Completely migrate to prism

[tompng]

Closes #1024
Fixes #75 and #286

Remove remaining Ripper dependency and completely migrate to Prism.

RubyLex#should_continue?

Instead of checking Ripper's state bits, check for IGNORED_NEWLINE token.
When code is continued to the next line, there's probably a IGNORED_NEWLINE token.

1 + IGNORED_NEWLINE 2
receiver. IGNORED_NEWLINE method
receiver IGNORED_NEWLINE .method

Prism.lex("1 +\n").value
=> [[INTEGER(1,0)-(1,1)("1"), 2], [PLUS(1,2)-(1,3)("+"), 1], [IGNORED_NEWLINE(1,3)-(2,0)("\n"), 1], [EOF(1,3)-(2,0)("\n"), 1]]

Prism.lex("1.+ \n").value
=> [[INTEGER(1,0)-(1,1)("1"), 2], [DOT(1,1)-(1,2)("."), 256], [PLUS(1,2)-(1,3)("+"), 16], [NEWLINE(1,4)-(2,0)("\n"), 1], [EOF(1,4)-(2,0)("\n"), 1]]

RubyLex#check_termination_in_prev_line

When multiline code is pasted (without bracketed-paste mode), IRB checks if the last line is connected to the previous line.

tap do
end # Code looks ended here
.itself # But its' connected

Before: Heuristic, find DOT token
After: Compare two syntax trees: parse(lines.join) and parse(lines[...-1].join)

RubyLex#check_code_syntax

Checks if the code is :valid, :unrecoverable_error or :recoverable_error.
Same as before, use syntax error messages and regexp patterns.

Basic rules

• meets end of file end-of-input unterminated is likely to be recoverable error
• is not allowed unexpected or invalid is likely to be unrecoverable error
• invalid or unexpected error location is near last token position, it may not be unrecoverable error

Checked with many recoverable-error-code collected by

1. Collect valid short code snippets from ruby/ruby/**/*.rb
2. Randomly split code snippets into two, check if the first half is invalid code and join with "\n" becomes valid