Nesting analysis using Prism

[tompng]

#1024
Migrate nesting analysis from Ripper to Prism

Original nesting calculation with Ripper

Tokenize with ripper
Parse the tokens mainly focusing on open/close tokens
Collect open tokens per line

New nesting calculation with Prism

Traverse syntax tree
Check open_loc and closing_loc for a node that makes nesting, create a token-like object for it
Collect open token-like objects per line

Gemfile

Exclude prism-1.8.0 because test fails. (ruby/prism#3851)

[st0012]

Can we nudge Prism maintainers to cut v1.8.1 for the fix, and add prism as a dependency and require 1.3.0+?

[st0012]

Can we make this or parse_by_line take (code, local_variables: []) and call Prism.parse_lex here?
I feel Prism should be hidden from the callers of this class. Then we only need to require it here instead of all the places that'd use a nesting parser.

[tompng]

In a followup pull request #1160, parse_lex_result will be also used in RubyLex#should_continue?.
To avoid parsing the same code two times in dynamic_prompt calculation, argument needs to be parse_lex_result.

@context.io.dynamic_prompt do |lines|
  parse_lex_result = Prism.parse_lex(code, scopes: [@context.local_variables])
  line_results = IRB::NestingParser.parse_by_line(parse_lex_result)
  ...
  line_results.map.with_index do |...|
    # This part requires lex result
    continue = @scanner.should_continue?(tokens_until_line, line, line_num_offset + 1)
    ...
  end
end

Maybe we can find a better way to separate after migrating to Prism, I think.

[st0012]

Oh you did this in #1091 already. I guess we just need to wait for the release then

[st0012]

Should this be:

Suggested change

	@heredocs[line_index]&.sort_by { |_node, (_line, col)| col }&.reverse_each do |elem|
	@heredocs[line_index]&.sort_by { |elem| elem.pos[1] }&.reverse_each do |elem|

If so, let's add a test case that catches the failure? Something like:

    def test_multiple_heredocs_same_line_ordering
      code = <<~'RUBY'
        x = <<B + <<A
        B
        A
      RUBY
      line_results = parse_by_line(code)

      _prev, opens_line1, _min = line_results[0]
      assert_equal(['<<B', '<<A'], opens_line1.map(&:tok))
    end

[tompng]

Nice catch! It was sorting with nil, no error was raised but not correctly sorted.
I'll add a test case that needs sort <<~A if <<B (<<B appears before <<A in the syntax tree)

[st0012]

If I understand the code correctly, these "tokens" are actually nesting elements? Can we update these variable names to reflect that?

[tompng]

Yes, it's nesting elements. Updated 👍

[st0012]

Suggested change

	# Heredocs appears in the ordef B,A,D,C in syntax tree, but should be processed in A,B,C,D order.
	# Heredocs appears in the order of B,A,D,C in syntax tree, but should be processed in A,B,C,D order.

[tompng]

🙈 (thanks)