Symbol with keyword value is lexed as keyword

[Earlopain]

In syntax_suggest, it lexes the code and counts various keywords. If it encounters a symbol like :module, it looks at state bits to decide if the token really is a keyword or not. ruby/syntax_suggest@10232e4

For the rewrite to native prism I don't want to rely on state at all. But I noticed that prism also lexes in the following way:

pp Prism.lex(":next").value
=> [[SYMBOL_BEGIN(1,0)-(1,1)(":"), 128], [KEYWORD_NEXT(1,1)-(1,5)("next"), 8], [EOF(1,5)-(1,5)(""), 2]]

# And similarly etc.
pp Prism.lex(":@foo").value
=> [[SYMBOL_BEGIN(1,0)-(1,1)(":"), 128], [INSTANCE_VARIABLE(1,1)-(1,5)("@foo"), 8], [EOF(1,5)-(1,5)(""), 2]]

Basically it happens in all the places that handle PM_CASE_KEYWORD (at least). Prism has the context, would it make sense to emit these as IDENTIFIER instead?

[kddnewton]

I really don't want to rely on lexing at all, if I'm honest. But if we are going to rely on it, I'd rather it match the output from ripper (at least for now) because it makes matching and testing way easier. I'm not ready to break with that until such a time as we have confidence we can reliably reproduce all state changes from CRuby (which I don't believe, specifically because of #3558).

[Earlopain]

syntax_suggest is not really possible without lexing. end on its own for example has zero usable information in the AST. I use ast to detect endless method definitions, heredocs and "continuable" code like

foo.
  bar

but have to use tokens for a few other things.

:module is actually the last test failure I currently have. I'll work around it in some way but it would be nice if it does the more obvious thing in the future.

---

Re #3558

It would be nice if you could tackle that some time. I tried, but didn't understand where the problem is. I think this is actually the only reported issue in the parser with real-world impact

[kddnewton]

Yeah, I agree I'd like to break from ripper on this, and hopefully soon. But I think we need to do more eco-system migration away from ripper first.

On 3558, yeah I can take a look. I tried a while ago and it was a bit difficult, but I'll keep going on it.