Consider using rustc_lexer

[CAD97]

rustc_lexer

rustc_lexer is now the lexer used by rustc and is on crates for reuse. rust-analyzer is already using it. It would make sense for proc-macro2 to also use rustc_lexer and only have to do bracket matching and trivia stripping.

I spent a little time trying to do so, but I'm not familiar enough with the current fallback parser to do so. I can attempt just ripping everything out and building from scratch on rustc_lexer, but I'd prefer to have a go-ahead before putting time into that.

[mystor]

That's good to know! This would probably help fix #126 and related issues as well.

FWIW you probably wouldn't need to rip out the entire fallback parser as it exists today. The only entry-point to it (if I remember correctly), which is the FromStr impl for TokenStream.

This code then goes on to lex the entire input string & generate a TokenStream out of it. With the new API, you'd iterate over the tokenize output and generate the corresponding TokenTree instances. Once it's gone you can drop the strnom module and about ~600 lines of fallback.

[CAD97]

Unfortunate news: rustc_lexer uses std::iter::from_fn, which was stabilized in 1.34.0, which is later than the current MSRV of 1.31.0.