Processing shebang and byte order mark

[mattheww]

Shebang

§18.1 (fls_s35hob3i7lr) provides the following syntax definitions:

SourceFile ::=
    ZeroWidthNoBreakSpace?
    Shebang?
    InnerAttributeOrDoc*
    Item*

Shebang ::=
    #! ~[NewLine]*

NewLine ::=
    \n

This grammar is ambiguous: a file starting with the following could be treated as starting with either a Shebang or an InnerAttributeOrDoc:

#![xxx]

This is an unusual case where the ambiguity isn't resolved "eagerly": rustc sees this as InnerAttributeOrDoc. So I think the FLS ought to say more here.

Also, I think putting a nonterminal inside ~[] isn't strictly following the notation described by §1.1.4:14 (fls_blvsfqeevosr). The definition of NewLine isn't used anywhere else.

Byte order mark

§18.1:2 (fls_ann3cha1xpek) reads

A Shebang does not have an effect on the compilation.

I suggest adding a similar statement for the ZeroWidthNoBreakSpace.

[mattheww]

Defining ZeroWidthNoBreakSpace and Shebang in chapter 18 rather than in chapter 2 (Lexical elements) seems a bit odd to me.

When the FLS was written the Reference was using a similar approach; it was changed in rust-lang/reference#1459 to treat dealing with these things as a separate pass before lexing proper.

I think the approach the FLS is using sits poorly with the first sentence of §2.2:1 (fls_d4nvxsvxj537):

The text of a source file is a sequence of separate lexical elements. The meaning of a program depends only on the particular sequence of lexical elements, excluding non-doc comments.

as we end up with some things in the source file which aren't LexicalElements. On the other hand the shebang and BOM have no effect on the program, so the second sentence is still true.