Lexical Format¶
Characters¶
The text format assigns meaning to source text, which consists of a sequence of characters. Characters are assumed to be represented as valid Unicode (Section 2.4) scalar values.
Tokens¶
The character stream in the source text is divided, from left to right, into a sequence of tokens, as defined by the following grammar.
Tokens are formed from the input character stream according to the longest match rule. That is, the next token always consists of the longest possible sequence of characters that is recognized by the above lexical grammar. Tokens can be separated by white space, but except for strings, they cannot themselves contain whitespace.
The set of keyword tokens is defined implicitly, by all occurrences of a terminal symbol in literal form, such as \(\def\mathdef1971#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef1971{keyword}\), in a syntactic production of this chapter.
Any token that does not fall into any of the other categories is considered reserved, and cannot occur in source text.
Note
The effect of defining the set of reserved tokens is that all tokens must be separated by either parentheses or white space. For example, \(\def\mathdef1972#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef1972{0\$x}\) is a single reserved token. Consequently, it is not recognized as two separate tokens \(\def\mathdef1973#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef1973{0}\) and \(\def\mathdef1974#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef1974{\$x}\), but instead disallowed. This property of tokenization is not affected by the fact that the definition of reserved tokens overlaps with other token classes.
White Space¶
White space is any sequence of literal space characters, formatting characters, or comments. The allowed formatting characters correspond to a subset of the ASCII format effectors, namely, horizontal tabulation (\(\def\mathdef1975#1{\mathrm{U{+}#1}}\mathdef1975{09}\)), line feed (\(\def\mathdef1976#1{\mathrm{U{+}#1}}\mathdef1976{0A}\)), and carriage return (\(\def\mathdef1977#1{\mathrm{U{+}#1}}\mathdef1977{0D}\)).
The only relevance of white space is to separate tokens. It is otherwise ignored.
Comments¶
A comment can either be a line comment, started with a double semicolon \(\def\mathdef1946#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef1946{{;}{;}}\) and extending to the end of the line, or a block comment, enclosed in delimiters \(\def\mathdef1944#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef1944{{(}{;}} \dots \def\mathdef1945#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef1945{{;}{)}}\). Block comments can be nested.
Here, the pseudo token \(\mathtt{eof}\) indicates the end of the input. The look-ahead restrictions on the productions for \(\href{../text/lexical.html#text-comment}{\mathtt{blockchar}}\) disambiguate the grammar such that only well-bracketed uses of block comment delimiters are allowed.
Note
Any formatting and control characters are allowed inside comments.