Characters
The text format assigns meaning to source text, which consists of a sequence of characters.
Characters are assumed to be represented as valid Unicode (Section 2.4) scalar values.
\[\begin{split}\begin{array}{llll}
\def\mathdef3372#1{{}}\mathdef3372{source} & \href{../text/lexical.html#text-source}{\mathtt{source}} &::=&
\href{../text/lexical.html#text-char}{\mathtt{char}}^\ast \\
\def\mathdef3372#1{{}}\mathdef3372{character} & \href{../text/lexical.html#text-char}{\mathtt{char}} &::=&
\def\mathdef3411#1{\mathrm{U{+}#1}}\mathdef3411{00} ~|~ \dots ~|~ \def\mathdef3412#1{\mathrm{U{+}#1}}\mathdef3412{D7FF} ~|~ \def\mathdef3413#1{\mathrm{U{+}#1}}\mathdef3413{E000} ~|~ \dots ~|~ \def\mathdef3414#1{\mathrm{U{+}#1}}\mathdef3414{10FFFF} \\
\end{array}\end{split}\]
Note
While source text may contain any Unicode character in comments or string literals,
the rest of the grammar is formed exclusively from the characters supported by the 7-bit ASCII subset of Unicode.
Tokens
The character stream in the source text is divided, from left to right, into a sequence of tokens, as defined by the following grammar.
\[\begin{split}\begin{array}{llll}
\def\mathdef3372#1{{}}\mathdef3372{token} & \href{../text/lexical.html#text-token}{\mathtt{token}} &::=&
\href{../text/lexical.html#text-keyword}{\mathtt{keyword}} ~|~ \href{../text/values.html#text-int}{\def\mathdef3393#1{{\mathtt{u}#1}}\mathdef3393{N}} ~|~ \href{../text/values.html#text-int}{\def\mathdef3399#1{{\mathtt{s}#1}}\mathdef3399{N}} ~|~ \href{../text/values.html#text-float}{\def\mathdef3407#1{{\mathtt{f}#1}}\mathdef3407{N}} ~|~ \href{../text/values.html#text-string}{\mathtt{string}} ~|~ \href{../text/values.html#text-id}{\mathtt{id}} ~|~
\def\mathdef3415#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3415{(} ~|~ \def\mathdef3416#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3416{)} ~|~ \href{../text/lexical.html#text-reserved}{\mathtt{reserved}} \\
\def\mathdef3372#1{{}}\mathdef3372{keyword} & \href{../text/lexical.html#text-keyword}{\mathtt{keyword}} &::=&
(\def\mathdef3417#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3417{a} ~|~ \dots ~|~ \def\mathdef3418#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3418{z})~\href{../text/values.html#text-idchar}{\mathtt{idchar}}^\ast
\qquad (\mathrel{\mbox{if}}~\mbox{occurring as a literal terminal in the grammar}) \\
\def\mathdef3372#1{{}}\mathdef3372{reserved} & \href{../text/lexical.html#text-reserved}{\mathtt{reserved}} &::=&
(\href{../text/values.html#text-idchar}{\mathtt{idchar}} ~|~ \href{../text/values.html#text-string}{\mathtt{string}})^+ \\
\end{array}\end{split}\]
Tokens are formed from the input character stream according to the longest match rule.
That is, the next token always consists of the longest possible sequence of characters that is recognized by the above lexical grammar.
Tokens can be separated by white space,
but except for strings, they cannot themselves contain whitespace.
Keyword tokens are defined either implicitly by an occurrence of a terminal symbol in literal form, such as \(\def\mathdef3419#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3419{keyword}\), in a syntactic production of this chapter, or explicitly where they arise in this chapter.
Any token that does not fall into any of the other categories is considered reserved, and cannot occur in source text.
Note
The effect of defining the set of reserved tokens is that all tokens must be separated by either parentheses, white space, or comments.
For example, \(\def\mathdef3420#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3420{0\$x}\) is a single reserved token, as is \(\def\mathdef3421#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3421{"a""b"}\).
Consequently, they are not recognized as two separate tokens \(\def\mathdef3422#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3422{0}\) and \(\def\mathdef3423#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3423{\$x}\), or \("a"\) and \("b"\), respectively, but instead disallowed.
This property of tokenization is not affected by the fact that the definition of reserved tokens overlaps with other token classes.
White Space
White space is any sequence of literal space characters, formatting characters, or comments.
The allowed formatting characters correspond to a subset of the ASCII format effectors, namely, horizontal tabulation (\(\def\mathdef3424#1{\mathrm{U{+}#1}}\mathdef3424{09}\)), line feed (\(\def\mathdef3425#1{\mathrm{U{+}#1}}\mathdef3425{0A}\)), and carriage return (\(\def\mathdef3426#1{\mathrm{U{+}#1}}\mathdef3426{0D}\)).
\[\begin{split}\begin{array}{llclll@{\qquad\qquad}l}
\def\mathdef3372#1{{}}\mathdef3372{white space} & \href{../text/lexical.html#text-space}{\mathtt{space}} &::=&
(\def\mathdef3427#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3427{~~} ~|~ \href{../text/lexical.html#text-format}{\mathtt{format}} ~|~ \href{../text/lexical.html#text-comment}{\mathtt{comment}})^\ast \\
\def\mathdef3372#1{{}}\mathdef3372{format} & \href{../text/lexical.html#text-format}{\mathtt{format}} &::=&
\href{../text/lexical.html#text-newline}{\mathtt{newline}} ~|~ \def\mathdef3428#1{\mathrm{U{+}#1}}\mathdef3428{09} \\
\def\mathdef3372#1{{}}\mathdef3372{newline} & \href{../text/lexical.html#text-newline}{\mathtt{newline}} &::=&
\def\mathdef3429#1{\mathrm{U{+}#1}}\mathdef3429{0A} ~|~ \def\mathdef3430#1{\mathrm{U{+}#1}}\mathdef3430{0D} ~|~ \def\mathdef3431#1{\mathrm{U{+}#1}}\mathdef3431{0D}~\def\mathdef3432#1{\mathrm{U{+}#1}}\mathdef3432{0A} \\
\end{array}\end{split}\]
The only relevance of white space is to separate tokens. It is otherwise ignored.
Comments¶
A comment can either be a line comment, started with a double semicolon \(\def\mathdef3392#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3392{{;}{;}}\) and extending to the end of the line, or a block comment, enclosed in delimiters \(\def\mathdef3390#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3390{{(}{;}} \dots \def\mathdef3391#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3391{{;}{)}}\). Block comments can be nested.
Here, the pseudo token \(\mathtt{eof}\) indicates the end of the input. The look-ahead restrictions on the productions for \(\href{../text/lexical.html#text-comment}{\mathtt{blockchar}}\) disambiguate the grammar such that only well-bracketed uses of block comment delimiters are allowed.
Note
Any formatting and control characters are allowed inside comments.