Lexical Format

Characters

The text format assigns meaning to source text, which consists of a sequence of characters. Characters are assumed to be represented as valid Unicode (Section 2.4) scalar values.

\[\begin{split}\begin{array}{llll} \def\mathdef3356#1{{}}\mathdef3356{source} & \href{../text/lexical.html#text-source}{\mathtt{source}} &::=& \href{../text/lexical.html#text-char}{\mathtt{char}}^\ast \\ \def\mathdef3356#1{{}}\mathdef3356{character} & \href{../text/lexical.html#text-char}{\mathtt{char}} &::=& \def\mathdef3395#1{\mathrm{U{+}#1}}\mathdef3395{00} ~|~ \dots ~|~ \def\mathdef3396#1{\mathrm{U{+}#1}}\mathdef3396{D7FF} ~|~ \def\mathdef3397#1{\mathrm{U{+}#1}}\mathdef3397{E000} ~|~ \dots ~|~ \def\mathdef3398#1{\mathrm{U{+}#1}}\mathdef3398{10FFFF} \\ \end{array}\end{split}\]

Note

While source text may contain any Unicode character in comments or string literals, the rest of the grammar is formed exclusively from the characters supported by the 7-bit ASCII subset of Unicode.

Tokens

The character stream in the source text is divided, from left to right, into a sequence of tokens, as defined by the following grammar.

\[\begin{split}\begin{array}{llll} \def\mathdef3356#1{{}}\mathdef3356{token} & \href{../text/lexical.html#text-token}{\mathtt{token}} &::=& \href{../text/lexical.html#text-keyword}{\mathtt{keyword}} ~|~ \href{../text/values.html#text-int}{\def\mathdef3377#1{{\mathtt{u}#1}}\mathdef3377{N}} ~|~ \href{../text/values.html#text-int}{\def\mathdef3383#1{{\mathtt{s}#1}}\mathdef3383{N}} ~|~ \href{../text/values.html#text-float}{\def\mathdef3391#1{{\mathtt{f}#1}}\mathdef3391{N}} ~|~ \href{../text/values.html#text-string}{\mathtt{string}} ~|~ \href{../text/values.html#text-id}{\mathtt{id}} ~|~ \def\mathdef3399#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3399{(} ~|~ \def\mathdef3400#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3400{)} ~|~ \href{../text/lexical.html#text-reserved}{\mathtt{reserved}} \\ \def\mathdef3356#1{{}}\mathdef3356{keyword} & \href{../text/lexical.html#text-keyword}{\mathtt{keyword}} &::=& (\def\mathdef3401#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3401{a} ~|~ \dots ~|~ \def\mathdef3402#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3402{z})~\href{../text/values.html#text-idchar}{\mathtt{idchar}}^\ast \qquad (\mathrel{\mbox{if}}~\mbox{occurring as a literal terminal in the grammar}) \\ \def\mathdef3356#1{{}}\mathdef3356{reserved} & \href{../text/lexical.html#text-reserved}{\mathtt{reserved}} &::=& (\href{../text/values.html#text-idchar}{\mathtt{idchar}} ~|~ \href{../text/values.html#text-string}{\mathtt{string}} ~|~ \def\mathdef3403#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3403{,} ~|~ \def\mathdef3404#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3404{;} ~|~ \def\mathdef3405#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3405{[} ~|~ \def\mathdef3406#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3406{]} ~|~ \def\mathdef3407#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3407{\{} ~|~ \def\mathdef3408#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3408{\}})^+ \\ \end{array}\end{split}\]

Tokens are formed from the input character stream according to the longest match rule. That is, the next token always consists of the longest possible sequence of characters that is recognized by the above lexical grammar. Tokens can be separated by white space, but except for strings, they cannot themselves contain whitespace.

Keyword tokens are defined either implicitly by an occurrence of a terminal symbol in literal form, such as \(\def\mathdef3409#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3409{keyword}\), in a syntactic production of this chapter, or explicitly where they arise in this chapter.

Any token that does not fall into any of the other categories is considered reserved, and cannot occur in source text.

Note

The effect of defining the set of reserved tokens is that all tokens must be separated by either parentheses, white space, or comments. For example, \(\def\mathdef3410#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3410{0\$x}\) is a single reserved token, as is \(\def\mathdef3411#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3411{"a""b"}\). Consequently, they are not recognized as two separate tokens \(\def\mathdef3412#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3412{0}\) and \(\def\mathdef3413#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3413{\$x}\), or \("a"\) and \("b"\), respectively, but instead disallowed. This property of tokenization is not affected by the fact that the definition of reserved tokens overlaps with other token classes.

White Space

White space is any sequence of literal space characters, formatting characters, comments, or annotations. The allowed formatting characters correspond to a subset of the ASCII format effectors, namely, horizontal tabulation (\(\def\mathdef3414#1{\mathrm{U{+}#1}}\mathdef3414{09}\)), line feed (\(\def\mathdef3415#1{\mathrm{U{+}#1}}\mathdef3415{0A}\)), and carriage return (\(\def\mathdef3416#1{\mathrm{U{+}#1}}\mathdef3416{0D}\)).

\[\begin{split}\begin{array}{llclll@{\qquad\qquad}l} \def\mathdef3356#1{{}}\mathdef3356{white space} & \href{../text/lexical.html#text-space}{\mathtt{space}} &::=& (\def\mathdef3417#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3417{~~} ~|~ \href{../text/lexical.html#text-format}{\mathtt{format}} ~|~ \href{../text/lexical.html#text-comment}{\mathtt{comment}})^\ast \\ \def\mathdef3356#1{{}}\mathdef3356{format} & \href{../text/lexical.html#text-format}{\mathtt{format}} &::=& \href{../text/lexical.html#text-newline}{\mathtt{newline}} ~|~ \def\mathdef3418#1{\mathrm{U{+}#1}}\mathdef3418{09} \\ \def\mathdef3356#1{{}}\mathdef3356{newline} & \href{../text/lexical.html#text-newline}{\mathtt{newline}} &::=& \def\mathdef3419#1{\mathrm{U{+}#1}}\mathdef3419{0A} ~|~ \def\mathdef3420#1{\mathrm{U{+}#1}}\mathdef3420{0D} ~|~ \def\mathdef3421#1{\mathrm{U{+}#1}}\mathdef3421{0D}~\def\mathdef3422#1{\mathrm{U{+}#1}}\mathdef3422{0A} \\ \end{array}\end{split}\]

The only relevance of white space is to separate tokens. It is otherwise ignored.

Comments

A comment can either be a line comment, started with a double semicolon \(\def\mathdef3376#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3376{{;}{;}}\) and extending to the end of the line, or a block comment, enclosed in delimiters \(\def\mathdef3374#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3374{{(}{;}} \dots \def\mathdef3375#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3375{{;}{)}}\). Block comments can be nested.

\[\begin{split}\begin{array}{llclll@{\qquad\qquad}l} \def\mathdef3356#1{{}}\mathdef3356{comment} & \href{../text/lexical.html#text-comment}{\mathtt{comment}} &::=& \href{../text/lexical.html#text-comment}{\mathtt{linecomment}} ~|~ \href{../text/lexical.html#text-comment}{\mathtt{blockcomment}} \\ \def\mathdef3356#1{{}}\mathdef3356{line comment} & \href{../text/lexical.html#text-comment}{\mathtt{linecomment}} &::=& \def\mathdef3376#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3376{{;}{;}}~~\href{../text/lexical.html#text-comment}{\mathtt{linechar}}^\ast~~(\href{../text/lexical.html#text-newline}{\mathtt{newline}} ~|~ \mathtt{eof}) \\ \def\mathdef3356#1{{}}\mathdef3356{line character} & \href{../text/lexical.html#text-comment}{\mathtt{linechar}} &::=& c{:}\href{../text/lexical.html#text-char}{\mathtt{char}} & (\mathrel{\mbox{if}} c \neq \def\mathdef3423#1{\mathrm{U{+}#1}}\mathdef3423{0A} \land c \neq \def\mathdef3424#1{\mathrm{U{+}#1}}\mathdef3424{0D}) \\ \def\mathdef3356#1{{}}\mathdef3356{block comment} & \href{../text/lexical.html#text-comment}{\mathtt{blockcomment}} &::=& \def\mathdef3374#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3374{{(}{;}}~~\href{../text/lexical.html#text-comment}{\mathtt{blockchar}}^\ast~~\def\mathdef3375#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3375{{;}{)}} \\ \def\mathdef3356#1{{}}\mathdef3356{block character} & \href{../text/lexical.html#text-comment}{\mathtt{blockchar}} &::=& c{:}\href{../text/lexical.html#text-char}{\mathtt{char}} & (\mathrel{\mbox{if}} c \neq \def\mathdef3425#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3425{;} \land c \neq \def\mathdef3426#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3426{(}) \\ &&|& \def\mathdef3427#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3427{;} & (\mathrel{\mbox{if}}~\mbox{the next character is not}~\def\mathdef3428#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3428{)}) \\ &&|& \def\mathdef3429#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3429{(} & (\mathrel{\mbox{if}}~\mbox{the next character is not}~\def\mathdef3430#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3430{;}) \\ &&|& \href{../text/lexical.html#text-comment}{\mathtt{blockcomment}} \\ \end{array}\end{split}\]

Here, the pseudo token \(\mathtt{eof}\) indicates the end of the input. The look-ahead restrictions on the productions for \(\href{../text/lexical.html#text-comment}{\mathtt{blockchar}}\) disambiguate the grammar such that only well-bracketed uses of block comment delimiters are allowed.

Note

Any formatting and control characters are allowed inside comments.

Annotations

An annotation is a bracketed token sequence headed by an annotation id of the form \(\def\mathdef3431#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3431{@id}\) or \(\def\mathdef3432#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3432{@"..."}\). No space is allowed between the opening parenthesis and this id. Annotations are intended to be used for third-party extensions; they can appear anywhere in a program but are ignored by the WebAssembly semantics itself, which treats them as white space.

Annotations can contain other parenthesized token sequences (including nested annotations), as long as they are well-nested. String literals and comments occurring in an annotation must also be properly nested and closed.

\[\begin{split}\begin{array}{llclll@{\qquad\qquad}l} \def\mathdef3356#1{{}}\mathdef3356{annotation} & \href{../text/lexical.html#text-annot}{\mathtt{annot}} &::=& \def\mathdef3433#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3433{(@}~\href{../text/lexical.html#text-annot}{\mathtt{annotid}} ~(\href{../text/lexical.html#text-space}{\mathtt{space}} ~|~ \href{../text/lexical.html#text-token}{\mathtt{token}})^\ast~\def\mathdef3434#1{\mbox{‘}\mathtt{#1}\mbox{’}}\mathdef3434{)} \\ \def\mathdef3356#1{{}}\mathdef3356{annotation identifier} & \href{../text/lexical.html#text-annot}{\mathtt{annotid}} &::=& \href{../text/values.html#text-idchar}{\mathtt{idchar}}^+ ~|~ \href{../text/values.html#text-name}{\mathtt{name}} \\ \end{array}\end{split}\]

Note

The annotation id is meant to be an identifier categorising the extension, and plays a role similar to the name of a custom section. By convention, annotations corresponding to a custom section should use the custom section’s name as an id.

Implementations are expected to ignore annotations with ids that they do not recognize. On the other hand, they may impose restrictions on annotations that they do recognize, e.g., requiring a specific structure by superimposing a more concrete grammar. It is up to an implementation how it deals with errors in such annotations.