3 Token Parsing🔗ℹ

The tokens used for grouping and indentation are distinct from other categories:

Other tokens are described by the grammar below, where a star (★) in the left column indicates the productions that correspond to terms or comments.

Numbers are supported directly in simple forms—decimal integers, decimal floating point, hexadecimal/octal/binary integers, and fractions—in all cases allowing _s between digits. A #{…} escape provides access to the full Racket S-expression number grammar. Special floating-point values use a # notation: #inf, #neginf, and #nan.

Boolean literals are #true and #false. The void value is #void.

Identifiers are formed from Unicode alphanumeric characters plus _ and emoji sequences, where the initial character must not be a numeric character (unless that numeric character starts an emoji sequence, as in 1 followed by U+FE0F and U+20E3). An identifier can also be prefixed with #%; such identifiers are intended for use for “internal” names that are not normally visible. An identifier prefixed with ~ (and without #%) forms a keyword, analogous to prefixing an identifier with #: in Racket.

Operators are formed from Unicode symbolic and punctuation characters other than the ones listed above as distinct tokens (plus a few more, like ", ', and single-character emoji sequences), but | or : is also allowed in an operator name as long as it is not by itself, and some # combinations like #' and #, are also operators. A multi-character operator cannot end in +, -, or . to avoid ambiguity in cases like 1+-2 (which is 1 plus -2, not 1 and 2 combined with a +- operator), unless the operator contains only +, -, or . (so ++, --, and ... are allowed). Also, a multi-character operator cannot end in :, since that creates an ambiguity with an operator just before a block, except that a sequence containing only : is allowed. A multi-character operator cannot end with / or contain // or /*, because that can create ambiguities with comments.

Implicit in the grammar is the usual convention of choosing the largest possible match at the start of a stream. Not reflected in the grammar is a set of delimiter requirements: numbers, #true, and #false must be followed by a delimiter. For example, 1x is a lexical error, because the x after 1 is not a delimiter. Non-alphanumeric characters other than _ are delimiters.

Certain ambiguities related to number and operator parsing are resolved by special rules. A number ends with a trailing . only if the . cannot be treated as the start of a multi-character operator; also, a . that is not part of a multi-character operator cannot appear after a number. The + and - characters as a number prefix versus an operator are also subject to a special rule: they are parsed as operators when immediately preceded by an alphanumeric character, _, ., ), ], or } with no whitespace in between. For example, 1+2 is 1 plus 2, but 1 +2 is 1 followed by the number +2.

When a #{…} escape describes an identifier S-expression, it is an identifier in the same sense as a shrubbery-notation identifier. The same holds for numbers, booleans, strings, byte strings, and keywords. A #{…} escape must not describe a pair, because pairs are used to represent a parsed shrubbery, and allowing pairs would create ambiguous or ill-formed representations.

For more details on @ parsing, see At-Notation Parsing, but the table below describes the shape of @ forms.