1.2 Compiling languages🔗ℹ

1.2.1 Compiling references to DSL bindings within Racket code🔗ℹ

Compilation in the Basic Tutorial: State Machine Language introduces the use of reference compilers.

By default, Racket code cannot reference names bound with DSL binding classes. To allow such references, specify a reference compiler for each class of bindings that should be usable in Racket code. The reference compiler is a syntax transformer that will be applied to compile the syntax including each reference.

When a reference appears in the head of a form, such as x in (x 1 2), the reference compiler receives the entire form to transform. If the reference appears in a set!, the reference compiler will be invoked if it is a set!-transformer?; otherwise the expansion of the set! form results in an error.

In all cases, the reference identifier in the syntax provided to the reference compiler is a compiled identifier.

DSL compilers specify the reference compilers to use by emitting compiled code containing with-reference-compiler forms.

syntax (with-reference-compilers     ([binding-class-id reference-compiler-expr] ...)   body ...+)

Declares the reference compilers to use when expanding DSL-bound identifiers of the specified binding classes when expanding body. Evaluates to body.

procedure (make-variable-like-reference-compiler   reference-stx      [ setter-stx])

→ set!-transformer?

reference-stx : (or/c syntax? (-> identifier? syntax?))

setter-stx : (or/c syntax? (-> syntax? syntax?)) = #f

Like make-variable-like-transformer, but works properly as a reference compiler that receives compiled identifiers.

If reference-stx is syntax, references expand to that syntax. If reference-stx is a procedure, it is called with the reference identifier to produce the reference’s expansion.

If setter-stx is syntax, it should be syntax that evaluates to a procedure. The procedure will be invoked with the new value for the variable.

If setter-stx is a procedure, it is called with the entire set! expression to produce its expansion.

If setter-stx is not provided, references within set! position raise a syntax error.

When a reference is used in the head position of a form such as (x 1 2), the variable-like reference compiler ensures that the form is parsed as a function application (not a macro call) and uses the reference-stx to expand only the identifier in head position.

Here is an example for a match DSL where pattern-bound variables cannot be mutated:

(syntax-spec

(host-interface/expression

(match target:racket-expr c:clause ...)

#'(with-reference-compilers ([pat-var (make-variable-like-reference-compiler (lambda (id) id))])

(let ([target-pv target])

(match-clauses target-pv c ...)))))

Alternately we could provide immutable-reference-compiler as the reference compiler, which behaves exactly the same.

value immutable-reference-compiler : set!-transformer?

A variable-like reference compiler that allows references but raises a syntax error when identifiers are used in set! expressions.

References expand to their compiled identifier.

value mutable-reference-compiler : set!-transformer?

A variable-like reference compiler that allows references as well as mutations via set! expressions.

References expand to their compiled identifier.

1.2.2 Compiled identifiers vs surface syntax🔗ℹ

The syntax of a DSL program in its initial, un-expanded, un-compiled state is called the surface syntax. During the expansion of a host interface usage, before your compiler is invoked, syntax-spec renames and compiles surface identifiers. The resulting identifiers are called compiled identifiers and have unique names and have special scopes according to your DSL’s binding rules.

Another concept that comes up when discussing identifiers and compilation is positive vs negative space. This has to do with macro-introduction scopes. To ensure macro hygiene, the Racket expander distinguishes between syntax that was introduced by a macro and syntax that originated from elsewhere. To do this, it adds an introduction scope to the macro invocation’s syntax, expands the invocation by running the macro’s transformer, and then flips the scope, removing it from syntax that has it and adding it to syntax that doesn’t.

During the expansion of the invocation, when a macro’s transformer is running, the macro transformer will see this introduction scope on the incoming syntax. This syntax is in negative space. After the scope is flipped off on the result, the syntax is in positive space. It’s important to note that positive vs negative space depends on the current introduction scope, as there may be many introduction scopes floating around. It is also possible to manually flip this scope in a transformer to convert syntax between positive and negative space.

1.2.3 Symbol collections🔗ℹ

Symbol collections allow compilers to track information related to dsl variables. Symbol collections expect to receive compiled identifiers in negative space.

1.2.3.1 Symbol tables🔗ℹ

procedure (symbol-table? v) → boolean?

v : any/c

Returns #t if v is a symbol table, #f otherwise.

procedure (in-symbol-table table)

→ (sequence/c identifier? (or/c syntax-datum? syntax?))

table : symbol-table?

Like in-free-id-table.

procedure (mutable-symbol-table? v) → boolean?

v : any/c

Returns #t if v is a mutable symbol table, #f otherwise.

syntax (define-persistent-symbol-table id)

Defines a (mutable) symbol table for global use. For example, if your DSL has a static type checker and you’re requiring typed identifiers between modules, you can store each identifier’s type in a persistent symbol table.

Can only be used at the top-level of a module.

procedure (local-symbol-table) → mutable-symbol-table?

Creates a (mutable) symbol table for local use.

procedure (syntax-datum? v) → boolean?

v : any/c

Roughly, returns #t if v is something that could be the result of syntax->datum, #f otherwise.

This includes pairs, vectors, symbols, numbers, booleans, etc.

procedure (symbol-table-set!   table             id             v            [ #:allow-overwrite? allow-overwrite?])   →   void?

table : mutable-symbol-table?

id : identifier?

v : (or/c syntax? syntax-datum?)

allow-overwrite? : any/c = #t

Like free-id-table-set!. Errors by default when setting the value of an identifier already present in the table. Pass #:allow-overwrite? #t to allow this. However, persistent symbol tables do not support this flag.

procedure (symbol-table-ref table id failure) → any/c

table : symbol-table?

id : identifier?

failure : any/c

Like free-id-table-ref

procedure (symbol-table-has-key? table id) → boolean?

table : symbol-table?

id : identifier?

Returns #t if table has an entry for id, #f otherwise.

procedure (immutable-symbol-table? v) → boolean?

v : any/c

Returns #t if v is an immutable symbol table, #f otherwise.

procedure (immutable-symbol-table) → immutable-symbol-table?

Creates an immutable, local symbol table. There are no persistent immutable symbol tables.

procedure (symbol-table-set   table       id       v      [ #:allow-overwrite? allow-overwrite?])

→ immutable-symbol-table?

table : immutable-symbol-table?

id : identifier?

v : (or/c syntax? syntax-datum?)

allow-overwrite? : any/c = #t

like free-id-table-set. Errors by default when setting the value of an identifier already present in the table. Pass #:allow-overwrite? #t to allow this.

procedure (symbol-table-remove table id) → immutable-symbol-table?

table : immutable-symbol-table?

id : identifier?

like free-id-table-remove

1.2.3.2 Symbol sets🔗ℹ

procedure (symbol-set? v) → boolean?

v : any/c

Returns #t if v is a symbol set, #f otherwise.

procedure (in-symbol-set table) → (sequence/c identifier?)

table : symbol-set?

Like in-free-id-set.

procedure (mutable-symbol-set? v) → boolean?

v : any/c

Returns #t if v is a mutable symbol set, #f otherwise.

syntax (define-persistent-symbol-set id)

Defines a (mutable) symbol set for global use like define-persistent-symbol-table.

procedure (local-symbol-set id ...) → mutable-symbol-set?

id : identifier?

Creates a local (mutable) symbol set containing the given identifiers.

procedure (symbol-set-add! s id) → void?

s : mutable-symbol-set?

id : identifier?

Like free-id-set-add!

procedure (symbol-set-member? s id) → boolean?

s : mutable-symbol-set?

id : identifier?

Like free-id-set-member?

procedure (immutable-symbol-set? v) → boolean?

v : any/c

Returns #t if v is an immutable symbol set, #f otherwise.

procedure (immutable-symbol-set id ...) → immutable-symbol-set?

id : identifier?

Creates a (local) immutable symbol set containing the given identifiers. There are no persistent immutable symbol sets.

procedure (symbol-set-add s id) → immutable-symbol-set?

s : immutable-symbol-set?

id : identifier?

Like free-id-set-add

procedure (symbol-set-remove s id) → immutable-symbol-set?

s : immutable-symbol-set?

id : identifier?

Like free-id-set-remove

procedure (symbol-set-union s ...) → immutable-symbol-set?

s : immutable-symbol-set?

Like free-id-set-union

procedure (symbol-set-intersect s0 s ...) → immutable-symbol-set?

s0 : immutable-symbol-set?

s : immutable-symbol-set?

Like free-id-set-intersect.

procedure (symbol-set-subtract s0 s ...) → immutable-symbol-set?

s0 : immutable-symbol-set?

s : immutable-symbol-set?

Like free-id-set-subtract.

1.2.4 Binding Operations🔗ℹ

procedure (compiled-identifier=? a-id b-id) → boolean?

a-id : identifier?

b-id : identifier?

Returns #t if the two compiled DSL identifiers correspond to the same binding, returns #f otherwise. Similar to free-identifier=?.

This is the equality used by symbol tables.

procedure (free-identifiers   stx      [ #:allow-host? allow-host?])

→ (listof identifier?)

stx : syntax?

allow-host? : boolean? = #f

Get a DSL expression’s free identifiers (deduplicated).

Host expressions currently are not supported.

procedure (alpha-equivalent?   stx-a             stx-b            [ #:allow-host? allow-host?])   →   boolean?

stx-a : syntax?

stx-b : syntax?

allow-host? : boolean? = #f

Returns #t if the two DSL expressions are alpha-equivalent, #f otherwise.

Host expressions currently are not supported.

syntax (get-racket-referenced-identifiers [binding-class-id ...] expr)

Returns an immutable symbol set containing identifiers of the specified binding classes that were referenced in racket (host) expressions in expr.

1.2.5 Expansion🔗ℹ

syntax (nonterminal-expander nonterminal-id)

nonterminal-id   :   identifier?

Produces an expander procedure for the specified nonterminal. This procedure expands macros down to the DSL’s core forms, binds identifiers in binding positions, and can be configured to compile and rename identifiers. It does not expand host expressions.

Expander procedure has contract (->* (syntax?) (#:should-rename? boolean?) syntax?). The default behavior is not to re-compile and re-rename identifiers. To do this, pass in #:should-rename? #t.

Can only be used with simple non-terminals.