5.2 Source Locations and Raw-Text Properties🔗ℹ

The result of parse-all records source locations in the syntax object representing a shrubbery form. Source-location information and other properties for a shrubbery (), [], {}, or '' is associated with the parens, brackets, braces, or quotes identifier in the representation. Similarly, source-location information and properties for a : block or | alternatives are recorded on a block or alts identifier. For all compound forms, including group and multi, the source location on the representation’s head identifier spans the compound’s form. For an operator, source-location information and properties are associated to the operator identifier, and not the wrapper op identifier. Each structuring identifier like group, block, parens, or op has the 'identifier-as-keyword syntax property as #t.

The result of parse-all has source locations copied to the S-expression “parentheses” of a syntax object representing a compound form, but only as a convenience; the intent is that information is more permanently and reliably associated to each compound form’s head identifier. Along similar lines, spanning source locations for compound forms tend not to be maintained as syntax objects are manipulated, and the intent is that spanning locations are reconstructed as needed, especially for group and multi forms. A useful convention may be to treat source locations the S-expression “parentheses” of a compound form as a cache for spanning information computed from the form’s content.

A syntax object produced by parse-all includes raw text properties that allow the original shrubbery text to be reconstructed from the syntax object. This raw text information is distributed among syntax objects in a way that is preserved to a useful degree on terms as they are rearranged by enforestation and macro expansion. The shrubbery-syntax->string function reconstructs source text based on raw text properties in a syntax object. The shrubbery/property module exports functions to access and update properties, which can be helpful to avoid typos that are all too easy when writing the key directly as a quoted symbol.

Raw-text property values are trees of strings: a string, an empty list, or a pair containing two trees. The source raw text is reconstructed through a preorder traversal of the tree. The parse-all function attaches raw text properties only to atom terms and the head identifiers of compound terms, not counting multi. A head op, multi, or parsed is normally not consulted for syntax properties, but parse-all associates empty raw text to a head op or multi. For a parsed representation, the third component of the parsed list is consulted for 'opaque-raw by functions like shrubbery-syntax->string.

The main raw text property key is 'raw, but the following list of all raw text keys is in the order that they contribute to reconstructed text:

• 'raw-prefix (use syntax-raw-prefix-property): records original text for whitespace and comments before a term or group. The parse-all function uses this property only when raw text cannot instead be associated with a preceding term or group as a suffix. When raw text must be associated as a prefix but could be attached to either a term or its enclosing group (because the term is first within its group), parse-all attaches the prefix to the group.

• 'raw-inner-prefix (use syntax-raw-inner-prefix-property): like 'raw-prefix, and after 'raw-prefix and still before 'raw, but with the intent that the raw text sticks to its term, instead of being shifted to a preceding term or enclosing group. The parse-all function uses this property only to record a @ that appears before a term for At-Notation Using @.

• 'raw (use syntax-raw-property): records the original text of an atomic term or the opening text for a compound term. For example, the input 0x11 will be parsed as the number 17 with a 'raw property value "0x11". The 'raw property for a parens representation is normally "(", while 'raw for a group representation is normally empty. The parse-all function also associated an empty 'raw on the op identifier of an operator representation, although that identifier is not normally consulted for properties (e.g., by shrubbery-syntax->string).

• 'raw-opaque-content (use syntax-raw-opaque-content-property): records raw text to use in place of the content of compound form. When the compound form also has the 'raw property, 'raw-opaque-content is shown after 'raw. A non-compound form can have 'raw-opaque-content, and it is combined with 'raw in that case, too. Note that the 'opaque-raw is a different property.

• 'opaque-raw (use syntax-opaque-raw-property): raw text that supersedes 'raw and 'raw-opaque-content. Furthermore, this property is recognized when present on the S-expression “parentheses” of a compound form or an op form, in which case properties on the leading identifier and/or compound form’s content are ignored. Although this property is intended for use on S-expression “parentheses” in a shrubbery representation, it is also recognized on atom terms. When 'opaque-raw is recognized on S-expression “parentheses”, then 'raw-prefix, 'raw-inner-prefix, 'raw-inner-suffix, and 'raw-suffix are also recognized (and those would otherwise be ignored).

• 'raw-tail (use syntax-raw-tail-property): records original text to appear after a term’s content. This property is intended for use with compound terms to hold the compound form’s closer. For example, 'raw-tail property for a parens representation is normally ")".

• 'raw-inner-suffix (use syntax-raw-inner-suffix-property): analogous to 'raw-inner-prefix, but for a suffix that should stick with its term. The parse-all function currently does not use this property.

• 'raw-suffix (use syntax-raw-suffix-property): records original text for whitespace and comments after a term or group. When a term is last in its group, the parse-all function uses 'raw-suffix on the group instead of the term.

Each of these properties is normally preserved (in the sense of a true fourth argument to syntax-property), except for 'opaque-raw, which is intended for use in intermediate, short-term mixtures of shrubbery forms and S-expressions.

For example, parse-all will parse the input 0x11 as the number 17 with a 'raw property value "0x11". The input "\u3BB" will be parsed as the string "λ" with a 'raw property value "\"\\u3BB\"".

For example, parse-all will parse the input  1 + 2 // done into the S-expression representation (multi (group 1 (op +) 2)). The syntax object for group will have a 'raw-prefix value equivalent to " " and a 'raw-suffix value equivalent to " // done", but possibly within a tree structure instead of a single string. The syntax object for 1 and will have a 'raw-suffix value equivalent to " ", while the syntax object for + and will have a 'raw-suffix value equivalent to "  " (i.e., two spaces).

The parse-all function will parse the input @x into an S-expression representation x with a 'raw-inner-prefix property "@". The parse-all function never produces a syntax object with 'raw-inner-suffix.

For example, the input (1 + 2) * 4 //done will be parsed into the S-expression representation (multi (group (parens (group 1 (op +) 2)) (op *) 4)). The syntax object for the outer group will have a 'raw-suffix value equivalent to " // done". The syntax object for parens will have a 'raw value equivalent to "(", a 'raw-tail value equivalent to ")", and a 'raw-suffix value equivalent to " ". The inner group syntax object will have no properties or ones with values that are equivalent to empty strings.

The 'opaque-raw property is useful in macro expansion to record a macro’s input to its output—not only in source location, but also in source text.