Token

Defined in: packages/engine/src/lexer/Token.ts:10

Lexer token produced by ExpressionLexer and consumed by parsers.

Each token carries:

• type: string name (e.g. “NUMBER”, “PLUS”) for diagnostics and parselet lookup
• typeId: integer ID for O(1) comparison in parser hot paths
• value/text: the token’s semantic value and raw source text
• offset/line/col: source position for error messages and highlighting

Properties

col

col: number;

Defined in: packages/engine/src/lexer/Token.ts:21

---

fault?

optional fault?: {
  code: string;
  message: string;
};

Defined in: packages/engine/src/lexer/Token.ts:50

Why a fused token stands for something the engine refuses to read, when it does.

Set by a normaliser rule that has decided a run is not what it is spelled like and will not be read as arithmetic either: a date literal naming no real day. The parselet for the fused type turns it into an Error value.

Carried here rather than packed into value, because value is what a parse error and the editor’s per-keystroke token API print, and a reader shown {"code":"...","message":"..."} is being shown the engine’s internals. value stays the text the reader typed.

code

readonly code: string;

message

readonly message: string;

---

line

line: number;

Defined in: packages/engine/src/lexer/Token.ts:20

---

lineBreaks

lineBreaks: number;

Defined in: packages/engine/src/lexer/Token.ts:19

---

mayNameVariable?

optional mayNameVariable?: boolean;

Defined in: packages/engine/src/lexer/Token.ts:63

On a rate denominator fused from a slash with nothing measured before it (100 / t, distance / t), that the word may be a variable’s name as well as a unit’s. A variable of that name, when one is defined, is divided by instead of read as the unit (#642). Set by the bare-rate rule; the parselet compiles the choice and the dependency graph records the read.

Also set on a line that is only sum or total (#742), which reads a variable of that name when the note defines one and totals the block above when it does not.

---

offset

offset: number;

Defined in: packages/engine/src/lexer/Token.ts:18

---

sourceEnd?

optional sourceEnd?: number;

Defined in: packages/engine/src/lexer/Token.ts:35

End offset of the source text this token was fused from, exclusive.

Only set on tokens the normalizer built by fusing several lexer tokens into one, and only there because those are the tokens whose value no longer describes the text they came from. 10 frames fuses into a FRAME_COUNT whose value is 10, so offset + value.length would cover two characters of a nine-character phrase.

Evaluation never reads this. It exists so that a consumer painting a line (see LanguageService.getSemanticTokens) can highlight a fused phrase as the one thing the parser will see rather than as its parts.

---

text

text: string;

Defined in: packages/engine/src/lexer/Token.ts:17

---

type

type: string;

Defined in: packages/engine/src/lexer/Token.ts:12

String token type (e.g., “NUMBER”, “PLUS”, “IDENT”). Used by ParseletRegistry string-keyed maps and error messages.

---

typeId

typeId: number;

Defined in: packages/engine/src/lexer/Token.ts:15

Integer token type ID for fast comparison in Parser hot path. Populated by the Lexer via registerTokenType(). Use tokenTypeId(type) to get a type’s ID.

---

value

value: string;

Defined in: packages/engine/src/lexer/Token.ts:16