Token

Defined in: packages/engine/src/lexer/Token.ts:10

Lexer token produced by ExpressionLexer and consumed by parsers.

Each token carries:

• type: string name (e.g. “NUMBER”, “PLUS”) for diagnostics and parselet lookup
• typeId: integer ID for O(1) comparison in parser hot paths
• value/text: the token’s semantic value and raw source text
• offset/line/col: source position for error messages and highlighting

Properties

col

col: number;

Defined in: packages/engine/src/lexer/Token.ts:21

---

line

line: number;

Defined in: packages/engine/src/lexer/Token.ts:20

---

lineBreaks

lineBreaks: number;

Defined in: packages/engine/src/lexer/Token.ts:19

---

offset

offset: number;

Defined in: packages/engine/src/lexer/Token.ts:18

---

sourceEnd?

optional sourceEnd?: number;

Defined in: packages/engine/src/lexer/Token.ts:35

End offset of the source text this token was fused from, exclusive.

Only set on tokens the normalizer built by fusing several lexer tokens into one, and only there because those are the tokens whose value no longer describes the text they came from. 10 frames fuses into a FRAME_COUNT whose value is 10, so offset + value.length would cover two characters of a nine-character phrase.

Evaluation never reads this. It exists so that a consumer painting a line (see LanguageService.getSemanticTokens) can highlight a fused phrase as the one thing the parser will see rather than as its parts.

---

text

text: string;

Defined in: packages/engine/src/lexer/Token.ts:17

---

type

type: string;

Defined in: packages/engine/src/lexer/Token.ts:12

String token type (e.g., “NUMBER”, “PLUS”, “IDENT”). Used by ParseletRegistry string-keyed maps and error messages.

---

typeId

typeId: number;

Defined in: packages/engine/src/lexer/Token.ts:15

Integer token type ID for fast comparison in Parser hot path. Populated by the Lexer via registerTokenType(). Use tokenTypeId(type) to get a type’s ID.

---

value

value: string;

Defined in: packages/engine/src/lexer/Token.ts:16