ExpressionLexer

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:422

Character-by-character tokenizer for expression text.

Scans a raw line/expression string into a stream of typed tokens (numbers, identifiers, operators, units, keywords, …), handling markdown-line classification (classifyLine), inline s`...` solve spans, and package-contributed vocabulary (registered via registerVocabulary/unregisterVocabulary, keywords operators, and units a package wants recognized as their own token types rather than falling through to generic identifiers).

Most consumers should use the higher-level Lexer wrapper, which adds streaming next()/peek() access over this class’s scan results.

Constructors

Constructor

new ExpressionLexer(localeCode?, lookup?): ExpressionLexer;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:512

Parameters

Parameter	Type	Default value
localeCode	string	'en'
lookup?	TokenLookup	undefined

Returns

ExpressionLexer

Properties

_inlineSolveSpans

_inlineSolveSpans: InlineSolveSpan[] = [];

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:469

Inline solve spans collected during the most recent tokenization pass. Populated by Symbol.iterator and consumed by scanDocument().

Methods

[iterator]()

iterator: Generator<Token, void, undefined>;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:879

Lazy token-by-token generator. Yields each token without allocating an intermediate Token[] array. Supports for…of and spread usage.

Usage: for (const t of lexer) { … } // lazy, no array allocation const tokens = […lexer]; // materializes via spread const tokens = lexer.tokenizeAll(); // materializes via Array.from()

IMPORTANT: This generator captures this.len ONCE at creation time (const len = this.len). scanDocument() relies on this behavior to scope tokenization to a single line by temporarily restricting this.len to the line end position before creating the iterator. Do NOT refactor to re-read this.len mid-loop without also updating scanDocument().

Returns

Generator<Token, void, undefined>

---

classifyLine()

classifyLine(lineText): LineClassification;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:1831

Classify a single line of markdown text.

Parameters

Parameter	Type
lineText	string

Returns

LineClassification

---

findInlineSolves()

findInlineSolves(lineText): InlineSolveSpan[];

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:1859

Find all inline solve markers in a line with precise coordinate mapping.

Parameters

Parameter	Type
lineText	string

Returns

InlineSolveSpan[]

---

getKeywords()

getKeywords(): Record<string, string>;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:1852

Every keyword this lexer currently recognizes, locale keywords (pi, sqrt, convert, …) merged with any plugin-contributed ones from registerVocabulary() (e.g. a package’s custom keywords), mapped to the token type they lex to. A snapshot copy, not a live reference mutating the return value has no effect on the lexer.

Returns

Record<string, string>

---

registerVocabulary()

registerVocabulary(plugin): void;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:538

Register a plugin to extend the lexer with custom tokens.

All registrations are additive, built-in patterns still work. Keywords, operators, and units from the plugin are merged with existing ones. Calling multiple times adds more entries.

Note: multi-word phrases are now handled by the TokenNormalizer (see IEnginePackage.normalizerRules), not the lexer.

Built-in tokens CANNOT be overridden. Throws a EngineError if the plugin attempts to register a keyword, operator, or unit that conflicts with a built-in one.

Parameters

Parameter	Type
plugin	LexerVocabulary

Returns

void

---

reset()

reset(input): void;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:683

Parameters

Parameter	Type
input	string

Returns

void

---

scanDocument()

scanDocument(text): ScanLineResult[];

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:717

Scan a full document text in a single pass, classifying each line and tokenizing non-skipped lines.

Replaces the separate classifyLine() + findInlineSolves() + per-line reset() + tokenizeAll() pattern with a single character-by-character walk through the entire document. Key benefits:

• Single reset(): this.pos, this.len, this.line, and this.lineStartPos are set once for the whole document, not per-line.
• Single classification: classifyLine() runs once per line inline; skipped lines are jumped over without tokenization.
• Shared tokenization: Non-skipped lines are tokenized using the existing state machine, yielding Token[] without per-line reset().
• Inline solve detection: findInlineSolves() is called only for lines that classifyLine() marks as having inline solves.

Tokenization is scoped to each line by temporarily restricting this.len to the line end position, so the [Symbol.iterator] generator naturally stops at the line boundary. After tokenization, this.len is restored and this.pos advances past the newline.

Parameters

Parameter	Type	Description
text	string	The full document text (with newlines).

Returns

ScanLineResult[]

Array of ScanLineResult, one per line, in document order.

---

tokenizeAll()

tokenizeAll(): Token[];

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:854

Tokenize an expression string into an array of Tokens.

Delegates to the lazy Symbol.iterator generator and collects all yielded tokens via Array.from(). For memory-sensitive use cases, prefer iterating the lexer directly with for…of to avoid array allocation.

Optimizations:

• CHAR_CLASS jump table (Uint8Array) → switch on small integers
• Direct character-code dispatch (c0 cached pattern)
• Mathematical digit parsing (integer math, not slice+parseFloat)
• Inline operator tokenizer with two-char peek-ahead
• Whitespace eliminated in-lexer (never emitted)
• 0-char and 1-char fast paths

Returns

Token[]

---

unregisterVocabulary()

unregisterVocabulary(plugin): void;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:643

Unregister a plugin, removing its custom tokens from the lexer.

This is the inverse of registerVocabulary(). All keywords, operators, and units registered by the plugin are removed. After unregistration, those tokens will revert to their default behavior (e.g., keywords become IDENT, operators become ERROR).

Calling unregisterVocabulary with a plugin that was never registered is safe, it simply has no effect.

Parameters

Parameter	Type
plugin	LexerVocabulary

Returns

void

---

hasExpressionIndicators()

static hasExpressionIndicators(
   input,
   start,
   end): boolean;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:1606

L1 expression gating: quickly determine if a line contains any characters that indicate an expression (digits, operators, currency, backticks, parentheses, etc.).

Pure prose lines (e.g., “The quick brown fox jumps over the lazy dog”) return false and can be skipped without full tokenization (L2).

This is a fast character-by-character scan that stops at the first expression indicator. Called once per line in classifyFromPositions().

Parameters

Parameter	Type
input	string
start	number
end	number

Returns

boolean