LexerVocabulary

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:315

Plugin interface for extending the ExpressionLexer with custom tokens.

Plugins can register:

• keywords: Map identifier strings to custom token types (checked after locale keywords).
• operators: Map multi-character operator sequences to custom token types.
• units: Register additional unit identifiers (checked alongside built-in units).

Multi-word phrase matching has been moved to the TokenNormalizer post-lexer stage. To register phrase patterns, use IEnginePackage.normalizerRules instead.

All registrations are additive, built-in patterns still work.

Properties

keywords?

optional keywords?: Record<string, string>;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:322

Keyword → tokenType mappings. Each key is a lowercase identifier that, when encountered, will emit the specified token type instead of IDENT. These are checked AFTER the locale’s built-in keywordMap, so locale keywords take priority.

---

operators?

optional operators?: Record<string, string>;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:330

Multi-character operator → tokenType mappings. Each key is the exact character sequence (e.g., ”::”, ”->”, ”=>”) and the value is the token type to emit. Two-character operators take priority during matching. Built-in operators (==, !=, >=, <=, <<, >>) always take priority.

---

rawLinePatterns?

optional rawLinePatterns?: {
  pattern: RegExp;
  tokenType: string;
}[];

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:368

Whole-line patterns matched against the RAW line text, BEFORE any per-character tokenization begins.

Every other extension point in this file (keywords/operators/ units, plus IEnginePackage.phrases/normalizerRules) transforms a token STREAM — they all assume the line is, at some granularity, valid Solve syntax. This hook exists for the one shape that isn’t: a package whose grammar captures arbitrary free-form text terminated by a fixed marker (e.g. a natural-language query ending in = ?), where the text itself (“distance to the moon”) would never tokenize or parse as a normal expression and must be captured verbatim instead — see packages/knowledge/ for the reference use.

Each entry’s pattern is tested (via RegExp.exec) against the full, untrimmed line text. If it matches AND capture group 1 is non-empty after trimming, the ENTIRE line becomes a single synthetic token of tokenType whose value/text is the trimmed capture group — the character-by-character scanner never runs for that line. Patterns are tried in registration order; the first match wins. A package registering a rule here still needs a prefixParselets entry for tokenType to actually consume the resulting token.

Because this bypasses tokenization entirely, a matching line can contain characters that would otherwise be lexer errors (unmatched quotes, stray symbols, …) — by design, since the whole point is to hand the package raw text the normal pipeline was never meant to parse.

pattern

pattern: RegExp;

tokenType

tokenType: string;

---

units?

optional units?: string[];

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:336

Additional unit identifiers to recognize (e.g., “gp”, “osrs”, “tile”). These are checked alongside the built-in knownUnits set.