TokenNormalizer

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:233

Token normalizer: applies NormalizerRules to a token stream.

Lifecycle

1. Registration: Rules are added via register and sorted by priority
2. Normalization: normalize applies rules greedily left-to-right
3. Cleanup: clear or unregister removes rules

Normalization algorithm

The normalizer uses a greedy left-to-right multi-pass algorithm:

• At each token position, rules are tried in priority order (highest first)
• When a rule matches, matched tokens are consumed and replaced
• Processing continues from the replacement position
• Multiple passes handle cascading matches (one rule’s output triggers another)
• Safety limits (NormalizerOptions.maxPasses) prevent infinite loops

Example

const normalizer = new TokenNormalizer();
normalizer.register(phraseRule);       // "to the power of" → CARET
normalizer.register(implicitMultRule); // "2 x" → "2 * x"
const normalized = normalizer.normalize(rawTokens);

Constructors

Constructor

new TokenNormalizer(options?): TokenNormalizer;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:265

Parameters

Parameter	Type	Description
options	NormalizerOptions	Configuration overrides for safety limits and diagnostic callbacks

Returns

TokenNormalizer

Accessors

ruleCount

Get Signature

get ruleCount(): number;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:323

Get the number of currently registered rules (excludes phrase trie entries).

Returns

number

Methods

addPhrase()

addPhrase(phrase, tokenType): void;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:339

Register a multi-word phrase for fusion into a single compound token.

This is the preferred way to add phrase patterns. It inserts into the internal PhraseTrie, which collapses all phrase rules into a single O(depth) trie walk per position, no separate rule scanning.

Parameters

Parameter	Type	Description
phrase	string	Multi-word phrase (e.g., “to the power of”, “abyssal whip”)
tokenType	string	Target token type after fusion (e.g., “CARET”, “ITEM”)

Returns

void

---

canStartPhrase()

canStartPhrase(word): boolean;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:361

Parameters

Parameter	Type
word	string

Returns

boolean

---

clear()

clear(): void;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:302

Remove all registered rules, resetting the normalizer to its initial state. Also clears the phrase trie.

Returns

void

---

getPhrases()

getPhrases(): Record<string, string>;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:357

Get all registered phrases and their target token types.

Exposes the full phrase trie structure for diagnostic rendering in the playground’s NormalizerTab. Returns ALL registered phrases, not just the ones that matched in the last evaluation.

Returns

Record<string, string>

---

normalize()

normalize(tokens, onFusion?): Token[];

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:393

Normalize a token stream by applying all registered rules.

Algorithm

Applies rules greedily left-to-right in multiple passes:

1. Sort rules by priority (descending)
2. Walk the token stream left to right
3. At each position, try rules in priority order
4. On match: consume matched tokens, insert replacements, restart from insert point
5. On no match: pass token through unchanged
6. Repeat until a full pass produces no changes, or maxPasses is reached

Fusion tracking

When a rule consumes more tokens than it produces, the normalizer calls onFusion with a TokenFusion record for diagnostic collection. This populates NormalizerOutput.fusions in the playground pipeline view.

Safety

If the normalized token count exceeds NormalizerOptions.maxTokens, an Error is thrown to prevent memory exhaustion from runaway rule expansion.

Parameters

Parameter	Type	Description
tokens	Token[]	Raw tokens from the lexer
onFusion?	(fusion) => void	Optional fusion callback (overrides NormalizerOptions.onFusion)

Returns

Token[]

Normalized tokens ready for parsing

Throws

If the normalized token count exceeds maxTokens

---

register()

register(rule): void;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:280

Register a normalization rule.

Rules are sorted by priority (descending) on each normalize call. Multiple rules can share the same priority, they are tried in registration order when priorities are equal.

Parameters

Parameter	Type	Description
rule	NormalizerRule	The rule to register

Returns

void

---

unregister()

unregister(ruleName): void;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:293

Unregister a normalization rule by its name.

If multiple rules share the same name, all are removed. This is safe to call with a name that doesn’t match any rule, it simply has no effect.

Parameters

Parameter	Type	Description
ruleName	string	The name of the rule to remove

Returns

void