TokenNormalizer

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:286

Token normalizer: applies NormalizerRules to a token stream.

Lifecycle

1. Registration: Rules are added via register and sorted by priority
2. Normalization: normalize applies rules greedily left-to-right
3. Cleanup: clear or unregister removes rules

Normalization algorithm

The normalizer uses a greedy left-to-right multi-pass algorithm:

• At each token position, rules are tried in priority order (highest first)
• When a rule matches, matched tokens are consumed and replaced
• Processing continues from the replacement position
• Multiple passes handle cascading matches (one rule’s output triggers another)
• Safety limits (NormalizerOptions.maxPasses) prevent infinite loops

Example

const normalizer = new TokenNormalizer();
normalizer.register(phraseRule);       // "to the power of" → CARET
normalizer.register(implicitMultRule); // "2 x" → "2 * x"
const normalized = normalizer.normalize(rawTokens);

Constructors

Constructor

new TokenNormalizer(options?): TokenNormalizer;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:356

Parameters

Parameter	Type	Description
options	NormalizerOptions	Configuration overrides for safety limits and diagnostic callbacks

Returns

TokenNormalizer

Properties

environment

environment: NormalizerEnvironment = {};

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:365

What the rules may read about the engine this normaliser serves, handed to every rule’s match. Empty for a normaliser no engine owns; the engine sets it once, at construction. See NormalizerEnvironment.

Accessors

ruleCount

Get Signature

get ruleCount(): number;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:519

Get the number of currently registered rules (excludes phrase trie entries).

Returns

number

Methods

addPhrase()

addPhrase(phrase, tokenType): void;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:602

Register a multi-word phrase for fusion into a single compound token.

This is the preferred way to add phrase patterns. It inserts into the internal PhraseTrie, which collapses all phrase rules into a single O(depth) trie walk per position, no separate rule scanning.

Parameters

Parameter	Type	Description
phrase	string	Multi-word phrase (e.g., “to the power of”, “abyssal whip”)
tokenType	string	Target token type after fusion (e.g., “CARET”, “ITEM”)

Returns

void

---

canStartPhrase()

canStartPhrase(word): boolean;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:624

Parameters

Parameter	Type
word	string

Returns

boolean

---

clear()

clear(): void;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:407

Remove all registered rules, resetting the normalizer to its initial state. Also clears the phrase trie.

Returns

void

---

getCandidateCounts()

getCandidateCounts(tokens): number[];

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:575

How many rules could fire at each position of tokens, against the total.

The point of the index is that most positions admit no rule at all, and this is what makes that visible rather than asserted.

Parameters

Parameter	Type
tokens	Token[]

Returns

number[]

---

getPhrases()

getPhrases(): Record<string, string>;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:620

Get all registered phrases and their target token types.

Exposes the full phrase trie structure for diagnostic rendering in the playground’s NormalizerTab. Returns ALL registered phrases, not just the ones that matched in the last evaluation.

Returns

Record<string, string>

---

getRuleShapes()

getRuleShapes(): {
  indexedSlots: number;
  name: string;
  priority: number;
  shape: readonly RuleSlot[];
  unshapedReason?: string;
}[];

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:544

Returns

{ indexedSlots: number; name: string; priority: number; shape: readonly RuleSlot[]; unshapedReason?: string; }[]

---

normalize()

normalize(tokens, onFusion?): Token[];

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:659

Normalize a token stream by applying all registered rules.

Algorithm

Applies rules greedily left-to-right in multiple passes:

1. Sort rules by priority (descending)
2. Walk the token stream left to right
3. At each position, try rules in priority order
4. On match: consume matched tokens, insert replacements, restart from insert point
5. On no match: pass token through unchanged
6. Repeat until a full pass produces no changes, or maxPasses is reached

Fusion tracking

When a rule consumes more tokens than it produces, the normalizer calls onFusion with a TokenFusion record for diagnostic collection. This populates NormalizerOutput.fusions in the playground pipeline view.

Safety

If the normalized token count exceeds NormalizerOptions.maxTokens, an Error is thrown to prevent memory exhaustion from runaway rule expansion.

Parameters

Parameter	Type	Description
tokens	Token[]	Raw tokens from the lexer
onFusion?	(fusion) => void	Optional fusion callback (overrides NormalizerOptions.onFusion)

Returns

Token[]

Normalized tokens ready for parsing

Throws

If the normalized token count exceeds maxTokens, or the stream is still changing after maxPasses passes (a rule chain that never settles): NORMALIZER_PASS_LIMIT_EXCEEDED, rather than a stream that is quietly whatever the last pass left.

---

register()

register(rule): void;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:378

Register a normalization rule.

Rules are sorted by priority (descending) on each normalize call. Multiple rules can share the same priority, they are tried in registration order when priorities are equal.

Parameters

Parameter	Type	Description
rule	NormalizerRule	The rule to register

Returns

void

---

unregister()

unregister(ruleName): void;

Defined in: packages/engine/src/normalizer/TokenNormalizer.ts:395

Unregister a normalization rule by its name.

If multiple rules share the same name, all are removed. This is safe to call with a name that doesn’t match any rule, it simply has no effect.

Parameters

Parameter	Type	Description
ruleName	string	The name of the rule to remove

Returns

void