LexerToken

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:232

A token, as the lexer produces it.

A class rather than an object literal because tokens are created on every keystroke and a shared hidden class keeps that path predictable for the engine running it.

Implements

• Token

Constructors

Constructor

new LexerToken(
   type,
   typeId,
   value,
   text,
   offset,
   lineBreaks,
   line,
   col,
   sourceEnd?): LexerToken;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:233

Parameters

Parameter	Type	Description
type	string	-
typeId	number	-
value	string	-
text	string	-
offset	number	-
lineBreaks	number	-
line	number	-
col	number	-
sourceEnd?	number	See Token.sourceEnd. Set only on normalizer-fused tokens.

Returns

LexerToken

Properties

col

col: number;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:241

Implementation of

Token.col

---

line

line: number;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:240

Implementation of

Token.line

---

lineBreaks

lineBreaks: number;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:239

Implementation of

Token.lineBreaks

---

offset

offset: number;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:238

Implementation of

Token.offset

---

sourceEnd?

optional sourceEnd?: number;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:243

See Token.sourceEnd. Set only on normalizer-fused tokens.

Implementation of

Token.sourceEnd

---

text

text: string;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:237

Implementation of

Token.text

---

type

type: string;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:234

String token type (e.g., “NUMBER”, “PLUS”, “IDENT”). Used by ParseletRegistry string-keyed maps and error messages.

Implementation of

Token.type

---

typeId

typeId: number;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:235

Integer token type ID for fast comparison in Parser hot path. Populated by the Lexer via registerTokenType(). Use tokenTypeId(type) to get a type’s ID.

Implementation of

Token.typeId

---

value

value: string;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:236

Implementation of

Token.value