LexerToken

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:425

A token, as the lexer produces it.

A class rather than an object literal because tokens are created on every keystroke and a shared hidden class keeps that path predictable for the engine running it.

Implements

• Token

Constructors

Constructor

new LexerToken(
   type,
   typeId,
   value,
   text,
   offset,
   lineBreaks,
   line,
   col,
   sourceEnd?,
   fault?,
   mayNameVariable?
): LexerToken;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:426

Parameters

Parameter	Type	Description
type	string	-
typeId	number	-
value	string	-
text	string	-
offset	number	-
lineBreaks	number	-
line	number	-
col	number	-
sourceEnd?	number	See Token.sourceEnd. Set only on normalizer-fused tokens.
fault?	{ code: string; message: string; }	See Token.fault. Set only on a fused token standing for a run the engine refuses to read.
fault.code?	string	-
fault.message?	string	-
mayNameVariable?	boolean	See Token.mayNameVariable. Set only on a rate denominator fused from a slash.

Returns

LexerToken

Properties

col

col: number;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:434

Implementation of

Token.col

---

fault?

optional fault?: {
  code: string;
  message: string;
};

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:438

See Token.fault. Set only on a fused token standing for a run the engine refuses to read.

code

readonly code: string;

message

readonly message: string;

Implementation of

Token.fault

---

line

line: number;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:433

Implementation of

Token.line

---

lineBreaks

lineBreaks: number;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:432

Implementation of

Token.lineBreaks

---

mayNameVariable?

optional mayNameVariable?: boolean;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:440

See Token.mayNameVariable. Set only on a rate denominator fused from a slash.

Implementation of

Token.mayNameVariable

---

offset

offset: number;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:431

Implementation of

Token.offset

---

sourceEnd?

optional sourceEnd?: number;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:436

See Token.sourceEnd. Set only on normalizer-fused tokens.

Implementation of

Token.sourceEnd

---

text

text: string;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:430

Implementation of

Token.text

---

type

type: string;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:427

String token type (e.g., “NUMBER”, “PLUS”, “IDENT”). Used by ParseletRegistry string-keyed maps and error messages.

Implementation of

Token.type

---

typeId

typeId: number;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:428

Integer token type ID for fast comparison in Parser hot path. Populated by the Lexer via registerTokenType(). Use tokenTypeId(type) to get a type’s ID.

Implementation of

Token.typeId

---

value

value: string;

Defined in: packages/engine/src/lexer/ExpressionLexer.ts:429

Implementation of

Token.value