Overview

The engine is a pipeline with a plugin system attached to it, plus a caching layer that makes it viable to run continuously.

Packages contributePipelineCaches skip

Document text

keywords, operators, unitsphrases and rewrite rulesparselets and binding powersnothingfunctions it can call

1Lexercharacters become tokens

2Normaliserthe token stream is rewritten

3Parsertokens become structure

4Compilerstructure becomes bytecode

5Virtual machinethe bytecode runs

Bytecode cachesame expression text, compiled before

Line cachenothing this line depends on changed

Value

Five stages, in order. A package contributes to four of them; the compiler is the exception, because it emits opcodes for whatever the parselets produced. Each bracket covers the stages that cache lets the engine skip entirely.

Stages

Text goes through five stages: lexing, normalisation, parsing, compilation, and execution. Each stage has a single responsibility and hands a well-defined structure to the next.

The unusual stage is normalisation. It sits between lexing and parsing and rewrites the token stream, fusing multi-word phrases into single tokens and making implicit operators explicit. It is what allows natural phrasing without turning ordinary words into reserved words.

Values

Execution produces a value: a type, a payload, and sometimes a unit. Errors and pending states are values too, which means they propagate through arithmetic instead of being coerced into numbers.

1. 1A valueType, payload, sometimes a unit.
2. 2Arithmetic keeps it5 km + 3 km
3. 3Errors are valuesThey propagate, they do not throw.
4. 4So is pendingWaiting is not the same as nothing.

12 km

1. 12 kmUom

Not a number with a label stuck to it. The type is part of the value and the unit travels with it, which is what lets the next operation reconcile the two sides rather than guess.

Every result

BackPauseNext

Packages

Every feature is a package, registered into shared registries at startup. The pipeline itself knows nothing about percentages, dates or currencies.

1. 1A package declaresA plain object, every field optional.
2. 2RegistrationInto shared registries, in order.
3. 3The pipeline reads themIt knows nothing about your feature.
4. 4Nothing is specialPercentages are a package. So are dates.

1. Vocabularykeywords, operators and units the lexer should know
2. Parseletshow its token types parse, and how tightly they bind
3. Functionswhat the virtual machine can call
4. Rules and categoriestoken rewrites, conversion targets, highlight categories

You declare only the part of the language you are adding. Arithmetic itself is a package with exactly this shape, which is the strongest evidence that these are the real extension points rather than a reduced set offered to outsiders.

Every feature

BackPauseNext

Caching

Three layers. Compiled bytecode is cached by expression text, on every entry point. Line results are cached by line, and a dependency graph tracks which lines depend on which variables, in the long-lived incremental evaluator an editor keeps: after an edit it re-runs the lines that depend on it and the lines on screen, and leaves the rest. A batch parseDocument call runs every line; see Performance.

1. 1ColdA line nobody has seen.
2. 2Bytecode cacheSame text, seen before.
3. 3Line cacheSame line, nothing it depends on changed.
4. 4TogetherWhy a keystroke is cheap.

1. Lex
2. Normalise
3. Parse
4. Compile
5. Execute

Every stage runs. This is the full cost, and it is paid once per distinct expression text.

Three layers

BackPauseNext

Further reading

The sibling pages here go deeper, aimed at contributors rather than users: the bytecode virtual machine, design decisions, the package system and the pipeline.