Editor integration

The language entry point provides what an editor needs, without assuming which editor.

import { LanguageService } from "solve-engine/language";

Completions

getCompletions(lineText, cursorOffset) answers the words that complete what the reader is typing, matched by prefix and capped at fifty. The candidates:

• keywords and function names (sqrt), and the call words packages declare, as functions (sha offers sha256);
• phrases, the aggregates among them, offered whole by their opening words (averag offers average of), and across the words already typed (net pres offers net present value of);
• units, and the units the document defines (1 sprint = 2 weeks makes spr offer sprint);
• the document’s variables, and a package’s own completionItems.

A phrase matched across typed words carries replaceLength, the number of characters before the cursor it replaces. The CodeMirror adapter, completionItemToOption, turns that into an apply that replaces them; another editor’s integration replaces that many characters itself. A phrase is offered by its opening words, not continued part-way through from the grammar.

A package author’s side of this is on highlighting and completions, including what normalizeForHighlighting costs.

The document’s own names

A note defines names as it goes: rent = 1200 makes rent a variable, hourly rate = $50 a name of two words, and f(x) = x * 2 a function called f. The language service offers those names as completions, and it uses them for one highlighting decision: a line holding a single bare word is coloured only when that word is a name the document defines, since otherwise any word of prose would look like code.

By default the service asks its engine, through engine.documentVariableNames(), which lists the names the document’s lines define and the engine still holds. It is the same list whichever way the host ran the note: parseDocument, evaluateLines, evaluateDocument, or a live incremental evaluator. So after the engine has run rent = 1200 and rate = 5, typing re offers rent and a line holding only rent is highlighted.

what the host did	re offers
parseDocument("rent = 1200\nrate = 5\nre")	rent
evaluateDocument(engine, "rent = 1200\nrate = 5\nre")	rent
evaluateExpression("rent = 1200")	nothing

What is not offered, and why:

• a name a line only reads (rent * missing does not make missing a name), including the half-typed word on the line being edited;
• a name set outside a document, with evaluateExpression, since that belongs to the host and not to the note;
• a name whose defining line was edited away or deleted, once the evaluator has run the edit;
• anything at all after engine.clear(), or from the previous note once the next document pass starts.

A host that highlights with a separate engine, one that never runs the note, passes variableNameSource so the service reads the evaluating engine’s names instead:

import { createEngine } from "solve-engine";
import { LanguageService } from "solve-engine/language";

const evaluatingEngine = createEngine(); // runs the note
const highlightEngine = createEngine(); // only colours lines

const service = new LanguageService(highlightEngine, {
  variableNameSource: () => evaluatingEngine.documentVariableNames(),
});

The function is called on every completion and on every line holding a lone word, so it should hand back names it already holds rather than build a list each time. documentVariableNames() reads the engine’s own table without copying it, and the default lone-word check asks the engine about the one word, engine.isDocumentVariableName(word), rather than walking every name.

Highlighting

Every token carries a category such as number, unit, function or operator. Map those categories to your own theme rather than hardcoding colours, so that a package adding new syntax is highlighted without further work.

The route from a line of text to a coloured line is four short steps, and the engine only owns the middle two.

1. 1A line of textWhat your editor has.
2. 2Semantic tokensgetSemanticTokens(line, number)
3. 3Class namestokenClassName(category)
4. 4ColourOrdinary CSS, in your stylesheet.

1. 12 km in miles

One string. No structure yet.

Your editor knows where the caret is and what the line says. It knows nothing about what any of it means, and it should not have to.

Editor integration

BackPauseNext

Highlighting reads a line and never runs it. Some lines change the document when they run: a running total (total += 5) adds to its total, an assignment (x = 3) sets a variable, and a unit definition (1 sprint = 2 weeks) adds a unit. The service checks that such a line is well formed and leaves it there, so an editor can highlight on every keystroke, as often as it likes, and no value in the document moves. Running the line is the evaluator’s job alone.

Every span is measured on the line exactly as it was handed over, so its from and to name the characters to colour, with no offset to add. Markdown in front of an expression is looked past and left uncoloured: a quote marker (> 1 + 2), a bullet (- 100 + 20, * 5 kg), a numbered item (1. 12 km) and a task box (- [ ] total += 5). A bullet’s - is markup, not a minus sign, which is also why the engine evaluates - 100 + 20 as 120. Written with no space after it, -100 + 20 is arithmetic, and its minus is coloured as one.

A line’s label (Total: 1 + 2) is prose the engine sets aside, so it is left uncoloured like the markdown in front of it. The conversion words to, in and into are coloured as keywords, and the colon of a clock time (12:30) as part of its number.

Off the main thread

A host that runs the engine in a worker gets the same calls from the worker client (getSemanticTokens, getCompletions, findReferences, getDefinition, rename, shiftLineReferences), answered by the worker’s engine. See every host call, off the thread.

References, rename and moving lines

The same service reads a whole document for the features that follow a name across lines: find every reference to a variable, go to its definition, show its value on hover, rename it without touching prose that shares the word, and keep line N references on their lines when lines are inserted or deleted. Each returns positions or text edits for the editor to apply. See reference-aware editing.

Checking a line parses

For deciding whether to decorate a line at all, there is a non-throwing check that avoids constructing an error object for the common case of prose that is not an expression. On a document that is mostly prose, that difference is substantial.

The check answers whether the line is well formed, not what it would come to, and it is read-only for the same reason highlighting is. A line that parses but would fail when run, such as total += nope where nope is never defined, is still a line the engine reads as an expression, so it answers yes, the same as 5 + nope does.