Spec M28 - mc lsp: the taught compiler as language server

Synthesis of the design panel (three proposals, two judges). Winner: "the parse is the index" - a token log plus two frame edges plus a small module vocabulary, with no change to src/ast.mc. Grafted: the diagnostic/resolution hook wired at the compiler's own name-resolution sites, the two-phase emission that keeps colours alive through a syntax error, the arena bracket in the long-lived process, the CPU guard and mc build --compiler-only.

1. Principle #

Only the project's taught compiler knows the developer's syntax, so the server IS that compiler in lsp mode. Nothing about any specific taught language enters src/. The server never learns what class, namespace, ref, . or a generic is; it reports what the parser knows: a token's class from the tables the language itself built, a declaration's position, a use's link, a diagnostic's range. Anything a module must tell the LSP that the core cannot deduce - that Box__Circle__4 is written Box<Circle,4>, that geo.Circle resolves to geo__Circle - goes through a generic mechanism justified by at least two imaginable languages, exactly as M21 SS1 requires. The core never mangles, never guesses, never splits a name on __.

Inert by construction (M21 SS6.5): with nothing registered and the index off, every tests/*.mc object and every --dump-ast/--dump-tokens byte is identical to the pre-M28 compiler's.

2. Server shape and protocol surface #

Three roles, two long-lived processes, one worker per parse. mc lsp [DIR] [--config F] is a subcommand of main(), beside mc build.

1. Front - the process the editor launches: framing, JSON, the document store, the last good index, the debounce timer. It reads mc.toml first; when [compiler].modules is non-empty it runs mc build --compiler-only and execvs the resulting taught compiler with the same lsp arguments, before answering initialize (so no document can exist yet). From that instant the server is the taught compiler. Because json.mc, index.mc, lspclass.mc and lsp.mc sit inside src/core.mc, every taught compiler is a language server for its own language at no extra cost, the same way it already carries mc build.
2. Zygote - forked by the front at startup, before the front touches the arena, and it never parses: its hp stays 0 for the life of the server. Pristineness is structural, not maintained.
3. Worker - one per parse, forked by the zygote. Registers the overlay, the error hook and the ref hook, then runs the ordinary front end with no reset code at all.

Why a process and not a reset. The compiler's globals are one-shot; src/driver.mc says so in its own header and spawns a second process for the second half of a taught build. Resetting means ~46 core counters and every module's own tables, which the core cannot see: examples/api/oop.mc has MAXIFACE 16 and a monotone nifaces, so an in-process loop dies with too many interfaces on parse

1. A fork resets tables the core will never know exist, in zero lines. Measured here with build/mc1 and a bare extern i64 fork();: 437 us per fork+waitpid from a clean parent, 474 us with 16 MiB dirty - flat, because COW is lazy. posix_spawn of mc lsp --lsp-index is the fallback where fork does not exist (M19/M20), measured 1.5-1.8 ms.

Protocol. JSON-RPC 2.0 over stdio, Content-Length framing, reader/writer in src/json.mc. Every field read or written is an integer or a string, so the absence of floating point costs nothing; js_int skips a fractional part so a client sending 1.0 is harmless. Positions are 0-based line plus UTF-16 character, computed in the worker, which holds every buffer. Exact request list, and nothing else:

Declined and advertised as such: incremental sync, semanticTokens/full/delta and /range, rename, formatting, folding, code actions, signature help, inlay hints, workspace/symbol, progress. Requests are handled strictly sequentially; LSP permits it and it removes all concurrency.

Reparse. didChange replaces the document and arms a 150 ms idle timer (poll's timeout - verified in this repository: a 150 ms poll on an empty pipe returns 0 after 151 ms). On fire, any in-flight worker is killed and reaped, then one new worker is forked. Exactly one worker exists at any moment, so the queue cannot grow. Queries are answered from the last blob without reparsing.

Guards. The worker sets setrlimit(RLIMIT_CPU, {N,N}) on itself, so a taught handler that loops forever is killed instead of wedging the editor - verified here: raw wait status 24 = SIGXCPU. A worker that dies without a complete blob leaves the previous index in place; the front says the index is stale rather than blanking the file.

3. Core changes #

Signatures, call sites, cost, and the second consumer that earns each one. src/ast.mc is not touched: provenance is keyed on the token log, never on a node field. That is deliberate - node_assign is a blind mem_copy of all ND_SIZE bytes, so a per-node position column is silently overwritten by node_copy_subst with the template's own position, and M21's ref rewrite discards nodes after the parse. It also leaves M21 SS5's objection to nd_uty standing instead of reopening it.

Total ~1585 lines: ~355 in existing files (lex.mc 183, parse.mc 78, hooks.mc 30, arena.mc 26, gen_arm64.mc 8, main.mc/driver.mc/core.mc/astdump.mc 30, ast.mc zero) and ~1230 in four new ones (json.mc 260, index.mc 300, lspclass.mc 75, lsp.mc 600). stage0 is untouched: mc lsp does not exist in C, exactly as --exe does not.

A rule for review: src/index.mc must stay compilable by the frozen seed build/mc0 with only arena.mc, lex.mc and ast.mc in front of it, because src/astdump.mc includes it for check-ast. Any use of a hooks.mc table inside it breaks that and belongs in lspclass.mc.

4. Provenance model #

Ranges. Every position is a token serial. lsp_tok appends one row per delivered token; p_mark() is the serial of the token the parser is on. A declaration's range is its own row, a use's range is its own row, a diagnostic's range is the row of the token the error was raised on. There is no second position system and no heuristic. tk_col derives the byte column from cbol; UTF-16 conversion happens once, in the worker, with a running cursor advanced in token order (tokens within a frame have increasing offsets), counting +1 per byte with (b & 0xC0) != 0x80 and +1 extra per byte >= 0xF0. The front never reads source text.

Token classes, with zero new tables. lsp_class(mark) is a pure function of the recorded id plus predicates hooks.mc already exports: id < 256 selects identifier/int/char/string/directive/hole; K_U8..K_VOID type, K_IF..K_EXTERN keyword, K_LPAR..K_ARROW operator or punctuation; id > K_ARROW means the lexeme was taught, and which table claims it says what it is - alias_find type, infix_is_taught operator-with-handler (so +, in the same table without a handler, is never mislabelled), syntax_find/syntax_stmt_find/syntax_expr_find/rule_find keyword, else te_word decides word versus punctuation. def_find marks a macro. An SC_TAUGHT bit rides along, giving the editor a taught modifier over the developer's own vocabulary. Because the recorded id already reflects every tok_add that had happened when the token was produced, classification is exact by construction: a name used as an identifier at line 10 and taught at line 100 records as an identifier and stays one.

Declarations, in three tiers. (1) Core, token-exact: the six p_decl sites of SS3.6. (2) Core, line-granularity: def_add (a DefEnt keeps no position) and top_add, which records every N_FUNC/N_GLOBAL/N_EXTERN/N_PROTO a module appends, from the node's own file/line. (3) Taught words: word_add records the position of the token the parser is on, so class Rect makes Rect navigable while syntax("class") inside user_init correctly records nothing - the module's vocabulary is not the program's. Tier 3 is skipped when a module already claimed the name, so cooperation upgrades instead of duplicating.

Uses, two passes, the compiler's answer winning. The worker runs both:

• Best-effort (lsp_resolve, after parse_unit): for each identifier row that is not itself a declaration, match by lexeme against the declaration table walking the scope chain outward, requiring declaration-before-use for locals and params and order-free at file scope. It is non-fatal, which is the point: gen_lower dies on the first unknown name, and an editor buffer is unresolved most of the time someone is typing.
• Authoritative (ref_note, during gen_lower): the compiler's own local_find/global_find/ func_find results overwrite the row's decl column and set a confirmed bit. What the editor shows is then what the compiler resolved, including its real shadowing.

A row that neither pass resolved is reported as "no definition" - never guessed.

Taught constructs. Four mechanisms, none of them about any language.

1. Frame origin (free). Every token inside an #include, a namespace body or a generic instantiation maps up to the token that caused the push. make slot<i64,3>; yields a frame named by the module (slot__i64__3 instantiated from prog.lx:27) whose origin is the real source range.
2. Frame inner (free). If the pushed buffer lies inside an ancestor frame's buffer - the module replayed a span recorded verbatim with p_skip_balanced - every offset converts to the ancestor's text by a constant delta, so go-to-definition inside Box<Circle,4> lands on the corresponding token of the Box template. When the module composes a new buffer instead (a mangled header prepended), containment fails and origin remains - which is exactly the gap M21 SS3 already documents, inherited rather than re-invented.
3. p_decl / p_ref for what no bookkeeping recovers: a module resolves geo.Circle to geo__Circle in its own table and lowers c.r to ld64(c + 8); the core sees three tokens and an arithmetic node. Panel measurement: 11 lines in examples/api/oop.mc upgrade the line-granularity answers to token-exact on w, h, area, Rect, Shape and link class Rect : Shape to the interface's own name token; 12 lines in lib/user_syntax_demo.mc's syntax_infix("~>") handler declare a member at first use and make every later use resolve to it, the obj.field case of M22, through a real Pratt hook.
4. p_decl_show for the developer's spelling: hover, outline and completion read Box<Circle,4> while the symbol is Box__Circle__4. The core stores it and hands it back; it never parses it.

Diagnostics. err_set intercepts all four funnels before _exit: the handler appends the record with the current token's range, flushes the blob and exits 0. A range inside a synthetic frame maps out through inner (exact) then origin (the use site), so Pair__i64__3 instantiated from prog.lx:15: unknown name lands on the instantiation with M21's prefix intact as the message. Two-phase emission keeps a broken file usable: the worker flushes the token log (phase 1, lex-only - --dump-tokens exits 0 on a file where --dump-rules dies at line 2) before the parse, so colours and the outline survive to the end of the file even when the parse stops at line

1. Everything goes through a Buf with one io_write: field-by-field emission of 13 000 records measured 86.5 ms against 2.3 ms buffered, 37x.

Determinism. Every table is linear in production order - token rows in lex order, files in push order, declarations and uses in walk order - searched back to front where last-wins is intended. No hashing, no pointer ordering, nothing iterated for output (docs/determinism.md rule 1). Two runs of --dump-index on the same bytes are byte-identical, which is what makes the goldens a diff.

5. Performance budget #

Measured. Rows marked (*) were re-measured in this repository against build/mc1; the rest come from the panel's prototypes on copies of src/.

Parse runs at ~14 MB/s of source, parse+resolve at ~8.5 MB/s, so a 100 ms budget covers ~850 KB of unit. Index overhead: +0.6% compiled in but off (inside noise), +2.6% while indexing. Blob: 24 B/token plus 48 B/declaration - 87 KB for the realistic unit, 2.4 MB for this compiler. Queries never reparse: definition is a binary search over the document's rows, references one linear pass (98 986 rows is ~0.1 ms), semanticTokens/full one filtered pass at ~250 ns/token. Front memory is flat because every request is bracketed by arena_mark/arena_release and the document text lives in a __bss slab; worker memory dies with the worker. The arena is the ceiling: this compiler reaches 24.8 MB of a 32 MiB heap with gen_lower, so a unit 30% larger fails - a diagnostic under the worker model, not a dead server. M23 lifts it.

No incrementality is built, and none is needed below ~5 k lines. Sub-unit incrementality would be unsound anyway: #define, #token, #infix, #rule, syntax*, type_alias and p_resplit_punct all mutate lexer and parser state as the parse proceeds, so file k's token stream (its lexeme boundaries included) is a function of files 1..k-1. There is no prefix-independent unit to cache.

6. What stays out, and why #

• More than one diagnostic per parse. err_at ends in _exit(1) and there is no recovery point in parse_unit. Recovery would change error text that tests/err/* and scripts/test.sh pin byte for byte. Mitigated, not solved, by phase 1 and by the flushed prefix. This is the single largest gap and it is inherited, not introduced.
• Anything needing a type. No type checker exists and M21 SS5 refused nd_uty and a core symbol table for reasons that still hold. So: no hover type for an arbitrary expression, no member completion after a taught ., no type errors, no signature help driven by argument types. Each is possible for a taught language if its module emits the facts and impossible for the core.
• Rename. A module derives several spellings from one name (w -> rect_w, set_rect_w, RECT_W) and only the module knows the derivation; renaming what the core resolved would silently corrupt source. Would need a p_rename_group hook. Not offered.
• Formatting, folding, code actions, comment colouring. skip_space discards comments and whitespace and they never become tokens; there is no lossless CST and no end position on a construct. M29's TextMate grammar colours comments, layered under semantic tokens.
• Workspace scope. One mc.toml is one unit is one server, no workspace/symbol; and the compiler has no translation unit smaller than [project].entry, so a file outside every closure gets an empty index and a message saying why.
• Un-registering a word. Structural: tok_add is append-only, so syntax_stmt("log") removes log from the identifier vocabulary program-wide and completion can never offer it as a variable. The core already says so plainly and the server repeats it verbatim.

7. Acceptance #

• --dump-index and --dump-sem are deterministic text dumps beside --dump-tokens/--dump-ast/ --dump-rules, diffed against goldens under tests/lsp/ by scripts/check-index.sh: the whole provenance model is testable with diff and no node client, in the project's own idiom.
• tests/lsp/client.mc - a scripted LSP client written in the language, driving a real server over a pipe and diffing replies - covers framing, JSON and the handlers. make check gains check-lsp.
• Against lib/user_syntax_demo.mc (exists today, built as a taught compiler by scripts/check-surface.sh): enum, unless, tmpl, make, bits, pipe come back as keyword + taught; ~>, .+ (code handlers) and <+> (Tier 1 #token + #infix) as operator + taught; + as a plain operator; p ~> len = 3 and p ~> at(2) resolve to the member declared at its first use and references on it lists every ~> site; make slot<i64,3>; produces a frame whose origin is the make site and whose diagnostics carry the instantiated from prefix mapped to a real range.
• Against examples/lang/main.lx through its taught compiler (M22): class, ref, namespace come back as taught words and . as a taught operator; definition of a method resolves to the class body line; references of a field lists every .field use; hover on a #define shows its folded value and on an instantiation shows Box<Circle,4>, not Box__Circle__4; documentSymbol nests methods under their class through p_decl_parent; a syntax error reports the right range and the file stays coloured to its end.
• Inertness gate (M21 SS6.5): with the index off, every tests/*.mc object and every --dump-ast output is byte-identical to the pre-M28 compiler's, and check-lex/check-ast stay green under the frozen seed build/mc0.
• make check green; golden rewritten once; stage0 untouched and under budget; docs/surface.md gains the module vocabulary of SS3.7 with the same emphasis M21 gave the lookahead contract: capture the mark while the parser is still on the token you mean.

8. Decisions (architect, 2026-09-03) #

1. DE_LEN: in, as a separate commit before the server (its own golden rewrite), since it is a byte-identical parse speedup unrelated to the LSP.
2. gen_lower on every keystroke: yes; [lsp] resolve = "parse" in mc.toml is the escape for very large units. The authoritative use links and unknown name are worth +66% of a ~50 ms parse.
3. Documents outside the entry's closure: scanned as their own root, reported as such; [lsp] unit = "file" for projects that want per-file navigation only.
4. p_decl kinds: start from LSP SymbolKind values, documented in docs/reference/hooks.md; append-only afterwards.
5. fork on Windows: accept the asymmetry; the --lsp-index spawn fallback is the Windows path and is documented, not hidden.
6. Diagnostics beyond the first: build the bounded rescan now (blank the offending line in the overlay, rescan, at most 3 iterations), because one squiggle at a time is the first thing every developer will complain about; it needs no core change.

Edit this page