Spec M21 - Tier 3 completion: expression/infix hooks, source record and replay, hygienic substitution

Goal: the smallest core surface that lets a .mc module build the whole of M22 - and languages that look nothing like M22 - without src/ learning any of it. All in src/*.mc; stage0 untouched.

1. Principle: mechanisms, never features #

Owner, 2026-09-03: mc gains only mechanisms here, never features. No notion of class, generic, ref, namespace or memory policy enters src/. Every mechanism below is justified by at least two imaginable languages, never by M22 alone. Three consequences that decide the design:

• Inert by construction. Every table starts empty, every new field is zero. An untaught compiler must produce byte-identical objects, --dump-ast and error text; that is the acceptance gate (6).
• No policy in the core. It never mangles a name, memoizes an instantiation, decides what . means or knows what a scope is; it hands out position, spans and a second source, and the module owns naming, typing, layout and ownership.
• What a module cannot route around is what earns a hook, in the compiler's own words.

2. The core surface #

2.1 void syntax_expr(uptr word, uptr fn) - expression position; handler i64 f() -> node index. Where: src/hooks.mc, a linear table in registration order searched back to front (last wins), with word_add refusing a core keyword - the shape of syntax/syntax_stmt; dispatched as the first thing in parse_primary, before the T_INT/T_STR/T_IDENT chain, with the parser stopped on the word (the handler consumes it). Why: C++ needs new T(a), sizeof(x), dynamic_cast<T>(e); Rust needs &mut x, match e { ... }, Some(v). #prefix cannot reach them - its template parses one operand with parse_unary into a fixed tree, reading neither a type nor an argument list, and it cannot build N_ADDR at all, since N_ADDR carries a name and not a child (#prefix "ref" &$1 is refused with & expects a name). Measured wall: f(new Point()) gives expression expected.

2.2 void syntax_infix(uptr word, i64 prec, uptr fn) - handler i64 f(i64 left) -> node index. Where: the Pratt entry gains one column (INF_FN, INF_SIZE 32 -> 40) with ie_fn/set_ie_fn, and infix_set clears it, so a #infix redefining the token drops the handler; in parse_expr's loop, right after if (ie_prec(e) < minprec) return lhs;, the core consumes the operator and does lhs = callp(ie_fn(e), lhs); continue;. Sharing #infix's table is the point: a taught . and a #infix operator sit in one comparable precedence order. infix_is_taught(tok) is added so err_name blames only operators that carry a handler, never +. Why: C# needs a.b, a?.b, a is T, a as T, whose right operand is a name or a type and whose lowering depends on the static type of the already parsed left operand; C needs a->b and a[i] (the core produces no N_INDEX at all). No template reaches this: #infix "." 12 left ld64($1 + $2) needs one program-wide #define per field at a fixed width and dies on p.x = 5; (left side of assignment must be a name) and on p.m() (wrong number of arguments); a #rule stmt: dies with unknown name, an ident hole being a name and never a value; a lexer fusion sees only ) in f(1).area.

2.3 uptr p_skip_balanced(i64 open, i64 close, uptr plen) - record a body without parsing it. Where: src/parse.mc, with cur on the opening token; it counts depth by consuming real tokens and returns the source bytes of the whole span, delimiters included, leaving the parser just past the closing token (unterminated dies at the opening token). Counting with the lexer instead of a byte scan is what makes a } inside a string or a comment harmless. Why: C++ and D templates capture a body now and check it per instantiation; Nim template, Ada generics and Rust macro_rules! are the same shape, as is any deferred region (Verilog generate, Zig comptime).

2.4 void p_push_source(uptr name, uptr text, i64 len) - a second source. Where: src/parse.mc, a wrapper over lex_push_mem(name, src, len, 0, p_line()), which already exists (M15 split it out for bundled includes); semantics are exactly #include's - the lexer pops on its own at the end of the buffer and name is what err_at prints for everything inside. Why: Zig's fn List(comptime T: type) type and C++ template instantiation both re-parse one recorded body per argument tuple; a namespace/import system pushes a second source from inside a declaration. Recording needs no core support (p_start() gives the byte pointer, the arena keeps the buffer); only re-entry does, because lex_push is file-only.

2.5 p_subst_reset() / p_subst_name(from, to) / p_subst_int(from, v) - hygienic substitution. Where: src/lex.mc; entries accumulate into a pending list, the next p_push_source binds it to the frame it pushes and the frame's pop discards it (MAXSUBST 16 per frame, nested frames independent). Applied in lex_next's identifier branch only, by exact lexeme, linear, in registration order; p_subst_name resolves to through word_id, so a type alias or a taught word yields the right token id, and p_subst_int yields T_INT with the value. Why: C# generics substitute a type parameter, C++ non-type template parameters substitute a literal used as an array bound; second family, a DSL/assembler macro substituting register names or a units language substituting a dimension exponent. Two properties come from doing it in the core, both correctness: substitution can never reach inside a string, a comment or a partial identifier (add_T is untouched), and an integer arrives as a token, so T items[N] folds in parse_dim - which tree substitution cannot do (u8 buf[$1]: array size must be a positive constant).

2.6 uptr p_start() / i64 p_depth() - the read side: tok_start(cur) and the lexer stack depth. Useless without 2.3/2.4 and required by any module that drives parse_top() over generated text (C++ templates, Zig comptime, a namespace body) and must know when the pushed source is exhausted.

2.7 void p_resplit_punct(i64 n) - the current punctuation token of length L > n becomes the punctuation formed by its first n bytes, the cursor rewinding to just after them. Why: C++ and C#/Java need >> split into two > inside a nested type-argument list, a longest-match lexing decision the parser cannot undo afterwards (List<List<i64>>: expected > after List<); Rust needs &&x read as two &. The core learns "a punctuation token may be re-split", not "generics exist".

2.8 Sizing (no code). MAXSYNTAX 32 -> 256 (one syntax_stmt per class and per instantiation), MAXALIAS 64 -> 256 (same reason), MAXOPEN 16 -> 32 (instantiation nesting), MAXSUBST 16. Every limit reachable mid-parse reports through err_at with a position, not die.

Cost. syntax_expr 35, syntax_infix 32, p_skip_balanced 22, p_push_source 4 (lex_push_mem already exists), p_subst_* 40, p_start/p_depth 3, p_resplit_punct 10 - ~146 lines total, ~110 non-comment, spread over src/hooks.mc, src/parse.mc and src/lex.mc.

3. Record/replay semantics and error attribution #

Record and replay. p_skip_balanced copies nothing into a token array: it returns the raw source span whose boundaries the real lexer found, and the span lives in the arena for the whole compilation. p_push_source pushes that span (or any buffer the module built) as a new lexer frame; parsing goes on normally and the frame pops by itself when exhausted. A handler that wants the generated declarations to reach the unit drives while (p_depth() != d0) top_add(parse_top());, which works from inside a function body, top_add appending to the unit list independently of the parser's position (nesting bounded by MAXOPEN).

The lookahead contract (to be stated in docs/surface.md). A push does not touch the pending lookahead token; the p_next() after the push discards it and reads the first token of the pushed source, so a handler must sit on the last token of its own construct when it pushes - exactly what do_directive does for #include (lex_include(path, line); next(); with cur still on the string). Two independent prototypes of this spec got it wrong first, which is why it is a rule.

Hygiene and guards. Substitution is by whole identifier, never by text, and the core never invents a name, so mangling is the module's; module temporaries use gensym_new() ($g<N>), and since the lexer cannot form an identifier containing $, capture is impossible. syntax_expr compares cp and tok_start(cur) around the callp and dies with syntax_expr handler consumed no tokens: <word> at the word's position, or syntax_expr handler produced no expression: <word> on a 0 return; syntax_infix needs no advance guard (the core consumes the operator first) but refuses a 0 return the same way. Both use err_at2.

Attribution. err_at prints lex_file(), which for a pushed frame is the string the module passed, so provenance costs zero core lines and beats a fixed one-level chain: the module composes Box__Circle__4 instantiated from prog.lx:15 and gets that prefix on every error inside, and nested instantiations compose because the module builds the name from the name it is already inside. Nodes built in the frame keep the string in nd_file, so a codegen error still names the instantiation after the frame popped. Gap: the line is relative to the generated text, so a module wanting the template's own line copies the span verbatim (line-for-line aligned) or emits a line map.

Determinism (docs/determinism.md rule 1). Every new table is a linear array in registration order, searched back to front where last-wins is intended; syntax_infix adds no table at all; no hashing, no pointer ordering, no iteration for output ordering. Instantiation memoization lives in the module, keyed by the argument lexemes in source order, so first-use order fixes generation order.

4. Feature to mechanism (all of M22) #

fn. The handler consumes fn, reads the name, builds the parameter list itself (so it accepts ref T x and class types and records each parameter's class), reads the optional -> T (absent = TY_VOID) and calls parse_function(ty, name, params); it holds the N_FUNC and rewrites the body before top_add, which is why on_func_end is redundant here. Gap: the body is parsed by core parse_stmt, so the module intercepts if/loop/return by walking the tree afterwards.

Classes, inheritance, virtual/override. Word 0 is the vtable, word 1 the refcount, the base's fields first; virtual takes a fresh slot, override reuses the base's (error naming the base when there is none), non-virtual methods are direct calls to a mangled Owner_m. Methods are parse_function with self prepended via param_new/list_append; the vtable is a generated global filled by a generated C_vt_init() with &Owner_m (M10 uptr). The class name is registered both as type_alias(name, TY_UPTR) (a type in params, globals, casts) and as syntax_stmt(name, &decl) (the module owns the local declaration and learns the local's class). Gap: register before the token after the body is lexed, or the next declaration dies with type expected at top level; and owning the declaration means owning local arrays of that type.

Interfaces. Unchanged from examples/api/oop.mc: a #define per slot, a generated dispatcher iface_m(self, ...) = callp(ld64(ld64(self) + IDX * 8), self, ...), a per-class vtable, a missing method as a compile-time error at the class's position. Only the spelling changes - obj.m(a) instead of iface_m(obj, a), through the . handler. Gap: none beyond M12.

Member access, assignment, methods, indexing. dot(left) reads the member name; the static class of left comes from the module's tables (a declared local or param) or a module side table keyed by node index for results it built. A method call lowers to callp(ld64(ld64(left) + SLOT), left, args) when virtual and to a direct call otherwise; a field lowers to ldW/stW at left + OFFSET. Member assignment works because = is deliberately not in the infix table: the Pratt loop has already stopped, so the handler peeks K_ASSIGN itself and emits the store, and core parse_stmt sees a plain expression statement. Gaps: the static type is known only for what the module declared, so an object crossing into core-parsed code is opaque; the side table must not live in nd_c/nd_d (dump_node walks those as node indices) and does not survive node_copy_subst.

Generics with where and const N. At the declaration the handler reads the parameter names, records the body with p_skip_balanced, stores it with the where clauses, registers syntax_stmt(name) and syntax_expr(name) for the use positions, and creates no class. At a use it parses < args > (splitting a closing >> with p_resplit_punct), builds the mangled name from the argument lexemes, returns early if that instantiation exists, checks each where clause against its own class/interface tables with its own error text, then calls p_subst_name("T", "Circle"), p_subst_int("N", 4), p_push_source("Box__Circle__4 instantiated from prog.lx:15", buf, len) and drives parse_top() until p_depth() returns; generic functions are the same from expression position. Constraints, mangling, memoization and inference are all module policy. Gap: a generic type in a core-parsed type position (core extern, a cast, a core function's parameter) has no hook; type_alias on the mangled name makes Box__Circle__4 work there, not Box<Circle,4> (see 5).

ref parameters. Call site: syntax_expr("ref", &h) requires an identifier and returns N_ADDR of it - exactly &y, which the core already lowers - after checking it is a local or a param. Callee: ref i64 x becomes an N_PARAM of TY_UPTR, and after parse_function the module rewrites the body in place, N_IDENT(x) -> ld64(x) and N_ASSIGN(x, e) -> st64(x, e') (value first, then the store), which also covers x += 1 from the prelude. Gap: ref a[i] / ref obj.f cannot use the ref spelling, because N_ADDR carries a name; M22 restricts ref to a local or a param.

Namespaces. namespace geo { ... } sets a module-side prefix and loops top_add(parse_top()) to the closing brace; the module's class and fn handlers consult the prefix when mangling, so reopening the namespace in another file merges by construction. import geo; is lex_include/p_push_source of geo.lx under the lookahead contract of 3, plus an implicit using; using geo; appends the prefix to a search list tried after the unqualified name; qualified geo.Circle is the . handler resolving against the module's namespace table, and a name that is both a local and a namespace is the ambiguity M22 asks to report. Gap: a namespace name registered as a word is reserved program-wide, so geo can never also be a variable - forced by the single lexer word table, not chosen. This is the least-proven row: every primitive is verified separately, the assembly is not.

Automatic memory. No new mechanism. syntax_stmt("{", &h) is accepted today (word_add refuses only K_U8..K_EXTERN = 256..269 and K_LBRACE is 272) and gives the module every statement-position block, nested ones included, without recursion (parse_block consumes its own brace) - that is what makes release per scope instead of per function. new C(...) is rt_alloc(C_SIZE) + vtable + rc = 1; x = e on a class-typed local becomes rc_inc/rc_dec around the store with a $gN temporary; each scope exit appends rc_dec for the names it declared; every return releases all live scopes except the returned name; rc_dec at zero calls virtual dispose, decrements class-typed fields, then frees to a size-class list in the program's rt.mc. Gaps: break N/continue must emit the right number of releases; the outermost block is tracked only because the module owns fn; cycles leak, as M22 says.

5. What stays out of the core, and why #

• syntax_type. Six core sites call type_of_token then do their own next(), so a consuming type hook changes that contract at every one, and parse_stmt calls it speculatively on a token that may not be a type - a handler that ate tokens during that lookahead is unrecoverable with one token of lookahead and no backtracking. That is why it is not eight lines, and M22 does not need it.
• on_func_end. Redundant once a module owns fn and receives the N_FUNC from parse_function; add it when a second consumer must transform core-parsed bodies.
• p_mark/p_reset (backtracking). Character rewind cannot cross a frame boundary and interacts badly with the lookahead; record-then-replay covers every legitimate need, forward-only. Cost: a taught language stays LL-ish with one token of lookahead.
• A lexer token hook (lex_hook). It runs before any scope exists: the qualified-name fusion it is built for silently rewrites a local named geo, and one pushback slot cannot back out of the three-token lookahead a.b.c needs. Wrong layer; p_resplit_punct is its narrow sound piece.
• A node type tag (nd_uty) and a core symbol table. They cost something on every compile, untaught ones included, which breaks inert-by-construction; a module that owns its declarations already knows its types.
• A generalized lvalue in parse_stmt/gen_assign. syntax_infix(".") handling = itself reaches the same result with no change to core semantics or core error text.
• Un-registering a word, scoped type names, overload resolution, separate compilation of templates, folding of template parameters (Box<T, N*2>), precise GC. Out: the first is structural (tok_add is append-only), the rest are where a surface mechanism becomes a second language inside the compiler - the risk docs/plan.md names.

6. Acceptance #

The demo is a toy language unrelated to M22, so generality is proven and not asserted. lib/user_syntax_demo.mc grows and lib/syntax_demo_test.mc keeps exiting 42:

• syntax_expr("bits", &h): bits i64 -> the type's width as a constant; reads a type in expression position, which no #prefix template can do.
• syntax_infix(".+", 9, &h): saturating add, alongside a #infix "<+>" in the same file - one program, one precedence order. syntax_infix("~>", 12, &h): p ~> len, right operand a name resolved in the module's own field table, including p ~> len = 3; and p ~> at(2).
• p_skip_balanced + p_subst_* + p_push_source: tmpl Pair<A, K> { ... } recorded, then make Pair<i64, 3>;, make Pair<u8, 2>; and a repeated make Pair<i64, 3>; memoized by the module. The body must contain (a) A slots[K]; - p_subst_int reaching parse_dim, (b) return "A is A"; - substitution never entering a string, (c) A_tag - substitution by whole identifier. p_resplit_punct: make Pair<Pair<i64,2>,3>; closing with >>.
• Attribution: tests/err/06x-tmpl.mc (outside scripts/test.sh, like 055 and 062) whose error reads Pair__i64__3 instantiated from <file>:<line>:N: <message>. Guards: tests/err/ cases for syntax_expr handler consumed no tokens, syntax_expr handler produced no expression, and a #infix redefining a taught operator dropping its handler.

scripts/check-surface.sh gains, after the existing M12 case: (1) the taught compiler compiles the extended lib/syntax_demo_test.mc with --exe and it exits 42; (2) the default compiler refuses the same source; (3) each tests/err/ case fails with the exact expected message; (4) the demo test compiled twice gives byte-identical objects; (5) inertness - with nothing registered, every tests/*.mc object and every --dump-ast output is byte-identical to the pre-M21 compiler's.

make check green (test, check-lex, check-ast, check-asm, check-obj, check-surface, test-exe, bootstrap at its fixed point); golden rewritten once; stage0 untouched and under budget; docs/surface.md Tier 3 gains the new registrations, the lookahead contract, and the note that a module side table keyed by node index must never live in nd_c/nd_d.

7. Decisions (owner and architect, 2026-09-03) #

1. Memory model (owner): reference counting with size-class free lists, cycles documented as a leak. Nothing in this spec depends on it.
2. Precedence table exposure (architect): yes. --dump-rules also lists every infix/prefix operator with precedence, associativity and whether a code handler is attached.
3. Colliding registrations (architect): refuse. A second syntax_infix on the same token is an error at user_init time (operator already taught: <tok>), consistent with #define and #rule duplicates being errors.
4. Instantiation budget (architect): raise MAXTOK to 2048 now (static array, no code); the other two limits follow 2.8.
5. Split delivery (architect): two commits — hooks (2.1-2.2, 2.6) then record/replay (2.3-2.5, 2.7) — each gated by the byte-identity check of 6(5) with the golden rewritten once per commit.

Edit this page