M41.5 — a module can participate in a function's parameter list

A follow-up in the style of M21.5 and M39.5: one gap a real consumer hit, closed with the smallest generic mechanism that closes it, and nothing else.

1. What was missing #

parse_params (src/parse.mc) accepted exactly one shape, type name, and it was the only position on the parse path with no hook at all:

• parse_top consults syntax_find on the declaration's first token, so a syntax handler either owns the whole declaration or sees nothing of it;
• word_add refuses the core type words (cannot redefine core keyword), so no keyed table — syntax, syntax_stmt, syntax_expr, syntax_infix, type_alias, type_new — can ever be reached from i64;
• on_stmt/on_jump run after a node exists and cannot consume tokens;
• syntax_lit is the numeric-literal position and nothing else.

The consumer is the ngen port of the teko language, written against docs/ against an unmodified compiler. It is blocked on two things it wants to spell with mc's own function syntax, and both are parameters:

i64 f(i64 x, i64 y = 10)     // a default parameter
i64 g(params i64 xs)         // a C#-style typed variadic

2. The decision #

The owner chose the hook over the features. mc does not grow default parameters and does not grow variadics; it grows the one registration that lets a module have them, the same way unless, class, 1.5 and rbit are not in the language either.

3. syntax_param(&f) #

Registration in src/hooks.mc, shaped like on_stmt's: a growable arena block (grow), an arena tag in the T_* registry (T_SYNPARAM, inserted after T_ONJUMP), a linear table walked in registration order — docs/determinism.md, rule 1.

Contract:

i64 f()      // -> the node index of an N_PARAM, or 0 = "the core handles this one"

Where it is consulted, and why there. At the head of parse_params's loop, right after the K_RPAR test and before type_of_token. That placement is the whole point:

• a parameter that opens with a word the module taught (params i64 xs) must reach the handler before the core demands a type — after type_of_token it is already type expected in parameter;
• a parameter that wants a trailer (i64 y = 10) must be read as a whole by whoever records the trailer: p_type(), p_ident(), then p_accept(K_ASSIGN) and parse_expr(0) for what the module keeps privately. A hook placed after the core had read type name would arrive with the = still unread and no way to attach it to anything.

Handlers run in registration order and the first non-zero answer wins. With none registered (nsynp == 0) the branch is not taken and not even the callp happens — the same short-circuit nonstmt, nonjump and nonlit have.

The name. syntax_param, not on_param: the on_* family runs after a node exists and cannot consume tokens, which is exactly what a default parameter has to do. This is a syntax hook that may own the parameter, like syntax_stmt and syntax_lit.

Three guards (the third added by the review, § 12), all raised at the parameter's own position:

The MAXPARAMS count and the at most 12 parameters diagnostic apply to what the handler returns exactly as they apply to what the core reads — the counting is outside the if.

4. p_decl_name() #

uptr p_decl_name() — the name of the declaration currently being parsed, 0 outside one. parse_top and parse_extern set it the moment the name is read (so it is already there when parse_params runs), parse_function sets it for the duration of the body and restores the previous value on the way out (§ 12), and top_add clears it when the declaration reaches the unit. void p_set_decl_name(uptr name) is the write side, for a handler that owns a declaration and reads the name itself (§ 12).

A handler needs it to know which function the parameter belongs to; without it a module could record a per-parameter fact and never say whose it was. lib/user_syntax_demo.mc keys its default table by it, which is what makes i64 f(i64 x, i64 y = 10) and i64 g(i64 x, i64 y = 30) — two declarations differing only in a default at the same parameter index — come out right.

The pointer is the one cur_name() made for that declaration and does not move while the declaration is read, so identity is a valid comparison and a handler can use it to notice that a new parameter list has started.

5. Acceptance #

1. A default parameter, end to end. i64 f(i64 x, i64 y = 10) spelled in the taught compiler; the module records the default from the handler and completes f(1) from a pass() with decl_find/decl_nparams. Exit 42.
2. A params-style parameter that opens with a taught word and lowers to uptr — the proof that the handler saw the word before type_of_token did.
3. p_decl_name() observed from inside the handler, with teeth: two functions differing only in the default at the same index.
4. The two guards, each a tests/err/ case with its exact message and a fixture handler in lib/user_syntax_demo.mc.
5. Inertness. With no syntax_param registered, --dump-ast and every object over tests/ are byte-identical to the frozen seed's (check-obj 32/32); and a module whose only registration is syntax_param and whose handler answers 0 for every parameter (lib/user_param_nop.mc) produces byte-identical trees and objects too.
6. stage0/ untouched; make check RC 0.

6. Implementation notes and deviations #

• cur_decl, not decl_name, is the global. src/parse.mc already has a function uptr decl_name(uptr msg) — the reader of a name in a declaration, #rule holes included — so the storage behind p_decl_name() is named cur_decl, next to cur_sect and cur_dylib.
• parse_extern sets it too, which the architect's design named only parse_top for. An extern is a top-level declaration and it calls parse_params; leaving it out would have made the documented contract ("the top-level declaration currently being parsed") false for exactly the declarations an FFI module cares about. One line.
• The two guards live in parse.mc, not in hooks.mc. run_syntax_param is the plain walk of the table, like run_syntax_lit; the guards need cp, tok_start, nnodes and nd_kind, all of which are the parser's and the AST's.
• The second guard uses err_at, not err_at2. By the time it fires the handler has consumed tokens, so cur_name() would name whatever token happens to be current — misleading. The position it reports is the parameter's first token, which is the useful one. The first guard keeps err_at2 with cur_name(), which there is the word, since nothing advanced.
• The demo handler claims every ordinary parameter and declines only on a non-type token and on void, so the core keeps type expected in parameter and parameter of type void word for word in the taught compiler as well.
• param_new puts the current token's line on the node, so a parameter the demo builds carries the line of the token after the name rather than of the type. Nothing prints an N_PARAM's line (dump_ast prints kind, type, name and val), and the demo uses the documented API on purpose; a module that wants the exact core line builds the node with node_new(N_PARAM, line, fl) itself.
• T_SYNPARAM renumbers the tags after it (T_BACKENDS..T_COUNT, 38 -> 39). Every use in src/ is symbolic; the visible effect is one more row in mc limits, named syntax_param, between on_jump and backends.

7. Cost #

git diff --numstat src/ for the core change:

M41.5 (second follow-up) — a core operator may be taught #

The same consumer, the same shape of gap: a registration mc accepts and then does not honour.

8. The defect #

syntax_infix("+", 10, &h) is accepted. word_add (src/hooks.mc) refuses only K_U8..K_EXTERN — the core keywords — and + is K_ADD, punctuation, outside that range. The handler is installed on the entry infix_set creates.

Then parse_unit() runs ops_init() as its first statement (src/parse.mc), ops_init goes through infix_set, and infix_set's last line is set_ie_fn(e, 0) — M21's rule that a #infix on a token drops the handler. Since user_init() runs before parse_unit(), the registration was wiped in silence.

Reproduced on this tree with a handler that cannot fail to be noticed:

// a throwaway module (not kept in the tree; the permanent proof is
// lib/user_coreop.mc), plus the two #include lines of a compiler that carries it
i64 probe_add(i64 left) { die("probe: the + handler fired"); return left; }
void user_init() { syntax_infix("+", 9, &probe_add); }

$ build/mc1 --exe mc_probe_add.mc -o build/mc-probe-add
$ build/mc-probe-add --exe prog.mc -o build/probe-prog    # i64 main() { return 1 + 2; }
$ ./build/probe-prog; echo $?
3

Exit 3: the die() never ran, 1 + 2 folded to 3. With the fix, the same two commands end at mc: probe: the + handler fired, exit 1. A module author reads the documentation, registers the operator, sees the program behave exactly as before and has nothing to debug — no diagnostic, no dump, nothing.

9. The decision: permit #

Not refuse. M24 already lets a module replace the machine slot that lowers + (lib/user_badmach.mc: v(50) + v(8) answers 42 instead of 58), so forbidding the same override one seam earlier — at the parser, where the operator is still syntax — would have been arbitrary. mc's position is that a module may take any decision the core makes, as long as the compiler that takes none behaves exactly as before.

Four requirements, all tested:

1. the core operator table exists before user_init(), so a syntax_infix on a core token finds the entry and installs its handler on it;
2. M21's rule is untouched — a #infix "+" written in the SOURCE still drops the handler (tests/err/066-infix-drops-handler.mc unchanged and still passing);
3. the refusal of a second syntax_infix on the same token stays (operator already taught);
4. with nothing registered, every object and every --dump-ast over tests/ is byte-identical to the frozen seed's.

10. The change: ops_init() is built on first use #

// src/parse.mc
i64 ops_ready = 0;

void ops_init() {
    if (ops_ready) return;
    ops_ready = 1;
    infix_set(K_OROR, 1, 0, 0);
    ...
}

// src/hooks.mc, in syntax_infix, before the lookup
    ops_init();

Five code lines: the global, the two-line guard, the call, and nothing else.

Why there and not next to user_init(). The obvious alternative is one line in src/cli.mc (ops_init(); user_init();). It was rejected for two reasons. The mechanical one: parse_unit() is reached from four entry points (src/cli.mc, src/driver.mc, src/astdump.mc, src/limits.mc), and user_init() from three of them, so "before user_init()" is not one place — it is three, and one of them (src/cli.mc) is being edited heavily on another branch. The real one: the table is an initialisation, not a phase. Making it idempotent and letting the first consumer trigger it means there is no ordering left to get wrong, in this compiler or in a recreated one that calls mc_main differently.

The guard also fixes a second, latent thing: ops_init used to re-fill the table on every parse_unit, so a second parse in the same process would have undone a #infix and every syntax_infix the first one had. No process parses two units today; now none can regress into it.

What the module's registration does to the entry. infix_set(tok, prec, 0, 0) + set_ie_fn(...) — the same two calls as before, now applied to the core's own row. So:

Two consequences worth writing down rather than hiding:

• word_is_taught(K_ADD) becomes true in that compiler, so err_name may answer name reserved by a syntax/type_alias registration: + where a name was expected. In a compiler that taught +, that is the truth.
• the rewrite is unconditional and program-wide. lib/user_coreop.mc says so in its own header: the functions its + lowers to cannot use + themselves, because they would call themselves. A module that wants an operator for its own type alone has to look at the operands and rebuild the core's node when they are not its own — which is a module's business, not the core's.

11. Proofs #

lib/user_coreop.mc (the module) + lib/mc_coreop.mc (the compiler that carries it), the lib/user_badmach.mc / lib/mc_badmach.mc shape, and scripts/check-surface.sh:

No tests/err/ case was added: the change introduces no new message. 066-infix-drops-handler.mc is unchanged and still passes with its exact text; the core-operator half of the same rule is asserted in check-surface.sh, because it needs a compiler that taught + and tests/err/ runs against the demo compiler.

12. Review findings and their fixes #

Three findings from the PR review of the two parts above. The whole code change is in src/parse.mc (+45/-2, 19 of the added lines neither comment nor blank); stage0/ was not touched.

(A) A handler that consumed tokens and then returned 0 was believed — HIGH #

param_syntax() ran its "consumed no tokens" guard only when the handler had returned a node:

i64 p = run_syntax_param();
if (p == 0) return 0;                        // ... with the cursor wherever the handler left it
if (cp == cp0 && tok_start(cur) == t0) err_at2(...);

0 means "the core handles this one", and the core then reads the parameter position it was standing in — but the handler may have moved it. Reproduced with a handler that reads peat i64 x and the comma after it and answers 0: i64 f(peat i64 x, i64 y, i64 z) came out of --dump-ast as

FUNC type=i64 name=f
  PARAM type=i64 name=y
  PARAM type=i64 name=z

a two-parameter function that compiled clean, linked, and returned 42 for f(4, 2) — a three-parameter declaration running with the wrong arity, no diagnostic anywhere.

The fix compares the cursor and the token start on both answers. When the handler declined after consuming, that is syntax_param handler consumed tokens and returned 0: <word>, at the parameter's own position; when it claimed without consuming, the message is the one M41.5 already had. The word in the detail is copied from the token the handler was given (xstrdup(t0, l0)), not from cur_name(), because by then the current token is somewhere else.

syntax_lit had the same latent shape (M24) and is fixed in the same commit: a handler may move the cursor with p_take_lit and then answer 0, and parse_primary would build its N_INT out of tok_val(cur) — a token whose span no longer covers what was read. Reproduced with a handler claiming a literal that ends in q: return 7q; compiled clean and exited 7, the q swallowed. Now syntax_lit handler consumed tokens and returned 0: <literal>.

The check is at the end of the chain, not per handler, because run_syntax_param/run_syntax_lit live in src/hooks.mc, which is included before src/parse.mc and cannot see cp or cur. A module that wants its consuming handler diagnosed must therefore not register a claiming handler after it — which is what lib/user_syntax_demo.mc does: sd_peat and sd_leat are registered last, on purpose, and their headers say why.

tests/err/073-param-consumed-zero.mc and tests/err/074-lit-consumed-zero.mc are the two cases, each asserted with its exact message in scripts/check-surface.sh. Both files are the repro: the 073 source is the one that used to compile clean and exit 42.

(B) p_decl_name() was blind inside a handler that owns the declaration — MEDIUM #

cur_decl was set in parse_top and parse_extern — the two places the core reads a declaration's name. A syntax handler that parses a container and declares each member with the public parse_params() + parse_function() reads those names itself, so p_decl_name() answered the enclosing declaration's name, or 0, for every member — while docs/reference/hooks.md recommended keying syntax_param bookkeeping by exactly that value.

Two changes:

• parse_function(ty, name, params) sets cur_decl = name for the duration of the body and restores the previous value on return. Restoring rather than clearing is what makes a declaration generated from inside a body (p_push_source + top_add) leave the outer one intact.
• void p_set_decl_name(uptr name) joins the public API: a handler that owns a declaration announces the member before calling parse_params(). p_decl_name() is trustworthy on the parse_top / parse_extern / parse_function path and only there, and the row in docs/reference/hooks.md says so.

The proof is in lib/user_syntax_demo.mc: capsule Name { … } declares its members with the public API, and the two members carry a default at the same parameter index with different values. With the announcement, --dump-ast shows INT val=10 in one call and INT val=30 in the other and the program exits 42; with the p_set_decl_name line commented out, the module's own guard fires — a default parameter needs a named declaration. (The container is capsule and not box because lib/syntax_demo_test.mc already declares a global named box, and a syntax registration reserves its word for the whole program.)

(C) The documented message text was missing the detail — LOW #

All four consumed no tokens messages are err_at2 calls and print : <word>; docs/reference/diagnostics.md wrote all four without it. The four rows — and syntax_expr handler produced no expression and syntax_infix handler produced no expression, which are err_at2 too — now carry the suffix, so the table matches what the compiler prints.

Edit this page