Spec M31 -- concurrency taught by a module: spawn, intent, await

Owner's test (2026-09-03): a developer must be able to add, from the surface, (1) threads with mutexes and semaphores, (2) spawn f(a) fire-and-forget talking over channels, (3) await res = f() with no async, where the call widens to Intent/Intent<T> and intent is legal only on a local. Non-negotiable: nothing about threads, channels, await or intents enters src/.

Design panel: three teams built working taught compilers against an unmodified build/mc1. This spec is the synthesis; the numbers below are theirs, on darwin arm64.

1. Verdict #

Yes. The core as it stands hosts the whole feature with zero changes to src/. Three independent spawn/intent/await compilers ran: one a parallel sum at 2.9x over serial, one examples/lang with atomic reference counting and its 14 tests still green, one a 100 000-object churn across four threads in a 4 MiB arena with live() == 0.

The hardest-sounding requirement -- await without async, in any function -- is free on this core under every design examined: await lowers to an ordinary call to a blocking runtime function. No CPS transform, no colouring, no core change. And the intent restriction is structural, not checked: registering the word with syntax_stmt and never with type_alias means it is not a type in any grammar position, so a return type, a parameter, a field or a generic argument reject it by construction (docs/surface.md, "Registration reserves the word for the whole program").

Three core gaps are real. None blocks a first delivery; all three are cheap, generic, and justified by languages that have nothing to do with threads. 2.4 lists what the panel claimed and this spec refuses, with evidence.

2. Core gaps #

2.1 decl_find -- ask the core about a declaration it already parsed #

i64 decl_find(uptr name);              // index of the N_FUNC/N_PROTO/N_EXTERN, -1 if none
i64 decl_ret(i64 d);                   // declared return type (TY_*)
i64 decl_nparams(i64 d);               // arity
i64 decl_param_type(i64 d, i64 i);     // i-th parameter's type

Widening is the feature, and it needs the callee's declared return type: void f() gives Intent and a trampoline that cannot bind a result (a naive version dies with value of type void), while a class-typed T decides whether the awaited local is reference counted. Two teams got this working only by reading unit_head, declared at src/parse.mc:132 and not inside the "public parser API (Tier 3)" block that starts at src/parse.mc:1629 -- a module reaching into a parser internal, and one that sees only what has been parsed so far.

Cost ~20 lines in src/parse.mc (a linear walk of unit_head plus four accessors), documented next to the Tier 3 block. Beyond concurrency: an FFI marshalling module generating glue for an extern needs every parameter type; M26 doc extraction and M28's LSP need the signature the core parsed.

2.2 on_jump -- a hook on the exit edges of a scope #

void on_jump(uptr fn);                 // i64 f(i64 n, i64 kind, i64 depth)

Called where the core creates an N_RETURN / N_BREAK / N_CONTINUE node, before any on_stmt hook and before another module can wrap it; kind is the node kind, depth the count of blocks open in the current function. Handlers run in registration order and may return a replacement.

Two teams independently built the same lock (m) { ... } by appending the unlock after the body and both reproduced the same defect: a return inside the body jumps over the unlock and the next call hangs forever. on_stmt is not a substitute -- the host module (examples/lang) already rewrites N_RETURN into a block for its own reference counting before a later-registered hook sees it, so a second module can neither recognise the jump nor place code without disturbing the host's release order: exactly the bookkeeping the memory model has for itself and a second module cannot get.

Cost ~35 lines (a depth counter in parse_block, a table shaped like on_stmt's, three call sites). Beyond concurrency: a defer / scope-guard module for any host language -- Go, Swift, Zig and D all put this at language level precisely because it must cover every exit edge -- and any coverage or tracing module needing a probe on early returns, not just at the closing brace.

2.3 A written and tested ABI contract #

No new mechanism: documentation plus one check-surface case. Four invariants carry every taught runtime, and violating one fails at run time with no diagnostic. (a) Parameters arrive in x0..x7 and the prologue does not clobber them, and (b) the epilogue leaves x0 alone -- both documented (lib/sys_svc.mc). (c) A zero-parameter, zero-local function has frame == 0, so gen_func NOPs the sub/add sp pair (src/gen_arm64.mc:1453) and its epilogue is exactly ldp x29, x30, [sp], #16 ; ret. (d) Generated code never writes x18..x28 (REG_BASE 9, REG_MAX 6, REG_TMP 8, REG_S1/S2 16/17 -- src/gen_arm64.mc:42-44; a full disassembly of build/mc1, 54 891 lines, mentions none of them). (c) and (d) are true and unwritten: an M17 allocator spilling depths into x19..x28, or a leaf-function optimisation dropping the stp, would silently corrupt every taught runtime.

Cost ~20 lines of test (grep --dump-asm for that epilogue; assert no x18..x28 destination in --dump-asm of src/mc.mc). Beyond concurrency: M30's DWARF unwinder, any debugger or profiler, and any future stackful-coroutine module read or fabricate frames on exactly these assumptions.

2.4 Claimed and refused #

3. Runtime design #

All of it is ordinary .mc in the module: externs, #opcode words, plain functions.

Platform layer, one #include per system, exactly as lib/sys.mc vs lib/sys_linux.mc. macOS: pthread_create/join/detach/self, pthread_mutex_*, pthread_cond_* from libSystem and -- since POSIX unnamed semaphores are stubbed out there -- dispatch_semaphore_create/wait/signal (DISPATCH_TIME_FOREVER spelled 0 - 1). Linux/musl (M16): the same pthread_* from libc.a plus sem_init/wait/post. Windows (M19), note only: CreateThread, SRWLOCK, WaitOnAddress. An mc uptr f(uptr) is a C void *(*)(void *) -- one integer in, one out -- so &f (M10) is a valid thread entry with no shim; opaque structs are reserved as bytes and passed by address (u8 m[64] for pthread_mutex_t, 56 + 8 on LP64; u8 c[48] for pthread_cond_t).

Atomics are whole functions whose operands happen to be in x0..x7 -- the lib/sys_svc.mc pattern, since #opcode folds constants only (#opcode argument not constant, src/gen_arm64.mc:870). These encodings were verified by three teams against Apple's disassembler, which prints ldaddal x1, x0, [x2], casal x0, x1, [x2], dmb ish:

#opcode movx(rd, rm)         0xAA0003E0 | (rm << 16) | rd          // orr rd, xzr, rm
#opcode ldaddal(rs, rt, rn)  0xF8E00000 | (rs << 16) | (rn << 5) | rt
#opcode casal(rs, rt, rn)    0xC8E0FC00 | (rs << 16) | (rn << 5) | rt
#opcode dmb_ish()            0xD5033BBF

i64  a_add(uptr p, i64 v)        { movx(2, 0); ldaddal(1, 0, 2); }   // returns the old value
i64  a_cas(i64 e, i64 n, uptr p) { casal(0, 1, 2); movx(0, 0); }     // returns the observed old
void a_fence()                   { dmb_ish(); }

Four threads x 200 000 increments give exactly 800 000 where the plain ld64/st64 version loses 570 000. LDADDAL/CASAL are ARMv8.1 (LSE); an ldaxr/stlxr retry loop split across two one-word #opcode functions is forbidden -- the intervening frame store and ret may clear the exclusive monitor, and it passed 5/5 on Apple silicon anyway, a silent portability trap. Until M24's #machine, the module is LSE-only and says so at build time.

Reference counting becomes rc_inc(p) = a_add(p + 8, 1), rc_dec(p) = a_add(p + 8, -1); the -AL variants give the acquire/release pair, so the counts need no separate fence. One team made exactly this change in examples/lang/lib/rt.mc (8 lines added, 3 changed) and the suite, churn test included, stayed green. The allocator is the other half: rt_alloc/rt_free walk unsynchronised free lists that two threads corrupt, so M31 ships one mutex around both (4.2 ns uncontended); per-thread arenas are section 8 question 5.

Channel: ring buffer + mutex + not-empty/not-full condvars; chan_recv returns 0 once closed and drained so a receive loop terminates, chan_close broadcasts both. Unbuffered round trip 2.2 us, buffered handoff 33 ns/item.

Intent: an ordinary reference-counted object of the host language -- word 0 a vtable whose slot 0 is the release function, word 1 the count -- so every existing rule (scope exit, break N, return, ref refusal) applies to an intent local with no new code. Fields: state (queued/running/done), fn, nargs, flags (result owns a reference / must be dropped), result, taken, runner slot, 8 packed argument words, 8 ownership tags; 208 bytes, inside the 256-byte free-list ceiling so intents recycle. No per-callee trampoline is needed: callp takes the pointer plus up to 7 arguments (callp expects 1 to 8 arguments, src/gen_arm64.mc:918), so one arity switch covers every callee. Past 7 arguments, or for a non-i64 result once M24 lands, the module generates a typed trampoline with p_subst_name

• p_push_source + top_add, verified to work from inside a statement handler mid-function-body.

Dispatch -- the decision. intent x = f(a) submits eagerly, at the call, to a pool of N workers. Lazy submission (record a thunk, run it at the await) makes await res = f() identical to res = f() and the feature vacuous: four 300 ms tasks as intents finish in 0.564 s against 1.513 s called directly. Eager submission alone starves a fixed pool the moment a task awaits, so await steals: it CASes the intent from QUEUED to RUNNING and, winning, runs it inline on the awaiting thread while the worker that later pops it skips it (verified: 8 self-awaiting tasks on 4 workers, no hang). Consequences, all intended -- back-to-back await r = f() degrades to a direct call plus bookkeeping; parallelism appears exactly when intents are left in flight; user code cannot assume which thread it runs on. spawn uses the same pool, but a spawned task is never awaited and so never stolen: the pool grows a worker when every worker is blocked in a runtime wait, capped at MAXTHREADS.

4. Surface design #

Registrations, all in user_init, all existing hooks: syntax_stmt for spawn, intent, await and lock, plus syntax_stmt("{") chained over the host's own handler (2.4, row 1), plus on_stmt and on_jump (2.2).

spawn_stmt  = "spawn" ident "(" [ expr { "," expr } ] ")" ";"
intent_stmt = "intent" ident "=" ident "(" [ expr { "," expr } ] ")" ";"
await_stmt  = "await" [ ident "=" ] ( ident | ident "(" [ expr { "," expr } ] ")" ) ";"
lock_stmt   = "lock" "(" expr ")" block

Lowering is one node each, folded into a single expression because a syntax_stmt handler returns one node and an N_BLOCK would scope the binding away (each helper returns the intent):

intent x = f(a,b)  ->  uptr x = it_submit(it_arg(it_arg(it_new(&f,2,fl),0,a,o0),1,b,o1));
spawn f(a)         ->  it_go(it_arg(it_new(&f,1,fl),0,a,o0));
await x;           ->  it_drop(x);
await r = x;       ->  T r = it_take(x);
await r = f(a)     ->  T r = it_call(it_submit(...));      // the intent is a temporary

T comes from decl_ret(decl_find(f)) (2.1), or from the host module's own table when the host declared f; void gives Intent and binding it is refused. For a class-typed T the flags say the result owns a reference, so await x; on such an intent must drop it, not merely take it (a first version leaked one object per discarded await).

The restriction. intent is a statement word and never a type_alias, so fn f(intent x), -> intent, class C { intent x; } and Box<intent> are unreachable. What remains is message quality: unaided the core says type expected in parameter, and three lines at the top of the host's own type reader fix that. The tag class the word widens to must be unnameable -- spell it $Intent, since the lexer never forms an identifier containing $; a nameable Intent lets fn h() -> Intent smuggle one out of a function, which one team verified is accepted today. That is the one fidelity hole the panel found, closed by naming, not by a core mechanism.

Errors the module owns: `intent` is not a type: it may only declare a local; an intent must be initialized by a call: x; the callee of spawn/intent/await must be a named function; unknown function: f; intent is never awaited in this scope: a; this intent was already awaited: a; this intent has no value to bind: use "await x;": a; await expects an intent: z; await while holding a lock: release it first. At run time: deadlock: await cycle, intent awaited twice, await of a null intent, await of a released intent, chan_send: the payload must be a reference counted object, not a scalar (a chan is an alias of uptr, so the payload rule -- objects, never scalars, because the send takes a reference of its own -- is enforced by the runtime and not by a type).

5. Safety #

• Ownership transfer. A direct call borrows because the caller outlives the callee; an intent's callee may run after the caller's scope is gone, so it_arg(it, i, v, own) captures -- own = 1 increments a borrowed value, own = 2 moves an already-owned temporary in with no increment -- and the intent's release drops what it holds. A channel likewise takes a reference of its own on send and hands it out on receive, so the sender's scope exit cannot free an object the consumer still holds. This, not atomic counts, closes the producer/consumer race: atomic rc_inc/rc_dec fixes the counter and does nothing for the transfer. It also fixes the documented lx leak "an owned temporary that is never bound leaks one reference".
• Dispose runs on whoever drops the last reference (the Arc/Drop rule). Under one allocator lock that is a convoy; under per-thread arenas it becomes memory migration and needs an owning-arena header -- which is why arenas are deferred rather than assumed cheap.
• Dropping an un-awaited intent blocks. it_release joins before freeing; the alternative is a worker writing into freed memory. The never awaited in this scope check (raised from the chained { handler) catches the straight-line case; a branch that awaits on only one path reaches the blocking release, which is safe and must be documented as loudly as the check. That check is textual, not a lifetime guarantee (found during delivery, 2026-09-03): it reads whether await was written in the block, never what became of the intent's value, so uptr saved = x; before the closing brace smuggles the pointer out -- an untyped local takes no reference -- and the block is freed at the scope's close all the same. The aliasing is the host language's own trait and reproduces in plain examples/lang with an ordinary object and no module on top; what the pool adds is that the freed block is handed to a live intent on another thread. it_release therefore poisons the state word before rt_free, so an intent operation on a stale pointer panics with await of a released intent rather than reading a stranger's block (examples/conc/tests/21-escaped-intent.lx). A raw ld64 on the escaped address is still a raw read of freed memory, and no lint in a language with ld64 changes that.
• Deadlock diagnostics. Each thread has a slot; a blocked thread records the intent it waits on, a running intent records its runner. Before each wait, walk the wait-for chain and panic with deadlock: await cycle if it returns to self (verified on a real two-task cycle, exit 70). Stealing cannot break such a cycle, so the detector is not redundant. await inside lock is refused at parse time -- the module owns both words, so it keeps a depth counter.
• Not solved, by construction: no Send/Sync, no data-race analysis. spawn f(&shared) compiles and races. A module sees only what it parsed, and M21 sec. 5 declined the mechanisms that would change that.

6. Acceptance #

1. Producer/consumer over a channel of class objects. The producer sends N objects and drops its own reference immediately; a consumer on another thread receives, uses and releases. Asserts the received sum and live() == 0 after a quiescence point.
2. Parallel sum, mutex vs atomics. The same partitioned sum over 8 M elements at 1/2/4/8 threads, once under a mutex and once with a_add, identical totals at every width, plus the racy version demonstrating the wrong answer.
3. await chain. intent a = f(x); intent b = g(y); await ra = a; await rb = b; plus a fused await w = h(z); and an Intent (void callee) discarded with await w;.
4. Error tests, one per intent restriction: return type, parameter, field, generic argument, plus fn h() -> $Intent proving the tag is unnameable; plus never-awaited, awaited-twice and await while holding a lock. Each asserts the exact message and exit 1.
5. spawn of a generic function. spawn Box<Circle,4>_fill(b, 3); through the host's own instantiation, proving the mangled callee resolves and the arity switch carries self.
6. Determinism. The same source compiled twice gives byte-identical .o and --exe for the same basename, --dump-asm diffs empty; every table the module memoises (trampolines, intent types) is linear and in first-use order, never a hash walk (docs/determinism.md rule 1).
7. Delivered under examples/ (a module for lx, or a sibling example) with its own test.sh in make check; src/ byte-for-byte unchanged, proved by check-obj 32/32 and the bootstrap fixed point.

Test hygiene: live() is racy once threads exist, so every assertion on it needs a quiescence point, and every concurrent test asserts a deterministic total (a sum, a count) or pins the pool to one worker. Program output under a scheduler is not deterministic; compiled output is.

7. Out of scope, with reasons #

• Green threads / M:N. A stackful runtime is encodable today -- one team built one (23 ns per switch, 20 000 live fibers, real migration between OS threads) on a four-instruction #opcode context switch. It stays out because it rests entirely on the unwritten invariants of 2.3; growable stacks are impossible (&x is an opaque uptr, so no roots, so no relocation: ~10^5 fibers at 64 KiB, not 10^6); a blocking syscall stalls a whole worker with no netpoller; and preemption is cooperative with real holes. Revisit after 2.3 lands, as its own milestone.
• Cancellation and timeouts, which change the typing rules (the result type of a cancelled intent, of a timed-out await) and were not prototyped. select, structured concurrency, and a typed chan<T> with member syntax -- the last wants the chainable syntax_infix of 2.4; free functions ship first. Floats through callp: everything packs as 8-byte integers until M24, after which a typed trampoline per callee costs the module nothing new once 2.1 exists.

8. Decisions (architect, 2026-09-03) #

1. Order of work: land 2.1, 2.2 and 2.3 first (about 75 lines, all generic), each inert for untaught programs (check-obj 32/32), then the example.
2. Host: a sibling example examples/conc whose taught compiler is examples/lang/lang.mc plus conc.mc ([compiler] modules = ["../lang/lang.mc", "conc.mc"]), so channels carry lx class objects, Chan<T>/Intent<T> use lx generics, and the ownership convention is tested where it matters. examples/lang itself stays unchanged.
3. Reserved words: accept spawn, await, intent, chan, lock as program-wide words of the concurrency module; documented in its README with the rename advice.
4. LSE: arm64 with LSE only; the module refuses to compile for a target without it with a clear message; ldaxr/stlxr sequences arrive with M24's #machine.
5. Allocator: one global mutex around rt_alloc/rt_free now; per-thread arenas are a later milestone that needs the reserved register of 2.4.

Edit this page