Spec M42 -- --exe on every host: the ELF executable writer, dynamic first

mc --exe writes a Mach-O executable and nothing else. src/cli.mc used to hardcode bname = "macho-exe"; the fix batch that precedes this milestone makes it resolve the host pair through the target registry and REFUSE when the executable slot is 0. That refusal is honest, and it is a step, not a destination: the slot is 0 for linux/aarch64, linux/x86_64, windows/aarch64 and windows/x86_64 because nobody has written the writer, not because the design forbids one.

The consequence is a promise that is false off macOS. docs/bootstrap.md prints standalone: the binary alone is the toolchain -- and on Linux and Windows it is not: the binary writes an object and an external ld.lld or lld-link finishes the job.

This milestone is step 1: ELF, both Linux architectures, dynamic linking first, with the static no-import case falling out as the degenerate form. PE/Windows is step 2 and a separate spec.

What already exists #

Why dynamic, and why it comes first #

The owner's requirement, and the reason is not convenience:

Dynamic linking removes the sysroot from the cross-compilation path. A dynamic executable needs no crt1.o (we bring our own _start), no crti.o/crtn.o (no init/fini arrays) and no libc.a (the library is resolved at load time). What it needs instead is names: the interpreter path, the DT_NEEDED soname, and the symbol names -- none of which is a file to download. This is the same insight that let scripts/sysroot-windows.sh build kernel32.lib with llvm-dlltool and no Windows SDK: an import list is a list of names.

So a Linux binary becomes cross-buildable from macOS with no Docker, no apk, and no cached sysroot. That is a bigger prize than --exe itself.

The static case is then not a separate feature: a program with no undefined symbols simply gets no PT_INTERP, no PT_DYNAMIC and no relocation sections. It falls out.

Design #

0. The risk is proved before the writer is written #

_start without crt1.o, in a DYNAMIC executable, is the one assumption that can sink the milestone, and it is cheap to test. Before a line of the writer exists, hand-assemble a dynamic ELF64 for each architecture -- our own _start, PT_INTERP naming the musl loader, one DT_NEEDED on libc.so, one imported symbol called through the PLT -- and run it under docker run --platform linux/arm64|linux/amd64 alpine:3.

This is the M39 discipline: there, the two-instruction context switch was written by hand and QEMU-tested before the RISC-V machine existed.

What the probe decides:

The probe's transcript goes in the spec's implementation notes either way.

Result (architect, 2026-09-04, run before this spec was committed). The probe was built -- scratchpad/m42probe/mkprobe.py, a struct.pack emitter with every field explicit, no linker and no assembler on the artefact -- and run on all four cells:

libc / archcontainerresult
musl / aarch64alpine:3 (3.24.1)PASS: ok, exit 42
musl / x86_64alpine:3PASS
glibc / aarch64debian:bookworm-slim (glibc 2.36)PASS
glibc / x86_64debian:bookworm-slimPASS

The glibc oracle has since moved to ubuntu:latest (post-merge review): the repository's Linux baseline is the newest Ubuntu on a 7.x kernel, and debian:bookworm-slim is two glibc releases behind it. Re-run 2026-09-04 on ubuntu:latest = Ubuntu 26.04 LTS, glibc 2.43, both architectures: PASS on both, unchanged. Every glibc oracle in scripts/ and in the tables below is ubuntu:latest; alpine:3 (3.24.1) stays the musl one.

A crt-less _start in a DT_BIND_NOW ET_EXEC calls write through a one-instruction PLT stub on every one of them. Neither libc needed __libc_start_main, so glibc is NOT deferred and decision 5 below is amended. Measured alongside: getconf PAGESIZE is 4096 in all four containers and a p_align of 64 KiB loads and runs on all four too, so 64 KiB is the safe alignment for aarch64 at zero cost; the SysV hash the emitter computes is array-identical to what ld.lld --hash-style=sysv produces for a reference binary of the same shape (nbucket=5, buckets and chains equal, both architectures); and a GOT of exactly ONE slot with .dynamic immediately after it is safe under BIND_NOW -- a --dump-got variant shows every loader writing only the slot and leaving the following words untouched. Each probe is 4312 bytes and two emissions are cmp-identical.

Not measured, and therefore carried into § Acceptance and § Risks rather than assumed: a 64 KiB-page aarch64 kernel (none available here); a libc call that needs more initialisation than write -- errno is TLS in both libcs, malloc and stdio need their own state -- which the writer's acceptance must exercise explicitly; and anything about PIE.

1. elf-exe and elf-exe-x86_64 #

Two backends over the same gen_lower + gen_encode_all every other writer consumes, registered in src/core_writers.mc and placed in the executable slot the two Linux targets leave at 0:

backend("elf-exe", &backend_elf_exe); backend("elf-exe-x86_64", &backend_elf_exe_x86); target("linux", "aarch64", "elf-obj", "elf-exe"); target("linux", "x86_64", "elf-obj-x86_64", "elf-exe-x86_64");

The object backend keeps picking the machine (machine_use as its first statement, the M17 step B rule); the executable backend does the same.

2. ET_EXEC at a fixed base, and DT_BIND_NOW #

Two simplifications, both with the M11 precedent of refusing the optional half of the format:

3. What the writer emits #

Segments, in one pass, page-aligned to 4 KiB (both Linux architectures accept 4 KiB; 64 KiB pages on aarch64 are a kernel configuration and p_align must be the maximum the image is valid under -- decide with the probe):

program headercontents
PT_PHDRthe program header table itself
PT_INTERPthe loader path, when there is at least one import
PT_LOAD r-xELF header, phdrs, .interp, .dynsym, .dynstr, hash, .rela.plt, .text, .plt, .rodata
PT_LOAD rw-.data, .got, .dynamic, and .bss as the gap between p_filesz and p_memsz
PT_DYNAMICthe .dynamic vector
PT_GNU_STACKPF_R | PF_W, never PF_X -- the executable-side counterpart of the .note.GNU-stack the fix batch adds to objects

The .dynamic vector carries DT_NEEDED (one per library), DT_STRTAB, DT_SYMTAB, DT_STRSZ, DT_SYMENT, DT_HASH, DT_PLTGOT, DT_JMPREL, DT_PLTRELSZ, DT_PLTREL, DT_FLAGS with DF_BIND_NOW, DT_FLAGS_1 with DF_1_NOW, DT_NULL.

DT_HASH and not DT_GNU_HASH: the SysV hash table is a handful of lines and every loader accepts it, while DT_GNU_HASH is a bloom filter plus sorted buckets for a lookup speed that does not matter at our symbol counts. A single-bucket table is legal; write the real one anyway, it is cheaper than explaining the shortcut.

Relocations resolved by the writer, in place, exactly as backend_exe.mc does for Mach-O: every reference to a defined symbol becomes a final address. Only references to UNDEFINED symbols survive into .rela.plt as R_AARCH64_JUMP_SLOT / R_X86_64_JUMP_SLOT against a .dynsym index.

4. Where the imports come from #

#dylib "path" (M12) is the existing surface for naming a library, and [libs]/[externs] in mc.toml (M14) is the project-level form. Both already carry what a DT_NEEDED needs. The interpreter path is the one new fact, and it is per-target and per-libc, so it is a TOML key with a default rather than a constant in the writer -- libc.so and /lib/ld-musl-<arch>.so.1 measured in alpine, with glibc's /lib/ld-linux-aarch64.so.1 and /lib64/ld-linux-x86-64.so.2 documented as the alternative.

5. The static case is the degenerate case #

A program whose undefined-symbol set is empty -- everything built on lib/sys_linux.mc, including tests/linux/070-nolibc.mc -- gets no PT_INTERP, no PT_DYNAMIC, no .dynsym, no .rela.plt and no PLT. The writer takes that path by counting imports, not by a flag.

Out of scope #

Files and estimated deltas #

filedelta
src/backend_elf_exe.mc+600..800new; the writer, both architectures
src/core_writers.mc+4 / -2two backend() calls, two target() slots filled
src/backend_elf.mc~0reused as-is; if anything is shared, it moves to src/objmodel.mc
tools/bundle.list+1mc/backend_elf_exe
scripts/test-linux.sh+40a --exe mode beside the object+link mode
docs/reference/objects.md, bundle.md, cli.md, build.md, docs/guide/50-cross-compile.md, docs/bootstrap.mdthe M15 promise corrected

stage0/ untouched. The five goldens move once (the bundle grows), under the usual rule.

Acceptance #

  1. It runs. The whole tests/*.mc suite built with mc --exe for linux/aarch64 and linux/x86_64 and EXECUTED in alpine:3 under Docker, exit codes and stdout compared against the macOS run -- the same corpus scripts/test-linux.sh runs today through ld.lld, with the linker removed from the path.
  2. The dynamic case is real, not simulated. At least one test imports from libc.so through PT_INTERP + DT_NEEDED + a JUMP_SLOT, and llvm-readelf -dr shows all three. A binary that imports nothing shows none of them. And one test goes past what the § 0 probe measured: it makes a libc call FAIL and reads errno (TLS), and it calls malloc -- proving the loader's own initialisation is enough for the state those need, with no crt object.
  3. No sysroot. The cross build of (1) runs with ~/.mc/sysroots/linux-* DELETED and no [linker] in the config. That is the milestone's whole point and it is a one-line proof.
  4. The stack is not executable. llvm-readelf -l shows GNU_STACK with RW and never E, on every binary in (1).
  5. Determinism. Two builds of the same source are cmp-identical; no clock, no path, no pointer order in the output.
  6. macOS does not move. make test-exe 32/32 with byte-identical binaries, check-obj 32/32 against the frozen seed, scripts/check-inert.sh clean across the change.
  7. Windows still refuses, with the message the fix batch introduced -- this milestone fills two of the four zero slots, not four.
  8. Docs. make check-docs green; docs/bootstrap.md's standalone claim now says on which hosts it holds.

Risks #

  1. _start without crt1.o in a dynamic image. The whole milestone rests on it; § 0 proves it before anything is written. Mitigation if it fails on musl: call __libc_start_main and pay the crt dependency back -- which would also mean the sysroot does NOT disappear, and the milestone's main prize is lost. Prove it first.
  2. glibc differs from musl. Contained by shipping musl first.
  3. Page size on aarch64. A 64 KiB-page kernel rejects a 4 KiB-aligned image. The probe showed a 64 KiB p_align costs nothing on the 4 KiB kernels available here, so the writer uses 64 KiB on aarch64 and 4 KiB on x86_64; a real 64 KiB-page kernel was NOT measured.
  4. DT_HASH correctness. A wrong bucket count silently resolves to the wrong symbol. Cross-check every table against llvm-readelf --hash-table of a clang-built dynamic binary.
  5. The writer duplicates backend_exe.mc in structure. Resist merging them prematurely: the formats diverge in every field. Share only what is genuinely format-neutral, and put it in src/objmodel.mc where M41 put the rest.

Implementation notes (written while building it) #

Every deviation from the design above, with its reason. Nothing in the design had to be abandoned: the probe's layout survived contact with the code and the writer emits its shape.

  1. Section headers ARE written (the § 3 table left the choice open, and the probe ran without them). No loader reads them -- PT_LOAD and PT_DYNAMIC are the whole contract -- but they cost a few kilobytes and they are what makes llvm-readelf --sections, llvm-nm, llvm-objdump -d and a debugger's backtrace read an mc binary, and what lets --dump-syms be compared against the file. A full .symtab/.strtab with final addresses goes with them, in the same spirit as macho-exe's LC_SYMTAB. An undefined symbol stays SHN_UNDEF with value 0 there rather than being aliased to its stub.
  1. The entry point. § 0's probe hand-wrote _start; the writer has to produce one. A program that defines _start itself keeps it (#include <sys_linux>, tests/linux/070-nolibc.mc), and anything else gets a synthesized .text.mcstart -- seven AArch64 instructions (28 B) or 34 x86-64 bytes -- that reads argc, argv and envp off the entry stack, calls main, and exits with a raw exit_group syscall. The syscall and not exit from libc, so the stub costs no import and the static case needs no PLT for it. With neither _start nor main: no main and no _start: cannot generate an executable.
  1. One PT_LOAD per Mach-O segment name, not exactly two. The § 3 table lists an r-x and an rw- load; the writer groups by segment name in order of first appearance, which is what backend_exe.mc already does, so #section __HOT __x gets a PT_LOAD of its own. __TEXT is PF_R|PF_X and everything else PF_R|PF_W. Two loads is what an ordinary program produces; tests/030-section.mc is what needs the general rule.
  1. Both segments are page-padded in the file, VM and offset alike, as macho-exe pads to 16 KiB. It costs up to p_align bytes of zeros (64 KiB on aarch64) and it makes p_offset == p_vaddr (mod p_align) true by construction rather than by an arithmetic argument.
  1. JUMP_SLOT really is the only dynamic relocation kind. A reference to an import -- a call, or &write in an expression -- resolves in place to the import's PLT stub address, the canonical address a linker gives an imported function in a non-PIE executable, so no GLOB_DAT is needed. That works because mc imports functions and never data. Correction (post-merge review): the example this note used to give, uptr p[] = { &write };, is not syntax this language has -- a global initializer must be a constant the compiler can fold, and mc answers initializer must be constant (exit 1) for it. The writer handles an R_UNSIGNED against an undefined symbol anyway, which is what a module emitting its own data would produce; what is not reachable is the source form, not the code path.
  1. DT_NEEDED is emitted for the default library even when every import is claimed by a #dylib, exactly as macho-exe always emits LC_LOAD_DYLIB for libSystem. Same rule, same spelling.
  1. Two TOML keys, not one. Decision 6 says the interpreter path is a key; glibc needs a second name, the DT_NEEDED soname, so [target].interp and [target].libc are both keys with musl defaults. They are read by src/driver.mc into two globals declared in src/objmodel.mc -- not in the writer -- because <mc/core_build> may be assembled without <mc/core_writers> and must still compile (scripts/check-parts.sh § 1b).
  1. --exe had to stop meaning macho-exe -- and by the time this milestone was rebased, it already had. src/cli.mc hardcoded the name; the post-M41 review batch (#15, on main) made it resolve the host pair through the target() registry, after user_init() so that a target a module registered counts, and refuse a 0 slot with <os> requires a linker: there is no direct executable. This milestone's own first draft resolved the same thing during argv parsing, i.e. BEFORE user_init(), which silently ignored a module's registration; that draft was dropped in the rebase and main's version kept, with its tests/proj/noexe.mc case (a taught compiler that re-registers the host pair with 0 in the exe slot) still passing. What M42 changes is only which slots are non-zero: the two Linux ones.

    The mc: linux requires [linker]: there is no direct executable diagnostic in mc build is gone for the same reason -- the slot is filled -- and scripts/check-build.sh asserts the Windows one instead.

  1. One cold-start seed had to grow. lim_seeds[T_BACKENDS] went 8 -> 16 in src/arena.mc. That table is FULL before the pre-scan can size it -- every built-in backend is registered from main(), before mc_main runs on_plan -- and <mc/core_writers> now registers eight, so a taught compiler adding one of its own doubled the table on every build (mc limits examples/kernel reported grew). Capacity only; no generated byte moves.

    lim_seeds[] is a positional list and M41.5 (#17, on main) inserted T_SYNPARAM before T_BACKENDS, so the row moved by one during the rebase: the 16 belongs at index 31, the fourth entry of the row that starts at T_ONSTMT, and a textual merge would have put it at index 30 -- on syntax_param -- with no conflict to show for it. It is checked by NAME, not by position: mc limits src/mc.mc prints backends 0 16 8 0 ok, reserved 16 and used 8.

  1. tests/linux/071-errno-malloc.mc uses fopen, not open, for the failing call. The reason first written here was a measurement, and the measurement does not hold. Re-measured 2026-09-04, with this milestone's own compiler, on all four cells (ubuntu:latest glibc 2.43 and alpine:3 musl, aarch64 and x86_64): a failing open("/no/such/file/at/all", 0, 0) declared extern i64 open(...) returns 0xffffffffffffffff in every one of them, and fd >= 0 is false in every one of them. The claim that glibc hands back 0x00000000ffffffff is not observed.

    What survives is weaker and still true: open returns an int, and both the AAPCS64 and the System V x86-64 ABI leave the bits above a 32-bit return value unspecified. Every libc measured here happens to sign-extend, but nothing in the ABI requires it, and mc has no narrow return types with which to say so. The hazard is latent, not active, and it is a pre-existing mc-wide question rather than something this milestone decides. fopen is kept anyway: it returns a pointer, all 64 bits of it, and it drags stdio in, which is more start-up state to prove -- which is the test's actual subject.

  1. The bootstrap chain still links. scripts/bootstrap-linux.sh keeps scripts/link-linux.sh for mc1l and mc2l: the SEED may be a published release older than this milestone, and the chain must work with whatever seed it is handed. Everything built afterwards uses --exe -- make check on a Linux host now runs test-exe, the whole suite through it. src/mc.linux-{aarch64,x86_64}.toml DID drop their [linker] and [sysroot], so cross-building the compiler for a Linux host from macOS needs nothing installed.

    --exe and --libc were added to that script in the post-merge review (note 12): with --exe every stage is written by the previous compiler and there is no ld.lld and no sysroot in the chain at all, which is the only road on a glibc host.

  1. mc --exe cannot say which libc, and that is deliberate. The two per-libc names are TOML keys, so the single-file CLI has no way to pass them and the writer's default is a CONSTANT: musl. Making it a probe of the machine -- "is there an /lib/ld-musl-<arch>.so.1 here?" -- would make the same source produce different bytes on two hosts, which is exactly what docs/determinism.md forbids. The consequence is real and is named rather than hidden: on a glibc Linux host, mc --exe prog.mc -o prog writes a binary that host cannot start, and the road there is mc build with [target].interp / [target].libc.

    Three scripts do that for the caller, by probing the loader on the disk (never the distribution's name): scripts/test-exe.sh (which prints which road it took), scripts/build-exe.sh (the helper check-toml and check-bundle use to make a binary) and scripts/bootstrap-linux.sh --libc glibc. Measured: scripts/test-exe.sh inside ubuntu:latest with a glibc-hosted mc is 31/31 via mc build, and inside alpine:3 with a musl-hosted one 31/31 via --exe (1 skipped, 032-svc, on both).

    Whether --exe should learn a --interp= / --libc= flag, or whether the host layer should answer for its own libc, is an architect's decision and is not taken here.

    CLOSED by the post-M42 patch (owner's decision, 2026-09-04). The flag exists and the host layer does NOT answer: a probe of the machine is still forbidden, for the reason above. What landed:

    • mc --exe (and --backend=elf-exe*) takes --libc=gnu|musl, --link=dynamic|static and --interp=PATH, mirroring three [target] keys word for word. The default is still the constant musl. Off Linux, with no --backend= naming a writer, each is refused (--libc applies to a linux target) rather than ignored.
    • [target].libc stopped being a soname and became a family, gnu or musl, which names the interpreter path and the DT_NEEDED soname together. The old spelling is refused with the migration in the message (libc must be gnu or musl (a soname is not a value: gnu is libc.so.6, musl is libc.so)). [target].interp stays as the explicit path override.
    • [target].link / --link= is new. static does not select the static image -- the import count still does -- it ASSERTS it, and a program that imports any symbol (libc or #dylib) is refused with static link with imports needs [linker]: see docs/build.md -- static linking (M46). The refusal is raised by the writer, which counts imports, and reported at the KEY's file:line:col through a reporter the driver installs (dyn_die, src/objmodel.mc) -- a writer must not know what TOML is, and a mc.toml diagnostic must not lose its position.
    • The vocabulary is now ONE: scripts/test-linux.sh, scripts/bootstrap-linux.sh and scripts/check-linux-host.sh take --libc musl|gnu (the value glibc is refused with the rename), and scripts/test-exe.sh and scripts/build-exe.sh lost their mc build detour entirely -- they are mc --exe --libc= now. The two src/mc.linux-<arch>-gnu.toml configs lost a key each.
    • docs/build.md § Linux targets carries the 3x2 matrix (libc gnu/musl/none x link dynamic/static) with the road per cell, and scripts/check-build.sh exercises every cell that is reachable without Docker -- both roads -- for 47 checks in all.

    Reviewed again (2026-09-05), two findings, closed in one commit.

    1. The refusal above was half a refusal. Its condition was "no --backend= and the host is not Linux", so on a Linux host the default OBJECT road took all three flags and did nothing with them (mc x.mc -o x.o --libc=gnu), and --backend=elf-obj --libc=gnu did the same on every host -- reproduced by cmp, the object with the flag and the object without it are byte for byte identical. Only the executable writer reads dyn_libc/dyn_interp/ dyn_static; PT_INTERP and DT_NEEDED are program-header fields and an object has neither. The gate is now three questions, moved to just after user_init() so that the second one can be asked of the target registry (backend_is_exe, src/hooks.mc) and a target a module registered answers for its own writer: is anything written at all (a --dump-* mode writes nothing -- applies to an executable: a --dump-* mode writes none), is it an executable (applies to an executable: use --exe), and is it a Linux one (the original applies to a linux target). The price of the move is that an unreadable entry now reports cannot open first, which is exactly what M39.5 accepted for [target].
    2. docs/reference/diagnostics.md still carried the row linux requires [linker]: there is no direct executable as a Linux fact. Since this milestone filled the two Linux exe slots the message belongs to any target whose exe slot is 0 -- windows/aarch64 and windows/x86_64 today, or a pair a module registers with target(os, arch, obj, 0). The row, its sibling in § 9, docs/reference/hooks.md's target() block (whose code sample still showed target("linux", "aarch64", "elf-obj", 0)) and the --libc glibc invocations left in docs/guide/90-linux-host.md -- a value the scripts refuse since the patch above -- were corrected together.

The four-cell run #

tests/013-putnum.mc (imports write) and tests/linux/071-errno-malloc.mc (a failing fopen, errno through __errno_location, malloc written at both ends, free):

libc / archcontainer013-putnum071-errno-malloc
musl / aarch64alpine:346368, exit 0errno=2 malloc ok, exit 0
musl / x86_64alpine:346368, exit 0errno=2 malloc ok, exit 0
glibc / aarch64ubuntu:latest46368, exit 0errno=2 malloc ok, exit 0
glibc / x86_64ubuntu:latest46368, exit 0errno=2 malloc ok, exit 0

Re-run 2026-09-04 on ubuntu:latest = Ubuntu 26.04 LTS / glibc 2.43 and alpine:3 = 3.24.1: all eight cells as in the table.

The compiler itself, dynamic against glibc #

M42's own acceptance ran the corpus against both libcs, but the COMPILER had only ever been built dynamically against musl (alpine:3, through make check-linux-host). The post-merge review added the glibc cell, and it is the same shape: scripts/check-linux-host.sh --libc glibc cross-builds mc from macOS with src/mc.linux-<arch>-gnu.toml (the musl config plus interp and libc, nothing else), copies it into ubuntu:latest and, with nothing installed in that container -- no make, no lld, no musl-dev -- runs scripts/bootstrap-linux.sh --libc glibc to its own fixed point, the whole suite through mc --exe, and the cross proof against the macOS build/mc2.o.

The golden is the SAME file the musl chain verifies (tests/golden/mc2-linux-<target>.sha256): an ELF ET_REL records no interpreter, so both roads must write the same object, and they do.

Decisions (architect) -- to ratify with the owner #

  1. Dynamic first, static as the degenerate case. Owner's requirement, adopted.
  2. ET_EXEC at a fixed base, PIE deferred and documented.
  3. DT_BIND_NOW, no lazy binding.
  4. DT_HASH, not DT_GNU_HASH.
  5. musl AND glibc from the start: the § 0 probe passed on both, on both architectures, with no __libc_start_main. The interpreter path and soname are the only per-libc facts (decision 6).
  6. The interpreter path is a TOML key with a per-target default, not a constant.
  7. [linker] stays supported and is the only route to a static libc link.
  8. No 1.0.0 on the back of this milestone: the owner's rule is that 1.0.0 waits for the whole roadmap and for coordination with the teko/ngen consumer.

Edit this page