mc

Spec M25 -- sysroots and cross-compilation resolution

Owner's direction (2026-09-03, restated 2026-09-04): cross-compiling must not make the developer hunt for files, and everything mc downloads must be checksummed, cached and reproducible. mc redistributes nothing. mingw-w64 import libraries matter for USER programs that need more than kernel32 (msvcrt, user32, ws2_32); the compiler itself never needs them.

Goal: one resolution chain in the driver -- explicit path, then the running system, then ~/.mc/sysroots/<target>, then a message with the exact command -- plus mc sysroot list|path|fetch|stub, so that a fresh checkout on any of the three hosts can build for any of the five targets without a hand-assembled directory, and so that a macOS box with no SDK can still link. Depends on M14 (driver/TOML), M16/M20 (the Linux and Windows targets), M37 and M38 (the host layer, on main at 2b83243).

What exists #

Design #

1. drv_sysroot becomes a chain #

One function, same signature, same single call site (drv_ph, src/driver.mc:356):

uptr drv_sysroot() {
    1. [sysroot].path            -> drv_path(p), and now CHECKED: if the directory has none
                                    of the target's marker files, name it and stop.
    2. host == target            -> the running system's own files (probes below)
    3. ~/.mc/sysroots/<os>-<arch>/  (or [sysroot].cache, or --sysroot-dir)
    4. sysroot_missing(os, arch) -> the instructions, exit 2
}

The result is cached in a global like drv_sdk_cache (src/driver.mc:66); the chain runs at most once per build and only when some [linker].args value mentions {sysroot} -- exactly the laziness {sdk} already has.

"Present" is a marker-file test, not a directory walk. mc has no opendir; it has open (src/arena.mc:5). One helper, path_exists(p) = open(p, O_RDONLY, 0) >= 0 then close, and a per-target marker list from the target registry: crt1.o + libc.a for linux-*, kernel32.lib for windows-*, usr/lib/libSystem.tbd for macos-*.

The per-OS "running system" probes, tried in order, first hit wins:

hostprobed, in order
linux/usr/lib/<arch>-linux-musl (Debian/Ubuntu musl-dev, the CI layout), /usr/lib/musl/lib, /usr/lib (Alpine, where apk add musl-dev puts all four)
macosdrv_sdk() for {sdk}; for {sysroot} on a macOS target, the same SDK path. Nothing to probe when host_has_sdk() is 0 -- that is what § 4 is for
windowsthe directory holding the compiler's own generated kernel32.lib: [sysroot].path, then build/sysroot/windows-<arch>. mc cannot regenerate it (that is llvm-dlltool), so the miss prints sh scripts/sysroot-windows.sh --arch <arch> or mc sysroot fetch windows-<arch>

The probe is skipped entirely when host_os()/host_arch() differ from the target, so a cross build never picks up the host's own libc.a.

~ and HOME. mc has no getenv. Two host functions join the interface, in the shape M38 set with host_exe_suffix() (src/host_macos.mc:78): host_home() returns the user's home directory or 0. macOS/Linux scan host_environ() -- a NUL-terminated array of KEY=VALUE pointers -- for HOME=. Windows cannot: host_environ() is 0 there (src/host_windows.mc:42), so host_windows.mc declares extern i64 GetEnvironmentVariableA(uptr name, uptr buf, i64 size) and asks for USERPROFILE; the name joins scripts/sysroot-windows.sh's kernel32.def (a 14th line) and lib/sys_windows_host.mc. [sysroot].cache in mc.toml and a --sysroot-dir DIR flag override, so CI never depends on HOME.

2. Downloading: mc spawns a downloader, it does not speak HTTP #

The language reaches sockets by extern and examples/api/lib/http.mc is HTTP/1.1 in plaintext over socket/bind/listen/accept -- a server, and no TLS anywhere. HTTPS is out of reach, and an http:// fetch of a checksummed file would still be a downgrade nobody should ship. So mc sysroot fetch spawns, through the existing drv_spawn:

hostdownloaderpresent because
macoscurl -fLsS -o FILE URL/usr/bin/curl ships with the system
linuxcurl, else wget -q -O FILE URLone of the two is on every distribution; both on the CI runners
windowscurl.exeshipped in System32 since Windows 10 1803; on PATH under Git Bash

drv_spawn returns the exit code and posix_spawnp searches PATH, so "no downloader" is cannot run curl today; M25 turns it into the offline message (§ 5). The checksum is computed by mc itself with src/sha256.mc over the downloaded bytes -- no shasum/sha256sum/ certutil divergence across three OSes, and the verification is part of the compiler rather than part of a script.

Extraction. Verified on this machine: /usr/bin/tar is bsdtar 3.5.3 - libarchive 3.7.4 zlib/1.2.12 liblzma/5.4.3, so macOS handles .tar.gz and .tar.xz out of the box. GNU tar on Linux handles both (-J). Windows 10 1803+ ships tar.exe (libarchive), which reliably handles .tar.gz and .zip; its .xz support is not something to rely on -- so any Windows-side archive is chosen as .zip. An Alpine .apk is a gzip tar (three concatenated gzip members) and tar -xzf reads it on all three, extracting usr/lib/... next to .PKGINFO.

3. Where the source list lives: a bundled .mc table, not a TOML file #

toml_parse fills one global table (src/toml.mc:65,415); parsing a second file during mc build would overwrite the project's config, and mc has no toml_parse_str to feed a bundled blob to. So the list is src/sysroots.mc -- rows of sysroot_src(target, kind, url, sha256, size, strip, files) registered in a table the same way target() and backend() are registered in src/main.mc:134-146. It costs no parser, it is deterministic to print, and mc sysroot list reads it with no I/O. docs/reference/sysroot.md carries the same rows in a table, and a scripts/check-sysroots.sh diffs the two the way check-bundle diffs the bundle -- which is how the URLs and hashes stay honest.

4. The three targets #

Linux (musl). Alpine's CDN, one file: https://dl-cdn.alpinelinux.org/alpine/v3.22/main/ <aarch64|x86_64>/musl-dev-<ver>.apk, pinned by version and sha256 -- not resolved through APKINDEX.tar.gz, which changes under us and would make a build non-reproducible (a mc sysroot fetch --index that reads APKINDEX to report a newer pin is a follow-up, not this milestone). musl-dev alone carries all four files; the musl package holds only the dynamic loader, which a static link does not use, so the plan row's "and musl-<ver>.apk" is dropped. Extract usr/lib/{crt1.o,crti.o,crtn.o,libc.a} into ~/.mc/sysroots/linux-<arch>/. A developer who needs more (libsqlite3.a for examples/api/mc.linux.toml, which today says "needs apk add sqlite-static") drops it in the same directory, and {sysroot}/libsqlite3.a resolves.

Windows (import libraries) -- synthesized by default, fetched on request. An import library is a list of names, which is why scripts/sysroot-windows.sh can already build kernel32.lib from a 13-line .def with llvm-dlltool and no download. mc knows the program's externs and which [libs] entry each belongs to (extern_lib_find, src/parse.mc:435), so it can write build/stubs/<lib>.def and spawn llvm-dlltool -- the exact Windows mirror of the Apple stub idea, covering user32/ws2_32/msvcrt for a user program with no download at all. Data exports would need a DATA keyword the synthesizer cannot infer; that is the documented gap, and the reason mc sysroot fetch windows-<arch> still exists: llvm-mingw's GitHub release (llvm-mingw-<date>-ucrt-<host>.tar.xz, ...-ucrt-x86_64.zip on Windows), from which only <triple>/lib/lib*.a is extracted (aarch64-w64-mingw32, x86_64-w64-mingw32). To be verified before this lands: lld-link -- the linker every Windows mc.toml names (docs/build.md:867) -- accepts mingw-style lib*.a import archives. lld does have MinGW import-library handling, but on the lld-link (link.exe-compatible) driver rather than ld.lld, and the acceptance below makes it an explicit step rather than an assumption. If it does not hold, the fetch extracts the .def files from the mingw-w64 source release (~10 MB, all of mingw-w64-crt/lib-common/*.def and lib-arm64/*.def) and llvm-dlltool builds the .lib locally -- lighter than a 400 MB toolchain and identical in shape to what already works.

macOS. Two roads, and the first one is "no road at all":

5. mc sysroot, the cache, and being offline #

A third dispatch line beside src/main.mc:151-153, into a new src/sysroot.mc:

mc sysroot list                     every registered target, its resolution and where from
mc sysroot path <os>-<arch>         the resolved directory on stdout, or exit 2
mc sysroot fetch <os>-<arch> [--yes] [--sysroot-dir DIR]
mc sysroot stub [DIR] [--config F]  write the .tbd/.def stubs for a project without linking

list is a fixed walk of the target registry plus path_exists probes -- deterministic output, no directory listing, safe in a golden test. fetch prints the plan first (URL, size, sha256, destination), then needs --yes: mc has no isatty, so there is no TTY prompt, and requiring the flag is the honest version of "asks for confirmation".

Cache layout, ~/.mc/sysroots/<os>-<arch>/<kind>/, with a sibling manifest.toml written by mc (source URL, sha256, size, the sysroots.mc row id, the extracted file names -- no date, docs/determinism.md) so list can say where a directory came from and check-sysroots.sh can re-verify it.

Offline / missing, one message, exit 2 (a code docs/reference/cli.md § Exit codes does not use yet -- 1 is diagnostics, 3 is the limits verdict):

mc.toml:14:8: no sysroot for linux-aarch64 [sysroot.path]
  tried: build/sysroot/linux-aarch64 (no crt1.o), ~/.mc/sysroots/linux-aarch64 (absent)
  run:   mc sysroot fetch linux-aarch64 --yes
  or:    curl -fLO https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/musl-dev-1.2.5-r10.apk
         sha256  <hex>
         tar -xzf musl-dev-1.2.5-r10.apk -C ~/.mc/sysroots/linux-aarch64 --strip-components=2 usr/lib

The same block comes out of a failed fetch (downloader missing, non-zero curl, checksum mismatch), so there is one text to get right and one to document.

6. CI #

Out of scope #

No TLS and no HTTP client in mc. No package manager: fetch knows a fixed pinned list, not a resolver. No APKINDEX resolution (a --index reporter is a follow-up). No frameworks in the macOS stub writer (libSystem and [libs] entries only). No .def DATA inference on Windows. No proxy configuration beyond what curl/wget read from the environment themselves. No wasm target (M27+). Nothing is fetched implicitly: mc build never downloads.

Files and estimated deltas #

filedelta
src/sysroot.mc (new): the chain, the probes, path_exists, mc sysroot~380
src/sysroots.mc (new): the pinned source rows + registry~120
src/stubs.mc (new): .tbd and .def writers over fs_at/extern_lib_find~200
src/driver.mc~+40: drv_sysroot becomes the chain; {stubs} placeholder in drv_ph
src/host_macos.mc, src/host_linux.mc, src/host_windows.mc+12 each: host_home(), host_downloader()
lib/sys_windows_host.mc, scripts/sysroot-windows.sh+15 / +1: GetEnvironmentVariableA
src/main.mc+2: the sysroot dispatch line, drv_usage gains its line
src/core.mc, tools/bundle.list, src/bundle_data.mc+3 entries, regenerated
scripts/check-sysroots.sh (new), scripts/check-stubs.sh (new)~60 + ~80
.github/workflows/ci.yml~+40 net (two fetch steps replacing two apt blocks, one stub step)
Makefile+15 (check-sysroots, check-stubs, wired into check)
tests/proj/stub.toml, tests/golden/sysroot-list.txtnew
docs/reference/sysroot.md (new, ~250), docs/guide/50-cross-compile.md, docs/build.md § [sysroot]/[linker], docs/reference/{cli,toml,hooks,diagnostics}.md, docs/guide/{90-linux-host,95-windows-host}.md, docs/ci.md, docs/plan.mdupdated

Acceptance #

  1. macOS with no SDK: with PATH stripped of xcrun, a program using write and sqlite3_open links with ld64.lld against build/stubs/*.tbd written by mc and runs; mc sysroot list shows macos-aarch64 stubs (synthesized).
  2. Fetch into a temp cache and build for Linux: mc sysroot fetch linux-aarch64 --yes --sysroot-dir $TMP writes four files whose sha256 match the manifest; mc build for linux/aarch64 then succeeds with no [sysroot].path in the config at all, and the binary runs in docker --platform linux/arm64.
  3. Offline: with a PATH that has no curl and no wget, mc sysroot fetch prints the manual block and exits 2; mc build for an unresolvable target prints the same block and exits 2 (not 1 -- docs/reference/cli.md gains the row).
  4. Running-system probe: on the Linux host jobs, mc build for the host target with no [sysroot] finds /usr/lib/<arch>-linux-musl and links; mc sysroot path linux-<arch> prints it.
  5. Windows import libs: mc sysroot stub on a program declaring MessageBoxA writes user32.def, llvm-dlltool turns it into user32.lib, and lld-link links it on the windows-11-arm runner. Separately, mc sysroot fetch windows-aarch64 --yes extracts llvm-mingw's aarch64-w64-mingw32/lib/lib*.a and a link against libws2_32.a succeeds -- the § 4 verification; if it fails, the .def-from-source variant lands instead and this spec's Decision 6 is amended in the same PR.
  6. mc sysroot list output is byte-stable against tests/golden/sysroot-list.txt on all three hosts (target names and resolution kinds only; absolute paths are printed by path, not by list).
  7. check-sysroots.sh proves src/sysroots.mc and docs/reference/sysroot.md agree; make check green on macOS, and the Linux and Windows subsets green.
  8. No behaviour change for a config that already sets [sysroot].path at a populated directory: every existing golden object is byte-identical.

Risks #

Decisions (architect, 2026-09-04 -- all twelve recommendations adopted) #

  1. The chain lives in drv_sysroot's one call site, not in a new placeholder. {sysroot} keeps meaning what it means; only how it is found changes. [sysroot].path still wins, and is now checked rather than passed through to the linker's error message.
  2. Spawn curl/wget/curl.exe; verify with mc's own src/sha256.mc. No HTTP in mc, and no dependence on three different checksum CLIs.
  3. The pinned source list is src/sysroots.mc, a bundled .mc table, not a TOML file. toml_parse has one global table (src/toml.mc:65) and no string entry point; a second parse during mc build would clobber the project config. docs/reference/sysroot.md mirrors it and a check script enforces the agreement.
  4. Cache at ~/.mc/sysroots/<os>-<arch>/, with manifest.toml and no dates. HOME via a new host_home(); USERPROFILE via GetEnvironmentVariableA on Windows, because host_environ() is 0 there. [sysroot].cache and --sysroot-dir override, and CI uses the override so no job depends on HOME.
  5. Missing sysroot = exit 2, with one message shared by build and fetch. Reserve 2 for "the environment is not ready", distinct from 1 (a diagnostic) and 3 (a limits verdict).

  6. Windows: synthesize import libraries from the program's own externs by default (llvm-dlltool over a generated .def, the exact mirror of the Apple .tbd road and of what scripts/sysroot-windows.sh already does), and keep mc sysroot fetch windows-* from llvm-mingw for the cases synthesis cannot cover. Pending the § 4 verification, prefer the mingw-w64 source release (.def files, ~10 MB) over the 400 MB binary toolchain.

    AMENDED 2026-09-04, after the verification § 4 asked for. lld-link DOES accept mingw-style lib*.a import archives, on both architectures, so the .def-from-source fallback is not needed and mc sysroot fetch windows-* pins the llvm-mingw release archives directly. What was run, on this machine:

    $ cat probe.mc
extern i64 htons(i64 v);
i64 main() { return htons(1); }

$ mc --backend=coff-obj-arm64 probe.mc -o probe.obj
$ llvm-nm probe.obj
                 U htons
00000000 T main
$ tar -xJf llvm-mingw-20260826-ucrt-ubuntu-22.04-aarch64.tar.xz --strip-components=3 \
      llvm-mingw-20260826-ucrt-ubuntu-22.04-aarch64/aarch64-w64-mingw32/lib/libws2_32.a
$ lld-link -machine:arm64 -subsystem:console -entry:main -nodefaultlib \
           -out:probe.exe probe.obj libws2_32.a
$ echo $?
0
$ llvm-readobj --coff-imports probe.exe
Format: COFF-ARM64
Import {
  Name: WS2_32.dll
  ImportLookupTableRVA: 0x2028
  ImportAddressTableRVA: 0x2038
  Symbol: htons (0)
}

    llvm-objdump -d probe.exe shows lld synthesized the ARM64 import thunk itself (adrp x16 / ldr x16, [x16, #0x38] / br x16). The same probe with --backend=coff-obj-x86_64, -machine:x64 and x86_64-w64-mingw32/lib/libws2_32.a also links with exit 0 and the same WS2_32.dll / htons import. Recorded in docs/reference/sysroot.md § 8 as well; the marker for a Windows sysroot therefore accepts kernel32.lib (what scripts/sysroot-windows.sh generates) or libkernel32.a (what the fetch unpacks).

  7. macOS: --exe first, stubs second, mirrors third. Say plainly in the guide that the built-in Mach-O writer needs no SDK at all, so the stubs matter only for the .o + ld road; and scope the stub acceptance to ld64.lld, since a machine with Apple's ld has an SDK. dyld_stub_binder goes into every synthesized libSystem.tbd.
  8. Linux: musl-dev only, pinned version + sha256, no APKINDEX resolution. Reproducibility beats freshness; an --index reporter can come later.
  9. CI: convert the two linux-* suite legs' sysroot step to mc sysroot fetch; leave the two linux-*-host legs on MC_SYSROOT=/usr/lib/<arch>-linux-musl (they are the running-system probe's regression) and leave all four Windows legs on scripts/sysroot-windows.sh. Add one macOS check-stubs step. Docker stays the local path.
  10. mc sysroot fetch requires --yes; there is no TTY prompt, because mc has no isatty and inventing one for this is not worth an extern.
  11. New reference page docs/reference/sysroot.md (the resolution chain, the cache layout, the pinned table, every message), with docs/guide/50-cross-compile.md gaining a short "you do not have to assemble this by hand" section that links to it. docs/README.md gains the row.
  12. Ship in three commits -- chain+probes, mc sysroot+fetch, stub writers -- so the no-network half can land and be used even if the fetch half slips.

Architect's additions: (a) mc build never downloads, ever -- only mc sysroot fetch --yes does; (b) exit code 2 is documented in docs/reference/cli.md before any script relies on it; (c) the three commits of Decision 12 are each gated by make check; (d) the lld-link-vs-mingw .a verification (acceptance 5) is the first thing the Windows half does, and the outcome is written into this file's Decision 6 in the same PR.

Deviations from this spec, as built #

  1. The "no sysroot" message carries no file:line:col. § 5 illustrates it as mc.toml:14:8: no sysroot for linux-aarch64 [sysroot.path]; what shipped is a bare mc: no sysroot for linux-aarch64 followed by the same tried:/run:/or: block (src/sysroot.mc, sysroot_missing(); docs/reference/sysroot.md § 5 and docs/reference/diagnostics.md document the shipped form).

    Why: the chain is not run from a key. It is run lazily, the first time some [linker].args value asks for {sysroot} (the same laziness {sdk} has), and there is no single position to blame — [sysroot].path may be absent, [target].os is what selected the target, and the argument that triggered it may be one of several. Worse, the same text is what mc sysroot path <target> prints, and that command reads no config at all: a position would have to be invented for half the callers. Every line of the message names an absolute directory, so nothing about which sysroot was wanted is lost — only which line asked, and that is one target per build.

    Threading the position of the {sysroot}-bearing [linker].args entry through drv_ph stays open as a follow-up; it is a toml_err_key-shaped change to the driver, not to this file.

Edit this page