Spec M50 -- the reproducible bench cell: one job per toolchain, a reference timed beside every row, and a dated JSON git blame can read

Written against minicompiler/mc main at 075dbec (M49 step D2), read-only session, 2026-09-13, on the same Apple M4 / macOS 26.6.2 host ../comparison.md measured. docs/plan.md's M50 row (../plan.md, line 420) names a shape -- bench/Dockerfile, one container per row, a fixed VPS, --cpus 1 --memory 512m -- and then says in its own text that the road is "an option, not a decision". This spec takes the measurements first (§ 1), and every decision below cites the number it rests on.

The milestone row, verbatim (docs/plan.md:420):

M50 | A reproducible bench cell (from docs/comparison.md): bench/Dockerfile with every compared toolchain pinned to an exact version, run one container per row on a fixed VPS with --cpus 1 --memory 512m, results committed as dated JSON so a regression is a git blame, not a chat log; replaces the two honest caveats docs/comparison.md § Conditions records today (shared host load, an unverified tree label). Option, not a decision: the soak (.github/workflows/bench-soak.yml, docs/comparison.md § "The hour under load") already runs one pinned-toolchain job per server on GitHub-hosted runners, so this cell could be a GitHub Actions job per row instead of Docker on the VPS | two consecutive runs of the cell agree within a documented tolerance; a deliberately regressed mc build fails the cell

The one-paragraph version. The two caveats have different cures and only one of them is about hardware. The tree label is cured by a digest: the cell measures a RELEASE tarball whose .sha256 it verifies and records, plus the SHA-256 of the binary it actually timed -- a number that cannot go stale the way bench/results.json:234's "tree at commit e5a1643" did. The shared host load is not cured by any machine this project can reach: the VPS is production and is itself a VM on shared hardware, a GitHub runner is shared by construction, and the developer's Mac was running a Lima VM while this spec was written. What cures it is the comparison rule: measured here, the same clang -O2 binary of the same program has timed 0.420 s, 0.49 s and 0.52-0.56 s in three sessions on one Mac (a 33% span with nothing changed), while the ratio of mc -O to a clang -O2 timed in the SAME run, as median-of-six after discarding the first repetition, agreed between two consecutive runs to -0.7% (and the --opt=0 / --opt=1 pair to -0.1%). So the cell's verdict is a ratio against a reference built and timed beside each row, absolute seconds are recorded and never gated, the unit of a job is a TOOLCHAIN (its install is the whole cost) rather than a row, and the road is GitHub Actions -- the soak's proven shape -- with Docker and the VPS kept as priced options for a road off GitHub. The teeth cost nothing to build: mc --opt=0 against mc --opt=1 is a deliberately regressed compiler that is already on main, and it measures 1.685x here.

0. What this stands on (read, not assumed) #

• The two caveats, in their own words (docs/comparison.md:43-58). (1) "One host, shared with other work. The load generator and the server run on the same ten cores over loopback, and make check runs were competing for the machine while some of this was measured"; bench/http/RESULTS.md § Notes 13 names three runs that "swing ~20% run-to-run with nothing in the logs to explain it" and ends "Repeat the outlier rows before reading anything into a difference under ~20%". (2) "The mc binary that ran the workload benchmark reports the tree it was built from as e5a1643" -- a commit that is real but is not what produced the binary; the page cites the verified 0648e1a instead and says so. Item 3 of the same block already promises this milestone: "The reproducible cell -- a fixed VPS, one Docker container per row, pinned toolchain versions, --cpus 1 --memory 512m -- is on the roadmap ... and will replace these numbers as the reference measurement."
• Where the stale label lives. bench/results.json:234: "version": "mc 0.0.0-dev (build/mc1, tree at commit e5a1643)", and bench/RESULTS.md:7 and :98 repeat it. It is a self-reported string with nothing to check it against; the JSON's host block (cpu, ram_bytes, os, date, timing_method) has no field for the identity of the compiler measured, and the binary's own digest is recorded nowhere.
• The existing harness. bench/measure.sh (12 lines) runs ONE command three times back to back under /usr/bin/time -l and keeps the best real plus that run's max RSS; bench/run.sh (68) is the portable contract check (make bench, Makefile:687) that needs only build/mc1 and clang and asserts the recorded stdout 8128903901837660708 / 3001134 / 39088169; bench/README.md:3-6 states the rule this milestone must not break: "Neither runs in make check: results depend on the host's CPU, load and installed toolchain versions, which make check must not."
• The soak is the protocol the cell would run, and it says so itself (bench/README.md:259-268): "The bench/Dockerfile-on-a-fixed-VPS cell of docs/comparison.md § 'Where to improve' is the answer to that caveat; this workflow is the protocol that cell would run." What .github/workflows/bench-soak.yml already does, and M50 should copy rather than invent: a plan job turning an input into a matrix (:75-88), one job per row with only that row's toolchain installed at a version written in one env: block (:50-72, :90-204), mc from a release asset with sha256sum -c before it is unpacked (:127-137), core pinning with taskset so the load generator never shares the measured cores (:213-219, bench/README.md:199-202), every observable fact recorded in facts.json (bench/README.md:214-216), per-row artifacts with 90-day retention plus a report job that downloads them all and writes tables and charts (:230-278), a weekly schedule and a workflow_dispatch, and "never on push or pull request: it costs an hour of runner time per server ... and measures nothing a commit can break" (:6-8).
• The dated-directory precedent. bench/soak/results/2026-09-06-34062566194/ holds that hour's RESULTS.md, results.json and 17 SVGs; bench/README.md:246-257 § "Recorded runs" says the directory is "dated so a later run sits beside it instead of overwriting it", and docs/comparison.md § "The hour under load" is a HAND-WRITTEN section that cites it. No generator writes into a doc page.
• The workload has no phase selector. bench/mc/bench.mc:68-76 is i64 main() calling mix(), primes() and fib(38) unconditionally, and the other five sources (50-59 lines each) have the same shape. M49 got its per-phase numbers "by trimming main in the scratchpad" (M49.md § 1.2) -- so the three numbers M49's acceptance is stated in (mix <= 0.28 s, primes <= 0.21 s, fib <= 0.18 s, § 9.2 there) cannot be re-measured by anything in the repository today.
• M49 hands its reference to this milestone, twice. M49.md D12: "the acceptance number is 1.3x on THIS host with the three phases recorded, and M50's cell becomes the reference when it exists"; § 10 row 5 (step D2, the x86-64 allocator): "not measured on this host (no x86-64 bench cell; M50's job)", and § 12 risk 11 defers the Win64 rdi/rsi question to "a measured decision ... when M50's x86-64 cell exists". The x86-64 allocator's only number today is instruction count: 148 042 -> 130 846 dump lines, -11.6% (§ Implementation notes 12 there). Also priced and not taken there, waiting on this cell: the third x86 peephole of § Implementation notes 2, "it belongs to whichever milestone owns an x86-64 bench cell (M50), with a number in front of it".
• The runners available (docs/ci.md:16-27, :131-141): macos-15 (arm64, Apple clang), ubuntu-24.04-arm (arm64), ubuntu-latest / ubuntu-24.04 (x86_64), windows-11-arm, windows-2025. The required status checks on main are the six ci.yml job names, and "Renaming a job means updating the branch protection in the same breath" -- a new workflow adds no required check unless someone adds it, which is what M50 wants.
• The VPS. docs/specs/M43.md:15: "The owner's VPS is Ubuntu 26.04 x86_64, kernel 7.0.0-30, root over ssh, 4 CPU / 15 GiB"; :55: "the VPS is itself a VM: nested virtualization is UNVERIFIED". It runs the site and the package registry (CLAUDE.md § State, minicompiler/mc-registry), and M43 used it as a sandbox oracle where the same box that costs 1.37 ms on the Lima arm64 VM costs 4.12 ms (M43.md:449, :581) -- a machine about three times slower per unit of work than the developer's own VM, with production traffic on it.
• docs/determinism.md § 6, "Fixed reference builds", is the rule this milestone is the benchmark's version of: "always compare builds made with the same flags". The cell's version is stronger and is § 4: always compare timings taken in the same run, on the same core, against a reference measured beside them.
• make check must stay untouched. bench/ is in no gate today (Makefile:680-688, .PHONY at :694 carries bench), and scripts/check-docs.sh:167 walks docs/**.md -- so this page's links are checked and any fenced ```mc block in it would be compiled. This spec has none.

1. The measurements (step 0), and what they decide #

1.1 What the recorded data already says, read from the files #

bench/results.json's A_results[].run_runs -- the three timed runs behind every "best of 3" in ../comparison.md § "The workload":

bench/http/results.json, per-run requests_per_second (18 row/configuration pairs): the spread runs from 0.9% (zig-threads keep-alive) to 228.5% (c-serial no-keep-alive: 4058, 4710, 13331 req/s), with six pairs above 15% -- the measurement bench/http/RESULTS.md § Notes 13 warns about by name.

Two "pinned" toolchains that moved between the two recorded measurements, on the same files:

• bench/go/go.mod says go 1.26, and the soak resolves it with actions/setup-go@v5 ... go-version-file -- so the workload ran on go1.26.4 and the soak on go1.26.7 (docs/comparison.md § Conditions, both blocks).
• .NET is pinned DOTNET_VERSION: "10.0.x" (bench-soak.yml:60): SDK 10.0.301 on the Mac, 10.0.400 on the runner.
• clang is whatever the host has: Apple clang 21.0.0 against Ubuntu clang 18.1.3.

And the reference row itself, the same clang -O2 binary of the same 50-line program, on the same Mac, in three sessions:

A 33% span, with nothing about clang, the source or the machine changed. That is the whole case against gating on absolute seconds, and docs/comparison.md carries both 0.49 and 0.420 on the same page today because of it.

1.2 What a cell run looks like on this host, measured for this spec #

Four binaries of the same workload, built from 075dbec: mc --exe, mc -O --exe, clang -O2, clang -O0. Seven repetitions, the four binaries interleaved inside each repetition (rep 1: mc-plain, mc-opt, c-O2, c-O0; rep 2: the same order), /usr/bin/time -p, nothing else asked of the machine -- except that this host is running a Lima VM with 4 of its 10 cores available to it, which is exactly the contamination the cell is for.

Run 1 (seconds):

Run 2, started immediately after run 1 finished, same command, same binaries:

Derived, dropping repetition 1 of each run (six kept):

And the three candidate verdict forms, each computed from the same six repetitions:

1.3 What that decides #

1. Discard the first repetition. It is 1.25-1.62x the best in nine of the ten recorded rows (§ 1.1) and 1.25-1.62x again here (§ 1.2, run 1): the 50 MB __bss sieve is first-touched and the binary's pages are cold. bench/measure.sh's best-of-3 hides this by taking the minimum; a cell that reports a median must drop it explicitly.
2. Never gate on absolute seconds. 0.420 / 0.49 / 0.52-0.56 for one unchanged binary (§ 1.1). Absolute times are recorded, reported, and used only as the machine's own health indicator (§ 4.4).
3. Do not use the median of per-repetition ratios. Its spread (9.1%) is WORSE than either binary's own (4.1% and 7.7%): two consecutive runs are not simultaneous, so their noise adds instead of cancelling. This falsified the first draft of § 4 and is why the rule is written the way it is.
4. Use the ratio of per-row medians against a reference measured in the same run. Two consecutive runs agree to -0.7% on mc -O / clang -O2, -0.8% on the plain road and -0.1% on the tooth pair; the worst row is clang -O0 at -3.2%. A documented tolerance of 5% on that ratio has margin over everything measured here, and 3% would have held for three of the four rows.
5. Seven repetitions, six kept, is enough for that tolerance on a 0.5-1.3 s workload, and costs about 21 s of timing per job. Nothing measured here asks for more.
6. Cross-session drift of the ratio is larger than back-to-back drift: mc -O / clang -O2 is 1.380-1.389 in this session against M49's 1.30 on the same host a week earlier -- about 7%, while every row moved ~30% in absolute terms. So the ±5% band is a within-cell, run-to-run band; a number from a different machine (or a machine whose background changed) is not comparable at all, which is what makes the in-run reference mandatory rather than nice.
7. The HTTP rows cannot meet a tolerance of this kind on shared hardware with a co-resident load generator: 0.9% to 228.5% (§ 1.1). They are out of the cell (§ 6), and the soak -- fixed rate, one runner per server, pinned cores -- is already the right protocol for them.
8. The teeth are free and measured. mc --opt=0 against mc --opt=1 on one source is 1.685x here (M49's own cross-session figures give 1.99x), with a run-to-run drift of 0.1%. A floor of 1.5x has margin in both directions, and it collapses to about 1.0 the moment the allocator stops allocating -- which is exactly "a deliberately regressed mc build fails the cell", with no patched compiler to maintain.

2. Where the cell runs -- decision (design question 1) #

The recommendation is (a), and the five lines of reasoning are these. The noise the plan row wants to remove cannot be removed by any machine this project can reach -- the VPS is production and is itself a shared VM, and the Mac is a desktop -- so the cure has to be the comparison rule, and a ratio to a reference timed in the same job holds to 0.7% (§ 1.2) on hardware that swings 33% between sessions. The soak already proves the whole mechanism on fourteen jobs: one toolchain per job at a pinned version, mc from a release with its .sha256 verified, taskset pinning, facts.json, per-row artifacts and a report job -- M50 is that workflow with oha replaced by six repetitions of a binary, which is less machinery, not more. Actions is the only road with both architectures, and M49 left two numbers explicitly waiting on an x86-64 cell (D2's allocator has never been timed anywhere, and its third peephole is priced on the condition that this cell exists). The VPS road costs an ssh secret, an operator and a cron outside CI, and puts a CPU-bound benchmark on the machine that serves minicompiler.dev. And Docker's real product -- one digest for the whole toolchain matrix -- is worth having only on a road where the versions cannot otherwise be reconstructed, which on Actions they can (D13 keeps it as an option, priced).

One deviation from the plan row's wording, stated plainly: the unit is a toolchain, not a row. A job's cost is its toolchain install (seconds of timing against minutes of setup-*), and the in-run reference must be inside the same job as the rows it is compared against, so a job carries its language's rows (mc plain and -O; clang -O2 and -O0; Zig ReleaseFast and Debug; ...) plus the reference. clang needs no install on any runner image, which is what makes the reference free everywhere. --cpus 1 --memory 512m becomes taskset -c 0 on the two Linux cells and nothing on macOS (D3): the workload is single-threaded and peaks at 51.3 MB RSS (94.6 MB for the C# JIT row), so a memory cap changes no number and a CPU quota is a share of a shared machine, not a speed.

3. Pinning (design question 2) #

3.1 One file, not two #

bench/cell/versions.env is the single source of truth, KEY=value lines, read by the workflow (cat bench/cell/versions.env >> "$GITHUB_ENV") and by the local road (. bench/cell/versions.env). The soak keeps its versions in the workflow's env: block, which is fine for a workflow-only measurement and would be a second copy the moment a local run has to agree with it.

Exact versions, and the two that are not exact today:

Every job records what the tools actually print into facts.json (go version, zig version, rustc -V, dotnet --version, clang --version | head -1, mc --version, mc --host), so a pin that silently resolves to something else is visible in the artefact rather than in nobody's memory.

3.2 How mc enters -- two roads, one digest rule #

• The default road is a release. gh release download of mc-<version>-<target>.tar.gz plus its .sha256, sha256sum -c BEFORE unpacking -- the soak's step verbatim (bench-soak.yml:127-137). This is the road that closes caveat 2 for good: the asset name carries the version, the checksum is the project's own published one, mc --version is recorded, and because every merged pull request cuts a tag (docs/ci.md § Versioning) the newest release is within one merge of main.
• The tree road is an input. workflow_dispatch with ref: builds the compiler from that commit -- on macos-15 that is make mc1, and the cell records git rev-parse HEAD. On the two Linux cells the tree road needs ci.yml's two-stage dance (cross-compile the ELF object on macOS, link it on the runner -- docs/ci.md § "The two-stage Linux design"); that is deferred and priced in § 9 rather than built in step 2, because the weekly history only ever needs a release.
• Whichever road, the cell records sha256 and size of the mc binary it timed, and of every binary it built. A digest cannot be stale. bench/results.json:234's failure mode is structurally impossible after that.

4. The comparison rule, the tolerance and the teeth (design question 3) #

4.1 One repetition schedule, written once #

Per job: build every row of its toolchain plus the reference (clang -O2 of bench/c/bench.c), then seven repetitions, rows interleaved within each repetition, in a fixed order, timed one at a time with nothing else running; on Linux each timed process is taskset -c 0. Repetition 1 is recorded and excluded from every statistic (§ 1.3 item 1). For each row the cell keeps all seven times, the best, the median and the max of the kept six.

4.2 The verdict is a ratio of medians #

For every row: ratio = median(row) / median(reference), both from the same six repetitions of the same job. The reference row's own ratio is 1.000 by construction, and clang -O0's is the second fixed point of the design (2.23-2.31 measured here, § 1.2) -- a row whose optimizer is famously off, which makes it a useful shape check.

Two runs of the same cell agree when, for every row, the ratio differs by no more than 5% relative. Measured basis: -0.8%, -0.7%, 0.0%, -3.2% for the four rows of § 1.2, back to back on a contaminated host. The number lives in bench/cell/versions.env as TOLERANCE=0.05 so the gate and the docs cannot disagree, and step 1's own two runs on this host are what ratify or tighten it (§ 9, gate 1): a spec cannot fix a tolerance it has only measured once, and this one has been measured twice on one host and zero times on a runner.

4.3 The teeth: --opt=0 against --opt=1 #

The mc job builds the same source three times -- plain, -O, and the reference -- and asserts median(plain) / median(-O) >= REGRESS_MIN with REGRESS_MIN=1.5 (measured 1.685x here, 1.99x in M49's own cross-session numbers, drift 0.1% between two runs). That is the plan row's "a deliberately regressed mc build fails the cell", built out of two flags that are both on main today:

• if the allocator, the peephole or the hoisting stops working, the ratio falls towards 1.0 and the assertion fails, in the same run, with no baseline to consult;
• the demonstration that the gate has teeth costs one dispatch with the -O row's flag set to --opt=0 (a one-flag regression): the cell must fail. That is step 3's own gate, and it needs no patched compiler to be kept alive in the tree.

4.4 What absolute seconds are for #

Recorded always, gated never (§ 1.3 item 2). The reference row's absolute median is the machine's health indicator: when it moves more than 10% from the newest committed run of the same cell, the run says so ("the machine moved: clang -O2 0.540 s -> 0.612 s") and the ratios are still read as valid. That sentence is what would have been printed between the three sessions of § 1.1, and it is the honest replacement for caveat 1: the cell does not claim a quiet machine, it measures whether the machine was the same one.

4.5 The regression comparison across runs #

bench/cell/gate.py compares a fresh results.json against the newest committed run of the same cell id (<os>-<arch>-<runner label>), never against a number written into a script: every row's ratio within TOLERANCE, the tooth above REGRESS_MIN, the reference's absolute time reported with the 10% note. A first run on a new cell id has nothing to compare against and says so (exit 0, "baseline recorded"): the x86-64 cell's first run is a baseline, not a regression (§ 11 risk 6).

5. The artefact, and how docs/comparison.md consumes it (design question 4) #

5.1 The directory and the schema #

bench/results/<date>-<run-id>/ -- the soak's shape (bench/soak/results/2026-09-06-34062566194/), so a later run sits beside an earlier one and git blame over results.json is the regression history the plan row asks for. Three files: results.json, facts.json, RESULTS.md.

results.json, one object per cell run:

{
  "cell": { "id": "linux-x86_64-ubuntu-latest", "os": "linux", "arch": "x86_64",
            "runner": "ubuntu-latest", "image": "ubuntu24 20260831.293.1",
            "cpu_model": "AMD EPYC 7763", "nproc": 4, "mem_kb": 16373452,
            "kernel": "6.17.0-1022-azure", "pinned_cpus": "0", "date": "2026-09-20T05:12:03Z",
            "run_id": "34062566194", "workflow_ref": "main" },
  "mc": { "road": "release", "version": "mc 0.15.15", "tag": "v0.15.15",
          "asset": "mc-0.15.15-linux-x86_64.tar.gz",
          "asset_sha256": "...", "binary_sha256": "...", "binary_bytes": 1181184,
          "commit": null, "host": "linux/x86_64" },
  "toolchains": { "clang": "Ubuntu clang version 18.1.3", "go": "go version go1.26.7 linux/amd64" },
  "reference": "c-O2",
  "reps": 7, "dropped": 1, "tolerance": 0.05, "regress_min": 1.5,
  "rows": [
    { "row": "mc-opt", "phase": "all", "build_cmd": "mc -O --exe bench/mc/bench.mc -o bin/mc-opt",
      "compile_s": 0.0041, "binary_bytes": 33461, "text_bytes": 1480,
      "binary_sha256": "...", "stdout_ok": true,
      "run_s": [1.01, 0.73, 0.75, 0.74, 0.75, 0.76, 0.75],
      "best": 0.73, "median": 0.75, "max": 0.76, "rss_bytes": 51347456,
      "ratio_to_reference": 1.389 }
  ],
  "verdict": { "status": "ok", "compared_to": "2026-09-13-34000000001",
               "worst_row": "c-O0", "worst_drift": -0.032,
               "tooth": 1.687, "reference_absolute_drift": 0.004 }
}

stdout_ok is bench/run.sh's own cross-check (8128903901837660708 / 3001134 / 39088169, or the phase's number) asserted per binary per repetition: a row whose output is wrong is not a fast row. facts.json is the soak's file, verbatim in spirit -- everything observable about the machine, the commands run, the sysctls -- and the two are separate for the same reason the soak separates them: one is the measurement, the other is the excuse.

5.2 The page is hand-edited, per promotion #

docs/comparison.md is not generated and gains no generator (D9). The precedent is in the tree: § "The hour under load" is hand-written from one soak run's RESULTS.md and links the dated directory. Two edits per promotion, one PR:

• § Conditions loses its two "read this with two things in mind" items and gains a cell of record line: the cell id, the run's dated directory, the mc tag with its asset checksum, and the reference row's absolute median with the tolerance the run used;
• the workload table gains the cell's numbers as a second block (one per cell id) beside the M4's own, which stays as the developer's host and is labelled as such.

Reasons, not taste: mcsite renders docs/**.md and scripts/check-docs.sh compiles every ```mc fence and resolves every link in it, so a generated table would put a benchmark run inside the documentation gate and make the page a build artefact; and git blame on a hand-written narrative is the history of what the project CLAIMS, while git blame on bench/results/*/results.json is the history of what it MEASURED. Those are two different histories and both are wanted.

5.3 Who commits it #

The workflow never pushes to main (permissions: contents: read, like the soak): it uploads bench-cell-<cell>-<run-id> and prints the one command that fetches the artefact into bench/results/<date>-<run-id>/. A human opens the docs-only PR, which is the merge discipline the repository already has (docs/ci.md § "Who touches what": the owner merges, squash only, and every merge cuts a tag). A workflow that committed results would bypass that and would make autotag.yml cut a version for a benchmark run.

6. Scope: the rows that are in, and the rows that are not (design question 5) #

In: the integer workload, six languages, eleven rows, three cells.

with clang -O2 of bench/c/bench.c built and timed in every job as the reference. Cells: macos-15 (arm64, Apple clang, mc --exe Mach-O), ubuntu-24.04-arm (arm64), ubuntu-latest (x86_64 -- the first timing M49 step D2's allocator has ever had). Six jobs per cell, eighteen jobs, each dominated by its toolchain install.

Also in, and it is new work: the three phases separately (mix, primes, fib), because M49's acceptance is stated per phase and nothing in the repository can produce those numbers today (§ 0). The six sources gain an optional argument -- no argument means today's behaviour and today's stdout, byte for byte, so bench/run.sh's contract and every recorded number stay comparable; m, p or f runs one phase and prints its own number. Dispatch is on the first byte, which costs 4-8 lines per language and no string comparison in mc.

Compile-side columns stay what bench/RESULTS.md § A records: compile seconds, compile RSS, binary bytes, __text bytes, and now the binary's SHA-256. A compile under 10 ms needs a better timer than /usr/bin/time (the recorded 4.1 ms is "min of 20 via perl Time::HiRes", bench/results.json host.timing_method); the cell keeps that method and records which one it used.

Out, with the reason:

• The HTTP throughput rows (bench/http/, bench/http2/): 0.9-228.5% recorded spread, a load generator sharing the cores it measures, a TIME_WAIT drain between runs, ab's HTTP/1.0 quirk, and 17 servers x 4 configurations x 3 runs of wall clock. No tolerance of the kind § 4 can state would hold. They stay a hand-run measurement, and the soak stays exactly as it is -- separate workflow, separate weekly hour, separate results directory. Giving the HTTP rows the same cell shape (one job per server, oha at a fixed rate, ratio to c-serial) is a later milestone's option, ~120 lines on top of the soak's own scripts, and it needs a measurement of its own before it is promised.
• The soak's protocol: untouched. M50 adds no input, no server and no column to it.
• make check: untouched (§ 7).
• Windows cells: mc --exe for Windows landed in M42 step 2, but bench/http/mc/* and the workload's <sys> road have never been built there and no recorded number exists to compare against. Out; one more cell later is a copy of the Linux job with -machine:x64.

7. What M50 does not do, and when the cell runs (design question 6) #

• No src/ code, no lib/, no stage0/, no tests/. git diff --stat src/ stage0/ lib/ tests/ is empty for the whole milestone. The compiler is measured, not changed: both roads the cell times (--opt=0, --opt=1) and the per-phase argument live in bench/, and the argument is in the benchmark's own six sources.
• make check does not change. The cell is not a gate: its numbers depend on the host's CPU and on installed toolchains, which is the rule bench/README.md:3-6 already states. Makefile gains bench-cell (the local road) and nothing in check:.
• When it runs: workflow_dispatch (with ref, cells, reps and mc-version inputs) and a weekly schedule at 06:00 UTC on Sunday, after the soak's 03:00 hour so the two never queue against each other. Not on a pull request (shared runners, and a benchmark that gates merges is a benchmark that gets ignored or a merge that blocks on noise) and not on a tag (a release must not wait on a measurement). What an optimizer milestone does instead is one line: gh workflow run bench-cell.yml -f ref=m52-inlining, before and after, and the two dated directories are the evidence -- which is precisely what M49 had to do by hand in a scratchpad.
• No new required status check on main (docs/ci.md:131-141).
• No Dockerfile in the recommended road (D13): the plan row's bench/Dockerfile is the right artefact for a road off GitHub and is priced in § 9 as an optional step, not built.

8. Acceptance (runnable; numbers, not adjectives) #

1. Two consecutive runs of the same cell agree within the documented tolerance: every row's ratio_to_reference within TOLERANCE (5%, ratified by step 1's own two runs), on each of the three cells; the two dated directories committed side by side so the agreement is in the repository and not in a log. Measured basis on this host: -0.8%, -0.7%, 0.0%, -3.2%.
2. A deliberately regressed mc fails the cell: one dispatch with the -O row built --opt=0 exits non-zero with tooth 1.00 below 1.50, and the same run with the flag restored exits 0 (1.5-2.0x expected; 1.685x measured here).
3. The two caveats are gone from docs/comparison.md § Conditions and replaced by a cell-of-record line naming the run, the mc tag, the asset checksum, the binary's digest and the reference's absolute median. No number on the page is attributed to an unverifiable label.
4. The x86-64 allocator has a time: mc -O / clang -O2 and mc plain / mc -O measured on ubuntu-latest, recorded, and read into docs/comparison.md -- the number M49 § 10 row 5 and § 12 risk 11 both defer to this cell. Whatever it is, it is the first one.
5. The three phases are measured separately on every cell, and M49 § 9.2's per-phase ceilings (mix <= 0.28 s, primes <= 0.21 s, fib <= 0.18 s on the M4) are re-stated as ratios to the reference so they survive a change of machine.
6. Every binary's stdout is the recorded cross-check in every repetition (8128903901837660708 / 3001134 / 39088169, or the phase's own number); bench/run.sh and make bench still pass unchanged.
7. results.json is complete enough to rebuild the run: the cell id, the CPU model, the kernel, the image, the pinned cores, every toolchain string as its tool prints it, every build command verbatim, every repetition's seconds, and a SHA-256 per binary.
8. make check is byte-for-byte the same gate: git diff Makefile touches only the new bench-cell target and .PHONY; check-docs green (every link in the new and edited pages resolves, no new ```mc fence); git diff --stat src/ stage0/ lib/ tests/ empty.
9. Runner cost recorded: wall-clock minutes per job and per run, so the weekly cron's price is a number in RESULTS.md and not a guess (estimate: 3-6 min per job, ~1 runner-hour per full run, against the soak's ~14).

9. Steps -- gated commits (design question 7) #

Needed to close the milestone: A, B, C.

10. Files and estimated deltas #

New code: about 730 lines outside src/ (python, sh, yml) plus ~45 in the benchmark sources. src/, lib/, stage0/, tests/: 0.

11. Risks #

1. Runner noise. Mitigated by the in-run reference, the interleaved repetitions, the discarded first repetition and a ratio verdict -- measured to 0.7% back to back on a contaminated host (§ 1.2). Not mitigated: a runner so busy that even the ratio moves; the reference's absolute time is what makes that visible (§ 4.4) rather than silent.
2. Image rotation and CPU-model change between weekly runs. ubuntu-latest moves; facts.json records the model and the image, and the gate reports a model change beside the drift instead of pretending the machines are the same. The ratios survive it; the absolute numbers do not, and are labelled.
3. Toolchain drift. The measured failure is already in the tree (go 1.26 -> two compilers, 10.0.x -> two SDKs). Cured by exact pins in one file plus recording what the tool prints; a silently moved pin shows up as a version string that differs from the previous run's.
4. The VPS is production. The reason it is not the road (§ 2). If the owner takes D13, the cell must be a second machine or a maintenance window, and the price is in that decision, not hidden in a script.
5. arm64 vs x86_64 asymmetry. M49's number is an Apple M4's; neither runner is one (ubuntu-24.04-arm is Neoverse, macos-15 is not an M4). The cell does not try to reproduce 1.30x: it records its own reference per cell id, and M49 D12's "M50's cell becomes the reference" is satisfied by naming which cell is the reference, not by matching an old number.
6. A first run has nothing to compare against. The gate says "baseline recorded" and exits 0 (§ 4.5). The x86-64 cell's first run is a baseline; reading a regression into it would be reading a regression into the only data point.
7. A phase argument changes the measured program. Mitigated by construction: no argument means the same three phases in the same order with the same stdout, so the recorded numbers and bench/run.sh's contract stay comparable; the phase mode is an extra entry, not a rewrite. stdout_ok per repetition is the check.
8. Runner minutes. ~1 hour per full run against the soak's ~14, weekly, on a public repository. Recorded (acceptance 9) so the cost is a number.
9. A benchmark nobody runs. The weekly cron plus ref dispatch is what keeps it alive; the one thing that would guarantee it is a required check on every PR, and § 7 refuses that on purpose. The mitigation is social and is written into bench/cell/README.md: an optimizer milestone dispatches the cell before and after, and cites the two dated directories.
10. macos-15 has no taskset. The macOS cell pins nothing and is therefore the noisiest of the three; it is kept because it is the only cell where mc --exe writes Mach-O and where Apple clang is the reference, which is the pair docs/comparison.md measured. Its tolerance may have to be wider; step 1's measurement on this Mac (±3.2% worst row, unpinned) says it probably does not.
11. 512 MiB and --cpus 1 are dropped from the plan row's wording (§ 2). If the owner wants them, they belong to D13's road, and the C# JIT row's 94.6 MB RSS plus the 50 MB sieve say 512 MiB is enough for the workload as written.

12. Decisions (architect) -- to ratify with the owner #

• D1 -- the cell runs on GitHub Actions, one job per TOOLCHAIN, three cells (macos-15, ubuntu-24.04-arm, ubuntu-latest). Recommend yes (§ 2). The VPS is production and is itself a shared VM; Actions is the only road with both architectures and it is the soak's proven shape.
• D2 -- the verdict is the ratio of per-row medians to a clang -O2 reference built and timed in the same job; absolute seconds are recorded and never gated. Recommend yes: measured 0.7% run-to-run against 33% cross-session on the same binary (§ 1.1, § 1.2).
• D3 -- 7 repetitions, rows interleaved, repetition 1 discarded, taskset -c 0 on Linux and nothing on macOS, no memory cap. Recommend yes (§ 1.3 items 1 and 5; § 2's last paragraph).
• D4 -- TOLERANCE = 0.05 on the ratio, provisional, ratified by step 1's two runs and stored in bench/cell/versions.env. Recommend yes; the measured back-to-back drift is 0.1-3.2%.
• D5 -- exact pins in one file (bench/cell/versions.env), and the two loose pins fixed: GO_VERSION=1.26.7, DOTNET_VERSION=10.0.400, clang-18 by version on Linux. Recommend yes: both loose pins have already resolved to two different compilers in two recorded runs (§ 1.1).
• D6 -- mc enters as a release tarball with its .sha256 verified (default) or as a tree at a ref (dispatch, macOS in step 2), and the cell always records the SHA-256 of the binary it timed. Recommend yes: this is what closes caveat 2, and a digest cannot go stale the way bench/results.json:234 did.
• D7 -- the teeth are --opt=0 against --opt=1 in one job, REGRESS_MIN = 1.5. Recommend yes: 1.685x measured here, 1.99x in M49's numbers, 0.1% run-to-run, and nothing to maintain.
• D8 -- the artefact is bench/results/<date>-<run-id>/{results.json,facts.json,RESULTS.md}, committed by a human in a docs-only PR; the workflow keeps contents: read. Recommend yes: the soak's precedent, and a workflow that pushed to main would cut a version for a benchmark.
• D9 -- docs/comparison.md stays hand-written; no generator writes into docs/. Recommend yes (§ 5.2): mcsite renders the page and check-docs gates it, and the two histories -- what the project claims and what it measured -- are better kept apart.
• D10 -- scope is the integer workload only; the HTTP rows and the soak stay out, unchanged. Recommend yes: 0.9-228.5% recorded spread against the workload's 2-8% (§ 1.1, § 1.3 item 7).
• D11 -- the six benchmark sources gain a phase argument, default behaviour and stdout unchanged. Recommend yes: M49's acceptance is per phase and nothing in the tree can measure that today (§ 0, § 6).
• D12 -- not in make check, not on a pull request, not on a tag: weekly cron at 06:00 UTC Sunday plus workflow_dispatch with a ref, and make bench-cell for the local road. Recommend yes (§ 7).
• D13 -- bench/cell/Dockerfile and the VPS road are deferred, priced (~130 lines plus an image build), and taken only on the owner's word. Recommend yes: on Actions the versions are already reconstructible, and the image's real value is a road off GitHub.
• D14 -- the cell records its own reference per cell id; M49's 1.30x stays an Apple M4 number and is labelled as one, and the x86-64 cell's first run is a baseline. Recommend yes (§ 11 risks 5 and 6). This is the ratification M49 D12 asks for, in the only form the hardware allows.
• D15 -- the reference row's absolute median is the machine's health indicator, reported with a 10% threshold, never gating. Recommend yes (§ 4.4): it is the honest replacement for caveat 1 -- the cell does not claim a quiet machine, it says whether the machine was the same one.

Implementation notes -- step A #

Written after the step landed, on the host § 1.2 measured: Apple M4, macOS 26.6.2, Darwin 25.6.0, 16 GiB, nothing pinned (macOS has no taskset, D3). Every number below is from six full runs of bench/cell/cell.py, taken as three consecutive PAIRS, 11 rows x 4 phases x 7 repetitions each.

1. All five toolchains were present here, so all eleven rows ran. Apple clang 21.0.0, go1.26.4 (versions.env pins 1.26.7 -- what CI installs), zig 0.16.0, rustc 1.96.0 and .NET SDK 10.0.301 (pinned 10.0.400). The two mismatches are exactly the drift D5 exists for and they are visible in facts.json rather than hidden: the cell records what each tool printed, never what the pin said. Nothing was faked and nothing was skipped for absence. cs-aot DID have to be un-skipped: NativeAOT links against libssl and libbrotli, which macOS does not ship, so the first two runs recorded it as skip cs-aot build failed (rc 1): ld: library 'ssl' not found -- the same wart bench/RESULTS.md § A already records. build.sh now sets LIBRARY_PATH to Homebrew's openssl@3 and brotli on Darwin, as that page did; where those are absent the row is skipped with the linker's own message, which is the behaviour the acceptance asks for.

1. Repetition 1 is dropped, and the gap lives on the phase that runs FIRST. Over every row of a run the median gap is only 1.03-1.10x, which would have made the drop look unnecessary; broken down per phase, all -- the first phase of each repetition, where the binary's pages and the 50 MB __bss sieve are first touched -- is 1.28x, 1.33x, 1.56x, 1.34x median with a worst row of 1.79x, and the three later phases of the same repetition are 1.01-1.12x. That is bench/results.json's own 1.25-1.62x band, and it is why results.json records rep1_over_kept_best_by_phase and not one diluted number.

1. The tolerance, measured -- and the one recommendation this step makes to step C. Worst per-row ratio drift inside a pair, with the count of rows of that phase over 5%:

   phase	pair 1	pair 2	pair 3	over 5% (of 10, 11, 11)
   all	2.44%	4.54%	4.76%	0, 0, 0
   mix	4.18%	4.97%	6.49%	0, 0, 1
   primes	3.56%	5.85%	8.36%	0, 1, 4
   fib	3.25%	6.84%	4.98%	0, 1, 0

   The median row moves 1.10-3.43% everywhere. The ratio's advantage over the absolute is real and smaller than § 1.2 suggested: back to back, the same runs' absolute medians moved by up to 15.77% (median 8.88% in pair 2) and the reference row alone by -9.34%, so the shape § 1.2 describes is reproduced -- the ratios held while the machine underneath did not -- but § 1.2's 0.7% was one pair of FOUR rows, and the worst of 44 rows is necessarily larger. In pair 3 the machine barely moved (reference -1.04%) and the worst ratio drift was still 8.36%: when both medians in the quotient are under 0.25 s, their independent scheduler noise ADDS instead of cancelling, which is § 1.3 item 3's finding one level down. So TOLERANCE = 0.05 is kept as D4 fixes it and it is right for the all phase (0 violations in three pairs, ~5% of margin) and wrong for the three short ones (7 of 99 rows over it). Recommended for step C, measured and not guessed: gate the band on all, report the per-phase ratios beside it, and re-measure on the two pinned Linux cells before widening anything there.

1. The teeth: REGRESS_PHASE, a deviation from § 4.3, decided by measurement. § 4.3 asserts the floor on the whole source; the tooth per phase over the six runs is

   phase	six runs	against the floor of 1.5
   all	1.554 … 1.619	holds by 4-8%
   mix	2.243 … 2.379	holds by 50%
   primes	1.049 … 1.093	would fail
   fib	0.996 … 1.037	would fail

   M49 moves mix by 2.3x, primes by 5% and fib by nothing at all (M49 § 10 rows 2-4: fib's cost is its call count), so all is a weighted average landing just above the floor -- and on the quieter host M49 step C measured, all was 0.793 / 0.548 = 1.45, which a 1.5 floor would have FAILED. versions.env therefore carries REGRESS_MIN=1.5 with REGRESS_PHASE=mix: the same floor, on the phase the optimizer is about, with half again of margin. § 1.2's own 1.685x was the all phase on a host running a Lima VM, which is why it read higher there than M49's 1.45.

1. The phase argument (D11) costs 78 added / 25 removed lines across the six sources and dispatches on the first byte with no string comparison: mix, primes, fib, and anything else -- an argument the caller mistyped included -- runs all three, so a human running it by hand is never handed silence. No argument is byte-for-byte today's output on all six (md5 c369e8b4679ca95f45fa46bbff03b7b7, which is bench/results.json's own output_md5_all_variants), verified per variant before and after. The one thing that does NOT stay byte-identical is the mc BINARY: i64 main() became i64 main(i64 argc, uptr argv), so gen_func emits two MTASK_PARAMs and the dispatch. Nothing gates a bench/ binary's bytes; risk 7's contract is the stdout, and it holds. Zig 0.16 was the only awkward one: std.process.args() no longer exists, and the iterator is std.process.Args.Iterator.initAllocator(init.minimal.args, init.gpa) -- probed against the installed standard library before the file was edited, not guessed.

1. build.sh prints its own run command. Its last line of stdout is run: <command>, so cell.py never learns that cs-jit is launched through the dotnet muxer (dotnet bench.dll) while every other row is a binary. The apphost next to bench.dll needs DOTNET_ROOT set unless dotnet lives in /usr/local/share/dotnet -- measured here as You must install .NET to run this application -- and the muxer on PATH is the same host process with nothing to configure.

1. Timing is time.monotonic() around one process at a time, with os.wait4 for max RSS, and results.json records that in a timing_method field rather than leaving it to be inferred. /usr/bin/time was not used: its 10 ms granularity is what forced bench/results.json's "min of 20 via perl Time::HiRes" for the sub-10 ms compiles, and monotonic has microsecond resolution on both hosts. ru_maxrss is bytes on Darwin and kibibytes on Linux; the cell normalises to bytes and says so. Compile seconds are one build on the same clock, recorded and never gated.

1. make check is untouched, which is acceptance 8: git diff main -- src/ stage0/ lib/ tests/ is empty, the Makefile diff is the new bench-cell target and its .PHONY entry, and check-docs is green -- the new pages are under bench/, which scripts/check-docs.sh does not walk, and no fenced ```mc block was added anywhere.

1. Results go to build/bench-cell/<date>-<id>/ by default, not to bench/results/. build/ is not tracked, so a local run stages nothing; --out bench/results/<date>-<run-id> is the directory step C commits. § 5.1 fixes that name and step C owns putting the first ones in it.

1. What step A deliberately did NOT build: bench/cell/gate.py (step C owns the comparison against the newest committed run of the same cell id -- the drift above was computed with a throwaway script, not with a committed one), .github/workflows/bench-cell.yml (step B), and any Dockerfile (D13, deferred). The built-in regression pair IS already reported and gated in every run, which is what this step owed.

Implementation notes -- step B #

Written after the step landed. Every number below is from three consecutive dispatches of the real workflow on GitHub's runners -- 34782461342, 34782812594, 34783020976, 21:00 to 21:13 UTC on 2026-09-13, 18 matrix jobs each. Nothing here was measured on the developer's Mac.

1. workflow_dispatch cannot reach a workflow that is not on the default branch. gh workflow run bench-cell.yml --ref m50-step-b is HTTP 404: workflow bench-cell.yml not found on the default branch, which is documented GitHub behaviour and not a repository setting. The three runs were therefore triggered by a push: trigger on the branch, added in its own commit and deleted before the branch was merged -- the merged file has workflow_dispatch and the cron and nothing else, as section 7 requires. Any later step that needs a run before its own merge pays the same one-commit price.

1. The runners are not the machines comparison.md measured, and the cell says so per run. macos-15 is an Apple M1 (Virtual), ubuntu-24.04-arm reports no model name at all (/proc/cpuinfo on arm64 Linux has none, so cpu_model is null -- the honest answer, not a gap), and ubuntu-latest came up as an AMD EPYC 7763 in the first run and an INTEL(R) XEON(R) PLATINUM 8573C in the second, twenty minutes later. Risk 2 happened while it was being watched, on its first day. It is in facts.json and in results.json's cell.cpu_model, and D14 is what makes it survivable: each cell records its own reference, and M49's 1.30x stays an Apple M4 number that no cell reproduces.

1. Acceptance 4, the first x86-64 timing of M49 step D2's allocator. ubuntu-latest, taskset -c 0, mc 0.15.34 from the release whose .sha256 the job verified, medians of six kept repetitions:

   	mc -O / clang -O2	mc plain / mc -O (the tooth)
   mix, run 2 (Xeon 8573C)	1.500	1.213
   mix, run 3 (Xeon 8573C)	1.447	1.193
   mix, run 1 (EPYC 7763)	1.45	1.182
   all, run 2	1.706	1.663
   all, run 3	1.690	1.299

   and the same rows on the two AArch64 cells, for the comparison that gives them meaning:

   cell	mc -O / clang -O2, mix	tooth, mix	mc -O / clang -O2, all
   linux-arm64 (Neoverse)	1.803 / 1.805	3.645 / 3.650	1.669 / 1.736
   macos-arm64 (M1 virtual)	1.465 / 1.420	2.305 / 2.422	1.499 / 1.453
   linux-x86_64 (Xeon)	1.500 / 1.447	1.213 / 1.193	1.706 / 1.690

   So the x86-64 allocator buys 19-21% on the phase it is about, where AArch64's buys 130-265%. That is not a surprise once M49 step D2 is read -- it allocates five registers (rbx, r12..r15) against AArch64's ten -- but until this run the only number that side had was an instruction count (-11.6%). Whatever it is, it is the first one, which is what section 8 item 4 asked for.

1. The floor had to become per-architecture, and that is the one design change step B makes. REGRESS_MIN = 1.5 was calibrated on AArch64 alone and x86-64 measures 1.18-1.21, so runs 1 and 2 FAILED on ubuntu-latest with tooth 1.181 below 1.50 -- the gate working exactly as designed, on a number nobody had measured when the threshold was written. versions.env gained REGRESS_MIN_X86_64=1.10 (7-9% below both measurements, 10% above the 1.00 the ratio collapses to when the allocator stops allocating) and cell.py reads REGRESS_MIN_<ARCH> before the global. REGRESS_PHASE needs no per-architecture value: on ubuntu-latest the tooth's run-to-run spread is 2.6% on mix against 28.7% on all, 69.3% on primes and 8.5% on fib, so mix is both the phase the optimizer is about and the only stable one. Run 3 is green on all three cells with the same three floors.

1. Two defects the first run found, both in step A's code, both invisible to the local road. (a) build.sh is handed the output directory and cds into the language's own directory for the go, zig and cs rows, so a RELATIVE --out -- which is what the workflow passes -- made -o <out>/go resolve against the wrong place and five of eleven rows were SKIPPED on every cell, with build.sh named a missing artefact and zig's own unable to open output directory. cell.py now makes the directory absolute before it builds anything; make bench-cell's default --out was already absolute, which is why six full local runs never saw it. Runs 2 and 3 have all eleven rows on all three cells and no skips at all. (b) The merge took each toolchain string with a blind update(), and a runner ships Go and Rust preinstalled -- so the five jobs that did NOT install them overwrote the pinned strings, and the first report claimed go1.24.13 and rustc 1.98.1 where the go and rust jobs had in fact installed 1.26.7 and 1.96.0. A merged string is now taken only from the job that owns the toolchain, plus clang, which every job uses as the reference. After the fix every cell reports exactly what versions.env pins: go1.26.7, rustc 1.96.0, dotnet 10.0.400, zig 0.16.0, Ubuntu clang 18.1.3 on Linux and Apple clang 17.0.0 on macOS. D5 is the reason this was caught rather than believed.

1. Acceptance 1 is met on ONE of the three cells, and that refutes step A's recommendation. Two consecutive complete runs (34782812594 and 34783020976), worst per-row ratio drift, 11 rows per cell, with the count over TOLERANCE in brackets:

   cell	all worst	over 5%	mix worst	primes worst	fib worst
   linux-arm64	4.80%	0	4.71%	6.39% (2)	0.51%
   macos-arm64	12.24%	2	7.19% (1)	32.22% (9)	8.30% (5)
   linux-x86_64	22.60%	1	5.12% (1)	26.48% (4)	24.31% (3)

   Step A measured three pairs on one Mac, found TOLERANCE = 0.05 sound on all and recommended gating there. On a shared runner it holds on linux-arm64 and fails on the other two. The mechanism is measured, not guessed: the reference is timed once per JOB, and between the two runs a job's own reference median moved by +31.81% (macos-arm64, the mc job), -23.57% (its go job) and -13.27% (linux-x86_64, the mc job) -- and inside one run the six per-job references of macos-arm64 spanned 0.556 to 0.719 s for the SAME clang -O2 binary, a 29% spread. A row's ratio carries the noise of its own job's reference, so comparing that ratio across runs adds two independent samples of runner load instead of cancelling them. What DOES hold across runs is a ratio whose numerator and denominator are two rows of the SAME job: the tooth drifts 0.13% (linux-arm64), 1.6% (linux-x86_64) and 5.1% (macos-arm64), and the mix median drift is 0.05-2.82% everywhere. Recommended to step C, which owns gate.py: gate on in-job ratios (the tooth, and a row against the reference of its own job in the SAME run), and treat a cross-run band as a report with a much wider tolerance -- 0.25 would cover everything measured here -- or drop it in favour of the per-cell history a committed results.json gives for free. Fixing the noise instead would mean one job per cell rather than one per toolchain, which is the trade section 2 priced and refused: a job's cost is its toolchain install.

1. Acceptance 9, the cron's price. Every job writes its own wall clock and the merged report carries the breakdown. Per job 35 s to 1m31s; per cell 4.12-6.18 min of runner time over its six jobs; 17.2 min of runner time for a full three-cell run (run 3: 5.71 + 5.71 + 6.15), in 2m25s to 3m1s of wall clock, because the 18 jobs run in parallel. The spec estimated 3-6 min per job and ~1 runner-hour per run: the real cost is about a third of that, because the timing itself is ~21 s and the toolchain install dominates. A weekly cron at 17 min of runner time is not a cost worth discussing -- the soak's hour per server is.

1. Acceptance 7, verbatim from a run. results.json carries the cell id, the runner label, the CPU model, nproc, mem_kb, the kernel string, the image (ubuntu24 20260907.300.1 -- the runner sets ImageOS/ImageVersion itself, so no step passes them in), the pinned cpus, the date, the run id and the workflow ref; the mc block with road: release, the tag, the asset name, the asset SHA-256 the job verified before unpacking and the SHA-256 and byte count of the binary that was timed (804135cf…f0d858, 1 446 104 B on linux-x86_64); every toolchain string as its tool printed it; the four pins; the schedule and both thresholds; then per (row, phase) the build and run commands, compile seconds, binary bytes, __text, the binary's SHA-256, stdout_ok, all seven repetition times, best/median/max of the kept six, max RSS and the ratio; then the verdict, the per-job reference medians and repetition 1's gap per phase. bench/results.json:234's "tree at commit e5a1643" failure mode is structurally impossible against that.

1. What step B deliberately did NOT build. No bench/cell/facts.py: the spec priced ~60 python for it, and cell.py already owned host_facts and the report writer, so a second file would have duplicated both -- what was added instead is the workflow-supplied identity (MC_ROAD, MC_TAG, MC_ASSET, MC_ASSET_SHA256, MC_COMMIT) and the runner's own ImageOS. No gate.py and no committed bench/results/ directory (step C). The tree road is built for macos-15 only, and the two Linux cells print that it is deferred, as section 3.2 says.

Implementation notes -- step C #

Written after the step landed, which closes the milestone. Every number below is from the two real dispatches of the workflow named in each item, or from the committed results.json files.

1. The comparison rule is the one step B's measurement supports, and it is narrower than § 4.2. bench/cell/gate.py gates the teeth and a wrong answer, and nothing else. The reason is not caution: it is that both medians in the tooth come from the same job of the same run, so the quotient carries no cross-run noise -- 0.13% / 1.6% / 5.1% drift measured by step B, and 0.25% / 1.6% / 4.4% measured again here between the committed run 34783020976 and this step's own 34785739260 (linux-arm64 3.650 -> 3.641, linux-x86_64 1.193 -> 1.212, macos-arm64 2.422 -> 2.315) -- while a row's ratio_to_reference carries the noise of its own job's reference, which moved +31.81% between two consecutive runs. So the per-row band is a REPORT with TOLERANCE as its threshold, printed worst-five-plus-a-count, and the reference's own absolute median is a NOTE at 10% (D15) beside a changed CPU model or image (risk 2). § 4.2's "two runs agree when every row's ratio differs by no more than 5%" survives as the thing the report measures; it is not what the exit code means.

1. A gate does not read its threshold out of the artefact it is gating. results.json records regress_min (the value cell.py resolved for that architecture), and gate.py deliberately ignores it and reads bench/cell/versions.env instead: a run that recorded a floor of 0.5 would otherwise pass its own tooth of 0.6. The per-architecture lookup is the same one cell.py does (REGRESS_MIN_<ARCH> before the global, aarch64 normalised to arm64), so the two cannot disagree about which floor an architecture has.

1. Acceptance 2 in full, both directions. workflow_dispatch reaches a workflow on a non-default branch as long as the FILE is on the default branch -- step B's note 1 is about the file, not the ref -- so gh workflow run bench-cell.yml --ref m50-step-c works now that step B is merged, and the run uses the branch's own workflow and scripts. The regression is § 4.3's own one-flag form, bench/cell/build.sh's opt="-O" set to opt="--opt=0" in a temporary commit dropped before the merge; --opt=0 is the default, so the two mc rows become the same binary and the tooth collapses to 1.0 exactly as designed.

   run	verdict	the tooth lines
   34785520475 (--opt=0)	FAILURE	linux-arm64: tooth 1.001 below 1.50 on phase mix; macos-arm64: tooth 0.981 below 1.50; linux-x86_64: tooth 1.006 below 1.10
   34785739260 (restored)	SUCCESS	3.641 / 1.50 (+143%), 2.315 / 1.50 (+54%), 1.212 / 1.10 (+10%), and ok 3 cell run(s) gated, no failures

   Two things worth keeping from the failing run. The three mc JOBS failed on their own (cell.py has gated the tooth since step A) and the report job's gate failed again on the merged per-cell reports, so the verdict is reached twice by two different readers of the same numbers. And the per-row REPORT named the regression from the other side -- mc-opt/mix ratio 1.805 -> 6.574 (+264.2%) on linux-arm64 -- which is what a committed per-cell history buys even on a band nobody gates.

1. bash -e is not bash -o pipefail. A run: step with no shell: key runs under bash -e, so gate.py | tee would have swallowed a non-zero exit. The gate step sets set -o pipefail explicitly, keeps the code in rc and ends with exit $rc, and it is the LAST step of report -- after the summary -- so a failure never hides the tables that produced it.

1. What is committed, and the one file that is not. bench/results/<date>-<run id>/<cell id>/ holds results.json and RESULTS.md, which is what gh run download of the merged report produces and what the workflow's own summary line tells a human to run. § 5.1 names a third file, facts.json, and it is NOT there: cell.py --merge writes two files per cell and the per-job facts.json stays in the 90-day artifacts. Nothing is lost that the verdict rests on -- the cell identity (cpu model, nproc, memory, kernel, image, pinned cpus, date, run id, workflow ref), the mc block with the verified asset digest and the timed binary's digest, every toolchain string as its tool PRINTED it, and the four pins are all inside results.json. Merging the facts blocks per cell is a later option; it would duplicate what results.json already carries.

1. One run is committed, not two. § 8 item 1 asks for "the two dated directories committed side by side"; what is here is step B's green run 34783020976, and the agreement between two consecutive runs is recorded in § Implementation notes -- step B item 6 (measured over 34782812594 and 34783020976) and re-measured in item 1 above. The weekly cron writes the second directory; committing this step's own green run instead would have made the gate compare a run against itself.

1. RESULTS.md's one relative link had to stop being relative. cell.py's report writer emitted a markdown link to ../cell/README.md, which is right from build/bench-cell/<run>/ and wrong from both bench/cell/report/<cell id>/ and the committed bench/results/<date>-<run>/<cell id>/ -- three depths, one link. It is now the repository-root path in backticks, correct from anywhere. The committed files of run 34783020976 keep the text that run produced: a measurement's artefact is not edited after the fact.

1. docs/comparison.md's caveats came out asymmetrically, and that is the honest shape. The tree label is GONE, because a digest cannot go stale and the page now carries the verified asset SHA-256 and the timed binary's. The shared host is not gone from the page -- it is REPLACED: the cell does not claim a quiet machine, it compares only numbers taken in the same run on the same core and reports the reference's own median so a reader can see when the machine moved. The page says so in those words, with the two facts that make it concrete (ubuntu-latest was an EPYC and then a Xeon twenty minutes apart; macos-15's per-job references spanned 29% inside one run). The HTTP tables keep their own ~20% caveat, unchanged and correctly scoped: § 6 puts those rows out of the cell on a measurement (0.9-228.5% recorded spread).

Edit this page