toolchain: a bootstrap finishes what it reassembles, and a sysroot is whole or made again

[Japabu]

Bootstrap recreates the primary's stage2 without its cargo link and without the clang src/clang.rs provisions. Three decisions downstream of that trusted the directory to have them:

• the primary bootstrapped on a moved mtime (target/stamps/compiler.stamp), not on a changed compiler;
• a sysroot counted as finished once it had SOURCES, so one cloned while stage2 had no cargo stayed broken for good;
• the owner's cargo run queued for the global lock behind other worktrees' sysroot builds after a bootstrap, to run the separate provisioning steps.

This closes issues/build/a-sysroot-cloned-during-a-toolchain-rebuild-never-gets-its-cargo.md, which it deletes: its exit condition is decision 2.

What changed, per decision

1. The primary bootstraps only when its compiler's content changed, when asked to, or when rustup has no toyos toolchain. The decision is toolchain::bootstrap(force_rebuild, current, toolchain_exists), a pure function. current is compiler::primary_is_current(rust/), which compares compiler::source(rust/) with the record rust/build/toyos-compiler. source is the git tree of compiler/, plus the working diff, plus untracked files, plus the src/llvm-project commit (clang from our own LLVM builds ToyOS's C, and toyos-cc is gone #541). A missing record means bootstrap.

   • rebuild_compiler removes the record (compiler::forget) before the bootstrap and writes it after reassemble. A stopped bootstrap is therefore run again by the primary, and a linked worktree gets choose's named refusal (run cargo run -- --build-only there) before any std build.
   • Removed: the mtime stamp, main's moved comparison beside it (primary_is_current subsumes it), the "record which compiler the toolchain is" act, and src/stamps.rs, which nothing reads now that toyos-cc is gone.
   • choose (a linked worktree against the primary) uses the same primary_is.
   • Why not identity.rs: every worktree's build system reads the record file. A format change would make each of those worktrees build a compiler of its own. Git's tree hash is already content identity.
   • record reads source(rust_dir) after the bootstrap it records, not before — a compiler/ edit made mid-bootstrap is folded into the record for a stage2 that does not contain it. Main has the same order, so this is filed rather than fixed here: issues/build/record-reads-compiler-source-after-the-bootstrap-it-records.md.

2. A toolchain directory is whole or it is not used. toolchain_defect is the one definition of whole: no narrated binary, rust-lld, and the C toolchain (clang::defect; clang::missing is gone). assert_toolchain_is_honest refuses by it.

   • One loop for a keyed product. buildlock::keyed_made is the use-or-make loop that sysroot::ensure and compiler::choose each carried inline. It makes the product while defect says it is not whole. It refuses a make that leaves the product not whole by that defect, and does not run make again. keyed_building and keyed_using are private now: keyed_made is their only caller outside buildlock's own tests, so a second use-or-make protocol cannot be written around them.
   • Sysroot. unfinished is "no SOURCES" or toolchain_defect. A sysroot that has SOURCES but is not whole is made again, all of it, under the key's exclusive lock. That includes one whose only defect is a dangling bin/cargo link: this is rare, and a sysroot has no repair path.
   • One policy for a compiler that is not whole. publish checks the compiler's stage2 before fill, and fill now runs build_std. A stopped bootstrap's stage2, without cargo or without clang, is refused before any std is built. The refusal names that stage2 and, for the primary's, cargo run -- --build-only in the primary checkout. Nothing is published. publish places no cargo of its own: complete and compiler::build_in_fork provision it in every stage2.

3. The owner's cargo run does not wait when no global change is needed. Both acts that run bootstrap (the toolchain and the hosted rustc) go through reassemble. reassemble runs complete() inside the same exclusive hold. complete() covers stage2's cargo, its clang, and the hosted sysroot's host target. Main's separate cargo and clang acts are gone.

   • An installed toolchain (Owner::Installed) keeps main's two steps, host target and cargo. Its clang is the artifact's own, and no LLVM sits beside it to provision from.

4. ensure and its tests share one production held. held(root, key, dir, make) is now the one function both call.

src/CLAUDE.md: main's "and never written again" is deleted from the sysroot bullet.

Gates

gate	exit
cargo test -p toyos-build --lib	0 (418 passed, 3 ignored)
cargo test --workspace --exclude toyos-build	0
cargo run -- --clippy	0 (10 invocations clean)

Build-system code only; no guest gate was run.

Negative controls

Each arm is a checked patch (git apply --check). For each one the mutated tree built (cargo test -p toyos-build --lib --no-run, exit 0), the named tests ran red (exit 101), and git apply -R left the tree clean. The line numbers are from the unmutated file.

arm	measured at	mutation	red
r1	c385c0c	delete publish's refusal of a compiler that is not whole (the review's "delete assert_toolchain_is_honest(&partial)": the check is now on stage2, before fill)	a_sysroot_is_whole_or_it_is_made_again at sysroot.rs:984 and a_sysroot_that_cannot_be_made_whole_is_made_once_and_refused at :1029. keyed_made refused sysroot fresh was made, and is not whole: … is missing cargo, which names neither the stage2 nor the remedy
r2	ef49a89	delete keyed_made's refusal after make	a_sysroot_that_cannot_be_made_whole_is_made_once_and_refused at :1043: the second make panicked a sysroot that was not whole was made again (left 2, right 1)
r2b	c385c0c	r1 and r2 together, the loop the review described	both tests fail and neither hangs: a sysroot that was not whole was made again: make 2 (:984), and the once assert (:1028)
r3	be5f500	if force_rebuild || !current → if force_rebuild	the_primary_bootstraps_when_asked_stale_or_missing: force_rebuild false, current false, toolchain_exists true, left None
r4	be5f500	delete crate::compiler::forget(rust_dir);	a_stopped_bootstrap_leaves_no_record: a stopped bootstrap left the record of the compiler before it
r5	be5f500	delete complete's host-target step	a_bootstrap_leaves_nothing_to_wait_on_a_sysroot_build_for: a bootstrap let its exclusive hold go with a global step left, …
m1	b05db87	fix 1 at its seam: primary_is_current becomes main's !(stamps::dir_changed(compiler/, stamp) || moved), and record writes the stamp where main wrote it (after bootstrap)	only_the_compiler_s_content_makes_the_primary_bootstrap: every file of compiler/ was rewritten with its own bytes, and the primary would bootstrap
m3	b05db87	all of fix 3: reassemble no longer completes, so completion runs under a hold of its own, as on main	a_bootstrap_leaves_nothing_to_wait_on_a_sysroot_build_for: a bootstrap let its exclusive hold go with a global step left, …
m3b	b05db87	complete/incomplete without clang	same test: a bootstrap let its exclusive hold go with stage2 not whole
r6	this head	held's || unfinished(dir) → || (!dir.join(SOURCES).is_file()).then(String::new)	a_sysroot_is_whole_or_it_is_made_again at sysroot.rs:990: assertion left == right failed: a sysroot without its cargo was trusted because it has SOURCES

No test depends on timing or on a dropped descriptor being closed. Every acquisition in keyed_made blocks, and none of the tests asserts a lock free.

Independent oracle: a recorded real failure, the owner's pasted wait on give the toyos toolchain its own cargo. The host's own timestamps agree with it: the stage2 cargo link appeared at 22:08:04, and sysroot 5dc157f7fac727be was made at 22:03:59 without cargo. m3 is that base behaviour, and its red names the queue.

🤖 Generated with Claude Code

[Japabu]

Review, round 1, at 8692732

CI: host is green at 8692732 (run 36466937181, conclusion success). git merge-tree against origin/main e3a1cdc is clean (6dd05d85).

BLOCKER

• src/sysroot.rs:388 — publish provisions cargo into the clone. That is a second code path: complete (toolchain.rs:356, now inside the bootstrap's hold) and compiler::build_in_fork already provision cargo in every stage2 before a clone can be taken. The only stage2 that reaches publish without cargo is a stopped bootstrap, and publish refuses that one on clang anyway, so one state gets two policies: heal the cargo, refuse the clang. Delete line 388 and its doc (379-381). The first half of a_sysroot_is_whole_or_it_is_made_again then asserts that a stage2 without cargo is refused by name and nothing is published.
• src/sysroot.rs:337 — held calls make() again until the sysroot is finished. Only publish's pre-fill assert (389) stops a stage2 without clang from rebuilding std forever, and no test covers that assert. Mutation: delete toolchain::assert_toolchain_is_honest(&partial); at 389, and a_sysroot_is_whole_or_it_is_made_again stays green because its stage2 has clang. Fix: held asserts finished(dir) after make() and names toolchain_defect. Add a test where the stage2 has no clang: held panics naming clang, with made == 1, and make panics if called a second time, so the mutation fails the test instead of hanging it.

NOTE

• src/toolchain.rs:452 — ensure's decision is untested, which answers the author's question. The patch if force_rebuild || !current → if force_rebuild passes every test; m1 only fails at the predicate. Move the decision into a pure fn (force_rebuild, current, toolchain_exists) -> Option<Bootstrap> and test that. Main's decision had no test either.
• src/toolchain.rs:466 — nothing removes the record when a bootstrap starts. A stopped --rebuild-toolchain or toolchain-missing bootstrap leaves the old record saying current, so the next build completes whatever stage2 the kill left. Removing primary_record before reassemble fixes both sides: the primary re-runs every stopped bootstrap, and compiler::choose gives linked worktrees a named refusal ("run cargo run -- --build-only there") before any std build.
• src/sysroot.rs:352 — the stopped-bootstrap case stops loudly; it is not a silent wait (389 panics naming the missing clang). But the panic comes after a full build_std, and it names the .partial clone instead of the primary's stage2 and the remedy. Check compiler.stage2 before build_std.
• src/sysroot.rs:328 — held is the same keyed_using/keyed_building loop that compiler::choose carries inline (compiler.rs:245). One helper could serve both.
• src/sysroot.rs:308 — finished now remakes a whole sysroot (6 std targets and libc) when only its bin/cargo link dangles, for example after the nightly toolchain is removed and host_cargo moves. That hits every key at once. The owner decides: accept the cost, or relink instead.
• src/clang.rs:119 — missing is defect(..).is_some(); delete it.
• src/toolchain.rs:1076 — the fixture never creates the hosted rustlib, so no test exercises the host-target part of complete/incomplete: deleting toolchain.rs:362-364 stays green.
• Growth: +317/−222 overall. Production is +198/−195, tests +118, and the issue file −26. Production is flat, and BLOCKER 1 plus the missing deletion would make it shrink.
• CI ran on a merge ref from before Host tests assert a lock free only after another process held it: the keyed-lock flake fixed #581 (b0c7efd) landed. The textual merge with e3a1cdc is clean, and Host tests assert a lock free only after another process held it: the keyed-lock flake fixed #581 touches only test helpers that the branch keeps using unchanged (estate, snapshot, write, git). The merge queue builds the combination.
• Merge b05db87 resolves every hunk. src/CLAUDE.md is main's text plus one clause (REMOVE below). Main's moved comparison and clang act are folded into primary_is_current/complete, build_c moved into fill, and check_installed_toolchain keeps main's two steps. Nothing the branch deletes (stamps, compiler.stamp, the two acts, the issue) is still read or cited.
• None of the three tests depends on timing or on a dropped fd being closed. Every acquisition in held blocks, none of them asserts a lock free, and the old 5 s recv_timeout is gone.
• No conflict with the LLVM-store recommendation. A keyed LLVM store moves where clang::provision reads clang from (today stage2/../llvm). The fix-3 fixture (toolchain.rs:1080) and the reason publish gives for refusing a stage2 without clang both assume that path, so both change with it.

REMOVE

• src/CLAUDE.md:22 — ", and never written again once whole": main's clause became false and was corrected instead of deleted.
• src/sysroot.rs:10-12 — "this machine's cargo," and "nothing writes it after its [SOURCES] file exists; one that has it and is not whole is not [finished], and is made again": rewritten prose, and it contradicts itself.
• src/compiler.rs:6-7 — "the primary bootstraps exactly when its compiler/ is no longer that": false, since force_rebuild and a missing toolchain also bootstrap.
• src/sysroot.rs:379-381 — "The cargo is this step's, never stage2's…": goes with BLOCKER 1, and it is false now that complete runs inside the hold.
• PR body, "What I am unsure of": this becomes main's record. Each bullet is either closed by BLOCKER 2 or the first two NOTEs, or belongs in issues/.

SEND BACK

[Japabu]

Review, round 2, at c385c0c

CI: host succeeded at c385c0c (run 36484645503, event pull_request, headSha c385c0c). git merge-tree --write-tree origin/main HEAD against e3a1cdc is clean (6c1607df, exit 0). origin/main is an ancestor of the head.

Round 1

• BLOCKER 1 (publish provisions its own cargo): CLOSED. publish places no cargo and refuses compiler.stage2 by toolchain_defect before fill. Arm r1 at c385c0c turns both sysroot tests red (exit 101).
• BLOCKER 2 (held re-makes without bound): CLOSED. keyed_made panics when a make leaves the product not whole. r2 at ef49a89 is red. r2b at c385c0c is red and bounded by most/once, not by a hang.
• The round-1 NOTEs are closed: the decision is a pure bootstrap (r3); forget runs before the bootstrap (r4); stage2 is checked before build_std (the test asserts !filled); choose and the sysroot share one loop; clang::missing is gone; the hosted rustlib fixture exists (r5); the dangling-cargo remake is recorded in one clause on unfinished.
• The round-1 REMOVEs are closed: main's "and never written again" is deleted rather than rewritten, sysroot.rs:10-12 is deleted, compiler.rs:6-7 is main's text again, and "What I am unsure of" is gone.
• rustup run on every build is main's cost, not a new one: origin/main src/toolchain.rs:404 computes toolchain_exists unconditionally in the same decide, and source is computed there too.
• Locks: every acquisition the new tests make goes through keyed_made, which blocks. No new test calls try_lock or keyed_idle, or asserts a lock free.

BLOCKER

• src/sysroot.rs:326 — The test goes around the call it claims to cover. ensure's || unfinished(&dir) has no test. a_sysroot_is_whole_or_it_is_made_again goes through a test-only copy of the call (held, sysroot.rs:952), which replaced round 1's production held that ensure itself called. The mutation - buildlock::keyed_made(root, Keyed::Sysroot, &key, || unfinished(&dir), || { / + buildlock::keyed_made(root, Keyed::Sysroot, &key, || (!dir.join(SOURCES).is_file()).then(String::new), || { is the deleted issue's bug restored, and every test stays green. Fix: make one production fn that both ensure and the tests call (round 1's held over keyed_made). This mutation must then turn a_sysroot_is_whole_or_it_is_made_again red.

NOTE

• src/buildlock.rs:268,275 — keyed_building and keyed_using are pub, but outside buildlock's own tests their only caller is keyed_made. Make them private so that a second use-or-make protocol cannot be written outside buildlock.
• src/toolchain.rs:401 — complete_toolchain is an abstraction with one caller (ensure, :516). Inline its act_if there.
• src/toolchain.rs:381 — record writes source(rust_dir) as it reads after the bootstrap, not as it read when the bootstrap began. A compiler/ edit made during the bootstrap is recorded as current for a stage2 that does not contain it. Main has the same order, so this is outside this branch: file it in issues/.
• Growth: +481/−240 overall. Production is +252/−210 (net +42, including stamps.rs −55), tests +228/−3, the issue file −26, and src/CLAUDE.md ±1. The growth is the pure decision, keyed_made, and forget/primary_is*/complete replacing the separate acts. The two NOTEs above are the cuts left.
• Proposed rule ("a test that exercises a loop must bound it with a panic"): no. The prompt already counts a negative control only by its exit code and log, so a hung arm has no measurement and is not counted, as happened this round. The Waits bullet already requires a loud bound. The sentence would restate both.

REMOVE

• src/toolchain.rs:470 — "Asked first, so an LLVM edit no commit holds is refused before any build.": this is main's comment rewritten. "First" relative to the rustup call orders nothing, and bootstrap(force_rebuild, current, …) taking current by value already makes every decide compute it.
• PR body, "In the first version of the r2b arm the wholeness test hung…": this paragraph is the measurement's history, not main's record. The r2b row stands on its own.

SEND BACK

[Japabu]

Review, round 3, at f23fea7

CI: host succeeded at f23fea7 (run 36488832770, job 109152240781, event pull_request). git merge-tree --write-tree origin/main HEAD against e3a1cdc is clean (cb336dde, exit 0). origin/main is an ancestor of the head. The reviewer ran cargo test -p toyos-build --lib at f23fea7: exit 0, 418 passed, 3 ignored.

Round 2

• BLOCKER src/sysroot.rs:326 (the test went around ensure's call): CLOSED. ensure (:331) and all four test call sites (:980, :989, :994, :1017, :1034) now go through the one production held (:316). The reviewer applied toolfix-r3/mutation.patch (|| unfinished(dir) → || (!dir.join(SOURCES).is_file()).then(String::new)) and ran cargo test -p toyos-build --lib sysroot::: exit 101. a_sysroot_is_whole_or_it_is_made_again is red at sysroot.rs:990 ("a sysroot without its cargo was trusted because it has SOURCES"), and a_sysroot_that_cannot_be_made_whole_is_made_once_and_refused is red as well. After git apply -R, git status --short is empty.
• NOTE buildlock.rs:268,275: CLOSED. keyed_building and keyed_using are private. Their callers are keyed_made and buildlock's own tests.
• NOTE toolchain.rs:401 complete_toolchain: CLOSED. It is inlined at :506.
• NOTE record order: CLOSED. It is filed as issues/build/record-reads-compiler-source-after-the-bootstrap-it-records.md, and its claim holds: record's only production caller is rebuild_compiler (toolchain.rs:384), which runs after reassemble.
• REMOVE toolchain.rs:470 and the PR body's r2b hang paragraph: CLOSED. Both are deleted.

Growth: +502/−241 overall. This round is production +18/−21, tests −5, and the issue +28.

BLOCKER

None.

NOTE

None.

REMOVE

• src/toolchain.rs:503-505 — "Completes a toolchain whose bootstrap was stopped before reassemble finished it; …" — false. A stopped bootstrap has no record (forget runs first), so "build the rust toolchain" runs it again, and its reassemble completes it. This act fires when host_cargo moves (for example, a nightly is installed) or when stage2 is changed outside a bootstrap. Delete the comment.
• PR body, point 3, "Completing a bootstrap that was stopped before it finished is one act_if left inline in ensure; …" — the same false claim.
• issues/build/record-reads-compiler-source-after-the-bootstrap-it-records.md:19-22 — "Main has the same order … carried forward unchanged." — this is the branch's story. After the merge, "this branch" names nothing.
• PR body, point 4 — this is review history ("Round 2 left ensure calling …"), not main's record. held being the one call is already in the code.
• PR body, Gates, "Rerun for round 3, whose head includes origin/main e3a1cdc already (…)" — it will rot.
• PR body, r6 row, "(the round-2 review's restored-bug mutation)" — review history.
• PR body, point 1, "current is asked before anything else, so an LLVM edit no commit holds is refused before any build, as main's decision did." — this is the ordering claim that round 2 removed from the code, surviving in the body.

LAND AFTER NAMED CHANGES