Skip to content

tests: a pty tethers QEMU and the judge's servers to the harness - #555

Merged
Japabu merged 10 commits into
mainfrom
wt/toyos-reap
Sep 29, 2026
Merged

Japabu merged 10 commits into
mainfrom
wt/toyos-reap

Conversation

@Japabu

@Japabu Japabu commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

A QEMU the harness started ran for hours after its harness was gone. The harness stops QEMU only in Drop, a SIGKILLed harness runs none, and its guest was reparented to init. The HTTPS judge's servers had the same shape.

What changed, per decision

One construction: toyos_build::tether::spawn (src/tether.rs). The child becomes the controlling process of a pseudo-terminal whose master only the harness holds. The kernel closes the master when the harness dies by any signal, and the hangup sends the child SIGHUP. The master is a field of whatever owns the child (QemuInstance, the judge's Server). No new binary, no per-test bookkeeping.

What the child owes. The tether holds only for a child that exits on SIGHUP, keeps the descriptor it inherited open, and starts no session of its own; spawn's contract says so. QEMU exits on SIGHUP (measured below). The judge's servers install no handler, so the default disposition ends them. Neither child forks.

Candidates, measured on the macOS dev host (QEMU 11.1.1; a held -S guest spawned by a holder process; the holder SIGKILLed; the QEMU pid polled for 5 s; ps before and after):

candidate QEMU after the holder's SIGKILL
stdin a pipe only the holder writes (QEMU reads EOF) alive 5 s later, PPID 1
a process group of its own alive 5 s later, PPID 1
-run-with exit-with-parent=on gone in 3.7 ms
controlling pty, master held by the holder gone in 3.1 ms

exit-with-parent is rejected. It ends QEMU alone, so the judge's servers would need a second mechanism. On Linux it is PR_SET_PDEATHSIG, which by prctl(2) follows the spawning thread rather than the process, so the two hosts would disagree about what a guest is tied to. The pty also ended a non-QEMU child (/bin/sleep) in 2.9 ms.

What pre_exec does, each piece measured. SIGHUP is unblocked and reset to default in the child. With it inherited blocked, QEMU survived its holder; with it inherited ignored, a child with no handler of its own (/bin/sleep) survived. The slave stays open across exec: closed at exec, QEMU survived on macOS. No hangup can precede TIOCSCTTY, because the child holds its own copy of the master until exec closes it.

Scope. Tethered: the processes that end only when the harness ends them, QEMU and the judge's servers. Builds and one-shot clients are not: a build killed mid-way can leave a toolchain half-written.

Drop's kill stays. SIGKILL cannot be caught, and Drop must reap QEMU before its NvmeClaim is released. A hangup ends QEMU too (dropping the master alone ended a held QEMU in 1.1–2.0 ms, exit 0), but it depends on QEMU's handler and prints terminating on signal 1 on the suite's stderr for every guest.

The owner's /tmp root goes with the test (TempDir::adopt). On main every boot takes TempDir::short("boot"), so the SIGKILLed --hold owner leaves /tmp/toyos-tmp-<pid>-<n>, and the test process has already made its one sweep of /tmp; ci guest's left_behind would red on it. toyos_tmpdir::TempDir::adopt(base, pid) moves every gone root of pid under base into the caller's directory, under the base's lock, so it goes when that does. A live root is never moved, and liveness stays the owner lock, never the pid.

adopt's liveness filter is proven, not assumed. an_adopted_root_goes_with_its_adopter now also makes a root named for the adopted pid — toyos-tmp-<pid>-9 — whose owner file the test itself holds locked, standing in for a live process that reused that pid, and asserts it survives adopt. Before this the test's only live root belonged to a different pid, so the prefix match alone kept it out and the liveness filter (gone_under, inside adopt) could be deleted without turning anything red. adopt and State::sweep now share that rename-into-reap-<name> step through one reap_into(base, dest, take) helper.

No QEMU test measures time (main's CLAUDE.md:97, #562). killed now returns Result<(), String>; the verdict rests on the event (the owner's stderr pipe reaching EOF) alone, and neither arm's print names how long that took.

The tests

Host arm: tether::tests::a_tethered_child_dies_with_its_owner (cargo test --lib). An owner process spawns a tethered child and is SIGKILLed. The owner inherits SIGHUP blocked and ignored, so the reset is exercised. The verdict is the owner's stderr pipe (std's Stdio::piped(), taken from the child) reaching EOF, which happens once every holder has exited, bounded by WITHIN (10 s). The owner runs in a process group of its own, which a tethered child leaves by setsid and an untethered one stays in. On a timeout the owner is not yet reaped, so its pid, and the group it leads, can be no other process's: the group is SIGKILLed, rather than the children's pids by number (their numbers went free once init reaped them). The refusal names the pids that still answer signal 0 and says whether every holder of the stderr then exited, which is how the test checks that nothing it started still runs. The owner runs with --test-threads 1. Owner::said waits with recv_timeout and matches its prefix with split_once, so a line libtest has already begun still matches. An Owner dropped is killed and reaped.

Guest arm: guest_dies_with_its_harness (Fast tier). The same judge, with the harness binary itself as the owner. The test builds the image into a toyos_tmpdir::TempDir it holds and runs the harness with --hold <image> and TMPDIR set to that directory. The owner boots the image Staged::Pristine and holds it until stdin ends. --hold is refused beside any other word. After the verdict, on every path and with the owner reaped, the test adopts the owner's /tmp roots into a TempDir::short of its own; its $TMPDIR scratch goes with the TempDir it set as the owner's TMPDIR. The green verdict rests on QEMU inheriting the harness's stderr; spawn_and_wait_ready says so where it sets that.

toyos-tmpdir: an_adopted_root_goes_with_its_adopter. A holder is SIGKILLed beside a live one, a gone root named for pid <pid>0, and a live root named toyos-tmp-<pid>-9 whose owner the test holds locked. adopt moves the killed root, with its directory and image, into the adopter; the other three stay, and the adopter's drop removes what it took.

The judge's servers. Only the QEMU site has an arm of its own; reverting the https.rs hunk alone leaves every test green. The site is kept because the servers are a process the harness has to end. They install no SIGHUP handler, which is the case the host arm's parked child exercises: a child with no handler, ended by the default disposition. That claim rests on the host arm.

Gates (head 6d82ff9, macOS)

  • cargo test -p toyos-build --lib: EXIT=0, 420 passed, 5 ignored.
  • cargo test --workspace --exclude toyos-build: EXIT=0.
  • cargo test --test toyos-build -- --list: EXIT=0; lists Fast guest_dies_with_its_harness.
  • cargo run -- --clippy: EXIT=0, 10 invocations clean.
  • cargo test -p toyos-tmpdir: EXIT=0, 8 passed in reclaim.

Linux: run 36351950439 (at d265676) logged [orphan] QEMU 31334 gone 1.55541ms after its harness's SIGKILL and PASS guest_dies_with_its_harness in guest (2). At 6d82ff9, branch nightly 36496779560 ran every guest shard, audio shard, tcg, host, build, portability-linux and portability-macos green; guest (2) logged [orphan] QEMU 31374 gone 2.532748ms after its harness's SIGKILL, PASS guest_dies_with_its_harness and nothing left in $TMPDIR or /tmp: every test took its scratch with it. portability-windows failed there as it does on main's own nightly (36400924827), tracked below.

Round-4 gates (head 46fd916, macOS)

  • cargo run -- --ci host: EXIT=0, [ci] Host: 54 step(s), all green.
  • cargo run -- --build-only: EXIT=0, Build finished.
  • guest-untethered.patch regenerated unchanged at reap-r4/guest-untethered.patch (round-3's content still applies cleanly): cargo test --test toyos-build --no-run on the mutated tree, EXIT=0; reverted, tree clean.
  • Guest arm at 46fd916, run by the orchestrator: guest_dies_with_its_harness EXIT=0; under guest-untethered.patch, EXIT=1.

Negative controls

Each control is a checked patch, shown to build, run, and restored in the same script, with the tree clean after it.

  • The unit test's owner spawning parked untethered (parked.spawn() for spawn(parked)), at 6d82ff9's content: EXIT=101 after 10.02 s, of its tethered children [35995], [35995] still answered; the owner's process group was killed, and every holder of its stderr then exited. No parked process remained (ps).
  • At d265676, each of these was EXIT=101 naming the child's pid: spawn returning a plain cmd.spawn(); the slave left close-on-exec; SIGHUP not unblocked; SIGHUP not reset to default; no TIOCSCTTY. An owner that never says its pid: EXIT=101 with the owner had not said "tethered " within 10s, after 11 s.
  • adopt moving nothing: EXIT=101, … was not adopted. The pid prefix without its trailing -: EXIT=101, another pid's root was adopted.
  • reap_into's liveness filter, where it now lives, shared by adopt and sweep: replacing its gone_under(base) call with a plain fs::read_dir(base) turns an_adopted_root_goes_with_its_adopter red — EXIT=101, a live root of a reused pid was adopted — restored, it is green.
  • Guest arm (tests/common/qemu.rs's tether::spawn reverted to qemu.spawn(), the field dropped, pid() kept), measured by the orchestrator on the dev host (macOS) at 6d82ff9: green arm EXIT=0, [orphan] QEMU 14834 gone 12.651708ms; the same command under the patch, EXIT=1, of its tethered children [18514], [18514] still answered; the owner's process group was killed, and every holder of its stderr then exited. Measured on macOS only: the red arm was never measured on Linux (see Unsure); the two hosts run the same QEMU 11.1.1, and the macOS control already shows a broken stdout does not end QEMU within WITHIN.

Independent oracles

  • The OS's process accounting. In the candidate measurements, ps shows each survivor with PPID 1 and each tethered child absent. During the host whole-revert arm at d265676, ps -axo pid,ppid,stat,etime,command taken 3 s into the wait shows the parked child alive with PPID 1 and its owner gone.
  • ci guest's own scratch check. src/ci.rs's left_behind judges the merged head's nightly shard that runs the guest arm, including /tmp.

Unsure

  • On macOS std marks a pipe close-on-exec only after the pipe exists, so a sibling's spawn in that window can hold the owner's stderr and red either arm falsely, never green it: filed as issues/build/a-pipe-made-on-macos-can-leak-into-a-sibling-spawn-and-red-the-tether-tests.md.
  • src/tether.rs does not compile on Windows; recorded in issues/build/the-build-system-does-not-compile-on-windows.md.
  • The guest-arm negative control's red arm has only ever run on macOS; nothing has forced it on Linux.

🤖 Generated with Claude Code

Japabu and others added 2 commits September 27, 2026 22:33
A QEMU the harness started ran for hours after its harness was gone: the
harness stops QEMU only in `Drop`, and a SIGKILLed harness runs none, so
its guest was reparented to init and ran on. The HTTPS judge's servers
had the same shape.

`toyos_build::tether::spawn` makes the child the controlling process of
a pseudo-terminal whose master only the harness holds. The kernel closes
the master when the harness dies by any signal, and the hangup sends the
child SIGHUP, on macOS and Linux alike. QEMU and the judge's servers are
spawned through it; `Drop`'s kill stays, so a clean end never waits on
the hangup.

Measured on this host (QEMU 11.1.1, a held `-S` guest, the holder
SIGKILLed): a stdin pipe's EOF leaves QEMU running (alive 5 s later), a
process group of its own leaves it running, `-run-with
exit-with-parent=on` ends it in 3.7 ms, the pty ends it in 3.1 ms. The
slave has to stay open in the child (closed at exec, QEMU survived), and
SIGHUP has to be unblocked (inherited blocked, QEMU survived) and reset
to default (inherited ignored, a child with no handler survived).
`exit-with-parent` was rejected: on Linux it is PR_SET_PDEATHSIG, which
follows the spawning thread, and it ends QEMU alone.

`a_tethered_child_dies_with_its_owner` is the host arm; the harness's
`guest_dies_with_its_harness` is the guest arm, whose owner is the
harness itself re-run with `--hold`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`Child::wait` closes the child's stdin, and the tethered child the host
arm parks reads that pipe to its end: with the mechanism reverted the
child still exited 4.6 ms after its owner's SIGKILL, on the pipe's EOF
rather than on a hangup, and the arm stayed green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu
Japabu marked this pull request as ready for review September 27, 2026 20:40
The owner's rules: only the orchestrator runs guest tests, and an agent cleans up every process it started before it reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator Author

Review, round 1, head aba9a95

Readiness. CI host passes at aba9a95 (run 36348983890, conclusion pass, [ci] Host: 48 step(s), all green). tether::tests::a_tethered_child_dies_with_its_owner ... ok ran on macos-latest (aarch64-apple-darwin). The guest arm guest_dies_with_its_harness has not run anywhere; it is owed to the orchestrator (see "Guest runs required").

Growth. +328/−11 (git diff --shortstat).

  • Production: +115/−7 (src/tether.rs:1-102, src/lib.rs, the qemu.rs and https.rs call sites).
  • Test support: +212/−3 (src/tether.rs:103-239, orphan.rs, testargs.rs, toyos.rs, pid(), sourcegate.rs).
  • CLAUDE.md: +1/−1.
  • Nothing in test support is dead. What should change is B3, not the size.

BLOCKER

  • B1 src/tether.rs:75-84 — The Linux peer path, and Linux's hangup semantics, have never run. The PR body says "CI's host lane runs the host arm there", which is false: ci.yml's host job is macos-latest, and nightly's host is macOS too. The first execution would be the nightly's ubuntu guest lane after the merge, where every QEMU spawn goes through this function, so a mistake reds all twelve shards and tcg. One cheap measurement settles it before landing: gh workflow run nightly.yml --ref wt/toyos-reap, with the guest shards and tcg green except for what --known-red names, and guest_dies_with_its_harness green in its shard's log.
  • B2 tests/common/orphan.rs:27-28 — The --hold owner takes a toyos_tmpdir root in the inherited $TMPDIR, and that root holds its per-boot image. The test SIGKILLs the owner, so the root stays. The parent harness has already done its once-per-process sweep (toyos-tmpdir/src/lib.rs:82), so nothing reclaims the root in this run. As a result, ci guest's unconditional "nothing left in $TMPDIR" step (src/ci.rs:642) reds the nightly shard that runs this test, unless some later process happens to be the first to sweep. The PR's "Unsure" section names the leak and accepts it. Fix: give the owner TMPDIR = a toyos_tmpdir::TempDir the test holds, and drop it after the verdict (the owner is dead and its lock released, so its root goes with it). Measurement: the shard's "nothing left in $TMPDIR" step is green in B1's nightly.
  • B3 src/tether.rs:154-170, :206-210 — Owner::said blocks on read_line with no bound. It also matches with strip_prefix, but when libtest runs one test thread, its pretty formatter writes test tether::tests::owner ... at the start of the same line. So with RUST_TEST_THREADS=1 inherited, or on a one-CPU host, tethered <pid> is never found. The owner reads stdin forever, and the host arm hangs with no verdict. This violates the Waits rule. The tree already solved this shape in toyos-tmpdir/tests/reclaim.rs:56-80: pass --test-threads 1 explicitly, match with split_once, and read through a channel with recv_timeout. Do the same. Measurement: RUST_TEST_THREADS=1 cargo test --lib tether::tests::a_tethered_child_dies_with_its_owner hangs at aba9a95, which the implementer shows with an outer bound, and exits 0 after the fix. An owner that never says the prefix makes said exit 101 with the reason.

NOTE

  • tests/common/https.rs:293-295 — No arm can go red on this site. Reverting it alone passes everything. The body's guest negative control reverts the qemu.rs and https.rs hunks together, but only the QEMU half can turn it red. Revert the qemu.rs hunk alone for that arm, and let the server's claim rest on the host arm.

  • tests/common/qemu.rs:4899 — The guest arm's green verdict is "the owner's stderr reached EOF" (src/tether.rs:182). That holds only while QEMU inherits the harness's stderr. The whole revert plus .stderr(Stdio::null()) here keeps guest_dies_with_its_harness green while QEMU is orphaned. State that invariant in one clause at this line.

  • src/tether.rs:27-29 — spawn's boundary contract does not say what the child owes, which is what "double fork" turns on. The child must:

    • exit on SIGHUP (QEMU does, measured; the https server installs no handler);
    • keep the descriptor it inherited, since on macOS a closed slave means no hangup (measured, pty-cloexec);
    • start no session of its own. Descendants in its process group get SIGHUP only when the leader exits (unmeasured). Neither current child forks, so no path exists today.

    Name these three in one clause.

  • src/tether.rs:106 — WITHIN is pub with no reader outside this file. Make it private.

  • tests/common/orphan.rs:30 — said waits for the owner's whole image build and boot. Its only bound is wait_for_ready's inside the owner; B3's recv_timeout must allow for that.

  • On the brief's questions, no finding:

    • Drop's kill still earns its place. SIGKILL cannot be caught, and Drop must reap before NvmeClaim is released (qemu.rs:3806-3810). Replacing it with a hangup would make that wait depend on QEMU's handler.
    • --hold is justified. Only the harness binary as owner covers spawn_and_wait_ready's call site, and the flag refuses every other word.
    • The pty tether is one rule in one function.

REMOVE

  • PR body — "on macOS and Linux alike". Linux is unmeasured (B1).
  • PR body, Unsure — "CI's host lane runs the host arm there". False.
  • PR body, Unsure — the scratch-root bullet. Moot once B2 is fixed.
  • src/tether.rs:1-2 — "cannot outlive the harness, however the harness ends". This overclaims: it holds only under the contract in the NOTE on :27-29.
  • tests/common/qemu.rs:3362 — "The QEMU process's pid." Narrates the signature.
  • .claude/agents/implementer.md:22 — "cargo test, never cargo run". cargo test is the QEMU harness (CLAUDE.md:71), so this line now contradicts aba9a95's "Agents never run QEMU". aba9a95's sentence itself is accurate.

Guest runs required (orchestrator), at the fixed head

  1. Dev host: cargo test --test toyos-build -- guest_dies_with_its_harness EXIT=0 with [orphan] QEMU <pid> gone …, then ps -p <pid> empty.
  2. Dev host, negative control: the qemu.rs hunk of reap-r1/guest/revert-mechanism.patch alone, shown to build, then the same command. Expected: EXIT=1 with still ran 10s … [pid] did, and were killed, and ps -p <pid> empty afterwards.
  3. Dev host: the Fast tier whole. Every guest now spawns through the tether. Green except known reds.
  4. Linux: the nightly on the branch (B1), including each shard's "nothing left in $TMPDIR" (B2).

SEND BACK

Japabu and others added 2 commits September 27, 2026 23:04
…arms to the orchestrator

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… with it

`Owner::said` read the owner's stdout with `read_line`, unbounded, and
matched with `strip_prefix`. With one libtest thread the pretty formatter
starts the owner's line with `test tether::tests::owner ... `, so the
prefix was never found and the host arm hung with no verdict:
`RUST_TEST_THREADS=1` at 930396e was still running after 60 s. The
owner's stdout is now read by a thread into a channel, `said` takes its
bound and waits with `recv_timeout`, and the prefix is matched with
`split_once` wherever on the line it starts. The owner is run with
`--test-threads 1` explicitly. An `Owner` dropped is killed and reaped,
so an owner a failed wait left running goes with the test.

The guest arm's owner took a `toyos_tmpdir` root in the inherited
`$TMPDIR`, and its SIGKILL left it there for `ci guest`'s "nothing left in
$TMPDIR" step. The owner's `$TMPDIR` is now a `TempDir` the test holds and
drops after the verdict. The test also builds the image there and hands
it to the owner (`--hold <image>`, booted `Staged::Pristine`), so the
owner builds nothing: its one wait is its boot, which its own
`wait_for_ready` ceiling ends at 10 s x its width floor of 2 x a fresh
process's host scale of 1 x the default two vCPUs' oversubscription of
at most 2, 40 s, inside `qemu::GUEST_WEDGED`, which bounds `said`.

`WITHIN` is private. `spawn`'s contract names what the child owes; the
module's opening overclaim and `pid`'s narrating doc are gone; the
invariant the guest arm's green rests on, QEMU inheriting the harness's
stderr, is stated where it is set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Japabu added a commit that referenced this pull request Sep 27, 2026
…ithout a slot

The same sentence as #555 places, so the two merge as one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu Japabu changed the title tests: every process the harness has to end dies with the harness tests: a pty tethers QEMU and the judge's servers to the harness Sep 27, 2026
Japabu added a commit that referenced this pull request Sep 28, 2026
…ows, and file the harness's misreport

quiesce_stops_the_machine was disabled behind a finding whose title and exit
rested on the harness's message that the guest asked for a reboot and stayed
up. PR #566's capture at 74f7d71 refutes that: writer 5's first pass ran
from 2.170 s to 7.434 s, the job printed "5 of 6 writers reached their loop
in 5s" at 6.874 s and exited 1, and it never printed "asking for the reset".
No stop began. PR #524's capture at 235c5a5 shows the same with "3 of 6",
and nightly run 36351950439 on PR #555 at d265676 with "4 of 6".

- The slow pass is the defect quiesce_dump_holds_the_stopped's issue already
  tracks, whose exit names a first write-and-fsync pass over 5 s. That issue
  is renamed to what both tests show, and gains these sightings. Both rows
  point at it.
- The stops finding is folded into it and deleted. It carried no durable line
  for a module header; its three sightings move with their evidence. Its
  a58abf5 sighting also had no stop: record, and whether that job printed
  its give-up line was not recorded.
- stopped_boot waits its whole QMP budget and then calls
  returned_to_firmware before it reads the console, so a job that never asked
  is reported as a guest that asked. In the #566 capture every scheduler
  heartbeat from 10.750 s to 253.244 s was idle, and the test went red after
  266 s. Filed as tooling, held by the orchestrator.
- The park issue is renamed: its records show 0 block operations open, so
  both threads were running, and one was the held thread, which
  last::hold keeps spinning while a sweep counts 2. It now names each
  sighting's PR and head, adds the 98e803c stop that gave up, labels the
  dispose_yield suspect as a hypothesis, and records that
  woken_by_its_threads has no enabled caller. Its exit asks for an
  instrument that names each thread still running, and for that coverage
  back.
- Deleted: the scratch log names and paths, "after a stop that reported
  every thread stopped", "on a loaded host", and the build/ park issue's
  "--known-red answers NO".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
@Japabu

Japabu commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator Author

Review, round 2, at d265676

Readiness. At d265676, PR CI host passes: run 36351594110 reports [ci] Host: 50 step(s), all green and tether::tests::a_tethered_child_dies_with_its_owner ... ok on macOS. The branch nightly, run 36351950439 on ubuntu-24.04, reports PASS guest_dies_with_its_harness in guest (2). The branch does not merge: see B4.

Growth. +372/−12 in total:

  • production +114/−7: src/tether.rs:1-101, lib.rs, the qemu.rs spawn site, https.rs;
  • test support +255/−3;
  • prose +3/−2.

The production growth is accepted.

Earlier BLOCKERs

  • B1 CLOSED at d265676. In run 36351950439, guest (2) logs [orphan] QEMU 31334 gone 1.55541ms after its harness's SIGKILL and PASS guest_dies_with_its_harness, so the TIOCGPTPEER path and the Linux hangup are measured end to end. Every guest shard and tcg spawn QEMU through tether::spawn, and all are green except:

    • guest (3): quiesce_stops_the_machine, which main disabled in cd2e630;
    • audio (2): see the NOTE.

    The host arm itself has still never run on Linux, because both host lanes are macOS. The guest arm covers the path. B1 must be measured again at the merged head (B4).

  • B2 CLOSED at d265676. The owner's $TMPDIR is now a TempDir (tests/common/orphan.rs:31), and guest (2) of run 36351950439 logs nothing left in $TMPDIR: every test took its scratch with it. Merging main reopens it as B4.

  • B3 CLOSED. src/tether.rs:163-181 waits with recv_timeout against a deadline and matches with split_once, and this_test passes --test-threads 1. Body measurements:

    • RUST_TEST_THREADS=1 gives EXIT=0; at 930396e the same run was still going at 60 s;
    • an owner that never prints the prefix exits 101 after 11 s.

BLOCKER

  • B4 — the branch does not merge with main, and main breaks the B2 fix.
    • git merge-tree --write-tree origin/main d2656765 conflicts in CLAUDE.md, src/sourcegate.rs, src/testargs.rs, tests/common/qemu.rs and tests/toyos.rs.
    • Main's left_behind (src/ci.rs:529-551) also reds on dead roots under /tmp.
    • On main, every boot takes TempDir::short("boot") (tests/common/qemu.rs:3147, :4805). So the SIGKILLed --hold owner leaves /tmp/toyos-tmp-<pid>-0 behind.
    • Nothing in the parent sweeps that root once the parent has taken its own first short root, because the sweep runs once per process. The shard then reds on left by a process that died during the steps above. Whether it does depends on test order in the 1-wide CI shards.
    • Fix: merge main, and have the test reclaim the owner's /tmp root after the verdict, the same way it reclaims its $TMPDIR root.
    • Measurement: a nightly on the merged head, where the shard that runs guest_dies_with_its_harness is green, including nothing left in $TMPDIR or /tmp.
  • B5 tests/common/qemu.rs:4912 — the guest arm's negative control has not been measured. The body gives only an expected result.
    • Why it might stay green: with the tether reverted, QEMU's stdin and stdout pipes still break when the owner dies. If QEMU ends on its own at a console write into the dead stdout within WITHIN (10 s), guest_dies_with_its_harness stays green and cannot tell the tether from its absence.
    • Patch: let (mut child, tether) = toyos_build::tether::spawn(qemu).expect(..) → let mut child = qemu.spawn().expect(..), and drop _tether. Keep pid().
    • The orchestrator runs this on the dev host at the merged head: cargo test --test toyos-build -- guest_dies_with_its_harness must exit 1 naming the pid, and ps -p <pid> must be empty afterwards.

NOTE

  • Merge B4 conflicts carefully:
    • sourcegate.rs: main's src/buildlock.rs count is now 1, not 2;
    • testargs.rs: main refuses --metal --nightly, so it must not return to the accepted list;
    • tests/toyos.rs: the host-slots block is deleted on main, and --hold goes after Run::begin() alone.
  • src/tether.rs:123-141 — io::pipe plus .stderr(write) plus drop(cmd) hand-rolls what Stdio::piped() and child.stderr.take() already do. With std closing the parent's write end, the verdict no longer depends on a dropped Command closing a descriptor, and two lines and a comment go.
  • src/tether.rs:198-203 — on failure, the test SIGKILLs pids by number after kill(pid, 0). Init has already reaped the orphaned child, so a live number can be a reused pid, and "a process of this test's own" is not checked.
  • src/tether.rs:108-215 — Owner is the third copy of the same pattern: re-run this test binary, read one prefixed line under a bound, SIGKILL it. The other two are toyos-tmpdir/tests/reclaim.rs's Holder and src/buildlock.rs:677. They are in different crates, so no merge now.
  • The macOS window before io::pipe gets close-on-exec (the body's "Unsure") can cause a false red in cargo test --lib, where sibling tests spawn processes. reclaim.rs serialises its own spawns (SPAWNING) for exactly this. Here it is recorded nowhere: file it in issues/ with an owner and an exit condition.
  • src/tether.rs, src/lib.rs:47 — this is a fourth unconditional Unix subsystem in toyos-build: portability-windows in run 36351950439 fails on src/tether.rs. Add it to issues/build/the-build-system-does-not-compile-on-windows.md, which names three.
  • Approach accepted:
    • A pty per child is the only measured candidate that ends both QEMU and the servers on both hosts.
    • A process group gets no signal when its parent dies (measured).
    • PR_SET_PDEATHSIG fires when the spawning thread exits, and guests are spawned from worker threads.
    • exit-with-parent is QEMU-only, so the servers would need a second path.
    • Nothing simpler covers both hosts without a helper process.
  • audio (2) in run 36351950439 (gate A, smp1 median 5765→6463, z=3.95) is not this branch's doing:
    • main's nightly 36400924827 passed with a fresh median of 6458;
    • audio (2) was red on main in runs 36290616312, 36278449733 and 36228604597;
    • in both runs the recorded sample shows 874 wakes against 1407 fresh, so the recorded sample itself is stale.
  • Main's .claude/agents/implementer.md:22 ("cargo test, never cargo run") contradicts main's CLAUDE.md:73. That is main's defect and needs its own brief, not this branch.

REMOVE

  • CLAUDE.md:72 — the whole hunk. Main already carries the rule, and efefec7 dropped the reporting clause as covered by "Leave the machine as you found it".
  • .claude/agents/implementer.md:22-23 — round 1 asked for this line to be deleted; it was rewritten. Now that main carries the rule, the change is outside this PR's fence.
  • tests/common/orphan.rs:29-30 — "what its SIGKILL leaves there goes when this does" is false on main because of the /tmp root (B4).
  • PR body, "Linux has not run." paragraph — false since run 36351950439.
  • PR body, "How the guest arm's bound on the owner is derived" — arithmetic over other modules' constants that goes stale with any ceiling change.
  • PR body, "Guest arm, run by the orchestrator" — an expected outcome is not a measurement.
  • PR body, "Unsure", the 511-ptys bullet — it states no consequence.

SEND BACK

Japabu and others added 2 commits September 29, 2026 00:54
Conflicts, each resolved to main where main overtook the branch:

- CLAUDE.md: main's text; the branch's hunk is dropped (main carries the
  rule). .claude/agents/implementer.md, which merged clean, is also main's:
  the branch's rewrite of that line was outside its fence.
- src/sourcegate.rs: current_exe's sites keep the branch's tether.rs and
  orphan.rs entries at main's buildlock.rs count of 1.
- src/testargs.rs: main's nightly/weekly-against-audio-gate/metal refusal
  stays, and `--metal --nightly` does not return to the accepted list;
  `--hold`'s refusal follows it.
- tests/common/qemu.rs: both imports, Tether and main's TempDir.
- tests/toyos.rs: main's deletion of the host-slots block; `--hold` goes
  after `Run::begin()` alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…killed through the owner's group

Review round 2 of #555.

B4: on main every boot takes `TempDir::short("boot")`, so the SIGKILLed
`--hold` owner leaves `/tmp/toyos-tmp-<pid>-<n>`, and the parent has
already made its one sweep of `/tmp`; `left_behind` in the guest job then
reds. `TempDir::adopt(base, pid)` moves every gone root of `pid` under
`base` into a directory of the caller's, under the base's lock, so it goes
when that does; `guest_dies_with_its_harness` adopts the owner's roots
into a `TempDir::short` after its verdict, on every path, the owner already
reaped. `an_adopted_root_goes_with_its_adopter` covers it: a live root and
a gone root of pid `<pid>0` stay. Controls, as checked patches that build:
`adopt` moving nothing reds with "was not adopted"; the prefix without its
trailing `-` reds with "another pid's root was adopted".

Owner: the stderr pipe is std's `Stdio::piped()`, taken from the child,
so the verdict no longer rests on a dropped `Command` closing a write end.
The owner runs in a process group of its own. On a failed wait it is not
yet reaped, so its pid, and the group it leads, can be no other process's;
the group is SIGKILLed rather than `pids` by number, whose owner's death
let init reap them and their numbers go free. The refusal then says
whether every holder of the stderr exited. Control, a checked patch that
builds: the unit test's owner spawning `parked` untethered exits 101 in
10.02 s naming the child's pid, "the owner's process group was killed,
and every holder of its stderr then exited", and no `parked` process
remains.

The orphan test's comment claiming its SIGKILL's leavings go with its
`$TMPDIR` is deleted: false once the owner has a `/tmp` root.

Filed: the macOS window before a pipe is close-on-exec can red the
tether tests; `src/tether.rs` as a fourth Unix-only subsystem in the
Windows issue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Review, round 3, at 6d82ff9

Readiness.

  • PR CI host at 6d82ff9: run 36496544598, conclusion success.
  • Branch nightly 36496779560 at 6d82ff9: every guest shard, audio shard, tcg, host, build, portability-linux and portability-macos concluded success. portability-windows failed, and it fails on main's nightly too (36400924827), tracked.
  • git merge-tree --write-tree origin/main 6d82ff9f is clean (tree 68b50996).

Growth. +473/−10 in total:

  • production +119/−7: src/tether.rs:1-101, lib.rs, the qemu.rs and https.rs sites;
  • test support +237/−3: Owner, orphan.rs, --hold, adopt, sourcegate;
  • tests +86;
  • issues +31.

Earlier BLOCKERs

  • B4 CLOSED.
    • The branch merges cleanly.
    • In nightly 36496779560, guest (2) checked out 6d82ff9 and logs [orphan] QEMU 31374 gone 2.532748ms after its harness's SIGKILL and PASS guest_dies_with_its_harness.
    • The same shard logs nothing left in $TMPDIR or /tmp: every test took its scratch with it and Guest("2/12"): 4 step(s), all green.
  • B5 CLOSED, on the dev host (macOS).
    • Green arm at 6d82ff9: EXIT=0, [orphan] QEMU 14834 gone 12.651708ms.
    • The same command under guest-untethered.patch: EXIT=1, with of its tethered children [18514], [18514] still answered; the owner's process group was killed, and every holder of its stderr then exited.
    • The patch puts the QEMU site back to the base's exact qemu.spawn() and drops the field. That is the whole mechanism on this arm's path: the https.rs hunk is not on it, and pid() is test support.
    • The control is measured on macOS only. Linux has the green arm and no red arm; see the NOTE.

BLOCKER

  • B6 toyos-tmpdir/src/lib.rs:118 — adopt's liveness filter can be deleted and an_adopted_root_goes_with_its_adopter stays green.
    • Why it passes: the test's live root belongs to a different pid, so the prefix filter alone keeps it out.
    • Why it matters: "A live root is left alone, whatever its name" is exactly what stops adopt from moving a live root out from under a process that reused the pid, in any worktree on the host. Doing that loses that process's data.
    • Patch: - for root in gone_under(base) { → + for root in fs::read_dir(base).unwrap().map(|e| e.unwrap().path()) {.
    • It must turn an_adopted_root_goes_with_its_adopter red. Suggested test change: add a root named toyos-tmp-<pid>-9 whose owner file the test holds locked, and assert that it stays.

NOTE

  • src/tether.rs:344-356, tests/common/orphan.rs:543 — main's CLAUDE.md:97 (No QEMU test measures time, and audio is judged on metal only #562, landed after this branch's merge base) says a QEMU test never measures how long something took. killed times the exit and the guest arm prints it. Return Result<(), String> and delete the Instant and both prints.
  • toyos-tmpdir/src/lib.rs:117-126 — adopt's loop repeats State::sweep's body: gone_under, then a rename to reap-<name>. One helper with a filter would serve both.
  • Guest arm on Linux: the red arm was never measured there. Nightly 36496779560 shows only the green arm. The two hosts run the same QEMU 11.1.1, and on macOS the control shows that a broken stdout does not end it within WITHIN, so this is accepted.
  • PR body: the B5 measurement is only in the orchestrator's run log. The body's guest control says only "the orchestrator runs it". Add the command, exit code and refusal line to main's record.

REMOVE

  • PR body, "The merged head's nightly has not run." — false since run 36496779560.
  • tests/common/orphan.rs:25 — "this process's one sweep of /tmp is behind it by this line, before the owner makes one". The order does not matter: adopt takes the root whichever way round it happens.

SEND BACK

B6: an_adopted_root_goes_with_its_adopter added a live root whose name
matches the adopted pid's prefix, so the test only passes because adopt's
liveness filter actually keeps it out; deleting that filter now turns it
red. adopt and State::sweep share the rename-into-reap-<name> step through
one reap_into helper, since writing both filters next to each other is
what made the gap visible.

NOTE: following main's #562, no QEMU test measures how long something
took. Owner::killed no longer times the exit; it returns Result<(), String>
and the verdict rests on the event alone. Deleted the two REMOVE'd lines
without rewriting them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Review, round 4, at 46fd916

Readiness. gh pr checks 555: host pass (run 36517133323) at 46fd916. gh pr view 555 --json mergeable: MERGEABLE. Guest arm at 46fd916, as the orchestrator measured it: guest_dies_with_its_harness EXIT=0, and EXIT=1 under guest-untethered.patch.

Growth. +498/−22 in total; round 4 is +47/−34. Production is src/tether.rs:1-101, the qemu.rs/https.rs sites, lib.rs and adopt. The rest is test support (Owner, orphan.rs, --hold, sourcegate), tests and two issue files.

Earlier BLOCKERs

  • B6 CLOSED. I copied toyos-tmpdir at 46fd916 into a scratch crate with TMPDIR pointed at a scratch base, then ran cargo test --test reclaim -- an_adopted_root_goes_with_its_adopter three times:

    • unmutated: EXIT=0;
    • with the patch below: EXIT=101, panicked at tests/reclaim.rs:354:5: a live root of a reused pid was adopted;
    • with the patch reversed (diff -r against the worktree clean): EXIT=0.

    The patch replaces adopt's reap_into(...) call with:

    for path in fs::read_dir(base).unwrap().map(|e| e.unwrap().path()) {
        let name = path.file_name().unwrap().to_string_lossy().into_owned();
        if name.starts_with(&prefix) {
            fs::rename(&path, self.path.join(format!("reap-{name}"))).unwrap();
        }
    }
    

Earlier NOTEs and REMOVEs

  • killed untimed: CLOSED. It returns Result<(), String>, and neither arm prints a duration.
  • adopt/sweep duplication: CLOSED. Both go through reap_into, and sweep's behaviour is unchanged.
  • The B5 measurement is in the PR body: CLOSED.
  • Both REMOVEs: CLOSED.
  • The Linux red arm of the guest control: still accepted, unchanged.

BLOCKER

None.

NOTE

  • toyos-tmpdir/src/lib.rs:313 — the parameter keep selects the roots that get reaped, the opposite of what its name says. Name it for what it takes.
  • PR body, Negative controls, B6 — it describes replacing gone_under(base) "in adopt", but at 46fd916 adopt has no gone_under: it lives in the shared reap_into. The same edit there also strips sweep's liveness filter, and would move the live roots of the host's real $TMPDIR. Record the head-level patch above instead.

REMOVE

  • toyos-tmpdir/src/lib.rs:311-312 — "Both adopt and sweep are this with a different filter and destination." It lists the callers and rots with the next one.
  • PR body — "since putting both filters side by side is what made the gap visible", the "(round-3 B6)" tags, and "Owner::killed timed the exit and both the host and guest arms printed the duration". This is review chronology in main's record.

LAND AFTER NAMED CHANGES

…ts caller list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu
Japabu enabled auto-merge September 29, 2026 06:34
@Japabu
Japabu added this pull request to the merge queue Sep 29, 2026
Merged via the queue into main with commit 65b5215 Sep 29, 2026
1 check passed
@Japabu
Japabu deleted the wt/toyos-reap branch September 29, 2026 07:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant