tests: a pty tethers QEMU and the judge's servers to the harness - #555
Conversation
A QEMU the harness started ran for hours after its harness was gone: the harness stops QEMU only in `Drop`, and a SIGKILLed harness runs none, so its guest was reparented to init and ran on. The HTTPS judge's servers had the same shape. `toyos_build::tether::spawn` makes the child the controlling process of a pseudo-terminal whose master only the harness holds. The kernel closes the master when the harness dies by any signal, and the hangup sends the child SIGHUP, on macOS and Linux alike. QEMU and the judge's servers are spawned through it; `Drop`'s kill stays, so a clean end never waits on the hangup. Measured on this host (QEMU 11.1.1, a held `-S` guest, the holder SIGKILLed): a stdin pipe's EOF leaves QEMU running (alive 5 s later), a process group of its own leaves it running, `-run-with exit-with-parent=on` ends it in 3.7 ms, the pty ends it in 3.1 ms. The slave has to stay open in the child (closed at exec, QEMU survived), and SIGHUP has to be unblocked (inherited blocked, QEMU survived) and reset to default (inherited ignored, a child with no handler survived). `exit-with-parent` was rejected: on Linux it is PR_SET_PDEATHSIG, which follows the spawning thread, and it ends QEMU alone. `a_tethered_child_dies_with_its_owner` is the host arm; the harness's `guest_dies_with_its_harness` is the guest arm, whose owner is the harness itself re-run with `--hold`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`Child::wait` closes the child's stdin, and the tethered child the host arm parks reads that pipe to its end: with the mechanism reverted the child still exited 4.6 ms after its owner's SIGKILL, on the pipe's EOF rather than on a hangup, and the arm stayed green. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The owner's rules: only the orchestrator runs guest tests, and an agent cleans up every process it started before it reports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review, round 1, head aba9a95Readiness. CI Growth. +328/−11 (
BLOCKER
NOTE
REMOVE
Guest runs required (orchestrator), at the fixed head
SEND BACK |
…arms to the orchestrator Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… with it `Owner::said` read the owner's stdout with `read_line`, unbounded, and matched with `strip_prefix`. With one libtest thread the pretty formatter starts the owner's line with `test tether::tests::owner ... `, so the prefix was never found and the host arm hung with no verdict: `RUST_TEST_THREADS=1` at 930396e was still running after 60 s. The owner's stdout is now read by a thread into a channel, `said` takes its bound and waits with `recv_timeout`, and the prefix is matched with `split_once` wherever on the line it starts. The owner is run with `--test-threads 1` explicitly. An `Owner` dropped is killed and reaped, so an owner a failed wait left running goes with the test. The guest arm's owner took a `toyos_tmpdir` root in the inherited `$TMPDIR`, and its SIGKILL left it there for `ci guest`'s "nothing left in $TMPDIR" step. The owner's `$TMPDIR` is now a `TempDir` the test holds and drops after the verdict. The test also builds the image there and hands it to the owner (`--hold <image>`, booted `Staged::Pristine`), so the owner builds nothing: its one wait is its boot, which its own `wait_for_ready` ceiling ends at 10 s x its width floor of 2 x a fresh process's host scale of 1 x the default two vCPUs' oversubscription of at most 2, 40 s, inside `qemu::GUEST_WEDGED`, which bounds `said`. `WITHIN` is private. `spawn`'s contract names what the child owes; the module's opening overclaim and `pid`'s narrating doc are gone; the invariant the guest arm's green rests on, QEMU inheriting the harness's stderr, is stated where it is set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ithout a slot The same sentence as #555 places, so the two merge as one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ows, and file the harness's misreport quiesce_stops_the_machine was disabled behind a finding whose title and exit rested on the harness's message that the guest asked for a reboot and stayed up. PR #566's capture at 74f7d71 refutes that: writer 5's first pass ran from 2.170 s to 7.434 s, the job printed "5 of 6 writers reached their loop in 5s" at 6.874 s and exited 1, and it never printed "asking for the reset". No stop began. PR #524's capture at 235c5a5 shows the same with "3 of 6", and nightly run 36351950439 on PR #555 at d265676 with "4 of 6". - The slow pass is the defect quiesce_dump_holds_the_stopped's issue already tracks, whose exit names a first write-and-fsync pass over 5 s. That issue is renamed to what both tests show, and gains these sightings. Both rows point at it. - The stops finding is folded into it and deleted. It carried no durable line for a module header; its three sightings move with their evidence. Its a58abf5 sighting also had no stop: record, and whether that job printed its give-up line was not recorded. - stopped_boot waits its whole QMP budget and then calls returned_to_firmware before it reads the console, so a job that never asked is reported as a guest that asked. In the #566 capture every scheduler heartbeat from 10.750 s to 253.244 s was idle, and the test went red after 266 s. Filed as tooling, held by the orchestrator. - The park issue is renamed: its records show 0 block operations open, so both threads were running, and one was the held thread, which last::hold keeps spinning while a sweep counts 2. It now names each sighting's PR and head, adds the 98e803c stop that gave up, labels the dispose_yield suspect as a hypothesis, and records that woken_by_its_threads has no enabled caller. Its exit asks for an instrument that names each thread still running, and for that coverage back. - Deleted: the scratch log names and paths, "after a stop that reported every thread stopped", "on a loaded host", and the build/ park issue's "--known-red answers NO". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
Review, round 2, at d265676Readiness. At d265676, PR CI Growth. +372/−12 in total:
The production growth is accepted. Earlier BLOCKERs
BLOCKER
NOTE
REMOVE
SEND BACK |
Conflicts, each resolved to main where main overtook the branch: - CLAUDE.md: main's text; the branch's hunk is dropped (main carries the rule). .claude/agents/implementer.md, which merged clean, is also main's: the branch's rewrite of that line was outside its fence. - src/sourcegate.rs: current_exe's sites keep the branch's tether.rs and orphan.rs entries at main's buildlock.rs count of 1. - src/testargs.rs: main's nightly/weekly-against-audio-gate/metal refusal stays, and `--metal --nightly` does not return to the accepted list; `--hold`'s refusal follows it. - tests/common/qemu.rs: both imports, Tether and main's TempDir. - tests/toyos.rs: main's deletion of the host-slots block; `--hold` goes after `Run::begin()` alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…killed through the owner's group Review round 2 of #555. B4: on main every boot takes `TempDir::short("boot")`, so the SIGKILLed `--hold` owner leaves `/tmp/toyos-tmp-<pid>-<n>`, and the parent has already made its one sweep of `/tmp`; `left_behind` in the guest job then reds. `TempDir::adopt(base, pid)` moves every gone root of `pid` under `base` into a directory of the caller's, under the base's lock, so it goes when that does; `guest_dies_with_its_harness` adopts the owner's roots into a `TempDir::short` after its verdict, on every path, the owner already reaped. `an_adopted_root_goes_with_its_adopter` covers it: a live root and a gone root of pid `<pid>0` stay. Controls, as checked patches that build: `adopt` moving nothing reds with "was not adopted"; the prefix without its trailing `-` reds with "another pid's root was adopted". Owner: the stderr pipe is std's `Stdio::piped()`, taken from the child, so the verdict no longer rests on a dropped `Command` closing a write end. The owner runs in a process group of its own. On a failed wait it is not yet reaped, so its pid, and the group it leads, can be no other process's; the group is SIGKILLed rather than `pids` by number, whose owner's death let init reap them and their numbers go free. The refusal then says whether every holder of the stderr exited. Control, a checked patch that builds: the unit test's owner spawning `parked` untethered exits 101 in 10.02 s naming the child's pid, "the owner's process group was killed, and every holder of its stderr then exited", and no `parked` process remains. The orphan test's comment claiming its SIGKILL's leavings go with its `$TMPDIR` is deleted: false once the owner has a `/tmp` root. Filed: the macOS window before a pipe is close-on-exec can red the tether tests; `src/tether.rs` as a fourth Unix-only subsystem in the Windows issue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review, round 3, at 6d82ff9Readiness.
Growth. +473/−10 in total:
Earlier BLOCKERs
BLOCKER
NOTE
REMOVE
SEND BACK |
B6: an_adopted_root_goes_with_its_adopter added a live root whose name matches the adopted pid's prefix, so the test only passes because adopt's liveness filter actually keeps it out; deleting that filter now turns it red. adopt and State::sweep share the rename-into-reap-<name> step through one reap_into helper, since writing both filters next to each other is what made the gap visible. NOTE: following main's #562, no QEMU test measures how long something took. Owner::killed no longer times the exit; it returns Result<(), String> and the verdict rests on the event alone. Deleted the two REMOVE'd lines without rewriting them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review, round 4, at 46fd916Readiness. Growth. +498/−22 in total; round 4 is +47/−34. Production is Earlier BLOCKERs
Earlier NOTEs and REMOVEs
BLOCKERNone. NOTE
REMOVE
LAND AFTER NAMED CHANGES |
…ts caller list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A QEMU the harness started ran for hours after its harness was gone. The harness stops QEMU only in
Drop, a SIGKILLed harness runs none, and its guest was reparented to init. The HTTPS judge's servers had the same shape.What changed, per decision
One construction:
toyos_build::tether::spawn(src/tether.rs). The child becomes the controlling process of a pseudo-terminal whose master only the harness holds. The kernel closes the master when the harness dies by any signal, and the hangup sends the childSIGHUP. The master is a field of whatever owns the child (QemuInstance, the judge'sServer). No new binary, no per-test bookkeeping.What the child owes. The tether holds only for a child that exits on
SIGHUP, keeps the descriptor it inherited open, and starts no session of its own;spawn's contract says so. QEMU exits onSIGHUP(measured below). The judge's servers install no handler, so the default disposition ends them. Neither child forks.Candidates, measured on the macOS dev host (QEMU 11.1.1; a held
-Sguest spawned by a holder process; the holder SIGKILLed; the QEMU pid polled for 5 s;psbefore and after):-run-with exit-with-parent=onexit-with-parentis rejected. It ends QEMU alone, so the judge's servers would need a second mechanism. On Linux it isPR_SET_PDEATHSIG, which by prctl(2) follows the spawning thread rather than the process, so the two hosts would disagree about what a guest is tied to. The pty also ended a non-QEMU child (/bin/sleep) in 2.9 ms.What
pre_execdoes, each piece measured.SIGHUPis unblocked and reset to default in the child. With it inherited blocked, QEMU survived its holder; with it inherited ignored, a child with no handler of its own (/bin/sleep) survived. The slave stays open acrossexec: closed at exec, QEMU survived on macOS. No hangup can precedeTIOCSCTTY, because the child holds its own copy of the master untilexeccloses it.Scope. Tethered: the processes that end only when the harness ends them, QEMU and the judge's servers. Builds and one-shot clients are not: a build killed mid-way can leave a toolchain half-written.
Drop's kill stays. SIGKILL cannot be caught, andDropmust reap QEMU before itsNvmeClaimis released. A hangup ends QEMU too (dropping the master alone ended a held QEMU in 1.1–2.0 ms, exit 0), but it depends on QEMU's handler and printsterminating on signal 1on the suite's stderr for every guest.The owner's
/tmproot goes with the test (TempDir::adopt). On main every boot takesTempDir::short("boot"), so the SIGKILLed--holdowner leaves/tmp/toyos-tmp-<pid>-<n>, and the test process has already made its one sweep of/tmp;ci guest'sleft_behindwould red on it.toyos_tmpdir::TempDir::adopt(base, pid)moves every gone root ofpidunderbaseinto the caller's directory, under the base's lock, so it goes when that does. A live root is never moved, and liveness stays the owner lock, never the pid.adopt's liveness filter is proven, not assumed.an_adopted_root_goes_with_its_adopternow also makes a root named for the adopted pid —toyos-tmp-<pid>-9— whose owner file the test itself holds locked, standing in for a live process that reused that pid, and asserts it survivesadopt. Before this the test's only live root belonged to a different pid, so the prefix match alone kept it out and the liveness filter (gone_under, insideadopt) could be deleted without turning anything red.adoptandState::sweepnow share that rename-into-reap-<name>step through onereap_into(base, dest, take)helper.No QEMU test measures time (main's
CLAUDE.md:97, #562).killednow returnsResult<(), String>; the verdict rests on the event (the owner's stderr pipe reaching EOF) alone, and neither arm's print names how long that took.The tests
Host arm:
tether::tests::a_tethered_child_dies_with_its_owner(cargo test --lib). An owner process spawns a tethered child and is SIGKILLed. The owner inheritsSIGHUPblocked and ignored, so the reset is exercised. The verdict is the owner's stderr pipe (std'sStdio::piped(), taken from the child) reaching EOF, which happens once every holder has exited, bounded byWITHIN(10 s). The owner runs in a process group of its own, which a tethered child leaves bysetsidand an untethered one stays in. On a timeout the owner is not yet reaped, so its pid, and the group it leads, can be no other process's: the group is SIGKILLed, rather than the children's pids by number (their numbers went free once init reaped them). The refusal names the pids that still answer signal 0 and says whether every holder of the stderr then exited, which is how the test checks that nothing it started still runs. The owner runs with--test-threads 1.Owner::saidwaits withrecv_timeoutand matches its prefix withsplit_once, so a line libtest has already begun still matches. AnOwnerdropped is killed and reaped.Guest arm:
guest_dies_with_its_harness(Fast tier). The same judge, with the harness binary itself as the owner. The test builds the image into atoyos_tmpdir::TempDirit holds and runs the harness with--hold <image>andTMPDIRset to that directory. The owner boots the imageStaged::Pristineand holds it until stdin ends.--holdis refused beside any other word. After the verdict, on every path and with the owner reaped, the test adopts the owner's/tmproots into aTempDir::shortof its own; its$TMPDIRscratch goes with theTempDirit set as the owner'sTMPDIR. The green verdict rests on QEMU inheriting the harness's stderr;spawn_and_wait_readysays so where it sets that.toyos-tmpdir:an_adopted_root_goes_with_its_adopter. A holder is SIGKILLed beside a live one, a gone root named for pid<pid>0, and a live root namedtoyos-tmp-<pid>-9whose owner the test holds locked.adoptmoves the killed root, with its directory and image, into the adopter; the other three stay, and the adopter's drop removes what it took.The judge's servers. Only the QEMU site has an arm of its own; reverting the
https.rshunk alone leaves every test green. The site is kept because the servers are a process the harness has to end. They install noSIGHUPhandler, which is the case the host arm'sparkedchild exercises: a child with no handler, ended by the default disposition. That claim rests on the host arm.Gates (head 6d82ff9, macOS)
cargo test -p toyos-build --lib: EXIT=0, 420 passed, 5 ignored.cargo test --workspace --exclude toyos-build: EXIT=0.cargo test --test toyos-build -- --list: EXIT=0; listsFast guest_dies_with_its_harness.cargo run -- --clippy: EXIT=0, 10 invocations clean.cargo test -p toyos-tmpdir: EXIT=0, 8 passed inreclaim.Linux: run 36351950439 (at d265676) logged
[orphan] QEMU 31334 gone 1.55541ms after its harness's SIGKILLandPASS guest_dies_with_its_harnessin guest (2). At 6d82ff9, branch nightly 36496779560 ran every guest shard, audio shard,tcg,host,build,portability-linuxandportability-macosgreen; guest (2) logged[orphan] QEMU 31374 gone 2.532748ms after its harness's SIGKILL,PASS guest_dies_with_its_harnessandnothing left in $TMPDIR or /tmp: every test took its scratch with it.portability-windowsfailed there as it does on main's own nightly (36400924827), tracked below.Round-4 gates (head 46fd916, macOS)
cargo run -- --ci host: EXIT=0,[ci] Host: 54 step(s), all green.cargo run -- --build-only: EXIT=0,Build finished.guest-untethered.patchregenerated unchanged atreap-r4/guest-untethered.patch(round-3's content still applies cleanly):cargo test --test toyos-build --no-runon the mutated tree, EXIT=0; reverted, tree clean.guest_dies_with_its_harnessEXIT=0; underguest-untethered.patch, EXIT=1.Negative controls
Each control is a checked patch, shown to build, run, and restored in the same script, with the tree clean after it.
parkeduntethered (parked.spawn()forspawn(parked)), at 6d82ff9's content: EXIT=101 after 10.02 s,of its tethered children [35995], [35995] still answered; the owner's process group was killed, and every holder of its stderr then exited. Noparkedprocess remained (ps).spawnreturning a plaincmd.spawn(); the slave left close-on-exec;SIGHUPnot unblocked;SIGHUPnot reset to default; noTIOCSCTTY. An owner that never says its pid: EXIT=101 withthe owner had not said "tethered " within 10s, after 11 s.adoptmoving nothing: EXIT=101,… was not adopted. The pid prefix without its trailing-: EXIT=101,another pid's root was adopted.reap_into's liveness filter, where it now lives, shared byadoptandsweep: replacing itsgone_under(base)call with a plainfs::read_dir(base)turnsan_adopted_root_goes_with_its_adopterred — EXIT=101,a live root of a reused pid was adopted— restored, it is green.tests/common/qemu.rs'stether::spawnreverted toqemu.spawn(), the field dropped,pid()kept), measured by the orchestrator on the dev host (macOS) at 6d82ff9: green arm EXIT=0,[orphan] QEMU 14834 gone 12.651708ms; the same command under the patch, EXIT=1,of its tethered children [18514], [18514] still answered; the owner's process group was killed, and every holder of its stderr then exited. Measured on macOS only: the red arm was never measured on Linux (see Unsure); the two hosts run the same QEMU 11.1.1, and the macOS control already shows a broken stdout does not end QEMU withinWITHIN.Independent oracles
psshows each survivor with PPID 1 and each tethered child absent. During the host whole-revert arm at d265676,ps -axo pid,ppid,stat,etime,commandtaken 3 s into the wait shows the parked child alive with PPID 1 and its owner gone.ci guest's own scratch check.src/ci.rs'sleft_behindjudges the merged head's nightly shard that runs the guest arm, including/tmp.Unsure
issues/build/a-pipe-made-on-macos-can-leak-into-a-sibling-spawn-and-red-the-tether-tests.md.src/tether.rsdoes not compile on Windows; recorded inissues/build/the-build-system-does-not-compile-on-windows.md.🤖 Generated with Claude Code