Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ toyos-swap = { path = "toyos-swap" }
# The watchdog's parameter name, so the gate over `kernel/src/params.rs` reads
# the same constant the kernel and the bootloader do.
toyos-tco = { path = "toyos-tco" }
# The one scratch directory under `$TMPDIR`: the harness's run takes one, and so does
# The one scratch directory, under `$TMPDIR` or, for a socket, `/tmp`: the harness's run takes one, and so does
# every test here; `--ci host` holds the host tests to leaving nothing behind.
toyos-tmpdir = { path = "toyos-tmpdir" }
toyos-blackbox = { path = "toyos-blackbox" }
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
status: expected-red
kind: tooling
opened: 2026-09-27
---

# `partition_claim_departure` exits 0 having said none of its refusals

Seen on 2026-09-27 in the nightly run 36336701867, job `guest (5)`, on PR
#541's head `1c0f0c753fb2c4c2d01caf51dc67b7f92ca4cd8e` — a diff that touches
none of `tests/common/partclaim.rs`, `tests/toyos-rust-tests/src/bin/partition_claimant.rs`
or the kernel's partition-claim code:

```
FAIL partition_claim_departure: departure: the guest exited 0 having said 0 of its 1 refusals:

FAIL partition_claim_departure (2s)
```

Re-run alone, immediately after, in the same session: green, with every role's
own line printed —

```
[partclaim] departure: 1 told; [kernel 0.757 cpu0] usb-quiesce: disk 0 SYNCHRONIZE CACHE ok
[partclaim] silent: 1 told; [kernel 0.710 cpu0] usb-quiesce: disk 0 SYNCHRONIZE CACHE ok
[partclaim] untold: 0 told; ...
PASS partition_claim_departure (7s)
ALONE partition_claim_departure: GREEN, and it was alone both times — nothing
the harness controls differed, so it failed once and passed once. That is a
rate and not a classification.
```

`cargo run -- --known-red partition_claim_departure` answered NO before this
row.

## What is known

The failure is `guest_verdict`'s (`tests/common/partclaim.rs`), on the
`departure` role — the first of the three `departed()` iterations in
`partition_claim_departure`. Its message, `"the guest exited 0 having said
{said} of its {refusals} refusals:\n{stdout}"`, prints with an empty tail: the
guest's own captured `stdout` held nothing at all — not one of the `departure`
role's own `println!`s (`"a write is reported and not flushed"`, `"the write
the device left under completed"`, the `refused with` lines `said()` prints,
or `"partition_claimant: PASS"`), yet the guest's exit code was 0.

An exit of 0 rules out a panic on a wrong assertion (`departure()`'s
`assert_eq!`s all `panic!` on mismatch, and a panic does not exit 0), so the
guest's own logic is not shown to have run into an unexpected state.

This is the same family flagged in
`issues/boot-media/partition-claim-gives-up-reds-beside-other-guests-and-is-green-alone.md`
— same test area, and that file already widens its scope to
`partition_claim_departure`. It is filed separately rather than folded in because the failure signature
differs: that file's evidence is a kernel-log line count coming up short
(`"{count} flushes were told of the loss, not {told}"`, matched against
lines the kernel actually printed) on the `silent` role, attributed to a
hypothesised unscoped global fsync deadman race; this is a `guest_verdict`
failure on the `departure` role with the guest's entire stdout capture
missing, which that hypothesis does not by itself explain.

## Exit condition

`guest_verdict`'s "exited 0 having said" refusal (`tests/common/partclaim.rs`)
carries the kernel window, as its non-zero-exit refusal already does, so the
next sighting is not blind; the mechanism named and fixed; and a test that
turns red on it deterministically. Then this row and its `src/redlist.rs`
entry are deleted.

## Owner

The partition-claim code, held by the orchestrator.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
status: open
kind: tooling
opened: 2026-09-28
---

# The `/tmp` leftover check can name another worktree's killed run

`src/ci.rs`'s `left_behind` names every root `toyos_tmpdir::gone_roots(short)`
finds that `before` does not: a `toyos_tmpdir::TempDir::short` whose owning
process died during the steps that ran between the two calls. `gone_roots`
reads `SHORT_BASE` (`/tmp`), and every worktree on the host shares that base
and its `GLOBAL` lock — the reclaim is deliberately cross-worktree
(`toyos-tmpdir/src/lib.rs`'s module header: "Every process that shares a
base — every worktree on the host — shares the lock file"). Nothing in
`gone_roots` or in `left_behind`'s call sites records which pids belong to
*this* job's own steps, so a root left by another worktree's harness,
SIGKILLed on the same host in the same window, is named as this job's own
leftover.

**Evidence:** `toyos-tmpdir`'s own tests demonstrate the mechanism the check
relies on — a SIGKILLed holder's root is reclaimed by the next process to make
a directory in the same base, whichever process that is
(`toyos-tmpdir/tests/reclaim.rs`'s `a_killed_process_is_reclaimed_and_a_live_one_is_never_touched`).
`left_behind` has no notion of a job or a worktree; it is a diff of
`gone_roots(short)` against a snapshot taken before the steps ran, over a base
every process on the host writes into. On the hosted runners the gate uses,
each job has its own host, so the check is exact there; a developer running
`cargo run -- --ci host` on a machine also running another worktree's harness
can see a red that is not this job's.

**Exit condition:** `left_behind` names only roots this job's own processes
created, or the check is scoped to a base this job does not share with another
worktree's harness; a test demonstrates the narrowed check passing a killed
run made outside the job's own process tree.
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
---
status: expected-red
kind: tooling
opened: 2026-09-27
---

# `i8042_mouse` ends four packets short with a clean exit; mechanism not known

Seen on 2026-09-27 in the orchestrator's Fast-tier run on PR #537's head
`06b926b1`, a diff that touches neither the i8042 driver nor this test:

```
FAIL i8042_mouse: 872 pointer events reached userland out of 876 packets injected, never more than 4 of them (12 bytes) outstanding against a 16-byte device queue
FAIL i8042_mouse (17s)
```

`cargo run -- --known-red i8042_mouse` answered NO. Earlier sightings of this
message are in `issues/build/parallel-tests-red-under-other-suites.md`'s
`i8042_mouse` entry.

## What is known

- **The run ended cleanly.** This message is reached only when
`run_test_paced` returned no error, so the test runner printed
`===TEST_END test_rs_i8042_mouse` with a tail other than `error=`: `exit=<n>`,
none, or one it cannot parse. A stall, a ceiling or a runner error ends in
the `STALLED` message instead.
- **The guest stopped reading mid-burst.** 876 injected is the four lead-in
packets plus 872 of `BURST`'s 1000. The shortfall, 4, equals `MOUSE_LEAD`:
the host always refills to `arrived + MOUSE_LEAD`, so any guest that stops
reading mid-burst leaves exactly that many unread.
- **Not the guest's `RUN_CEILING`.** The test took 17 s, and the ceiling is
60 s from `===I8042_MOUSE_READY===`.
- **The capture that would tell is not kept.** The message carries neither
the guest's stdout, the kernel's serial window, nor the exit code, and
`i8042_mouse` never reads `exit_code` before this count. So the log's missing
`mev done` line says nothing. The boot is `Profile::Metal`, whose 16550 is
the console on stdio, so it writes no `uart-*.log`. No `Boot parameter:` line
in the run's kept serial logs carries `i8042-trace`.

Two paths in the tree end the guest this way, and nothing captured tells
them apart:

- **The guest's own end rule, met by a misframed packet.**
`tests/toyos-rust-tests/src/bin/i8042_mouse.rs` exits 0 on the first event
without the right button after one with it. `toyos_ps2::mouse`'s decoder
resets to a head on any gap between bytes longer than `PACKET_GAP_NS` (5 ms).
Measured on the host by feeding the burst's bytes to `MouseDecoder`: one such
gap between a −1 packet's head `0x18` and its `dx` `0xFF` takes `0xFF` as a
head. The decoder emits `buttons=0x07`, discards the next packet's `0x01 0x00`,
and then emits the following −1 with `buttons=0x00`. That is a press and
release of the right button. Whether a gap that long in guest time lands
inside a packet on this host is not measured.
- **A non-zero exit**, which this test does not read.

## Exit condition

The shortfall refusal (`i8042_mouse` in `tests/toyos.rs`) carries
`result.stdout` and `result.exit_code`, so the next sighting is not blind; the
mechanism is named, and a deterministic test is red on it. Then this file and
its `src/redlist.rs` row are deleted.

## Owner

The i8042/input path, held by the orchestrator.
69 changes: 53 additions & 16 deletions src/ci.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
//! open; `cargo run` only notes one, because a build must not stop for brew.

use std::io::{BufRead, BufReader, Write};
use std::path::Path;
use std::path::{Path, PathBuf};
use std::process::Command;

use crate::arch::Arch;
Expand Down Expand Up @@ -427,6 +427,8 @@ fn run_control(root: &Path, control: &Control) -> Result<String, String> {
/// host triple for the same reason.
fn host(root: &Path) -> Vec<Step> {
let tmp = toyos_tmpdir::TempDir::new("ci-host");
let short = Path::new(toyos_tmpdir::SHORT_BASE);
let before = toyos_tmpdir::gone_roots(short);
// Before any thread: nothing in this process reads the environment
// concurrently with the write, and every child inherits it.
std::env::set_var("TMPDIR", tmp.path());
Expand Down Expand Up @@ -491,18 +493,19 @@ fn host(root: &Path) -> Vec<Step> {
steps.push(step("the toyos SDK", || {
cargo(root, &["test", "--manifest-path", "toyos/Cargo.toml", "--target", &host_triple])
}));
steps.push(step("nothing left in $TMPDIR", || left_behind(&tmp)));
steps.push(step("nothing left in $TMPDIR or /tmp", || left_behind(&tmp, short, &before)));
steps
}

/// What `tmp` holds but the lock `toyos_tmpdir` keeps in it, as a refusal.
/// What `tmp` holds but the lock `toyos_tmpdir` keeps in it, and every root
/// under `short` whose process is gone that `before` does not name.
///
/// Refuses if `tmp` holds no [`toyos_tmpdir::GLOBAL`] at all: every step above
/// makes at least one `toyos_tmpdir::TempDir`, which always writes that lock
/// file first, so its absence means this `$TMPDIR` never saw the steps at
/// all — the guard reading an empty directory it was never given, rather than
/// one every test actually cleaned.
fn left_behind(tmp: &Path) -> Result<String, String> {
fn left_behind(tmp: &Path, short: &Path, before: &[PathBuf]) -> Result<String, String> {
let mut left: Vec<String> = std::fs::read_dir(tmp)
.map_err(|e| format!("read {}: {e}", tmp.display()))?
.map(|e| e.map(|e| e.file_name().to_string_lossy().into_owned()))
Expand All @@ -518,15 +521,28 @@ fn left_behind(tmp: &Path) -> Result<String, String> {
}
left.retain(|name| name != toyos_tmpdir::GLOBAL);
left.sort();
if left.is_empty() {
let mut dead: Vec<String> = toyos_tmpdir::gone_roots(short)
.into_iter()
.filter(|root| !before.contains(root))
.map(|root| root.display().to_string())
.collect();
dead.sort();
let mut said = Vec::new();
if !left.is_empty() {
said.push(format!(
"left in {} by the steps above, each written past a `toyos_tmpdir::TempDir` or held \
past its test: {}",
tmp.display(),
left.join(", ")
));
}
if !dead.is_empty() {
said.push(format!("left by a process that died during the steps above: {}", dead.join(", ")));
}
if said.is_empty() {
return Ok("every test took its scratch with it".into());
}
Err(format!(
"left in {} by the steps above, each written past a `toyos_tmpdir::TempDir` or held past \
its test: {}",
tmp.display(),
left.join(", ")
))
Err(said.join("; "))
}

/// What protects `main` is configured outside the repository, so it is read
Expand Down Expand Up @@ -610,6 +626,8 @@ fn suite_args(args: &[&str]) -> Vec<String> {
/// every boot image — survives past the last step, which reds on it.
fn guest(root: &Path, suite: &[String]) -> Vec<Step> {
let tmp = toyos_tmpdir::TempDir::new("ci-guest");
let short = Path::new(toyos_tmpdir::SHORT_BASE);
let before = toyos_tmpdir::gone_roots(short);
// Before any thread, same as `host`: every child this process spawns below
// inherits this, and nothing here reads the environment concurrently with
// the write.
Expand All @@ -634,7 +652,7 @@ fn guest(root: &Path, suite: &[String]) -> Vec<Step> {
}
// Unconditional: whatever stopped earlier, this $TMPDIR is still this
// process's own to judge, and a leak past a failing suite is still a leak.
steps.push(step("nothing left in $TMPDIR", || left_behind(&tmp)));
steps.push(step("nothing left in $TMPDIR or /tmp", || left_behind(&tmp, short, &before)));
steps
}

Expand Down Expand Up @@ -861,7 +879,6 @@ fn at_tip(ls_remote: &str, head: &str) -> Result<(), String> {
#[cfg(test)]
mod tests {
use super::*;
use std::path::PathBuf;

/// A deterministic control on `host`'s `std::env::set_var("TMPDIR", ...)`:
/// delete that line and every child writes to the real `$TMPDIR` instead of
Expand All @@ -870,21 +887,41 @@ mod tests {
#[test]
fn left_behind_refuses_a_tmpdir_that_never_saw_the_lock() {
let tmp = toyos_tmpdir::TempDir::new("left-behind-blind");
let refusal = left_behind(&tmp).expect_err("an untouched $TMPDIR is a red, not a pass");
let short = toyos_tmpdir::TempDir::new("left-behind-blind-short");
let refusal =
left_behind(&tmp, &short, &[]).expect_err("an untouched $TMPDIR is a red, not a pass");
assert!(refusal.contains(toyos_tmpdir::GLOBAL), "{refusal}");
}

/// The lock is the one thing a `$TMPDIR` keeps; anything else is named.
#[test]
fn the_host_job_names_what_its_tests_left_behind() {
let tmp = toyos_tmpdir::TempDir::new("left-behind");
let short = toyos_tmpdir::TempDir::new("left-behind-short");
std::fs::write(tmp.join(toyos_tmpdir::GLOBAL), b"").unwrap();
assert!(left_behind(&tmp).is_ok());
assert!(left_behind(&tmp, &short, &[]).is_ok());
std::fs::create_dir(tmp.join("forkcheck-1-current")).unwrap();
let refusal = left_behind(&tmp).expect_err("a directory left behind is a red");
let refusal = left_behind(&tmp, &short, &[]).expect_err("a directory left behind is a red");
assert!(refusal.contains("forkcheck-1-current"), "{refusal}");
}

/// A short root whose process died while the steps ran is named; one already
/// dead before them is not the steps'.
#[test]
fn the_job_names_a_short_root_a_step_left_when_it_died() {
let tmp = toyos_tmpdir::TempDir::new("left-behind-died");
let short = toyos_tmpdir::TempDir::new("left-behind-died-short");
std::fs::write(tmp.join(toyos_tmpdir::GLOBAL), b"").unwrap();
let earlier = format!("{}1-0", toyos_tmpdir::ROOT_PREFIX);
std::fs::create_dir(short.join(&earlier)).unwrap();
let before = toyos_tmpdir::gone_roots(&short);
assert!(left_behind(&tmp, &short, &before).is_ok());
let died = format!("{}2-0", toyos_tmpdir::ROOT_PREFIX);
std::fs::create_dir(short.join(&died)).unwrap();
let refusal = left_behind(&tmp, &short, &before).expect_err("a dead step's root is a red");
assert!(refusal.contains(&died) && !refusal.contains(&earlier), "{refusal}");
}

fn repo_root() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR"))
}
Expand Down
7 changes: 3 additions & 4 deletions src/qemu.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,9 @@
//!
//! # QMP, and the machine that has already stopped
//!
//! The socket is `/tmp/toyos-qmp.sock`. A harness test booted with
//! `BootOptions { qmp: true }` leaves one under
//! the run's lane directory, `$TMPDIR/toyos-tmp-<pid>-*/tests-*/lane-<n>/`
//! while the run lives (`tests/common/lane.rs`), which is how a frozen guest is read
//! The socket is `/tmp/toyos-qmp.sock`, and a harness boot's is
//! `/tmp/toyos-tmp-<pid>-*/boot-<n>/qmp.sock` (`toyos_tmpdir::TempDir::short`)
//! while its guest lives, which is how a frozen guest is read
//! without a `cargo run` at all: `human-monitor-command` with `info registers
//! -a` gives every vCPU's `RIP`, `RFL` and `HLT`, and that is what tells a
//! halted-awaiting-interrupt machine from a wedged one.
Expand Down
Loading
Loading