Skip to content

Disable flaky netd_refused_accept (hung 2341s waiting on a wake) - #579

Merged
Japabu merged 5 commits into
mainfrom
wt/toyos-netdred
Sep 28, 2026
Merged

Japabu merged 5 commits into
mainfrom
wt/toyos-netdred

Conversation

@Japabu

@Japabu Japabu commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Round 3

  • Deleted (not rewrote) the four remaining false tests/common/qemu.rs
    passages the round-3 review named: the GUEST_WEDGED doc comment's
    "Width is what a ceiling on a slow guest..." paragraph and its "Price a
    new waiting check against this number" line, ceiling_verdict's "So a
    guest still talking is now never ended by ceiling..." sentence, and
    screendump_while_rendering's "or the [GUEST_WEDGED] backstop" — all
    false once a ceiling exceeds GUEST_WEDGED, since the real backstop is
    ceiling.max(GUEST_WEDGED).
  • ceiling_self_check gained the case the round-3 NOTE named: a ceiling
    above GUEST_WEDGED with a guest still talking past GUEST_WEDGED but
    short of that ceiling must not be ended at GUEST_WEDGED. Mutating
    qemu.rs:685's let backstop = ceiling.max(GUEST_WEDGED); to
    let backstop = GUEST_WEDGED; fails the new case (checks::serial_vocabulary)
    with "a guest talking past GUEST_WEDGED but short of a higher ceiling was
    ended anyway — the backstop did not follow a ceiling above GUEST_WEDGED";
    restoring the line turns it green again.

Test plan

  • cargo run -- --known-red netd_refused_accept reports the test
    disabled, pointing at the new issue file.
  • cargo test -p toyos-build --lib — exit 0, 398 passed, 0 failed, 2
    ignored.
  • cargo test --test toyos-build -- --list — exit 0.
  • cargo test --test toyos-checks -- serial_vocabulary — exit 0 (green);
    exit 101 with the named mutation applied, restored and re-verified
    green.
  • Not run: any guest suite (agents never run QEMU).

🤖 Generated with Claude Code

https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j

Orchestrator Fast tier at #562's head 2a9c77e timed the test out at "timed
out after 2341s, with the guest still talking 8s ago", stuck on the wake for
the connection an accept refused for room left; #562 touches nothing this
test runs. A flaky test is disabled at once behind its filed issue
(issues/kernel/netd-refused-accept-hung-waiting-for-a-wake-that-never-came.md),
which also records the unconfirmed reading that the host's dial ended on a
reset rather than a graceful close, returning netd's listener to Listen with
no wake owed — not established without a kept console.

A second, unrelated sighting from the same run: the hang cost the full
39-minute ceiling backstop rather than the much smaller GUEST_QUIET silence
window, because the guest's periodic console lines never let it go quiet.
tests/CLAUDE.md already documents this as intended, so it is filed as a
tooling note on the cost rather than a defect
(issues/build/a-stuck-but-chatty-guest-costs-the-full-ceiling-backstop-before-the-harness-ends-it.md).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
@Japabu
Japabu marked this pull request as ready for review September 28, 2026 15:59
@Japabu

Japabu commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator Author

Review, round 1, at 11ada4e

CI: host SUCCESS at 11ada4e (run 36447730406). cargo run -- --known-red netd_refused_accept exit 0: "YES, disabled — it does not run", naming the new issue. The redlist row is in alphabetical order and has the same shape as its neighbours. Net diff +68/-0: no production code, 4 redlist lines, 64 tracker lines.

These facts in the netd issue were checked against the tree and the log and hold:

  • Log lines 1160, 1166 and 1168.
  • netd_refused_accept.rs:63.
  • listen.rs:73-85.
  • "identical to main": md5 of the guest binary's source, listen.rs, tests/netcase/system.toml, netcase_against_host, fn netd_refused_accept and PatternServer is equal at d4e4102, 2a9c77e and origin/main.
  • The unconfirmed reading is labelled as unconfirmed.

Every qemu.rs/toyos.rs line the harness issue cites resolves.

BLOCKER

  • issues/build/a-stuck-but-chatty-guest-costs-the-full-ceiling-backstop-before-the-harness-ends-it.md — delete the file. It records no weakness:

    • The body says "Nothing here says the design is wrong".
    • It restates what the tree already says: tests/CLAUDE.md ("only a far backstop stands behind a guest that keeps talking") and tests/common/qemu.rs:505-508 and :523-528 ("the real ceiling of any failing wait on a shared boot").
    • It is kind: tooling, status: open with nothing owed. issues/README.md: "if the body says otherwise, the kind is what is wrong".
    • It has no owner and no exit condition.

    The weakness this sighting does measure is a different one. The backstop ceiling.max(GUEST_WEDGED) (qemu.rs:702) paid 2341 s: the width- and host-scaled 120 s budget. GUEST_WEDGED's own contract (qemu.rs:516-522) says the backstop is "Not budget-scaled, and that is the point of the pair … scaling it would only make that state cost an hour at width 12". The two contracts contradict each other, and this log measured the cost. File that, with an owner and an exit condition, or file nothing.

NOTE

  • issues/kernel/netd-refused-accept-hung-waiting-for-a-wake-that-never-came.md:31 — nothing in the tree can meet the exit condition ("established from a kept console"), and the issue does not say so:

    • On its error path, netcase_against_host (tests/toyos.rs:10113-10121) returns result.stdout and drops result.serial.
    • The netcase boot writes no uart-*.log. None of the 104 files kept in target/red-run-serial/toyos-tmp-59761-0 mentions netd.
    • The test now runs nowhere.

    Name the instrument that is owed (keep result.serial on that path), or the exit cannot happen.

  • issues/kernel/netd-refused-accept-hung-waiting-for-a-wake-that-never-came.md — wrong area, and the owner line names a path, not a holder:

    • Nothing in the file implicates the kernel.
    • Its netd-listener siblings live under issues/hardware/: a-connect-between-two-accepts-is-reset.md, an-accept-that-never-reaches-netd-strands-its-listener.md and a-handshake-nobody-finishes-holds-a-listeners-port-shut.md.
    • They name their owner as "whoever holds issues/design-debt/toyos-has-its-own-network-stack.md".
    • Fix: git mv, move the redlist row with it, and use the sibling owner line.

REMOVE

  • issues/kernel/netd-refused-accept-hung-waiting-for-a-wake-that-never-came.md:20-21 — "so it likely ended on a reset rather than a graceful close". dial (tests/toyos.rs:9996-10003) answers Ok(written) for every write error except a stall, and a passing run's dial also ends Ok(N). Ok cannot tell a reset from a FIN.
  • issues/kernel/netd-refused-accept-hung-waiting-for-a-wake-that-never-came.md:31 — "held by the orchestrator" says assigned on an expected-red entry.
  • Both issues — the orch-runs/562r5-fast.log` lines ~1160-1169 citation. It is a session scratchpad path no reader can resolve. The quoted FAIL line is the evidence.
  • PR body, second bullet from "That issue records the log evidence as fact" to "kept console." — it summarises the file in main's merge record.
  • PR body, third bullet — it goes with the deleted file.

SEND BACK

…'s area and owner, and two removed claims

The deleted tooling issue restated tests/CLAUDE.md's own design and owed
nothing. In its place: GUEST_WEDGED's own doc says the wedged-but-chatty
backstop is "Not budget-scaled", but ceiling_verdict's backstop is
ceiling.max(GUEST_WEDGED) and ceiling is already width/host-scaled, so once
a scaled ceiling exceeds 300s the backstop is that scaled ceiling, not the
number the doc names — exactly the 2341s netd_refused_accept paid.

The netd issue moves to issues/hardware/ beside its listener siblings, takes
their owner line, and its exit condition now names the instrument it needs:
netcase_against_host drops result.serial on its error path, so no kept
console can exist until that path keeps it. Also removed: the "likely ended
on a reset" claim (Ok cannot tell a reset from a FIN), "held by the
orchestrator" on an expected-red entry, and both files' scratchpad-path
citations, keeping the quoted FAIL line as the evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
@Japabu

Japabu commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator Author

Review, round 2, at 9ac43db

CI: host COMPLETED SUCCESS at 9ac43db (run 36449911663). cargo run -q -- --known-red netd_refused_accept exited 0 with "YES, disabled — it does not run" and named issues/hardware/netd-refused-accept-hung-waiting-for-a-wake-that-never-came.md. Net diff is +72/-0: 4 redlist lines, 68 tracker lines, no production code.

Round 1

  • BLOCKER (the harness issue) — OPEN.
    • The old file is deleted, and nothing in the tree cites it (git grep at HEAD).
    • Its replacement's exit condition can be met while the contradiction stands, and nothing gates either branch of it. See below.
  • NOTE (no instrument in the netd exit) — CLOSED. The exit now names keeping result.serial on netcase_against_host's error path. At tests/toyos.rs:10114-10122 both error arms drop it. The line after them, :10123, keeps it only on success.
  • NOTE (area and owner) — CLOSED. The file is under issues/hardware/, the redlist row moved with it, and the owner line is the same as its siblings'.
  • REMOVE "likely ended on a reset" — OPEN. It was rewritten, not deleted: "which path the reset actually took" still asserts that a reset happened.
  • REMOVE "held by the orchestrator" — CLOSED.
  • REMOVE the orch-runs/ citation — CLOSED.
  • REMOVE both PR-body passages — CLOSED.

The contradiction, checked against every caller of the backstop

  • ceiling_verdict has two callers, and both pass a scaled ceiling:
    • run_test_paced, via budget_smp(timeout, self.smp) at qemu.rs:3615. The sighting took this path: netcase_against_host → run_test, 120 s.
    • screendump_while_rendering at qemu.rs:3314.
  • These use GUEST_WEDGED raw, unscaled:
    • guest_liveness → await_guest (qemu.rs:533, :980), which settle_null_sink_client_exits reaches at toyos.rs:3376.
    • QmpHold::held at update.rs:193.
    • await_machine at update.rs:264.
  • So the contradiction is real: for run_test_paced a stuck-and-chatty guest costs max(scaled ceiling, 300 s), and 2341 s is what that cost.

BLOCKER

  • issues/build/guest-wedgeds-not-budget-scaled-contract-is-broken-by-its-own-pairing-with-ceiling.md:9-18,30-33 — the exit condition cannot close what the issue names, for three reasons:
    • The intent is already recorded. The comment on the .max() itself (qemu.rs:696-701) says the scaling is intended: "Never below the per-test ceiling, so a long test whose own budget already exceeds it is not cut short … ceiling already carries that". So the exit's "if scaling … is actually intended" is already answered by the tree.
    • The first branch brings back what 696-701 prevents. Ending at the unscaled 300 s kills a talking guest below its own scaled budget.
    • The second branch leaves the false claims standing. Deleting only :516-522 leaves :525-529 ("the real ceiling of any failing wait"), :663 ("a talking one runs to the GUEST_WEDGED backstop") and :818-819 ("the shape every real test has") false.
    • Neither branch is gated. By reading, ceiling_self_check stays green under - let backstop = ceiling.max(GUEST_WEDGED); / + let backstop = GUEST_WEDGED;. Every case either has ceiling ≤ 300 s or checks a talking guest past both 300 s and its ceiling, so either rule gives the same verdict.
    • Fix: rewrite the exit condition to require all of these:
      • one backstop rule;
      • every statement of it in qemu.rs agreeing with that rule;
      • a ceiling_self_check case with a ceiling above GUEST_WEDGED and a talking guest between the two, which turns red under the rule that was rejected.
    • Or file nothing, as round 1 allowed.

NOTE

  • issues/build/guest-wedgeds-not-budget-scaled-contract-is-broken-by-its-own-pairing-with-ceiling.md:35 — "whoever next touches ceiling_verdict" names nobody. The one who pays the 2341 s is whoever runs the guest suites, and two siblings in issues/build/ already say Owner: orchestrator.

REMOVE

  • issues/build/guest-wedgeds-not-budget-scaled-contract-is-broken-by-its-own-pairing-with-ceiling.md:24-28 — the list of GUEST_WEDGED's other users. It omits guest_liveness/await_guest (qemu.rs:533, :980), the main unscaled user. It cites toyos.rs:3368, which is a doc comment. And it will go stale as callers change.
  • issues/hardware/netd-refused-accept-hung-waiting-for-a-wake-that-never-came.md:25-26 — "so which path the reset actually took is not established". It asserts a reset that Ok(585728) cannot distinguish from a FIN.

SEND BACK

… prose, and stop the netd issue asserting a reset

The harness issue's exit condition could close while the contradiction it
named stood, and both its branches were already ruled out by qemu.rs's own
comment on `ceiling_verdict`'s backstop (:696-701): the pairing with
`ceiling` is intended, so the backstop already is the scaled ceiling once
that ceiling exceeds GUEST_WEDGED — exactly what the 2341s netd_refused_accept
paid. So the issue is deleted, and the prose that called that backstop
"Not budget-scaled" (qemu.rs:517, :522, :525-528), or its destination "the
GUEST_WEDGED backstop" (:662-663), or its ceiling-below-backstop shape "every
real test has" (:818-819), is deleted rather than rewritten: it was false of
the tree, not the tree's own guard.

The netd issue's last sentence claimed "which path the reset actually took"
was left unestablished — but Ok(585728) cannot tell a reset from a graceful
close, so no reset was ever established to have a path. Deleted along with
the rest of that clause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
@Japabu

Japabu commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator Author

Review, round 3, at 5a38b52

CI: host COMPLETED SUCCESS at 5a38b52 (run 36454480065). cargo run -q -- --known-red netd_refused_accept exited 0 with "YES, disabled — it does not run" and named issues/hardware/netd-refused-accept-hung-waiting-for-a-wake-that-never-came.md. git merge-tree --write-tree origin/main HEAD exited 0 with no conflicts. Net diff +46/-16: no production code, redlist +4, tracker +33, tests/common/qemu.rs +9/-16 (doc comments only).

Round 2

  • BLOCKER (the harness issue) — CLOSED under the orchestrator's ruling. The file is deleted. git grep for both harness-issue names exits 1. The contract sentences that called the backstop unscaled are deleted.
  • NOTE (harness issue owner) — CLOSED. The file is gone.
  • REMOVE (harness issue's user list) — CLOSED. The file is gone.
  • REMOVE ("which path the reset actually took") — CLOSED. What remains ("A reset would…") is a conditional under "Unconfirmed reading".

The five deleted qemu.rs sentences

All five were false or misleading, so deleting them was right:

  • Old :517, "Not budget-scaled". True of the constant as guest_liveness, QmpHold::held and await_machine use it. False of the ceiling_verdict backstop it is paired with. So it misleads.
  • Old :522, "cost an hour at width 12". run_test_paced already pays that: the 2341 s.
  • Old :525-528, "real ceiling of any failing wait". False for any ceiling above 300 s. The PR Two CI reds fixed at their owners: the removal nobody waited for, and the i8042 line that was not the test's #96 provenance was chronology.
  • Old :662-663, "the GUEST_WEDGED backstop". False for any ceiling above 300 s.
  • Old :818-819, "the shape every real test has". False. Its first half is already carried by the assert message at :815.

What is left reads grammatically and leaves nothing dangling. Some of it is still false, below.

BLOCKER

None.

NOTE

  • tests/common/qemu.rs:696 — the mutation - let backstop = ceiling.max(GUEST_WEDGED); / + let backstop = GUEST_WEDGED; still leaves ceiling_self_check green. No case in it has a ceiling above GUEST_WEDGED and a talking guest between the two. It is outside this PR's fence under the ruling, and nothing in the tree now records it.

REMOVE

  • tests/common/qemu.rs:655-657 — "So a guest still talking is now never ended by ceiling: the per-test budget bites only a guest that has also gone quiet". False whenever the ceiling is above GUEST_WEDGED: the backstop is then ceiling, and it ended a talking guest at 2341 s. This is the same falsehood deleted at old :662-663, and this branch rewrote line 657.
  • tests/common/qemu.rs:517-522 — rewritten lines. They argue for a conclusion ("not scaled") that was deleted. "it is never judged by this at all" is false: a talking guest whose ceiling is under 300 s is judged by GUEST_WEDGED at :696.
  • tests/common/qemu.rs:524 — "Price a new waiting check against this number" is a rewritten line. It is false for both ceiling_verdict callers, whose price is ceiling.max(GUEST_WEDGED).
  • tests/common/qemu.rs:3285-3286 — "or the [GUEST_WEDGED] backstop". The same phrase was deleted at old :662-663 as false. screendump_while_rendering passes budget_smp(timeout, self.smp) (:3308) into ceiling_verdict, so its backstop is ceiling.max(GUEST_WEDGED).

LAND AFTER NAMED CHANGES

Japabu and others added 2 commits September 28, 2026 19:14
…ceiling_self_check gets the case that would have caught the round-2 NOTE

Each was false whenever a ceiling exceeds GUEST_WEDGED, the same falsehood
already deleted elsewhere in round 2 or contradicted by ceiling_verdict's own
backstop computation (ceiling.max(GUEST_WEDGED)): a talking guest is not
always spared by `ceiling`, GUEST_WEDGED does not judge every talking guest,
a new check's price is not this constant alone, and screendump_while_rendering's
backstop is not GUEST_WEDGED but ceiling.max(GUEST_WEDGED).

ceiling_self_check gained the case the NOTE named: a ceiling above
GUEST_WEDGED with a guest talking past GUEST_WEDGED but still short of that
ceiling must not be ended at GUEST_WEDGED. Mutating qemu.rs:685's
`let backstop = ceiling.max(GUEST_WEDGED);` to `let backstop = GUEST_WEDGED;`
fails the new case with "a guest talking past GUEST_WEDGED but short of a
higher ceiling was ended anyway — the backstop did not follow a ceiling above
GUEST_WEDGED"; restoring the line turns it green again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
@Japabu
Japabu added this pull request to the merge queue Sep 28, 2026
Merged via the queue into main with commit a2c74f2 Sep 28, 2026
1 check passed
@Japabu
Japabu deleted the wt/toyos-netdred branch September 28, 2026 17:46
Japabu added a commit that referenced this pull request Sep 28, 2026
Brings in #572 (host QEMU's edk2), #580 and #579 (disabled reds), #549
(a kill never waits on its victim) and #566 (metaltalk redial).

- src/redlist.rs: one row each for user_copy_races_munmap and
  quiesce_leaves_the_volume_whole, which both sides added. main's new rows
  stay (netd_refused_accept, quiesce_wakes_on_the_last_teardown,
  root_chunk_refused_on_a_usb_stick, syscall_window_nmi). The rows for
  tests or issues this branch deleted go (hda_tone, doom_sound_flood,
  latency_wake, sched_check_build), and so does lan_swap, whose issue
  main deleted with swap_netd's and swap_crash_rolls_back's rows.
- The two issue files both sides added take main's text.
- tests/common/power.rs: main's woken_by_the_held_thread, shared by the
  new quiesce_wakes_on_the_last_teardown, without the two clock verdicts
  this branch took off QEMU (stopped_the_machine in stopped_boot, and
  woken_by_its_threads).
- tests/common/qemu.rs: qemu_command takes main's firmware_vars and has
  no audio_wav, so profile_argv passes six paths. The
  too_many_arguments allow goes, because seven parameters do not
  trigger it.
- kill_while_blocked.rs: main's text. After #549 a kill does not park
  in retire_task, so this branch's doc for arm 4 was false. main's arm
  also has no clock.
- tests/toyos.rs check_rust_result: this branch's single-print form,
  which already carries the stdout main added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant