Skip to content

Disable four flaky tests behind their filed defects - #565

Merged
Japabu merged 4 commits into
mainfrom
wt/toyos-flakes
Sep 28, 2026
Merged

Japabu merged 4 commits into
mainfrom
wt/toyos-flakes

Conversation

@Japabu

@Japabu Japabu commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • The netd-swap redial failure is one mechanism, seen on lan_swap,
    swap_netd and swap_crash_rolls_back across six sightings. Every
    re-claimed netd worked end to end on every sighting that shows a console;
    only the host's redial gave up first. src/metaltalk.rs's Stream::redial
    dials logd again at once on every refusal with no event to wait on,
    spending a fixed 64-dial ceiling against a guest-side gap the swap does not
    bound — the compromise
    issues/diagnostics/a-swaps-redial-asks-again-with-no-event-to-wait-on.md
    already names. Filed one issue,
    issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md
    (status: expected-red), folding in three prior issue files and a bullet
    that traced the same defect on other days; those three files and the bullet
    are deleted so the three Disabled rows cite one issue.
  • usb_transport_break's AnotherStick red is the already-traced
    issues/kernel/a-held-disk-waits-for-a-pass-no-cpu-takes-when-every-cpu-is-in-a-call-on-it.md
    — its status moves to expected-red and a Disabled row is added naming
    it. That row also hides the test's three other open red modes, now listed
    in the issue as related records:
    issues/kernel/a-shutdown-on-a-held-usb-disk-left-a-cpu-deaf-to-a-tlb-shootdown.md,
    issues/build/usb-transport-break-flushedstick-can-break-after-the-reboot.md
    and issues/boot-media/a-disk-whose-port-went-away-panics-the-boot-at-roots-hold.md.

Test plan

  • cargo test --lib — EXIT=0 (384 passed, 0 failed, 1 ignored)
  • cargo run -- --clippy — EXIT=0, 10 invocations clean

🤖 Generated with Claude Code

https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j

…break with their filed defects

The three swap tests share one mechanism, seen today on 557r2-fast.log and
563r3-fast.log: `Stream::redial` (src/metaltalk.rs) dials logd again at once
on every refusal with no event to wait on, spending a fixed 64-dial ceiling
against a guest-side gap the swap does not bound — a race that landed on all
three names in one day. usb_transport_break's AnotherStick red is the known
held-disk-vs-vfs-lock mechanism already traced on the xhci-wake branch
(554r6-usb_transport_break.log); its issue moves to expected-red rather than
duplicating that branch's own sighting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
@Japabu
Japabu marked this pull request as ready for review September 28, 2026 06:12
@Japabu

Japabu commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator Author

Review r1 of #565 at de58423.

Gate: CI run 36385290264 (host, pull_request) at de58423 completed with conclusion success, and its --ci host and --ci gate-stage steps both succeeded. The branch adds no test and targets no hardware.
Growth: git diff --shortstat origin/main...HEAD is 3 files, +107 −1. Production code +0. src/redlist.rs has +16 lines of rows. Issues are +91 −1. The branch could delete ~98 lines of sibling issues (BLOCKER 1) and does not.

BLOCKER

  • issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:85-90 — This is a fifth file for one defect, and it keeps the other four ("not folded in here"). The tree already carries three: issues/build/lan-swap-redial-spent-its-ceiling-on-a-nightly-shard.md (its own line 44 says it "should go on Test suite, first pass: a disabled list replaces the quarantine, sysret waits on its report, update_* reboot through the power connector #542's disabled list … citing this file"), issues/build/swap-crash-rolls-back-reds-when-its-redial-spends-its-ceiling-under-load.md, and issues/build/swap-crash-rolls-back-redial-turned-away-once-on-mains-nightly.md. The third is a finding, which its next review must promote or fold. The fourth is the swap_netd bullet at issues/build/parallel-tests-red-under-other-suites.md:509. The three rows cite one issue, and the three sibling files and the bullet are deleted in this change. Every one of their exits reduces to the same redial. The rest of parallel-tests-red-under-other-suites.md stays as it is.
  • issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:75-83 — The exit is not fix-shaped and no owner is named. "A deterministic test red on it" is a reproducer. The actual fix is deferred to issues/diagnostics/a-swaps-redial-asks-again-with-no-event-to-wait-on.md, whose exit (a guest-side event from logd/netd) is more work than re-enabling needs. Under the owner's rule, the cited issue has to state (a) an owner, written the way the sibling expected-red files write it (held by …), and (b) a one-line fix-shaped exit. Suggested exit: Stream::redial gives up on its time bound alone, never on a dial count, and a swap whose refusal window is staged long is green. The three rows come off with that fix.

NOTE

  • src/metalswap.rs:55 — Is this a harness defect? Yes. The guest completed all three swaps (both consoles). The red comes from the host's own count, and a count is neither the event nor a timeout. The event is "a dial logd admits". It is only observable by dialing, and each dial already blocks on the machine's answer. Waiting on that event, bounded by a timeout that fails loudly, means dialing until admitted within the swap's window (120 s in tests/common/swap.rs:33). That change mostly deletes code: TURNED_AWAY_CEILING leaves the redial path, redial loses its ceiling, the judge's ceiling finding at src/metalswap.rs:322-327 goes, and turned_away is kept as a reported figure.
  • src/metalswap.rs:55 — Should that fix be made here? No. The owner's rule is disable at once, and the fix changes a verdict. It owes its own negative control, which this branch has not run: an actuator that holds the old netd for about 1 s after CARRIER_LEAVING should be red with the ceiling restored and green without it. It also owes an independent oracle: the guest console's second logd: serving this boot's log to 10.0.2.2:… line after the swap. Give it to a separate branch whose brief is the one cited issue. The diagnostics issue stays open for the spin itself.
  • tests/common/swap.rs:84 — The fix branch has a cheap measurement to take first. The stream reader already prints each turned-away dial's ended N ms after it opened, turned away: <how> under echo, and the rig reads with echo off. The consoles already bound the answer. In swap_netd (563r3-fast.log), 64 dials ended between netd is being replaced at 3.061 and logd listening at 3.387 with none admitted, so each dial took 5 ms or less. Green swaps in the same runs turned away 3, 6 and 22. The dials got faster on the red runs, not slower. That fits the window where logd refuses while the old netd is still up, and it does not fit host contention.
  • src/redlist.rs:46,75,79 — These three rows remove every QEMU run of an accepted netd swap, and with it the only rollback test. swap_refusals and the swap_once_sshd_answers tests never redial. The fix branch is the only way that coverage comes back.
  • src/redlist.rs:87 — On main, issues/kernel/a-held-disk-waits-for-a-pass-no-cpu-takes-when-every-cpu-is-in-a-call-on-it.md carries no sighting of the AnotherStick red this row disables for. That sighting is on unmerged 430b077. The row also hides three other open red modes of this test: issues/kernel/a-shutdown-on-a-held-usb-disk-left-a-cpu-deaf-to-a-tlb-shootdown.md, issues/build/usb-transport-break-flushedstick-can-break-after-the-reboot.md and issues/boot-media/a-disk-whose-port-went-away-panics-the-boot-at-roots-hold.md. The held-disk issue's exit is fix-shaped, so this row stands.

REMOVE

  • issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:10 — a /private/tmp session scratchpad path. It points at nothing on main.
  • issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:22 — "green on about ten other Fast-tier runs": an unmeasured count.
  • issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:30-31 — "arms Stream::redial the moment init answers accepted" is false. It is armed after logd's CARRIER_LEAVING and the go (src/metalswap.rs:193-214).
  • issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:31,33,36 — line numbers, which rot.
  • issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:41-43 — narration ("reached on all three names in one day").
  • issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:45-51 — The table mixes two events in one column. lan_swap's 2.975 s is logd's netd is being replaced, while init stops pid 5 at 3.115 s (557r2-fast.log). The other two rows are init's stopping.
  • issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:53-62,71-73 — The host-contention story is a guess the consoles contradict. Contention slows a dial, so fewer dials fit a gap, not 64. [build-lock]/[host-builds] lines appear around green swaps in 557r2-fast.log too. The siblings record the same red on nightly shards, which run one guest per machine.
  • PR body — "That sighting is already recorded on wt/toyos-xhciwake (commit 430b077e, not yet merged to main)" rots when that branch lands. The same line is in the commit message.
  • PR body — "a CPU spent ~4 s in logd's create holding vfs::lock(), so no CPU took the scheduler pass" states an inference as fact. The record it cites says "That it spun on the VFS lock is inferred, not measured".
  • PR body — scratchpad log names (557r2-fast.log, 563r3-fast.log, 554r6-usb_transport_break.log) and the "Not run: … never run per policy" line. Neither belongs in main's record.

SEND BACK

Japabu and others added 2 commits September 28, 2026 08:35
…e its owner and a fix-shaped exit

Three separate issue files and a bullet all traced the same swap-redial
defect to the same fixed dial ceiling; the review named it a fifth file for
one mechanism. Their true content folds into
issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md
as prior sightings, and the three files and the bullet are deleted so the
three disabled rows cite one issue.

The exit condition was a reproducer, not a fix, and named no owner. It is now
one line naming the redial's time-bound fix, owned the way the other
expected-red issues write it. The host-contention paragraph is replaced with
the review's measurement: 64 dials were refused in 326 ms with none taking
more than 5 ms, while green swaps were refused 3, 6 and 22 times — the
refusal window, not contention.

The held-disk issue lists usb_transport_break's three other open red modes
as related records, since its redlist row now hides them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
@Japabu

Japabu commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator Author

Review r2 of #565 at 4a7e805.

Gate: CI run 36387429942 (host, pull_request) at 4a7e805 completed with conclusion success. Its --ci host and --ci gate-stage steps both succeeded. The branch adds no test and targets no hardware.
Growth: git diff --shortstat origin/main...HEAD is 7 files, +96 −107, so the branch is 11 lines net smaller. Production code is +0. src/redlist.rs gains 16 lines of rows. Issues are +80 −107.

Round 1 BLOCKERs

  • CLOSED — the siblings. lan-swap-redial-spent-its-ceiling-on-a-nightly-shard.md, swap-crash-rolls-back-reds-when-its-redial-spends-its-ceiling-under-load.md and swap-crash-rolls-back-redial-turned-away-once-on-mains-nightly.md are deleted, and so is the swap_netd bullet in parallel-tests-red-under-other-suites.md. An rg over the tree at 4a7e805 finds no path or bare name of the three deleted files. All three Disabled rows cite the one issue.
  • CLOSED — the owner and the exit. issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:68-71 carries the one-line exit ("gives up on its time bound alone, never on a dial count"), and "held by the orchestrator" is in.

BLOCKER
(none)

NOTE

  • issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:70 — The owner is another issue, not code. That issue's exit (a guest-side event, with the ceiling and the count deleted) is not this one's, so this issue can close while that one stays open. The siblings name code, for example "Owner: the sleep path, kernel/src/sched". Name src/metaltalk.rs's Stream::redial when the file is next touched.
  • issues/diagnostics/a-swaps-redial-asks-again-with-no-event-to-wait-on.md:19-22 — Its "What bounds it" paragraph goes false once wt/toyos-redial deletes TURNED_AWAY_CEILING. That branch owns deleting it, not this one.
  • src/redlist.rs:95 — The AnotherStick sighting the row disables for is still only on origin/wt/toyos-xhciwake (430b077), not in the cited issue on main. As in r1, the issue's exit is fix-shaped, so the row stands.

REMOVE

  • issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:12-13,57 — "today's" and the scratchpad log names 557r2-fast.log and 563r3-fast.log. They point at nothing on main, and "today" rots.
  • issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:27-28 "(on the two Restored boots)" and :49-50 "on the two Restored/crash-rollback tests" — false. Only swap_crash_rolls_back judges Expect::Restored (tests/common/swap.rs:324). swap_netd and lan_swap judge InService.
  • issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:21-24 — The block is quoted as output, but no run printed "InService/Restored". It is two lines merged into one.
  • issues/build/a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md:61-64 — Cut from "not at host contention:" to the end of the paragraph. It argues against the guess r1 removed, and the numbers before it already carry the point.
  • issues/kernel/a-held-disk-waits-for-a-pass-no-cpu-takes-when-every-cpu-is-in-a-call-on-it.md:49,51,53 — The three parentheticals repeat the file names they follow.
  • PR body — "today and earlier" rots.
  • PR body — "A same-day measurement replaces the earlier host-contention guess: … not at host contention." It repeats the issue and narrates the review.
  • PR body — "The issue names its owner ("held by the orchestrator") and a one-line, fix-shaped exit: …" narrates review compliance.
  • PR body — The unchecked "No QEMU run (out of scope — …)" item. r1 cut this line, and it came back reworded.

LAND AFTER NAMED CHANGES

…nd false parentheticals

The redial-ceiling issue's exit names src/metaltalk.rs's Stream::redial as
its owner instead of the diagnostics issue it already cites elsewhere, since
that issue's own exit is unrelated and can close independently. Drops
today-dated scratchpad log names that point at nothing on main, a fabricated
"each ended" log block no run actually printed, and three false claims about
which tests judge Restored vs InService. The held-disk issue's three related
records drop parentheticals that only repeated the file names they follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
@Japabu
Japabu enabled auto-merge September 28, 2026 07:00
@Japabu
Japabu added this pull request to the merge queue Sep 28, 2026
Merged via the queue into main with commit 002af8c Sep 28, 2026
1 check passed
Japabu added a commit that referenced this pull request Sep 28, 2026
- rust: the fork's 62fa74d7a50 merges main's pin 1b236638a90 (#541) into
  this branch's fee9fa6f7bb; the two share no file, and the merge over main's
  pin is this branch's six std files alone.
- issues/build/a-lane-s-tap-socket-path-outgrows-sun-len-on-the-dev-host.md:
  deleted as #560 deleted it. This branch's hunk was the two path lengths
  PR #536's lanes hit and an A/B on $TMPDIR's length; #560's short TempDir
  under /tmp is the fix for both lengths, so nothing of it is owed.
- issues/build/swap-crash-rolls-back-redial-turned-away-once-on-mains-nightly.md:
  deleted as #565 deleted it, folding it into
  a-swaps-redial-races-a-hard-dial-ceiling-against-an-unbounded-guest-gap.md.
  This branch's hunks: the status moved to expected-red and the known-red
  line (both carried by #565's redlist rows for swap_crash_rolls_back and
  lan_swap); the mechanism, a probation-long gap against a ceiling of 64
  refusals and not a time (carried by the folded issue's "What the code
  shows"); and the #536 sightings, added there as a seventh bullet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
@Japabu
Japabu deleted the wt/toyos-flakes branch September 28, 2026 09:46
Japabu added a commit that referenced this pull request Sep 28, 2026
Takes #560, #541, #565, #563, #569 and #570. `src/ci.rs` and
`tests/toyos.rs` merge without conflict; `rust` takes main's pin, 1b236638,
since this branch carries no fork commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant