Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
status: open
kind: tooling
opened: 2026-09-28
---

# `wait_until` for init's final word never wakes once the stream is dead

`Stream::wait_until` (`src/metaltalk.rs:260`) is woken by each line as it
lands, and by nothing else. `wait_for_connection` (`src/metaltalk.rs:289`)
computes its own `dialing` — `redial.is_some() || current.is_some() ||
unopened.is_none()` — and returns `None` at once once a redial has ended with
no connection: the stream can never carry another line. `metalswap::swap`'s
call to `wait_until` for init's final word (`src/metalswap.rs:214`) has no
such exit, so it waits out its whole `window` even after the stream has
reached that same dead state.

Predates this branch: a redial that ends without naming a cause already left
`wait_until` waiting before `src/metaltalk.rs:383` started ending a forward's
redial at its first refusal.

Evidence: hold-red took 124 s this way, run at PR #566's `7e06a657`
(`566r2-hold-red.log:507`, orchestrator's round-2 review job).

## Exit condition

Give `wait_until` the same exit `wait_for_connection` already has: return
`None` once the stream's own `dialing` state goes false, instead of waiting
out `by`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
status: open
kind: defect
opened: 2026-09-28
---

# Connects are turned away for 6 s after `logd` listens again

In a swap of netd, connects to `logd`'s port keep being turned away for about
6 s after `logd` itself has already re-bound the port and init has said the
new netd is in service. In a hold-red run at PR #566's `7e06a657`, netd
re-bound `41337` at 1.758 s and init said `in service` at 6.739 s
(`566r2-hold-red.log:330`, `:332`); the matching hold-green run (guest-
identical, since the branch's fix is host-only) only had its redial admitted
at 7.721 s (`566r2` hold-green oracle line). That is what makes the redial's
window long after a swap — not `logd`'s closed listener during the swap
itself, which is announced and handled.

## Exit condition

Whatever holds a bound listener from accepting for those ~6 s is named, and
either removed or bounded by a measured floor — shown by a run whose
admission time tracks `logd`'s own re-bind and init's `in service`, not
trailing it by seconds.
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
---
status: open
status: assigned
kind: defect
opened: 2026-09-25
---

# A swap's redial asks again with no event to wait on

Held by the orchestrator.

A swap of netd ends the host's log stream with no FIN and no reset, and
`src/metaltalk.rs`'s `Stream::redial` dials `logd` again. Nothing the machine
sends says when `logd` listens again: the stream and the ssh channel both die
Expand All @@ -16,12 +18,15 @@ forward, accepted and closed before a line — is asked again at once. On a LAN
that is a question for the name on the link, which the old netd answers at
once, and a `connect` per round trip for the whole gap.

What bounds it: every dial turned away is counted, refusals included
(`Stream::turned_away`), and a redial gives up at
`metalswap::TURNED_AWAY_CEILING`, which the swap's judge reds on by name
(`a_redial_counts_every_refusal_and_gives_up_at_its_ceiling`). The T14 is
unmeasured, and a refusal there costs a LAN round trip rather than QEMU's
forward's.
The T14 is unmeasured, and a refusal there costs a LAN round trip rather than
QEMU's forward's. Its redial asks the name on the link again after every dial
turned away, as soon as the old netd answers the last ask, so its questions
may go out faster than RFC 6762 §5.2's floor between two queries
(`ASK_WAIT`); that rate is unmeasured on metal.

The forward's cost is measured: a hold-green run turned away 8125 dials in
7019 ms, and the guest logged 16231 of its 19034 interrupts over that boot, all
on cpu0.

## Exit condition

Expand All @@ -30,4 +35,5 @@ machine sends when `logd` can admit a reader again after a swap of netd —
for example `logd` keeping its listener across the swap and holding the
connections it accepts until the new netd serves, or netd announcing its
exit on a channel that outlives it — and `Stream::redial` dials once per
such event, with the ceiling and the count deleted.
such event, with the count deleted, so no redial asks the link faster than
§5.2's floor.
24 changes: 24 additions & 0 deletions issues/hardware/the-t14-redial-re-asks-mdns-after-every-refusal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
status: open
kind: defect
opened: 2026-09-28
---

# The T14 redial re-asks mDNS after every refusal

`open` (`src/metaltalk.rs:383`, `:419`, `:452`) sets `on_the_link` on the
first failed dial and never clears it, so once the old or new netd starts
sending refusals or resets, every dial that follows first asks the link for
the name again (`ask_the_link`) before it redials. Without a dial ceiling to
stop that loop, a gap where the machine keeps answering with a refusal sends
multicast queries at LAN round-trip rate for as long as `wait_secs` runs —
below the RFC 6762 §5.2 floor between two queries (`ASK_WAIT`) that the code
itself cites.

## Exit condition

A timer-free fix: set `on_the_link` only on a host-absent failure
(`not_yet_reachable`'s `EHOSTDOWN`, `EHOSTUNREACH`, `ENETUNREACH`, or a
failure seen after `WAITED`). A refusal or reset is the machine answering at
that address, so the next dial goes there again with no new ask, and only a
dial that finds nobody home asks the link once more.
3 changes: 1 addition & 2 deletions src/metal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1791,8 +1791,7 @@ impl Talking {
};
println!("asking for {peer:?}'s log");
let at = self.dir.join(file);
crate::metaltalk::Stream::connect(peer, &at, true, by, crate::metalswap::TURNED_AWAY_CEILING)
.map_err(Refusal::Cable)
crate::metaltalk::Stream::connect(peer, &at, true, by).map_err(Refusal::Cable)
}
}

Expand Down
26 changes: 8 additions & 18 deletions src/metalswap.rs
Original file line number Diff line number Diff line change
Expand Up @@ -46,14 +46,6 @@ const REFUSED_WORD: Duration = Duration::from_secs(10);
/// a missing line is a red verdict rather than a longer wait.
const CARRIER_WORD: Duration = Duration::from_millis(toyos_swap::ANSWER_MS);

/// How many dials a stream's first dial, or a swap's redial, may have turned
/// away — a failed connect, or a connection closed before a line — before it
/// gives up and the boot or the swap is red. Nothing the machine sends says
/// when `logd` listens again after a swap, so a redial asks again at once and
/// this ceiling is the one thing that stops it spinning unseen: the recorded
/// compromise `issues/diagnostics/a-swaps-redial-asks-again-with-no-event-to-wait-on.md`.
pub const TURNED_AWAY_CEILING: usize = 64;

/// What asking for one swap came to.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Swapped {
Expand Down Expand Up @@ -211,7 +203,11 @@ pub fn swap(
}
}
if accepted && service == toyos_logstream::CARRIER {
stream.redial(window, TURNED_AWAY_CEILING);
let (left, seen, redialed) = (window.saturating_sub(began.elapsed()), stream.connections(), Instant::now());
stream.redial(left);
if stream.wait_for_connection(seen, left).is_some() {
println!(" swap: `logd` admitted the stream again {} ms after the redial", redialed.elapsed().as_millis());
}
}
let mut outcome_ms = None;
if let Some(until) = until {
Expand Down Expand Up @@ -319,13 +315,7 @@ pub fn judge(heard: &Swapped, expect: Expect) -> Result<Vec<String>, Vec<String>
heard.connections.1
)),
}
if heard.turned_away >= TURNED_AWAY_CEILING {
bad.push(format!(
"the stream's redial was turned away {} time(s), its ceiling of {TURNED_AWAY_CEILING}, \
and gave up",
heard.turned_away
));
} else if heard.service == toyos_logstream::CARRIER {
if heard.service == toyos_logstream::CARRIER {
said.push(format!(
"the stream's dials were turned away {} time(s), refusals included, from the ask to \
the end",
Expand Down Expand Up @@ -605,8 +595,8 @@ mod tests {
elsewhere.words.last_mut().unwrap().1 = "/tmp/swap/other/netd as pid 12".into();
assert!(judge(&elsewhere, Expect::InService).is_err());
let mut spun = heard(Expect::InService);
spun.turned_away = TURNED_AWAY_CEILING;
assert!(judge(&spun, Expect::InService).is_err(), "a redial that reached its ceiling");
spun.turned_away = 10_000;
assert!(judge(&spun, Expect::InService).is_ok(), "how many dials were turned away is no verdict");
}

#[test]
Expand Down
Loading
Loading