netd: a listener's socket that left Listen is handed over or listens again, and a refused accept leaves its owner a wake - #559
Conversation
…again A listener is one smoltcp socket that becomes the connection it accepts, so its port listens only while that socket is in Listen. netd woke the owner only when a pass saw the socket Established, and accept took only Established. A peer whose handshake-closing ACK and FIN land in one pass moves the socket SynReceived -> CloseWait with no pass seeing it Established; an ACK and a reset in one pass leave it Closed. Either way the owner is never woken, the socket never listens again, and every later SYN on the port is answered with a reset, with nothing logged on either side. The rule is `listen::Listening`, one type the pass and accept both use. A socket in Established or CloseWait holds a connection its owner is woken for and takes; a Closed one listens again. `settle` is the one function that reads a listener's socket state, and it writes: it is what puts a Closed socket back into Listen. Taken from bdfc2c5 (userland/netd only); `connection_waiting` is renamed `settle` because it re-listens a closed socket. Host negative controls on listen.rs, each measured on bdfc2c5's tree with a checked patch that built and was restored: CloseWait answered like Listen, Closed not re-listened, accept keeping the wake; each exits 101. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eighty peers connect through QEMU's forward and leave at once, half of them with a zero linger so the close is a reset; then `echo` over ssh must be answered. The forward finishes the guest's handshake after the host has closed and sends the FIN straight behind the last ACK, which is how a listener's socket goes SynReceived -> CloseWait with no pass seeing it Established. A netd that wakes its owner only for Established leaves port 22 resetting every SYN for the rest of the boot. It is the committed arm for netd's wiring of `listen::Listening`: the host tests in userland/netd call `Listening` directly and stay green when the pass stops using it. A reset ask is retried at once, since a hasty handshake still in flight shuts the port without sshd hearing of it; a second reset waits on sshd saying something, since a connection netd holds for sshd shuts the port until sshd takes it. A port that stays shut ends the wait as a guest gone quiet, which is the red. No fixed delay: the measured actuator this comes from paced its rounds and retried its ask on a one-second sleep, and neither is kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… is room
The accept's room refusal and its missing-pipes refusal returned before
the accept spent the owner's wake. The owner had already read its wake
byte, but netd still held it as given, so no pass announced the waiting
connection again: a std server refused for room blocked in its next
accept for the rest of the boot while the connection held the port. sshd
got out only because it rebinds on any accept error.
The rule, in `listen::Listening`: an accept spends the owner's wake
whatever it answers, a refusal included, and a wake is owed only for a
connection there is room to take. `Listening::accept` takes the room and
answers `Accept::{Take, NoRoom, Nothing}`, so the room refusal cannot skip
the spend; the missing-pipes refusal comes after it. Room is in the wake
condition so that an owner refused for room is woken again when room
returns and not on every pass before it, which spending alone would do:
each refused accept would be answered by a wake on the next pass.
`serve_piped_listeners` moves after `process_pending`, which frees room
when a pending connect ends: the wake condition is read after everything
in a pass that changes it, with nothing between it and the wait.
Arms: the host test
an_accept_refused_for_room_is_woken_again_when_room_returns, and the
machine test netd_refused_accept (tests/netcase), whose guest refuses an
accept for its pipes and one for room and waits on the wake each leaves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on smoltcp 0.12's interface over a hand-played wire (the harness of userland/netd/src/listen/tests.rs): after one SYN and 600 s of silence the listener's socket was still SynReceived, having sent 72 SYN-ACKs, and another peer's SYN was answered with a reset. Filed as a defect, and both listener defects are named under the network-stack track, whose holder owns them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review round 1 of Gate. CI BLOCKER
NOTE
REMOVE
The brief's questions
Runs required at the new headRun each with
SEND BACK |
…sten.rs `Listening::wake` returns the bytes the pass writes and records the wake it returns, so `main.rs` reads no socket state and writes what it is handed. That is sound because a pass whose owed write fails ends the listener. `Listening::accept` takes the pipes the request carried as an `Option` and answers `NoPipes` after spending the wake, so the pipes refusal is ordered against the spend inside the one function and is host-tested (`an_accept_refused_for_its_pipes_is_woken_again`). An accept with nothing waiting answers `Nothing` whatever the room: there is no connection to refuse, and netd no longer says it refused one. `process_pending` returns the room the pass ends with and `serve_piped_listeners` takes it, so serving before the pending work is freed does not compile; the comment that stated the order goes. `netd_refused_accept` loses its pipes phase, which the host suite now carries; it keeps the room phase, the one arm for `main.rs`'s room wiring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its red is the harness's quiet guard, which is no verdict, and its green can go red on a correct netd: a hasty peer's handshake left in `SynReceived` shuts port 22 until smoltcp retransmits, and the test's reset text and `sshd: ` prefix predicate decide the outcome on timing. The listener's wake rule it guarded is now whole in `listen.rs`, where the host suite reddens on every mutation of it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
std's accept reads the wake before it reaches netd, so a failure in between spends a wake netd still counts as held. Filed under the network-stack track. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review round 2 of Round 1 BLOCKERs
BLOCKER
NOTE
REMOVE
The brief's questions
SEND BACK |
`wake` blocked on `read_nonblock` against a poller timeout, and the two connects passed a 30s deadline: three clocks inside a test whose verdict a lost wake must fail by hanging the harness, not by racing a guest clock against netd. `wake` now blocks on `listener.notify.read`, refusing `Ok(0)` by name (netd closed the listener), and both connects pass `timeout_ms: 0` (`main.rs:1181` sets no deadline for that). `WITHIN`, `Duration`, `await_until` and `READABLE` go with them, and so does the doc's claim that netd_refused_accept's clocks are liveness guards: it has none now. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`settle` must run before `room` and `woken` are read, because it is what relists a socket its peer reset: skip it under `room` false and a reset socket never listens again while room is out, and skip it under `woken` true and a socket reset while its owner holds a wake stays closed until the accept reaches it. Both orders passed every committed test, because `accept` calls `settle` again on its own path and papered over `wake`'s skip. `a_peer_that_resets_before_it_is_taken_frees_the_port` now calls `wakes` with no room, which only reaches the next peer's SYN if `wake` relisted the port; `a_wake_spent_on_a_reset_connection_announces_the_next` now sends the next peer's SYN before the accept, so the accept's own `settle` cannot mask a `wake` that left the socket closed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`handle_tcp_accept_piped` looks up the listener once and holds no borrow of it across the intervening inserts, so the second lookup before the replacement handle is written cannot fail. The `if let` treated that as a skippable case; `expect` says why it never is. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Review round 3 of Gate at
|
| part | lines | net |
|---|---|---|
production (listen.rs +88, main.rs +41 −44) |
+129 −44 | +85 |
tests (listen/tests.rs +314, netd_refused_accept.rs +130, toyos.rs +26 −3) |
+470 −3 | +467 |
| issues | +59 | +59 |
Round 3 is +17 −24. The production growth is unchanged from round 2, which accepted it.
REMOVE
- PR body, item 5 — This narrates the test's past waits (a 20 s poller wait and 30 s connects), which never existed on
main. - PR body, item 6 — "both orders passed every committed test …" and the "now asks / now sends" sentences are review chronology. The
m-roomandm-wokenrows already record what the item pins. - PR body, the paragraph "
m-roomandm-wokenare round-3 verification patches … the two the review named" — this is review chronology. - PR body, the "round-2 result at
ac948e6a" column, including the "superseded" cells — false at this head. - PR body, "Fast tier at
ac948e6a: 395 passed, 2 failed …" — false at this head. - PR body, the bold paragraph "All of the above guest numbers are round-2's … The orchestrator's guest run list for this head …" — false at this head.
LAND AFTER NAMED CHANGES
…ytes goes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in #537, #553 (every kernel panic halts, usbd and poison deleted, the no-kernel-threads track), #559 and #561. The kernel now spawns one thread outside the actuator build, klogd: this branch deleted iod and main deleted usbd, and neither side's replacement is a kthread. Conflicts, each resolved against both sides' hunks: - kernel/src/drivers/nvme.rs, kernel/src/iod.rs (modify/delete): deleted. Main's changes to them were the `mm::policy::MmioPolicy` import rename and dropping `OnPanic::Recover` from iod's spawn, adaptations to code this branch removes; neither carries behaviour to move elsewhere. - kernel/src/sched/kthread.rs: main's row table without panic policy. MAX_KERNEL_TASKS is 1 (klogd), and 2 + MAX_LOG_SHARDS in the actuator build (klogd, lognest, one logstorm per shard, which can run in one boot). - kernel/src/main.rs: neither `usbd::start()` nor `iod::start()`; main's panic handler without recovery, this branch's storage phase. - kernel/src/quiesce.rs: main's header, which names no kernel thread. - kernel-loom/src/lib.rs: neither `poison` (main) nor `durability` (here). - tests/toyos.rs: `heap_ceiling_bounds` (main's rename) without `cache_eviction` (deleted here); `blocked_dump` and `klogd_hosted` ask for klogd alone; `klogd_panic_halts` is main's, whose usbd arm and recover row are gone with usbd and poison. - issues/build/a-lane-s-tap-socket-path-outgrows-sun-len-on-the-dev-host.md: main's body and `kind: tooling`, this branch's two-shapes measurement; `status: open`, since no disabled row names `lan_mdns_answer` on either side and `--known-red` answers NO, so the quarantine paragraph goes. Beyond the conflicts: toyos-inventory gets the `description` main's hostws gate now requires of every workspace package, and the no-kernel-threads track loses K5, which this branch meets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
The branch continues from a worktree made from main, so main is this merge's first parent and every hash of wt/toyos-install stays reachable. netd takes main's side whole: #559 landed the listener fix and its machine test on its own, which supersedes this branch's copy of listen.rs, its tests and its main.rs wiring. The listener issue takes main's text, which names its owner and `listen::settle`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6rME2DoqwjcYFStYHHY4j
What changed, and why
1. A listener's socket that left
Listenis handed over or listens again (userland/netd/src/listen.rs). A netd listener is one smoltcp socket that becomes the connection it accepts. netd woke the owner only when a pass saw the socketEstablished. A peer whose handshake ACK and FIN land in one pass takes itSynReceived→CloseWait, and one whose ACK and RST do takes it toClosed; neither is ever seenEstablished. The owner was never woken, the socket never listened again, and the port reset every later SYN with nothing logged.listen::Listeningis now the one rule that the pass and accept both use.EstablishedandCloseWaithold a connection, andClosedlistens again.settleis the one function that reads the socket's state, and it writes: it puts aClosedsocket back intoListen.2. The pass writes the wake it is handed.
Listening::wake(socket, room)returns the bytes to write and records the wake it returns.main.rsreads no socket state and holds no wake bookkeeping. Recording the wake before the write is sound because a pass whose owed write fails ends the listener.3. A refused accept spends its owner's wake, and the wake is owed again only when there is room. The room refusal and the missing-pipes refusal returned before the accept spent the wake. The owner had already read its wake byte, but netd still counted it as held. A std server refused for room would then block in its next
acceptfor the rest of the boot.Listening::accept(socket, room, pipes)takes the pipes the request carried as anOptionand answersAccept::{Take(pipes), NoPipes, NoRoom, Nothing}after it has spent the wake, so no refusal can skip the spend. Room is part of the wake condition: without it, each accept refused for room would be answered by a new wake. An accept with nothing waiting answersNothingwhatever the room, since there is no connection to refuse.4. The wake is served with the room the pass ends with.
process_pending, which frees room when a pending connect ends, returns that room, andserve_piped_listenerstakes it as a parameter. Serving before the pending work does not compile.5. A silent skip becomes a named invariant.
handle_tcp_accept_pipedlooked up its listener withif let Some(pl) = ...a second time, after nothing had removed it frompiped_listenerssince the first lookup.expectnow says why that lookup cannot fail.6. Issues.
issues/hardware/a-handshake-nobody-finishes-holds-a-listeners-port-shut.mdnames its owner.issues/hardware/an-accept-that-never-reaches-netd-strands-its-listener.mdrecords the client half of the same strand: std's accept reads the wake before it reaches netd, so a failure in between spends a wake netd still counts.issues/design-debt/toyos-has-its-own-network-stack.mdnames all three listener defects as its own, witha-connect-between-two-accepts-is-reset.md.Gates
At
dbd8b021, measured before the one-line deletion of an unreachable match arm innetd_refused_accept.rs(97eaa363, which builds every guest binary:cargo test --test toyos-build -- --listexit 0).cargo run -- --ci host(50 steps, clippy with warnings denied among them;userland/netd's 19 host tests, 6 of themlisten::tests)cargo run -- --build-onlycargo test --test toyos-build -- --list: builds every guest binary,netd_refused_acceptamong them, checks registration and boots nothingnetd_refused_acceptat head, run by the orchestratorlan_mdns_answer(a harness socket path over SUN_LEN on macOS, not this diff)host, run 36355783543Negative controls and the oracle
Every patch is applied as a checked patch and not committed. Each mutated tree was shown to build: netd for the host (
cargo test --no-run) and the whole image (cargo run -- --build-only), both exit 0. The tree was then restored at the same head.Host arms:
netd's host suite,cargo test --target <host>inuserland/netd.host-base-ruleListeninggiven the base's rule whole:Establishedonly, no re-listen, room ignored by the wake, refusals before the spendlisten::tests, among thema_peer_that_closes_with_its_last_ack_is_a_connection: "a connection the peer half-closed was never announced";a_peer_that_resets_before_it_is_taken_frees_the_port: "the port answered the next peer … Rst";an_accept_refused_for_its_pipes_is_woken_again: "an owner refused for its pipes was never woken again"host-m1-closewait-not-a-connectionCloseWaitread likeListena_peer_that_closes_with_its_last_ack_is_a_connection: "a connection the peer half-closed was never announced"host-m2-closed-not-relistenedClosednot listening againa_peer_that_resets_before_it_is_taken_frees_the_port: "the port answered the next peer … Rst"host-m3-accept-keeps-the-wakean_accept_refused_for_room_is_woken_again_when_room_returns: "an owner refused for room was never woken again";an_accept_refused_for_its_pipes_is_woken_again;a_wake_spent_on_a_reset_connection_announces_the_nexthost-wake-unrecordedself.woken |= oweddeleteda_finished_handshake_is_owed_one_wake: "a connection its owner holds a wake for was announced twice"host-no-room-nothing-waitsNoRoomwith nothing waitingan_accept_refused_for_room_is_woken_again_when_room_returns: "an accept with nothing waiting was refused for room"refused-room-unowedan_accept_refused_for_room_is_woken_again_when_room_returns: "the owner was woken for a connection there is no room to take"refused-room-keeps-wakean_accept_refused_for_room_is_woken_again_when_room_returns: "an owner refused for room was never woken again"refused-pipes-firstan_accept_refused_for_its_pipes_is_woken_again: "an owner refused for its pipes was never woken again"m-roomwake's order toroom && settle(..) && !self.wokena_peer_that_resets_before_it_is_taken_frees_the_portatlisten/tests.rs:263: "the port answered the next peer … Rst"m-wokenwake's order to!self.woken && settle(..) && rooma_wake_spent_on_a_reset_connection_announces_the_next, panicking insidesyn(listen/tests.rs:196): "a SYN from 5002 was answered … Rst"Machine arms, run by the orchestrator at
dbd8b021; each patch applied as a checked patch and reversed clean.refused-revertuserland/netdback toorigin/mainrefused-room-keeps-wakerefused-room-unowed:57: "netd woke its owner for a connection there is no room to take"pass-room-trueroomreplaced bytrue:57: "netd woke its owner for a connection there is no room to take"accept-room-trueroomreplaced bytrue:45: "an accept with every connection taken"Independent oracle. smoltcp's own
Interfaceand TCP state machine, driven over a hand-played wire (listen/tests.rs): the states are smoltcp's, not ours. RFC 9293 §3.10.7.4 ("Eighth, check the FIN bit"): a FIN in SYN-RECEIVED enters CLOSE-WAIT.🤖 Generated with Claude Code