feat(uring): add the thread-per-core worker skeleton - #3014
Conversation
PR 2 of the io_uring plan on #2875: the blocker-flush milestone. moq-uring grows from an empty spike crate into the worker the relay will eventually pin per core: - Worker: a SINGLE_ISSUER | DEFER_TASKRUN | COOP_TASKRUN ring with a hard Linux 6.12 floor (probed via the MIN_TIMEOUT feature bit; a legible error names the kernel and the container-seccomp caveat, and there is no fallback path). - Parking: a futex word per worker. Remote wakes are an atomic store plus one futex(2) wake only while the worker is parked (a FUTEX_WAIT SQE armed on the word); the awake fast path is syscall-free. - Timers: a userspace heap the worker sweeps; the earliest deadline rides io_uring_enter as an absolute CLOCK_MONOTONIC timeout (ABS_TIMER), so timers cost zero SQEs. Handle implements moq_net::Timers, so moq_net::runtime::Deadline works on the worker as-is. - Local spawn: kio::Tasks drives !Send futures; its slot wakers are thread-safe and funnel into the futex word. - udp::Socket: multishot recvmsg from a registered provided-buffer ring consumed incrementally (IOU_PBUF_RING_INC) with UDP_GRO, and pooled sendmsg with an explicit UDP_SEGMENT cmsg per send. Packets borrow the receive pool (drop = release = backpressure); send staging buffers are owned by id and released on completion, the shape SENDMSG_ZC needs later. - Validation: a raw-quiche echo (tests/echo.rs) plus the echo_quiche ablation benchmark (multishot/GRO/GSO toggled one at a time), both kernel-gated to skip loudly below 6.12. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first compile and run of the PR-2 skeleton surfaced four issues: - io-uring 0.7 names the BUF_MORE accessor cqueue::buffer_more, not buf_more. - Two borrow-check failures: decode_addr sliced by family.len() while family was mutably borrowed, and on_recv_multi needed a single reborrow of the RefMut guard so rx.bufs and rx.hdr split as disjoint fields. - The bench's #[path] include of tests/support/quiche.rs sat inside the inline linux module, so the OS had to resolve .. through the nonexistent benches/linux directory. Hoisted to the crate root. - echo_streams read 64 KiB then echoed it back with ?, but quiche's stream_send returns Err(Done) once the connection send capacity (cwnd at first) hits zero, killing the server mid-handshake-rampup. Reads are now bounded by stream_capacity so the echo write always fits; a capacity-blocked stream stays readable and retries after the next flush/receive round. An empty fin write is safe at zero capacity, per quiche's stream_do_send. Plus rustfmt over the cfg(linux) code (macOS rustfmt skipped it) and an intra-doc link fix (set is a trait method, not inherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B59URZxYVSkEBaZQU6ck5G
The ablation benchmark killed the receive path within seconds on the all-on config: the multishot recvmsg completed with EFAULT. The mechanism, confirmed by tracing buffer state at failure (remainders of 114 and 206 bytes against a 208 byte recvmsg header): with incremental consumption the kernel releases a provided buffer only when it hits exactly zero bytes left, and io_recvmsg_prep_multishot fails with EFAULT the moment the selected buffer's leftover tail is smaller than the recvmsg header (16 bytes + msg_namelen + msg_controllen). Sooner or later a tail lands in that fatal window, so INC fundamentally cannot back a multishot recvmsg; it was designed for byte-stream recv. The echo test never cycled buffers enough to hit the window, which is what the sustained benchmark is for. Receive now uses a classic provided-buffer ring: one whole buffer per completion, each sized for a worst-case GRO coalesce plus the header (default 16 buffers). Same Packet borrowing and backpressure; the incremental offset bookkeeping is gone. The bench that caught this is the regression test: it now survives sustained transfer in every ablation. Loopback numbers (1 MiB echoed per iteration): all-on 3.21ms, oneshot 3.59ms, no-gro 5.49ms, no-gso 10.86ms, all-off 22.79ms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B59URZxYVSkEBaZQU6ck5G
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 53ffbc8319
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| /// Send `buf[..len]` to `to` as datagrams of `segment` bytes (the last may | ||
| /// be short). Fire-and-forget: the buffer returns to the pool when the | ||
| /// kernel completes, and a failed send surfaces on the next pool acquire. | ||
| pub fn send(&self, mut buf: TxBuf, len: usize, to: SocketAddr, segment: usize) -> io::Result<()> { |
There was a problem hiding this comment.
TxBuf records its owning socket, but send accepts any TxBuf and creates the release lease from self.shared. If a caller acquires from socket A and sends through socket B, completion returns A's buffer ID to B while permanently removing it from A. B can then contain duplicate or out-of-range IDs, causing a later acquire to panic or hand out two mutable handles to the same allocation. Make the buffer's socket determine the send target, or reject mismatched ownership.
AGENTS.md reference: AGENTS.md:L158-L162
Useful? React with 👍 / 👎.
| pub fn handle(&self) -> Handle { | ||
| Handle { | ||
| shared: self.shared.clone(), | ||
| } |
There was a problem hiding this comment.
A cloned Handle owns the same Rc<Shared> as the worker, so after drop(worker) the retained handle can still successfully call udp, spawn, and create timers. There is no worker left to submit or dispatch these operations, and SockShared::worker.upgrade() also succeeds because the handle itself keeps Shared alive, leaving receives and tasks pending forever while sends report success. Encode the worker lifetime in the handle or track the stopped state and reject further operations.
AGENTS.md reference: AGENTS.md:L158-L163
Useful? React with 👍 / 👎.
| let sin6 = unsafe { name.as_ptr().cast::<libc::sockaddr_in6>().read_unaligned() }; | ||
| Some(SocketAddr::from((sin6.sin6_addr.s6_addr, u16::from_be(sin6.sin6_port)))) |
There was a problem hiding this comment.
For IPv6 link-local or scoped multicast traffic, the kernel supplies the required interface in sin6_scope_id, but this tuple conversion constructs a SocketAddrV6 with scope ID zero and also discards flow information. Packet::from() therefore cannot be used as a reply destination on multi-interface hosts, commonly producing an unreachable route or selecting the wrong interface. Construct SocketAddrV6 explicitly with the decoded scope ID.
Useful? React with 👍 / 👎.
| if self.shared.config.gso { | ||
| send_one(&shared, &self.shared, &lease, base, len, to, Some(segment as u16))?; |
There was a problem hiding this comment.
When GSO is enabled, a segment greater than u16::MAX can pass the current validation and is silently truncated here. For example, a roughly 64 KiB payload with segment = 65_537 is validated as one segment but encoded as a one-byte UDP_SEGMENT, producing tens of thousands of implied segments and an error instead of the requested datagram. Validate the representable range before staging the operation or use a segment-size newtype.
AGENTS.md reference: AGENTS.md:L158-L162
Useful? React with 👍 / 👎.
| .map_err(|err| match err.raw_os_error() { | ||
| Some(libc::ENOSYS) | Some(libc::EPERM) | Some(libc::EACCES) => Error::Unsupported(format!( |
There was a problem hiding this comment.
On kernels old enough not to recognize one of the requested setup flags, io_uring_setup returns EINVAL before the MIN_TIMEOUT feature check can run. This mapping reports that as Error::Io, although the documented contract and the tests treat an old kernel as Error::Unsupported; fallback callers will therefore fail instead of selecting the tokio stack, and the kernel-gated tests panic rather than skip on those systems.
Useful? React with 👍 / 👎.
1 participant
Summary
PR 2 of the io_uring plan on #2875 (#2875 (comment)), stacked on #3007: the moq-uring worker skeleton, the blocker-flush milestone. The crate grows from the empty M3 spike into:
Worker: one per pinned thread, owning aSINGLE_ISSUER | DEFER_TASKRUN | COOP_TASKRUNring. Hard Linux 6.12 floor, probed via theMIN_TIMEOUTfeature bit;Worker::newrefuses older kernels with a legible error naming the running kernel and the container-seccomp caveat. No fallback path by design.futex(2)wake only when the worker is actually parked (aFUTEX_WAITSQE armed on the word); wakes while awake are syscall-free. All wakers the worker mints (main future, task slots, timer waiters) funnel into this word.io_uring_enteras an absolute timeout (ABS_TIMER), so timers cost zero SQEs, per the monoio/glommio survey in the plan.Handleimplementsmoq_net::Timers, somoq_net::runtime::Deadlineworks on the worker unchanged (validating the PR-1 trait shape against a non-tokio runtime).kio::Tasksdrives!Sendfutures; its slot wakers are already thread-safe, which sidesteps the classic Rc-waker unsoundness of local executors.udp::Socket: RX is one persistent multishotrecvmsgper socket from a registered provided-buffer ring (one worst-case-sized buffer per completion) withUDP_GRO; aPacketborrows the pool and its drop is both release and backpressure. TX issendmsgwith an explicitUDP_SEGMENTcmsg on every send (never the socket default, cf. tokio-quiche: always set UDP_SEGMENT cmsg when calling sendmsg() cloudflare/quiche#2060), staged in a pool of buffers owned by id and released on completion, the shapeSENDMSG_ZC's deferred-reclaim NOTIF model slots into later.gro/gso/multishottoggles exist for the ablation matrix.Worker::dropsync-cancels and reaps terminal CQEs before any of it frees, and leaks rather than frees if the kernel refuses to finish.tests/echo.rsruns a raw-quiche echo over the worker (handshake, 512 KiB each way, quiche's own timeout driving the timer path, server as a spawned task), in both all-on and all-off configurations.benches/echo_quiche.rsis the ablation matrix (just rs bench-echo). Both skip loudly below the kernel floor, which includes GitHub-hosted runners (ubuntu-24.04 is kernel 6.8).Design amendment vs the plan: no
IOU_PBUF_RING_INCThe plan called for incremental provided-buffer consumption (few large buffers matching GRO's 100B-64KB completion variance). The ablation benchmark disproved it within seconds of sustained load: incremental consumption cannot back a multishot
recvmsg. The kernel releases an INC buffer only when it reaches exactly zero bytes left, andio_recvmsg_prep_multishotfails the receive withEFAULTthe moment the selected buffer's leftover tail is smaller than the recvmsg header (16 B +msg_namelen+msg_controllen= 208 B here). Sooner or later a tail lands in that fatal(0, 208)window; two independent failures showed remainders of 114 and 206 bytes. INC is a byte-streamrecvfeature. RX is now a classic provided ring: one whole buffer per completion, each sized for a full GRO coalesce plus the header (default 16 x ~64 KiB), samePacketborrowing and backpressure. The echo test alone never cycled buffers enough to hit the window, which is exactly what the sustained benchmark was for.Validation results (Linux 7.1.3, loopback, 1 MiB echoed per iteration)
GSO is the dominant win, GRO second, multishot receive a modest but real gain; the ordering matches the M0 syscall-level
bench-udpmatrices.COOP_TASKRUNcombines fine withDEFER_TASKRUNon this kernel (ring setup succeeds and everything runs with both flags).just checkandjust testpass (7/7 moq-uring tests, including the kernel-gated echo in default and fully-ablated configurations; full changed-scope suite 3485 passed).cargo fmtran on Linux so the cfg'd code is formatted; clippy + rustdoc clean.Public API
New (experimental,
publish = false):Worker,Config,Handle,Timer,Error, andudp::{Config, Socket, Packet, TxBuf}.Handleimplementsmoq_net::Timers. No changes to any published crate; no wire changes, so no draft updates.(Written by Claude Fable 5)
🤖 Generated with Claude Code