diff --git a/.claude/skills/verify-reconnect/SKILL.md b/.claude/skills/verify-reconnect/SKILL.md index ef1b52da40c..8f5ac127e8a 100644 --- a/.claude/skills/verify-reconnect/SKILL.md +++ b/.claude/skills/verify-reconnect/SKILL.md @@ -26,9 +26,10 @@ Each rule is an INVARIANT; the harm says why it matters. All but `NoOrphan`, `Re ```sh quest gate just rs tla moq-net/verify/reconnect/Handover.tla quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartWait.cfg +quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartHandoff.cfg ``` -The first checks every rule over four sessions: about 1.4M distinct states, under a minute. The second checks the same rules under `Restart::Wait` over three sessions: about 300k states. Both pass at the sync commit. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed. +The first checks every rule over four sessions: about 1.4M distinct states, under a minute. The second and third check the same rules under `Restart::Wait` and `Restart::Handoff` over three sessions: about 300k states each. All pass at the sync commit. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed. ## Open findings @@ -40,7 +41,7 @@ Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on th ## Coverage -Sync commit: `bd617cc1eebc93e0c76a362cc83f2f3852c50244`. Run `git log bd617cc1ee.. -- rs/moq-net/src/reconnect.rs rs/moq-net/src/reconnect/ rs/moq-tokio/src/connection.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. +Sync commit: `c3682eae32c19362b99292d2a1c2ddb80679dd2f`. Run `git log c3682eae32.. -- rs/moq-net/src/reconnect.rs rs/moq-net/src/reconnect/ rs/moq-tokio/src/connection.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. | Code | Model | | --- | --- | @@ -48,10 +49,10 @@ Sync commit: `bd617cc1eebc93e0c76a362cc83f2f3852c50244`. Run `git log bd617cc1ee | `rs/moq-net/src/reconnect.rs` `established`, and `walk_failed` ending the first walk | `Established`, `Failed`, `Abandon` | | `rs/moq-net/src/reconnect.rs` `connected`, `fallback`, `failed`, `probed` | `Connected`, `Failed`, `ProbeFails`, `ProbeGetsThrough` | | `rs/moq-net/src/reconnect.rs` `end`, `closed`, `done`, `Predecessor` | `EndGoaway`, `EndClosed`, `Done`, `Retire`, `IdleClose` | -| `rs/moq-net/src/reconnect.rs` `Restart::Wait` | `LiveEnds` with `Restart = "wait"` (`RestartWait.cfg`) | +| `rs/moq-net/src/reconnect.rs` `Restart::Wait`, `Restart::Handoff` | `LiveEnds` and `WaitOut` with `Restart = "wait"` (`RestartWait.cfg`) and `"handoff"` (`RestartHandoff.cfg`) | | `rs/moq-net/src/reconnect/redirect.rs` `Redirect::target` | `Refused` and `Endpoints` | | `rs/moq-tokio/src/connection.rs` `run`, `Shared::connected`, `migrating`, `retired`, `offline`, `end` | `Connected`, `Apply`, `OnMigrating`, `OnRetired`, `OnOffline`, `OnDone` | ## Not covered -One configured address, so the walk across several, `ATTEMPT`, and `Verdict::Settled` and `Unusable` are left out. So are the retry window and `Failure::Exhausted`, tokens, connecting once, fixed targets, and redirect-loop scoring (`Redirects`): a session's healthiness is chosen freely instead, which covers a looped one. Backoff delays and jitter are left out; every deadline can pass in any order. `Connection::close`, `Monitor`, bandwidth forwarding, and the QUIC fresh-port retry are left out, and a session's subscriptions reduce to whether the predecessor is idle. A pass covers four sessions under `Restart::Redial` and three under `Restart::Wait`, with two endpoints a GOAWAY may name and one it may not. The tests in `reconnect/tests.rs` and `moq-tokio`'s connection tests own the timing. +One configured address, so the walk across several, `ATTEMPT`, and `Verdict::Settled` and `Unusable` are left out. So are the retry window and `Failure::Exhausted`, tokens, connecting once, fixed targets, and redirect-loop scoring (`Redirects`): a session's healthiness is chosen freely instead, which covers a looped one. Backoff delays and jitter are left out; every deadline can pass in any order. `Connection::close`, `Monitor`, bandwidth forwarding, and the QUIC fresh-port retry are left out, and a session's subscriptions reduce to whether the predecessor is idle. A pass covers four sessions under `Restart::Redial` and three under `Restart::Wait` and `Restart::Handoff`, with two endpoints a GOAWAY may name and one it may not. The tests in `reconnect/tests.rs` and `moq-tokio`'s connection tests own the timing. diff --git a/doc/bin/relay/cluster.md b/doc/bin/relay/cluster.md index b0ea559f5b3..3731435500d 100644 --- a/doc/bin/relay/cluster.md +++ b/doc/bin/relay/cluster.md @@ -216,9 +216,11 @@ subscriptions go elsewhere when another route serves them, and an existing one moves to another route of the same stream if there is one, or else finishes on the draining peer. The drain is that relay's own state, so it still advertises the routes onward at their real cost. A -relay keeps its session to a peer draining to restart until that peer closes -it, and redials only then, so it reaches the replacement rather than the -draining process. +relay whose session to a peer draining to restart has lived past +`backoff.healthy` redials it once, at once, which reaches a replacement serving +beside the draining process ([handoff](/bin/relay/config#handoff)). A session +drained before that, such as the redial when the draining process answered it +itself, stays until that process closes it, and the relay redials only then. A route's price is the publisher's production cost plus the link costs it crosses. A live publisher seeds 0, and caching never changes it. diff --git a/doc/bin/relay/config.md b/doc/bin/relay/config.md index 3c3906b6cd7..8330db7a7c6 100644 --- a/doc/bin/relay/config.md +++ b/doc/bin/relay/config.md @@ -74,7 +74,8 @@ predecessor's QUIC connections, so clients reconnect at once instead of waiting out the idle timeout. Keep it secret, and give it to one relay only: any holder resets every other holder's connections, so live relays behind one address would reset each other's clients unless the load balancer routes by -connection ID (`[listen.lb] id`, QUIC-LB). Relays on different addresses gain nothing from sharing +connection ID (`[listen.lb] id`, QUIC-LB), as the two relays of a +[handoff](#handoff) do. Relays on different addresses gain nothing from sharing it, since a client accepts a reset only from its relay's address. Create one with `openssl rand -hex 32 > reset.key`. @@ -411,6 +412,45 @@ Its `ending` count distinguishes sessions already closed before the deadline but still reporting their end. The live tally separates open sessions from that cleanup, and the summary's elapsed time stays the same for late observers. +## \[handoff] + +```toml +[handoff] +socket = "/run/moq-relay/handoff.sock" # Hand every socket to the relay that replaces this one. Linux only. +supervise = false # Run under a stand-in main process, for a systemd restart. +``` + +With `socket` set, a starting relay first connects to it. When a relay already +serves there, the new one takes over every listening socket the old one holds +instead of binding its own: the QUIC port, `[listen.tcp]`, `[listen.unix]`, +`[web]`, and `[internal]`. A listener whose address changed binds +afresh. Once the new relay is ready it takes every new QUIC handshake, and the +old one stops accepting and drains as on SIGTERM ([Shutdown](#shutdown)): its +sessions stay with it until they leave or `drain_timeout` passes. Cluster peers +redial once on that GOAWAY and reach the new relay, so the drain does not wait +on them. A session that stays on the old relay anyway, such as a client that +ignores GOAWAY, loses the routes it shared with those peers once they leave. +Either relay exiting leaves the other's sessions alone, and if one dies without +closing its sessions, the other answers their packets with a stateless reset, +which their clients accept when both relays share `[listen.reset] secret`. + +The QUIC port is handed on as a reuseport group with room for two relays, each +steering its own connections by connection ID, so handoff needs +`[runtime] workers` (tokio or io\_uring) and halves the most workers a relay +runs to 128. The new relay serves the old one's worker count, logging a warning +when the config asks for another. A relay still serving beside the one it +replaced refuses a third until that one exits. A relay dialing static or API +cluster peers needs `[connect] bind` on port 0: their shared dial socket on a +fixed port is bound at startup and not handed on. Packets on the QUIC port that +are not QUIC, such as WebRTC, go to the newest relay, so a WebRTC session on the old +relay ends at the handoff. + +Both relays name the same node while the old one drains. Each counts the +sessions it serves, and the new one holds back its [stats](#stats) broadcast +until the old one exits. Both must run as the same user. Memory and file +descriptors double while both run. [Production](/setup/prod#upgrades) shows the +systemd unit settings that make a restart a handoff. + ## \[log] ```toml diff --git a/doc/concept/stats.md b/doc/concept/stats.md index 48079a28fa8..34a6b013db5 100644 --- a/doc/concept/stats.md +++ b/doc/concept/stats.md @@ -186,7 +186,10 @@ generate more stats. The broadcast stays announced for the producer's life. Each announcement carries a fresh [instance](/concept/moq-lite#publisher-instances) on its route, so a restarted node is a new broadcast at the same path: it replaces the -old one, and no relay serves it groups cached under the old instance. A reader +old one, and no relay serves it groups cached under the old instance. A relay +taking over through [handoff](/bin/relay/config#handoff) announces its broadcast +only once the relay it replaced exits, so a node has one instance at a time; +its counters run from its start, and its first publish carries them. A reader treats each instance as its own set of counters. The instance rides moq-lite 07 announcements; over anything else the receiving session mints one per announcement, so a reader sees the same change as an end and a start at the path. diff --git a/doc/setup/prod.md b/doc/setup/prod.md index f0b871e3b5f..61971e4dd0b 100644 --- a/doc/setup/prod.md +++ b/doc/setup/prod.md @@ -61,6 +61,56 @@ sudo sysctl --system macOS uses `kern.ipc.maxsockbuf`; Windows sizes each socket directly. +## Upgrades + +On Linux a relay upgrades without cutting its sessions: the new process takes +over the old one's sockets and serves every new session while the old one +drains (see [`[handoff]`](/bin/relay/config#handoff)). Set `[runtime] workers`, +then make `systemctl restart moq-relay` the upgrade with the drop-in the package +installs as `/usr/share/moq-relay/handoff.conf`: + +```bash +sudo mkdir -p /etc/systemd/system/moq-relay.service.d +sudo cp /usr/share/moq-relay/handoff.conf /etc/systemd/system/moq-relay.service.d/ +sudo systemctl daemon-reload +``` + +The relay running when you install it started without a stand-in, so the first +restart after that cuts its sessions as before; every restart after it hands +off. + +```ini +[Service] +ExecStart= +ExecStart=/usr/bin/moq-relay --handoff-supervise /etc/moq-relay/relay.toml +Environment=MOQ_HANDOFF_SOCKET=/run/moq-relay/handoff.sock +KillMode=process +RestartKillSignal=SIGUSR2 +RuntimeDirectoryPreserve=restart +``` + +`--handoff-supervise` keeps a stand-in as the unit's main process. A restart +sends it SIGUSR2, and it passes that on and exits at once, leaving its relay to +drain while the next relay takes over and reports ready. A stop sends SIGTERM, +which the stand-in passes to its relay, as it does SIGINT and SIGUSR1 (a heap +profile's). A relay left draining is not signaled again and exits at the end of +its own drain, and a restart before then fails until it has, which +`Restart=on-failure` retries. systemd logs that the old relay "remains running +after unit stopped": that is the drain. + +If the next relay never takes over, such as on a bad config or a crash at +startup, the old relay keeps serving outside the unit until `drain_timeout` +passes, then drains and exits like any other. Until then `systemctl stop` +cannot reach it and a fresh start cannot bind its port. `systemctl status +moq-relay` lists it among the unit's processes, and the journal shows the next +relay's error; fix that and restart to hand off to it, or stop the old relay +sooner with `kill -TERM `. + +Both relays must run as the same user. `DynamicUser=` adopts an existing system +user named after the unit, so create one to keep the UID across restarts: +`sudo useradd --system --no-create-home moq-relay`. On NixOS, set +`services.moq-relay.handoff = true`. + ## Scaling out Connect relays into a [cluster](/bin/relay/cluster) to serve multiple regions diff --git a/nix/Cargo.nix b/nix/Cargo.nix index 88fe4c5e1e2..1aebfbfaf75 100644 --- a/nix/Cargo.nix +++ b/nix/Cargo.nix @@ -20399,6 +20399,11 @@ rec { "serde" ]; } + { + name = "libc"; + packageId = "libc"; + target = { target, features }: ("linux" == target."os" or null); + } { name = "moq-auth"; packageId = "moq-auth"; @@ -20425,7 +20430,6 @@ rec { { name = "moq-sock"; packageId = "moq-sock"; - optional = true; target = { target, features }: ("linux" == target."os" or null); } { @@ -20679,7 +20683,6 @@ rec { ]; features = { "_uring" = [ - "dep:moq-sock" "dep:moq-uring" "dep:web-transport-proto" ]; diff --git a/nix/modules/moq-relay.nix b/nix/modules/moq-relay.nix index a44e9d1dc81..54c4f9294d6 100644 --- a/nix/modules/moq-relay.nix +++ b/nix/modules/moq-relay.nix @@ -199,6 +199,16 @@ in description = "State directory for keys and runtime data"; }; + handoff = lib.mkOption { + type = lib.types.bool; + default = false; + description = '' + Upgrade in place: a restart starts the new relay beside the old one, which + hands over its sockets and drains instead of cutting its sessions. Serves + QUIC from one worker per core unless MOQ_RUNTIME_WORKERS says otherwise. + ''; + }; + heapDumpPrefix = lib.mkOption { type = lib.types.str; default = "/tmp/moq-relay.heap"; @@ -302,6 +312,8 @@ in "network.target" ] ++ lib.optional cfg.auth.enable "moq-auth.service"; + # A changed unit restarts rather than stopping first, so a switch hands off. + stopIfChanged = !cfg.handoff; serviceConfig = { # The relay reports ready only once it serves. @@ -330,6 +342,16 @@ in # Network capabilities for binding to ports < 1024 AmbientCapabilities = lib.optional (cfg.port < 1024) "CAP_NET_BIND_SERVICE"; + } + // lib.optionalAttrs cfg.handoff { + # Only the stand-in main process is signaled: on a restart it exits and leaves + # its relay draining, and on a stop it passes SIGTERM to the relay. A relay no + # successor takes over from drains once drain_timeout passes. + KillMode = "process"; + RestartKillSignal = "SIGUSR2"; + # The handoff socket lives here and must outlive the restart. + RuntimeDirectory = "moq-relay"; + RuntimeDirectoryPreserve = "restart"; }; environment = { @@ -343,6 +365,11 @@ in MOQ_CONNECT_TLS_INSECURE = lib.boolToString cfg.cluster.disableTlsVerify; } + // lib.optionalAttrs cfg.handoff { + MOQ_HANDOFF_SOCKET = "/run/moq-relay/handoff.sock"; + MOQ_HANDOFF_SUPERVISE = "true"; + MOQ_RUNTIME_WORKERS = lib.mkDefault "auto"; + } // lib.optionalAttrs (cfg.tls.generate != [ ]) { # TLS configuration MOQ_LISTEN_TLS_GENERATE = lib.concatStringsSep "," cfg.tls.generate; diff --git a/packaging/moq-relay/handoff.conf b/packaging/moq-relay/handoff.conf new file mode 100644 index 00000000000..736a95212e9 --- /dev/null +++ b/packaging/moq-relay/handoff.conf @@ -0,0 +1,15 @@ +# Upgrade in place: `systemctl restart moq-relay` starts the new relay beside +# the old one, which hands over its sockets and drains. Needs `[runtime] workers` +# in relay.toml. Install as /etc/systemd/system/moq-relay.service.d/handoff.conf, +# then run `systemctl daemon-reload`. See https://doc.moq.dev/setup/prod#upgrades. +[Service] +ExecStart= +ExecStart=/usr/bin/moq-relay --handoff-supervise /etc/moq-relay/relay.toml +Environment=MOQ_HANDOFF_SOCKET=/run/moq-relay/handoff.sock +# Only the stand-in main process is signaled: on a restart it exits and leaves +# its relay draining, and on a stop it passes SIGTERM to the relay. A relay no +# successor takes over from drains once drain_timeout passes. +KillMode=process +RestartKillSignal=SIGUSR2 +# The handoff socket lives here and must outlive the restart. +RuntimeDirectoryPreserve=restart diff --git a/packaging/moq-relay/nfpm.yaml b/packaging/moq-relay/nfpm.yaml index f4a76acd9f5..26da6f144f8 100644 --- a/packaging/moq-relay/nfpm.yaml +++ b/packaging/moq-relay/nfpm.yaml @@ -40,6 +40,12 @@ contents: file_info: mode: 0644 + # Opt-in: copied into moq-relay.service.d to make a restart a handoff. + - src: packaging/moq-relay/handoff.conf + dst: /usr/share/moq-relay/handoff.conf + file_info: + mode: 0644 + # Per-packager overrides for distro-specific runtime dependencies. overrides: deb: diff --git a/quest/m3/relay-restart.md b/quest/m3/relay-restart.md deleted file mode 100644 index 425edc3ccdf..00000000000 --- a/quest/m3/relay-restart.md +++ /dev/null @@ -1,69 +0,0 @@ -# [XL] moq-relay restarts without dropping sessions - -## Goal - -An operator upgrades a relay by starting the new process beside the old one -on the same UDP port. New sessions go to the new process. Existing sessions -stay with the old process until they end or its drain deadline passes. The -upgrade itself takes seconds, and a session that would have ended inside the -deadline is never cut. - -## Plan - -Decided 2026-11-09 (maintainer, `/quest-plan`): ✅ planned now, built after -downstream's batched rollouts (rejected: building it first; not building -it). m3, because a downstream deployment's rollouts wait on it: its relays -restart one at a time, and each restart waits minutes for sessions to drain, -so the drain sets the length of every rollout and every rollback. - -Prior art, read 2026-11-09: - -- Meta's Socket Takeover passes the UDP socket to the new process over a - UNIX socket, and the new process forwards packets for old connections to - the old process, keyed on the QUIC connection ID - ([Zero Downtime Release, SIGCOMM 2020](https://par.nsf.gov/servlets/purl/10211855)). -- Cloudflare's udpgrm puts both processes in one `SO_REUSEPORT` group and - routes with eBPF: new flows to the newest socket generation, existing - flows to their owner, optionally by a generation encoded in the - connection ID - ([udpgrm](https://blog.cloudflare.com/quic-restarts-slow-problems-udpgrm-to-the-rescue)). -- GOAWAY with a new URI, which moq-lite already carries, moves a client that - can reconnect. It stays the path for a session that outlives the deadline. - -Guidance: - -- Say how a new handshake reaches the new process. Nothing does today: a - client's Initial carries a CID it invented, `moq_sock::shard` steers only - by server-issued CIDs and binds its group once, QUIC-LB forwarding never - forwards long headers, and the io_uring backend refuses `listen.lb.id` - because shard steering and QUIC-LB compete for the CID bytes. Choose - between inheriting the sockets by fd passing, an eBPF steer as in udpgrm, - or reusing the shard index or QUIC-LB server ID as the generation; do not - add a third CID layout. Say whether `reset.secret` stays. Cover both - backends, `moq-tokio` and `moq-uring`. -- Decide what happens to cluster peer sessions: handed to the new process, - or re-dialed. Today a peer session does not leave on GOAWAY and holds a - drain to its bound. Measured on a staging mesh: a relay with no inbound - peer stopped about two minutes after its stop signal, and every relay - with one stopped about seven minutes after it, having waited out the full - 300 s session window that opened at the two-minute mark. So inbound peers - add about five minutes to each restart and to each wave of a batched - rollout. - Decided 2026-11-11 (maintainer, `/quest-plan`): ✅ remove that cost - here, by running the new process beside the old (rejected as an interim: - the draining relay closing inbound peer sessions once its other sessions - have left, a forced close that works against this design; a separate - quest for it). -- Two processes share one node identity for the length of a drain. Stats, - routes and announcements must count and name each session once across - them. -- Memory and file descriptors double during the overlap; measure it. -- Linux only. Integrate with systemd so the unit's restart is the upgrade. -- A crash of either process must not take the other's sessions with it. - -Public API: relay configuration for the overlap and its deadline, additive. -Wire: none expected; the connection ID's layout is local to a relay. - -## Related - -- [GOAWAY migration follow-ups](/quest/m3/goaway-migration-followups.md) - the client's side of leaving a draining session diff --git a/quest/m4/relay-handoff-routes.md b/quest/m4/relay-handoff-routes.md new file mode 100644 index 00000000000..e6f42b07c98 --- /dev/null +++ b/quest/m4/relay-handoff-routes.md @@ -0,0 +1,33 @@ +# [L] Keep cluster routes for sessions that stay on a draining relay + +## Goal + +During a [handoff](/doc/bin/relay/config.md), a session that stays on the old +relay through its drain, such as a third-party client that ignores GOAWAY, +keeps reaching the cluster, and the cluster keeps reaching it. Today the +cluster peers that dialed the old relay redial the new one at once and retire +their old session once it carries no subscription, or after `goaway.handover` +(10 s by default). From then on a broadcast published only on the old relay is +unreachable from the rest of the cluster, and a subscriber left on the old +relay loses every broadcast it reached through those peers, until the old +relay's drain ends. + +## Plan + +- Recommended: link the two relays for the overlap. The handoff connection + between them already exists; the new relay runs a moq-lite session to the + old one over a socket it carries, and each treats the other as a cluster + peer until the old one exits, so routes cross between generations as they + cross between nodes. Check first how the route rule treats two processes + that share the node's identity, since a shared hop would read each other's + routes as loops. +- Rejected: keep a peer's old session past idle until the old relay closes it. + That holds the old relay to its drain deadline again, which is what handoff + removes. + +Exercise it with a publisher that ignores GOAWAY on the old relay and a +subscriber on another node, past `goaway.handover`. + +Picked by the maintainer on 2026-11-15: ✅ keep this quest in m4 (rejected: +drop it), and ✅ the relay-restart PR landed without it (rejected: hold that PR +for this link). diff --git a/quest/m5/relay-handoff-webrtc.md b/quest/m5/relay-handoff-webrtc.md new file mode 100644 index 00000000000..1c60ba85664 --- /dev/null +++ b/quest/m5/relay-handoff-webrtc.md @@ -0,0 +1,27 @@ +# [L] Keep WebRTC flows across a relay handoff + +## Goal + +An embedder serving WebRTC on the relay's QUIC port (`Relay::webrtc`) keeps +each WebRTC session across a [handoff](/doc/bin/relay/config.md), as QUIC +sessions are kept: the old relay serves its flows until they end or its drain +deadline passes, and new flows reach the new relay. Today the reuseport filter +hashes every non-QUIC datagram onto the newest generation, so a WebRTC +session the old relay holds loses its ICE, DTLS, and RTP at the handoff. + +## Plan + +Classic BPF keeps no per-flow state, so the filter alone cannot tell the old +relay's flows from new ones. Options: + +- The new relay hands a non-QUIC datagram from a source its demux pin table + does not hold to the old relay over a socket the handoff carries, and the old + relay answers from its own member, as Meta's Socket Takeover forwards QUIC. + Recommended: no eBPF loader, and the demux already pins each source. +- An eBPF reuseport program with a map of the sources the old relay pinned. + Needs `CAP_BPF` and a loader. +- Steer by the ICE username fragment, which only STUN checks carry, not DTLS + or RTP. Rejected. + +Kept in m5 with no consumer yet (maintainer, 2026-11-14): ✅ keep it in m5 +(rejected: drop it until an embedder serves WebRTC on a relay that hands off). diff --git a/rs/moq-net/src/reconnect.rs b/rs/moq-net/src/reconnect.rs index 5a3c57e95ec..dd49e0a1c92 100644 --- a/rs/moq-net/src/reconnect.rs +++ b/rs/moq-net/src/reconnect.rs @@ -154,6 +154,16 @@ pub enum Restart { /// draining process again. The peer must close the session itself, as a relay does at /// its drain deadline. Wait, + /// Redial the same addresses once, as [`Self::Redial`] does, and keep serving on the + /// drained session meanwhile. A redial that reaches the draining process again, which + /// drains it before it is [healthy](Backoff::healthy), then serves as under + /// [`Self::Wait`] until the peer closes it. + /// + /// Right for an address that names one node which may start its replacement beside the + /// draining process, as a relay with `[handoff]` does: the one redial reaches the + /// replacement. A node that restarts in place costs a single extra session to the + /// draining process rather than one per backoff. + Handoff, } /// The whole connect policy for one peer. @@ -585,7 +595,16 @@ impl + From> Reco // Checked before the close: a GOAWAY means the session is still up, and // migrating from it is not the same as reconnecting after it died. if let Some(Poll::Ready(Ok(msg))) = live.goaway.as_ref().map(|goaway| goaway.poll(waiter)) { - if self.config.restart == Restart::Wait && msg.uri().is_empty() { + // Under `Handoff`, a session drained before it was healthy is a redial that + // reached the draining process rather than a replacement, so another would + // only reach it again. + let wait = match self.config.restart { + Restart::Redial => false, + Restart::Wait => true, + Restart::Handoff => now.saturating_duration_since(live.since) < self.config.backoff.healthy, + }; + // A one-shot policy ends on any GOAWAY instead, as `once` promises. + if wait && !self.config.once && msg.uri().is_empty() { tracing::info!("peer is restarting; serving until it closes the session"); live.goaway = None; } else { diff --git a/rs/moq-net/src/reconnect/tests.rs b/rs/moq-net/src/reconnect/tests.rs index c08da1a6807..1126634bd79 100644 --- a/rs/moq-net/src/reconnect/tests.rs +++ b/rs/moq-net/src/reconnect/tests.rs @@ -792,6 +792,106 @@ async fn a_waited_restart_redials_only_once_the_session_closes() { assert_eq!(run.events, ["offline"]); } +/// A one-shot policy ends cleanly on a GOAWAY naming no URI, as on any other, under +/// every restart policy and whether or not the session was healthy. +#[kio::test] +async fn a_one_shot_policy_ends_on_a_restart() { + for restart in [Restart::Redial, Restart::Wait, Restart::Handoff] { + let peer = url("tcp://peer:1/"); + let mut config = config(INITIAL, Some(Duration::from_secs(1))); + config.restart = restart; + config.once = true; + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let mut run = Run::default(); + let end = drive(&mut machine, &[(peer, Peer::Goaway(String::new()))], &mut run, |_| { + false + }) + .await; + assert!(matches!(end, Some(Ok(()))), "{restart:?}: {end:?}"); + assert_eq!(run.dials.len(), 1, "{restart:?} redialed"); + } +} + +/// The actions a policy asks for until it only waits, by name. +async fn settle(machine: &mut Reconnect) -> Vec<&'static str> { + let mut events = Vec::new(); + while let Ok(action) = kio::clock::timeout(Duration::from_secs(60), next(machine)).await { + events.push(match action { + Action::Migrating => "migrating", + Action::Retired => "retired", + Action::Offline(_) => "offline", + other => panic!("expected the policy to wait, got {other:?}"), + }); + } + events +} + +/// Under [`Restart::Handoff`] a healthy session's empty-URI GOAWAY redials the same address once +/// while the drained session serves, so a replacement already serving beside the draining +/// process takes over and the drained session retires once it carries nothing. +#[kio::test] +async fn a_handoff_restart_moves_to_a_replacement() { + let peer = url("tcp://peer:1/"); + let mut config = config(INITIAL, Some(Duration::from_secs(1))); + config.restart = Restart::Handoff; + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let mut run = Run::default(); + drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| { + !run.accepts.is_empty() + }) + .await; + + kio::clock::sleep(Duration::from_secs(2)).await; + let goaway = kio::clock::now(); + run.servers[0].drain().send(goaway::Goaway::new()).unwrap(); + drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| { + run.accepts.len() == 2 + }) + .await; + assert_eq!(run.accepts[1].1, goaway, "the redial waited"); + assert_eq!(run.events, ["migrating"]); + assert_eq!(settle(&mut machine).await, ["retired"]); + assert!(machine.session().is_some(), "the replacement serves"); + assert_eq!(run.dials.len(), 2); +} + +/// Under [`Restart::Handoff`] a redial that reaches the draining process again, drained before +/// it is healthy, changes nothing while the peer keeps it: no further redial. Its close then +/// reconnects like any drop. +#[kio::test] +async fn a_handoff_restart_redials_once_until_the_session_closes() { + let peer = url("tcp://peer:1/"); + let mut config = config(INITIAL, Some(Duration::from_secs(1))); + config.restart = Restart::Handoff; + let mut machine = Reconnect::new(config, vec![peer.clone()], kio::clock::now()).unwrap(); + let mut run = Run::default(); + drive(&mut machine, &[(peer.clone(), Peer::Serve)], &mut run, |run| { + !run.accepts.is_empty() + }) + .await; + + kio::clock::sleep(Duration::from_secs(2)).await; + run.servers[0].drain().send(goaway::Goaway::new()).unwrap(); + drive( + &mut machine, + &[(peer.clone(), Peer::Goaway(String::new()))], + &mut run, + |run| run.accepts.len() == 2, + ) + .await; + assert_eq!(run.events, ["migrating"]); + assert_eq!(settle(&mut machine).await, ["retired"], "the policy left the session"); + assert_eq!(run.dials.len(), 2, "redialed the draining process again"); + + run.servers.remove(1).abort(Error::GoawayTimeout); + run.events.clear(); + drive(&mut machine, &[(peer, Peer::Serve)], &mut run, |run| { + run.accepts.len() == 3 + }) + .await; + assert_eq!(run.events, ["offline"]); +} + /// A session closed for a protocol violation or a refused credential reconnects like any /// other close: the policy does not judge the error's kind. #[kio::test] diff --git a/rs/moq-net/verify/reconnect/Handover.tla b/rs/moq-net/verify/reconnect/Handover.tla index 0f9ad8f38dd..fd4f495d1e2 100644 --- a/rs/moq-net/verify/reconnect/Handover.tla +++ b/rs/moq-net/verify/reconnect/Handover.tla @@ -28,7 +28,7 @@ CONSTANTS Endpoints, \* where a dial can land Start, \* the configured target Refused, \* a GOAWAY URI `Redirect::target` refuses - Restart, \* `Config::restart`: "redial" or "wait" + Restart, \* `Config::restart`: "redial", "wait", or "handoff" MaxSessions \* sessions that connect in one behaviour Sessions == 1..MaxSessions @@ -334,18 +334,26 @@ EndClosed == \* `Phase::Live`: a GOAWAY is checked before the close. Goaway(s) == watch /\ sess[s].goaway # NoGoaway +\* `Restart::Wait` serves out a restart in place; `Restart::Handoff` does so +\* only on a session drained before it was healthy, and redials an older one +\* once, as `Restart::Redial` does. +WaitOut(s) == + /\ sess[s].goaway = "" + /\ \/ Restart = "wait" + \/ Restart = "handoff" /\ sess[s].age = 0 + LiveEnds == /\ queue = <<>> /\ ~PredDue /\ phase = "live" /\ \/ /\ Goaway(live) - /\ Restart = "wait" /\ sess[live].goaway = "" - \* `Restart::Wait`: drop the latch and serve until the peer closes. + /\ WaitOut(live) + \* Drop the latch and serve until the peer closes. /\ watch' = FALSE /\ UNCHANGED <> /\ CallerSame \/ /\ Goaway(live) - /\ ~(Restart = "wait" /\ sess[live].goaway = "") + /\ ~WaitOut(live) /\ EndGoaway(live, sess[live].goaway) \/ /\ ~Goaway(live) /\ ~sess[live].open @@ -461,7 +469,7 @@ ProbeGetsThrough == \/ /\ ~Goaway(live) /\ phase' = "dialing" /\ timer' = [timer EXCEPT !["dial"] = FALSE, !["handover"] = FALSE] - \* Its watcher moves with it, unless `Restart::Wait` dropped it. + \* Its watcher moves with it, unless `WaitOut` dropped it. /\ sess' = [Abort(sess, pred) EXCEPT ![live].sent = TRUE, ![live].watched = watch] /\ pred' = live /\ live' = None diff --git a/rs/moq-net/verify/reconnect/RestartHandoff.cfg b/rs/moq-net/verify/reconnect/RestartHandoff.cfg new file mode 100644 index 00000000000..75377d9e5a3 --- /dev/null +++ b/rs/moq-net/verify/reconnect/RestartHandoff.cfg @@ -0,0 +1,21 @@ +\* Handover.cfg's rules under `Restart::Handoff`, where a GOAWAY naming no URI +\* redials once off a healthy session and keeps one drained before it was +\* healthy serving until the peer closes it. +CONSTANTS + Endpoints = {"a", "b"} + Start = "a" + Refused = "far" + Restart = "handoff" + MaxSessions = 3 + +SPECIFICATION Spec +VIEW View + +INVARIANTS + NoStaleSession + StatusTruthful + OfflineMeansNothingServes + NoOrphan + PresenceBalanced + FirstWalkFailsFast + RedirectFollowed diff --git a/rs/moq-quic/src/cid_generator.rs b/rs/moq-quic/src/cid_generator.rs index 379177bf8f5..f310e135cbc 100644 --- a/rs/moq-quic/src/cid_generator.rs +++ b/rs/moq-quic/src/cid_generator.rs @@ -43,6 +43,8 @@ pub struct RandomConnectionIdGenerator { cid_len: usize, lifetime: Option, shard: Option, + /// How many equal generations split the shard's group; see [`Self::set_generations`]. + generations: u16, } impl Default for RandomConnectionIdGenerator { @@ -51,6 +53,7 @@ impl Default for RandomConnectionIdGenerator { cid_len: 8, lifetime: None, shard: None, + generations: 1, } } } @@ -71,10 +74,12 @@ impl Shard { (stride * count + u32::from(self.index)) as u8 } - /// Whether `cid` carries this shard's prefix and the marker. - fn owns(self, cid: &[u8]) -> bool { + /// Whether `cid` carries the marker and the prefix of this shard's position in any of + /// `generations` equal generations of the group. + fn owns(self, cid: &[u8], generations: u16) -> bool { + let period = self.count / generations; cid.len() > SHARD_MARKER_BYTE - && u16::from(cid[0]) % self.count == self.index + && u16::from(cid[0]) % period == self.index % period && cid[SHARD_MARKER_BYTE] & SHARD_MARKER == SHARD_MARKER } } @@ -127,9 +132,37 @@ impl RandomConnectionIdGenerator { "a {}-byte CID cannot carry the shard marker", self.cid_len ); + assert!( + count.is_multiple_of(self.generations), + "{count} shards cannot split into {} generations", + self.generations + ); self.shard = Some(Shard { index, count }); self } + + /// Also accept the CIDs of the shard at this one's position in each of `generations` + /// equal generations of the group, such as the members a predecessor process served. + /// + /// A group shared by successive processes gives each its own generation, so their CIDs + /// never collide. A steering filter folds a generation's packets onto the same position of + /// the one left once that generation is gone, and [`validate`](ConnectionIdGenerator::validate) + /// then accepts them, so the endpoint answers with a stateless reset (verifiable when both + /// share a reset key) rather than leaving the client to its idle timeout. Defaults to 1. + /// + /// Panics if `generations` is zero or does not divide the shard count. + pub fn set_generations(&mut self, generations: u16) -> &mut Self { + assert!(generations > 0, "a group holds at least one generation"); + if let Some(shard) = self.shard { + assert!( + shard.count.is_multiple_of(generations), + "{} shards cannot split into {generations} generations", + shard.count + ); + } + self.generations = generations; + self + } } impl ConnectionIdGenerator for RandomConnectionIdGenerator { @@ -146,7 +179,7 @@ impl ConnectionIdGenerator for RandomConnectionIdGenerator { fn validate(&self, cid: ConnectionId) -> Result<(), InvalidCid> { match self.shard { - Some(shard) if !shard.owns(&cid) => Err(InvalidCid), + Some(shard) if !shard.owns(&cid, self.generations) => Err(InvalidCid), _ => Ok(()), } } @@ -311,6 +344,35 @@ mod tests { assert!(first.validate(unmarked).is_err(), "took a CID without the marker"); } + /// Across generations, a shard takes its own position's CIDs from the other generation, which + /// reach it only once that generation is gone, and still refuses every other position's. + #[test] + fn a_generation_takes_its_position_from_the_other() { + let mut newer = sharded(5, 8); + newer.set_generations(2); + for _ in 0..32 { + newer.validate(sharded(1, 8).generate_cid()).unwrap(); + let own = newer.generate_cid(); + newer.validate(own).unwrap(); + for other in [0, 2, 3, 4, 6, 7] { + assert!( + newer.validate(sharded(other, 8).generate_cid()).is_err(), + "took position {other}'s CID" + ); + } + } + // One generation is the default: the same position's CID from the other half is refused. + assert!(sharded(5, 8).validate(sharded(1, 8).generate_cid()).is_err()); + } + + #[test] + #[should_panic] + fn generations_must_split_the_group_evenly() { + RandomConnectionIdGenerator::default() + .set_shard(0, 6) + .set_generations(4); + } + #[test] #[should_panic] fn a_shard_past_the_group_is_refused() { diff --git a/rs/moq-relay/Cargo.toml b/rs/moq-relay/Cargo.toml index afa31ee1ef2..9eb0596ab7d 100644 --- a/rs/moq-relay/Cargo.toml +++ b/rs/moq-relay/Cargo.toml @@ -61,7 +61,7 @@ uds = ["moq-tokio/uds"] # This is off by default: the relay is useful without io_uring. io-uring = ["_uring", "moq-tokio/driver", "moq-uring/quic"] # The listener machinery shared by the feature above. -_uring = ["dep:moq-sock", "dep:moq-uring", "dep:web-transport-proto"] +_uring = ["dep:moq-uring", "dep:web-transport-proto"] [dependencies] anyhow = { workspace = true, features = ["backtrace"] } @@ -111,7 +111,10 @@ url = { workspace = true, features = ["serde"] } usage-rs = { workspace = true, features = ["completions", "config"] } [target.'cfg(target_os = "linux")'.dependencies] -moq-sock = { workspace = true, optional = true } +# `geteuid`, so a handoff only trusts a relay of the same user. +libc = { workspace = true } +# The reuseport group and descriptor passing a handoff carries the sockets with. +moq-sock = { workspace = true } moq-uring = { workspace = true, optional = true } # The WebTransport CONNECT response the io_uring listener answers with. web-transport-proto = { workspace = true, optional = true } diff --git a/rs/moq-relay/src/cluster.rs b/rs/moq-relay/src/cluster.rs index 8bdb62f6db6..823e48420e0 100644 --- a/rs/moq-relay/src/cluster.rs +++ b/rs/moq-relay/src/cluster.rs @@ -863,6 +863,17 @@ pub struct Config { } impl Config { + /// Whether this relay dials a static or API peer. A static list naming only this + /// relay dials nothing, so it runs standalone rather than as a mesh that finishes + /// at once. + pub(crate) fn dials_statics(&self) -> bool { + let own = self.node.as_deref().map(PeerKey::canonicalize); + self.connect + .iter() + .any(|peer| own.as_ref() != Some(&PeerKey::canonicalize(&peer.url))) + || self.api.url.is_some() + } + /// Released spellings this config was parsed from, each paired with what replaced it. pub fn deprecated(&self) -> moq_tokio::cli::Deprecated { let mut found = moq_tokio::cli::Deprecated::default(); @@ -1041,6 +1052,12 @@ pub struct Options { /// Stats publishing on the cluster's origin. Disabled by default. pub stats: crate::stats::Config, + /// Holds the stats publisher back until this resolves, or its sender drops, so a + /// relay serving beside the one it replaces announces the node's broadcast only once + /// that one is gone. `None` publishes from the start. Counters still accumulate + /// meanwhile, and the first publish carries them. + pub stats_hold: Option>, + /// How peers are reached in place of the network, for tests on `kio::clock::sim`: /// every peer dial becomes a [`Dial`](moq_tokio::sim::Dial) the test routes. /// `None` dials over sockets. @@ -1449,6 +1466,7 @@ impl Cluster { client, peer, stats, + stats_hold, #[cfg(feature = "test-support")] network, #[cfg(all(feature = "test-support", feature = "quic"))] @@ -1497,7 +1515,13 @@ impl Cluster { // Publish through the same origin cluster peers read from. let stats_prefix = stats.reserved()?; let (stats_producer, publisher) = stats.build(origin.clone())?; - work.push(Child::new("stats publisher", publisher.run())); + work.push(Child::new("stats publisher", async move { + if let Some(hold) = stats_hold { + // A dropped sender releases it as well as a send. + let _ = hold.await; + } + publisher.run().await; + })); let stats = stats_producer.registry().clone(); let nodes = crate::nodes::Nodes::default(); tracing::info!(hop_id = %origin.hop(), "cluster initialized"); @@ -1838,15 +1862,7 @@ impl Cluster { } } - // A static list naming only this relay dials nothing, so it runs standalone - // rather than as a mesh that finishes at once. - let own = node.as_deref().map(PeerKey::canonicalize); - let statics = self - .config - .connect - .iter() - .any(|peer| own.as_ref() != Some(&PeerKey::canonicalize(&peer.url))) - || self.config.api.url.is_some(); + let statics = self.config.dials_statics(); let can_dial = statics || lan; if !can_dial { tracing::info!("no cluster peers configured; running standalone"); @@ -2303,13 +2319,16 @@ impl Cluster { // replacement. // // A GOAWAY naming no URI is a relay draining to restart. A peer URL names - // that one node, so a redial would only reach the draining process again - // and replay both announce tables there for the rest of its window. Its - // routes already rank below every other, so the session serves on until - // the node closes it, and the redial after that finds the replacement. + // that one node, so the loop redials it once: a node that hands its port to + // a replacement serving beside it (see `handoff`) is reached that way, and + // the drained session leaves within its handover. A redial that reaches the + // draining process again is drained at once, and serves on until the node + // closes it rather than redialing for the rest of its window; its routes + // already rank below every other, and the redial after that close finds the + // replacement. let reconnect = client .with_reconnect(true) - .with_restart(moq_tokio::Restart::Wait) + .with_restart(moq_tokio::Restart::Handoff) .connect(addrs.clone()); let monitor = reconnect.monitor(); let mut observation = monitor.state(); diff --git a/rs/moq-relay/src/config.rs b/rs/moq-relay/src/config.rs index d5e8772b5b5..7b6e4d89e9b 100644 --- a/rs/moq-relay/src/config.rs +++ b/rs/moq-relay/src/config.rs @@ -95,6 +95,12 @@ pub struct Config { #[serde(default)] pub internal: internal::Config, + /// Handing this relay's sockets to the relay that replaces it, so an upgrade + /// serves the new process beside the old one. Disabled unless `handoff.socket` is set. + #[usage(flatten)] + #[serde(default)] + pub handoff: crate::handoff::Config, + /// How long accepted sessions may keep running after a shutdown signal, e.g. /// "10s" or "500ms". The first signal sends every session a GOAWAY and waits /// up to this long for clients to reconnect elsewhere before force-closing @@ -321,6 +327,7 @@ impl Config { // is checked by `Cluster::new`, which embedders call directly. self.quic.validate().context("invalid [quic]")?; self.check_runtime()?; + self.check_handoff()?; self.stats.check(&self.cluster)?; #[cfg(feature = "cluster-lan")] self.check_lan()?; @@ -372,6 +379,34 @@ impl Config { Ok(()) } + /// Refuse a handoff this relay could not carry out. + fn check_handoff(&self) -> anyhow::Result<()> { + // A stand-in whose relay cannot take over would leave the old relay serving + // after every restart, and the new one failing to bind beside it. + anyhow::ensure!( + !self.handoff.supervise || self.handoff.socket.is_some(), + "handoff.supervise needs handoff.socket" + ); + if self.handoff.socket.is_none() { + return Ok(()); + } + #[cfg(not(target_os = "linux"))] + anyhow::bail!("handoff.socket is Linux-only"); + // Static and API peers share one dial socket on a fixed port, bound at startup and + // not handed on: a successor could not bind it beside the relay it replaces. + anyhow::ensure!( + !self.cluster.dials_statics() || self.connect.resolve().bind.port() == 0, + "handoff.socket cannot hand on the cluster's dial socket: give --connect-bind port 0" + ); + // The QUIC port is handed on as a reuseport group with room for a successor, + // which only a worker group binds. + anyhow::ensure!( + self.runtime.workers().is_some() || self.runtime.io_uring() || !self.listen.serves_quic(), + "handoff.socket requires runtime.workers: the QUIC port is handed on as a worker group" + ); + Ok(()) + } + /// Refuse LAN discovery this relay cannot advertise or dial. #[cfg(feature = "cluster-lan")] fn check_lan(&self) -> anyhow::Result<()> { @@ -866,6 +901,44 @@ root = ["ca.pem"] "[runtime]\nworkers = 1\nio_uring = true\n", "runtime.io_uring requires listen.bind", ); + + // A stand-in with no relay to hand over to would leave the old one serving. + refuse( + "[handoff]\nsupervise = true\n", + "handoff.supervise needs handoff.socket", + ); + #[cfg(all(target_os = "linux", feature = "_quic"))] + { + // The QUIC port is handed on as a worker group, which the shared listener is not. + let quic = "[listen]\nbind = \"127.0.0.1:0\"\n"; + let handoff = "[handoff]\nsocket = \"/run/moq-relay/handoff.sock\"\nsupervise = true\n"; + refuse(&format!("{quic}{handoff}"), "handoff.socket requires runtime.workers"); + resolve(&format!("{quic}[runtime]\nworkers = 1\n{handoff}")); + // With no QUIC listener there is no group to hand on. + resolve(&format!("[listen.tcp]\nbind = \"127.0.0.1:0\"\n{handoff}")); + // Static and API peers dial from one socket on a fixed port, which a successor + // could not bind beside its predecessor; an ephemeral port is bound per session. + let quic = format!("{quic}[runtime]\nworkers = 1\n{handoff}"); + for peers in [ + "[cluster]\nconnect = [\"https://b.example\"]\n", + "[cluster.api]\nurl = \"https://api.example/peers\"\n", + ] { + refuse( + &format!("{quic}{peers}[connect]\nbind = \"0.0.0.0:4444\"\n"), + "cannot hand on the cluster's dial socket", + ); + resolve(&format!("{quic}{peers}[connect]\nbind = \"0.0.0.0:0\"\n")); + } + // A list naming only this relay dials nothing. + resolve(&format!( + "{quic}[cluster]\nnode = \"https://a.example\"\nconnect = [\"https://a.example\"]\n[connect]\nbind = \"0.0.0.0:4444\"\n" + )); + } + #[cfg(not(target_os = "linux"))] + refuse( + "[handoff]\nsocket = \"/run/moq-relay/handoff.sock\"\n", + "handoff.socket is Linux-only", + ); } /// LAN discovery needs a QUIC port and an identity to advertise. diff --git a/rs/moq-relay/src/handoff.rs b/rs/moq-relay/src/handoff.rs new file mode 100644 index 00000000000..059e0c8ab48 --- /dev/null +++ b/rs/moq-relay/src/handoff.rs @@ -0,0 +1,1027 @@ +//! Handing a relay's sockets to the relay that replaces it. +//! +//! An upgrade starts the new relay beside the old one rather than after it. With +//! `handoff.socket` set, a starting relay first connects to that Unix socket. When +//! an older relay answers, the new one takes every listening socket the old one +//! holds by file descriptor instead of binding its own: nothing queued on them is +//! lost, and the QUIC port's reuseport group keeps the numbering its connection IDs +//! name (see [`moq_sock::shard`]). The new relay then serves, takes every new QUIC +//! handshake, and tells the old one, which stops accepting and drains: each session +//! gets a GOAWAY and stays until it leaves or the drain deadline passes. A cluster +//! peer that redials (see [`moq_tokio::Restart::Handoff`]) reaches the new relay, so the +//! drain no longer waits on inbound peers. +//! +//! The two relays share the node's identity for the length of the drain. Each +//! serves its own sessions on its own origin, so a session is counted and named by +//! the one that holds it. The new relay holds its stats broadcast back until the old +//! one exits, so the node's broadcast has one instance at a time and the old one's +//! last counters are read before the new one replaces it. +//! +//! A relay with a predecessor still serving beside it refuses a successor: the +//! group has room for two generations. Either relay exiting leaves the other's +//! sessions alone, since each holds every socket. A successor that dies while its +//! predecessor still drains hands new handshakes back to it, and the next successor +//! takes over from it. +//! +//! # Protocol +//! +//! Over the stream socket, the old relay sends a big-endian `u32` length and a JSON +//! hello, then one byte per batch of at most [`moq_sock::fd::MAX`] descriptors, +//! each batch attached to its byte, in the order the hello lists them. The new relay +//! answers with one byte once it serves, and both hold the connection open: its end +//! is how each learns the other exited. Both check that the peer runs as the same +//! user. + +#[cfg(target_os = "linux")] +use std::net::SocketAddr; +use std::path::PathBuf; + +use serde::{Deserialize, Serialize}; + +/// Handing this relay's sockets to the relay that replaces it. Linux only. +#[derive(usage_rs::Args, Clone, Debug, Default, Deserialize, Serialize)] +#[serde(deny_unknown_fields, default)] +#[non_exhaustive] +pub struct Config { + /// A Unix socket where this relay serves its sockets to the relay that replaces + /// it, and where it first looks for the relay it replaces. Unset disables handoff. + /// Needs `runtime.workers`, since the QUIC port is handed on as a reuseport group. + #[usage( + long = "handoff-socket", + name = "handoff-socket", + env = "MOQ_HANDOFF_SOCKET", + setting = "handoff.socket" + )] + #[usage(value_hint = usage_rs::ValueHint::AnyPath)] + #[serde(default, skip_serializing_if = "Option::is_none")] + pub socket: Option, + + /// Run the relay as the child of a process that stands in for it as a service's + /// main process. The stand-in passes SIGTERM and SIGINT on, forwards readiness, and + /// exits with the relay; on SIGUSR2 it exits at once and leaves the relay running, + /// which drains anyway if no successor takes over within `drain_timeout`. + /// A systemd unit with `KillMode=process` and `RestartKillSignal=SIGUSR2` then + /// restarts by starting the next relay beside this one, which takes over through + /// `handoff.socket`. Needs `handoff.socket` and a service manager's `NOTIFY_SOCKET`. + #[usage( + long = "handoff-supervise", + env = "MOQ_HANDOFF_SUPERVISE", + setting = "handoff.supervise", + bool_value + )] + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + pub supervise: bool, +} + +/// The flag that runs a stand-in, which its relay must not see again. +#[cfg_attr(not(target_os = "linux"), allow(dead_code))] +const SUPERVISE_FLAG: &str = "--handoff-supervise"; + +/// The arguments a stand-in starts its relay with: its own, but for the flag that +/// made it a stand-in. +#[cfg_attr(not(target_os = "linux"), allow(dead_code))] +fn relay_args(args: impl IntoIterator) -> Vec { + let mut positional = false; + args.into_iter() + .filter(|arg| { + if positional { + return true; + } + if arg == "--" { + positional = true; + return true; + } + let Some(arg) = arg.to_str() else { + return true; + }; + !(arg == SUPERVISE_FLAG + || arg + .strip_prefix(SUPERVISE_FLAG) + .is_some_and(|rest| rest.starts_with('='))) + }) + .collect() +} + +/// What one handed-on socket serves, which is how the successor finds it again. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub(crate) enum Role { + /// The QUIC port's reuseport group. + Quic, + /// `listen.tcp.bind`. + Tcp, + /// `listen.unix.bind`. + Unix, + /// `web.http.listen`. + WebHttp, + /// `web.https.listen`. + WebHttps, + /// `internal.listen`. + Internal, + /// `handoff.socket` itself. + Control, +} + +#[cfg(target_os = "linux")] +/// The protocol version a [`Hello`] speaks; a successor refuses any other. +const VERSION: u32 = 1; + +#[cfg(target_os = "linux")] +/// The byte a successor sends once it serves. +const TOOK: u8 = b'T'; + +#[cfg(target_os = "linux")] +/// The byte each batch of descriptors rides on. +const BATCH: u8 = b'F'; + +#[cfg(target_os = "linux")] +/// What the old relay sends first. +#[derive(Debug, Default, Serialize, Deserialize)] +struct Hello { + version: u32, + /// Why the old relay will not hand off, in place of sockets. + #[serde(default, skip_serializing_if = "Option::is_none")] + refused: Option, + #[serde(default)] + sockets: Vec, +} + +#[cfg(target_os = "linux")] +/// One handed-on socket, or the QUIC group's members. +#[derive(Clone, Debug, Serialize, Deserialize)] +struct Entry { + role: Role, + /// The address an internet socket is bound to. + #[serde(default, skip_serializing_if = "Option::is_none")] + addr: Option, + /// The path a Unix socket is bound at. + #[serde(default, skip_serializing_if = "Option::is_none")] + path: Option, + /// The QUIC group's shape. + #[serde(default, skip_serializing_if = "Option::is_none")] + group: Option, + /// How many descriptors this entry carries, after the previous entry's. + fds: usize, +} + +#[cfg(target_os = "linux")] +/// A reuseport group's shape, beside its members' descriptors. +#[derive(Clone, Copy, Debug, Serialize, Deserialize)] +struct Group { + /// Members per generation. + count: u16, + /// The generation the sender serves. + generation: u16, + /// Whether the port lock follows the members. + lock: bool, +} + +/// What a serving [`Handoff`] tells the rest of the relay. +#[cfg_attr(not(target_os = "linux"), allow(dead_code))] +pub(crate) struct Events { + /// Set once a successor took over: stop accepting and drain. + pub handed: tokio::sync::watch::Sender, + /// Dropped once no predecessor serves beside this relay. + pub alone: Option>, + /// Starts this relay's drain. + pub drain: crate::shutdown::Trigger, + /// SIGUSR2, which a stand-in passes on as it leaves this relay to its successor. + #[cfg(unix)] + pub left: Option, + /// How long a successor has to take over once the stand-in left, before this + /// relay drains anyway: `drain_timeout`. + pub window: std::time::Duration, +} + +#[cfg(target_os = "linux")] +/// Whether a socket bound to `held` serves a listener configured for `requested`: +/// the same address, where an ephemeral request takes whatever port was resolved. +fn serves(held: SocketAddr, requested: SocketAddr) -> bool { + held.ip() == requested.ip() && (requested.port() == 0 || requested.port() == held.port()) +} + +#[cfg(target_os = "linux")] +pub use imp::supervise; +#[cfg(target_os = "linux")] +pub(crate) use imp::{Handoff, Sockets}; + +#[cfg(target_os = "linux")] +mod imp { + use std::collections::HashMap; + use std::io; + use std::net::SocketAddr; + use std::os::fd::{AsFd, BorrowedFd, OwnedFd}; + use std::path::{Path, PathBuf}; + + use anyhow::Context; + use moq_sock::shard; + use tokio::io::Interest; + use tokio::net::UnixStream; + + use super::{BATCH, Config, Entry, Events, Hello, Role, TOOK, VERSION, serves}; + + /// The sockets a starting relay binds or takes over, and the copies it keeps to + /// hand on in turn. Disabled without `handoff.socket`, when every method returns + /// `None` and each listener binds as it always has. + pub(crate) struct Sockets { + /// `None` when handoff is off. + path: Option, + /// What the predecessor handed over and nothing has taken yet. + inherited: HashMap)>, + /// What a successor gets, in the order it is sent. + exports: Vec<(Entry, Vec)>, + /// Steers this relay's generation of the QUIC group. + steering: Option, + /// The connection to the relay this one replaces. + predecessor: Option, + } + + impl Sockets { + /// Handoff off: every listener binds as it always has. + pub(crate) fn disabled() -> Self { + Self { + path: None, + inherited: HashMap::new(), + exports: Vec::new(), + steering: None, + predecessor: None, + } + } + + /// Find the relay this one replaces, taking its sockets when one answers. + pub(crate) async fn open(config: &Config) -> anyhow::Result { + let Some(path) = &config.socket else { + return Ok(Self::disabled()); + }; + let mut sockets = Self { + path: Some(path.clone()), + ..Self::disabled() + }; + + let stream = match UnixStream::connect(path).await { + Ok(stream) => stream, + // Nobody serves there: this relay is the first. + Err(err) if matches!(err.kind(), io::ErrorKind::NotFound | io::ErrorKind::ConnectionRefused) => { + tracing::info!(path = %path.display(), "no relay to take over from; binding every listener"); + return Ok(sockets); + } + Err(err) => { + return Err(err).with_context(|| format!("failed to reach the relay at {}", path.display())); + } + }; + same_user(&stream).context("refusing the relay at the handoff socket")?; + + let (hello, fds) = receive(&stream) + .await + .with_context(|| format!("failed to take over from the relay at {}", path.display()))?; + let mut fds = fds.into_iter(); + for entry in hello.sockets { + let taken = fds.by_ref().take(entry.fds).collect(); + sockets.inherited.insert(entry.role, (entry, taken)); + } + tracing::info!( + path = %path.display(), + sockets = ?sockets.inherited.keys().collect::>(), + "taking over from the relay already serving" + ); + sockets.predecessor = Some(stream); + Ok(sockets) + } + + /// Whether handoff is on. + pub(crate) fn enabled(&self) -> bool { + self.path.is_some() + } + + /// Whether this relay took over from one still serving. + pub(crate) fn replacing(&self) -> bool { + self.predecessor.is_some() + } + + /// The predecessor's socket for `role`, if it serves `requested`. One that + /// serves something else is dropped, and the listener binds afresh. + fn take(&mut self, role: Role, requested: impl Fn(&Entry) -> bool) -> Option<(Entry, Vec)> { + let (entry, fds) = self.inherited.remove(&role)?; + if requested(&entry) { + return Some((entry, fds)); + } + tracing::warn!( + ?role, + ?entry, + "the predecessor's socket serves a different address; binding afresh" + ); + None + } + + /// Keep a copy of `socket` to hand on as `entry`. + fn export(&mut self, entry: Entry, socket: BorrowedFd<'_>) -> anyhow::Result<()> { + let fd = socket + .try_clone_to_owned() + .context("failed to keep a copy to hand on")?; + self.exports.push((Entry { fds: 1, ..entry }, vec![fd])); + Ok(()) + } + + /// The QUIC port's reuseport group on `addr`: the predecessor's, or a fresh one + /// of `count` members with room for a successor. + #[cfg(any(feature = "_quic", feature = "_uring"))] + pub(crate) fn group(&mut self, addr: SocketAddr, count: u16) -> anyhow::Result> { + if !self.enabled() { + return Ok(None); + } + // Both generations share the one byte the steering filter reads. + let most = shard::MAX_SHARDS / 2; + if count > most { + tracing::warn!(count, most, "serving fewer workers, to leave room for a successor"); + } + let count = count.min(most); + let inherited = self.take(Role::Quic, |entry| { + entry.addr.is_some_and(|held| serves(held, addr)) && entry.group.is_some() + }); + let bound = match inherited { + Some((entry, mut fds)) => { + let group = entry.group.expect("checked by take"); + let lock = match group.lock { + true => fds.pop(), + false => None, + }; + if group.count != count { + tracing::warn!( + predecessor = group.count, + configured = count, + "serving the predecessor's worker count: a group's size cannot change across a handoff" + ); + } + let handoff = shard::Handoff { + addr: entry.addr.expect("checked by take"), + count: group.count, + generation: group.generation, + sockets: fds, + lock, + }; + shard::Bound::adopt(handoff).context("failed to adopt the predecessor's QUIC sockets")? + } + None => { + let mut config = shard::Config::new(addr, count); + config.handoff = true; + config + .bind() + .with_context(|| format!("failed to bind the reuseport group on {addr}"))? + } + }; + + let handoff = bound.handoff().context("failed to keep the QUIC sockets to hand on")?; + let lock = handoff.lock.is_some(); + let mut fds = handoff.sockets; + fds.extend(handoff.lock); + self.exports.push(( + Entry { + role: Role::Quic, + addr: Some(handoff.addr), + path: None, + group: Some(super::Group { + count: handoff.count, + generation: handoff.generation, + lock, + }), + fds: fds.len(), + }, + fds, + )); + self.steering = Some(bound.steering().context("failed to steer the QUIC sockets")?); + Ok(Some(bound)) + } + + /// The tokio QUIC workers: on the predecessor's group, or a fresh one with room + /// for a successor when handoff is on. + #[cfg(feature = "_quic")] + pub(crate) fn workers( + &mut self, + server: moq_tokio::server::Config, + config: moq_tokio::worker::Config, + ) -> anyhow::Result { + if !self.enabled() { + return Ok(moq_tokio::worker::Workers::bind(server, config)?); + } + let addr = match &server.listen.bind { + Some(bind) => bind.resolve().context("failed to resolve listen.bind")?, + None => moq_tokio::server::DEFAULT_BIND, + }; + let group = self.group(addr, config.count)?.expect("handoff is on"); + Ok(moq_tokio::worker::Workers::from_group(server, group, config.pin)?) + } + + /// A TCP listener for `role` on `addr`: the predecessor's, or a fresh one. + pub(crate) fn tcp(&mut self, role: Role, addr: SocketAddr) -> anyhow::Result> { + if !self.enabled() { + return Ok(None); + } + let listener = match self.take(role, |entry| entry.addr.is_some_and(|held| serves(held, addr))) { + Some((_, mut fds)) => { + let fd = fds.pop().context("the predecessor sent no socket")?; + let listener = std::net::TcpListener::from(fd); + listener.set_nonblocking(true)?; + listener + } + None => moq_tokio::bind::tcp(addr).with_context(|| format!("failed to bind {addr}"))?, + }; + let entry = Entry { + role, + addr: Some(listener.local_addr()?), + path: None, + group: None, + fds: 1, + }; + self.export(entry, listener.as_fd())?; + Ok(Some(listener)) + } + + /// A Unix listener for `role` at `path`: the predecessor's, or a fresh one. + pub(crate) fn unix( + &mut self, + role: Role, + path: &Path, + ) -> anyhow::Result> { + if !self.enabled() { + return Ok(None); + } + let listener = match self.take(role, |entry| entry.path.as_deref() == Some(path)) { + Some((_, mut fds)) => { + std::os::unix::net::UnixListener::from(fds.pop().context("the predecessor sent no socket")?) + } + None => bind_unix(path)?, + }; + listener.set_nonblocking(true)?; + let entry = Entry { + role, + addr: None, + path: Some(path.to_path_buf()), + group: None, + fds: 1, + }; + self.export(entry, listener.as_fd())?; + Ok(Some(listener)) + } + + /// Take the handoff socket itself, ready to serve a successor. `None` without + /// handoff. + pub(crate) fn finish(mut self) -> anyhow::Result> { + let Some(path) = self.path.clone() else { + return Ok(None); + }; + let control = self.unix(Role::Control, &path)?.expect("handoff is on"); + // Whatever the predecessor held that this relay no longer serves stays with + // the predecessor until it exits. + for (role, _) in self.inherited.drain() { + tracing::info!(?role, "not serving a listener the predecessor held"); + } + Ok(Some(Handoff { + control, + exports: self.exports, + steering: self.steering, + predecessor: self.predecessor, + })) + } + } + + /// Bind a Unix listener at `path`, replacing a socket file nothing listens on. + /// + /// Only called once a connect to the path found nobody, or for a listener the + /// predecessor did not hold, so a stale file is a crashed relay's. + fn bind_unix(path: &Path) -> anyhow::Result { + use std::os::unix::fs::FileTypeExt; + match std::fs::symlink_metadata(path) { + Ok(meta) if meta.file_type().is_socket() => std::fs::remove_file(path) + .with_context(|| format!("failed to remove the stale socket at {}", path.display()))?, + Ok(_) => anyhow::bail!("{} exists and is not a socket", path.display()), + Err(err) if err.kind() == io::ErrorKind::NotFound => {} + Err(err) => return Err(err).with_context(|| format!("failed to inspect {}", path.display())), + } + std::os::unix::net::UnixListener::bind(path).with_context(|| format!("failed to bind {}", path.display())) + } + + /// A relay's handoff while it serves: the sockets it hands on, how it steers new + /// handshakes, and the relay it replaces. + pub(crate) struct Handoff { + control: std::os::unix::net::UnixListener, + exports: Vec<(Entry, Vec)>, + steering: Option, + predecessor: Option, + } + + impl Handoff { + /// Take every new QUIC handshake from the relay this one replaces, while it + /// keeps the connections it holds. A no-op without one. + pub(crate) fn steer_here(&self) -> anyhow::Result<()> { + if self.predecessor.is_none() { + return Ok(()); + } + self.steer(shard::Steer::Beside) + .context("failed to steer new handshakes here") + } + + /// Tell the relay this one replaces that this one serves, so it stops + /// accepting and drains. A no-op without one. + pub(crate) async fn took(&self) { + let Some(predecessor) = &self.predecessor else { + return; + }; + // A predecessor gone already is noticed as the connection's end. + match send(predecessor, &[TOOK], &[]).await { + Ok(()) => tracing::info!("serving beside the relay this one replaces; it drains now"), + Err(err) => tracing::warn!(%err, "the relay this one replaces is gone"), + } + } + + /// Steer this relay's generation, when it serves QUIC. + fn steer(&self, steer: shard::Steer) -> io::Result<()> { + match &self.steering { + Some(steering) => steering.steer(steer), + None => Ok(()), + } + } + + /// Serve successors and watch the relay this one replaces, for as long as + /// this relay runs. Only an error returns. + pub(crate) async fn run(self, events: Events) -> anyhow::Result { + let Events { + handed, + mut alone, + drain, + mut left, + window, + } = events; + // When this relay drains if no successor took over since its stand-in left. + let mut takeover: Option = None; + let control = tokio::net::UnixListener::from_std(self.control.try_clone()?) + .context("failed to serve the handoff socket")?; + let mut predecessor = self.predecessor.as_ref(); + if predecessor.is_none() { + alone.take(); + } + let mut successor: Option = None; + + loop { + tokio::select! { + () = closed(predecessor), if predecessor.is_some() => { + tracing::info!("the relay this one replaced exited"); + predecessor = None; + alone.take(); + if let Err(err) = self.steer(shard::Steer::Alone) { + tracing::warn!(%err, "failed to take back the replaced relay's connections"); + } + } + Some(()) = signaled(left.as_mut()), if successor.is_none() && takeover.is_none() => { + tracing::info!(?window, "left by the stand-in; draining unless a relay takes over first"); + takeover = Some(kio::clock::Instant::now().saturating_add(window)); + } + () = until(takeover), if successor.is_none() && takeover.is_some() => { + // A failed upgrade: the service no longer tracks this relay, so it must not + // serve on indefinitely, holding the port the next start needs. + tracing::warn!(?window, "no relay took over after the stand-in left; draining"); + takeover = None; + drain.start(); + } + () = closed(successor.as_ref()), if successor.is_some() => { + tracing::warn!("the relay that took over exited before this one; taking new handshakes back"); + successor = None; + if let Err(err) = self.steer(shard::Steer::Alone) { + tracing::warn!(%err, "failed to take new handshakes back"); + } + } + accepted = control.accept(), if successor.is_none() => { + let stream = match accepted { + Ok((stream, _)) => stream, + Err(err) => { + tracing::warn!(%err, "failed to accept on the handoff socket"); + continue; + } + }; + match self.offer(&stream, predecessor.is_some()).await { + Ok(()) => { + successor = Some(stream); + handed.send_replace(true); + drain.start(); + } + Err(err) => { + tracing::warn!(err = %format!("{err:#}"), "a relay failed to take over"); + // It may have steered new handshakes to itself before failing. Not when + // refused for serving beside a predecessor, which still holds the other + // generation's connections. + if predecessor.is_none() + && let Err(err) = self.steer(shard::Steer::Alone) + { + tracing::warn!(%err, "failed to take new handshakes back"); + } + } + } + } + } + } + } + + /// Hand every socket to the relay on `stream`, and wait until it serves. + async fn offer(&self, stream: &UnixStream, beside: bool) -> anyhow::Result<()> { + same_user(stream)?; + let mut hello = Hello { + version: VERSION, + ..Default::default() + }; + if beside { + hello.refused = Some( + "the relay there still serves beside the one it replaced; retry once that one exits".to_string(), + ); + let body = serde_json::to_vec(&hello)?; + send(stream, &frame(&body), &[]).await?; + anyhow::bail!("refused: still serving beside the relay this one replaced"); + } + + hello.sockets = self.exports.iter().map(|(entry, _)| entry.clone()).collect(); + let body = serde_json::to_vec(&hello)?; + send(stream, &frame(&body), &[]).await?; + let fds: Vec> = self + .exports + .iter() + .flat_map(|(_, fds)| fds.iter().map(AsFd::as_fd)) + .collect(); + for batch in fds.chunks(moq_sock::fd::MAX) { + send(stream, &[BATCH], batch).await?; + } + tracing::info!(sockets = fds.len(), "handed every socket to the relay taking over"); + + let mut byte = Vec::new(); + let mut unexpected = Vec::new(); + match recv(stream, &mut byte, &mut unexpected).await? { + 0 => anyhow::bail!("it exited before serving"), + _ if byte.first() == Some(&TOOK) => {} + _ => anyhow::bail!("it answered {byte:?}"), + } + tracing::info!("the relay that took over serves; draining"); + Ok(()) + } + } + + /// Prefix `body` with its big-endian length. + fn frame(body: &[u8]) -> Vec { + let mut framed = u32::try_from(body.len()) + .expect("a hello fits u32") + .to_be_bytes() + .to_vec(); + framed.extend_from_slice(body); + framed + } + + /// Read a [`Hello`] and every descriptor it lists. + async fn receive(stream: &UnixStream) -> anyhow::Result<(Hello, Vec)> { + let mut data = Vec::new(); + let mut fds = Vec::new(); + let hello = loop { + if data.len() >= 4 { + let len = u32::from_be_bytes(data[..4].try_into().expect("four bytes")) as usize; + if data.len() >= 4 + len { + let hello: Hello = serde_json::from_slice(&data[4..4 + len]).context("unreadable hello")?; + data.drain(..4 + len); + break hello; + } + } + anyhow::ensure!(recv(stream, &mut data, &mut fds).await? > 0, "it closed the connection"); + }; + anyhow::ensure!( + hello.version == VERSION, + "it speaks handoff version {}, not {VERSION}", + hello.version + ); + if let Some(refused) = hello.refused { + anyhow::bail!("it refused: {refused}"); + } + + let total: usize = hello.sockets.iter().map(|entry| entry.fds).sum(); + let batches = total.div_ceil(moq_sock::fd::MAX); + while data.len() < batches || fds.len() < total { + anyhow::ensure!( + recv(stream, &mut data, &mut fds).await? > 0, + "it closed the connection with {} of {total} sockets sent", + fds.len() + ); + } + anyhow::ensure!(fds.len() == total, "it sent {} sockets, not {total}", fds.len()); + Ok((hello, fds)) + } + + /// Send all of `data`, with `fds` riding on its first byte. + async fn send(stream: &UnixStream, mut data: &[u8], mut fds: &[BorrowedFd<'_>]) -> io::Result<()> { + while !data.is_empty() { + stream.writable().await?; + match stream.try_io(Interest::WRITABLE, || moq_sock::fd::send(stream.as_fd(), data, fds)) { + Ok(sent) => { + data = &data[sent..]; + fds = &[]; + } + Err(err) if err.kind() == io::ErrorKind::WouldBlock => {} + Err(err) => return Err(err), + } + } + Ok(()) + } + + /// Read what is available into `data`, and any descriptors into `fds`. Zero at + /// the end of the stream. + async fn recv(stream: &UnixStream, data: &mut Vec, fds: &mut Vec) -> io::Result { + let mut buf = [0; 4096]; + loop { + stream.readable().await?; + match stream.try_io(Interest::READABLE, || moq_sock::fd::recv(stream.as_fd(), &mut buf, fds)) { + Ok(read) => { + data.extend_from_slice(&buf[..read]); + return Ok(read); + } + Err(err) if err.kind() == io::ErrorKind::WouldBlock => {} + Err(err) => return Err(err), + } + } + } + + /// Resolve once the relay on `stream` is gone; never without one. + /// The next SIGUSR2; never without the handler. + async fn signaled(signal: Option<&mut tokio::signal::unix::Signal>) -> Option<()> { + match signal { + Some(signal) => signal.recv().await, + None => std::future::pending().await, + } + } + + /// Wait for `deadline`; never without one. + async fn until(deadline: Option) { + match deadline { + Some(deadline) => kio::clock::sleep_until(deadline).await, + None => std::future::pending().await, + } + } + + async fn closed(stream: Option<&UnixStream>) { + let Some(stream) = stream else { + return std::future::pending().await; + }; + let mut data = Vec::new(); + let mut fds = Vec::new(); + // Anything it sends after the handoff is ignored; only its end counts. + while let Ok(read) = recv(stream, &mut data, &mut fds).await { + if read == 0 { + return; + } + data.clear(); + fds.clear(); + } + } + + /// Stand in for the relay as a service's main process, per `handoff.supervise`: + /// start the relay as a child with this process's arguments, pass it SIGTERM, + /// SIGINT, and SIGUSR1 (a heap profile's), forward its readiness, and return its + /// exit code once it exits. On SIGUSR2 pass it on and return 0 at once, leaving the + /// relay to drain beside the one that replaces it, or on its own once + /// `drain_timeout` passes with none taking over. Before the relay is ready, SIGUSR2 + /// stops it instead, as it serves nothing a successor could take. + /// + /// A service manager takes readiness only from its main process, which the relay + /// no longer is; it also takes it from nothing else, so a relay left draining can + /// never speak for the service. That needs a service manager's `NOTIFY_SOCKET`. + pub async fn supervise(config: &Config) -> anyhow::Result { + use std::os::unix::process::ExitStatusExt; + + use tokio::signal::unix::{SignalKind, signal}; + + let path = config + .socket + .as_ref() + .context("handoff.supervise needs handoff.socket")?; + // Before the relay starts, so no signal for it is missed. + let mut terminate = signal(SignalKind::terminate())?; + let mut interrupt = signal(SignalKind::interrupt())?; + let mut profile = signal(SignalKind::user_defined1())?; + let mut leave = signal(SignalKind::user_defined2())?; + + let exe = std::env::current_exe().context("failed to find this binary")?; + let mut command = tokio::process::Command::new(exe); + command + .args(super::relay_args(std::env::args_os().skip(1))) + .env("MOQ_HANDOFF_SUPERVISE", "false") + // Its own process group: a signal to the stand-in's group, such as a terminal's + // Ctrl-C, reaches the relay once, through `pass`, since a second ends its drain. + .process_group(0); + + let manager = std::env::var_os("NOTIFY_SOCKET") + .context("handoff.supervise needs a service manager's NOTIFY_SOCKET (systemd Type=notify)")?; + let mut name = path.clone().into_os_string(); + name.push(format!(".{}.notify", std::process::id())); + let name = PathBuf::from(name); + let _ = std::fs::remove_file(&name); + let socket = + tokio::net::UnixDatagram::bind(&name).with_context(|| format!("failed to bind {}", name.display()))?; + command.env("NOTIFY_SOCKET", &name); + let notify = Notify { name, socket, manager }; + + let mut child = command.spawn().context("failed to start the relay")?; + let pid = child.id().context("the relay exited at once")? as libc::pid_t; + tracing::info!(pid, "supervising the relay"); + + let mut message = [0; 4096]; + // Whether the relay reported READY, and so listens for SIGUSR2. + let mut ready = false; + let code = loop { + tokio::select! { + status = child.wait() => { + let status = status.context("failed to wait for the relay")?; + tracing::info!(%status, "the relay exited"); + // One a signal ended exits as a shell reports it, 128 plus the signal. + break status.code().or_else(|| status.signal().map(|signal| 128 + signal)).unwrap_or(1); + } + Some(()) = terminate.recv() => pass(pid, libc::SIGTERM), + Some(()) = interrupt.recv() => pass(pid, libc::SIGINT), + Some(()) = profile.recv() => pass(pid, libc::SIGUSR1), + Some(()) = leave.recv() => { + if ready { + tracing::info!(pid, "leaving the relay to drain beside the one replacing it"); + pass(pid, libc::SIGUSR2); + break 0; + } + tracing::info!(pid, "restarted before the relay was ready; stopping it"); + pass(pid, libc::SIGTERM); + } + Ok(len) = notify.socket.recv(&mut message) => { + let message = &message[..len]; + ready |= std::str::from_utf8(message).is_ok_and(|text| text.lines().any(|line| line == "READY=1")); + if let Err(err) = notify.forward(message) { + tracing::warn!(%err, "failed to forward the relay's notification"); + } + } + } + }; + let _ = std::fs::remove_file(¬ify.name); + Ok(code) + } + + /// Pass `signal` to the relay at `pid`. Not yet reaped, since the wait for it has + /// not returned, so the pid is still its own. + fn pass(pid: libc::pid_t, signal: libc::c_int) { + // SAFETY: kill has no memory-safety preconditions. + if unsafe { libc::kill(pid, signal) } == -1 { + tracing::warn!(pid, signal, err = %io::Error::last_os_error(), "failed to signal the relay"); + } + } + + /// Where a stand-in's relay sends its notifications, and the service manager's + /// socket they are forwarded to. + struct Notify { + name: PathBuf, + socket: tokio::net::UnixDatagram, + manager: std::ffi::OsString, + } + + impl Notify { + /// Send `message` on to the service manager, as this process's own. + fn forward(&self, message: &[u8]) -> io::Result<()> { + use std::os::linux::net::SocketAddrExt; + let socket = std::os::unix::net::UnixDatagram::unbound()?; + let manager = self.manager.as_encoded_bytes(); + match manager.strip_prefix(b"@") { + Some(name) => { + let addr = std::os::unix::net::SocketAddr::from_abstract_name(name)?; + socket.send_to_addr(message, &addr)?; + } + None => { + socket.send_to(message, Path::new(&self.manager))?; + } + } + Ok(()) + } + } + + /// Refuse a peer that runs as another user: the sockets are this user's. + fn same_user(stream: &UnixStream) -> anyhow::Result<()> { + let peer = stream.peer_cred().context("failed to read the peer's credentials")?; + // SAFETY: geteuid has no preconditions and cannot fail. + let euid = unsafe { libc::geteuid() }; + anyhow::ensure!(peer.uid() == euid, "it runs as uid {}, not {euid}", peer.uid()); + Ok(()) + } +} + +/// Off Linux, handoff is refused when configured, and otherwise does nothing. +#[cfg(not(target_os = "linux"))] +mod imp { + use super::{Config, Role}; + + /// Handoff is Linux-only: with no socket configured every method declines. + pub(crate) struct Sockets; + + impl Sockets { + pub(crate) async fn open(config: &Config) -> anyhow::Result { + anyhow::ensure!(config.socket.is_none(), "handoff.socket is Linux-only"); + Ok(Self) + } + + pub(crate) fn replacing(&self) -> bool { + false + } + + #[cfg(feature = "_quic")] + pub(crate) fn workers( + &mut self, + server: moq_tokio::server::Config, + config: moq_tokio::worker::Config, + ) -> anyhow::Result { + Ok(moq_tokio::worker::Workers::bind(server, config)?) + } + + pub(crate) fn tcp( + &mut self, + _role: Role, + _addr: std::net::SocketAddr, + ) -> anyhow::Result> { + Ok(None) + } + + #[cfg(all(feature = "uds", unix))] + pub(crate) fn unix( + &mut self, + _role: Role, + _path: &std::path::Path, + ) -> anyhow::Result> { + Ok(None) + } + + pub(crate) fn finish(self) -> anyhow::Result> { + Ok(None) + } + } + + /// Never called off Linux, where the config refuses `handoff.supervise`. + pub async fn supervise(_config: &Config) -> anyhow::Result { + anyhow::bail!("handoff.supervise is Linux-only") + } + + /// Never built off Linux. + pub(crate) enum Handoff {} + + impl Handoff { + pub(crate) fn steer_here(&self) -> anyhow::Result<()> { + match *self {} + } + + pub(crate) async fn took(&self) { + match *self {} + } + + pub(crate) async fn run(self, _events: super::Events) -> anyhow::Result { + match self {} + } + } +} + +#[cfg(not(target_os = "linux"))] +pub use imp::supervise; +#[cfg(not(target_os = "linux"))] +pub(crate) use imp::{Handoff, Sockets}; + +#[cfg(test)] +mod tests { + use super::*; + + /// The relay a stand-in starts sees every argument but the one that made it a + /// stand-in, including after a `--`, where that spelling is a positional. + #[test] + fn a_stand_in_starts_its_relay_without_its_flag() { + let args = [ + "--handoff-supervise", + "--handoff-socket=/run/h.sock", + "--handoff-supervise=true", + "--handoff-supervisor", + "relay.toml", + "--", + "--handoff-supervise", + ]; + let kept = relay_args(args.map(std::ffi::OsString::from)); + assert_eq!( + kept, + [ + "--handoff-socket=/run/h.sock", + "--handoff-supervisor", + "relay.toml", + "--", + "--handoff-supervise" + ] + ); + } + + /// A handed-on socket's address serves the listener configured for it, and an + /// ephemeral port takes whatever the old relay resolved. + #[cfg(target_os = "linux")] + #[test] + fn a_held_socket_serves_the_same_address() { + let held: SocketAddr = "127.0.0.1:4443".parse().unwrap(); + assert!(serves(held, "127.0.0.1:4443".parse().unwrap())); + assert!(serves(held, "127.0.0.1:0".parse().unwrap())); + assert!(!serves(held, "127.0.0.1:443".parse().unwrap())); + assert!(!serves(held, "[::]:4443".parse().unwrap())); + } +} diff --git a/rs/moq-relay/src/internal.rs b/rs/moq-relay/src/internal.rs index ee81584144e..7b3d25cbbf2 100644 --- a/rs/moq-relay/src/internal.rs +++ b/rs/moq-relay/src/internal.rs @@ -161,6 +161,19 @@ impl Internal { self } + /// Serve `listener`, already bound, in place of binding `internal.listen`. + pub(crate) fn with_bound(mut self, listener: Option) -> anyhow::Result { + if let Some(listener) = listener { + self.addr = Some( + listener + .local_addr() + .context("failed to resolve internal bind address")?, + ); + self.listener = Some(listener); + } + Ok(self) + } + /// Bind the configured listener now, so [`addr`](Self::addr) reports an ephemeral port before serving. pub fn bind(mut self) -> anyhow::Result { if let Some(listen) = self.config.listen diff --git a/rs/moq-relay/src/lib.rs b/rs/moq-relay/src/lib.rs index fdafb0f6ec5..c349e9ee7c4 100644 --- a/rs/moq-relay/src/lib.rs +++ b/rs/moq-relay/src/lib.rs @@ -20,6 +20,7 @@ mod config; mod connection; mod critical; mod fetch; +pub mod handoff; mod http_client; mod incoming; pub mod internal; diff --git a/rs/moq-relay/src/main.rs b/rs/moq-relay/src/main.rs index 9c46f87567c..ee96529a436 100644 --- a/rs/moq-relay/src/main.rs +++ b/rs/moq-relay/src/main.rs @@ -12,5 +12,10 @@ async fn main() -> anyhow::Result<()> { // embedder gets it by calling one function instead of copying this file. // Extra routes and cloned handles go on the returned `Relay`; `run` keeps // the listeners and workers. - Relay::load(Config::load()?).await?.run().await + let config = Config::load()?; + if config.handoff.supervise { + let code = moq_relay::handoff::supervise(&config.handoff).await?; + std::process::exit(code); + } + Relay::load(config).await?.run().await } diff --git a/rs/moq-relay/src/relay.rs b/rs/moq-relay/src/relay.rs index 4376d2d3472..677a8bb5a3d 100644 --- a/rs/moq-relay/src/relay.rs +++ b/rs/moq-relay/src/relay.rs @@ -121,6 +121,12 @@ pub struct Relay { /// nothing on tokio serves QUIC: a stream-only relay, or the io_uring workers. #[cfg(feature = "_quic")] demux: Option, + /// The sockets this relay hands to the relay that replaces it. `None` unless + /// `handoff.socket` is set. + handoff: Option, + /// Releases the stats publisher once the relay this one replaced has exited. + /// `None` unless it took over from one still serving. + alone: Option>, } impl Relay { @@ -168,6 +174,10 @@ impl Relay { let (shutdown_trigger, shutdown) = shutdown::Observer::new(drain_timeout); let handshake_timeout = config.listen.resolved_timeout(); + // Before anything binds: a relay already serving hands over its sockets, which + // every listener below takes in place of binding its own. + let mut sockets = crate::handoff::Sockets::open(&config.handoff).await?; + // Bind the QUIC workers first: they own the listen address when configured, // so the server below must not also try to bind it. #[cfg_attr( @@ -198,7 +208,8 @@ impl Relay { server.quic = config.quic.clone(); server.admission = shutdown.admission.clone(); server.demux = demux.clone(); - Some(moq_tokio::worker::Workers::bind(server, worker).context("failed to start the QUIC workers")?) + let workers = sockets.workers(server, worker); + Some(workers.context("failed to start the QUIC workers")?) } _ => None, }; @@ -221,7 +232,10 @@ impl Relay { server.listen = listen.clone(); server.quic = config.quic.clone(); server.tls = served.clone(); - Some(crate::uring::Workers::bind(server, worker).context("failed to start the io_uring QUIC workers")?) + Some( + crate::uring::Workers::bind(server, worker, &mut sockets) + .context("failed to start the io_uring QUIC workers")?, + ) } _ => None, }; @@ -232,6 +246,13 @@ impl Relay { let quic_owned_elsewhere = workers_addr.is_some(); let mut server_config = moq_tokio::server::Config::default(); + if let Some(addr) = listen.tcp.bind { + server_config.streams.tcp = sockets.tcp(crate::handoff::Role::Tcp, addr)?; + } + #[cfg(all(feature = "uds", unix))] + if let Some(path) = &listen.unix.bind { + server_config.streams.unix = sockets.unix(crate::handoff::Role::Unix, path)?; + } server_config.listen = listen; server_config.tls = served; server_config.quic = config.quic.clone(); @@ -294,6 +315,13 @@ impl Relay { .with_stats(config.stats); cluster_options.client = config.limits.client; cluster_options.peer = config.limits.peer; + // The node's stats broadcast has one instance at a time: the relay this one + // replaces keeps it until it exits, and its last counters are read first. + let alone = sockets.replacing().then(|| { + let (alone, hold) = tokio::sync::oneshot::channel(); + cluster_options.stats_hold = Some(hold); + alone + }); let (cluster, driver) = cluster::Cluster::new(cluster_options)?; let mut cluster = cluster.with_connect(config.connect.clone(), config.quic.clone()); if let Some(advertise) = advertise { @@ -302,13 +330,21 @@ impl Relay { let sessions = crate::session::Registry::new(); let (ready, _) = tokio::sync::watch::channel(Startup::Starting); + let http = match config.web.http.listen { + Some(addr) => sockets.tcp(crate::handoff::Role::WebHttp, addr)?, + None => None, + }; + let https = match config.web.https.listen { + Some(addr) => sockets.tcp(crate::handoff::Role::WebHttps, addr)?, + None => None, + }; let web = web::Web::new(auth.clone(), cluster.clone(), certificates, config.web) .with_shutdown(shutdown.clone()) .with_listen(&config.listen) .with_quic(&config.quic) .with_sessions(sessions.clone()) .with_tls_acceptor(server.tls_acceptor()); - let web = web.bind()?; + let web = web.with_bound(http, https)?.bind()?; // `bind`, not `listen`: the TCP/Unix accept loops handshake as soon as // they run, and `load` is not yet willing to take a session. `run` // starts them on its first accept. @@ -319,7 +355,12 @@ impl Relay { // when unconfigured. Every listener that performs a real accept(2) reports here, // web and stream alike: a stream-only relay has no web listener at all, and the // point is that whichever socket goes quiet is the one a scrape can see. + let held = match config.internal.listen { + Some(addr) => sockets.tcp(crate::handoff::Role::Internal, addr)?, + None => None, + }; let internal = internal::Internal::new(config.internal, cluster.stats.clone()) + .with_bound(held)? .with_timeout(handshake_timeout) .with_cluster(&cluster) .with_sessions(sessions.clone()) @@ -338,6 +379,9 @@ impl Relay { None => internal, }; + // Last, once every listener holds its socket: the handoff socket itself. + let handoff = sockets.finish()?; + // `kind` so the QUIC line is distinguishable from the web listeners', which // log the same way from `web::Web::serve` and may sit on a different port. match addr { @@ -368,6 +412,8 @@ impl Relay { uring, #[cfg(feature = "_quic")] demux, + handoff, + alone, }) } @@ -586,6 +632,8 @@ impl Relay { workers, #[cfg(all(target_os = "linux", feature = "_uring"))] uring, + handoff, + alone, .. } = self; @@ -618,12 +666,31 @@ impl Relay { }) .transpose()?; + // Take every new QUIC handshake from the relay this one replaces: the workers + // are bound, so what arrives queues until they serve just below. Not once this + // relay is itself stopping. + if let Some(handoff) = handoff.as_ref() + && shutdown.deadline().is_none() + { + handoff.steer_here()?; + } + // After the last step that can fail, since a handler outlives a // failed `run`, and before anything accepts a session a signal would kill // without a GOAWAY. - // Nothing awaits from here to the drain below, so a signal caught in between - // starts the drain at its first poll, as one just after READY would. + // Nothing awaits from here to the drain below but the note to the relay this + // one replaces, so a signal caught in between starts the drain at its first + // poll, as one just after READY would. let mut handlers = signals.then(shutdown::Signals::new).transpose()?; + // Before READY too, which is when a stand-in starts passing SIGUSR2 on. + #[cfg(unix)] + let left = match signals && handoff.is_some() { + true => Some( + tokio::signal::unix::signal(tokio::signal::unix::SignalKind::user_defined2()) + .context("failed to listen for SIGUSR2")?, + ), + false => None, + }; #[cfg(all(target_os = "linux", feature = "_uring"))] if let Some(uring) = uring.as_mut() { @@ -632,6 +699,12 @@ impl Relay { // An embedder that already stopped the relay gets a drain, not a READY. if shutdown.deadline().is_none() { + // The relay this one replaces stops accepting and drains, so READY means + // this one serves in its place. Its stream and web listeners are this + // relay's too, and queue what arrives until the loops below accept. + if let Some(handoff) = handoff.as_ref() { + handoff.took().await; + } ready.send_replace(Startup::Serving); #[cfg(unix)] @@ -716,6 +789,10 @@ impl Relay { let quic_on_workers = has_workers || uring_serving; #[cfg(not(all(target_os = "linux", feature = "_uring")))] let quic_on_workers = has_workers; + // Set once the relay replacing this one serves: every accept loop stops, since + // that relay accepts on the same sockets, and the drain starts. + let (handed_tx, handed) = tokio::sync::watch::channel(false); + let serve_shared = { let idle = quic_on_workers && server.accept_health().is_empty(); let server = &mut server; @@ -723,22 +800,55 @@ impl Relay { let auth = auth.clone(); let shutdown = shutdown.clone(); let sessions = sessions.clone(); + let mut handed = handed.clone(); // Boxed like the serving futures in the `select!` below. Box::pin(async move { - match idle { - true => std::future::pending().await, - false => serve_listening(server, cluster, auth, shutdown, sessions).await, + if idle { + return std::future::pending().await; + } + let served = tokio::select! { + res = serve_listening(server, cluster, auth, shutdown, sessions) => Some(res), + true = handed_set(&mut handed) => None, + }; + match served { + Some(res) => res, + None => { + server.close_streams().await; + std::future::pending().await + } } }) }; + let handoff = { + let drain = shutdown_trigger.clone(); + let window = shutdown.drain_timeout; + async move { + let Some(handoff) = handoff else { + return std::future::pending().await; + }; + let events = crate::handoff::Events { + handed: handed_tx, + alone, + drain, + #[cfg(unix)] + left, + window, + }; + match handoff.run(events).await { + Ok(never) => match never {}, + Err(err) => Err::<(), _>(err), + } + } + }; // Box the large serving futures: a debug build copies each one into this poll's // stack frame several times, and an embedder's main thread pays for that frame. let result = tokio::select! { Err(err) = Box::pin(started.run()) => Err(err).context("cluster failed"), - Err(err) = Box::pin(web.serve(web_routes)) => Err(err).context("web server failed"), - Err(err) = internal.serve(internal_routes) => Err(err).context("internal server failed"), + Err(err) = Box::pin(until_handed(handed.clone(), web.serve(web_routes))) => Err(err).context("web server failed"), + Err(err) = until_handed(handed.clone(), internal.serve(internal_routes)) => Err(err).context("internal server failed"), Err(err) = serve_shared => Err(err).context("server failed"), + Err(err) = Box::pin(handoff) => Err(err).context("handoff failed"), Err(err) = quic_workers => Err(err).context("QUIC workers failed"), err = uring_failed => Err(err).context("io_uring QUIC workers failed"), Err(err) = jemalloc => Err(err).context("jemalloc profiler failed"), @@ -768,6 +878,29 @@ impl Relay { } } +/// Serve until this relay hands its listeners to the relay replacing it, then stop +/// accepting and pend: that relay accepts on the same sockets from here on, and the +/// connections already accepted run on as tasks of their own. Serves on for good +/// once `handed` can no longer change. +async fn until_handed( + mut handed: tokio::sync::watch::Receiver, + serve: impl Future>, +) -> anyhow::Result { + let served = tokio::select! { + res = serve => Some(res), + true = handed_set(&mut handed) => None, + }; + match served { + Some(res) => res, + None => std::future::pending().await, + } +} + +/// Whether `handed` was set, rather than its sender dropped unset. +async fn handed_set(handed: &mut tokio::sync::watch::Receiver) -> bool { + handed.wait_for(|handed| *handed).await.is_ok() +} + /// Two-stage shutdown: the first signal, or an embedder firing /// [`shutdown::Trigger::start`], starts the drain broadcast (every session sends /// GOAWAY and waits for its peer to leave). Returns from [`Relay::run`] once diff --git a/rs/moq-relay/src/uring.rs b/rs/moq-relay/src/uring.rs index 0896bdfaf38..d75c2c0707a 100644 --- a/rs/moq-relay/src/uring.rs +++ b/rs/moq-relay/src/uring.rs @@ -128,10 +128,15 @@ pub struct Workers { } impl Workers { - /// Bind one socket per worker on the address `listen` describes. + /// Bind one socket per worker on the address `listen` describes, or serve the + /// group a handoff carries from the relay this one replaces. /// /// Shares one served identity with every member; stream listeners are served separately. - pub fn bind(server: moq_tokio::server::Config, config: moq_tokio::worker::Config) -> anyhow::Result { + pub(crate) fn bind( + server: moq_tokio::server::Config, + config: moq_tokio::worker::Config, + sockets: &mut crate::handoff::Sockets, + ) -> anyhow::Result { // Before any TLS loads or socket binds, so a bad `[quic]` has no side effects. server.quic.validate()?; // Every setting by name, so one added to `listen` fails to compile here until @@ -210,8 +215,11 @@ impl Workers { // concurrently-constructing same-UID group (the bind probe covers one // that already holds it), refuses a size the steering filter could not // address, and binds members in the order the kernel numbers them by. - let mut group = moq_sock::shard::bind(requested, config.count) - .with_context(|| format!("failed to bind the reuseport group on {requested}"))?; + let mut group = match sockets.group(requested, config.count)? { + Some(group) => group, + None => moq_sock::shard::bind(requested, config.count) + .with_context(|| format!("failed to bind the reuseport group on {requested}"))?, + }; let count = group.count(); let mut members = Vec::with_capacity(count as usize); while let Some(member) = group.member().context("failed to clone a reuseport member")? { @@ -336,7 +344,9 @@ impl Workers { }; for member in std::mem::take(&mut self.members) { - let index = member.shard().index(); + // The worker's position in this process, not its slot in the group: a + // successor serving the second generation counts from zero too. + let index = member.worker(); let core = cores.get(index as usize % cores.len().max(1)).copied(); let stop = Arc::new(Stop::default()); let (ready_tx, ready_rx) = std::sync::mpsc::channel(); @@ -556,7 +566,7 @@ struct Spawn { /// alone would not notice one, since the parent holds a sender of its own and /// the channel therefore never closes. fn run_worker(spawn: Spawn) { - let index = spawn.member.shard().index(); + let index = spawn.member.worker(); let stop = spawn.stop.clone(); let failures = spawn.failures.clone(); @@ -592,7 +602,7 @@ fn serve_worker(spawn: Spawn) -> bool { start, failures, } = spawn; - let index = member.shard().index(); + let index = member.worker(); if let Some(core) = core { if moq_sock::cpu::pin(core) { tracing::debug!(index, core = core.id(), "pinned io_uring QUIC worker"); @@ -976,8 +986,12 @@ mod tests { server.listen.tls.generate = vec!["localhost".into()]; } set(&mut server.listen); - let err = Workers::bind(server, moq_tokio::worker::Config::new(1)) - .expect_err("the workers cannot advertise a preferred address"); + let err = Workers::bind( + server, + moq_tokio::worker::Config::new(1), + &mut crate::handoff::Sockets::disabled(), + ) + .expect_err("the workers cannot advertise a preferred address"); assert!(err.to_string().contains(setting), "{err:#}"); } } @@ -991,8 +1005,12 @@ mod tests { server.listen.bind = Some("127.0.0.1:0".parse().unwrap()); server.listen.tls.generate = vec!["localhost".into()]; server.listen.lb.nonce = Some(8); - let err = Workers::bind(server, moq_tokio::worker::Config::new(1)) - .expect_err("a QUIC-LB nonce without an id is refused"); + let err = Workers::bind( + server, + moq_tokio::worker::Config::new(1), + &mut crate::handoff::Sockets::disabled(), + ) + .expect_err("a QUIC-LB nonce without an id is refused"); assert!(err.to_string().contains("--listen-quic-lb-nonce"), "{err:#}"); } @@ -1084,8 +1102,12 @@ mod tests { fn bind_refuses_an_empty_keylog_path() { let mut server = moq_tokio::server::Config::default(); server.quic.keylog = Some(std::path::PathBuf::new()); - let err = - Workers::bind(server, moq_tokio::worker::Config::new(1)).expect_err("an empty keylog path must be refused"); + let err = Workers::bind( + server, + moq_tokio::worker::Config::new(1), + &mut crate::handoff::Sockets::disabled(), + ) + .expect_err("an empty keylog path must be refused"); assert!( matches!(err.downcast_ref(), Some(moq_tokio::Error::KeylogUnset)), "{err:#}" @@ -1107,7 +1129,12 @@ mod tests { let mut server = moq_tokio::server::Config::default(); server.listen.bind = Some("127.0.0.1:0".parse().unwrap()); server.listen.tls.generate = vec!["localhost".into()]; - let workers = Workers::bind(server, moq_tokio::worker::Config::new(2)).unwrap(); + let workers = Workers::bind( + server, + moq_tokio::worker::Config::new(2), + &mut crate::handoff::Sockets::disabled(), + ) + .unwrap(); let cluster = cluster::Cluster::spawn(cluster::Options::default()).unwrap(); let workers = workers .spawn( diff --git a/rs/moq-relay/src/web.rs b/rs/moq-relay/src/web.rs index b641392a4c3..df7c0403cd9 100644 --- a/rs/moq-relay/src/web.rs +++ b/rs/moq-relay/src/web.rs @@ -235,6 +235,28 @@ impl Web { } } + /// Serve these listeners, already bound, in place of binding `http.listen` and + /// `https.listen`. + pub(crate) fn with_bound( + mut self, + http: Option, + https: Option, + ) -> anyhow::Result { + if let Some(listener) = http { + let addr = listener.local_addr().context("failed to resolve HTTP bind address")?; + self.addrs.http = Some(addr); + tracing::info!(%addr, kind = "http", "listening"); + self.http_listener = Some(listener); + } + if let Some(listener) = https { + let addr = listener.local_addr().context("failed to resolve HTTPS bind address")?; + self.addrs.https = Some(addr); + tracing::info!(%addr, kind = "https", "listening"); + self.https_listener = Some(listener); + } + Ok(self) + } + /// Bind the configured listeners now, so [`addrs`](Self::addrs) reports ephemeral ports before serving. pub fn bind(mut self) -> anyhow::Result { let https = &self.config.https; diff --git a/rs/moq-relay/tests/integration/cluster_drain.rs b/rs/moq-relay/tests/integration/cluster_drain.rs index da2c66eeae0..ca768162811 100644 --- a/rs/moq-relay/tests/integration/cluster_drain.rs +++ b/rs/moq-relay/tests/integration/cluster_drain.rs @@ -1,5 +1,6 @@ -//! A draining relay's cluster peers stay on the session it drains instead of -//! redialing it, and reconnect once its replacement is up. +//! A draining relay's cluster peers redial it once, which reaches a relay serving +//! beside it when one took over, and otherwise stay on the redialed session until +//! it closes and reconnect once its replacement is up. //! //! Three relays on kio's simulator: B drains while A and C peer with it. Each peer //! dials B through a [`Net`], which counts the sessions B accepts, and @@ -38,18 +39,39 @@ async fn learns(step: &str, origin: &moq_net::origin::Producer, name: &str) { .expect("resolves"); } +/// Past the healthy bar a peer's session must clear for a GOAWAY to read as a +/// restart's, which the peer redials at once. +const HEALTHY: Duration = Duration::from_secs(2); + /// A relay serving as `b`, publishing `b`, that drains over [`DRAIN`] once `trigger` -/// fires. `exited` resolves once its drain ends and the host is down, as -/// [`moq_relay::Relay::run`] returns and drops its listener. +/// fires. `exited` resolves once its drain ends, as [`moq_relay::Relay::run`] +/// returns. struct Relay { cluster: cluster::Cluster, trigger: shutdown::Trigger, exited: kio::clock::sim::JoinHandle<()>, + /// Whether `b` is still this relay's listener, to take down as it exits. + owned: std::sync::Arc, _route: moq_net::origin::Route, } impl Relay { + /// The relay serving `b`, which takes the host down as it exits and drops its + /// listener. fn up(net: &Net) -> Self { + Self::serve(net) + } + + /// A relay taking over `b` from `old`, as a handoff does: new dials reach it, the + /// sessions `old` holds stay there, and the listener is this relay's to drop. + fn beside(net: &Net, old: &Self) -> Self { + old.owned.store(false, std::sync::atomic::Ordering::SeqCst); + net.down("b"); + Self::serve(net) + } + + fn serve(net: &Net) -> Self { + let owned = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(true)); let cluster = memory::spawn_cluster(net.options(&[])); let route = cluster .origin @@ -58,14 +80,18 @@ impl Relay { let (trigger, observer) = shutdown::Observer::new(DRAIN); let _accepts = net.relay("b", &cluster, &memory::public(&["**"], &[], &[]), &observer); let net = net.clone(); + let exits = owned.clone(); let exited = kio::clock::sim::spawn(async move { observer.drained().await; - net.down("b"); + if exits.load(std::sync::atomic::Ordering::SeqCst) { + net.down("b"); + } }); Self { cluster, trigger, exited, + owned, _route: route, } } @@ -81,16 +107,11 @@ async fn peer(net: &Net, name: &str) -> (cluster::Cluster, moq_net::origin::Rout (cluster, route) } -#[test] -fn draining_relay_keeps_its_peers_until_it_restarts() { - memory::run(draining_relay_keeps_its_peers_until_it_restarts_inner); -} - -async fn draining_relay_keeps_its_peers_until_it_restarts_inner() { - let net = Net::new(); - let b = Relay::up(&net); - let (a, _a_route) = peer(&net, "a").await; - let (c, _c_route) = peer(&net, "c").await; +/// B with A and C peered to it, each over one session, past the healthy bar. +async fn mesh(net: &Net) -> (Relay, [(cluster::Cluster, moq_net::origin::Route); 2]) { + let b = Relay::up(net); + let (a, a_route) = peer(net, "a").await; + let (c, c_route) = peer(net, "c").await; // The mesh converges: each side learns the other's broadcasts, over one session each. learns("A learns b", &a.origin, "b").await; @@ -99,13 +120,30 @@ async fn draining_relay_keeps_its_peers_until_it_restarts_inner() { learns("B learns c", &b.cluster.origin, "c").await; kio::clock::sim::idle().await; assert_eq!(net.accepted("b"), 2); + kio::clock::sleep(HEALTHY).await; + (b, [(a, a_route), (c, c_route)]) +} + +#[test] +fn draining_relay_keeps_its_peers_until_it_restarts() { + memory::run(draining_relay_keeps_its_peers_until_it_restarts_inner); +} - // B drains for its whole window: its peers hold their sessions to the end. +async fn draining_relay_keeps_its_peers_until_it_restarts_inner() { + let net = Net::new(); + let (b, _peers) = mesh(&net).await; + + // B drains for its whole window. Each peer redials once, which reaches B again + // with nothing beside it, and holds that session to the end. b.trigger.start(); within("B exits after its drain", DRAIN * 2, b.exited) .await .expect("relay task"); - assert_eq!(net.accepted("b"), 2, "a peer redialed the draining relay"); + assert_eq!( + net.accepted("b"), + 4, + "a peer redialed the draining relay other than once" + ); // Nothing answers while B is down. Then its replacement comes up. kio::clock::sleep(Duration::from_secs(5)).await; @@ -116,5 +154,40 @@ async fn draining_relay_keeps_its_peers_until_it_restarts_inner() { learns("B' learns a", &b2.cluster.origin, "a").await; learns("B' learns c", &b2.cluster.origin, "c").await; kio::clock::sim::idle().await; - assert_eq!(net.accepted("b"), 4, "a peer dialed the replacement other than once"); + assert_eq!(net.accepted("b"), 6, "a peer dialed the replacement other than once"); +} + +#[test] +fn a_relay_beside_the_draining_one_takes_its_peers() { + memory::run(a_relay_beside_the_draining_one_takes_its_peers_inner); +} + +/// The upgrade a handoff makes: B' takes over B's address while B drains. Each peer +/// redials once and reaches B', so B exits as soon as they leave rather than holding +/// its drain open to the deadline. +async fn a_relay_beside_the_draining_one_takes_its_peers_inner() { + let net = Net::new(); + let (b, _peers) = mesh(&net).await; + + let b2 = Relay::beside(&net, &b); + let started = kio::clock::now(); + b.trigger.start(); + within("B exits once its peers leave", DRAIN, b.exited) + .await + .expect("relay task"); + assert!( + started.elapsed() < DRAIN / 2, + "B held its drain open for {:?}", + started.elapsed() + ); + + learns("B' learns a", &b2.cluster.origin, "a").await; + learns("B' learns c", &b2.cluster.origin, "c").await; + kio::clock::sim::idle().await; + assert_eq!(net.accepted("b"), 4, "a peer dialed B' other than once"); + + // B's exit left the address to B': a fresh peer still reaches it. + let (_d, _d_route) = peer(&net, "d").await; + learns("B' learns d", &b2.cluster.origin, "d").await; + assert_eq!(net.accepted("b"), 5, "the fresh peer did not reach B'"); } diff --git a/rs/moq-relay/tests/integration/handoff.rs b/rs/moq-relay/tests/integration/handoff.rs new file mode 100644 index 00000000000..826ac8010f4 --- /dev/null +++ b/rs/moq-relay/tests/integration/handoff.rs @@ -0,0 +1,633 @@ +//! Upgrading a relay in place: the new relay takes over the old one's sockets and +//! serves beside it while the old one drains. +//! +//! Linux-only, as handoff is: the QUIC port is handed on as a reuseport group, +//! steered by a socket filter. +#![cfg(all(target_os = "linux", feature = "_quic"))] + +use std::path::{Path, PathBuf}; +use std::time::Duration; + +use futures::FutureExt; +use moq_net::stats::{Presence, Registry, Tier}; +use moq_relay::{Config, Relay}; + +const TIMEOUT: Duration = Duration::from_secs(10); + +/// A self-signed certificate on disk, which both relays load: workers refuse +/// `listen.tls.generate`. +fn certificate(dir: &Path) -> (PathBuf, PathBuf) { + let key = rind_rcgen::KeyPair::generate(&rcgen::PKCS_ECDSA_P256_SHA256).expect("keypair"); + let mut params = rcgen::CertificateParams::new(vec!["localhost".to_string()]).expect("cert params"); + params.serial_number = Some(1u64.into()); + let cert = params.self_signed(&key).expect("self-signed cert"); + let cert_path = dir.join("cert.pem"); + let key_path = dir.join("key.pem"); + std::fs::write(&cert_path, cert.pem()).expect("write cert"); + std::fs::write( + &key_path, + pem::encode(&pem::Pem::new("PRIVATE KEY", key.to_pkcs8().secret_pkcs8_der())), + ) + .expect("write key"); + (cert_path, key_path) +} + +/// The QUIC backend a relay serves on. +#[derive(Clone, Copy, Debug)] +enum Backend { + Tokio, + #[cfg(feature = "_uring")] + Uring, +} + +/// One config for every relay, as an upgrade restarts with the same file: each +/// listener on an ephemeral port, which a relay taking over reads as "whatever +/// the old one bound". +fn config(dir: &Path, cert: &Path, key: &Path, backend: Backend) -> Config { + let mut config = Config::default(); + config.listen.bind = Some("127.0.0.1:0".parse().unwrap()); + config.listen.tls.cert = vec![cert.to_path_buf()]; + config.listen.tls.key = vec![key.to_path_buf()]; + config.listen.tcp.bind = Some("127.0.0.1:0".parse().unwrap()); + config.web.http.listen = Some("127.0.0.1:0".parse().unwrap()); + config.internal.listen = Some("127.0.0.1:0".parse().unwrap()); + config.runtime.workers = moq_relay::runtime::Workers::Count(std::num::NonZeroU16::new(2).unwrap()); + config.runtime.pin = false; + config.auth.public = vec![moq_auth::Pattern::all()]; + config.stats.enabled = true; + config.stats.node = Some("test".into()); + config.handoff.socket = Some(dir.join("handoff.sock")); + match backend { + Backend::Tokio => {} + #[cfg(feature = "_uring")] + Backend::Uring => config.runtime.io_uring = true, + } + config +} + +/// The sessions of the relay behind `registry`. +fn presence(registry: &Registry) -> Presence { + registry + .snapshot() + .sessions() + .into_iter() + .find(|(tier, _)| *tier == Tier::default()) + .map_or(Presence::default(), |(_, presence)| presence) +} + +/// Sessions the relay behind `registry` serves now. +fn active(registry: &Registry) -> u64 { + presence(registry).active() +} + +/// Wait until the relay behind `registry` serves `count` sessions. +async fn serving(registry: &Registry, count: u64, what: &str) { + kio::clock::timeout(TIMEOUT, async { + while active(registry) != count { + tokio::task::yield_now().await; + } + }) + .await + .unwrap_or_else(|_| panic!("{what}: serving {} sessions, not {count}", active(registry))); +} + +fn dial() -> moq_tokio::connect::Config { + let mut config = moq_tokio::connect::Config::default(); + config.tls.insecure = Some(true); + config.bind = Some("127.0.0.1:0".parse().unwrap()); + config +} + +async fn established(client: moq_tokio::Client, url: &str) -> moq_tokio::Connection { + kio::clock::timeout(TIMEOUT, client.connect(url.parse::().unwrap()).established()) + .await + .expect("connect timeout") + .expect("connect failed") +} + +/// A session that leaves on the relay's GOAWAY. +async fn connect(url: &str) -> moq_tokio::Connection { + established(dial().init(Default::default()).unwrap().with_reconnect(false), url).await +} + +/// A session that stays through the relay's drain until the relay closes it. +async fn stays(url: &str, quic: moq_tokio::quic::Config) -> moq_tokio::Connection { + established(dial().init(quic).unwrap().with_restart(moq_tokio::Restart::Wait), url).await +} + +/// Wait for the GOAWAY a draining relay sends `connection`. +async fn drained(connection: &moq_tokio::Connection) { + let draining = connection.draining().unwrap(); + let goaway = kio::clock::timeout(TIMEOUT, kio::wait(|waiter| draining.poll(waiter))) + .await + .expect("the old relay never drained") + .unwrap(); + assert_eq!(goaway.uri(), ""); +} + +#[tokio::test] +async fn a_new_relay_takes_over_while_the_old_one_drains() { + takes_over(Backend::Tokio, Backend::Tokio).await; +} + +#[cfg(feature = "_uring")] +#[tokio::test] +async fn an_io_uring_relay_takes_over() { + if crate::runtime_uring::supported() { + takes_over(Backend::Uring, Backend::Uring).await; + } +} + +/// Both backends hand on the same group, with connection IDs in the same layout. +#[cfg(feature = "_uring")] +#[tokio::test] +async fn a_relay_takes_over_across_backends() { + if crate::runtime_uring::supported() { + takes_over(Backend::Tokio, Backend::Uring).await; + } +} + +/// The new relay serves every listener the old one did, on the same sockets: new +/// sessions reach it while the old one's stay, until those leave and it exits. +/// Only then does the new relay publish the node's stats, and only then may a +/// third relay take over. +async fn takes_over(from: Backend, to: Backend) { + kio::clock::host_time(); // real relays on real sockets + moq_tokio::crypto::install_default().unwrap(); + let dir = tempfile::tempdir().unwrap(); + let (cert, key) = certificate(dir.path()); + + let old = Relay::load(config(dir.path(), &cert, &key, from)) + .await + .unwrap() + .with_signals(false); + let addr = old.addr().unwrap(); + let tcp = old.tcp_addr().unwrap(); + let http = old.web_addrs().http.unwrap(); + let internal = old.internal().addr().unwrap(); + let old_stats = old.cluster().stats.clone(); + // Holds the old relay's drain open, as a session that stays to the deadline would: + // a client leaves on GOAWAY at once. + let staying = old.shutdown().admission().enter().unwrap(); + let ready = old.ready(); + let old_running = tokio::spawn(old.run()); + ready.wait().await.unwrap(); + + let quic = format!("https://{addr}/handoff"); + let held = connect(&quic).await; + serving(&old_stats, 1, "the old relay").await; + + let new = Relay::load(config(dir.path(), &cert, &key, to)) + .await + .unwrap() + .with_signals(false); + assert_eq!(new.addr(), Some(addr)); + assert_eq!(new.tcp_addr(), Some(tcp)); + assert_eq!(new.web_addrs().http, Some(http)); + assert_eq!(new.internal().addr(), Some(internal)); + let new_stats = new.cluster().stats.clone(); + let new_origin = new.cluster().origin.consume(); + let trigger = new.shutdown_trigger().clone(); + let ready = new.ready(); + let new_running = tokio::spawn(new.run()); + ready.wait().await.unwrap(); + + // The old relay drains once the new one serves. + drained(&held).await; + + // New sessions, by either transport, reach the new relay. + let fresh = connect(&quic).await; + let stream = connect(&format!("tcp://{tcp}/handoff")).await; + serving(&new_stats, 2, "the new relay").await; + assert_eq!( + presence(&old_stats).sessions_started, + 1, + "a new session reached the draining relay" + ); + + // The group has room for two generations, both in use. + let refused = Relay::load(config(dir.path(), &cert, &key, to)) + .await + .err() + .expect("a third relay took over while the old one served"); + assert!(format!("{refused:#}").contains("refused"), "{refused:#}"); + + // One instance of the node's stats at a time: the old relay's, so far. + let mut stats = new_origin.follow(".stats/node/test", None).unwrap(); + assert!( + stats.available().now_or_never().is_none(), + "published the node's stats beside the old relay" + ); + + drop((held, staying)); + kio::clock::timeout(TIMEOUT, old_running) + .await + .expect("the old relay outlived its last session") + .unwrap() + .unwrap(); + kio::clock::timeout(TIMEOUT, stats.available()) + .await + .expect("the node's stats stayed held once the old relay exited") + .unwrap() + .unwrap(); + + // Every handshake reaches the new relay now, and a third may take over. + let last = connect(&quic).await; + serving(&new_stats, 3, "the new relay alone").await; + + trigger.start(); + drop((fresh, stream, last)); + kio::clock::timeout(TIMEOUT, new_running) + .await + .expect("the new relay outlived its sessions") + .unwrap() + .unwrap(); +} + +/// A new relay that dies while the old one drains takes none of the old one's +/// sessions with it: they still reach the old relay, and the next relay takes over +/// from it on the same sockets. +#[tokio::test] +async fn the_old_relay_outlives_a_new_one_that_dies() { + kio::clock::host_time(); // real relays on real sockets + moq_tokio::crypto::install_default().unwrap(); + let dir = tempfile::tempdir().unwrap(); + let (cert, key) = certificate(dir.path()); + // Neither an idle session nor the drain deadline ends a session within the test's + // timeout, so only a close that reaches the old relay ends one there. + let mut quic = moq_tokio::quic::Config::default(); + quic.idle_timeout = Some(Duration::from_secs(600).into()); + let config = || { + let mut config = config(dir.path(), &cert, &key, Backend::Tokio); + config.quic = quic.clone(); + config.drain_timeout = Some(Duration::from_secs(600).into()); + config + }; + + let old = Relay::load(config()).await.unwrap().with_signals(false); + let addr = old.addr().unwrap(); + let tcp = old.tcp_addr().unwrap(); + let old_stats = old.cluster().stats.clone(); + let staying = old.shutdown().admission().enter().unwrap(); + let ready = old.ready(); + let old_running = tokio::spawn(old.run()); + ready.wait().await.unwrap(); + + let quic_url = format!("https://{addr}/handoff"); + let first = stays(&quic_url, quic.clone()).await; + let second = stays(&quic_url, quic.clone()).await; + serving(&old_stats, 2, "the old relay").await; + + let new = Relay::load(config()).await.unwrap().with_signals(false); + let ready = new.ready(); + let new_running = tokio::spawn(new.run()); + ready.wait().await.unwrap(); + drained(&first).await; + + // The new relay dies, and every socket it held goes with it. + new_running.abort(); + assert!(new_running.await.unwrap_err().is_cancelled()); + + // A session of the old relay still reaches it: its close arrives there. + drop(first); + serving(&old_stats, 1, "the old relay after the new one died").await; + + // The next relay takes over from the old one, on the same sockets. + let next = Relay::load(config()) + .await + .expect("the old relay refused the next one") + .with_signals(false); + assert_eq!(next.addr(), Some(addr)); + assert_eq!(next.tcp_addr(), Some(tcp)); + let next_stats = next.cluster().stats.clone(); + let trigger = next.shutdown_trigger().clone(); + let ready = next.ready(); + let next_running = tokio::spawn(next.run()); + ready.wait().await.unwrap(); + let fresh = connect(&quic_url).await; + let stream = connect(&format!("tcp://{tcp}/handoff")).await; + serving(&next_stats, 2, "the next relay").await; + + drop((second, staying)); + kio::clock::timeout(TIMEOUT, old_running) + .await + .expect("the old relay outlived its last session") + .unwrap() + .unwrap(); + + trigger.start(); + drop((fresh, stream)); + kio::clock::timeout(TIMEOUT, next_running) + .await + .expect("the next relay outlived its sessions") + .unwrap() + .unwrap(); +} + +/// A stand-in that takes its relay with it when dropped, such as on a failed +/// assertion; a relay it already left is [`Reap`]'s. +struct StandIn(tokio::process::Child); + +impl Drop for StandIn { + fn drop(&mut self) { + if let Some(pid) = self.0.id() { + for relay in children(pid) { + // SAFETY: kill has no memory-safety preconditions. + unsafe { libc::kill(relay, libc::SIGKILL) }; + } + } + } +} + +impl std::ops::Deref for StandIn { + type Target = tokio::process::Child; + fn deref(&self) -> &Self::Target { + &self.0 + } +} + +impl std::ops::DerefMut for StandIn { + fn deref_mut(&mut self) -> &mut Self::Target { + &mut self.0 + } +} + +/// A relay a stand-in left to this test, killed if the test ends before reaping it. +struct Reap(Option); + +impl Reap { + /// Wait for the relay to exit, reaping it so the guard never signals a reused pid. + async fn exited(&mut self, what: &str) -> libc::c_int { + let relay = self.0.expect("reaped already"); + let status = tokio::task::spawn_blocking(move || { + let mut status = 0; + // SAFETY: waitpid writes only to `status`. + let reaped = unsafe { libc::waitpid(relay, &mut status, 0) }; + assert_eq!(reaped, relay); + status + }); + let status = kio::clock::timeout(TIMEOUT, status) + .await + .unwrap_or_else(|_| panic!("{what} kept serving")) + .unwrap(); + self.0 = None; + status + } +} + +impl Drop for Reap { + fn drop(&mut self) { + if let Some(relay) = self.0 { + // SAFETY: kill has no memory-safety preconditions; the pid is not yet reaped, + // so it is still the relay's. + unsafe { libc::kill(relay, libc::SIGKILL) }; + } + } +} + +/// Start `moq-relay --handoff-supervise` with its config in the environment, which +/// is all it reads: the test's own would reach it too. +fn stand_in(dir: &Path, cert: &Path, key: &Path, notify: &Path) -> StandIn { + stand_in_draining(dir, cert, key, notify, "10s") +} + +/// [`stand_in`], with a `drain_timeout` of `drain`. +fn stand_in_draining(dir: &Path, cert: &Path, key: &Path, notify: &Path, drain: &str) -> StandIn { + let child = tokio::process::Command::new(env!("CARGO_BIN_EXE_moq-relay")) + .arg("--handoff-supervise") + .env_clear() + .env("MOQ_LISTEN", "127.0.0.1:0") + .env("MOQ_LISTEN_TLS_CERT", cert) + .env("MOQ_LISTEN_TLS_KEY", key) + .env("MOQ_RUNTIME_WORKERS", "1") + .env("MOQ_AUTH_PUBLIC", "**") + .env("MOQ_HANDOFF_SOCKET", dir.join("handoff.sock")) + .env("NOTIFY_SOCKET", notify) + .env("MOQ_DRAIN_TIMEOUT", drain) + .kill_on_drop(true) + .spawn() + .expect("start the stand-in"); + StandIn(child) +} + +/// A restart whose next relay never takes over, such as one that fails to start, +/// leaves the old relay outside the service: it drains once `drain_timeout` +/// passes with no successor, rather than holding the port forever. +#[tokio::test] +async fn a_relay_no_successor_takes_over_from_drains() { + kio::clock::host_time(); // real relays in real processes + // SAFETY: prctl with these arguments has no memory-safety preconditions. + assert_eq!(unsafe { libc::prctl(libc::PR_SET_CHILD_SUBREAPER, 1, 0, 0, 0) }, 0); + + let dir = tempfile::tempdir().unwrap(); + let (cert, key) = certificate(dir.path()); + let notify_path = dir.path().join("notify"); + let notify = tokio::net::UnixDatagram::bind(¬ify_path).unwrap(); + + let mut stand_in = stand_in_draining(dir.path(), &cert, &key, ¬ify_path, "1s"); + ready(¬ify).await; + let relays = children(stand_in.id().unwrap()); + assert_eq!(relays.len(), 1, "the stand-in runs one relay: {relays:?}"); + let mut left = Reap(Some(relays[0])); + + let started = kio::clock::now(); + // SAFETY: kill has no memory-safety preconditions. + assert_eq!( + unsafe { libc::kill(stand_in.id().unwrap() as libc::pid_t, libc::SIGUSR2) }, + 0 + ); + let status = exited(&mut stand_in, "the stand-in").await; + assert_eq!(status.code(), Some(0), "{status}"); + + let status = left.exited("a relay no successor took over from").await; + assert!( + libc::WIFEXITED(status) && libc::WEXITSTATUS(status) == 0, + "the relay exited with {status:#x}" + ); + assert!( + started.elapsed() >= Duration::from_secs(1), + "drained before its window for a successor passed" + ); +} + +/// Start a relay process with its config in the environment, serving QUIC on `port` +/// with a shared reset key, and sessions that outlast the test unless closed. +fn relay(dir: &Path, cert: &Path, key: &Path, port: u16, notify: &Path) -> tokio::process::Child { + tokio::process::Command::new(env!("CARGO_BIN_EXE_moq-relay")) + .env_clear() + .env("MOQ_LISTEN", format!("127.0.0.1:{port}")) + .env("MOQ_LISTEN_TLS_CERT", cert) + .env("MOQ_LISTEN_TLS_KEY", key) + .env("MOQ_LISTEN_RESET_SECRET", dir.join("reset.key")) + .env("MOQ_QUIC_IDLE_TIMEOUT", "600s") + .env("MOQ_DRAIN_TIMEOUT", "600s") + .env("MOQ_RUNTIME_WORKERS", "2") + .env("MOQ_AUTH_PUBLIC", "**") + .env("MOQ_HANDOFF_SOCKET", dir.join("handoff.sock")) + .env("NOTIFY_SOCKET", notify) + .kill_on_drop(true) + .spawn() + .expect("start the relay") +} + +/// A new relay killed while the old one drains leaves its clients to the old one, +/// which takes their packets back and, sharing the reset key, answers them with a +/// stateless reset, so they reconnect rather than wait out their idle timeout. +#[tokio::test] +async fn a_killed_relay_s_clients_are_reset_by_the_one_it_replaced() { + kio::clock::host_time(); // real relays in real processes + moq_tokio::crypto::install_default().unwrap(); + let dir = tempfile::tempdir().unwrap(); + let (cert, key) = certificate(dir.path()); + std::fs::write(dir.path().join("reset.key"), "ab".repeat(32)).unwrap(); + let notify_path = dir.path().join("notify"); + let notify = tokio::net::UnixDatagram::bind(¬ify_path).unwrap(); + let port = std::net::UdpSocket::bind("127.0.0.1:0") + .unwrap() + .local_addr() + .unwrap() + .port(); + let url = format!("https://127.0.0.1:{port}/handoff"); + let mut quic = moq_tokio::quic::Config::default(); + quic.idle_timeout = Some(Duration::from_secs(600).into()); + // Keeps sending, so a reset comes back at once. + quic.keep_alive = Some(Duration::from_millis(200).into()); + + let _old = relay(dir.path(), &cert, &key, port, ¬ify_path); + ready(¬ify).await; + // Holds the old relay in its drain once the new one takes over. + let _kept = stays(&url, quic.clone()).await; + + let mut new = relay(dir.path(), &cert, &key, port, ¬ify_path); + ready(¬ify).await; + let mut orphan = stays(&url, quic).await; + + new.kill().await.unwrap(); + kio::clock::timeout(TIMEOUT, async { + while orphan.status().await.unwrap() != moq_tokio::Status::Disconnected {} + }) + .await + .expect("the killed relay's client was never reset"); +} + +/// Wait for a READY notification on `notify`, as a service manager does. +async fn ready(notify: &tokio::net::UnixDatagram) { + let mut message = [0; 1024]; + kio::clock::timeout(TIMEOUT, async { + loop { + let len = notify.recv(&mut message).await.unwrap(); + if std::str::from_utf8(&message[..len]) + .unwrap() + .lines() + .any(|line| line == "READY=1") + { + return; + } + } + }) + .await + .expect("no READY reached the service manager"); +} + +/// The processes whose parent is `parent`. +fn children(parent: u32) -> Vec { + std::fs::read_dir("/proc") + .unwrap() + .filter_map(|entry| { + let pid: libc::pid_t = entry.ok()?.file_name().to_str()?.parse().ok()?; + let stat = std::fs::read_to_string(format!("/proc/{pid}/stat")).ok()?; + // The command name is parenthesized and may hold anything; the parent's pid + // is the second field after it. + let rest = stat.get(stat.rfind(')')? + 2..)?; + let ppid: u32 = rest.split(' ').nth(1)?.parse().ok()?; + (ppid == parent).then_some(pid) + }) + .collect() +} + +async fn exited(child: &mut tokio::process::Child, what: &str) -> std::process::ExitStatus { + kio::clock::timeout(TIMEOUT, child.wait()) + .await + .unwrap_or_else(|_| panic!("{what} never exited")) + .unwrap() +} + +/// What a service manager does across a restart with `KillMode=process` and +/// `RestartKillSignal=SIGUSR2`: the stand-in exits on SIGUSR2 and leaves its relay +/// serving, the next stand-in's relay takes over and reports ready through it, and +/// the old relay exits once drained. A stop's SIGTERM drains the relay and exits. +#[tokio::test] +async fn a_restart_starts_the_new_relay_beside_the_old() { + kio::clock::host_time(); // real relays in real processes + // Adopt the relay a stand-in leaves behind, as the service manager does, so its + // exit is this test's to wait on. + // SAFETY: prctl with these arguments has no memory-safety preconditions. + assert_eq!(unsafe { libc::prctl(libc::PR_SET_CHILD_SUBREAPER, 1, 0, 0, 0) }, 0); + + let dir = tempfile::tempdir().unwrap(); + let (cert, key) = certificate(dir.path()); + let notify_path = dir.path().join("notify"); + let notify = tokio::net::UnixDatagram::bind(¬ify_path).unwrap(); + + let mut old = stand_in(dir.path(), &cert, &key, ¬ify_path); + ready(¬ify).await; + let relays = children(old.id().unwrap()); + assert_eq!(relays.len(), 1, "the stand-in runs one relay: {relays:?}"); + let relay = relays[0]; + let mut left = Reap(Some(relay)); + + // The restart's stop: the stand-in leaves, its relay stays. + // SAFETY: kill has no memory-safety preconditions. + assert_eq!( + unsafe { libc::kill(old.id().unwrap() as libc::pid_t, libc::SIGUSR2) }, + 0 + ); + let status = exited(&mut old, "the old stand-in").await; + assert_eq!(status.code(), Some(0), "{status}"); + let mut status = 0; + // SAFETY: waitpid writes only to `status`. + let reaped = unsafe { libc::waitpid(relay, &mut status, libc::WNOHANG) }; + assert_eq!(reaped, 0, "the old relay exited with its stand-in"); + + // The restart's start: the new relay takes over, and the old one drains. + let mut new = stand_in(dir.path(), &cert, &key, ¬ify_path); + ready(¬ify).await; + let status = left.exited("the old relay, after the new one took over,").await; + assert!( + libc::WIFEXITED(status) && libc::WEXITSTATUS(status) == 0, + "the old relay exited with {status:#x}" + ); + + // A stop drains the relay, and the stand-in exits with it. + // SAFETY: kill has no memory-safety preconditions. + assert_eq!( + unsafe { libc::kill(new.id().unwrap() as libc::pid_t, libc::SIGTERM) }, + 0 + ); + let status = exited(&mut new, "the new stand-in").await; + assert_eq!(status.code(), Some(0), "{status}"); + + // SIGUSR1, a heap profile's signal, reaches the relay rather than ending the + // stand-in. Built without jemalloc, the relay dies of it, and the stand-in exits + // as a shell reports that. + #[cfg(not(feature = "jemalloc"))] + { + let mut last = stand_in(dir.path(), &cert, &key, ¬ify_path); + ready(¬ify).await; + // SAFETY: kill has no memory-safety preconditions. + assert_eq!( + unsafe { libc::kill(last.id().unwrap() as libc::pid_t, libc::SIGUSR1) }, + 0 + ); + let status = exited(&mut last, "the last stand-in").await; + assert_eq!(status.code(), Some(128 + libc::SIGUSR1), "{status}"); + } + + // Each stand-in removed the socket its relay notified it on. + let left: Vec<_> = std::fs::read_dir(dir.path()) + .unwrap() + .map(|entry| entry.unwrap().file_name()) + .filter(|name| name.to_string_lossy().ends_with(".notify")) + .collect(); + assert!(left.is_empty(), "{left:?}"); +} diff --git a/rs/moq-relay/tests/integration/main.rs b/rs/moq-relay/tests/integration/main.rs index 63182023c65..244392033ef 100644 --- a/rs/moq-relay/tests/integration/main.rs +++ b/rs/moq-relay/tests/integration/main.rs @@ -23,6 +23,7 @@ mod drain; mod embed; #[cfg(feature = "test-support")] mod goaway_cluster; +mod handoff; mod hidden_cluster; mod implementation; mod lan_mesh; diff --git a/rs/moq-sock/README.md b/rs/moq-sock/README.md index 7e5a44d8b59..e3df2c04d76 100644 --- a/rs/moq-sock/README.md +++ b/rs/moq-sock/README.md @@ -14,7 +14,13 @@ runtimes (`moq-tokio` and `moq-uring`). A `Shard` names the slot a member holds, which the endpoint serving it names in every connection id it issues, alongside a marker no SRT data header carries (`moq_quic::RandomConnectionIdGenerator::set_shard`). Anything else - on the port, including other protocols, keeps the kernel's 4-tuple hash. + on the port, including other protocols, hashes its source address and port + onto one member of the newest generation. With `Config::handoff` the group + holds a second generation for the process that replaces this one, which + adopts every member by file descriptor and serves beside it, each + generation keeping its own connections; `Bound::steering` sends new + handshakes to either. +- **`fd`**: passing open sockets to another process over a Unix stream socket. - **`cpu`**: pinning worker threads to cores. - **`demux`**: classifying each datagram on a shared port by its first byte (RFC 7983), so QUIC and the WebRTC media path (ICE, DTLS, RTP) share diff --git a/rs/moq-sock/src/fd.rs b/rs/moq-sock/src/fd.rs new file mode 100644 index 00000000000..0c2c765fce8 --- /dev/null +++ b/rs/moq-sock/src/fd.rs @@ -0,0 +1,178 @@ +//! Passing open file descriptors between processes over a Unix stream socket. +//! +//! The kernel copies each descriptor sent this way into the receiver, which +//! then holds the same open socket the sender does: a listener or a reuseport +//! member keeps its place, its queue, and its connections. A process hands its +//! sockets to its successor this way (see [`crate::shard::Handoff`]). +//! +//! Both calls work on non-blocking sockets and report +//! [`io::ErrorKind::WouldBlock`], so an async runtime can drive them from its +//! readiness loop. + +use std::io; +use std::os::fd::{AsRawFd, BorrowedFd, FromRawFd, OwnedFd}; + +/// The most descriptors one [`send`] may carry: the kernel's `SCM_MAX_FD`. +pub const MAX: usize = 253; + +/// Space for one `SCM_RIGHTS` message carrying [`MAX`] descriptors. +const CONTROL: usize = unsafe { libc::CMSG_SPACE((MAX * size_of::()) as u32) } as usize; + +/// Send `data` with `fds` attached, returning how many bytes went out. +/// +/// The descriptors arrive with the first byte the receiver reads of this call, +/// even when the write is partial, so resend the rest without them. At most +/// [`MAX`] descriptors, and at least one byte of `data` to carry them. +pub fn send(socket: BorrowedFd<'_>, data: &[u8], fds: &[BorrowedFd<'_>]) -> io::Result { + if fds.len() > MAX { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + format!("{} descriptors in one message; the kernel takes {MAX}", fds.len()), + )); + } + if data.is_empty() && !fds.is_empty() { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "descriptors need at least one byte to ride on", + )); + } + + let mut iov = libc::iovec { + iov_base: data.as_ptr() as *mut libc::c_void, + iov_len: data.len(), + }; + let mut control = Control([0; CONTROL]); + // SAFETY: an all-zero msghdr is a valid empty message. + let mut msg: libc::msghdr = unsafe { std::mem::zeroed() }; + msg.msg_iov = &mut iov; + msg.msg_iovlen = 1; + + if !fds.is_empty() { + let len = fds.len() * size_of::(); + msg.msg_control = control.0.as_mut_ptr().cast(); + // SAFETY: CMSG_SPACE is a pure size computation. + msg.msg_controllen = unsafe { libc::CMSG_SPACE(len as u32) } as _; + // SAFETY: `msg_control` points at `control`, which has room for one + // header and `fds.len()` descriptors, so the first header and its data + // are in bounds. + unsafe { + let header = libc::CMSG_FIRSTHDR(&msg); + (*header).cmsg_level = libc::SOL_SOCKET; + (*header).cmsg_type = libc::SCM_RIGHTS; + (*header).cmsg_len = libc::CMSG_LEN(len as u32) as _; + let data = libc::CMSG_DATA(header).cast::(); + for (index, fd) in fds.iter().enumerate() { + data.add(index).write_unaligned(fd.as_raw_fd()); + } + } + } + + // SAFETY: `msg` points at `iov` and `control`, both alive for the call. + let sent = unsafe { libc::sendmsg(socket.as_raw_fd(), &msg, libc::MSG_NOSIGNAL) }; + match sent { + -1 => Err(io::Error::last_os_error()), + sent => Ok(sent as usize), + } +} + +/// Receive into `buf`, appending any descriptors that arrived with it to `fds`. +/// +/// Returns how many bytes were read; zero means the peer closed. Each received +/// descriptor is close-on-exec. A message whose descriptors did not fit is an +/// error, since the kernel has already closed the ones it dropped. +pub fn recv(socket: BorrowedFd<'_>, buf: &mut [u8], fds: &mut Vec) -> io::Result { + let mut iov = libc::iovec { + iov_base: buf.as_mut_ptr().cast(), + iov_len: buf.len(), + }; + let mut control = Control([0; CONTROL]); + // SAFETY: an all-zero msghdr is a valid empty message. + let mut msg: libc::msghdr = unsafe { std::mem::zeroed() }; + msg.msg_iov = &mut iov; + msg.msg_iovlen = 1; + msg.msg_control = control.0.as_mut_ptr().cast(); + msg.msg_controllen = CONTROL as _; + + // SAFETY: `msg` points at `buf` and `control`, both alive for the call. + let read = unsafe { libc::recvmsg(socket.as_raw_fd(), &mut msg, libc::MSG_CMSG_CLOEXEC) }; + if read == -1 { + return Err(io::Error::last_os_error()); + } + + // Take ownership of everything that arrived before judging the message, so an + // error below still closes them. + // SAFETY: the kernel filled `msg_control` with `msg_controllen` bytes of + // well-formed headers, which the CMSG macros walk within those bounds. + unsafe { + let mut header = libc::CMSG_FIRSTHDR(&msg); + while !header.is_null() { + if (*header).cmsg_level == libc::SOL_SOCKET && (*header).cmsg_type == libc::SCM_RIGHTS { + let data = libc::CMSG_DATA(header).cast::(); + let len = (*header).cmsg_len as usize - libc::CMSG_LEN(0) as usize; + for index in 0..len / size_of::() { + fds.push(OwnedFd::from_raw_fd(data.add(index).read_unaligned())); + } + } + header = libc::CMSG_NXTHDR(&msg, header); + } + } + if msg.msg_flags & libc::MSG_CTRUNC != 0 { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "a message carried more descriptors than fit", + )); + } + Ok(read as usize) +} + +/// A control buffer aligned for `cmsghdr`, as the CMSG macros require. +#[repr(C, align(8))] +struct Control([u8; CONTROL]); + +#[cfg(test)] +mod tests { + use super::*; + use std::io::{Read, Write}; + use std::os::fd::AsFd; + use std::os::unix::net::UnixStream; + + /// A descriptor sent across arrives as the same open file: what one side + /// writes, the other reads. + #[test] + fn a_descriptor_arrives_as_the_same_open_file() { + let (left, right) = UnixStream::pair().unwrap(); + let (mut ours, theirs) = UnixStream::pair().unwrap(); + assert_eq!(send(left.as_fd(), b"x", &[theirs.as_fd()]).unwrap(), 1); + drop(theirs); + + let mut buf = [0; 8]; + let mut fds = Vec::new(); + assert_eq!(recv(right.as_fd(), &mut buf, &mut fds).unwrap(), 1); + assert_eq!(&buf[..1], b"x"); + assert_eq!(fds.len(), 1); + + let mut received = UnixStream::from(fds.pop().unwrap()); + received.write_all(b"hello").unwrap(); + let mut got = [0; 5]; + ours.read_exact(&mut got).unwrap(); + assert_eq!(&got, b"hello"); + } + + /// The kernel's limit is enforced before anything is sent, and the most it + /// allows arrives whole. + #[test] + fn the_most_descriptors_arrive_whole() { + let (left, right) = UnixStream::pair().unwrap(); + let files: Vec<_> = (0..=MAX).map(|_| UnixStream::pair().unwrap().0).collect(); + let borrowed: Vec<_> = files.iter().map(AsFd::as_fd).collect(); + assert_eq!( + send(left.as_fd(), b"x", &borrowed).unwrap_err().kind(), + io::ErrorKind::InvalidInput + ); + assert_eq!(send(left.as_fd(), b"x", &borrowed[..MAX]).unwrap(), 1); + + let mut fds = Vec::new(); + recv(right.as_fd(), &mut [0; 1], &mut fds).unwrap(); + assert_eq!(fds.len(), MAX); + } +} diff --git a/rs/moq-sock/src/lib.rs b/rs/moq-sock/src/lib.rs index 6074e1c7fda..c2036f2796b 100644 --- a/rs/moq-sock/src/lib.rs +++ b/rs/moq-sock/src/lib.rs @@ -6,12 +6,16 @@ //! id, so a worker-per-core listener keeps each connection on the socket that //! owns it; [`cpu`] pins those workers; [`udp`] (forked from quinn-udp) sends //! and receives with GSO, GRO, and ECN; [`demux`] splits one socket's datagrams -//! between QUIC and the WebRTC media path. Both `moq-tokio` and `moq-uring` build -//! their worker groups on this crate, so the group-formation invariants live -//! here once, in the group itself rather than in each caller. +//! between QUIC and the WebRTC media path; [`fd`] passes open sockets to a +//! successor process, which serves a handed-off group beside its predecessor. +//! Both `moq-tokio` and `moq-uring` build their worker groups on this crate, so +//! the group-formation invariants live here once, in the group itself rather +//! than in each caller. pub mod bind; pub mod cpu; pub mod demux; +#[cfg(target_os = "linux")] +pub mod fd; pub mod shard; pub mod udp; diff --git a/rs/moq-sock/src/shard.rs b/rs/moq-sock/src/shard.rs index 47896662fc3..390050b222d 100644 --- a/rs/moq-sock/src/shard.rs +++ b/rs/moq-sock/src/shard.rs @@ -36,7 +36,8 @@ //! 8-byte connection IDs whose first byte is congruent to the member's //! [`Shard::index`] modulo [`Shard::count`], and whose fourth byte has both //! bits of `0x18` set, a pattern no valid SRT data header carries, so a short -//! header from another protocol on the port keeps the 4-tuple hash. +//! header from another protocol on the port is balanced by its source address +//! and port instead, one flow to one member. //! `moq_quic::RandomConnectionIdGenerator::set_shard` issues exactly those. //! The kernel selects `socks[A % count]` for a returned accumulator `A`, so //! feeding it that byte lands on the owner. @@ -56,9 +57,32 @@ //! BPF toolchain in the build, and no map to keep alive across restarts. The //! cost is that the kernel selects by *position*, so the group has to be bound //! once, in order, and never resized, which is why [`bind`] does all of it. +//! +//! # Handing the port to a successor +//! +//! A group bound with [`Config::handoff`] holds two generations of members, one +//! per process, so an upgraded process can serve the port beside the one it +//! replaces. Each process serves one generation and issues connection IDs whose +//! index names a member of it, so the predecessor's connections keep reaching +//! the predecessor. The successor takes every member by file descriptor +//! ([`Bound::handoff`], [`Bound::adopt`]) rather than binding its own: both +//! processes then hold every member, so neither one exiting closes a socket and +//! renumbers the survivors, which is what positional steering cannot survive. +//! +//! A client's first Initial carries a connection ID it invented, which names no +//! member, so those packets go to whichever generation last called +//! [`Bound::steer`]: the newest process takes every new handshake. A Handshake +//! or short-header packet carries an ID the server chose and reaches its exact +//! member while two generations serve ([`Steer::Beside`]), or the newest +//! generation's member of the same position once the other is gone +//! ([`Steer::Alone`]), which answers it with a stateless reset (see +//! [`Shard::generations`]) so its client reconnects rather than waiting out its +//! idle timeout. use std::io; use std::net::{SocketAddr, UdpSocket}; +#[cfg(target_os = "linux")] +use std::os::fd::OwnedFd; /// One member's slot in a group of sockets sharing a port via `SO_REUSEPORT`. /// @@ -75,6 +99,7 @@ use std::net::{SocketAddr, UdpSocket}; pub struct Shard { index: u16, count: u16, + generations: u16, } impl Shard { @@ -87,6 +112,14 @@ impl Shard { pub fn count(self) -> u16 { self.count } + + /// How many processes' generations split the group evenly: two when bound for a + /// [handoff](self#handing-the-port-to-a-successor), else one. A connection ID naming + /// this member's position in another generation reaches this member only once that + /// generation is gone, so it is this member's to reset. + pub fn generations(self) -> u16 { + self.generations + } } /// Why a reuseport group could not be formed. @@ -134,63 +167,119 @@ pub enum Error { /// The steering filter could not be attached. #[error("failed to steer the reuseport group")] Steer(#[source] io::Error), + + /// A handed-off group did not describe the sockets it carried. + #[error("cannot adopt the handed-off reuseport group: {0}")] + Adopt(String), } -/// Bind a steered group of `count` sockets sharing `addr`. -/// -/// Everything a steered group has to get right happens here rather than in its -/// caller, because none of it is visible in a socket afterwards: -/// -/// - The port is locked before the first member binds and stays locked until the -/// returned group and every socket it handed out are gone, so a second -/// same-UID group cannot interleave into this one. Hold the group for as long -/// as its sockets are served. -/// - The member count is checked once, against what the steering filter can -/// address, and never changes. -/// - Members bind in index order, which is the order the kernel numbers them -/// by, and the filter is attached before any socket is released. -/// -/// A `count` of zero is a group of one: a lone member is a valid group. -/// -/// An ephemeral port (`0`) is resolved first, with a plain bind that is -/// dropped again, and the group then takes that port like a named one. -/// Never a reuseport bind of port `0`: Linux counts a port held only by -/// same-UID reuseport sockets as free for one, so the kernel could hand -/// back a port another group holds and this one would silently join it. A -/// plain bind conflicts with every holder. Another socket taking the port -/// before the first member binds fails that bind with -/// [`io::ErrorKind::AddrInUse`]. +/// Bind a steered group of `count` sockets sharing `addr`, with no room for a +/// successor: [`Config::bind`] with [`Config::handoff`] off. pub fn bind(addr: SocketAddr, count: u16) -> Result { - let count = count.max(1); - if count > MAX_SHARDS { - return Err(Error::Count { count, max: MAX_SHARDS }); + Config::new(addr, count).bind() +} + +/// The shape of a steered group, for [`Config::bind`]. +#[derive(Clone, Copy, Debug)] +#[non_exhaustive] +pub struct Config { + /// The address every member binds. Port `0` resolves an ephemeral port first. + pub addr: SocketAddr, + /// How many members this process serves. Zero is a group of one. + pub count: u16, + /// Also bind a second generation of `count` members, held in reserve for a + /// successor process to [`adopt`](Bound::adopt) and serve beside this one. + /// Doubles the members the steering filter has to address. Defaults to off. + pub handoff: bool, +} + +impl Config { + /// A group of `count` members on `addr`, with no room for a successor. + pub fn new(addr: SocketAddr, count: u16) -> Self { + Self { + addr, + count, + handoff: false, + } } - let addr = match addr.port() { - 0 => crate::bind::udp(crate::bind::Udp::new(addr)) - .and_then(|socket| socket.local_addr()) - .map_err(Error::Resolve)?, - _ => addr, - }; + /// Bind the group and steer it. + /// + /// Everything a steered group has to get right happens here rather than in its + /// caller, because none of it is visible in a socket afterwards: + /// + /// - The port is locked before the first member binds and stays locked until the + /// returned group and every socket it handed out are gone, so a second + /// same-UID group cannot interleave into this one. Hold the group for as long + /// as its sockets are served. + /// - The member count is checked once, against what the steering filter can + /// address, and never changes. + /// - Members bind in index order, which is the order the kernel numbers them + /// by, and the filter is attached before any socket is released. + /// + /// This process serves the first generation and takes every new handshake. + /// + /// An ephemeral port (`0`) is resolved first, with a plain bind that is + /// dropped again, and the group then takes that port like a named one. + /// Never a reuseport bind of port `0`: Linux counts a port held only by + /// same-UID reuseport sockets as free for one, so the kernel could hand + /// back a port another group holds and this one would silently join it. A + /// plain bind conflicts with every holder. Another socket taking the port + /// before the first member binds fails that bind with + /// [`io::ErrorKind::AddrInUse`]. + pub fn bind(self) -> Result { + let count = self.count.max(1); + let generations = match self.handoff { + true => 2, + false => 1, + }; + let total = count.saturating_mul(generations); + if u32::from(count) * u32::from(generations) > u32::from(MAX_SHARDS) { + return Err(Error::Count { + count: total, + max: MAX_SHARDS, + }); + } - let port = addr.port(); - let lock = Lock::acquire(port).map_err(|_| Error::Overlap { port })?; + let addr = match self.addr.port() { + 0 => crate::bind::udp(crate::bind::Udp::new(self.addr)) + .and_then(|socket| socket.local_addr()) + .map_err(Error::Resolve)?, + _ => self.addr, + }; - let mut sockets = Vec::with_capacity(usize::from(count)); - for index in 0..count { - let socket = member(addr, index).map_err(|source| Error::Bind { index, source })?; - sockets.push((Shard { index, count }, socket)); - } + let port = addr.port(); + let lock = Lock::acquire(port).map_err(|_| Error::Overlap { port })?; + + let mut slots = Vec::with_capacity(usize::from(total)); + for index in 0..total { + slots.push(member(addr, index).map_err(|source| Error::Bind { index, source })?); + } - let (_, last) = sockets.last().expect("a group always has at least one member"); - attach(last, count).map_err(Error::Steer)?; + let group = Bound { + slots, + count, + generation: 0, + next: 0, + addr, + _lock: lock, + }; + // The last member to bind is the first point the filter covers the whole array. + group.steer(Steer::Alone).map_err(Error::Steer)?; + Ok(group) + } +} - Ok(Bound { - sockets, - next: 0, - addr, - _lock: lock, - }) +/// Which connections the newest generation receives, set by [`Bound::steer`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Steer { + /// Another generation still serves its connections: every connection ID a + /// server chose reaches the member that issued it. + Beside, + /// No other generation serves: every packet reaches this generation, and one + /// for a connection the other generation issued is refused by its connection + /// ID generator rather than queued on a socket nobody reads. + Alone, } /// A complete steered group that retains every member socket. @@ -198,11 +287,17 @@ pub fn bind(addr: SocketAddr, count: u16) -> Result { /// [`member`](Self::member) returns duplicate handles to the retained sockets. /// Dropping one handle stops serving that slot without removing it from the /// kernel group or renumbering any survivor. Dropping this group closes every -/// retained member together and releases the port. +/// retained member together and releases the port, unless a successor that +/// [`adopt`](Self::adopt)ed it still holds them. #[derive(Debug)] pub struct Bound { - sockets: Vec<(Shard, UdpSocket)>, - next: usize, + /// Every member in kernel order: one generation of `count`, or two with a handoff. + slots: Vec, + /// Members per generation, which is how many this process serves. + count: u16, + /// The generation this process serves. + generation: u16, + next: u16, addr: SocketAddr, /// Held for exactly as long as the complete socket group, and released by @@ -211,9 +306,9 @@ pub struct Bound { } impl Bound { - /// How many sockets share the port. + /// How many sockets this process serves. pub fn count(&self) -> u16 { - self.sockets.len().try_into().expect("group count fits u16") + self.count } /// The address every member holds, with an ephemeral port already resolved. @@ -221,33 +316,203 @@ impl Bound { self.addr } + /// How many generations the group holds members for: two when bound for a handoff. + fn generations(&self) -> u16 { + u16::try_from(self.slots.len() / usize::from(self.count)).expect("at most 256 members") + } + /// The next socket to serve, or `None` once every member was handed out. /// /// The group retains the underlying socket, so dropping the returned handle /// cannot resize the kernel's reuseport array. pub fn member(&mut self) -> io::Result> { - let Some((shard, socket)) = self.sockets.get(self.next) else { + if self.next >= self.count { return Ok(None); + } + let index = self.generation * self.count + self.next; + let socket = self.slots[usize::from(index)].try_clone()?; + let shard = Shard { + index, + count: u16::try_from(self.slots.len()).expect("at most 256 members"), + generations: self.generations(), }; - let socket = socket.try_clone()?; + let worker = self.next; self.next += 1; - Ok(Some(Socket { shard: *shard, socket })) + Ok(Some(Socket { shard, worker, socket })) + } + + /// Make this process's generation the one every new handshake reaches. + /// + /// [`Config::bind`] steers its group already. A successor calls this once it + /// serves, with [`Steer::Beside`] while its predecessor still serves and + /// [`Steer::Alone`] once it is gone. The filter belongs to the whole group, so + /// the latest call from either process wins. + pub fn steer(&self, steer: Steer) -> io::Result<()> { + let first = self.slots.first().expect("a group always has at least one member"); + attach(first, self.program(steer)) + } + + /// A handle that steers this group for this process's generation, for whoever + /// decides when new handshakes move once the group itself is serving elsewhere. + pub fn steering(&self) -> io::Result { + let first = self.slots.first().expect("a group always has at least one member"); + Ok(Steering { + socket: first.try_clone()?, + count: self.count, + total: u16::try_from(self.slots.len()).expect("at most 256 members"), + base: self.generation * self.count, + }) + } + + fn program(&self, steer: Steer) -> Program { + Program { + count: self.count, + total: u16::try_from(self.slots.len()).expect("at most 256 members"), + base: self.generation * self.count, + exact: steer == Steer::Beside, + } + } + + /// Duplicate every member and the port lock for a successor to [`adopt`](Self::adopt). + /// + /// Refused with [`io::ErrorKind::Unsupported`] for a group bound without + /// [`Config::handoff`], which has no generation to give away. + #[cfg(target_os = "linux")] + pub fn handoff(&self) -> io::Result { + if self.generations() < 2 { + return Err(io::Error::new( + io::ErrorKind::Unsupported, + "the reuseport group was bound without room for a successor", + )); + } + let sockets = self + .slots + .iter() + .map(|socket| socket.try_clone().map(OwnedFd::from)) + .collect::>()?; + let lock = self + ._lock + .as_ref() + .map(|lock| lock._file.try_clone().map(OwnedFd::from)) + .transpose()?; + Ok(Handoff { + addr: self.addr, + count: self.count, + generation: self.generation, + sockets, + lock, + }) } + + /// Serve the generation a predecessor's [`handoff`](Self::handoff) left free. + /// + /// Nothing is steered here: the predecessor keeps every new handshake until + /// this process is ready to serve and calls [`steer`](Self::steer). + #[cfg(target_os = "linux")] + pub fn adopt(handoff: Handoff) -> Result { + let Handoff { + addr, + count, + generation, + sockets, + lock, + } = handoff; + let refuse = |reason: String| Err(Error::Adopt(reason)); + if count == 0 || u32::from(count) * 2 > u32::from(MAX_SHARDS) { + return refuse(format!("{count} members per generation")); + } + if generation > 1 { + return refuse(format!("generation {generation} of two")); + } + if sockets.len() != usize::from(count) * 2 { + return refuse(format!("{} sockets for two generations of {count}", sockets.len())); + } + let mut slots = Vec::with_capacity(sockets.len()); + for (index, socket) in sockets.into_iter().enumerate() { + let socket = UdpSocket::from(socket); + match socket.local_addr() { + Ok(local) if local == addr => slots.push(socket), + Ok(local) => return refuse(format!("member {index} is bound to {local}, not {addr}")), + Err(err) => return refuse(format!("member {index} is not a bound UDP socket: {err}")), + } + } + Ok(Self { + slots, + count, + generation: 1 - generation, + next: 0, + addr, + _lock: lock.map(|file| Lock { + _file: std::fs::File::from(file), + }), + }) + } +} + +/// Steers a group for one process's generation, as [`Bound::steer`] does, from a +/// handle of its own: holds one member open, which keeps nothing else alive. +#[derive(Debug)] +pub struct Steering { + socket: UdpSocket, + count: u16, + total: u16, + base: u16, +} + +impl Steering { + /// See [`Bound::steer`]. + pub fn steer(&self, steer: Steer) -> io::Result<()> { + attach( + &self.socket, + Program { + count: self.count, + total: self.total, + base: self.base, + exact: steer == Steer::Beside, + }, + ) + } +} + +/// A steered group on its way to a successor process: every member in kernel +/// order and the port lock, from [`Bound::handoff`] for [`Bound::adopt`]. +/// +/// Carries file descriptors, so send it over a Unix socket with +/// [`crate::fd::send`] and rebuild it from what [`crate::fd::recv`] returns. +#[cfg(target_os = "linux")] +#[derive(Debug)] +pub struct Handoff { + /// The address every member holds. + pub addr: SocketAddr, + /// Members per generation. + pub count: u16, + /// The generation the sender serves; the successor takes the other one. + pub generation: u16, + /// Every member, two generations of `count`, in the order the kernel numbers them. + pub sockets: Vec, + /// The lock holding the port against a second group, if the sender had one. + pub lock: Option, } /// One serving handle released by a complete [`Bound`] group. #[derive(Debug)] pub struct Socket { shard: Shard, + worker: u16, socket: UdpSocket, } impl Socket { - /// This socket's position in the group. + /// The slot this socket holds in the group, which its connection IDs name. pub fn shard(&self) -> Shard { self.shard } + /// This socket's position among the ones its process serves, from zero. + pub fn worker(&self) -> u16 { + self.worker + } + /// Release the UDP handle while the [`Bound`] group retains its member. pub fn into_inner(self) -> UdpSocket { self.socket @@ -472,27 +737,41 @@ fn prepare(dir: std::path::PathBuf) -> Option { Some(dir) } -/// The largest group the steering filter can address. +/// The largest group the steering filter can address, both generations counted. /// /// It selects with one byte of the connection ID, so a member past 255 could /// never be named: the filter's `byte % count` could never return that index. pub const MAX_SHARDS: u16 = 256; -/// Attach the steering filter to a socket in a reuseport group of `count`. +/// What the steering filter computes for one group. +#[derive(Clone, Copy, Debug)] +struct Program { + /// Members per generation. + count: u16, + /// Members in the group, every generation counted. + total: u16, + /// The first member of the generation that takes new handshakes. + base: u16, + /// Steer a server-chosen connection ID to the member that issued it, rather than + /// to the same position in the newest generation. + exact: bool, +} + +/// Attach the steering filter to the reuseport group `socket` belongs to. /// -/// Applies to the group, so once is enough, and the last member to bind is the -/// first point where that is true. +/// Applies to the group, so once is enough, and replaces whatever filter the +/// group had: a successor retargets new handshakes this way. #[cfg(target_os = "linux")] -fn attach(socket: &UdpSocket, count: u16) -> io::Result<()> { +fn attach(socket: &UdpSocket, program: Program) -> io::Result<()> { use std::os::fd::AsRawFd; - let program = program(count); + let insns = program.insns(); let fprog = libc::sock_fprog { - len: program.len() as u16, - filter: program.as_ptr() as *mut libc::sock_filter, + len: insns.len() as u16, + filter: insns.as_ptr() as *mut libc::sock_filter, }; - // SAFETY: `fprog` points at `program`, which outlives the call, and its `len` + // SAFETY: `fprog` points at `insns`, which outlives the call, and its `len` // is that slice's length. let res = unsafe { libc::setsockopt( @@ -508,12 +787,12 @@ fn attach(socket: &UdpSocket, count: u16) -> io::Result<()> { return Err(io::Error::last_os_error()); } - tracing::debug!(count, "steering the reuseport group by connection ID"); + tracing::debug!(?program, "steering the reuseport group by connection ID"); Ok(()) } #[cfg(not(target_os = "linux"))] -fn attach(_socket: &UdpSocket, _count: u16) -> io::Result<()> { +fn attach(_socket: &UdpSocket, _program: Program) -> io::Result<()> { // Unreachable in practice: a member only gets here once `bind::udp` accepted // `SO_REUSEPORT`, which is Linux-only. Err(io::Error::new( @@ -522,74 +801,140 @@ fn attach(_socket: &UdpSocket, _count: u16) -> io::Result<()> { )) } -/// The classic-BPF program that maps a QUIC packet to its owner's index. -/// -/// A long header is QUIC when it carries v1 or v2. A short header is QUIC when -/// its fixed bit and relay CID marker are set. For either, the first destination -/// CID byte is reduced modulo the group size. Everything else returns an invalid -/// index, which asks reuseport to use its 4-tuple hash. The packet starts at -/// offset 0 because the kernel pulls the UDP header off before running this. -/// -/// A load past the end of the datagram aborts the program with 0, which -/// reuseport takes as member 0, so anything too short for every load is -/// rejected first. -#[cfg(target_os = "linux")] -fn program(count: u16) -> [libc::sock_filter; 18] { - /// Long-header form bit in the first byte of a QUIC packet. - const LONG_HEADER: u32 = 0x80; - /// Fixed bit on short headers. Greasing is disabled on sharded endpoints. - const FIXED_BIT: u32 = 0x40; - const VERSION_1: u32 = 0x0000_0001; - const VERSION_2: u32 = 0x6b33_43cf; - /// The SRT key-encryption bits overlaid by the fourth destination CID byte. - const MAGIC_MASK: u32 = 0x18; - /// 1 first byte + 4 version bytes + 1 length byte precede a long header's - /// destination connection ID. - const LONG_DCID: u32 = 6; - /// A short header's destination connection ID follows the first byte. - const SHORT_DCID: u32 = 1; - /// The shortest datagram every load below stays inside. - const MIN_LEN: u32 = LONG_DCID + 1; - - fn insn(code: u32, jt: u8, jf: u8, k: u32) -> libc::sock_filter { - libc::sock_filter { - code: code as u16, - jt, - jf, - k, +impl Program { + /// The classic-BPF program that maps a QUIC packet to its member's index. + /// + /// A long header is QUIC when it carries v1 or v2. A short header is QUIC when + /// its fixed bit and relay CID marker are set. A short header and a Handshake + /// packet carry a connection ID the server chose, so its first byte names a + /// member: modulo the group size when [`exact`](Self::exact), else modulo + /// one generation's size, offset into the newest. An Initial or 0-RTT packet + /// carries the one the client invented, which names nothing, so it always + /// goes to the newest generation, the same member for every retransmission. + /// Everything else (another protocol sharing the port, or a QUIC version this + /// cannot read) hashes its source address and port into the newest + /// generation, so one flow stays on one member and never lands on a + /// generation nobody reads. The kernel's own 4-tuple hash would pick from + /// every member, the reserve generation included. The packet starts at + /// offset 0 because the kernel pulls the UDP header off before running + /// this; the addresses are read back through the network header. + /// + /// A load past the end of the datagram aborts the program with 0, which + /// reuseport takes as member 0, so anything too short for every load is + /// rejected first. Jumps only go forward, which is why the two reductions + /// are spelled out twice. + #[cfg(target_os = "linux")] + fn insns(self) -> [libc::sock_filter; 42] { + /// Long-header form bit in the first byte of a QUIC packet. + const LONG_HEADER: u32 = 0x80; + /// Fixed bit on short headers. Greasing is disabled on sharded endpoints. + const FIXED_BIT: u32 = 0x40; + /// The long header's packet type bits. + const TYPE_MASK: u32 = 0x30; + /// QUIC v1's Handshake packet type (RFC 9000). + const HANDSHAKE_V1: u32 = 0x20; + /// QUIC v2's Handshake packet type (RFC 9369). + const HANDSHAKE_V2: u32 = 0x30; + const VERSION_1: u32 = 0x0000_0001; + const VERSION_2: u32 = 0x6b33_43cf; + /// The SRT key-encryption bits overlaid by the fourth destination CID byte. + const MAGIC_MASK: u32 = 0x18; + /// 1 first byte + 4 version bytes + 1 length byte precede a long header's + /// destination connection ID. + const LONG_DCID: u32 = 6; + /// A short header's destination connection ID follows the first byte. + const SHORT_DCID: u32 = 1; + /// The shortest datagram every load below stays inside. + const MIN_LEN: u32 = LONG_DCID + 1; + /// Loads at this offset read from the IP header rather than the payload. + const NET: u32 = libc::SKF_NET_OFF as u32; + /// Where the source address sits in an IPv4 header. + const V4_SRC: u32 = 12; + /// The last word of the source address in an IPv6 header. + const V6_SRC: u32 = 20; + /// The fixed IPv6 header, which a UDP header follows on every flow worth balancing. + const V6_LEN: u32 = 40; + + fn insn(code: u32, jt: u8, jf: u8, k: u32) -> libc::sock_filter { + libc::sock_filter { + code: code as u16, + jt, + jf, + k, + } } - } - [ - // A = datagram length; too short to be QUIC uses the kernel hash. - insn(libc::BPF_LD | libc::BPF_W | libc::BPF_LEN, 0, 0, 0), - insn(libc::BPF_JMP | libc::BPF_JGE | libc::BPF_K, 0, 12, MIN_LEN), - // A = packet[0] - insn(libc::BPF_LD | libc::BPF_B | libc::BPF_ABS, 0, 0, 0), - // Long header? Jump to its version check. - insn(libc::BPF_JMP | libc::BPF_JSET | libc::BPF_K, 7, 0, LONG_HEADER), - // A short header must retain QUIC's fixed bit. - insn(libc::BPF_JMP | libc::BPF_JSET | libc::BPF_K, 0, 9, FIXED_BIT), - // A = packet[4], the fourth destination CID byte. - insn(libc::BPF_LD | libc::BPF_B | libc::BPF_ABS, 0, 0, SHORT_DCID + 3), - insn(libc::BPF_ALU | libc::BPF_AND | libc::BPF_K, 0, 0, MAGIC_MASK), - // A short header without the relay CID marker uses the kernel hash. - insn(libc::BPF_JMP | libc::BPF_JEQ | libc::BPF_K, 0, 6, MAGIC_MASK), - // A = packet[1] - insn(libc::BPF_LD | libc::BPF_B | libc::BPF_ABS, 0, 0, SHORT_DCID), - insn(libc::BPF_ALU | libc::BPF_MOD | libc::BPF_K, 0, 0, u32::from(count)), - insn(libc::BPF_RET | libc::BPF_A, 0, 0, 0), - // Long headers steer only supported QUIC versions. - insn(libc::BPF_LD | libc::BPF_W | libc::BPF_ABS, 0, 0, 1), - insn(libc::BPF_JMP | libc::BPF_JEQ | libc::BPF_K, 2, 0, VERSION_1), - insn(libc::BPF_JMP | libc::BPF_JEQ | libc::BPF_K, 1, 0, VERSION_2), - // An out-of-range result asks reuseport to use its 4-tuple hash. - insn(libc::BPF_RET | libc::BPF_K, 0, 0, u32::from(count)), - // A = packet[6] - insn(libc::BPF_LD | libc::BPF_B | libc::BPF_ABS, 0, 0, LONG_DCID), - insn(libc::BPF_ALU | libc::BPF_MOD | libc::BPF_K, 0, 0, u32::from(count)), - insn(libc::BPF_RET | libc::BPF_A, 0, 0, 0), - ] + let count = u32::from(self.count); + let base = u32::from(self.base); + // A server-chosen ID either names its exact member, or the member at the same + // position in the newest generation. + let (server_mod, server_add) = match self.exact { + true => (u32::from(self.total), 0), + false => (count, base), + }; + + [ + // 0: A = datagram length; too short to be QUIC is hashed as a flow. + insn(libc::BPF_LD | libc::BPF_W | libc::BPF_LEN, 0, 0, 0), + insn(libc::BPF_JMP | libc::BPF_JGE | libc::BPF_K, 0, 25, MIN_LEN), + // 2: A = packet[0] + insn(libc::BPF_LD | libc::BPF_B | libc::BPF_ABS, 0, 0, 0), + // Long header? Jump to its version check at 10. + insn(libc::BPF_JMP | libc::BPF_JSET | libc::BPF_K, 6, 0, LONG_HEADER), + // 4: A short header must retain QUIC's fixed bit. + insn(libc::BPF_JMP | libc::BPF_JSET | libc::BPF_K, 0, 22, FIXED_BIT), + // 5: A = packet[4], the fourth destination CID byte. + insn(libc::BPF_LD | libc::BPF_B | libc::BPF_ABS, 0, 0, SHORT_DCID + 3), + insn(libc::BPF_ALU | libc::BPF_AND | libc::BPF_K, 0, 0, MAGIC_MASK), + // 7: A short header without the relay CID marker is hashed as a flow. + insn(libc::BPF_JMP | libc::BPF_JEQ | libc::BPF_K, 0, 19, MAGIC_MASK), + // 8: A = packet[1], then the server-chosen reduction at 20. + insn(libc::BPF_LD | libc::BPF_B | libc::BPF_ABS, 0, 0, SHORT_DCID), + insn(libc::BPF_JMP | libc::BPF_JA, 0, 0, 10), + // 10: Long headers steer only supported QUIC versions. + insn(libc::BPF_LD | libc::BPF_W | libc::BPF_ABS, 0, 0, 1), + insn(libc::BPF_JMP | libc::BPF_JEQ | libc::BPF_K, 4, 0, VERSION_1), + insn(libc::BPF_JMP | libc::BPF_JEQ | libc::BPF_K, 0, 14, VERSION_2), + // 13: v2: a Handshake loads the server-chosen ID at 19, else the client's at 23. + insn(libc::BPF_LD | libc::BPF_B | libc::BPF_ABS, 0, 0, 0), + insn(libc::BPF_ALU | libc::BPF_AND | libc::BPF_K, 0, 0, TYPE_MASK), + insn(libc::BPF_JMP | libc::BPF_JEQ | libc::BPF_K, 3, 7, HANDSHAKE_V2), + // 16: v1: the same split by v1's type. + insn(libc::BPF_LD | libc::BPF_B | libc::BPF_ABS, 0, 0, 0), + insn(libc::BPF_ALU | libc::BPF_AND | libc::BPF_K, 0, 0, TYPE_MASK), + insn(libc::BPF_JMP | libc::BPF_JEQ | libc::BPF_K, 0, 4, HANDSHAKE_V1), + // 19: A = packet[6], a Handshake's first destination CID byte. + insn(libc::BPF_LD | libc::BPF_B | libc::BPF_ABS, 0, 0, LONG_DCID), + // 20: A server-chosen ID names its member. + insn(libc::BPF_ALU | libc::BPF_MOD | libc::BPF_K, 0, 0, server_mod), + insn(libc::BPF_ALU | libc::BPF_ADD | libc::BPF_K, 0, 0, server_add), + insn(libc::BPF_RET | libc::BPF_A, 0, 0, 0), + // 23: A = packet[6], the ID the client invented, into the newest generation. + insn(libc::BPF_LD | libc::BPF_B | libc::BPF_ABS, 0, 0, LONG_DCID), + insn(libc::BPF_ALU | libc::BPF_MOD | libc::BPF_K, 0, 0, count), + insn(libc::BPF_ALU | libc::BPF_ADD | libc::BPF_K, 0, 0, base), + insn(libc::BPF_RET | libc::BPF_A, 0, 0, 0), + // 27: Not QUIC: hash the flow into the newest generation, by IP version. + insn(libc::BPF_LD | libc::BPF_B | libc::BPF_ABS, 0, 0, NET), + insn(libc::BPF_ALU | libc::BPF_RSH | libc::BPF_K, 0, 0, 4), + insn(libc::BPF_JMP | libc::BPF_JEQ | libc::BPF_K, 5, 0, 6), + // 30: IPv4: X = the header length, A = the UDP source port, X = A, A = the source address. + insn(libc::BPF_LDX | libc::BPF_B | libc::BPF_MSH, 0, 0, NET), + insn(libc::BPF_LD | libc::BPF_H | libc::BPF_IND, 0, 0, NET), + insn(libc::BPF_MISC | libc::BPF_TAX, 0, 0, 0), + insn(libc::BPF_LD | libc::BPF_W | libc::BPF_ABS, 0, 0, NET + V4_SRC), + insn(libc::BPF_JMP | libc::BPF_JA, 0, 0, 3), + // 35: IPv6: the same past the fixed header. + insn(libc::BPF_LD | libc::BPF_H | libc::BPF_ABS, 0, 0, NET + V6_LEN), + insn(libc::BPF_MISC | libc::BPF_TAX, 0, 0, 0), + insn(libc::BPF_LD | libc::BPF_W | libc::BPF_ABS, 0, 0, NET + V6_SRC), + // 38: A = (address ^ port) % count + base. + insn(libc::BPF_ALU | libc::BPF_XOR | libc::BPF_X, 0, 0, 0), + insn(libc::BPF_ALU | libc::BPF_MOD | libc::BPF_K, 0, 0, count), + insn(libc::BPF_ALU | libc::BPF_ADD | libc::BPF_K, 0, 0, base), + insn(libc::BPF_RET | libc::BPF_A, 0, 0, 0), + ] + } } #[cfg(test)] @@ -649,7 +994,14 @@ mod tests { for index in 0..COUNT { let member = group.member().unwrap().expect("a socket per member"); - assert_eq!(member.shard(), Shard { index, count: COUNT }); + assert_eq!( + member.shard(), + Shard { + index, + count: COUNT, + generations: 1 + } + ); assert_eq!(member.into_inner().local_addr().unwrap(), addr); } assert!(group.member().unwrap().is_none(), "the group cannot be resized"); @@ -663,8 +1015,93 @@ mod tests { assert_eq!(group.count(), 1); } + /// Which member received something, given every send targets exactly one. + #[cfg(target_os = "linux")] + fn receiver(group: &[UdpSocket]) -> Option { + use std::os::fd::AsRawFd; + + // Block until a member is readable: the filter runs on the send, but the + // datagram lands in the member's queue asynchronously. + let mut fds: Vec<_> = group + .iter() + .map(|socket| libc::pollfd { + fd: socket.as_raw_fd(), + events: libc::POLLIN, + revents: 0, + }) + .collect(); + let n = unsafe { libc::poll(fds.as_mut_ptr(), fds.len() as libc::nfds_t, 5_000) }; + assert!(n >= 0, "poll failed: {}", std::io::Error::last_os_error()); + let mut buf = [0u8; 64]; + group.iter().position(|socket| socket.recv_from(&mut buf).is_ok()) + } + + /// Send `packet` to `addr` from a fresh source port, so nothing but its + /// connection ID can be steering it. + #[cfg(target_os = "linux")] + fn send(packet: &[u8], addr: SocketAddr) { + let sender = crate::bind::udp(crate::bind::Udp::new("127.0.0.1:0".parse().unwrap())).unwrap(); + sender.send_to(packet, addr).unwrap(); + } + + /// A short header carrying `id`. + #[cfg(target_os = "linux")] + fn short(id: [u8; CID_LEN]) -> Vec { + let mut packet = vec![0x40]; + packet.extend_from_slice(&id); + packet + } + + /// A long header of `first` byte and `version`, addressed to `id`. + #[cfg(target_os = "linux")] + fn long(first: u8, version: [u8; 4], id: [u8; CID_LEN]) -> Vec { + let mut packet = vec![first, version[0], version[1], version[2], version[3], CID_LEN as u8]; + packet.extend_from_slice(&id); + packet + } + + const V1: [u8; 4] = [0, 0, 0, 1]; + const V2: [u8; 4] = [0x6b, 0x33, 0x43, 0xcf]; + + /// Every packet a client sends with a connection ID the server chose. + #[cfg(target_os = "linux")] + fn chosen(id: [u8; CID_LEN]) -> [(&'static str, Vec); 3] { + [ + ("short", short(id)), + ("v1 handshake", long(0xe0, V1, id)), + ("v2 handshake", long(0xf0, V2, id)), + ] + } + + /// Every packet a client sends with the connection ID it invented. + #[cfg(target_os = "linux")] + fn invented(id: [u8; CID_LEN]) -> [(&'static str, Vec); 4] { + [ + ("v1 initial", long(0xc0, V1, id)), + ("v1 0-rtt", long(0xd0, V1, id)), + ("v2 initial", long(0xd0, V2, id)), + ("v2 0-rtt", long(0xe0, V2, id)), + ] + } + + /// Serve every member of `group`, non-blocking. + #[cfg(target_os = "linux")] + fn serve(group: &mut Bound) -> (Vec, Vec) { + let mut sockets = Vec::new(); + let mut shards = Vec::new(); + while let Some(member) = group.member().expect("clone retained socket") { + assert_eq!(usize::from(member.worker()), sockets.len(), "workers count from zero"); + shards.push(member.shard()); + let socket = member.into_inner(); + socket.set_nonblocking(true).unwrap(); + sockets.push(socket); + } + (sockets, shards) + } + /// The whole point, end to end against a real kernel: a packet carrying a - /// member's connection ID has to reach that member and no other. + /// member's connection ID has to reach that member and no other, and a client's + /// retransmitted Initial has to reach the same member as its first. /// /// This is what the 4-tuple hash cannot do. Every packet here is sent from a /// different ephemeral source port, so under the default hashing the landing @@ -675,120 +1112,242 @@ mod tests { const COUNT: u16 = 4; let mut group = bind("127.0.0.1:0".parse().unwrap(), COUNT).unwrap(); - let mut sockets = Vec::new(); - let mut shards = Vec::new(); - while let Some(member) = group.member().expect("clone retained socket") { - shards.push(member.shard()); - let socket = member.into_inner(); - socket.set_nonblocking(true).unwrap(); - sockets.push(socket); - } + let (sockets, shards) = serve(&mut group); let addr = group.addr(); - /// Which member received something, given every send targets exactly one. - fn receiver(group: &[UdpSocket]) -> Option { - use std::os::fd::AsRawFd; - - // Block until a member is readable: the filter runs on the send, but the - // datagram lands in the member's queue asynchronously. - let mut fds: Vec<_> = group - .iter() - .map(|socket| libc::pollfd { - fd: socket.as_raw_fd(), - events: libc::POLLIN, - revents: 0, + for (index, shard) in shards.into_iter().enumerate() { + let id = cid(shard); + for (form, packet) in chosen(id).into_iter().chain(invented(id)) { + send(&packet, addr); + assert_eq!( + receiver(&sockets), + Some(index), + "{form} for member {index} landed on the wrong socket" + ); + } + } + } + + /// Two processes serving one port through a handoff, against a real kernel: + /// each keeps its own connections, the newest takes every new handshake, and + /// the predecessor leaving renumbers nothing. + #[test] + #[cfg(target_os = "linux")] + fn a_successor_serves_beside_its_predecessor() { + const COUNT: u16 = 3; + + let mut config = Config::new("127.0.0.1:0".parse().unwrap(), COUNT); + config.handoff = true; + let mut old = config.bind().unwrap(); + let addr = old.addr(); + let (old_sockets, old_shards) = serve(&mut old); + assert_eq!(old.count(), COUNT, "a process serves one generation"); + assert_eq!( + old_shards, + (0..COUNT) + .map(|index| Shard { + index, + count: COUNT * 2, + generations: 2, }) - .collect(); - let n = unsafe { libc::poll(fds.as_mut_ptr(), fds.len() as libc::nfds_t, 5_000) }; - assert!(n >= 0, "poll failed: {}", std::io::Error::last_os_error()); - let mut buf = [0u8; 64]; - group.iter().position(|socket| socket.recv_from(&mut buf).is_ok()) + .collect::>(), + "the first process serves the first generation, named against both" + ); + + // Until the successor steers, it receives nothing at all. + let mut new = Bound::adopt(old.handoff().unwrap()).unwrap(); + let (new_sockets, new_shards) = serve(&mut new); + assert_eq!( + new_shards, + (COUNT..COUNT * 2) + .map(|index| Shard { + index, + count: COUNT * 2, + generations: 2, + }) + .collect::>(), + "the successor serves the generation its predecessor left free" + ); + let invented_id: [u8; CID_LEN] = rand::random(); + for (form, packet) in invented(invented_id) { + send(&packet, addr); + assert!( + receiver(&old_sockets).is_some(), + "a {form} skipped the serving generation" + ); } - for (index, shard) in shards.into_iter().enumerate() { - let id = cid(shard); + new.steer(Steer::Beside).unwrap(); + for (generation, sockets, shards) in [(0, &old_sockets, &old_shards), (1, &new_sockets, &new_shards)] { + for (index, shard) in shards.iter().enumerate() { + for (form, packet) in chosen(cid(*shard)) { + send(&packet, addr); + assert_eq!( + receiver(sockets), + Some(index), + "a {form} for generation {generation} member {index} left its member" + ); + } + } + } + for _ in 0..16 { + let id: [u8; CID_LEN] = rand::random(); + for (form, packet) in invented(id) { + send(&packet, addr); + let first = receiver(&new_sockets); + assert!(first.is_some(), "a new {form} reached the predecessor"); + // A retransmission repeats the ID it invented, so it reaches the same member. + send(&packet, addr); + assert_eq!(receiver(&new_sockets), first, "a retransmitted {form} changed members"); + } + } - // A short header: form bit clear, connection ID straight after. - let mut short = [0; CID_LEN + 1]; - short[0] = 0x40; - short[1..].copy_from_slice(&id); - let long = |version: [u8; 4]| { - let mut packet = [0; CID_LEN + 6]; - packet[..6].copy_from_slice(&[0xc0, version[0], version[1], version[2], version[3], CID_LEN as u8]); - packet[6..].copy_from_slice(&id); - packet + // The predecessor exits. Its sockets stay open in the successor, so nothing + // renumbers, and once alone the successor takes its stale packets too, steered + // through a handle of its own as a process does once its workers own the group. + drop(old_sockets); + drop(old); + new.steering().unwrap().steer(Steer::Alone).unwrap(); + for (index, shard) in new_shards.iter().enumerate() { + for (form, packet) in chosen(cid(*shard)) { + send(&packet, addr); + assert_eq!(receiver(&new_sockets), Some(index), "a {form} left member {index}"); + } + // The predecessor's member at the same position. + let stale = Shard { + index: shard.index() - COUNT, + ..*shard }; - let v1 = long([0, 0, 0, 1]); - let v2 = long([0x6b, 0x33, 0x43, 0xcf]); - - for (form, packet) in [("short", &short[..]), ("v1", &v1[..]), ("v2", &v2[..])] { - // A fresh source port per packet, so nothing but the connection ID - // can be steering this. - let sender = crate::bind::udp(crate::bind::Udp::new("127.0.0.1:0".parse().unwrap())).unwrap(); - sender.send_to(packet, addr).unwrap(); + for (form, packet) in chosen(cid(stale)) { + send(&packet, addr); assert_eq!( - receiver(&sockets), + receiver(&new_sockets), Some(index), - "{form} header for member {index} landed on the wrong socket" + "a stale {form} for member {index} was not folded onto its position" ); } } } - /// Datagram protocols that share the port stay on the kernel's 4-tuple hash, - /// even when fields at the QUIC connection-ID offsets change or are missing. + /// A group bound without room for a successor has none to give, and a + /// handoff that does not describe its sockets is refused. + #[test] + #[cfg(target_os = "linux")] + fn a_handoff_needs_a_second_generation() { + let group = bind("127.0.0.1:0".parse().unwrap(), 2).unwrap(); + assert_eq!(group.handoff().unwrap_err().kind(), io::ErrorKind::Unsupported); + + let mut config = Config::new("127.0.0.1:0".parse().unwrap(), 2); + config.handoff = true; + let group = config.bind().unwrap(); + let mut handoff = group.handoff().unwrap(); + handoff.sockets.pop(); + assert!(matches!(Bound::adopt(handoff), Err(Error::Adopt(_)))); + + let mut handoff = group.handoff().unwrap(); + handoff.addr.set_port(handoff.addr.port().wrapping_add(1)); + assert!(matches!(Bound::adopt(handoff), Err(Error::Adopt(_)))); + + // Two generations must both fit the byte the filter reads. + let mut config = Config::new("127.0.0.1:0".parse().unwrap(), MAX_SHARDS / 2 + 1); + config.handoff = true; + assert!(matches!(config.bind(), Err(Error::Count { max: MAX_SHARDS, .. }))); + } + + /// Datagram protocols that share the port are balanced by flow, even when + /// fields at the QUIC connection-ID offsets change or are missing, and only + /// across the generation taking new handshakes: a member nobody reads would + /// swallow the flow. #[test] #[cfg(target_os = "linux")] fn non_quic_flows_stay_on_one_member() { + use std::net::IpAddr; + const COUNT: u16 = 4; - let mut group = bind("127.0.0.1:0".parse().unwrap(), COUNT).unwrap(); - let mut sockets = Vec::new(); - while let Some(member) = group.member().expect("clone retained socket") { - let socket = member.into_inner(); - socket - .set_read_timeout(Some(std::time::Duration::from_millis(100))) - .unwrap(); - sockets.push(socket); + + /// Send each protocol's flow to `addr` and require that every packet of it, + /// truncated or not, lands on one member of `serving`, among `group`. + fn flows(addr: SocketAddr, group: &[UdpSocket], serving: std::ops::Range) { + let flows = [ + ("rtp", [0x80, 96, 0, 1, 0, 0, 0, 1, 0x12, 0x34, 0x56, 0x78]), + // SRT data with KK=0b10 (no encryption); R is toggled below. + ("srt", [0x40, 0, 0, 1, 0x10, 0, 0, 1, 0x12, 0x34, 0x56, 0x78]), + ("stun", [0, 1, 0, 0, 0x21, 0x12, 0xa4, 0x42, 1, 2, 3, 4]), + ("dtls", [22, 0xfe, 0xfd, 0, 0, 0, 0, 0, 0, 0, 0, 0]), + ]; + let local = SocketAddr::new(addr.ip(), 0); + for (name, mut packet) in flows { + // A flow hashed off the generation's first member, so a filter that + // faults to its first index shows up. Every flow lands in the generation. + let (sender, owner) = (0..64) + .find_map(|_| { + let sender = crate::bind::udp(crate::bind::Udp::new(local)).unwrap(); + sender.send_to(&packet, addr).unwrap(); + let owner = receiver(group).expect("one member receives the flow"); + assert!( + serving.contains(&owner), + "{name} reached member {owner}, outside {serving:?}" + ); + (owner != serving.start).then_some((sender, owner)) + }) + .expect("a flow hashed off the first member"); + + for sequence in 2..18u8 { + packet[3] = sequence; + if name == "srt" { + packet[4] ^= 0x04; // retransmission flag + } + sender.send_to(&packet, addr).unwrap(); + assert_eq!(receiver(group), Some(owner), "{name} packet {sequence} changed members"); + } + + // Too short to reach the connection-ID offsets the filter reads. + sender.send_to(&packet[..2], addr).unwrap(); + assert_eq!( + receiver(group), + Some(owner), + "a truncated {name} packet changed members" + ); + } + + // Flows spread across the generation: 64 flows on one member of four is + // a 4^-63 chance, not a balance. + let owners: std::collections::HashSet<_> = (0..64) + .map(|_| { + let sender = crate::bind::udp(crate::bind::Udp::new(local)).unwrap(); + sender.send_to(&flows[2].1, addr).unwrap(); + receiver(group).expect("one member receives the flow") + }) + .collect(); + assert!(owners.len() > 1, "every flow reached one member: {owners:?}"); } - let flows = [ - ("rtp", [0x80, 96, 0, 1, 0, 0, 0, 1, 0x12, 0x34, 0x56, 0x78]), - // SRT data with KK=0b10 (no encryption); R is toggled below. - ("srt", [0x40, 0, 0, 1, 0x10, 0, 0, 1, 0x12, 0x34, 0x56, 0x78]), - ("stun", [0, 1, 0, 0, 0x21, 0x12, 0xa4, 0x42, 1, 2, 3, 4]), - ("dtls", [22, 0xfe, 0xfd, 0, 0, 0, 0, 0, 0, 0, 0, 0]), - ]; - for (name, mut packet) in flows { - let mut buf = [0; 64]; - // A flow hashed off member 0, so a filter that faults to index 0 shows up. - let (sender, owner) = (0..64) - .find_map(|_| { - let sender = crate::bind::udp(crate::bind::Udp::new("127.0.0.1:0".parse().unwrap())).unwrap(); - sender.send_to(&packet, group.addr()).unwrap(); - let owner = sockets - .iter() - .position(|socket| socket.recv_from(&mut buf).is_ok()) - .expect("one member receives the flow"); - (owner != 0).then_some((sender, owner)) + for host in [ + IpAddr::from([127, 0, 0, 1]), + IpAddr::from(std::net::Ipv6Addr::LOCALHOST), + ] { + let mut config = Config::new(SocketAddr::new(host, 0), COUNT); + config.handoff = true; + let old = config.bind().unwrap(); + let addr = old.addr(); + // Every member, the unserved generation included, so a stray lands somewhere seen. + let group: Vec = old + .handoff() + .unwrap() + .sockets + .into_iter() + .map(|fd| { + let socket = UdpSocket::from(fd); + socket.set_nonblocking(true).unwrap(); + socket }) - .expect("a flow hashed off member 0"); + .collect(); - for sequence in 2..18u8 { - packet[3] = sequence; - if name == "srt" { - packet[4] ^= 0x04; // retransmission flag - } - sender.send_to(&packet, group.addr()).unwrap(); - sockets[owner] - .recv_from(&mut buf) - .unwrap_or_else(|err| panic!("{name} packet {sequence} changed shards: {err}")); - } + flows(addr, &group, 0..usize::from(COUNT)); - // Too short to reach the connection-ID offsets the filter reads. - sender.send_to(&packet[..2], group.addr()).unwrap(); - sockets[owner] - .recv_from(&mut buf) - .unwrap_or_else(|err| panic!("truncated {name} packet changed shards: {err}")); + let new = Bound::adopt(old.handoff().unwrap()).unwrap(); + new.steer(Steer::Beside).unwrap(); + flows(addr, &group, usize::from(COUNT)..usize::from(COUNT) * 2); } } @@ -865,22 +1424,78 @@ mod tests { #[test] #[cfg(target_os = "linux")] fn the_program_branches_to_the_right_loads() { - let program = program(4); - assert_eq!(program.len(), 18); + /// Where a jump at `at` lands when taken (`jt`) or not (`jf`). + fn target(at: usize, offset: u8) -> usize { + at + 1 + usize::from(offset) + } - // Instruction 1 sends a short datagram to the kernel-hash return at 14. + let program = Program { + count: 4, + total: 8, + base: 4, + exact: true, + } + .insns(); + const HASH: usize = 27; assert_eq!(program[1].k, 7, "every load must stay inside the datagram"); - assert_eq!(program[1].jf, 12, "a short datagram must use the kernel hash"); - assert_eq!(program[14].k, 4, "the kernel-hash return is out of range"); - // Instruction 3 falls through to the short-header checks, and jumps to - // the long-header version load at 11. - assert_eq!(program[3].jt, 7, "long header must land on the version load"); - assert_eq!(program[3].jf, 0, "short header must fall through"); + assert_eq!(target(1, program[1].jf), HASH, "a short datagram is hashed as a flow"); + assert_eq!( + target(3, program[3].jt), + 10, + "long header must land on the version load" + ); + assert_eq!(target(3, program[3].jf), 4, "short header must fall through"); + assert_eq!(target(4, program[4].jf), HASH, "no fixed bit is hashed as a flow"); assert_eq!(program[5].k, 4, "short header reads the fourth CID byte"); + assert_eq!(target(7, program[7].jf), HASH, "no marker is hashed as a flow"); assert_eq!(program[8].k, 1, "short header reads the shard byte"); - assert_eq!(program[11].k, 1, "long header reads its version"); - assert_eq!(program[15].k, 6, "long header reads past version and length"); - assert_eq!(program[9].k, 4, "the short modulus is the group size"); - assert_eq!(program[16].k, 4, "the long modulus is the group size"); + assert_eq!( + target(9, program[9].k as u8), + 20, + "short header takes the server reduction" + ); + assert_eq!(program[10].k, 1, "long header reads its version"); + assert_eq!(target(11, program[11].jt), 16, "v1 checks v1's packet types"); + assert_eq!(target(12, program[12].jt), 13, "v2 checks v2's packet types"); + assert_eq!(target(12, program[12].jf), HASH, "another version is hashed as a flow"); + for (at, handshake) in [(15, 0x30), (18, 0x20)] { + assert_eq!(program[at].k, handshake, "the Handshake type"); + assert_eq!(target(at, program[at].jt), 19, "a Handshake reads the server's ID"); + assert_eq!(target(at, program[at].jf), 23, "anything else reads the client's ID"); + } + assert_eq!(program[19].k, 6, "a Handshake reads past version and length"); + assert_eq!(program[23].k, 6, "an Initial reads past version and length"); + assert_eq!( + (program[20].k, program[21].k), + (8, 0), + "beside, a server's ID names its member" + ); + assert_eq!( + (program[24].k, program[25].k), + (4, 4), + "a client's ID reaches the newest generation" + ); + assert_eq!(program[HASH].k, libc::SKF_NET_OFF as u32, "a flow reads its IP version"); + assert_eq!(target(29, program[29].jt), 35, "IPv6 reads past its fixed header"); + assert_eq!(target(29, program[29].jf), 30, "IPv4 reads its header length"); + assert_eq!(target(34, program[34].k as u8), 38, "IPv4 joins the flow reduction"); + assert_eq!( + (program[39].k, program[40].k), + (4, 4), + "a flow reaches the newest generation" + ); + + let alone = Program { + count: 4, + total: 8, + base: 4, + exact: false, + } + .insns(); + assert_eq!( + (alone[20].k, alone[21].k), + (4, 4), + "alone, a server's ID folds onto the newest" + ); } } diff --git a/rs/moq-tokio/src/listen.rs b/rs/moq-tokio/src/listen.rs index d5caa4de969..d1777645c0f 100644 --- a/rs/moq-tokio/src/listen.rs +++ b/rs/moq-tokio/src/listen.rs @@ -14,8 +14,9 @@ pub enum Bind { } impl Bind { + /// The socket address this names, resolving a host now. #[cfg(feature = "quic")] - pub(crate) fn resolve(&self) -> std::io::Result { + pub fn resolve(&self) -> std::io::Result { match self { Self::Addr(addr) => Ok(*addr), Self::Host(host, port) => crate::util::resolve(Some(&format!("{host}:{port}")), ""), @@ -243,7 +244,8 @@ pub struct Reset { /// A server restarted with the same secret resets its predecessor's connections, /// so peers reconnect at once instead of waiting out their idle timeout. Every /// holder can reset every other holder's connections, so share it between live - /// servers only behind one address that routes by connection ID (QUIC-LB). + /// servers only behind one address that routes by connection ID, as QUIC-LB or a + /// handoff's reuseport group does. /// Unset draws a random key per process. #[usage( name = "listen-reset-secret", diff --git a/rs/moq-tokio/src/quic/backend.rs b/rs/moq-tokio/src/quic/backend.rs index 7d8e264d1f0..88a23c159d5 100644 --- a/rs/moq-tokio/src/quic/backend.rs +++ b/rs/moq-tokio/src/quic/backend.rs @@ -750,7 +750,8 @@ impl Server { ); endpoint_config.cid_generator = Some(Arc::new(move || { let mut cids = moq_quic::RandomConnectionIdGenerator::default(); - cids.set_shard(shard.index(), shard.count()); + cids.set_shard(shard.index(), shard.count()) + .set_generations(shard.generations()); Box::new(cids) })); } else if let Some(load_balancer) = load_balancer { diff --git a/rs/moq-tokio/src/server.rs b/rs/moq-tokio/src/server.rs index 6a8015de5f9..a69ddc21b4f 100644 --- a/rs/moq-tokio/src/server.rs +++ b/rs/moq-tokio/src/server.rs @@ -108,7 +108,7 @@ impl crate::listen::Config { /// Default bind address used when [`crate::listen::Config::bind`] is not set. #[cfg(feature = "quic")] -pub(crate) const DEFAULT_BIND: net::SocketAddr = +pub const DEFAULT_BIND: net::SocketAddr = net::SocketAddr::V6(net::SocketAddrV6::new(net::Ipv6Addr::UNSPECIFIED, 443, 0, 0)); /// Which listeners a [`Server`] opens, out of the ones its config describes. @@ -202,12 +202,37 @@ pub struct Config { #[cfg(feature = "quic")] pub demux: crate::quic::demux::Demux, + /// Stream listeners the caller already holds, served in place of binding the + /// configured addresses. + #[cfg(any(feature = "tcp", all(feature = "uds", unix)))] + pub streams: Streams, + /// The simulated host the QUIC endpoint binds on, at the configured bind port, in place of /// a UDP socket. For tests on `kio::clock::sim`; see [`crate::quic::driver::sim`]. #[cfg(all(feature = "sim", feature = "quic"))] pub sim: Option, } +/// Stream listeners bound outside the [`Server`], for [`Config::streams`]: the +/// sockets a process took over from the one it replaces, which go on accepting +/// the connections already queued on them. +/// +/// Each still needs its listener configured (`listen.tcp.bind`, +/// `listen.unix.bind`), which says how to serve it; the socket replaces only the +/// bind. A Unix socket's path belongs to whoever bound it, so a listener served +/// from one never unlinks it. +#[cfg(any(feature = "tcp", all(feature = "uds", unix)))] +#[derive(Debug, Default)] +#[non_exhaustive] +pub struct Streams { + /// Served for `listen.tcp.bind`. Must be listening. + #[cfg(feature = "tcp")] + pub tcp: Option, + /// Served for `listen.unix.bind`. Must be listening. + #[cfg(all(feature = "uds", unix))] + pub unix: Option, +} + impl Config { /// Build the [`Server`] this config describes. pub fn init(self) -> crate::Result { @@ -234,6 +259,8 @@ impl Config { stats: self.stats.clone(), admission: self.admission.clone(), demux: self.demux.clone(), + #[cfg(any(feature = "tcp", all(feature = "uds", unix)))] + streams: Streams::default(), #[cfg(feature = "sim")] sim: self.sim.clone(), } @@ -290,6 +317,8 @@ impl Server { stats, #[cfg(feature = "quic")] demux, + #[cfg(any(feature = "tcp", all(feature = "uds", unix)))] + streams: prebound, #[cfg(all(feature = "sim", feature = "quic"))] sim, } = config; @@ -407,6 +436,23 @@ impl Server { // Collect the configured stream listeners (at most one TCP, one Unix). #[cfg(any(feature = "tcp", all(feature = "uds", unix)))] let mut stream_binds = Vec::new(); + #[cfg(any(feature = "tcp", all(feature = "uds", unix)))] + let Streams { + #[cfg(feature = "tcp")] + tcp: mut prebound_tcp, + #[cfg(all(feature = "uds", unix))] + unix: mut prebound_unix, + } = prebound; + #[cfg(feature = "tcp")] + if prebound_tcp.is_some() && !(build_streams && config.tcp.bind.is_some()) { + return Err(Error::NoBackend("server::Config::streams.tcp needs --listen-tcp-bind")); + } + #[cfg(all(feature = "uds", unix))] + if prebound_unix.is_some() && !(build_streams && config.unix.bind.is_some()) { + return Err(Error::NoBackend( + "server::Config::streams.unix needs --listen-unix-bind", + )); + } #[cfg(feature = "tcp")] if let Some(addr) = config.tcp.bind.filter(|_| build_streams) { let tls = match config.tcp.tls == Some(true) { @@ -418,11 +464,11 @@ impl Server { )?), false => None, }; - stream_binds.push(StreamBind::Tcp(addr, tls)); + stream_binds.push(StreamBind::Tcp(addr, tls, prebound_tcp.take())); } #[cfg(all(feature = "uds", unix))] if let Some(path) = config.unix.bind.clone().filter(|_| build_streams) { - stream_binds.push(StreamBind::Unix(path)); + stream_binds.push(StreamBind::Unix(path, prebound_unix.take())); } // `None` (or an all-empty allowlist) means the listener enforces nothing. #[cfg(all(feature = "uds", unix))] @@ -891,6 +937,18 @@ impl Listener { crate::runtime::scope(self.server.shutdown()).await; } + /// Stop accepting on the stream (`tcp`/`unix`) listeners and release their + /// sockets, leaving QUIC and every accepted session alone. + /// + /// For a process handing its listeners to a successor that holds the same + /// sockets: what is queued on them is the successor's to accept. A handshake + /// still in flight is dropped, and its client dials again. Later + /// [`accept`](Self::accept) calls see no stream listener. + #[cfg(any(feature = "tcp", all(feature = "uds", unix)))] + pub async fn close_streams(&mut self) { + crate::runtime::scope(self.server.streams.shutdown()).await; + } + /// The address the QUIC listener bound to, useful when the config asked for /// port 0. /// @@ -1014,11 +1072,13 @@ fn stream_versions(base: &moq_net::Versions) -> moq_net::Versions { /// A configured stream listener (`--listen-tcp-bind` / `--listen-unix-bind`). #[cfg(any(feature = "tcp", all(feature = "uds", unix)))] enum StreamBind { - /// With its TLS when the listener serves `tls://`. + /// With its TLS when the listener serves `tls://`, and the socket to serve + /// when the caller bound it. #[cfg(feature = "tcp")] - Tcp(net::SocketAddr, Option), + Tcp(net::SocketAddr, Option, Option), + /// With the socket to serve when the caller bound it. #[cfg(all(feature = "uds", unix))] - Unix(PathBuf), + Unix(PathBuf, Option), } /// The TLS a `--listen-tcp-tls` listener serves, reloading its certificate @@ -1075,7 +1135,7 @@ impl StreamBind { #[cfg(feature = "tcp")] Self::Tcp(..) => "tcp", #[cfg(all(feature = "uds", unix))] - Self::Unix(_) => "unix", + Self::Unix(..) => "unix", } } } @@ -1156,12 +1216,15 @@ impl StreamListeners { let alpns = self.versions.alpns(); match bind { #[cfg(feature = "tcp")] - StreamBind::Tcp(addr, tls) => { + StreamBind::Tcp(addr, tls, prebound) => { if tls.is_none() && !addr.ip().is_loopback() { tracing::warn!(%addr, "tcp listener bound to a non-loopback address; qmux is UNENCRYPTED, ensure the network is trusted"); } - let mut listener = crate::tcp::Listener::bind(addr) - .await? + let listener = match prebound { + Some(listener) => crate::tcp::Listener::from_std(listener)?, + None => crate::tcp::Listener::bind(addr).await?, + }; + let mut listener = listener .with_protocols(alpns) .with_quic(&self.quic) .with_accept_health(health); @@ -1176,9 +1239,12 @@ impl StreamListeners { pending.push(BoundListener::Tcp(listener)); } #[cfg(all(feature = "uds", unix))] - StreamBind::Unix(path) => { - let listener = crate::unix::Listener::bind(&path) - .await? + StreamBind::Unix(path, prebound) => { + let listener = match prebound { + Some(listener) => crate::unix::Listener::from_std(listener, &path)?, + None => crate::unix::Listener::bind(&path).await?, + }; + let listener = listener .with_protocols(alpns) .with_quic(&self.quic) .with_accept_health(health); @@ -1935,6 +2001,72 @@ mod tests { assert_eq!(window.stream, 256 * 1024); } + /// Stream sockets the caller holds, as a process holds the ones it took over from its + /// predecessor, are served in place of binding: a session dialed before the server + /// existed is accepted off the queue. Closing the streams then leaves the caller's + /// sockets listening and the Unix path in place, for whoever serves them next. + #[cfg(all(feature = "tcp", feature = "uds", unix))] + #[tokio::test] + async fn held_stream_sockets_are_served_and_left_to_their_holder() { + kio::clock::host_time(); // real sockets + const TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); + + let tcp = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let addr = tcp.local_addr().unwrap(); + // Short, unique, and under /tmp: macOS caps AF_UNIX paths near 104 bytes. + let path = PathBuf::from(format!("/tmp/moq-tokio-held-{}.sock", std::process::id())); + let _ = std::fs::remove_file(&path); + let unix = std::os::unix::net::UnixListener::bind(&path).unwrap(); + + let client = crate::connect::Config::default() + .init(Default::default()) + .expect("client init"); + let _queued = client.connect(format!("tcp://{addr}/").parse::().unwrap()); + + let mut listen = crate::listen::Config::default(); + listen.tcp.bind = Some(addr); + listen.unix.bind = Some(path.clone()); + let mut config = Config { + listen, + ..Default::default() + }; + config.streams.tcp = Some(tcp.try_clone().unwrap()); + config.streams.unix = Some(unix.try_clone().unwrap()); + let mut listener = config.init().expect("server init").listen().await.expect("listen"); + assert_eq!(listener.tcp_local_addr(), Some(addr), "served the held socket"); + kio::clock::timeout(TIMEOUT, listener.accept()) + .await + .expect("the queued session never arrived") + .expect("accept"); + + let _unix = client.connect(format!("unix://{}", path.display()).parse::().unwrap()); + kio::clock::timeout(TIMEOUT, listener.accept()) + .await + .expect("the unix session never arrived") + .expect("accept"); + + listener.close_streams().await; + assert!(path.exists(), "closing the streams unlinked a path it did not bind"); + std::os::unix::net::UnixStream::connect(&path).expect("the held unix socket stopped listening"); + std::net::TcpStream::connect(addr).expect("the held tcp socket stopped listening"); + assert!( + tokio::net::TcpListener::bind(addr).await.is_err(), + "the holder's socket was released" + ); + + drop((tcp, unix)); + let _ = std::fs::remove_file(&path); + } + + /// A held socket with no listener configured for it would be served by nothing. + #[cfg(feature = "tcp")] + #[test] + fn a_held_stream_socket_needs_its_listener() { + let mut config = Config::default(); + config.streams.tcp = Some(std::net::TcpListener::bind("127.0.0.1:0").unwrap()); + assert!(matches!(config.init_streams(), Err(Error::NoBackend(_)))); + } + /// A millisecond track: the frames written here carry timestamps. #[cfg(all(feature = "uds", unix))] fn timed() -> moq_net::track::Info { diff --git a/rs/moq-tokio/src/tcp.rs b/rs/moq-tokio/src/tcp.rs index 2118569c527..f8afde9ac33 100644 --- a/rs/moq-tokio/src/tcp.rs +++ b/rs/moq-tokio/src/tcp.rs @@ -355,6 +355,20 @@ impl Listener { .await } + /// Serve a listening socket the caller bound, such as one taken over from the + /// process this one replaces. + pub(crate) fn from_std(listener: std::net::TcpListener) -> Result { + listener.set_nonblocking(true)?; + Ok(Self { + listener: tokio::net::TcpListener::from_std(listener)?, + protocols: Vec::new(), + health: crate::accept::Health::new("tcp"), + session: crate::qmux_default(), + tls: None, + serve: None, + }) + } + /// Serve TLS with `config`, built from `serve` with an ALPN list that already /// names each `qmux-01.` pair to accept. pub(crate) fn with_tls(mut self, serve: crate::tls::Serve, config: std::sync::Arc) -> Self { diff --git a/rs/moq-tokio/src/unix.rs b/rs/moq-tokio/src/unix.rs index 9b3fb22a8d8..2c8eac940a6 100644 --- a/rs/moq-tokio/src/unix.rs +++ b/rs/moq-tokio/src/unix.rs @@ -266,10 +266,12 @@ fn socket_path(url: &Url) -> Option { /// Listens for incoming qmux connections on a Unix domain socket. /// /// Credentials are checked against the server's allowlist before the qmux handshake. -/// The socket file is removed on drop. +/// The socket file is removed on drop when this listener bound it. pub(crate) struct Listener { listener: tokio::net::UnixListener, path: PathBuf, + /// Whether this listener bound the socket file, and so removes it on drop. + owned: bool, protocols: Vec, health: crate::accept::Health, /// The settings each accepted qmux session starts with. @@ -299,6 +301,22 @@ impl Listener { Ok(Self { listener, path, + owned: true, + protocols: Vec::new(), + health: crate::accept::Health::new("unix"), + session: crate::qmux_default(), + }) + } + + /// Serve a listening socket the caller bound at `path`, such as one taken over + /// from the process this one replaces. The path stays the caller's, so it is + /// left in place on drop. + pub(crate) fn from_std(listener: std::os::unix::net::UnixListener, path: impl AsRef) -> Result { + listener.set_nonblocking(true)?; + Ok(Self { + listener: tokio::net::UnixListener::from_std(listener)?, + path: path.as_ref().to_path_buf(), + owned: false, protocols: Vec::new(), health: crate::accept::Health::new("unix"), session: crate::qmux_default(), @@ -399,7 +417,9 @@ impl Listener { impl Drop for Listener { fn drop(&mut self) { // Best-effort: don't leave a stale socket file behind. - let _ = fs::remove_file(&self.path); + if self.owned { + let _ = fs::remove_file(&self.path); + } } } diff --git a/rs/moq-tokio/src/worker/group.rs b/rs/moq-tokio/src/worker/group.rs index 7419608563b..1dc6936b920 100644 --- a/rs/moq-tokio/src/worker/group.rs +++ b/rs/moq-tokio/src/worker/group.rs @@ -71,35 +71,46 @@ impl Workers { /// The group's shared certificate reloads run on the caller's Tokio runtime, else /// the fallback, which must outlive the group; see [`Listen::load`](crate::tls::Listen::load). pub fn bind(mut server: crate::server::Config, config: Config) -> Result { - if server.tls.is_some() && server.listen.tls.configured() { - return Err(crate::tls::Error::ConflictingServe.into()); - } - if server.tls.is_none() { - server.tls = Some(server.listen.tls.load()?); - server.listen.tls = Default::default(); - } - - let cores = config.pin.then(cores).unwrap_or_default(); - // One resolution for the whole group. Each worker resolves its own config // otherwise, so a DNS answer that rotates between queries would hand // members different addresses and fail the bind on whichever member drew a - // fresh one. A worker group always opens QUIC, including when the source - // server config also names a stream listener. + // fresh one. let requested = materialize_bind(&mut server.listen)?; // The group owns everything a reuseport group has to get right: it takes // the port before the first member binds and holds it until the group is // dropped, refuses a size the steering filter could not address, and // binds one member per slot in the order the kernel numbers them by. - let mut group = moq_sock::shard::bind(requested, config.count).map_err(|err| match err { - moq_sock::shard::Error::Count { count, max } => Error::WorkerCount { count, max }, - moq_sock::shard::Error::Overlap { .. } => Error::WorkerOverlap { addr: requested }, - moq_sock::shard::Error::Resolve(err) - | moq_sock::shard::Error::Bind { source: err, .. } - | moq_sock::shard::Error::Steer(err) => crate::quic::Error::BindSocket(err).into(), - err => crate::quic::Error::BindSocket(std::io::Error::other(err)).into(), - })?; + let group = moq_sock::shard::bind(requested, config.count).map_err(|err| group_error(err, requested))?; + Self::from_group(server, group, config.pin) + } + + /// Serve every member of a reuseport group the caller bound, or adopted from + /// the process it replaces, one worker per member this process serves. + /// + /// Like [`Self::bind`], with the group in place of [`Config::count`]: build + /// one with [`moq_sock::shard::Config`], or take a predecessor's with + /// [`moq_sock::shard::Bound::adopt`]. `server.listen.bind` is replaced with the + /// group's address. The group stays reachable through [`Self::sockets`], to + /// steer it or hand it on. + /// + /// Fails if a member was already taken with [`moq_sock::shard::Bound::member`]. + pub fn from_group(mut server: crate::server::Config, mut group: moq_sock::shard::Bound, pin: bool) -> Result { + if server.tls.is_some() && server.listen.tls.configured() { + return Err(crate::tls::Error::ConflictingServe.into()); + } + if server.tls.is_none() { + server.tls = Some(server.listen.tls.load()?); + server.listen.tls = Default::default(); + } + + let cores = pin.then(cores).unwrap_or_default(); + + // The requested address, with an ephemeral port resolved. A worker group + // always opens QUIC, including when the source server config also names a + // stream listener, so the bind is written back for every worker to build. + let addr = group.addr(); + server.listen.bind = Some(crate::listen::Bind::Addr(addr)); let count = group.count(); let shared = Arc::new(Shared::default()); @@ -108,7 +119,9 @@ impl Workers { let mut certificates = None; while let Some(member) = group.member().map_err(crate::quic::Error::BindSocket)? { - let index = member.shard().index(); + // The worker's position in this process, not its slot in the group: a + // successor serving the second generation pins from the first core too. + let index = member.worker(); // `max(1)` because an empty core list means pinning is off, not that // there are no workers. let core = cores.get(index as usize % cores.len().max(1)).copied(); @@ -125,8 +138,19 @@ impl Workers { workers.push(worker); } - // The requested address, with an ephemeral port resolved. - let addr = group.addr(); + // A member taken before this call has no worker here, yet the group still steers + // its share of the port to it. + if workers.len() != usize::from(count) { + return Err(crate::quic::Error::BindSocket(std::io::Error::new( + std::io::ErrorKind::InvalidInput, + format!( + "{} of the group's {count} members were taken before it was served", + usize::from(count) - workers.len() + ), + )) + .into()); + } + tracing::info!(workers = count, pinned = !cores.is_empty(), %addr, "bound QUIC workers"); Ok(Self { @@ -139,6 +163,12 @@ impl Workers { }) } + /// The reuseport group the workers serve, to [steer](moq_sock::shard::Bound::steer) + /// or [hand to a successor](moq_sock::shard::Bound::handoff). + pub fn sockets(&self) -> &moq_sock::shard::Bound { + &self._group + } + /// The address every worker is bound to. pub fn local_addr(&self) -> SocketAddr { self.addr @@ -200,6 +230,18 @@ impl Workers { } } +/// The worker-group error for a reuseport group that could not form on `requested`. +fn group_error(err: moq_sock::shard::Error, requested: SocketAddr) -> Error { + match err { + moq_sock::shard::Error::Count { count, max } => Error::WorkerCount { count, max }, + moq_sock::shard::Error::Overlap { .. } => Error::WorkerOverlap { addr: requested }, + moq_sock::shard::Error::Resolve(err) + | moq_sock::shard::Error::Bind { source: err, .. } + | moq_sock::shard::Error::Steer(err) => crate::quic::Error::BindSocket(err).into(), + err => crate::quic::Error::BindSocket(std::io::Error::other(err)).into(), + } +} + /// Resolve the one QUIC address shared by every worker and write it back into /// the per-worker config so a separately configured stream listener cannot /// make [`Server::build`] treat the worker as stream-only. @@ -269,6 +311,12 @@ impl Group { self.addr } + /// The reuseport group the workers serve, to [steer](moq_sock::shard::Bound::steer) + /// or [hand to a successor](moq_sock::shard::Bound::handoff). + pub fn sockets(&self) -> &moq_sock::shard::Bound { + &self._group + } + /// The certificates the workers are serving, tracking hot reloads. /// /// Every member resolves through this same live identity. @@ -624,7 +672,7 @@ impl Worker { core: Option, shared: Arc, ) -> Result { - let index = member.shard().index(); + let index = member.worker(); let (ready_tx, ready_rx) = std::sync::mpsc::channel(); let (stop_tx, stop_rx) = tokio::sync::oneshot::channel(); let (spawn_tx, spawn_rx) = tokio::sync::mpsc::unbounded_channel(); @@ -723,7 +771,7 @@ fn run( mut spawn: tokio::sync::mpsc::UnboundedReceiver, shared: Arc, ) { - let index = member.shard().index(); + let index = member.worker(); if let Some(core) = core { pin(index, core); } diff --git a/rs/moq-tokio/tests/integration/worker.rs b/rs/moq-tokio/tests/integration/worker.rs index 37b184a32b1..fded5d28bf4 100644 --- a/rs/moq-tokio/tests/integration/worker.rs +++ b/rs/moq-tokio/tests/integration/worker.rs @@ -869,3 +869,23 @@ fn shutting_down_outside_a_runtime_joins_the_workers() { futures::executor::block_on(workers.split().shutdown()); assert!(owned.is_disjoint(&open_sockets()), "the group left the port bound"); } + +/// A group whose member was taken before it was served would leave that member's +/// share of the port to nobody, so serving it is refused. +#[cfg(target_os = "linux")] +#[tokio::test] +async fn a_group_missing_a_member_is_refused() { + let _ = rind_rustls::default_provider().unwrap().install_default(); + + let dir = tempfile::tempdir().expect("tempdir"); + let (cert, key) = certificate(dir.path()); + for taken in [1, 2] { + let mut group = moq_sock::shard::bind("127.0.0.1:0".parse().unwrap(), 2).unwrap(); + for _ in 0..taken { + group.member().unwrap().expect("a member"); + } + let mut server = moq_tokio::server::Config::default(); + server.listen = listen_config(&cert, &key, 0); + Workers::from_group(server, group, false).expect_err("served a group missing a member"); + } +} diff --git a/rs/moq-uring/src/quic/driver/mod.rs b/rs/moq-uring/src/quic/driver/mod.rs index e537c10d1ed..f07d3066fd5 100644 --- a/rs/moq-uring/src/quic/driver/mod.rs +++ b/rs/moq-uring/src/quic/driver/mod.rs @@ -105,7 +105,8 @@ pub(crate) fn endpoint_config( config.cid_generator = Some(Arc::new(move || { let mut cids = moq_quic::RandomConnectionIdGenerator::default(); if let Some(shard) = shard { - cids.set_shard(shard.index(), shard.count()); + cids.set_shard(shard.index(), shard.count()) + .set_generations(shard.generations()); } Box::new(cids) }));