Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
8da8349
chore(quest): claim quest/m3/relay-restart.md
kixelated Nov 12, 2026
17e711a
feat(moq-sock): a reuseport group can hold a successor's generation
kixelated Nov 13, 2026
5ba834e
feat(moq-net): a waited restart redials once to find a replacement
kixelated Nov 13, 2026
251de15
feat(moq-tokio): serve a caller's reuseport group and stream sockets
kixelated Nov 13, 2026
b74213b
feat(moq-sock): steer a group's generation from a handle the relay keeps
kixelated Nov 13, 2026
dd8264a
feat(moq-relay): hand every socket to the relay that replaces it
kixelated Nov 13, 2026
d9bc628
docs(moq-relay): upgrade a relay in place under systemd and NixOS
kixelated Nov 13, 2026
0173325
fix(moq-relay): build the handoff's worker group off Linux too
kixelated Nov 13, 2026
9ade0c5
test(moq-net): model a waited restart's single redial
kixelated Nov 13, 2026
f60afe9
test(moq-net): move the reconnect model's sync commit to this branch'…
kixelated Nov 13, 2026
d769776
test(moq-relay): the old relay outlives a new one that dies mid-handoff
kixelated Nov 13, 2026
bfc8c01
fix(moq-relay): refuse a handoff that cannot hand on the cluster's di…
kixelated Nov 13, 2026
34e7130
fix(moq-relay): a handoff stand-in passes SIGUSR1 to its relay
kixelated Nov 13, 2026
dc0a0c8
Merge origin/main into quest/m3/relay-restart
kixelated Nov 14, 2026
69ef3b3
feat(moq-net): Restart::Handoff redials once off a healthy session; W…
kixelated Nov 14, 2026
62a64d3
chore(nix): regenerate Cargo.nix after merging main
kixelated Nov 14, 2026
2db695f
fix(moq-sock,moq-quic): reset a gone generation's connections after a…
kixelated Nov 14, 2026
6bfd5e2
fix(moq-quic): use is_multiple_of
kixelated Nov 14, 2026
ddb13e1
Merge remote-tracking branch 'origin/main' into t3/relay-restart-hand…
kixelated Nov 14, 2026
21d4277
fix(moq-relay): a relay no successor takes over from drains; review f…
kixelated Nov 15, 2026
8a33fb7
Merge origin/main into quest/m3/relay-restart
kixelated Nov 15, 2026
e9238c7
chore(verify): move the reconnect model's sync commit to the base
kixelated Nov 15, 2026
b3a49b0
quest: record the maintainer's picks for relay-handoff-routes
kixelated Nov 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions .claude/skills/verify-reconnect/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,10 @@ Each rule is an INVARIANT; the harm says why it matters. All but `NoOrphan`, `Re
```sh
quest gate just rs tla moq-net/verify/reconnect/Handover.tla
quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartWait.cfg
quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartHandoff.cfg
```

The first checks every rule over four sessions: about 1.4M distinct states, under a minute. The second checks the same rules under `Restart::Wait` over three sessions: about 300k states. Both pass at the sync commit. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed.
The first checks every rule over four sessions: about 1.4M distinct states, under a minute. The second and third check the same rules under `Restart::Wait` and `Restart::Handoff` over three sessions: about 300k states each. All pass at the sync commit. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed.

## Open findings

Expand All @@ -40,18 +41,18 @@ Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on th

## Coverage

Sync commit: `bd617cc1eebc93e0c76a362cc83f2f3852c50244`. Run `git log bd617cc1ee.. -- rs/moq-net/src/reconnect.rs rs/moq-net/src/reconnect/ rs/moq-tokio/src/connection.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.
Sync commit: `c3682eae32c19362b99292d2a1c2ddb80679dd2f`. Run `git log c3682eae32.. -- rs/moq-net/src/reconnect.rs rs/moq-net/src/reconnect/ rs/moq-tokio/src/connection.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.

| Code | Model |
| --- | --- |
| `rs/moq-net/src/reconnect.rs` `Reconnect::poll` (queue, predecessor, phases), `follow` | `Pop`, `Retire`, `IdleClose`, `FollowPred`, `Dial`, `Abandon`, `BackoffEnds`, `LiveEnds`, `ProbeAsk`, `ProbeAbandon` |
| `rs/moq-net/src/reconnect.rs` `established`, and `walk_failed` ending the first walk | `Established`, `Failed`, `Abandon` |
| `rs/moq-net/src/reconnect.rs` `connected`, `fallback`, `failed`, `probed` | `Connected`, `Failed`, `ProbeFails`, `ProbeGetsThrough` |
| `rs/moq-net/src/reconnect.rs` `end`, `closed`, `done`, `Predecessor` | `EndGoaway`, `EndClosed`, `Done`, `Retire`, `IdleClose` |
| `rs/moq-net/src/reconnect.rs` `Restart::Wait` | `LiveEnds` with `Restart = "wait"` (`RestartWait.cfg`) |
| `rs/moq-net/src/reconnect.rs` `Restart::Wait`, `Restart::Handoff` | `LiveEnds` and `WaitOut` with `Restart = "wait"` (`RestartWait.cfg`) and `"handoff"` (`RestartHandoff.cfg`) |
| `rs/moq-net/src/reconnect/redirect.rs` `Redirect::target` | `Refused` and `Endpoints` |
| `rs/moq-tokio/src/connection.rs` `run`, `Shared::connected`, `migrating`, `retired`, `offline`, `end` | `Connected`, `Apply`, `OnMigrating`, `OnRetired`, `OnOffline`, `OnDone` |

## Not covered

One configured address, so the walk across several, `ATTEMPT`, and `Verdict::Settled` and `Unusable` are left out. So are the retry window and `Failure::Exhausted`, tokens, connecting once, fixed targets, and redirect-loop scoring (`Redirects`): a session's healthiness is chosen freely instead, which covers a looped one. Backoff delays and jitter are left out; every deadline can pass in any order. `Connection::close`, `Monitor`, bandwidth forwarding, and the QUIC fresh-port retry are left out, and a session's subscriptions reduce to whether the predecessor is idle. A pass covers four sessions under `Restart::Redial` and three under `Restart::Wait`, with two endpoints a GOAWAY may name and one it may not. The tests in `reconnect/tests.rs` and `moq-tokio`'s connection tests own the timing.
One configured address, so the walk across several, `ATTEMPT`, and `Verdict::Settled` and `Unusable` are left out. So are the retry window and `Failure::Exhausted`, tokens, connecting once, fixed targets, and redirect-loop scoring (`Redirects`): a session's healthiness is chosen freely instead, which covers a looped one. Backoff delays and jitter are left out; every deadline can pass in any order. `Connection::close`, `Monitor`, bandwidth forwarding, and the QUIC fresh-port retry are left out, and a session's subscriptions reduce to whether the predecessor is idle. A pass covers four sessions under `Restart::Redial` and three under `Restart::Wait` and `Restart::Handoff`, with two endpoints a GOAWAY may name and one it may not. The tests in `reconnect/tests.rs` and `moq-tokio`'s connection tests own the timing.
8 changes: 5 additions & 3 deletions doc/bin/relay/cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,9 +216,11 @@ subscriptions go elsewhere when another route serves them, and an existing one
moves to another route of the same stream if there is one, or else finishes on
the draining peer. The drain is that relay's
own state, so it still advertises the routes onward at their real cost. A
relay keeps its session to a peer draining to restart until that peer closes
it, and redials only then, so it reaches the replacement rather than the
draining process.
relay whose session to a peer draining to restart has lived past
`backoff.healthy` redials it once, at once, which reaches a replacement serving
beside the draining process ([handoff](/bin/relay/config#handoff)). A session
drained before that, such as the redial when the draining process answered it
itself, stays until that process closes it, and the relay redials only then.

A route's price is the publisher's production cost plus the link costs it
crosses. A live publisher seeds 0, and caching never changes it.
Expand Down
42 changes: 41 additions & 1 deletion doc/bin/relay/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,8 @@ predecessor's QUIC connections, so clients reconnect at once instead of
waiting out the idle timeout. Keep it secret, and give it to one relay
only: any holder resets every other holder's connections, so live relays behind
one address would reset each other's clients unless the load balancer routes by
connection ID (`[listen.lb] id`, QUIC-LB). Relays on different addresses gain nothing from sharing
connection ID (`[listen.lb] id`, QUIC-LB), as the two relays of a
[handoff](#handoff) do. Relays on different addresses gain nothing from sharing
it, since a client accepts a reset only from its relay's address.
Create one with `openssl rand -hex 32 > reset.key`.

Expand Down Expand Up @@ -411,6 +412,45 @@ Its `ending` count distinguishes sessions already closed before the deadline
but still reporting their end. The live tally separates open sessions from
that cleanup, and the summary's elapsed time stays the same for late observers.

## \[handoff]

```toml
[handoff]
socket = "/run/moq-relay/handoff.sock" # Hand every socket to the relay that replaces this one. Linux only.
supervise = false # Run under a stand-in main process, for a systemd restart.
```

With `socket` set, a starting relay first connects to it. When a relay already
serves there, the new one takes over every listening socket the old one holds
instead of binding its own: the QUIC port, `[listen.tcp]`, `[listen.unix]`,
`[web]`, and `[internal]`. A listener whose address changed binds
afresh. Once the new relay is ready it takes every new QUIC handshake, and the
old one stops accepting and drains as on SIGTERM ([Shutdown](#shutdown)): its
sessions stay with it until they leave or `drain_timeout` passes. Cluster peers
redial once on that GOAWAY and reach the new relay, so the drain does not wait
on them. A session that stays on the old relay anyway, such as a client that
ignores GOAWAY, loses the routes it shared with those peers once they leave.
Either relay exiting leaves the other's sessions alone, and if one dies without
closing its sessions, the other answers their packets with a stateless reset,
which their clients accept when both relays share `[listen.reset] secret`.

The QUIC port is handed on as a reuseport group with room for two relays, each
steering its own connections by connection ID, so handoff needs
`[runtime] workers` (tokio or io\_uring) and halves the most workers a relay
runs to 128. The new relay serves the old one's worker count, logging a warning
when the config asks for another. A relay still serving beside the one it
replaced refuses a third until that one exits. A relay dialing static or API
cluster peers needs `[connect] bind` on port 0: their shared dial socket on a
fixed port is bound at startup and not handed on. Packets on the QUIC port that
are not QUIC, such as WebRTC, go to the newest relay, so a WebRTC session on the old
relay ends at the handoff.

Both relays name the same node while the old one drains. Each counts the
sessions it serves, and the new one holds back its [stats](#stats) broadcast
until the old one exits. Both must run as the same user. Memory and file
descriptors double while both run. [Production](/setup/prod#upgrades) shows the
systemd unit settings that make a restart a handoff.

## \[log]

```toml
Expand Down
5 changes: 4 additions & 1 deletion doc/concept/stats.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,10 @@ generate more stats.
The broadcast stays announced for the producer's life. Each announcement
carries a fresh [instance](/concept/moq-lite#publisher-instances) on its
route, so a restarted node is a new broadcast at the same path: it replaces the
old one, and no relay serves it groups cached under the old instance. A reader
old one, and no relay serves it groups cached under the old instance. A relay
taking over through [handoff](/bin/relay/config#handoff) announces its broadcast
only once the relay it replaced exits, so a node has one instance at a time;
its counters run from its start, and its first publish carries them. A reader
treats each instance as its own set of counters. The instance rides moq-lite 07
announcements; over anything else the receiving session mints one per
announcement, so a reader sees the same change as an end and a start at the path.
Expand Down
50 changes: 50 additions & 0 deletions doc/setup/prod.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,56 @@ sudo sysctl --system

macOS uses `kern.ipc.maxsockbuf`; Windows sizes each socket directly.

## Upgrades

On Linux a relay upgrades without cutting its sessions: the new process takes
over the old one's sockets and serves every new session while the old one
drains (see [`[handoff]`](/bin/relay/config#handoff)). Set `[runtime] workers`,
then make `systemctl restart moq-relay` the upgrade with the drop-in the package
installs as `/usr/share/moq-relay/handoff.conf`:
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

```bash
sudo mkdir -p /etc/systemd/system/moq-relay.service.d
sudo cp /usr/share/moq-relay/handoff.conf /etc/systemd/system/moq-relay.service.d/
sudo systemctl daemon-reload
```

The relay running when you install it started without a stand-in, so the first
restart after that cuts its sessions as before; every restart after it hands
off.

```ini
[Service]
ExecStart=
ExecStart=/usr/bin/moq-relay --handoff-supervise /etc/moq-relay/relay.toml
Environment=MOQ_HANDOFF_SOCKET=/run/moq-relay/handoff.sock
KillMode=process
RestartKillSignal=SIGUSR2
RuntimeDirectoryPreserve=restart
```

`--handoff-supervise` keeps a stand-in as the unit's main process. A restart
sends it SIGUSR2, and it passes that on and exits at once, leaving its relay to
drain while the next relay takes over and reports ready. A stop sends SIGTERM,
which the stand-in passes to its relay, as it does SIGINT and SIGUSR1 (a heap
profile's). A relay left draining is not signaled again and exits at the end of
its own drain, and a restart before then fails until it has, which
`Restart=on-failure` retries. systemd logs that the old relay "remains running
after unit stopped": that is the drain.

If the next relay never takes over, such as on a bad config or a crash at
startup, the old relay keeps serving outside the unit until `drain_timeout`
passes, then drains and exits like any other. Until then `systemctl stop`
cannot reach it and a fresh start cannot bind its port. `systemctl status
moq-relay` lists it among the unit's processes, and the journal shows the next
relay's error; fix that and restart to hand off to it, or stop the old relay
sooner with `kill -TERM <pid>`.

Both relays must run as the same user. `DynamicUser=` adopts an existing system
user named after the unit, so create one to keep the UID across restarts:
`sudo useradd --system --no-create-home moq-relay`. On NixOS, set
`services.moq-relay.handoff = true`.

## Scaling out

Connect relays into a [cluster](/bin/relay/cluster) to serve multiple regions
Expand Down
7 changes: 5 additions & 2 deletions nix/Cargo.nix

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 27 additions & 0 deletions nix/modules/moq-relay.nix
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,16 @@ in
description = "State directory for keys and runtime data";
};

handoff = lib.mkOption {
type = lib.types.bool;
default = false;
description = ''
Upgrade in place: a restart starts the new relay beside the old one, which
hands over its sockets and drains instead of cutting its sessions. Serves
QUIC from one worker per core unless MOQ_RUNTIME_WORKERS says otherwise.
'';
};

heapDumpPrefix = lib.mkOption {
type = lib.types.str;
default = "/tmp/moq-relay.heap";
Expand Down Expand Up @@ -302,6 +312,8 @@ in
"network.target"
]
++ lib.optional cfg.auth.enable "moq-auth.service";
# A changed unit restarts rather than stopping first, so a switch hands off.
stopIfChanged = !cfg.handoff;

serviceConfig = {
# The relay reports ready only once it serves.
Expand Down Expand Up @@ -330,6 +342,16 @@ in

# Network capabilities for binding to ports < 1024
AmbientCapabilities = lib.optional (cfg.port < 1024) "CAP_NET_BIND_SERVICE";
}
// lib.optionalAttrs cfg.handoff {
# Only the stand-in main process is signaled: on a restart it exits and leaves
# its relay draining, and on a stop it passes SIGTERM to the relay. A relay no
# successor takes over from drains once drain_timeout passes.
KillMode = "process";
RestartKillSignal = "SIGUSR2";
# The handoff socket lives here and must outlive the restart.
RuntimeDirectory = "moq-relay";
RuntimeDirectoryPreserve = "restart";
};

environment = {
Expand All @@ -343,6 +365,11 @@ in

MOQ_CONNECT_TLS_INSECURE = lib.boolToString cfg.cluster.disableTlsVerify;
}
// lib.optionalAttrs cfg.handoff {
MOQ_HANDOFF_SOCKET = "/run/moq-relay/handoff.sock";
MOQ_HANDOFF_SUPERVISE = "true";
MOQ_RUNTIME_WORKERS = lib.mkDefault "auto";
}
// lib.optionalAttrs (cfg.tls.generate != [ ]) {
# TLS configuration
MOQ_LISTEN_TLS_GENERATE = lib.concatStringsSep "," cfg.tls.generate;
Expand Down
15 changes: 15 additions & 0 deletions packaging/moq-relay/handoff.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Upgrade in place: `systemctl restart moq-relay` starts the new relay beside
# the old one, which hands over its sockets and drains. Needs `[runtime] workers`
# in relay.toml. Install as /etc/systemd/system/moq-relay.service.d/handoff.conf,
# then run `systemctl daemon-reload`. See https://doc.moq.dev/setup/prod#upgrades.
[Service]
ExecStart=
ExecStart=/usr/bin/moq-relay --handoff-supervise /etc/moq-relay/relay.toml
Environment=MOQ_HANDOFF_SOCKET=/run/moq-relay/handoff.sock
# Only the stand-in main process is signaled: on a restart it exits and leaves
# its relay draining, and on a stop it passes SIGTERM to the relay. A relay no
# successor takes over from drains once drain_timeout passes.
KillMode=process
Comment thread
kixelated marked this conversation as resolved.
Resolved
RestartKillSignal=SIGUSR2
# The handoff socket lives here and must outlive the restart.
RuntimeDirectoryPreserve=restart
6 changes: 6 additions & 0 deletions packaging/moq-relay/nfpm.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,12 @@ contents:
file_info:
mode: 0644

# Opt-in: copied into moq-relay.service.d to make a restart a handoff.
- src: packaging/moq-relay/handoff.conf
dst: /usr/share/moq-relay/handoff.conf
file_info:
mode: 0644

# Per-packager overrides for distro-specific runtime dependencies.
overrides:
deb:
Expand Down
Loading