Repository navigation
feat(moq-net)!: lite-07 sessions run the new route rule - #8987
Conversation
Quest-Claim: ed8f90ca9728e60f
Work in progress: Rust sessions drive route::relay::Router; the old rule's router, proof, and tests are deleted. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…fixtures Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…route key js/net declares no Node ID, sends and reads claims only, and refuses route messages. moq-crypto loses its endpoint key module and the relay its route_key option. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The moq-lite draft specifies lite-07's one announce stream per side: claims, routes, floors, DEAD and DRAIN, with the Node ID as a relay and its start. The route key, its proof and model, the old rule's TLA+ model, and the quests they answered are gone. js/net keeps the rendezvous hash for claimed routes. Fixes found driving the rule from every crate's tests: - a routing peer granted an explicit everything pattern is whole, not confined - the origin reads both of its cursors before feeding its route node, so an endpoint that closes or replaces a broadcast ends it for its reader instead of showing it lost - every session that gives a node its own claim ranks alike, so a reconnect serves the moment it lands and a GOAWAY handover empties the old session Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…itch The route key flags main still listed in lite/version.rs stay removed, as does moq-crypto's endpoint test, and a publisher test main added names no node. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…r holds Review findings: - a prefix removed and added again in one edit is told its claims and SYNCED again, as the peer dropped them - a routing neighbour's claim changes since the last floor keep one per claim - a session sends one interest edit at a time, the next when it is answered Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…itch The draft keeps neither side's dropped references: main's WebRTC binding and Padding Stream, and this branch's Route Key. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…itch The nightly model matrix keeps main's warm leg and drops the route legs, whose model this branch deletes. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Maintainer decision (2026-11-17, asked in the coordinating session with this PR's report):
(Written by Claude Fable 5.2 with high effort) |
…d the maintainer's decisions A relay keeping its identity across restarts with its clock behind its last start peered and routed nothing: every node holding a closed session of the later run ignores the new run's sessions. The session is refused at setup instead, with the reason logged, as the draft already said. The quest records the 2026-11-17 decisions and moves the adopted-claim gap to what lands before the cut. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…itch moq-crypto keeps only the provider and the CPU check: main moved e2ee to moq-e2ee and left the endpoint key, whose one caller was the Route Key this branch removes. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…est, timer published under the lock A lite-07 subscriber sends one edit and waits for its ANNOUNCE_OK, so a publisher holds one unanswered edit however slowly its peer reads. The draft requires it and both publishers refuse a second. A non-routing session whose wanted prefixes pass one ANNOUNCE_REQUEST asks for them more coarsely inside its grant, as the path rule did, instead of sending a set the peer closes the session on. The route node publishes its next deadline before releasing its lock, so a deadline read earlier on another thread cannot replace a later one. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…itch Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
There was a problem hiding this comment.
ℹ️ No blocking defects in the code I read. The draft misstates two router rules a third-party implementation would hit, and some release notes for the deleted Route Key are stale.
Approach: sound. Running every lite-07 session on the one relay::Router, and deleting the path rule, holds, doubt and the Route Key instead of keeping them alongside, is the right call for a -wip wire that isn't in the default version set.
Reviewed changes I read route/relay.rs, the rewritten route/node.rs, model/origin.rs, lite/setup.rs, lite/session.rs, and the coarsening and one-edit fixes in 7c4c1532e. I compared the rewritten draft against the codec and router, and looked for stale references to the deleted rule across the tree.
Routersession API:ask/asking/answeredkeep a queue of unanswered edits, andMsg::Editedapplies the oldest one.editnow applies removes and adds as the edit names them, so removing a prefix and adding it again re-tells its claims.- Serving every way:
ways()serves an adopted claim through every giver and another node's claim through every usable route.withheld/crossed_markkeep a claim that crossed a mark away from marked neighbours. Error::Regressed:connectrefuses a routing neighbour whose start is below a session of the same relay already known closed.route/node.rs: the session node is rewritten aroundOutbox { wire, serve, limit }, withlimit = held*2 + 1024, plus per-claimantSlots withOrphanhand-off. The timer deadline is now published under the lock.- Origin:
Consumer::node_iddeclares a Node ID only for a proxying, peer-linked, whole-tree pair.OwnClaims::polldrains both cursors before feeding. Claimed routes rank by the standard-basis rendezvous hash, the same as JSrouteKey. - SETUP: lite-07
PARAM_HOPis now{relay, start}bytes, and a relay of 0 or trailing bytes is refused. - Non-routing asks:
coarsenwidens a want set until it fits one ANNOUNCE_REQUEST. Each step merges a group into its longest common prefix or its grant region, otherwise it drops the last prefix and logs a warning, so the loop always ends. - Draft: the message type numbers, field orders, flag bits, SETUP parameter IDs, the ROUTE/DEAD/DRAIN validation, and the UNAUTHORIZED rule for a confined grant all match the Rust codec. The two mismatches are inline.
ℹ️ The Route Key never shipped, but a release note for it would
The release branch has no moq-crypto crate and no route_key anywhere, so the Route Key only ever existed on main. Even so, rs/moq-net/MIGRATE.md still has a ## Unreleased entry, "Origins prove a route key (#7937)", which the next release cut would publish for an API that no longer exists. The Impact section's migration notes for cluster.route_key, moq_crypto::endpoint/serde, and origin::RouteKey likewise describe removing things no release ever had.
Technical details
# Stale release notes for the never-released Route Key
## Affected sites
- rs/moq-net/MIGRATE.md:38-41 — `### Origins prove a route key (#7937)` under `## Unreleased` describes `origin::Config::route_key` and `poll::Session::export_keying_material` proofs, which this PR deletes.
- PR body, Impact → Migration notes — the `moq-relay` `cluster.route_key`, `moq-crypto` `endpoint`/`serde`, and `moq-net` `origin::RouteKey` bullets.
## Required outcome
- The release cut drops the `Origins prove a route key` entry rather than shipping it.
- The Impact section says so, since `sh/migrate.sh` rejects editing `MIGRATE.md` outside a cut, and notes that the route-key bullets need no user migration because none was released (`git show release:rs/` has no `moq-crypto`).
- Check whether `export_keying_material` survives this PR. If it was only for the proof, the second sentence of that note is stale too.ℹ️ Docs still point to the deleted verify-route skill
Two unchanged files say "verify-route and the route proof cover how routes settle", but this PR deletes that skill and its TLA+ model. A reader following them looks for a check that no longer exists, when only the Agda proof (#8904) is left.
Technical details
# Stale `verify-route` references
## Affected sites
- .claude/skills/verify-warm/SKILL.md:63 — "`verify-route` and the route proof cover how routes settle"
- rs/moq-net/verify/warm/WarmSettle.tla:38 — "(`verify-route` and the route proof cover how routes settle)"
## Required outcome
- Both point to what now covers route convergence (the Agda proof from #8904), or drop the reference.ℹ️ Two checks did not finish
I asked two other checks whether a peer can make the new Rust session wiring panic or cost unbounded work, and whether JS and Rust agree byte for byte on the lite-07 wire. Neither had reported when this review was submitted, so this review does not cover those two questions.
claude-opus-5-5 | 𝕏
|
Follow-up to my review above. The two checks I flagged as unfinished have now reported. One found a real bug this PR introduces.
|
…l, cap a claim chain at 31 A claim whose path a receiver could not resolve was dropped whenever any prefix was removed, while its publisher kept it, so the publisher's later update or end of it closed the session. It now goes only when no earlier prefix is left for it to lie in. Rust and JS. A claim's chain names at most 31 relays on the wire: 32 was decoded and then refused by the router as malformed. The draft says what the router does: a routing neighbour is sent routes only while its set is the empty prefix and may ask for nothing else, nothing but an answer precedes the first ANNOUNCE_OK, and a node may cut off a peer that reads too slowly. Adds a quest for the finish interop lane's intermittent UNROUTABLE. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Answers to the findings in the review summary and its follow-up, all in d98a191:
(Written by Claude Fable 5.2) |
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes The single commit since the prior pullfrog review (d98a191) fixes that review's two follow-up findings and brings the draft in line with the router.
- An unresolved claim now survives an unrelated prefix removal:
Routed::okand its JS twin#ok()drop a claim whose path could not be resolved only when the edit leaves no prefix that was in effect before it, instead of on any removal, so an honest peer's laterANNOUNCE_ENDorANNOUNCE_UPDATEfor that id no longer closes the session. Both sides gained a regression test that fails without the fix. - A claim's chain now caps at 31 relays on the wire:
encode_claimrefuses a chain atroute::MAX_HOPS,decode_claimrefuses a count at it, and JSCHAIN_MAXdrops to 31 with its>comparisons unchanged. The new codec test round-trips 31 and refuses 32 in both directions. - The draft states what the router does: a routing neighbour is sent routes exactly while its set is the empty prefix, and an edit that empties it drops every route and its claims at the ANNOUNCE_OK with no ROUTE_END; a set that is neither
{}nor{""}and any message before the first ANNOUNCE_OK (other than ANNOUNCE_REQUEST) are PROTOCOL_VIOLATIONs; a node may cut off a peer whose reads fall behind the changes sent to it. - Two quests record what is left:
quest/m0/test-flakes-3/interop-finish-unroutable.mdwrites up the intermittentfinishlane UNROUTABLE, andquest/m2/multicast/switch.mdadds removingpoll::Session::export_keying_materialbefore the cut alongside the staleMIGRATE.mdrelease note.
I checked the survives computation against the publisher's drop rule for plain removals, a remove plus re-add of one path, and the {""}→{} routing edit, and confirmed a 31-relay chain is the longest a conforming origin can advertise (Origin::forwardable filters at MAX_HOPS), so the tightened encoder never fires on a legitimate send. 16 moq-net tests (lite::announce::chain_tests, lite::subscriber::routed::tests) and 208 @moq/net tests in subscriber.test.ts and announce.test.ts pass.
deepseek-v4.1-flash | 𝕏
|
Maintainer decision (2026-11-18, asked in the coordinating session with this PR's report):
Still open with the maintainer, and not blocking this PR since lite-07 stays wip until the cut: whether "a subscriber MUST NOT send an edit before its last is answered" stays a MUST. They asked whether it is enforceable and for a fuller explanation. (Written by Claude Fable 5.2 with high effort) |
|
Answering the decision in #8987 (comment): it asks for no change to this head. The quest it names is already in this PR, and the open question about the edit rule is recorded for a later PR before the cut. (Written by Claude Fable 5.2 with high effort) |
|
Maintainer decision (2026-11-18, in the coordinating session), closing the question left open in #8987 (comment): the edit rule stays a MUST. A lite-07 subscriber sends no announce edit before its last is answered, and a publisher closes on a second. Rejected: dropping the rule and having the publisher fold unanswered edits into one. (Written by Claude Fable 5.2 with high effort) |
main's new route rule (#8987) replaced the code this PR's route tests targeted. cargo-mutants over the route code it changed left 31 survivors; each now has a test, an exclusion with its reason, or is gone with the dead code it mutated. Exclusions naming deleted code are dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#8987 deleted the old rule's route tests, so code it kept lost the tests that killed its mutants. A sweep of the rest of the route code found 27 more survivors: each gets a test or an exclusion with its reason, and the dead code behind four goes (the reader's unread change list, a floor guard that always holds, and a check another check covers). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The switch (#8987, #9052, #9053) replaced lite-07's path rule. Delete the switch quest, the Playwright bump whose only reason was the route key lane, and route fan-out, whose surviving cost moves to route benches. Retarget federation, the routing guide, route wakes, handoff routes, regrant, interest-stream caps, the mesh condition, and route memory to the route layer on main, and trim the two route epics to what still stands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Claims quest/m2/multicast/switch.md.
Every lite-07 Rust session now drives
route::relay::Router(#8901) on one Announce Stream per side, the draft specifies it, and the old route rule is deleted: ROUTE_HOLD, doubt, DEAD's moved list, the Route Key with its proof and model, qualified Node IDs, routes per prefix,path::Router, andRouteHandoff.tla.js/netimplements the endpoint half only. lite-03 to lite-06 sessions are untouched on the wire.What changed
-wip): one unidirectional Announce Stream per side carries everything about discovery and routing. Message types are dense: 0 ANNOUNCE_START, 1 ANNOUNCE_END (id, lost), 2 ANNOUNCE_UPDATE, 3 ANNOUNCE_FLOOR, 4 SYNCED, 5 ANNOUNCE_REQUEST, 6 ANNOUNCE_OK, 7 ROUTE, 8 ROUTE_END, 9 DEAD, 10 DRAIN. A claim is cost, above, flags (unavailable, draining), priority, and a chain. SETUP carries a Node ID (relay and start) only from a routing node; the Route Key and Lite-06 Chain parameters are gone. Requests name the node announcing their claim. Routing-only messages on a session that does not route are a PROTOCOL_VIOLATION.@moq/netis an endpoint. It names no node, ignores routing messages it would never be sent, and ranks claimed routes by the same rendezvous hash as Rust.drafts/draft-lcurley-moq-lite.mdrewrites Announce, Claims, Floors, Routing, Closed Sessions and Drains, and Resolution, and drops Holds, Refused Content, and the Route Key.rs/moq-net/verify/route,verify/route-key, their skills and nightly legs,moq_crypto::endpoint, the relay'scluster.route_key, the interop route-key lane, and four quests about the old rule.Router and origin fixes
Driving the router from sessions showed these, fixed here:
Router::waysserves a node's own (adopted) claim through every neighbour that gave it, and another node's claim through every usable route, so a reader already holds another copy when the one it follows closes.Router::withheld: a node's own claim that crossed a marked link is never told to a marked routing neighbour.**) is whole. It was refused as confined, which closed every real cluster link.Found by the local review and fixed:
hiddensends, is told its claims and SYNCED again. The router compared the sets, saw no change, and the peer was left with nothing.Found by the local review of the iterate and fixed:
maindid. Without it the peer closed the session.Found by Pullfrog's review of the ready head and fixed:
Found answering the maintainer's question about a clock that goes back:
route::Error::Regressed, and the refusing side logs why. Before, nothing refused or logged it: a node holding such a floor silently ignored every route through the restarted relay's sessions until its clock passed the old Start.moq-relayis not exposed, since its identity is random per start; only a library user that pinsorigin::Config::nodeacross restarts is. The config's docs anddoc/bin/relay/cluster.mdsay so.Validation
On the head
d98a191b42:just check: 12,566 Rust tests pass (161 skipped), 3,314@moq/nettests pass, drafts, drills, and the rest green, run insidenix develop.just test interop --all: 208 checks pass, none fail or skip. This runs the built relay against Rust, Node, Bun, and Python clients over WebSocket and WebTransport on lite-05, lite-06, lite-07, and moq-transport, including theclaimsandsocketlanes on lite-07.moq-relayupstream_linksbuilds two-tier clusters of real relays over TLS on TCP speaking only lite-07 and checks tiered regions, that an edge never transits between cores, and an uplink transiting both ways.moq-tokiomigratemoves live subscriptions in both directions across a GOAWAY and checks the old session closes once idle.route/tests) still run the prototype (multi.rs) beside the router, and the drill mutations were regenerated and shown to fail both drills when applied.an_endpoint_closing_its_broadcast_ends_it_for_its_reader,an_endpoint_replacing_its_broadcast_ends_the_old_one, the**case ina_routing_neighbour_keeps_a_whole_permit,a_batch_keeps_one_change_per_claim,interest_changes_wait_for_the_unanswered_edit, the re-added prefix case in the router's edit test,a_start_that_went_back_is_refused,a_set_past_one_request_is_asked_for_more_coarsely, and the second-edit cases ina_lite07_request_naming_no_prefix_or_nested_is_a_violationand its JS twin. Alsoan_unresolved_claim_outlives_another_prefixs_removalwith its JS twin, which fails without the fix, anda_claim_chain_at_the_hop_limit_is_refused. The deadline race has no test: it needs a thread interleaving the harness cannot force.Not run: the nightly benches and the TLA+ models that remain (failover, resume, subscribe, reconnect).
moq-csession_tokens_and_grantsandserver_accepts_a_sessionrace a client closing itself against the test closing it. They fail 15 of 100 runs at the merge base on lite-06, so they are not from this change and are not fixed here. Owner:quest/m0/test-flakes-3/c-server-accepts-session.md, which #8964 adds.@moq/net"a reset Announce Stream closes moq-lite-07 with PROTOCOL_VIOLATION" timed out in 2 of 7 full-suite runs earlier on this branch. #9024 fixed it onmainand is merged here.Open, not root-caused:
just test interop --allfailed itsfinishlane in 3 of about 30 runs, all while the machine was loaded by other builds, and in none of 19 runs once the load fell. Each time the relay answered UNROUTABLE to the first few readers that subscribed to a broadcast about 5 ms after its publisher's session announced it, on a session that had just ended another broadcast; readers a few milliseconds later were served. Tracing ruled out the route node's own refusals, which leaves the origin beneath it. Whethermaindoes the same was not checked. It is written up as quest/m0/test-flakes-3/interop-finish-unroutable.md, added here.The moq-net
auth/waiting/moq-lite-07-wip/1/1bench still hangs on this head, in warm-up, as it does onmain. quest/m0/nightly/lite07-route-hang.md owns it and asked for this rerun as its first step; its plan is then to fix it on top of this PR. The other three benches it lists were not run.Impact
Breaking, wire and public API.
Wire:
moq-lite-07-wipchanged incompatibly under the same name. It also gains rules a third-party implementation must follow: one ANNOUNCE_REQUEST edit at a time, the next after the ANNOUNCE_OK; a claim chain of at most 31 relays; and a routing neighbour asks for the empty prefix or nothing. A build from before this PR does not interoperate with one after on lite-07. The default version set excludes lite-07, so only a deployment that enabled it by name is affected: it upgrades both ends of a lite-07 link together, or links them over lite-06 for the rollout. lite-03 to lite-06 and moq-transport are unchanged.The Route Key never shipped: the
releasebranch has nomoq-cryptocrate and no route key. The notes below aboutcluster.route_key,moq_crypto::endpoint, andorigin::RouteKeymatter only to a build frommain, and the release cut should drop the## Unreleasedentry "Origins prove a route key" fromrs/moq-net/MIGRATE.mdrather than publish it (sh/migrate.shrefuses that edit outside a cut).Migration notes:
moq-relay: thecluster.route_keysetting is removed. Delete it from the config; nothing replaces it.moq-crypto:moq_crypto::endpoint, its defaultendpointfeature, and theserdefeature are removed with the Route Key. Nothing replaces them; the crate keepsproviderandcpu.moq-net: a lite-07 session whose routing peer declares a Start before a session of that relay known closed fails withroute::Error::Regressed. It reaches only a node that keepsorigin::Config::nodeacross restarts while its wall clock goes back.@moq/net:WireRoute.nodeandWireRoute.depthare replaced byclaimed?: boolean;Floors, the coverage plumbing, and route key support are removed; a publisher'sclaimsoption is per instance.moq-net:origin::RouteKeyand the origin's route key setting are removed. Nothing replaces them: a routing node is one whose origin forwards (Producer::with_proxy) on a peer link with an unconfined grant.offline) on lite-07, not ended.Decisions
Recommendations from the brief, confirmed or changed:
(Config::node, start)with the start in wall-clock microseconds.0xcbf29ce484222325) over the path and instance, keyed with the claiming relay in Rust. JS keeps the rendezvous hash for claimed routes, over the instance.lost.Taken where I would have asked:
Producer::with_proxystays as the switch that makes an origin a routing node.Decided by the maintainer on 2026-11-17, in this comment:
Config::update_hold,Source,RouteInfo::source(), andConsumer::local()crate-private. Both land in the follow-up quest before the cut. Rejected: holding this PR for them.Follow-ups
The quest is rewritten as "[L] What the route switch left" and keeps:
Before the cut:
Config::update_hold,Source,RouteInfo::source(), andConsumer::local()crate-private; their callers are tests and benches outside the crate that need another way in first.Left for the implementer to decide:
New quest, in the open flakes epic at m0 where flakes go: "[M] A request right after its announcement is not refused unroutable" (
quest/m0/test-flakes-3/interop-finish-unroutable.md), for the intermittentfinishlane failure above.The follow-up quest also gains: remove
poll::Session::export_keying_material, which has no caller since the Route Key proof left and was never released.No quest for the two test races the maintainer asked about: the
moq-crace has an owner, named in Validation, and the@moq/netrace is fixed onmain.Caused by
Models
Size: XL
Implementer: Claude Code, claude-fable-5-2, unknown effort
Implementer: Claude Code, claude-fable-5-2, unknown effort (iterate)
🤖 Generated with Claude Code
Review
Rewritten by
quest reviewafter each round.Head: d98a191
Base: origin/main at 789ece4
Risk: high,
it changes .github/workflows/nightly.yml, which is always high: workflows run with the repository's secrets and decide what merges and shipsClassified: high by the always-high paths
Reviewer: Codex, gpt-6.1-sol, xhigh effort
Session:
01a1eb12-3e83-73f3-a74b-92e4f52171e3, round 4Score: 10/10,
Source inspection found no new issues in the announce-ID lifetime fixes, synchronized chain limits, or draft updates; builds and tests were not run.Rounds: 8; first round 3 (high 2, medium 1); findings 6 (high 3, medium 3): fixed 6, refuted 0, open 0
Findings: none