From 1e370ccc697fa7409f45553d0538bbcb7f70f1e6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 01:43:38 -0800 Subject: [PATCH] quest: re-plan the routing quests for the path rule, and plan Babel's removal The route rule is the path, a withdrawal naming the dead hop, holds, and a timed doubt (routes epic Plan item 27). The quests around it still planned for Babel's distance, the seqno, ROUTE_REQUEST, and the ROUTE_END echo, and the removal itself had no quest. - scope.md builds the path router with a route per scope, sans-IO, with no wire change. - switch.md (new) puts it on the lite-07-wip wire, the draft, the Rust sessions, and js/net, and deletes Babel's router. - interest-set, route-stream, and acceptance follow the path rule and wait on the router or the switch. - route-inductive stays in m1, rewritten for the path argument. - Wording in m3 and m5 quests that cited feasibility. Co-Authored-By: Claude --- quest/m1/route-inductive.md | 41 ++++-- quest/m1/route-trace-validation.md | 13 +- quest/m2/cluster-routing/README.md | 4 +- quest/m2/cluster-routing/routes/README.md | 4 +- quest/m2/cluster-routing/routes/acceptance.md | 35 +++-- .../cluster-routing/routes/handoff-revival.md | 8 +- .../m2/cluster-routing/routes/interest-set.md | 34 +++-- .../m2/cluster-routing/routes/route-stream.md | 52 ++++--- quest/m2/cluster-routing/routes/scope.md | 138 ++++++++++-------- quest/m2/cluster-routing/routes/switch.md | 127 ++++++++++++++++ quest/m3/mesh/overlay.md | 4 + quest/m3/route-split-horizon.md | 7 +- quest/m5/browser-mesh.md | 5 +- 13 files changed, 344 insertions(+), 128 deletions(-) create mode 100644 quest/m2/cluster-routing/routes/switch.md diff --git a/quest/m1/route-inductive.md b/quest/m1/route-inductive.md index d188f9ff58f..941d0cf0a84 100644 --- a/quest/m1/route-inductive.md +++ b/quest/m1/route-inductive.md @@ -2,12 +2,15 @@ ## Goal -An inductive invariant of the route model implies `LoopFree`, `NoRevival`, -`NoFalseEnd`, `NoRegression`, and `Truthful`. Apalache checks in Nightly that -it holds initially and that every step preserves it from any state satisfying -it, on each mesh shape of up to five relays. The rules then hold for runs of -any length and any number of changes, where TLC's pass covers one claim, two -revisions, and two session opens or closes. +An inductive invariant of the route model implies `NoRevival`, `NoFalseEnd`, +`NoRegression`, `Truthful`, and the loop rule the model proves for the path +rule: no relay uses a route whose path names it, and selected next hops loop +only within the bound +[the model](/quest/m2/cluster-routing/routes/scope-model.md) found. Apalache +checks in Nightly that it holds initially and that every step preserves it +from any state satisfying it, on each mesh shape of up to five relays. The +rules then hold for runs of any length and any number of changes, where +TLC's pass covers one claim, two revisions, and two session opens or closes. ## Plan @@ -23,14 +26,26 @@ moq-net): a fix quest that joins [Finalize moq-lite-07](/quest/m2/lite07-finalize.md) then. -What it removes is the bound on the run, not on the mesh: sequence numbers, -revisions, and churn are unbounded, while the relay count and queue lengths +Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for +Babel), asked what happens to this quest now that the route rule is the path +([routes epic](/quest/m2/cluster-routing/routes/README.md) Plan item 27): +✅ keep it in m1, rewritten for the path rule (rejected: m4, since the path +argument is simpler than Babel's; deleting it and resting on the bounded +TLC runs and the sweeps). The claim rules are where the stale copies were, +and they are what a bounded run covers least. `LoopFree` leaves the Goal as +an invariant: under the path rule selected next hops can loop for a moment +when two unrelated things fail at once, which the maintainer accepted, so +the invariant implies the bound the model states instead. + +What it removes is the bound on the run, not on the mesh: revisions, +sessions, and churn are unbounded, while the relay count and queue lengths stay constants the config states. Say both bounds in the `verify-route` skill. -- Start from the argument Babel and DUAL rest on: along selected next hops the - (seqno, distance) pair strictly improves, and every message in flight carries - a value no better than its sender's feasibility distance when it was sent. - Strengthen until Apalache finds no counterexample to induction. +- Start from the path argument: no relay holds or selects a route whose path + names it, every route in flight extends the path its sender held when it + sent it, and every withdrawal in flight names a hop whose session closed. + The doubt's timer is a step that may fire at any moment, as the model has + it. Strengthen until Apalache finds no counterexample to induction. - A counterexample to induction starts from a state that may be unreachable. If TLC cannot reach it, the invariant is too weak: strengthen it. If it can, replay it on `route::tests::Sim` as the skill says before calling it a bug. @@ -40,6 +55,8 @@ stay constants the config states. Say both bounds in the `verify-route` skill. the invariant also says a route is held only from a node the holder shares a link with: every route sent adds its sender to hops that did not name it ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)). Start there. +- Apalache checks safety. That every hold ends is a liveness rule the model + checks with TLC; leave it there and say so in the skill. - Run it through the `tla` recipes with Apalache from the flake. If five relays does not fit Nightly, record the largest mesh that does and its time. diff --git a/quest/m1/route-trace-validation.md b/quest/m1/route-trace-validation.md index 0aad11a4d7d..a7652add5fe 100644 --- a/quest/m1/route-trace-validation.md +++ b/quest/m1/route-trace-validation.md @@ -29,9 +29,15 @@ Decided 2026-11-10 (maintainer, a later /quest-plan the same day): This is TLA+ trace validation (Cirstea, Kuppe, Loillier, Merz, 2024, used on CCF and etcd's raft): `route::tests::Sim` records each step as the model action it corresponds to, with the variables the model keeps, and TLC checks -that the recorded trace is a behaviour of the spec. The model fixes interest -at the root and drops scopes, so the sweeps either restrict to what it models -or the model grows to cover them; pick whichever costs less and say which. +that the recorded trace is a behaviour of the spec. Where the model covers +less than the sweeps do, the sweeps either restrict to what it models or the +model grows to cover them; pick whichever costs less and say which. + +Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for +Babel): the traces come from the path router +([routes epic](/quest/m2/cluster-routing/routes/README.md) Plan item 27). +Once the model has the path rule, a trace of Babel's router is no behaviour +of it, so this waits for the router too. Run it on the sweeps' existing seeds in CI only if it stays fast; otherwise in Nightly with the sweeps' fresh seeds. @@ -41,3 +47,4 @@ Public API: none. Wire: none. ## Required - [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - replaces who a relay believes and adds two prefixes and a scoped relay, so traces are validated against the rule that stays +- [The router runs the path rule with a route per scope](/quest/m2/cluster-routing/routes/scope.md) - the router `Sim` records, running the rule the model has diff --git a/quest/m2/cluster-routing/README.md b/quest/m2/cluster-routing/README.md index cf4311089af..25125310d7e 100644 --- a/quest/m2/cluster-routing/README.md +++ b/quest/m2/cluster-routing/README.md @@ -51,7 +51,9 @@ Decided with the maintainer, 2026-10-13: holds, and a timed doubt ([routes epic](/quest/m2/cluster-routing/routes/README.md) Plan item 27). A detour no longer waits on the publisher, and today's router keeps Babel - until [the router quest](/quest/m2/cluster-routing/routes/scope.md) lands. + until [the switch](/quest/m2/cluster-routing/routes/switch.md) lands. From + then, "feasible" in this Plan reads as: the route's path does not name the + relay and no doubt keeps the relay off it. - An origin declares its own routing identity, fresh per process or tab restart and shared across its live sessions. It is unverified. Decided 2026-11-02 (maintainer, /quest-unblock): no key proof is planned; if a use case needs a provable identity, moq-lite-07's diff --git a/quest/m2/cluster-routing/routes/README.md b/quest/m2/cluster-routing/routes/README.md index 9040e088286..75860de58cb 100644 --- a/quest/m2/cluster-routing/routes/README.md +++ b/quest/m2/cluster-routing/routes/README.md @@ -45,7 +45,7 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout 26. ✅ Decided 2026-11-11 (maintainer, /quest-plan after [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews), asked whose copy of a publisher's claims a relay believes: **a route is shared per publisher and per scope, and a relay believes a route by that route's own path** ([the prototype](https://github.com/moq-dev/moq/pull/8533), [the model](/quest/m2/cluster-routing/routes/scope-model.md), then [the router](/quest/m2/cluster-routing/routes/scope.md)). A scope is the prefix a relay asked a neighbour for when its grant there is narrower than its interest. Today only a feasible neighbour counts, one strictly cheaper than the distance the relay remembers, so a relay's claims move whenever a cost or seqno does: it can forget a claim for good, a mesh with two grants never settles after one seqno advance, and behind a scoped neighbour it never shows what a farther one knows. Only a broadcast some relay carries gets its own route, and that route propagates ([cache-aware routing](/quest/m3/cache-routing/README.md) Plan item 8). What keeps a route free of loops and stale copies is item 27. The contract's bullet on which neighbours certify stands until the model quest rewrites it. First decided the same day as any neighbour whose path to the publisher does not pass through the relay; tried on the router, two relays behind a scoped relay then keep a dead claim alive through each other, and the relay behind a scoped neighbour shows a claim it has no route to request. Rejected: that rule; believing a neighbour that is cheaper than the relay is now, which fixed the storm and the forgotten claim with no wire change but rests belief on distance; a path per broadcast as lite-06 has; link-state among relays; routes to relays. -27. ✅ Decided 2026-11-11 (maintainer, on [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533), which measured it), asked what keeps a route free of loops and stale copies: **the route's path, a withdrawal that names the hop that died, holds, and a timed doubt. Babel's distance, the seqno, ROUTE_REQUEST, and the ROUTE_END echo go.** None of it is built: [the model](/quest/m2/cluster-routing/routes/scope-model.md) proves it, [the router](/quest/m2/cluster-routing/routes/scope.md) builds it, and the contract below describes today's router until they do. The rule, as the prototype in `rs/moq-net/src/route/tests/paths.rs` runs it: +27. ✅ Decided 2026-11-11 (maintainer, on [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533), which measured it), asked what keeps a route free of loops and stale copies: **the route's path, a withdrawal that names the hop that died, holds, and a timed doubt. Babel's distance, the seqno, ROUTE_REQUEST, and the ROUTE_END echo go.** None of it is built: [the model](/quest/m2/cluster-routing/routes/scope-model.md) proves it, [the router](/quest/m2/cluster-routing/routes/scope.md) builds it, [the switch](/quest/m2/cluster-routing/routes/switch.md) puts it on the wire (item 28), and the contract below describes today's router until they do. The rule, as the prototype in `rs/moq-net/src/route/tests/paths.rs` runs it: - A hop names a relay and the session it learned the route on. A relay uses the cheapest route whose path does not name it, at once. - A relay that loses a session names that hop, and every relay drops every route through it, whoever offered it. One message per session covers every publisher and lists the routes that moved to another path. - A relay with no route left holds its claims, unavailable, until each neighbour that learned the route from it has answered: with another way, or with nothing once those behind it have answered. Then it lets go. @@ -53,6 +53,8 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps and no model. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). +28. ✅ Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for Babel), asked how item 27's removal is split and ordered: **the router first with no wire change, then a switch of the sessions, and the switch lands before the route stream.** [The router](/quest/m2/cluster-routing/routes/scope.md) runs the path rule with a route per scope, sans-IO, beside today's router. [The switch](/quest/m2/cluster-routing/routes/switch.md) puts it on the lite-07-wip wire, the draft, the Rust sessions, and `js/net`, and deletes Babel's router. [The interest set](/quest/m2/cluster-routing/routes/interest-set.md) then edits the path router only, and [the route stream](/quest/m2/cluster-routing/routes/route-stream.md) follows the switch with no echo or route request to carry. The switch's recommended wire puts the answer that ends a hold on the publisher half, behind the routes its sender sent first, so the path rule needs no order between streams that today's sessions lack for one prefix. [Acceptance](/quest/m2/cluster-routing/routes/acceptance.md) is of the switched sessions, and [the inductive invariant](/quest/m1/route-inductive.md) stays in m1, rewritten for the path, without `LoopFree` as an invariant. Rejected: the router quest in m1; one quest for router, wire, draft, sessions, and `js/net`; the route stream first, which builds the echo and ROUTE_REQUEST on the new stream and then deletes them; one wire change for the path rule and the one stream together; moving the inductive invariant to m4; deleting it. + The route layer's core landed in [moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676): a crate-private, sans-IO `Router` in `rs/moq-net/src/route/`, generic over the diff --git a/quest/m2/cluster-routing/routes/acceptance.md b/quest/m2/cluster-routing/routes/acceptance.md index 0680fd9e660..f543654485f 100644 --- a/quest/m2/cluster-routing/routes/acceptance.md +++ b/quest/m2/cluster-routing/routes/acceptance.md @@ -12,11 +12,18 @@ replace are retired. ## Plan +Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for +Babel): acceptance is of the path rule +([epic](/quest/m2/cluster-routing/routes/README.md) Plan item 27), on the +sessions [the switch](/quest/m2/cluster-routing/routes/switch.md) leaves. +The seqno request fixtures are replaced below, since the rule has no seqno +and no request. + - Port the forwarding fixtures (`Forward`, `Aggregate`, pool ties, and the sweeps over them) to the implementation, then delete the model. -- The earlier simulator persisted seqnos across restart. Re-run with the - settled fresh routing incarnation and the stale-state rules; do not carry - the old assumption into acceptance. +- The earlier simulator persisted seqnos across restart, and the path rule + has none. Re-run with a fresh routing identity per restart and the + stale-state rules; do not carry the old assumption into acceptance. - Retire [moq-dev/moq#5375](https://github.com/moq-dev/moq/pull/5375)'s quarantine, [moq-dev/moq#5391](https://github.com/moq-dev/moq/pull/5391)'s dedup, and the 300 ms hold only once the fixtures below demonstrate their @@ -53,12 +60,15 @@ link flap with many broadcasts behind one origin costs one ROUTE change per link; a ring of three where a stale neighbour cannot revive an ended broadcast; a two-uplink mesh that never carries CDN routes back to the CDN; an island reached through the CDN after a partition; an unknown Route ID -refused; a five-node ring whose preferred origin link fails recovers its -longer backup through a seqno request, duplicates suppressed, while that -backup never loses its session; a ring where a node's own preferred route -points back at the requester forwards the request to it, which carries it -on along its own route, and an unanswerable request stops once no usable -route is left. Failover is measured +refused; a five-node ring whose preferred origin link fails moves to its +longer backup, at once where the backup's path avoids the far end of the +closed link and once the doubt lifts where it runs through it, while that +backup never loses its session; a relay with several neighbours that dies +is named in one withdrawal per session, and no relay moves onto a route +through it before the doubt lifts; two relays whose only routes run through +each other both let go, neither holding for the other; and a relay that +holds lets go once the neighbours behind it have answered, including one +whose session closes without an answer. Failover is measured on broadcast liveness while the next hop changes, including a new next hop whose ROUTE arrives before its ANNOUNCE set, and reports per-path retractions and re-announcements beside the ROUTE budget; eventual route @@ -85,7 +95,12 @@ per-broadcast table rewrites or poll unrelated fronts. Measure messages, bytes, wakes/select calls, CPU, replay work, and retained state. One broadcast per origin is an explicit case. Coordinate with Route wakes and Announce replay; benchmark smoke alone does not enforce complexity, so add operation-count -regressions. Bound pending requests and freshness state under same-clock churn. +regressions. Bound held routes, doubts, and remembered dead hops under +same-clock churn. + +## Required + +- [lite-07 sessions run the path router](/quest/m2/cluster-routing/routes/switch.md) - the sessions and the route rule these fixtures accept ## Related diff --git a/quest/m2/cluster-routing/routes/handoff-revival.md b/quest/m2/cluster-routing/routes/handoff-revival.md index 04cbe0ff62a..0ddd0b6b82a 100644 --- a/quest/m2/cluster-routing/routes/handoff-revival.md +++ b/quest/m2/cluster-routing/routes/handoff-revival.md @@ -26,9 +26,11 @@ Decided the same day (maintainer, /quest-plan): ✅ plan it. ([epic](/quest/m2/cluster-routing/routes/README.md), handoff contract). If the fixture's `revivals` count misses an excuse, fix the count and say so. - Otherwise fix the cause and keep the shrunk case as a test. -- If the cause is in what - [the route per scope](/quest/m2/cluster-routing/routes/scope.md) deletes, - pin it as a `residual_` test naming that quest instead of fixing it twice. +- If the cause is in what the path rule replaces (the distance, the seqno, + route requests, the echo, or which neighbours certify a region), pin it as + a `residual_` test naming + [the route per scope](/quest/m2/cluster-routing/routes/scope.md), whose + router must pass it, instead of fixing it twice. Public API: none. Wire: none. diff --git a/quest/m2/cluster-routing/routes/interest-set.md b/quest/m2/cluster-routing/routes/interest-set.md index fd471af3227..3a526388684 100644 --- a/quest/m2/cluster-routing/routes/interest-set.md +++ b/quest/m2/cluster-routing/routes/interest-set.md @@ -5,8 +5,9 @@ The route model (`rs/moq-net/verify/route/RouteHandoff.tla`) and the router (`rs/moq-net/src/route/`) treat a neighbour as one session, keyed once for both directions, whose interest is a set of prefixes edited in place. TLC -checks every ordering of an edit against the routes, batches, floors, and -echoes sharing one FIFO per direction, and the existing rules hold: an edit +checks every ordering of an edit against the routes, withdrawals, batches, +floors, and hold answers sharing one FIFO per direction, and the existing +rules hold: an edit that swaps prefixes for a wider or narrower one keeps every claim both cover, ends nothing, revives nothing, and sends nothing twice. Nothing on the wire or in the sessions changes here; [the route stream](/quest/m2/cluster-routing/routes/route-stream.md) @@ -18,6 +19,18 @@ Decided 2026-11-10 (maintainer, /quest-plan on lite-07 routing). The model goes first because the set's edit rule is the one new rule in this redesign, and every earlier routing bug was an ordering nobody had enumerated. +Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for +Babel): ✅ this edits the path router, after +[the route per scope](/quest/m2/cluster-routing/routes/scope.md) builds it, +and the model as +[scope-model](/quest/m2/cluster-routing/routes/scope-model.md) leaves it +(rejected: landing first on Babel's router, which +[the switch](/quest/m2/cluster-routing/routes/switch.md) deletes). The path +rule has no echo and no route request +([epic](/quest/m2/cluster-routing/routes/README.md) Plan item 27), so an edit +is checked against a withdrawal that names a dead hop and against a hold and +its answer instead. + Settled: 1. ✅ **A stamp covers one prefix, named by its id** (rejected: a stamp @@ -62,19 +75,19 @@ The edit rule those three imply, to prove rather than assume: recommended: it replaces them, so there is one floor per route per prefix. - A route stays on the stream while some live prefix presents a claim of it (Plan item 15, over the set). One whose last claim an edit drops ends with - ROUTE_END after the marker, echoed as usual, with no floor first, since - nothing was withdrawn. + ROUTE_END after the marker, with no floor first, since nothing was + withdrawn. Work: -- Start from the model and `Sim` with link costs and hops - ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)), and move the echo from its own queue onto the neighbour's FIFO, as - that PR's first commit (`81c0698295`) did: with one stream per side nothing - overtakes anything. +- Start from the model and the router with the path rule. Everything a side + says to a neighbour shares one FIFO per direction, the hold answer + included: with one stream per side nothing overtakes anything. - The model has two prefixes and a scoped relay since [it has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md). Give it edits in flight: add, remove, widen, narrow, and an edit that - crosses a claim change, a route change, a ROUTE_END, and its echo. Keep it + crosses a claim change, a route change, a ROUTE_END, a withdrawal that + names a dead hop, and a hold with its answer. Keep it tractable by bounding edits as it bounds flaps. Where Settled item 1 says the merge rule is unchanged, read it as that quest's rule: a floor names its route's prefix, and the newest floor wins among one publisher's routes. @@ -104,8 +117,9 @@ Public API: none (the router is crate-private). Wire: none. ## Required - [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - the two prefixes and the rule for whose claims a relay believes, which an edit must preserve +- [The router runs the path rule with a route per scope](/quest/m2/cluster-routing/routes/scope.md) - the router this edits, with a floor per route ## Related -- [The router keeps a route per scope](/quest/m2/cluster-routing/routes/scope.md) - rewrites how the router keeps a floor per prefix; whichever lands second adapts +- [lite-07 sessions run the path router](/quest/m2/cluster-routing/routes/switch.md) - deletes Babel's router; whichever lands second adapts - [Widening an interest ends no live claim](/quest/m2/cluster-routing/routes/widen-false-end.md) - a false END on an interest change today, which the edit in place may remove diff --git a/quest/m2/cluster-routing/routes/route-stream.md b/quest/m2/cluster-routing/routes/route-stream.md index d629aff1ac1..e037f83c664 100644 --- a/quest/m2/cluster-routing/routes/route-stream.md +++ b/quest/m2/cluster-routing/routes/route-stream.md @@ -4,11 +4,11 @@ In a lite-07 session each endpoint opens at most one unidirectional Announce Stream and sends everything it says about routing on it, in order: the -prefixes it wants, the routes, claims, floors, and SYNCED that answer the -peer's prefixes, its ROUTE_END echoes, and its route requests. No routing -message can overtake another, so production is the route model's one FIFO per -direction by construction, and a ROUTE_END echo always arrives behind the -routes its sender sent first. A change of token scope edits the prefix set +prefixes it wants, and the routes, withdrawals, hold answers, claims, floors, +and SYNCED that answer the peer's prefixes. No routing message can overtake +another, so production is the route model's one FIFO per direction by +construction, whatever prefixes a side asked for. A change of token scope +edits the prefix set instead of reopening a request. `announced()` in Rust and `routes()` in JS keep their APIs, and lite-06 and older and moq-transport are untouched. @@ -21,10 +21,21 @@ the streams its peer opened, and QUIC orders no stream against another. The first count to infinity the route model found is one message overtaking another across that split, and the router keys each stream as its own neighbour (`route/node.rs`), where the model and `Sim` have one neighbour per -session. The fix does not rest on this: the hops a route carries -([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)) hold with the echo unordered, on today's streams. This removes the ordering hazard as a class and makes a +session. This removes the ordering hazard as a class and makes a token-scope change an edit. +Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for +Babel): ✅ [the switch](/quest/m2/cluster-routing/routes/switch.md) to the +path rule lands first, on today's streams, and this follows (rejected: this +first, which builds the echo and ROUTE_REQUEST on the new stream and then +deletes them; one wire change for both, which joins two XL quests). The path +rule ([epic](/quest/m2/cluster-routing/routes/README.md) Plan item 27) has +no echo and no route request, and the switch puts the hold answer on the +publisher half, so a side's routing messages for one prefix already share a +stream. What is left for this quest is the order between a side's prefixes, +which still ride a stream each, one neighbour per session in the router, +and the edit. + Settled: 1. ✅ **One one-way stream per side** (rejected: one bidirectional Announce @@ -55,19 +66,17 @@ departure is a choice to list there: reopening means no generation of streams to order against each other. - The bidirectional Announce stream leaves lite-07. - What a side sends as a subscriber: ANNOUNCE_REQUEST, now an edit (prefix - ids to remove, then prefixes to add, each with its `Hidden`), ROUTE_REQUEST, - and the echo. What it sends as a publisher: one ANNOUNCE_OK per edit it - applied, the marker where the edit takes effect, then ROUTE_START, UPDATE, - HOLD, and END, the ANNOUNCE messages, ANNOUNCE_FLOOR with a Prefix ID, and - SYNCED with a Prefix ID. ANNOUNCE_START names the Prefix ID its suffix is - relative to, and a compression base shares it. -- The echo needs its own message type. Both sides' Route IDs now meet on one - stream, so a ROUTE_END for the sender's own route and the echo of the - peer's cannot share a type. + ids to remove, then prefixes to add, each with its `Hidden`). What it + sends as a publisher: one ANNOUNCE_OK per edit it applied, the marker where + the edit takes effect, then ROUTE_START, UPDATE, HOLD, and END, the + withdrawal that names a dead hop and the hold answer as + [the switch](/quest/m2/cluster-routing/routes/switch.md) shaped them, the + ANNOUNCE messages, ANNOUNCE_FLOOR with a Prefix ID, and SYNCED with a + Prefix ID. ANNOUNCE_START names the Prefix ID its suffix is relative to, + and a compression base shares it. - A side's Route IDs, Announce IDs, and Prefix IDs are one counter each per stream. A message names the peer's id where it answers the peer: a Prefix - ID in ANNOUNCE_START, ANNOUNCE_FLOOR, and SYNCED, and a Route ID in the - echo. + ID in ANNOUNCE_START, ANNOUNCE_FLOOR, and SYNCED. - A removed prefix may still be named until the ANNOUNCE_OK for the edit that removed it, and never after. The subscriber holds a removed prefix's claims against its cap until then. @@ -106,12 +115,12 @@ per-announce-stream timer becomes per session) and the Announce Stream). Tests: every ordering hazard the old shape had, on real sessions with mocked -time. An echo never overtakes a route; a swap edit shows no claim offline and +time. A hold answer never overtakes a route, across prefixes too; a swap edit shows no claim offline and re-sends nothing both sets cover; a narrowing edit drops only what it uncovers; a scope renewal that changes the region keeps every still-covered claim shown; a second Announce stream, a nested prefix, and a set over the -size limit each close the session; a peer that withholds ANNOUNCE_OK or -echoes holds no more than the announce cap. Replace +size limit each close the session; a peer that withholds ANNOUNCE_OK or a +hold answer holds no more than the announce cap. Replace `tests/integration/lazy_discovery.rs`'s two-stream cases with edits. Update the draft's Streams tables, Announce, Floors, Routing, the message @@ -128,6 +137,7 @@ Public API: none expected. Wire: lite-07-wip only. ## Required - [The route model and router take an interest as an edited set of prefixes](/quest/m2/cluster-routing/routes/interest-set.md) - the edit rule proven in the model and the router that implements it +- [lite-07 sessions run the path router](/quest/m2/cluster-routing/routes/switch.md) - the messages this moves onto one stream, with no echo or route request among them ## Related diff --git a/quest/m2/cluster-routing/routes/scope.md b/quest/m2/cluster-routing/routes/scope.md index 4ac76089c36..7ba6ab25eed 100644 --- a/quest/m2/cluster-routing/routes/scope.md +++ b/quest/m2/cluster-routing/routes/scope.md @@ -1,19 +1,27 @@ -# [XL] The router keeps a route per scope +# [XL] The router runs the path rule with a route per scope ## Goal -`route::Router` keys a route by its publisher and the prefix it covers, and -believes a route that did not cross it, as -[the model](/quest/m2/cluster-routing/routes/scope-model.md) settled. A relay -shows a claim exactly while such a route covers it and the newest revision a -covering route gave it still has the claim, two relays never keep a -dead route alive through each other, a change of path or seqno alone changes -no claim a reader sees, and a relay behind a scoped neighbour reaches what a +`route::Router` runs the rule +[the model](/quest/m2/cluster-routing/routes/scope-model.md) settled, sans-IO, +and `route::tests::Sim` drives it. A route is keyed by its publisher and the +prefix it covers and carries its path. A relay uses the cheapest route whose +path does not name it, drops every route through a hop a withdrawal names, +holds its claims while the neighbours that learned a route from it have yet +to answer, and stays off a node in doubt until the timer lifts it. It keeps +no distance, seqno, route request, or echo. + +A relay shows a claim exactly while such a route covers it and the newest +revision a covering route gave it still has the claim, two relays never keep +a dead route alive through each other, a change of path alone changes no +claim a reader sees, and a relay behind a scoped neighbour reaches what a farther one offers. The `residual_` tests in -`rs/moq-net/src/route/tests/residual.rs` pass -as plain tests, a sweep over meshes with scoped relays holds the rules at -every quiet point, and the draft and the epic's contract say what the router -does. +`rs/moq-net/src/route/tests/residual.rs` pass as plain tests on it, and a +sweep over meshes with scoped relays holds the rules at every quiet point. + +Sessions, the wire, the draft, and `js/net` keep today's router until +[the switch](/quest/m2/cluster-routing/routes/switch.md), so nothing a peer +or an application sees changes here. ## Plan @@ -26,67 +34,69 @@ Decided 2026-11-11 (maintainer, on [moq-dev/moq#8533](https://github.com/moq-dev/moq/pull/8533)): the route rule changes too. Babel's distance, the seqno, ROUTE_REQUEST, and the ROUTE_END echo go, for the rule in the -[epic](/quest/m2/cluster-routing/routes/README.md)'s Plan item 27. That -removal is not planned yet. Plan it with `/quest-plan` before this quest -starts, as its own quest or as work here: it touches the wire (a session in -each hop, a message that names a dead hop, no ROUTE_REQUEST and no echo), -the draft, `js/net`, and most of `route/mod.rs`. The prototype -(`route/tests/paths.rs`, `compare.rs`, `copies.rs`) is then promoted or -deleted. +[epic](/quest/m2/cluster-routing/routes/README.md)'s Plan item 27. + +Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for +Babel), asked how that removal is split: ✅ two quests in this epic. This one +builds the router with no wire change, and +[the switch](/quest/m2/cluster-routing/routes/switch.md) moves the sessions, +the wire, the draft, and `js/net` onto it and deletes Babel's router +(rejected: the same split with this quest in m1, since it breaks nothing; +one quest for everything, which is one PR across the router, the wire, the +draft, the Rust sessions, and `js/net`). + +The two rules share a tree until the switch. Pick how, and say why in the +PR: the prototype (`route/tests/paths.rs`, `compare.rs`, `copies.rs`) +promoted out of `tests/` into a second router, or one router generic over +the rule. Either way the claim side is one implementation, not a copy: the +views, floors, `show`, and the interest handling are what the fixtures of +the last month fixed, and the rule changes only which routes feed them. +Sessions construct today's rule and nothing on the wire changes. Whatever +of the prototype the router replaces is deleted here. Work: -- Key `rx`, the selection, `sent`, and the views in `route/mod.rs` by route. - A view then has one floor, so delete `certifiers`, `certified`, `prune`, - `loose`, and the per-region floors, and keep the newest floor among one +- Route selection by the rule of the epic's Plan item 27, as the model + states it: the path with a session in each hop, the withdrawal naming a + dead hop and listing the routes that moved, holds and their answers, and + the doubt with its timer on the router's mocked clock. The timer's length + starts from the maintainer's 1 s; the model and the sweeps settle it. +- Key `rx`, the selection, `sent`, and the views by route. A view then has + one floor, so `certifiers`, `certified`, `prune`, `loose`, and the + per-region floors have no counterpart, and the newest floor wins among one publisher's routes as the model has it. Keep the reader's view a function of what the router holds, computed incrementally the way `show` is today, with the test build comparing it against the whole computation. - Keep a session's teardown about linear in the prefixes it held, as [the relay session hour](/quest/m0/ci/nightly-relay-session-hour.md) makes it. -- Remove what #8487 added to work around the old rule: `recompute`'s `prune` - switch, which skips pruning on a change of hops or RTT alone. Keep the - qualified-key test in `infeasible` unless the new rule makes it moot, and - say which. -- Serve a request through the route the model's rule picks, in - `route/node.rs`'s `home`, in place of every feasible route whose neighbour - `carries` the claim. -- Flip each `residual_` test, and add the review harness's scoped and - two-prefix families as a sweep beside - `sweep_no_route_outgrows_a_loop_free_path`, checking at every quiet point - that each relay shows what the rule says and nothing waits. Compare a few - thousand scripts against the router before this change and explain every - difference in the PR. -- Wire (lite-07-wip): whatever the model settled for naming a route's - prefix. List it under Impact and run `just test interop --all`. -- Draft: Routing says a route covers a prefix, and Floors says a floor - certifies its route's prefix. Replace the Floors condition "every peer - whose last advertised route to it is feasible here has answered the - current request" with the settled rule. The hold rule in Routing changes - with the route rule (epic Plan item 27). Add a changelog bullet. -- Change js/net's route layer (`js/net/src/lite/publisher.ts`, - `js/net/src/lite/subscriber.ts`, `js/net/src/lite/floors.ts`, - `js/net/src/route.ts`) in the same PR. -- Make js/net's origin apply the covering rule (`covering` in - `route/view.rs`) when it picks among a session's routes, as its consume - cache does through `Floors.covers`. Decided 2026-11-12 (maintainer, on - [moq-dev/moq#8443](https://github.com/moq-dev/moq/pull/8443)): ✅ leave it to - this rewrite (rejected: port the per-node merged view in #8443). Until then, - when a newer floor withdraws a node's claim above `live/x` while another - node claims `live` with a different instance, `forwardAnnounced` still - offers the withdrawn claim, and an origin request for `live/x` fails - `Unroutable` until the lagging stream's floor lands. Recommended: the - origin asks a session's route, per request, whether it covers the - requested path (the session's `Floors.covers`) and skips one that does not, - since an announcement at `live` cannot say it stops short of `live/x`. - Test it with the two-stream, two-instance case from #8443's review. -- Rewrite the epic's bullet on who certifies as built, and update the - `verify-route` skill's coverage and sync commit. -- Measure route and claim messages per change with - `rs/moq-net/tests/integration/mesh_sweep.rs`'s benchmark before and after. - -Public API: none (the router is crate-private). Wire: lite-07-wip. +- What #8487 added to work around the old rule has no counterpart either: + `recompute`'s `prune` switch, which skips pruning on a change of hops or + RTT alone. Keep the qualified-key test in `infeasible` only if the new + rule still needs it, and say which. +- Expose the route the model's rule picks for a request, as a crate-private + query, so the switch can serve a request through it in `route/node.rs`'s + `home`. `Sim` checks it reaches the publisher without a loop (`Reaches`). +- Run the handoff fixtures and both sweeps on the new rule. Flip each + `residual_` test, and add the review harness's scoped and two-prefix + families as a sweep beside `sweep_no_route_outgrows_a_loop_free_path`, + checking at every quiet point that each relay shows what the rule says and + nothing waits. Compare a few thousand scripts against today's router and + explain every difference in the PR. +- Report route and claim messages per change on `Sim` for both rules, on the + meshes `compare.rs` holds. The switch measures real sessions. +- Update the `verify-route` skill's coverage and sync commit for the rule + `Sim` now runs. + +Left to [the switch](/quest/m2/cluster-routing/routes/switch.md): the wire, +the draft, `route/node.rs` and the sessions, `js/net`, the epic's contract +as built, and the deletion of today's router. + +Public API: none (the router is crate-private). Wire: none. ## Required - [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - the rule, proven on two prefixes and a scoped relay + +## Related + +- [lite-07 sessions run the path router](/quest/m2/cluster-routing/routes/switch.md) - puts this router on the wire and deletes the one it replaces diff --git a/quest/m2/cluster-routing/routes/switch.md b/quest/m2/cluster-routing/routes/switch.md new file mode 100644 index 00000000000..6bf7b48af1c --- /dev/null +++ b/quest/m2/cluster-routing/routes/switch.md @@ -0,0 +1,127 @@ +# [XL] lite-07 sessions run the path router + +## Goal + +Every lite-07 session, in Rust and in `js/net`, routes by the rule of the +[epic](/quest/m2/cluster-routing/routes/README.md)'s Plan item 27, on the +router [the route per scope](/quest/m2/cluster-routing/routes/scope.md) +built. A route on the wire carries its path with a session in each hop and +the prefix it covers. A relay that loses a session names that hop in one +message, and a relay answers a neighbour that holds for it. Babel's +distance, the seqno, ROUTE_REQUEST, and the ROUTE_END echo are gone from +lite-07-wip, the draft, both codecs, and the tree: today's router is +deleted, and one route rule remains. + +A detour no longer waits on the publisher, a relay that dies costs one +withdrawal per session in place of a seqno round, and the draft and the +epic's contract say what the sessions do. lite-06 and older and +moq-transport are untouched. + +## Plan + +Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for +Babel): + +- ✅ The removal is two quests in this epic: the router first with no wire + change, then this switch (rejected: the router quest in m1; one quest for + everything). +- ✅ The switch lands on today's streams, before + [the route stream](/quest/m2/cluster-routing/routes/route-stream.md) + (rejected: the route stream first, which builds the echo and ROUTE_REQUEST + on the new stream and then deletes them; one wire change for both, which + joins two XL quests and waits on the interest set). The path rule sends no + routing message on the subscriber half, so the ordering the route stream + was planned to give the echo is not needed here. + +Recommended wire, for the implementer to confirm or improve in the PR; each +departure is a choice to list there. Where the model settled something +else, the model wins: + +- ROUTE_START and ROUTE_UPDATE carry the path as hops, each a Node ID and + the session that node learned the route on, and the prefix the route + covers as the model settled it. The seqno and the down-only bit go. The + metric stays as the cost that ranks routes, and no longer decides which + are safe. The hop limit stays the wire's 32. +- One message names a dead hop. A relay sends it once per session it loses, + to every neighbour, covering every publisher, and it lists the routes that + moved to another path, as the prototype has it. +- The answer that ends a hold rides the publisher half: the stream that + carries the answering side's routes to the relay that waits. It is then + ordered behind every route its sender sent first, on today's streams. A + relay waits only on a neighbour it listens to, since no other can hand a + stale route back. +- ROUTE_HOLD stays as the model has it (epic Plan items 14 and 19). +- ROUTE_REQUEST and the echo are removed with their types. The Node ID a + request names (epic Plan item 2) stays. +- The doubt is local: nothing on the wire carries it. Its timer is the + router's, settled by the model and the sweeps from the maintainer's 1 s. + +Work: + +- `route/node.rs` drives the new router. `home` serves a request through + the route the rule picks, in place of every feasible route whose neighbour + `carries` the claim. +- Delete today's router with what only it used: the feasibility distance and + its drain, seqno requests and their pending state, the echo, and their + fixtures. A fixture that states a claim rule moves to the new router if + [the route per scope](/quest/m2/cluster-routing/routes/scope.md) did not + move it already. +- The Rust codec and sessions (`rs/moq-net/src/lite/`), and `js/net`'s route + layer (`js/net/src/lite/publisher.ts`, `js/net/src/lite/subscriber.ts`, + `js/net/src/lite/floors.ts`, `js/net/src/route.ts`) in the same PR. A + browser is an endpoint: it sends its own route, refuses a route whose path + names it, and applies a withdrawal that names a hop. +- Make js/net's origin apply the covering rule (`covering` in + `route/view.rs`) when it picks among a session's routes, as its consume + cache does through `Floors.covers`. Decided 2026-11-12 (maintainer, on + [moq-dev/moq#8443](https://github.com/moq-dev/moq/pull/8443)): ✅ leave it to + this rewrite (rejected: port the per-node merged view in #8443). Until then, + when a newer floor withdraws a node's claim above `live/x` while another + node claims `live` with a different instance, `forwardAnnounced` still + offers the withdrawn claim, and an origin request for `live/x` fails + `Unroutable` until the lagging stream's floor lands. Recommended: the + origin asks a session's route, per request, whether it covers the + requested path (the session's `Floors.covers`) and skips one that does not, + since an announcement at `live` cannot say it stops short of `live/x`. + Test it with the two-stream, two-instance case from #8443's review. +- [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)'s warm-route + sweep (`route/tests/cache.rs`) and `Router::eligible` rest on the + feasibility distance. The reopened + [cache-aware routing proof](/quest/m2/cache-routing-proof.md) rewrites them + for the path; if it has not landed, move the sweep onto the new router's + query here and say what it no longer checks. +- Draft: Routing states the path rule, the withdrawal, holds, and the doubt. + Floors says a floor certifies its route's prefix, and the condition "every + peer whose last advertised route to it is feasible here has answered the + current request" becomes the settled rule. Remove ROUTE_REQUEST and the + echo from the message sections and the stream tables, and add a changelog + bullet. +- Rewrite the epic's contract as built, mark its Plan items 3, 7, 20, and + the echo and drain of item 25 as superseded by item 27, and update the + `verify-route` skill's rules, coverage, and sync commit. +- Tests on real sessions with mocked time: a detour that avoids the far end + of a closed link is taken at once, and one through it after the doubt; a + relay with several neighbours that dies reaches every relay as withdrawals + and no relay moves onto a route through it early; a hold ends behind a + neighbour that answers and behind one whose session closes; an unknown + hop named in a withdrawal changes nothing; a path at the hop limit is + refused. +- Measure route and claim messages per change with + `rs/moq-net/tests/integration/mesh_sweep.rs`'s benchmark before and after, + and put both in the PR. +- Run `just check`, `just test interop --all`, and `just drafts check`. + +Rollout: lite-07 is `-wip`, so a relay on this wire refuses one on the old +by misparse, not by version. Relays pinned to `main` roll together, as each +earlier lite-07 wire change did. + +Public API: none expected. Wire: lite-07-wip only. + +## Required + +- [The router runs the path rule with a route per scope](/quest/m2/cluster-routing/routes/scope.md) - the router these sessions drive + +## Related + +- [Each side says everything about routing on one stream](/quest/m2/cluster-routing/routes/route-stream.md) - moves these messages onto one stream per side afterwards +- [Route layer acceptance](/quest/m2/cluster-routing/routes/acceptance.md) - proves the switched sessions end to end diff --git a/quest/m3/mesh/overlay.md b/quest/m3/mesh/overlay.md index 5714c5adc95..db04459458f 100644 --- a/quest/m3/mesh/overlay.md +++ b/quest/m3/mesh/overlay.md @@ -47,6 +47,10 @@ no cores, do a ring of edges? I just want this to work by default, which I guess means everything is a core unless stated otherwise." Rejected: pricing alone; shaping alone; ranking links to cheaper nodes by priority, whose outcome under the feasibility rule depends on which route arrived first. +That reason was written under Babel. Under the path rule +([routes epic](/quest/m2/cluster-routing/routes/README.md) Plan item 27) a +relay takes the cheapest route whose path does not name it, whatever arrived +first. - Every node is a core unless the embedding application says it is an edge. Every node forwards; the role only changes which links it keeps and what diff --git a/quest/m3/route-split-horizon.md b/quest/m3/route-split-horizon.md index e2199c49e65..b7337ff83a9 100644 --- a/quest/m3/route-split-horizon.md +++ b/quest/m3/route-split-horizon.md @@ -5,8 +5,11 @@ A relay never advertises a route back over the session it learned it from. Since [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023), each neighbor holds two stream keys, so the router advertises a route back to its -source. That is loop-free by the feasibility condition, but it doubles route -traffic between every pair of relays. +source. That is loop-free, since the source refuses a route whose path names +it ([routes epic](/quest/m2/cluster-routing/routes/README.md) Plan item 27; +by the feasibility condition until +[the switch](/quest/m2/cluster-routing/routes/switch.md) lands), but it +doubles route traffic between every pair of relays. ## Plan diff --git a/quest/m5/browser-mesh.md b/quest/m5/browser-mesh.md index 670b5eee695..f89224ccda6 100644 --- a/quest/m5/browser-mesh.md +++ b/quest/m5/browser-mesh.md @@ -16,7 +16,10 @@ on the local network so please plan that." - ✅ The forwarding engine is the Rust router, run in the tab through the [Rust-to-TypeScript port](/quest/m5/rs2ts-js.md) (rejected: handwriting - feasibility, route requests, holds, and echoes in `js/net`, a second + the forwarding rule in `js/net`, which was feasibility, route requests, + holds, and echoes when this was decided and is paths, withdrawals, holds, + and the doubt since the + [routes epic](/quest/m2/cluster-routing/routes/README.md)'s Plan item 27, a second implementation of the part of the protocol that has produced the most bugs; pulling the port forward to m4). So the tab runs the router the route model checks, and the [P2P epic](/quest/m4/p2p/README.md)'s "no second