Repository navigation
fix(moq-net): never stitch two broadcasts (release) - #9043
Conversation
A front resumed a broadcast through another route to the same publisher when its source closed, and spliced a better route in while serving. A publisher that re-creates a broadcast at the same path restarts its group numbers, so the relay stitched the new broadcast onto the old one's sequence and viewers waited on groups that never came. A front now ends when its source closes or when another route becomes best; readers re-request and get a fresh front. Nothing splices a second source into a front, and a relay always reads through the route it advertises. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A front never changes routes now; it ends. A request arriving after another route became best, before the driver ended the front, joined the front about to end. Admission now checks the route the front's source came through, not just its publisher. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A front that ends while a route still serves its path changed nothing the announce cursors could see, so a browser or downstream relay holding the ended broadcast never requested a fresh one: it stalled offline, or a downstream relay re-spliced the new broadcast at the old position. The front now emits Bounce before End when it served, and the origin re-announces the exact path (a retraction, then the route again) to every cursor presenting it. Covering-prefix routes are left alone. Failover across relays is no longer seamless: the GOAWAY cluster tests now assert the old subscription ends and a fresh request reads the new leg, and the cluster doc says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As main does for a lite-06 (minted) instance: asking the peer again could reach another broadcast under the same path, since the peer's front may have ended and a fresh one taken over, so the track ends with the error and a returning reader is not re-spliced. A local broadcast's copy still re-splices, as on main, since it is the one broadcast published here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Important
Two findings verified at this head: a serving front that ends with best == None (a route renamed in place) still leaves its consumers stalled, and broadcast_route_migration in moq-tokio panics on the new contract, so just check and CI fail.
Approach: sound for a release-line fix — a front serves one source for its whole life, so a re-created broadcast can never be stitched onto an old one, at the documented cost of a visible restart on every failover.
Reviewed changes
- One source per front:
Front::selectedends a serving front on any route change andsource_closedalways ends it, so nothing is re-spliced. - Announce bounce: an ending front emits
Action::Bounce, andOriginState::bouncere-announces the exact path (retraction, then the route) to each cursor presenting it there, so consumers re-request; covering prefixes are deliberately left alone. - Join gate:
RemoteFront::pinnow carries the route the front requested through, and a request joins only while that route is still the best. - Peer copy end: a peer's track copy dying after delivering now aborts the track instead of re-querying the same source; a local copy still re-splices.
- Tests: front/origin tests rewritten to the restart contract, new returning-publisher and bounce tests, plus
rejoin.rsandgoaway_cluster.rsupdated.
⚠️ moq-tokio's route-migration test still asserts the seam this PR removes
broadcast_route_migration (rs/moq-tokio/tests/broadcast.rs:991) reads the standby's groups off the same subscription after the preferred session dies, and asserts the path never retracts. At this head it panics with track closed at broadcast.rs:970, and sh/rs/select.sh selects moq-tokio as a dependant of moq-net, so just check and CI run it. Rework it the way goaway_cluster.rs was reworked.
Technical details
# moq-tokio route-migration test asserts the removed seam
## Affected sites
- `rs/moq-tokio/tests/broadcast.rs:991` `broadcast_route_migration` — line 1117 `read_payloads(&mut sub, 2)` expects `["b2","b3"]` on the subscription that served `a1`; lines 1131-1133 assert every later announce is active.
- Failure at this head: `cargo test -p moq-tokio --test broadcast -- broadcast_route_migration` -> panic `track closed` at `rs/moq-tokio/tests/broadcast.rs:970` (`recv_group` returned `Ok(None)`).
- `rs/moq-tokio/tests/broadcast.rs:1406` `route_reannounce_test` is unaffected: the first hop stays `0x4444` and the route updates in place, so the front keeps serving; `broadcast_route_reannounce*` pass at this head.
## Required outcome
- The test asserts the new contract: the old subscription ends without any standby group, the path retracts and re-announces, and a fresh `request_broadcast` reads `b2`, `b3`, and `b4`.
- Otherwise the release branch ships a test that asserts the opposite of the code, and the PR gate stays red.
## Open questions for the human
- Whether `moq-tokio`'s cross-route migration is now a documented non-goal (drop the test) or should get a drill for the restart contract.ℹ️ Public rustdoc still promises the seamless re-splice
Consumer::request_broadcast (rs/moq-net/src/model/origin.rs:4696-4701) still says the front "re-splices through the best route sharing its first hop at a group boundary, invisibly to subscribers", which is exactly what this PR removes. Same for OriginState::fronts (origin.rs:3666-3670), the front.rs module doc (lines 3-5), and Identity::Local/Publisher (front.rs:142-154, "safe to resume through").
Technical details
# Docs describing the removed re-splice
## Affected sites
- `rs/moq-net/src/model/origin.rs:4696-4701` — public `Consumer::request_broadcast` doc, "re-splices ... invisibly to subscribers".
- `rs/moq-net/src/model/origin.rs:3666-3670` — `OriginState::fronts` field doc, "re-splices it through routes sharing its first hop".
- `rs/moq-net/src/model/front.rs:3-5` — module doc, "re-splicing when the source is replaced".
- `rs/moq-net/src/model/front.rs:142-154` — `Identity::Local` ("a newer local publisher takes over while the incumbent is live") and `Identity::Publisher` ("safe to resume through").
## Required outcome
- Each describes one source per front and that any route change ends it and bounces the path, matching `doc/bin/relay/cluster.md` and the code.deepseek-v4.1-flash | 𝕏
… one source A route renamed in place (its first hop changed, path unchanged) shows the announce cursors only an update, so a front ending on it left the old broadcast announced active. The front now bounces in that case too. The front, origin, and request_broadcast docs still described resuming through routes sharing a first hop; they now say a front serves one source and ends with a bounce. moq-tokio's route migration test asserts the new contract: the old subscription ends and a fresh request reads the new route. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed review 5949783260 in 88f1371:
(Written by Claude Opus 5.5) |
There was a problem hiding this comment.
ℹ️ No critical issues. The delta is sound and both of its new tests pass here; one stale internal comment still describes the re-splice this PR removes.
Approach: unchanged, and sound for the release line: one source per front, and this commit closes the last end path that did not bounce.
Reviewed changes
Changes since the prior pullfrog review (83590a3), verified at 88f1371:
- No-candidate bounce:
Front::selected'sNonearm now pushesBouncebeforeEndwhenever the front had a source (self.resolved). A route renamed in place, or an anonymous route gaining a first hop, therefore retracts and re-announces its path instead of leaving consumers on an ended broadcast whose path stays active. A cursor that presents only a covering prefix still sees nothing, sinceTableCursor::bouncerequirescurrent.prefix == path. - Doc pass: the
front.rsmodule doc,Identity::LocalandIdentity::Publisher,OriginState::fronts, and the publicConsumer::request_broadcastdoc now describe one source per front and the bounce, replacing the re-splice promises. - Regression test:
a_route_renamed_in_place_bounces_its_announcecovers the arm (it fails with the guard removed, which I checked), and the two front tests that end through it now expect[Bounce, End]. moq-tokiotest reworked:broadcast_route_migrationnow asserts that the old subscription ends without any standby group, that a fresh request readsb2,b3, andb4, and that the path ends announced.
I ran the moq-net front and origin tests for this arm plus cargo test -p moq-tokio --test broadcast -- broadcast_route_migration --exact at this head; all pass.
ℹ️ One comment still promises the re-splice this PR removes
Consumer::request_broadcast's body comment (rs/moq-net/src/model/origin.rs:4773-4776) still says the watcher "re-splices the front through routes sharing its first hop for as long as one serves", four lines below the public doc that was just corrected. A reader following it would reintroduce the seam this PR closes.
Technical details
# Stale re-splice comment in `request_broadcast`
## Affected sites
- `rs/moq-net/src/model/origin.rs:4773-4776` — "The watcher materializes the path from the best covering route, resolves the channel, and re-splices the front through routes sharing its first hop for as long as one serves." It predates this PR (`98d19aa9c`) and the doc pass in `88f13713` missed it.
## Required outcome
- The comment states the current contract: the watcher materializes the path from the best covering route and serves that one source until it ends or another route becomes best, when the front bounces the path and ends.
## Suggested approach
- Reuse the sentence in the public doc directly above (`origin.rs:4696-4700`).(Written by DeepSeek V4.1 Flash)
deepseek-v4.1-flash | 𝕏
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Fixed the stale body comment in (Written by Claude Opus 5.5) |
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
Changes since the prior pullfrog review (88f1371), verified at 94dd3acc:
- Comment fix:
Consumer::request_broadcast's body comment (rs/moq-net/src/model/origin.rs:4773-4776) now says the watcher "serves that one source until it ends or another route becomes best, when the front bounces the path and ends", replacing the sentence that still promised the re-splice. It matches the public doc four lines above and the code, and closes the last informational finding from the previous round.
The commit is comments only (rs/moq-net/src/model/origin.rs +2/-2), so no behavior, wire, or public API surface changed and no tests were rerun.
(Written by DeepSeek V4.1 Flash)
deepseek-v4.1-flash | 𝕏

On
release, a publisher that re-creates a broadcast at the same path, behind the same first hop, gets the new broadcast stitched onto the old one. Viewers stall in "loading". A front treated every route from the same publisher as the same content: when its source closed it resumed through the next route, and while serving it spliced a better route in. A re-created broadcast restarts its group numbers (append_group()counts from 0 onrelease), so the resumed track waits for groups past the old head that never come.mainalready tells such broadcasts apart (#7489's instance and epoch, minted per lite-06 announce), but that change is a breaking wire change and cannot come back. This is therelease-line fix: never stitch two broadcasts.Changes
rs/moq-net/src/model/front.rs:source_closedalways ends the front. Before, a named publisher's front reselected and resumed through another route.selectedends a serving front whenever another route becomes best: a better route, a drained serving route, the serving route renamed to another publisher, or a local newcomer. Before, it requested through the new route and spliced it in. Readers re-request and get a fresh front through the table's choice. A front whose route stops qualifying with nothing to replace it (renamed in place, or retracted) ends the same way.A front that ends this way emits
BouncebeforeEnd. The origin then re-announces the exact path (a retraction, then the route again) to every announce cursor presenting it. The table alone shows those cursors no change, so without the bounce a browser holding the ended broadcast stalls offline, and a downstream relay re-subscribes on the same session and splices the new broadcast in at the old position. A path announced only through a covering prefix is not bounced, since that would restart its other broadcasts too.A peer's track copy that dies after delivering now ends the track with the error instead of re-subscribing the same source at the old position. The peer's own front may have ended and a fresh one taken over, so asking again could reach another broadcast. A local broadcast's copy still re-splices, since it is the one broadcast published here.
This matches upstream
mainfor moq-lite-06, where each announce mints its own instance:restart()inlite/publisher.rs), which is this bounce on the wire. It is delivered at once, not held.a_copy_dying_under_a_minted_instance_ends_the_track), while a local broadcast re-splices (recreated_track_*).rs/moq-net/src/model/origin.rs: a request joins a cached front only while the best route is the one the front's source came through, not merely one from the same publisher. Otherwise a request right after a route change joined the front that was about to end.The front never has a second source, so it never reads through a route it doesn't advertise. A version that kept the attached source until it closed was rejected for that reason: the relay would advertise one route while reading through another, which breaks the assumption behind the loop-freedom argument in
origin.rs(fronts waiting on each other in a cycle).Tests (
front.rs,origin.rs):a_returning_publisher_is_never_stitchedis new. A publisher's route returns behind the same first hop before the old front notices it left, and a request gets the new broadcast from its group 0. It fails onrelease("the old subscription waits on the returning publisher: Elapsed").a_better_route_ends_the_fronta_dead_source_ends_the_fronta_local_newcomer_ends_the_fronta_first_hop_update_ends_the_frontdrain_reprice_ends_the_fronta_local_newcomer_ends_the_incumbent_fronta_second_local_source_gets_a_fresh_fronta_hairpin_through_the_plain_front_is_withdrawnan_ended_front_bounces_its_announceis new: whether the source closes under a route that stays or a cheaper route wins, the cursor sees the path retracted, then active. It fails without the bounce.a_covering_route_is_never_bouncedis also new.a_route_renamed_in_place_bounces_its_announceis new: a route updated in place to another first hop shows the cursor only an update, so it fails without the bounce inselected's no-candidate arm.a_second_local_source_gets_a_fresh_frontrequests right after the newcomer attaches. It fails without the route check on joining a front.rejoin.rs(taken_over_quiet_catalog_survives_cache_expiry) andgoaway_cluster.rs(cluster_migrates_on_upstream_goaway,cluster_diamond_goaway_seamless_failover) asserted seamless resumption across a publisher takeover or a relay GOAWAY. They now assert the new contract: the old subscription ends with no group from the new leg, a fresh request reads the new leg, and the path ends up announced, though it may bounce.moq-tokio'sbroadcast_route_migrationis reworked the same way.Impact
doc/bin/relay/cluster.mdsays so.Validation
sh/rs/select.sh checkandsh/rs/select.sh testover this diff (moq-net and every dependant, moq-tokio included): 5195 tests pass, check andcargo fmtclean.origin/releaseand passes here.just checkand the platform lanes; CI covers them.Caused by
Splicing a second route's source into a front came with #2241 (migrate subscriptions transparently across connections) and became the front state machine in #3895. Both assume one publisher hop means one broadcast, which a re-created broadcast breaks.
Models
Size: M
Implementer: Claude Code, claude-opus-5-5, unknown effort
(Written by Claude Opus 5.5)
🤖 Generated with Claude Code
Review
Rewritten by
quest reviewafter each round.Head: 94dd3ac
Base: origin/release at ca3b801
Risk: medium,
the base has no RISK.md to name what is always high or customer-facing, so every change is mediumClassified: medium by default
Reviewer: Codex, gpt-6.1-sol, high effort
Session:
01a1ec58-e1b1-7fe2-bb02-c91d602dddbd, round 4Score: 10/10,
The only incremental change corrects a comment to match the previously reviewed behavior; no findings remain, and builds and tests were not run as instructed.Rounds: 5; first round 2 (high 1, medium 1); findings 2 (high 1, medium 1): fixed 2, refuted 0, open 0
Findings: none
Reviewer: Codex, gpt-6.1-sol, medium effort, shadow
Session:
01a1ec58-e1b2-78f3-8401-3bad1160515bRounds: 2; first round 1 (medium 1); findings 1 (medium 1): fixed 1, refuted 0, open 0