Skip to content

fix(moq-net): never stitch two broadcasts (release) - #9043

Merged
kixelated merged 6 commits into
releasefrom
fix/release-no-resume
Nov 18, 2026
Merged

kixelated merged 6 commits into
releasefrom
fix/release-no-resume

Conversation

@kixelated

@kixelated kixelated commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

On release, a publisher that re-creates a broadcast at the same path, behind the same first hop, gets the new broadcast stitched onto the old one. Viewers stall in "loading". A front treated every route from the same publisher as the same content: when its source closed it resumed through the next route, and while serving it spliced a better route in. A re-created broadcast restarts its group numbers (append_group() counts from 0 on release), so the resumed track waits for groups past the old head that never come.

main already tells such broadcasts apart (#7489's instance and epoch, minted per lite-06 announce), but that change is a breaking wire change and cannot come back. This is the release-line fix: never stitch two broadcasts.

Changes

rs/moq-net/src/model/front.rs:

  • source_closed always ends the front. Before, a named publisher's front reselected and resumed through another route.

  • selected ends a serving front whenever another route becomes best: a better route, a drained serving route, the serving route renamed to another publisher, or a local newcomer. Before, it requested through the new route and spliced it in. Readers re-request and get a fresh front through the table's choice. A front whose route stops qualifying with nothing to replace it (renamed in place, or retracted) ends the same way.

  • A front that ends this way emits Bounce before End. The origin then re-announces the exact path (a retraction, then the route again) to every announce cursor presenting it. The table alone shows those cursors no change, so without the bounce a browser holding the ended broadcast stalls offline, and a downstream relay re-subscribes on the same session and splices the new broadcast in at the old position. A path announced only through a covering prefix is not bounced, since that would restart its other broadcasts too.

  • A peer's track copy that dies after delivering now ends the track with the error instead of re-subscribing the same source at the old position. The peer's own front may have ended and a fresh one taken over, so asking again could reach another broadcast. A local broadcast's copy still re-splices, since it is the one broadcast published here.

This matches upstream main for moq-lite-06, where each announce mints its own instance:

  • Another instance winning a path makes the lite publisher send an end and a fresh start (restart() in lite/publisher.rs), which is this bounce on the wire. It is delivered at once, not held.
  • A copy dying under a minted instance ends its track (a_copy_dying_under_a_minted_instance_ends_the_track), while a local broadcast re-splices (recreated_track_*).

rs/moq-net/src/model/origin.rs: a request joins a cached front only while the best route is the one the front's source came through, not merely one from the same publisher. Otherwise a request right after a route change joined the front that was about to end.

The front never has a second source, so it never reads through a route it doesn't advertise. A version that kept the attached source until it closed was rejected for that reason: the relay would advertise one route while reading through another, which breaks the assumption behind the loop-freedom argument in origin.rs (fronts waiting on each other in a cycle).

Tests (front.rs, origin.rs):

  • a_returning_publisher_is_never_stitched is new. A publisher's route returns behind the same first hop before the old front notices it left, and a request gets the new broadcast from its group 0. It fails on release ("the old subscription waits on the returning publisher: Elapsed").
  • Tests that asserted resumption or a splice now assert that the front ends and a new request gets a fresh front:
    • a_better_route_ends_the_front
    • a_dead_source_ends_the_front
    • a_local_newcomer_ends_the_front
    • a_first_hop_update_ends_the_front
    • drain_reprice_ends_the_front
    • a_local_newcomer_ends_the_incumbent_front
    • a_second_local_source_gets_a_fresh_front
    • a_hairpin_through_the_plain_front_is_withdrawn
  • an_ended_front_bounces_its_announce is new: whether the source closes under a route that stays or a cheaper route wins, the cursor sees the path retracted, then active. It fails without the bounce. a_covering_route_is_never_bounced is also new.
  • a_route_renamed_in_place_bounces_its_announce is new: a route updated in place to another first hop shows the cursor only an update, so it fails without the bounce in selected's no-candidate arm.
  • a_second_local_source_gets_a_fresh_front requests right after the newcomer attaches. It fails without the route check on joining a front.
  • rejoin.rs (taken_over_quiet_catalog_survives_cache_expiry) and goaway_cluster.rs (cluster_migrates_on_upstream_goaway, cluster_diamond_goaway_seamless_failover) asserted seamless resumption across a publisher takeover or a relay GOAWAY. They now assert the new contract: the old subscription ends with no group from the new leg, a fresh request reads the new leg, and the path ends up announced, though it may bounce. moq-tokio's broadcast_route_migration is reworked the same way.
  • Tests that only exercised a second source's splice are deleted: the takeover and warm-head splicing tests, and the refusals of a successor or replacement copy.

Impact

  • Wire: none.
  • Public API: none. Only crate-private front logic changes.
  • Behaviour: failover across relays is no longer seamless. When the serving route drains (a relay GOAWAY included), dies, or is beaten by a better one, the path bounces. Viewers see the broadcast go offline and come back, and they re-subscribe. In-flight subscriptions drain the old copy until it ends, then end. Before, they were spliced onto the new route, which for a re-created broadcast was wrong. doc/bin/relay/cluster.md says so.
  • Announce churn: a bounce reaches every cursor presenting the exact path, including peer relays, once per front that ends while still routed.

Validation

  • sh/rs/select.sh check and sh/rs/select.sh test over this diff (moq-net and every dependant, moq-tokio included): 5195 tests pass, check and cargo fmt clean.
  • The new test fails on origin/release and passes here.
  • Not run here: the full just check and the platform lanes; CI covers them.

Caused by

Splicing a second route's source into a front came with #2241 (migrate subscriptions transparently across connections) and became the front state machine in #3895. Both assume one publisher hop means one broadcast, which a re-created broadcast breaks.

Models

Size: M
Implementer: Claude Code, claude-opus-5-5, unknown effort

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: 94dd3ac

  • Base: origin/release at ca3b801

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1ec58-e1b1-7fe2-bb02-c91d602dddbd, round 4

  • Score: 10/10, The only incremental change corrects a comment to match the previously reviewed behavior; no findings remain, and builds and tests were not run as instructed.

  • Rounds: 5; first round 2 (high 1, medium 1); findings 2 (high 1, medium 1): fixed 2, refuted 0, open 0

  • Findings: none

  • Reviewer: Codex, gpt-6.1-sol, medium effort, shadow

  • Session: 01a1ec58-e1b2-78f3-8401-3bad1160515b

  • Rounds: 2; first round 1 (medium 1); findings 1 (medium 1): fixed 1, refuted 0, open 0

kixelated and others added 3 commits November 17, 2026 17:09
A front resumed a broadcast through another route to the same publisher
when its source closed, and spliced a better route in while serving. A
publisher that re-creates a broadcast at the same path restarts its group
numbers, so the relay stitched the new broadcast onto the old one's
sequence and viewers waited on groups that never came.

A front now ends when its source closes or when another route becomes
best; readers re-request and get a fresh front. Nothing splices a second
source into a front, and a relay always reads through the route it
advertises.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A front never changes routes now; it ends. A request arriving after
another route became best, before the driver ended the front, joined the
front about to end. Admission now checks the route the front's source
came through, not just its publisher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A front that ends while a route still serves its path changed nothing
the announce cursors could see, so a browser or downstream relay holding
the ended broadcast never requested a fresh one: it stalled offline, or a
downstream relay re-spliced the new broadcast at the old position. The
front now emits Bounce before End when it served, and the origin
re-announces the exact path (a retraction, then the route again) to every
cursor presenting it. Covering-prefix routes are left alone.

Failover across relays is no longer seamless: the GOAWAY cluster tests
now assert the old subscription ends and a fresh request reads the new
leg, and the cluster doc says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 18, 2026 01:52
As main does for a lite-06 (minted) instance: asking the peer again could
reach another broadcast under the same path, since the peer's front may
have ended and a fresh one taken over, so the track ends with the error
and a returning reader is not re-spliced. A local broadcast's copy still
re-splices, as on main, since it is the one broadcast published here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated enabled auto-merge November 18, 2026 02:02

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

Two findings verified at this head: a serving front that ends with best == None (a route renamed in place) still leaves its consumers stalled, and broadcast_route_migration in moq-tokio panics on the new contract, so just check and CI fail.

Approach: sound for a release-line fix — a front serves one source for its whole life, so a re-created broadcast can never be stitched onto an old one, at the documented cost of a visible restart on every failover.

Reviewed changes

  • One source per front: Front::selected ends a serving front on any route change and source_closed always ends it, so nothing is re-spliced.
  • Announce bounce: an ending front emits Action::Bounce, and OriginState::bounce re-announces the exact path (retraction, then the route) to each cursor presenting it there, so consumers re-request; covering prefixes are deliberately left alone.
  • Join gate: RemoteFront::pin now carries the route the front requested through, and a request joins only while that route is still the best.
  • Peer copy end: a peer's track copy dying after delivering now aborts the track instead of re-querying the same source; a local copy still re-splices.
  • Tests: front/origin tests rewritten to the restart contract, new returning-publisher and bounce tests, plus rejoin.rs and goaway_cluster.rs updated.

⚠️ moq-tokio's route-migration test still asserts the seam this PR removes

broadcast_route_migration (rs/moq-tokio/tests/broadcast.rs:991) reads the standby's groups off the same subscription after the preferred session dies, and asserts the path never retracts. At this head it panics with track closed at broadcast.rs:970, and sh/rs/select.sh selects moq-tokio as a dependant of moq-net, so just check and CI run it. Rework it the way goaway_cluster.rs was reworked.

Technical details
# moq-tokio route-migration test asserts the removed seam

## Affected sites
- `rs/moq-tokio/tests/broadcast.rs:991` `broadcast_route_migration` — line 1117 `read_payloads(&mut sub, 2)` expects `["b2","b3"]` on the subscription that served `a1`; lines 1131-1133 assert every later announce is active.
- Failure at this head: `cargo test -p moq-tokio --test broadcast -- broadcast_route_migration` -> panic `track closed` at `rs/moq-tokio/tests/broadcast.rs:970` (`recv_group` returned `Ok(None)`).
- `rs/moq-tokio/tests/broadcast.rs:1406` `route_reannounce_test` is unaffected: the first hop stays `0x4444` and the route updates in place, so the front keeps serving; `broadcast_route_reannounce*` pass at this head.

## Required outcome
- The test asserts the new contract: the old subscription ends without any standby group, the path retracts and re-announces, and a fresh `request_broadcast` reads `b2`, `b3`, and `b4`.
- Otherwise the release branch ships a test that asserts the opposite of the code, and the PR gate stays red.

## Open questions for the human
- Whether `moq-tokio`'s cross-route migration is now a documented non-goal (drop the test) or should get a drill for the restart contract.

ℹ️ Public rustdoc still promises the seamless re-splice

Consumer::request_broadcast (rs/moq-net/src/model/origin.rs:4696-4701) still says the front "re-splices through the best route sharing its first hop at a group boundary, invisibly to subscribers", which is exactly what this PR removes. Same for OriginState::fronts (origin.rs:3666-3670), the front.rs module doc (lines 3-5), and Identity::Local/Publisher (front.rs:142-154, "safe to resume through").

Technical details
# Docs describing the removed re-splice

## Affected sites
- `rs/moq-net/src/model/origin.rs:4696-4701` — public `Consumer::request_broadcast` doc, "re-splices ... invisibly to subscribers".
- `rs/moq-net/src/model/origin.rs:3666-3670` — `OriginState::fronts` field doc, "re-splices it through routes sharing its first hop".
- `rs/moq-net/src/model/front.rs:3-5` — module doc, "re-splicing when the source is replaced".
- `rs/moq-net/src/model/front.rs:142-154` — `Identity::Local` ("a newer local publisher takes over while the incumbent is live") and `Identity::Publisher` ("safe to resume through").

## Required outcome
- Each describes one source per front and that any route change ends it and bounces the path, matching `doc/bin/relay/cluster.md` and the code.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using deepseek-v4.1-flash | 𝕏

Comment thread rs/moq-net/src/model/front.rs Resolved
@kixelated
kixelated disabled auto-merge November 18, 2026 02:07
… one source

A route renamed in place (its first hop changed, path unchanged) shows the
announce cursors only an update, so a front ending on it left the old
broadcast announced active. The front now bounces in that case too.

The front, origin, and request_broadcast docs still described resuming
through routes sharing a first hop; they now say a front serves one source
and ends with a bounce. moq-tokio's route migration test asserts the new
contract: the old subscription ends and a fresh request reads the new route.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Addressed review 5949783260 in 88f1371:

  • Renamed-in-place stall: fixed; see the inline reply.
  • broadcast_route_migration: rewritten to the restart contract. The old subscription ends without a standby group, a fresh request_broadcast (looped until it is not the old broadcast) reads b2, b3, then b4, and the path ends announced active. On the open question: I kept it as a drill for the restart contract rather than dropping it, since it is the only moq-tokio test that kills a route under a live subscriber.
  • Stale rustdoc: request_broadcast, OriginState::fronts, the front.rs module doc, and Identity::Local/Publisher now say a front serves one source and ends with a bounce.

sh/rs/select.sh test over this diff runs 5195 tests and all pass, moq-tokio included.

(Written by Claude Opus 5.5)

@kixelated
kixelated enabled auto-merge November 18, 2026 02:26

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues. The delta is sound and both of its new tests pass here; one stale internal comment still describes the re-splice this PR removes.

Approach: unchanged, and sound for the release line: one source per front, and this commit closes the last end path that did not bounce.

Reviewed changes

Changes since the prior pullfrog review (83590a3), verified at 88f1371:

  • No-candidate bounce: Front::selected's None arm now pushes Bounce before End whenever the front had a source (self.resolved). A route renamed in place, or an anonymous route gaining a first hop, therefore retracts and re-announces its path instead of leaving consumers on an ended broadcast whose path stays active. A cursor that presents only a covering prefix still sees nothing, since TableCursor::bounce requires current.prefix == path.
  • Doc pass: the front.rs module doc, Identity::Local and Identity::Publisher, OriginState::fronts, and the public Consumer::request_broadcast doc now describe one source per front and the bounce, replacing the re-splice promises.
  • Regression test: a_route_renamed_in_place_bounces_its_announce covers the arm (it fails with the guard removed, which I checked), and the two front tests that end through it now expect [Bounce, End].
  • moq-tokio test reworked: broadcast_route_migration now asserts that the old subscription ends without any standby group, that a fresh request reads b2, b3, and b4, and that the path ends announced.

I ran the moq-net front and origin tests for this arm plus cargo test -p moq-tokio --test broadcast -- broadcast_route_migration --exact at this head; all pass.

ℹ️ One comment still promises the re-splice this PR removes

Consumer::request_broadcast's body comment (rs/moq-net/src/model/origin.rs:4773-4776) still says the watcher "re-splices the front through routes sharing its first hop for as long as one serves", four lines below the public doc that was just corrected. A reader following it would reintroduce the seam this PR closes.

Technical details
# Stale re-splice comment in `request_broadcast`

## Affected sites
- `rs/moq-net/src/model/origin.rs:4773-4776` — "The watcher materializes the path from the best covering route, resolves the channel, and re-splices the front through routes sharing its first hop for as long as one serves." It predates this PR (`98d19aa9c`) and the doc pass in `88f13713` missed it.

## Required outcome
- The comment states the current contract: the watcher materializes the path from the best covering route and serves that one source until it ends or another route becomes best, when the front bounces the path and ends.

## Suggested approach
- Reuse the sentence in the public doc directly above (`origin.rs:4696-4700`).

(Written by DeepSeek V4.1 Flash)

Pullfrog  | Fix it ➔ | View workflow run | Using deepseek-v4.1-flash | 𝕏

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Fixed the stale body comment in request_broadcast (review 5949998224) in 94dd3ac, reusing the public doc's wording.

(Written by Claude Opus 5.5)

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Changes since the prior pullfrog review (88f1371), verified at 94dd3acc:

  • Comment fix: Consumer::request_broadcast's body comment (rs/moq-net/src/model/origin.rs:4773-4776) now says the watcher "serves that one source until it ends or another route becomes best, when the front bounces the path and ends", replacing the sentence that still promised the re-splice. It matches the public doc four lines above and the code, and closes the last informational finding from the previous round.

The commit is comments only (rs/moq-net/src/model/origin.rs +2/-2), so no behavior, wire, or public API surface changed and no tests were rerun.

(Written by DeepSeek V4.1 Flash)

Pullfrog  | View workflow run | Using deepseek-v4.1-flash | 𝕏

@kixelated
kixelated merged commit 2ab31a1 into release Nov 18, 2026
21 checks passed
@kixelated
kixelated deleted the fix/release-no-resume branch November 18, 2026 03:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant