Skip to content

fix(moq-net): a finished track a read holds outlives its linger, and the failover and resume models mirror the code - #9048

Merged
moq-bot[bot] merged 3 commits into
mainfrom
quest/m1/precut-fixes/failover-models-sync
Nov 18, 2026
Merged

moq-bot[bot] merged 3 commits into
mainfrom
quest/m1/precut-fixes/failover-models-sync

Conversation

@kixelated

@kixelated kixelated commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Quest failover-models-sync (from the 2026-11-17 pre-cut audit) found three problems:

  • verify-failover and verify-resume still named their old sync commits, though about 60 commits had touched the covered code since.
  • FailoverChain.tla did not model the linger.
  • A relay forgot a finished track after its 30 s linger, even while a fetch or group read still held it. A closed track counted as unread (TrackWeak::is_wanted), and TrackWeak::abort_unwanted reports a closed track as gone. Once the track was forgotten, its resume producer concluded, so a held read whose upstream route then died failed instead of moving to another route.

Approach

  • Reproduced first, in the model. I modelled the pre-fix linger as Forget plus a guided search over Mesh.cfg (a state constraint that prunes only states off the path). It broke NoSkipWhileHeld in 32 steps:
    1. E fails over to B, which serves through A.
    2. E fetches group 0 from B, and B fetches it from A.
    3. The track ends everywhere, and B's linger runs out while E's fetch still holds the track.
    4. A dies, and B can no longer move the fetch to O.
    5. The client's group fails, although E still has a route.
  • Reproduced in code. The new test a_held_finished_track_outlives_its_linger (model/origin.rs) fails on main: no replacement route is asked for the held finished track after the linger.
  • Fix (model/origin.rs):
    • A finished track whose reads still move with the front (resume::Producer::moves) counts as read while anything holds it (TrackIo::read).
    • Its demand edge waits on the holder count even after the close.
    • The Forget arm keeps a track that is still read.
    • Failed and concluded tracks keep the old behavior: unread once closed. a_track_ending_while_read_lets_its_front_retire pins that.
  • kio: ProducerWeak::poll_held_changed is poll_wanted_changed that keeps counting consumers after the channel closes.
  • FailoverChain.tla (beyond the linger, two drifts that fix(moq-net): a relay whose track ended moves a fetch off a dying route #8896 introduced, found by auditing every commit since the old sync commit):
    • Forget lets a relay's finished track go only once no downstream copy reads it (Reads).
    • A missing group's fetch goes to the copy the track ended on until that copy fails (Answering, Route::answering).
    • That copy is never let go while the track is finished (LetGo, Phase::Finished).
    • Demand counts only open downstream subscriptions, and Subscribe waits for a reader.
  • ResumeReader.tla: no behavioral drift since 103052e. Three comments now name the current code (Phase::Finished instead of the removed Route::finished, Copy::bound, latest_key).
  • Skills: both sync commits move forward, the Coverage tables gain the new symbols, and the state counts are updated.
  • The quest file is deleted.

Decisions

Each can be undone in review:

  1. Only a finished track that still moves is held by its readers. A failed or concluded track owes its holders nothing, and an existing test deliberately lets it retire while held.
  2. An app that keeps a finished track's handle forever keeps that track's front alive until it drops the handle. The front cannot tell a fetch in flight from an idle handle. Per rs/AGENTS.md, a parked clone keeping something alive is the holder's bug.
  3. The guided search config is not shipped. It needs a Guide operator in the model. Mesh.cfg stays sampled, as fix(moq-net): a relay whose track ended moves a fetch off a dying route #8896 settled.
  4. Sync commits:
    • failover names 61f8956839 (newest main commit touching its files) plus this PR, which mirrors its own change.
    • resume names 74e05ddc70. This PR's track.rs change is outside the resume model.

Validation

  • FailoverChain.cfg: pass, 2,588,648 distinct states (2.0M before), 55 s.
  • Leave.cfg: pass, 2,831,737 distinct states, 66 s.
  • Mesh.cfg, 200k-trace sample at depth 100: pass, 80M states, 7 min.
  • Guided Mesh.cfg:
    • With the pre-fix linger: NoSkipWhileHeld violated at depth 32.
    • With the fix: exhausted 8,380,829 distinct states, depth 66, pass.
  • ResumeReader.cfg: pass, 14,334,670 distinct states. ThreeRoutes.cfg: pass, 7,053,230.
  • a_held_finished_track_outlives_its_linger fails on main (the front rerouted the held track: Elapsed) and passes here.
  • held_changed_outlives_the_close (kio).
  • just rs test -p moq-net -p kio --all-features: 4304 passed.
  • just check: pass.
  • Not reproduced end to end over the wire. In the failover_chain.rs relay chain, a relay's track parks rather than finishes, because downstream readers leave before the copy closes. The origin test and the model cover the path instead.

Impact

  • Public API: kio gains ProducerWeak::poll_held_changed (additive). moq-net: none; everything changed is crate-private.
  • Wire: none.

Caused by

  • moq-dev/moq#8896 - added the finished-track reroute for the reads still holding a track, but left Forget dropping a closed track whatever held it (found by the pre-cut audit).

Follow-ups

None.

Models

Size: M
Implementer: Claude Code, claude-opus-5-5 (arm B)

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

Review

Rewritten by quest review after each round.

  • Head: 19af1ad

  • Base: origin/main at d8ec56d

  • Risk: high, The branch adds a public kio polling API that downstream libraries can depend on, so reverting after adoption could require coordinated changes in those libraries.

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1eca5-53b2-7601-ad5d-5810cb20b352, round 2

  • Score: 8/10, No actionable defects were found in the incremental changes or merge resolution, while builds, tests, and model checks remain unverified as instructed.

  • Raised the risk to high: The branch adds a public kio polling API that downstream libraries can depend on, so reverting after adoption could require coordinated changes in those libraries.

  • Rounds: 2; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

kixelated and others added 2 commits November 17, 2026 17:33
…the failover and resume models mirror the code

The front's driver forgot a finished track once its linger ran out even while a
fetch or group read still held it, since a closed track counted as unread and
`abort_unwanted` reports a closed track gone. A held read then could no longer
move off a dying route. A finished track whose reads move with the front is now
read while anything holds it.

FailoverChain.tla models the linger (`Forget`), where a finished track's fetch
goes first (`Route::answering`), and that it keeps the copy it ended on; the
pre-fix linger broke NoSkipWhileHeld in Mesh. ResumeReader.tla's comments name
the current code. Both skills' sync commits move forward.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title The failover and resume models mirror the code, and an ended track a reader holds keeps rerouting fix(moq-net): a finished track a read holds outlives its linger, and the failover and resume models mirror the code Nov 18, 2026
…/failover-models-sync

# Conflicts:
#	.claude/skills/verify-failover/SKILL.md
@kixelated
kixelated marked this pull request as ready for review November 18, 2026 03:39
@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 8 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 668e6104-a1e5-4a8c-a3fe-fc1c5f18b100
Commits

Reviewing files that changed from the base of the PR and between d8ec56d and 19af1ad.

Files selected for processing: 12

  • .claude/skills/verify-failover/SKILL.md
  • .claude/skills/verify-resume/SKILL.md
  • quest/m1/precut-fixes/failover-models-sync.md
  • rs/kio/src/consumer.rs
  • rs/kio/src/tests.rs
  • rs/kio/src/weak.rs
  • rs/moq-net/src/model/front.rs
  • rs/moq-net/src/model/origin.rs
  • rs/moq-net/src/model/resume.rs
  • rs/moq-net/src/model/track.rs
  • rs/moq-net/verify/failover/FailoverChain.tla
  • rs/moq-net/verify/resume/ResumeReader.tla
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated

Copy link
Copy Markdown
Collaborator Author

Maintainer pick: /quest-merge.

(Written by Claude Opus 5.5 with high)

@kixelated
kixelated enabled auto-merge November 18, 2026 03:42

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound — the front now decides whether a track is read from the consumer count rather than the open-channel demand level, so a finished track's resume producer survives its linger while a fetch or group read still holds it, and both TLA+ models are moved to match the code they cover.

Reviewed changes

The whole 12-file diff was read end to end, with the driver, resume, kio, and model sides cross-checked against each other.

  • The linger — TrackIo::read treats a finished track whose reads still move as read while any consumer holds it; the demand edge polls the holder count in that case, and Action::Forget keeps the track and re-feeds Wanted so the linger re-arms once the holder leaves. The new a_held_finished_track_outlives_its_linger pins both halves and genuinely fails on main.
  • kio — additive ProducerWeak::poll_held_changed counts consumers past closure, with held_changed_outlives_the_close covering the drain and the wake. The extraction leaves poll_wanted_changed unchanged.
  • Failover chain model — Forget/Reads, Answering, the LetGo guard on the copy the track ended on, reader-only Demand, and Subscribe waiting for a reader all line up with Route::answering, Recover::poll_fetch, and Phase::Finished. The added last field is maintained at every record construction.
  • Resume model and skills — ResumeReader.tla is comment-only and the renamed symbols exist; the Coverage rows validate, and the sync commits are ancestors of main with no covered-file commits after them.

ℹ️ Validation is the author's report, not a local run

Cargo and TLC cannot run in this environment, so this review is static: the model and test results above are as reported in the PR. The static checks that did run are listed below.

Technical details
# What was checked without running the suite

## Verified
- `python3 sh/coverage-tables.py` passes with the new Coverage rows.
- `61f8956839` and `74e05ddc70` are ancestors of `origin/main`, and `git log <sync>..origin/main --` over each skill's covered files is empty.
- `Route::answering` picks `Phase::Finished.last` while that copy has not failed and the serving copy otherwise, matching `Answering`.
- `Recover::poll_fetch` fetches from `route.copy` (the `short` arm) while `Fetching` goes through `answering()` (the `missing` arm).
- `Copy::bound` and `Copy::send` do not `reopen()`; `Reader::sync` reads `Phase::Finished.last`; `Action::Reroute` already guards `is_held()`.
- `TrackWeak::abort_unwanted` has exactly one caller, the `Forget` arm this PR fixes.
- No lost wakeup in the new edge: `Action::Conclude` wakes `io.ready` and a close wakes the consumer waiter list, so every moving-to-non-moving phase change still reaches `Unwanted`.
- A relay's `Forget` drops `routes` and the retained copy, so the held-handle pin chain unwinds with the client.
- `js/net` has no linger/`Forget` driver, so no JS counterpart is owed.

## Environment
- `cargo` cannot build here: the pinned `1.98.1` toolchain install fails with a `clippy-preview` conflict, and TLC needs nix.

Pullfrog  | View workflow run | Using deepseek-v4.1-flash | 𝕏

@moq-bot
moq-bot Bot added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 099f215 Nov 18, 2026
8 checks passed
@moq-bot
moq-bot Bot deleted the quest/m1/precut-fixes/failover-models-sync branch November 18, 2026 04:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant