Skip to content

fix(net): a closed endpoint's sessions end as closed, never unroutable - #8900

Merged
kixelated merged 3 commits into
mainfrom
quest/m0/test-flakes-3/py-socket-abort
Nov 16, 2026
Merged

kixelated merged 3 commits into
mainfrom
quest/m0/test-flakes-3/py-socket-abort

Conversation

@kixelated

@kixelated kixelated commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

py/moq-rs/tests/test_socket.py::test_abort_and_endpoint_close_are_errors still failed on main about 1 run in 6 (10 of 60 alone): after a.close(), the peer's read raised Protocol: unroutable instead of Closed. moq-dev/moq#8825 did not fix it.

The cause is in the origin's front (rs/moq-net/src/model/front.rs), not the socket endpoint. Closing an endpoint resets its session tracks with Stream(Session(Cancel)) and then closes its broadcasts. The front's driver reads the copy's end and, separately, whether its source is closing (is_closing()). When the driver read the end after the close had begun, it took the undelivered copy's end as a closing source's answer, "not a verdict", kept nothing, and the front then ended with no replacement route, so the track ended Unroutable. When it read the end first, the track ended with the reset. The cross-thread timing of that one read decided the error. Any local publisher that aborts a track before writing to it and then closes its broadcast could hit this, not only sockets.

Approach

Front::refuse now keeps a closing source's answer as the track's lost error, unless an earlier loss is already recorded. It is still not a refusal: no source is marked refused, and a replacement is asked fresh. But if the front ends first, or the replacement refuses, the track ends with that answer, not with Unroutable or the refusal. That is what moq-dev/moq#8373 already does for a copy that dies mid-serve.

The reset reaches the front in one of two ways. If the copy was spliced in, it arrives as the copy's end (TrackEnded). If the copy was reset before its metadata was read, the driver reports it as the query's answer (TrackInfo(Err)). Both go through refuse's closing branch, so both are covered.

The regression test a_closing_source_reset_reports_the_loss drives the state machine through both orderings, with no timing involved. It checks each against a front ending with no replacement and against a replacement that refuses. Without the fix it fails, reporting the replacement's NotFound.

This deletes the quest and its link in quest/m2/sockets/remove.md.

Decisions

  • Fixed in moq-net instead of skipping the test. The quest's Plan (2026-11-16) said not to fix the endpoint, but to skip the test until socket removal deletes it. That assumed the fault was in the endpoint. It is in the front, which stays after sockets leave, and the session brief asked for a fix at the source. Recommended: keep the fix. The alternative is reverting to a skip owned by /quest/m2/sockets/remove.md, which would leave the front's ordering bug in place for every local publisher.

Impact

  • No public API or wire change. Behavior change: when a source is closing, its answer for a track is kept, whether that is an undelivered copy's reset or a refusal of the query. If nothing replaces the source, or a replacement refuses, the track ends with that answer instead of Unroutable or the replacement's refusal.
  • JS has no closing-source rule in its origin, so it has no counterpart change.

Alternatives

  • Read closing when the copy ends, not when the driver gets to it. A copy's end and its source's close are separate channels, so this needs a snapshot across both. That costs more and still gives no ordering for a peer across a session.
  • Treat the closing source's undelivered end as a refusal. That would undo the rule that a source on its way out gives no verdict, and would stop a replacement from being asked.

Validation

  • .tmp/loop.sh ran the test alone, each time in a fresh process, on a --features testing,openh264 build. On the base: 3/30 and 7/30 failed, every failure MoqError.Protocol: unroutable. After each fix commit: 0/100.
  • The new unit test fails with the get_or_insert line removed (it reports NotFound instead of the reset) and passes with it.
  • cargo test -p moq-net --all-features: all pass. The full Python suite (moq-rs/tests, moq-ffi/tests): 128 passed. just check: pass on the head. just ci: pass on the first commit.
  • just test interop --all was not run: nothing changed in wire, moq-ffi, or gateways. verify-failover's spec models neither closing sources nor undelivered refusals, so it was not rerun.

Caused by

  • moq-dev/moq#3895: a closing source's undelivered copy ending left the track nothing to end with, so a front ending after it reported Unroutable (found by check).
  • moq-dev/moq#8373: kept the loss for a copy that delivered, but not for one that delivered nothing (found by check).

Follow-ups

None.

Models

Size: S
Implementer: Claude Code, claude-opus-5-5, medium effort (arm A)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: 8646071

  • Base: origin/main at 8f8eec3

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1e40f-ef84-7322-b9dd-a8dccd4d12c9, round 2

  • Score: 10/10, The head preserves the reset through both event orderings, with focused regression coverage and no new issues found by code inspection.

  • Rounds: 2; first round 1 (medium 1); findings 1 (medium 1): fixed 1, refuted 0, open 0

  • Findings: none

  • The last round's findings, at 466f735:

    • Fixed: medium, rs/moq-net/src/model/front.rs:631: Preserve the reset when the copy dies before metadata is processed; The closing branch of refuse now preserves the error in track.lost, and a_closing_source_reset_reports_the_loss covers both TrackEnded and TrackInfo(Err) with no replacement and a refusing replacement.

kixelated and others added 2 commits November 16, 2026 02:39
…t when its closing source has no replacement

A front treated an undelivered copy's end from a closing source as no verdict
and kept nothing, so when nothing replaced the source the track ended
Unroutable. Whether the driver saw the source closing when it read the copy's
end decided the outcome, so a socket endpoint's close reached its peer as
unroutable about 1 run in 6. Keep the reset as the track's loss, as a copy
lost mid-serve already is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title A closed endpoint ends its streams as Closed, never unroutable fix(net): a closed endpoint's sessions end as closed, never unroutable Nov 16, 2026
The driver reports a copy reset before its metadata was read as the query's
answer, not the copy's end, so the same race reached refuse's closing branch.
Keep every closing source's answer as the track's loss there, unless an earlier
loss stands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 16, 2026 11:29

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the closing-source fix in Front::refuse, its regression test, and the quest cleanup.

  • Closing source's answer kept as the loss: refuse now does track.lost.get_or_insert(refusal.0) before its early return. This means a front ending with no replacement (origin.rs:5223 reads front.lost()) or a replacement that refuses (refuse takes lost into refusal) reports the reset rather than Unroutable or the replacement's error. The source is still not marked refused, and a replacement is still queried fresh.
  • Both delivery orderings covered: the reset reaches refuse through track_ended (undelivered Err) and through track_info (Err before metadata). The new a_closing_source_reset_reports_the_loss test drives both orderings, each with no replacement and with a replacement that refuses.
  • Stale lost is still bounded: a successful splice clears it (track_info Ok at front.rs:545), parking an Idle or Querying track clears it (front.rs:790), and a later mid-serve loss overwrites it. Keeping it in the closing branch therefore does not outlive the track's next real answer.
  • Quest cleanup: removes quest/m0/test-flakes-3/py-socket-abort.md and its link in quest/m2/sockets/remove.md.

One trade-off is documented in the PR's Decisions and Impact and needs no action. If a replacement refuses, it is reported with the closing source's error, and its own refusal and Retry Interval are dropped. This matches what #8373 already does for the mid-serve case.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The front model now retains a closing-source refusal as track loss when no earlier loss exists, without recording the source as having refused the track. Tests cover reset errors received through TrackEnded and TrackInfo, replacement refusal, and front termination without a replacement. The quest file about the Python socket test failure and its conditional checklist instruction are removed.

Merge Risk: Minimal · up to 86460

This change makes a closing endpoint's sessions end with the original close error instead of an unroutable error. The change is localized, has regression tests for both event orderings, and has no public API or wire change. No merge-blocking risk was found.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: a closed endpoint's sessions now end as closed rather than unroutable. The changeset preserves a closing source's reset error in the front's track.lost, which fixes th…
Description check Passed The description states the problem (a flaky socket test failing with Protocol: unroutable instead of Closed), names the root cause in rs/moq-net/src/model/front.rs, and describes the Front::refuse cha…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: a closed endpoint's sessions now end as closed rather than unroutable. The changeset preserves a closing source's reset error in the front's track.lost, which fixes that outcome. The title is specific and matches the code change in rs/moq-net/src/model/front.rs.

Full details: Description check

Explanation

The description states the problem (a flaky socket test failing with Protocol: unroutable instead of Closed), names the root cause in rs/moq-net/src/model/front.rs, and describes the Front::refuse change, the regression test a_closing_source_reset_reports_the_loss, the quest deletion, and validation. It relates directly to the changeset.

Finishing Touches

Simplify code

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8a9be465-3582-4367-bce4-3f7667d086ce
Commits

Reviewing files that changed from the base of the PR and between 7101e7a and 8646071.

Files selected for processing: 3

  • quest/m0/test-flakes-3/py-socket-abort.md
  • quest/m2/sockets/remove.md
  • rs/moq-net/src/model/front.rs
Files with no reviewable changes: 2

  • quest/m0/test-flakes-3/py-socket-abort.md
  • quest/m2/sockets/remove.md

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review.

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit 8646071c379d6e8842a381bce266e445e151aca7.

No actionable defect found in this static review. I traced both the undelivered TrackEnded error and pre-metadata TrackInfo(Err) through Front::refuse, then checked the driver's Action::End handling. Retaining the first loss in the closing-source branch preserves the reset without marking that source refused or preventing a replacement query. A refusing replacement uses the retained loss; a successful splice clears it, and parking an idle/querying track abandons it. The change reuses the existing recovery/error-provenance mechanism rather than adding endpoint-specific timing logic.

The new deterministic state-machine test covers both entry paths, each with no replacement and with a replacement that refuses. It checks the stored loss in the no-replacement case; the driver separately uses that loss when ending the track.

Limits: GitHub source/diff inspection only. I did not run the Rust/Python suites, repeated-process flake reproduction, interop, or the failover model, and have not independently reproduced the reported passing runs. This is a COMMENT, not an approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant