Repository navigation
fix(net): a lite-07 route layer flushes the claim a widened grant shows - #9039
Conversation
Quest-Claim: 696d92e65ad02f0b
The publish side buffered what a widened grant newly showed after its flush, then rested with it unsent: the grant's wakeup was spent. Also repairs two bench fixtures: moq-lite-07's Announce Stream holds a uni stream grant beside the delivery-credit groups, and an attached token arrives as the request's token since #8923. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-07 route hang Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Decisions from the maintainer:
(Written by Claude Opus 5.5 with high) |
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 3 minutes. View limit details
No actionable comments were generated in the recent review. WalkthroughThe patch changes Lite-07 route-stream polling, adds a regression test, and adjusts session benchmark stream credit. It changes the auth benchmark fixture to use the admission request token for its grant prefix. It also adds a resume-history investigation document and removes the Lite-07 route-hang quest document. Merge Risk:
|
There was a problem hiding this comment.
✅ No new issues found.
Approach: sound. Each hang is fixed at its own cause. The route-layer fix is general: the pass now flushes before it rests whenever the writer holds bytes, rather than special-casing poll_permit. That covers any later path that buffers without a node message behind it.
Reviewed changes
I reviewed all 6 files at 8227645. I also ran the new regression test with the fix and with the route_stream.rs hunk reverted.
Running::poll_sendflush before rest:Answer::poll_permitruns after the pass'spoll_flush. Before this change, ROUTE_STARTs that a widened grant'ssyncbuffered were left unflushed oncepoll_mountswent Pending. The newcontinuecannot spin:Writer::poll_flushclears the buffer when it returns Ready, sois_flushed()then holds.- When it returns Pending, the pass breaks.
- The
Open | Endingguard keepsClosing/Closedout of the loop. routed.releasenow runs only after a successful flush, which matches every other path.
- Regression test
a_widened_grant_flushes_the_claim_it_shows: confirmed locally. It passes with the fix and fails without it ("the claim the grant now shows stayed buffered"). Separate auth links for the publish half and the subscriber isolate the Publish-direction wakeup. session_delivery_creditfixture: lite-07 gets one extra uni stream for the persistent Announce Stream. This matches the existingsupport::held_uni_streamsin the integration tests.relay_auth_refreshdecider: it now readsadmission.request.tokeninstead of thejwt=query, which matches how #8923 delivers in-band tokens.- Quest files: removes the finished quest and splits the lite-06 browser
resume-historyflake into its own quest.
claude-opus-5-5 | 𝕏
|
quest land watch: the merge queue removed this pull request at 8227645 because its The land fixer works on it in the background, and the watcher re-enqueues it at the back of the queue once it is fixed or judged flaky, or hands it back here with what it needs.
… .fetch_update(Ordering::AcqRel, Ordering::Acquire, |live| {
1148 + .try_update(Ordering::AcqRel, Ordering::Acquire, |live| {
|
warning: use of deprecated method `std::sync::atomic::Atomic::<u64>::fetch_update`: renamed to `try_update` for consistency
--> rs/moq-net/src/transport/boxed.rs:164:5
|
164 | .fetch_update(Ordering::AcqRel, Ordering::Acquire, |paused| {
| ^^^^^^^^^^^^
|
help: replace the use of the deprecated method
|
164 - .fetch_update(Ordering::AcqRel, Ordering::Acquire, |paused| {
164 + .try_update(Ordering::AcqRel, Ordering::Acquire, |paused| {
|
warning: `moq-net` (lib) generated 7 warnings (run `cargo fix --lib -p moq-net` to apply 7 suggestions)
Finished `dev` profile [unoptimized] target(s) in 8.10s
mbx[cache]: object cache: 0 hits, 2 misses; 0 B downloaded, 0 B uploaded, 11.7 MiB stored locally
mbx[cache]: 8.65s session; 0s estimated compiler time avoided (summed across compilations)
mbx[cache]: Cargo artifact reuse and CI cache archive transfers are not included above
mbx[savings]: 2d 5h refunded over 4512 builds. no receipt necessary.
##[endgroup]
##[group]test rs
sh/rs/select.sh test "$LIST"
rs: selecting hang libmoq moq-archive moq-audio moq-bench moq-boy moq-cli moq-e2ee moq-ffi moq-gst moq-hls moq-json moq-loc moq-mux moq-net moq-net-tests moq-relay moq-rtmp moq-rtsp moq-srt moq-stats moq-tokio moq-transcode moq-uring moq-video moq-webrtc qmux
KIO_DENY_HOST_TIMERS=1 CARGO_BUILD_WARNINGS=deny RUSTC_WORKSPACE_WRAPPER='/nix/store/jqx7yhf9hnxfi9lgb1bfl608xyjq57jm-rust-minimal-1.98.1/bin/clippy-driver' bash sh/rs/nextest.sh cargo nextest run --locked --tests "$@"
Compiling moq-net v0.3.10 (/var/cache/github-runner-work/moq/moq/rs/moq-net)
Compiling moq-quic v0.1.0 (/var/cache/github-runner-work/moq/moq/rs/moq-quic)
Compiling moq-video v0.1.10 (/var/cache/github-runner-work/moq/moq/rs/moq-video)
Compiling moq-relay v0.17.2 (/var/cache/github-runner-work/moq/moq/rs/moq-relay)
Compiling moq-json v0.5.8 (/var/cache/github-runner-work/moq/moq/rs/moq-json)
Compiling qmux v0.7.0 (/var/cache/github-runner-work/moq/moq/rs/qmux)
Compiling moq-loc v0.2.19 (/var/cache/github-runner-work/moq/moq/rs/moq-loc)
Compiling moq-uring v0.0.12 (/var/cache/github-runner-work/moq/moq/rs/moq-uring)
Compiling moq-archive v0.0.11 (/var/cache/github-runner-work/moq/moq/rs/moq-archive)
Compiling moq-e2ee v0.0.26 (/var/cache/github-runner-work/moq/moq/rs/moq-e2ee)
Compiling hang v0.21.11 (/var/cache/github-runner-work/moq/moq/rs/hang)
Compiling moq-stats v0.2.12 (/var/cache/github-runner-work/moq/moq/rs/moq-stats)
Compiling moq-tokio v0.19.23 (/var/cache/github-runner-work/moq/moq/rs/moq-tokio)
Compiling moq-mux v0.10.11 (/var/cache/github-runner-work/moq/moq/rs/moq-mux)
Compiling moq-net-tests v0.0.0 (/var/cache/github-runner-work/moq/moq/rs/moq-net-tests)
Compiling moq-audio v0.1.10 (/var/cache/github-runner-work/moq/moq/rs/moq-audio)
Compiling moq-hls v0.5.11 (/var/cache/github-runner-work/moq/moq/rs/moq-hls)
Compiling moq-rtmp v0.3.11 (/var/cache/github-runner-work/moq/moq/rs/moq-rtmp)
Compiling moq-srt v0.3.11 (/var/cache/github-runner-work/moq/moq/rs/moq-srt)
Compiling moq-rtsp v0.0.1 (/var/cache/github-runner-work/moq/moq/rs/moq-rtsp)
Compiling moq-webrtc v0.3.12 (/var/cache/github-runner-work/moq/moq/rs/moq-webrtc)
Compiling moq-transcode v0.1.10 (/var/cache/github-runner-work/moq/moq/rs/moq-transcode)
Compiling moq-gst v0.4.12 (/var/cache/github-runner-work/moq/moq/rs/moq-gst)
Compiling libmoq v0.7.0 (/var/cache/github-runner-work/moq/moq/rs/libmoq)
Compiling moq-cli v0.14.2 (/var/cache/github-runner-work/moq/moq/rs/moq-cli)
error[E0063]: missing field `routing` in initializer of `lite::subscriber::route_stream::Publish`
--> rs/moq-net/src/lite/subscriber.rs:9373:4
|
9373 | route_stream::Publish {
| ^^^^^^^^^^^^^^^^^^^^^ missing `routing`
For more information about this error, try `rustc --explain E0063`.
error: could not compile `moq-net` (lib test) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
Timing report saved to /var/cache/github-runner-work/moq/moq/target/cargo-timings/cargo-timing-20261118T063656081Z-efcac1b57bcf5093.html
error: command `/nix/store/aa12cxnxswf80dqkihsb2pmblh1c5v89-cargo-1.98.1-x86_64-unknown-linux-gnu/bin/cargo '--color=auto' test --no-run --message-format json-render-diagnostics --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/hang#0.21.11' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/libmoq#0.7.0' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-archive#0.0.11' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-audio#0.1.10' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-bench#0.1.0' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-boy#0.5.12' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-cli#0.14.2' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-e2ee#0.0.26' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-ffi#0.4.12' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-gst#0.4.12' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-hls#0.5.11' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-json#0.5.8' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-loc#0.2.19' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-mux#0.10.11' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-net#0.3.10' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-net-tests#0.0.0' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-relay#0.17.2' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-rtmp#0.3.11' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-rtsp#0.0.1' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-srt#0.3.11' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-stats#0.2.12' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-tokio#0.19.23' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-transcode#0.1.10' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-uring#0.0.12' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-video#0.1.10' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/moq-webrtc#0.3.12' --package 'path+file:///var/cache/github-runner-work/moq/moq/rs/qmux#0.7.0' --package moq-video --package moq-audio --examples --tests --benches --features 'moq-video/capture moq-audio/capture moq-audio/playback moq-cli/play moq-cli/openh264 moq-boy/openh264 moq-hls/server moq-json/bench moq-mux/fuzz moq-net/async moq-net/fuzz moq-relay/test-support moq-rtmp/fuzz moq-stats/bench moq-uring/_probe moq-uring/fuzz moq-uring/quic moq-video/openh264 qmux/fuzz' --timings --locked` exited with code 101
mbx[cache]: object cache: 2 hits, 52 misses, 32 bypassed; 0 B downloaded, 0 B uploaded, 1.9 GiB stored locally
mbx[cache]: 1m 26s session; 2.20s estimated compiler time avoided (summed across compilations)
mbx[cache]: bypass reasons: 32 unsupported-crate-type
mbx[cache]: Cargo artifact reuse and CI cache archive transfers are not included above
mbx[savings]: 2.5 TiB logical of build debris has left the building.
error: recipe `test` failed on line 342 with exit code 101
error: recipe `test-changed` failed on line 99 with exit code 101
##[endgroup]
##[error]ci: test rs failed
ci: wall and CPU seconds per module
120 169 check rs
87 367 test rs (failed)
3 37 check biome
28 12 check markdown
18 23 check gh
87 80 check carla_bridge
0 0 check drill
2 3 check quest
18 17 check shell
1 0 check large_files
0 0 check text
1 0 check coverage_tables
0 0 check toml
3 2 check nix
0 0 check justfile
208 716 whole run
error: recipe `ci` failed on line 120 with exit code 101
##[error]Process completed with exit code 101.
##[group]Run actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9
with:
name: cargo-timings
path: target/cargo-timings/cargo-timing-*.html
if-no-files-found: ignore
retention-days: 14
compression-level: 6
overwrite: false
include-hidden-files: false
archive: true
env:
BASE: c422cfe8474c9c523a8d7205fcf3ccb217a3949e
CHECK_FAIL_FAST: 1
NEXTEST_TEST_THREADS: 24
MBX_CACHE_LINKS: 1
MBX_REMOTE_URL:
##[endgroup]
With the provided path, there will be 6 files uploaded
Artifact name is valid!
Root directory input is valid!
Uploading artifact: cargo-timings.zip
Beginning upload of artifact content to blob storage
Uploaded bytes 302291
Finished uploading artifact content to blob storage!
SHA256 digest of uploaded artifact is e57076cfbc11254dac4962f5b8c921dd50ae553eb76267d6d30d486577a24187
Finalizing artifact upload
Artifact cargo-timings successfully finalized. Artifact ID 14404128831
Artifact cargo-timings has been successfully uploaded! Final size is 302291 bytes. Artifact ID is 14404128831
Artifact download URL: https://github.com/moq-dev/moq/actions/runs/44252361858/artifacts/14404128831
Post job cleanup.
##[start-action display=Run jdx/mr-boxington-action@88acf2bb50552c7c508cbac13e5f8c576b95b1de;id=__04719235-3171-4854-a94a-adb831182710.mbx]
##[end-action id=__04719235-3171-4854-a94a-adb831182710.mbx;outcome=skipped;conclusion=skipped;duration_ms=0]
Post job cleanup.
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git version
git version 2.56.0
Temporarily overriding HOME='/var/cache/github-runner-work/_temp/66ea62f3-7c0f-4cf8-8b35-b0939d13a23d' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git config --global --add safe.directory /var/cache/github-runner-work/moq/moq
Removing SSH command configuration
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git config --local --name-only --get-regexp core\.sshCommand
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
Removing HTTP extra header
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
Removing includeIf entries pointing to credentials config files
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir:
[command]/nix/store/i2m3w4ly2w6grrmk1j59rb7zp09bssia-git-2.56.0/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
A job completed hook has been configured by the self-hosted runner administrator
##[group]Run '/nix/store/qrn6ds64bzvwbm5nwf245ifz2c6jb9xw-ci-cache-gc.sh/bin/ci-cache-gc.sh'
shell: /nix/store/lfm00fnq66xfgbn099vqckabyl5jllz6-bash-interactive-5.3p20/bin/bash --noprofile --norc -e -o pipefail {0}
##[endgroup]
evicted 134 objects and 46 action results (2.2 GiB logical); 50.0 GiB logical remain
dropped 12 session event streams
evicted 0 objects and 0 action results (0 B logical); 50.0 GiB logical remain
Cleaning up orphan processes
(written by quest land watch) |
|
quest land watch: armed auto-merge on this pull request at 1261e85, so it joins the back of the merge queue once its required checks pass, with the land fixer's fix ( (written by quest land watch) |
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
Since 8227645, the branch merged main. One test line changed to compile against main's new field.
routing: falseina_widened_grant_flushes_the_claim_it_shows:mainaddedroute_stream::Publish::routing. Setting it tofalsekeeps the test's intent:- The publish half stays a non-routing peer, matching the subscriber's
PeerSetupwith no Node ID. truewould hit the whole-grant refusal atroute_stream.rs:235against the startingGrant::default.- It matches the other non-routing fixture at
route_stream.rs:463.
- The publish half stays a non-routing peer, matching the subscriber's
claude-opus-5-5 | 𝕏

Three of the four Nightly benches in the quest now complete. They had three separate causes, not the single shared one the plan assumed. The fourth (the browser smoke) is lite-06, intermittent, and split out as its own quest.
Changes
auth/waitinghang). InRunning::poll_send(rs/moq-net/src/lite/subscriber/route_stream.rs),Answer::poll_permitruns after the pass's flush. When a widened grant newly shows a claim the node had already told, it buffers the Start, finds nothing else queued, and rests with the bytes unflushed. The grant's wakeup was spent on that pass, so nothing ever flushes them. The rest point now loops back to flush while the writer holds bytes. That covers this path and any later one that buffers without a node message behind it.session_delivery_creditfixture (rs/moq-net/benches/session/mod.rs). The viewer was givencredituni streams, plus one only for draft-22's persistent SETUP stream. On lite-07 the relay's Announce Stream is also persistent and took the only grant, so no group stream could ever open. The fixture now counts it too. This was a bench setup bug, not a protocol one.relay_auth_refreshfixture (rs/moq-relay/benches/auth.rs). Since moq-dev/moq#8923, an attached token arrives asrequest.token, not asjwt=in the query. The bench's decider still read the query, so every token got the setup token'sinitialgrant. The bench then updated the wrong lease and waited forever on token0's watch. The decider now reads the token.lite07-route-hang.mdand addsresume-history-control.mdfor the browser smoke. That failure isjs/net/bench/resume-history.tson lite-06, where the control intermittently keeps the serving route reachable. It failed on 11-14 and 11-17 and passed on 11-15 and 11-16, and I could not reproduce it locally in about 36 full sweeps.JS needs no counterpart change:
js/net/src/lite/answer.tshandles a grant change by poking a resync loop that awaitsstream.send, so it never buffers and then rests.Validation
main(cargo bench -p moq-net --features async --bench auth -- auth/waiting/moq-lite-07-wip/1/1was stuck in warm-up for over 15 minutes). Per the plan, I re-ran it on feat(moq-net)!: lite-07 sessions run the new route rule #8987's head (d98a191b) and it still hung. With tracing on feat(moq-net)!: lite-07 sessions run the new route rule #8987, iteration 5 loggedpublisher::routed: announce wait/5/0and the server never decoded it. Debug prints in a unit test then showedpoll_sendresting withis_flushed() == falseright after the permit change.lite::subscriber::tests::a_widened_grant_flushes_the_claim_it_shows: aRouteStreamwhose publish half holds back a hidden claim, then a grant that widens with no other change. It fails without the fix ("the claim the grant now shows stayed buffered") and passes with it. Two integration-level attempts usingauthorizepassed either way, because a Subscribe-direction edit flushed the bytes as a side effect. That is why the test drives the route layer directly with separate auth links.auth/waitingsweep (all 9 cells), the fullsession_delivery_credit/moq-lite-07-wipsweep (9 cells), and the full moq-relayauthbench (relay_auth_refreshandrelay_auth_queue) all complete. On a scratch checkout of feat(moq-net)!: lite-07 sessions run the new route rule #8987's head, the same route-stream hunk applies cleanly andauth/waitingcompletes there too.bun micro.ts --bench resume-historyin Chrome, 3 runs × 4 rounds × 2 variants, all passed (the flake did not reproduce).cargo test -p moq-net --features asyncandjust checkboth pass (11794 tests).Decisions
mainnow; moq-dev/moq#8987 picks the fix up when it next mergesmain(rejected: hold as a draft until feat(moq-net)!: lite-07 sessions run the new route rule #8987 lands). The plan expected feat(moq-net)!: lite-07 sessions run the new route rule #8987's rewrite ofroute_stream.rsto discard amainfix, but the hunk applies cleanly to its head and fixes the bench there too.quest/m0/nightly/resume-history-control.md; this PR lands only the three fixes (rejected: keep it here, blocking the fixes on an unreproduced flake). This overrides the plan's no-split Goal, which assumed the four benches shared a root cause.Follow-ups
Caused by
Running::poll_sendwithAnswer::poll_permitbuffering after the flush and a rest that ignores unflushed bytes (found by check, Nightly bisect).request.tokenwithout updating the relay auth bench's decider (found by check).session_delivery_crediton lite-07: the fixture's uni stream budget predates lite-07 keeping a persistent Announce Stream. I did not confirm which PR first made that stream persistent.Impact
Public API: none. Wire: none (the same bytes go out, just without the lost wakeup).
Models
Size: M
Implementer: Claude, claude-opus-5-5, unknown effort (arm A)
🤖 Generated with Claude Code
(Written by Claude Opus 5.5)
Review
mechanical fix by codex, gpt-6.1-sol, medium effort:
Merged base and initialized routing: false in the non-routing test fixture to preserve its intent; ran nix develop --command just ci base and git diff --check.(head 1261e85)Rewritten by
quest reviewafter each round.Head: 8227645
Base: origin/main at 4bcae73
Risk: medium,
the base has no RISK.md to name what is always high or customer-facing, so every change is mediumClassified: medium by default
Reviewer: Codex, gpt-6.1-sol, high effort
Session:
01a1ed46-00b9-7fc0-bc46-7388997cc62b, round 1Score: 9/10,
No actionable regressions were found; the flush fix and benchmark corrections appear sound from source review, but builds and tests were not run, as instructed.Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0
Findings: none