Skip to content

fix(moq-net): the relay's drop poll stops locking the track for a revision it rarely needs - #8979

Merged
kixelated merged 8 commits into
mainfrom
quest/m2/relay-cpu
Nov 17, 2026
Merged

kixelated merged 8 commits into
mainfrom
quest/m2/relay-cpu

Conversation

@kixelated

@kixelated kixelated commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m2/relay-cpu.md.

Problem

The relay memory pairing (moq-dev/moq#8943) measured main's relay at +43% CPU per delivered frame in chat and +35% in fanout against release, the opposite of the 2026-11-05 pairing. The cut waits on a named cause.

Approach

Attributed with just bench-profile instructions chat 8 tokio-workers origin/release by owner (transport, session, origin). Transport is 29k instructions per frame cheaper than release, session 8k cheaper, origin (moq_net::model) 62k dearer. The net growth is lock contention: futex syscalls rose from 2.4k to 19.7k instructions per frame, nearly all on the track's shared TrackState lock. Every subscriber wakes on each group at once, and the publisher's track side (TrackRun::poll_track) takes that lock several times per pass to forward drops and judge misses, three passes per group. Contention grows with workers, so the host matters (8 pinned CPUs on 2026-11-05, 32 here).

  • Fix: resume::Reader::poll_drop read the serving copy's stale revision through the track lock on every poll, even with no received stale run to check (moq-dev/moq#8867). It now reads it only when a received run is held.
  • Bench harness: bench/fanout/{model,session} build again against the timestamp-started group API (moq-dev/moq#8638, moq-dev/moq#8961) and the private Hop; the README says which revisions the model harness builds against.
  • Record: .github/release-perf.md gains a Relay CPU section with the attribution for both workloads, the fix's numbers, and the remaining cost recorded as the price of drop forwarding and fills. The quest file is deleted.

Impact

  • No public API or wire change.

Alternatives

  • A spare chunk of sent packets (the quest's transport lead, moq-dev/moq#6936): cuts 803 to 200 instructions per packet, about 0.4% of a chat frame, but holds 8 KiB on every idle connection (16 MiB across chat's 2,001). Measured and dropped.
  • Cutting the remaining contention now (reuse a judgement at an unchanged track version, or one lock per pass): reworks the drop cursor's judgement and its deadline wakes. Planned as Track lock passes instead.

Decisions

  • Maintainer, from moq-dev/moq#8943: keep this quest in m2, and the release cut waits on it (rejected: moving it to m3; dropping it). This PR completes the quest and deletes its file, so the decision is recorded here rather than in its Plan.
  • Mine: the remaining cost (drop forwarding and fills judged per subscriber under the track lock) is recorded as the cost of those features, per the 2026-10-21 release-perf policy. Whether the cut ships with it or waits on cutting it is the follow-up's open decision below.

Validation

All on fedora (32 CPUs, shared, one-minute load 24 to 66); per-round numbers are in release-perf.md.

  • just bench-profile instructions chat 8 tokio-workers "<release> <main>", 4 alternating rounds: futex instructions per frame release 2.3k, main 8.5k, the fix 4.7k; total 161.2k / 164.8k / 164.0k.
  • just bench-relay <main> quic, 5 rounds: the fix lowers chat relay CPU per frame by a median 11.5% (four of five pairs lower); fanout and video within noise.
  • just bench-relay <release> quic against this exact head, 5 rounds (addressing the automated review on bd49646): chat +8.7% (-5.3 to +25.3; +8.7 to +25.3 in the three pairs at the offered rate), fanout +18.8% (-6.8 to +38.2), video -15.2%. The fix lost 0.3% to 1.1% of chat's groups during startup, release none.
  • just bench-profile instructions fanout 8 tokio-workers "<release> <main>", 3 rounds: release 170.7k, main 219.7k, the fix 218.7k instructions per frame; the growth is origin (+73k) and its lock (futex 12.6k to 33.8k), which the fix does not reach.
  • just bench-fanout <main> with callgrind: single-threaded counts unchanged (session 113.7k to 113.6k per group at 512), which shows the fix's effect is contention, not work; the call counts give the passes per group.
  • cargo nextest run -p moq-net --all-features (4,051 passed), just check, just ci.

Caused by

  • moq-dev/moq#8867 - read the stale revision through the shared track lock on every drop poll, even with no received run to check (found by check)

Follow-ups

  • Track lock passes (m3, L): a live subscriber's track passes stop contending for the track lock, closing the remaining chat (+8.7%) and fanout (+18.8%) gap. Kept; decided 2026-11-17 (maintainer, comment): the cut does not wait on it, it follows the cut in m3 (rejected: moving it to m2). Recorded in its Plan by 0b40501.
  • Chat startup loss (m3, M): explain the groups main's relay loses while chat's viewers join. Kept (maintainer, 2026-11-17). No open decisions.

Models

Size: M
Implementer: Claude Code, claude-opus-5-5, unknown effort

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

Review

Rewritten by quest review after each round.

  • Head: 9edfed2

  • Base: origin/main at 93af504

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1e761-9e07-7490-a016-a8b1b513c3bf, round 4

  • Score: 9/10, The incremental change records the release decision, and the merge introduces no actionable findings in the branch’s own changes; builds and tests were not run, as instructed.

  • Rounds: 4; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

kixelated and others added 3 commits November 16, 2026 15:40
Quest-Claim: 5361f58f45b58427
…received run needs a check

The fan-out harnesses also build against the start-addressed group API again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ntion

Records the attribution and the fix's measurements in release-perf.md, finishes
the relay CPU quest, and plans the track-lock passes and the chat startup loss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title The relay's chat and fanout CPU growth since release is explained before the cut fix(moq-net): the relay's drop poll stops locking the track for a revision it rarely needs Nov 17, 2026
@kixelated
kixelated marked this pull request as ready for review November 17, 2026 02:44

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: bd496465ab3366233ea2c1a48d6799a08b54777c.

I’d land the lock fix. When there are no queued received stale runs, there’s nothing for the revision to invalidate. Once a run is queued, the next poll still reads the revision before forwarding it. No code blocker found.

I wouldn’t call the release comparison cleared yet. The reported medians still show +13.9% chat and +17.5% fanout CPU per delivered frame versus release (respectively +10.4% and +7.8% pinned). Some chat rounds mixed startup losses and backlog into the measured window, and the measured variant kept the spare chunk this PR drops. Before accepting the residual for the cut, pair the exact candidate with release on the same host, report startup group loss separately, and compare steady-state windows at the same delivery rate. That does not require landing the larger lock rework first.

Static review only; I did not rerun benchmarks or tests.

pullfrog[bot]
pullfrog Bot previously approved these changes Nov 17, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the drop-poll fix in resume::Reader::poll_drop, the two bench/fanout harness updates, and the perf record plus quest bookkeeping.

  • Drop poll skips the track lock when nothing was received: poll_drop now reads stale_revision() only when received is non-empty. While it is empty, received_checked keeps its last value, so the first poll after a run is received may recheck received one extra time. That recheck costs O(received), and the result stays correct. stale_revision is a monotonic counter (track.rs, stale_revision += 1), a run joins received only after stale_held reported it held, and sync still resets received_checked to None when the generation changes. A matching revision therefore still proves the run is held.
  • Bench harness builds again: bench/fanout/{model,session} now call append_group(start), Timestamp::from_millis, group::Producer::close(), and origin::Config::new(NonZeroU64). All four match the current moq-net signatures.
  • Perf record: .github/release-perf.md adds the Relay CPU attribution and the fix's numbers. It also says the release-to-fix runs included the spare-chunk transport change that was not taken (about 0.4% of a chat frame).
  • Quests: deletes quest/m2/relay-cpu.md and adds the m3 follow-ups relay-track-lock.md and chat-startup-loss.md.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The changes update Reader::poll_drop stale-revision checks and adjust the fan-out benchmark code and documentation. They add a relay performance comparison and quest documents on chat startup loss and track-lock contention. They also remove the M2 relay CPU quest document and its index entry.

Merge Risk: Minimal · up to bd496

This change reduces lock contention in the relay's drop polling and updates benchmarks and performance documentation. No public API or wire change is stated. Merge risk is minimal.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the change: the relay's drop poll in moq-net no longer takes the track lock for a revision it rarely needs. The raw summary confirms resume.rs Reader::poll_drop now reads the stale rev…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description is specific and on-topic. It explains the relay CPU regression against release, the cause (shared TrackState lock contention), the fix to resume::Reader::poll_drop, the bench harness u…
Full details: Title check

Explanation

The title names the change: the relay's drop poll in moq-net no longer takes the track lock for a revision it rarely needs. The raw summary confirms resume.rs Reader::poll_drop now reads the stale revision only when received contains stale runs, which matches the title's scope. The fix prefix and wording are specific and accurate.

Full details: Description check

Explanation

The description is specific and on-topic. It explains the relay CPU regression against release, the cause (shared TrackState lock contention), the fix to resume::Reader::poll_drop, the bench harness updates, the release-perf record, and the quest file deletion. It also lists validation runs, alternatives, and follow-ups. This matches the changeset summaries, so the check passes.

Finishing Touches

Simplify code

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e0581485-b810-46ba-b828-2b1c23fa4f94
Commits

Reviewing files that changed from the base of the PR and between ba9c4c6 and bd49646.

Files selected for processing: 9

  • .github/release-perf.md
  • bench/README.md
  • bench/fanout/model/src/main.rs
  • bench/fanout/session/src/main.rs
  • quest/m2/README.md
  • quest/m2/relay-cpu.md
  • quest/m3/chat-startup-loss.md
  • quest/m3/relay-track-lock.md
  • rs/moq-net/src/model/resume.rs
Files with no reviewable changes: 1

  • quest/m2/relay-cpu.md

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 1 remain after this review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pullfrog[bot]
pullfrog Bot previously approved these changes Nov 17, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Since the last review at bd49646, the branch merged origin/main and updated the fan-out bench harnesses to match main's group API.

  • Timestamp-started groups in the bench harnesses: bench/fanout/{model,session} now call track.append_group(at) and write_frame(at, …) with at = Timestamp::from_millis(start), then group.finish(). These calls match track::Producer::append_group(&self, start: Timestamp), Timestamp::from_millis(u64) -> Result, and group::Producer::finish(self) on the current head. Both harnesses create their track with Timescale::MILLI, so the rescale in append_group can't fail.
  • README revision floor: bench/README.md now says the model harness builds against revisions since #8961, and the example command compares against origin/main.
  • Quest index: the merge removed the relay-cpu link from quest/m2/README.md, since this PR deletes that quest file.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

…butes fanout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Addressed the automated review on bd49646: paired this exact head (no spare chunk) with release over 5 rounds of just bench-relay <release> quic. Median relay CPU per frame: chat +8.7% (+8.7 to +25.3 in the three pairs at the offered 20,000 frames/s), fanout +18.8%, video -15.2%. Startup loss is now reported apart (0.3% to 1.1% of chat's groups on the fix, none on release) and owned by the chat startup loss follow-up. fanout is attributed too: the same track-lock work, which this fix does not reach. Whether the cut ships with the remaining gap is left as the open decision on the Track lock passes follow-up, recommended no.

(Written by Claude Opus 5.5)

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 499791330c07cb93cb9eccf3f50cea4e19f0ae45.

The new same-host pairing addresses the measurement request: it compares the fix without the spare chunk and separates startup loss. At the offered delivery rate, the three chat pairs still cost +8.7%, +22.1%, and +25.3% CPU/frame; fanout's five-pair median is +18.8%. Video improves by 15.2% at the median. Chat’s startup loss is reduced but not gone (0.3–1.1% of groups versus none on release).

I’d land the lock fix. Treat the remaining CPU cost and the startup delivery loss as separate release decisions, not as cleared benchmarks. In particular, don’t dismiss the startup loss on the strength of the steady-state runs. I reviewed the recorded results; I did not rerun them.

pullfrog[bot]
pullfrog Bot previously approved these changes Nov 17, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Since the last review at d762859, the branch only updated docs.

  • Fanout attribution in the perf record: .github/release-perf.md now splits the owner table and futex numbers by chat and fanout, and says the contention is work under the track lock as well as waiting for it.
  • Exact-head release pairing: the release-to-fix medians come from 5 rounds, and the text notes which chat rounds ran below the offered rate and the startup group loss.
  • Follow-up quest goal: quest/m3/relay-track-lock.md now covers fanout as well as chat, with matching numbers in its decision text.

No code changed, so the earlier poll_drop review still holds.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 17, 2026
@kixelated

Copy link
Copy Markdown
Collaborator Author

Maintainer decision (2026-11-16, /quest-run): merge. Keep both follow-ups. For quest/m3/relay-track-lock.md: ✅ the cut does not wait on it, and it follows the cut in m3 (rejected: moving it to m2). Keep quest/m3/chat-startup-loss.md.

(Written by Opus 5.5)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@moq-bot
moq-bot Bot removed this pull request from the merge queue due to a manual request Nov 17, 2026
@moq-bot

moq-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

quest land watch: ended this pull request's pick, taking it out of the queue, since its head 4997913 is held by decision by kixelated: Maintainer decision (2026-11-16, /quest-run): merge. Keep both follow-ups. For quest/m3/relay-track-lock.md: ✅ the cut d (#8979 (comment)). Push a fix, or answer each by a thread reply or a comment linking its review or decision, then land it again.

(written by quest land watch)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Answered the maintainer decision in 0b40501: quest/m3/relay-track-lock.md records in its Plan that the cut does not wait on it and it follows the cut in m3 (moving it to m2 rejected), and drops its Maintainer: Waiting item. Both follow-ups are kept; ## Follow-ups says so. Head 9edfed2 adds that commit and a conflict-free merge of main.

(Written by Claude Opus 5.5)

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Since the last review at 4997913, the branch updated one quest file and merged origin/main. The merge changed nothing else in this PR's files.

  • Release decision recorded: quest/m3/relay-track-lock.md now records the maintainer's 2026-11-17 decision that the release cut does not wait on this work, which follows the cut in m3. The "Waiting" section that asked for that decision is removed.

No code changed, so the earlier approval of the poll_drop fix still holds.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit 8c9e135 Nov 17, 2026
10 checks passed
@kixelated
kixelated deleted the quest/m2/relay-cpu branch November 17, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant