Skip to content

fix(qmux)!: egress bills only the bytes the link took - #8930

Merged
kixelated merged 2 commits into
mainfrom
fix/qmux-bills-what-the-link-took
Nov 17, 2026
Merged

kixelated merged 2 commits into
mainfrom
fix/qmux-bills-what-the-link-took

Conversation

@kixelated

@kixelated kixelated commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

moq-dev/moq#8845 left qmux's poll_acked at Acked::Unsupported, so coding::Writer bills qmux (WebSocket) egress by the bytes the stream accepted. The decision behind it assumed the two counts "differ only while the link is blocked and holds accepted bytes". They also differ for good when a reset or a lost session discards what the mux still holds: those bytes never leave the relay, but they stay billed. That breaks stats' own contract that egress counts "never what a reset or a lost session discarded".

It shows end to end. A local relay served one broadcast to a moq export client over ws://, behind a TCP proxy counting every byte the relay wrote, including the tail after the client left. One run billed 1,618,638 payload bytes while the relay wrote 1,603,091 bytes to the socket, framing included. The same broadcast over QUIC and WebTransport billed 1,596,309. So about 22 kB, roughly one frame the client's exit left in the mux, was billed and never sent.

Approach

  • transport::poll::Acked gains Sent: the transport handed every byte below the offset to a reliable link, which delivers it unless the link fails. It is not an acknowledgment and says nothing of when the peer got the bytes, so the 2026-10-22 decision holds: a lag or delivery-delay frontier still treats qmux as having no acknowledgments (quest/m3/qos/lag-histogram.md now says so).
  • qmux's SendStream mints moq_quic::send::Offsets and answers poll_acked from its mux, which counts stream data as acknowledged once the transport takes its record. It answers from acked_len() first, so a reset or STOP_SENDING keeps the prefix that went out, as the QUIC adapters do.
  • coding::Writer's Billing bills through a Sent offset as it does a Delivered one.

What the link took but TCP never wrote stays billed if the link dies mid-flush. That is at most one driver batch (64 KiB) per session, the same bound as the session's wire_tx counter and qmux's delivered() before #8845.

Impact

  • moq-net: transport::poll::Acked::Sent is new. Acked is #[non_exhaustive], so this is additive.
  • qmux (breaking): SendStream's poll::SendStream::Offset changes from () to moq_quic::send::Offset, and it implements poll_acked. Migration: code that names the associated type takes the new one from offset(); code generic over SendStream is unaffected.
  • Behavior: qmux egress bytes/frames bill what the link took, not what the stream accepted, so a reset or lost session no longer bills the bytes it discarded. The open-stream debug log still counts only Delivered streams as acknowledged.
  • Wire: none.

Alternatives

  • qmux answers Delivered once the link took the bytes. It is the smallest change, but it is what 2026-10-22 rejected: a future lag histogram would read TCP write times as acknowledgments.
  • Keep Unsupported and bill an unsupported stream only once it finishes. That never over-bills, but it drops the sent prefix of every reset group, such as each group a lagging viewer skips.
  • Keep Unsupported and accept billing discarded bytes. Egress then over-bills after every reset with bytes still queued.

Decisions

Confirmed by the maintainer on 2026-11-17.

  • ✅ A distinct Acked::Sent that billing counts and acknowledgment consumers ignore (rejected: qmux answers Delivered; bill unsupported streams only on finish; keep billing accepted bytes). This revisits the 2026-11-15 choice in quest/m2/api-ack-one-primitive.md ("Rejected: giving the hook qmux's 'the link took it' meaning"), because its premise missed the reset case above. The hook still has one primitive; Sent is a separate answer, not Delivered.

Caused by

  • moq-dev/moq#8845 - kept qmux at Acked::Unsupported, so its egress billed the bytes a reset or lost session discarded (found by check).

Validation

  • just ci passed: every check module, and test rs ran 12,853 tests.
  • New qmux session::tests::bytes_count_as_sent_once_the_link_takes_them, over each qmux version: bytes the peer read answer Sent, bytes a stalled link holds stay pending, and after a reset the sent prefix still answers Sent while the held tail fails. On main it cannot compile, since qmux mints no offsets.
  • coding::writer::tests::egress_bills_acknowledged_payload now also runs with a stream answering Sent. Without the writer's Sent arm it fails (left: (200, 2), expected (100, 1)), billing the taken bytes instead.
  • End to end, with this commit on top of 80ca9787e: the WebSocket scenario from Problem, three runs, each billing at most what the proxy saw the relay write: 1,596,309 ≤ 1,603,041, 1,596,307 ≤ 1,603,036, and 1,618,636 ≤ 1,625,415. The third billed the same extra frame as the failure above, but this time the socket carried it.
  • Local review: Codex gpt-6.1-sol, medium effort, one round, no findings.

Follow-ups

None.

Models

Size: S
Implementer: Claude Code, claude-opus-5-5, unknown effort
Implementer: Claude Code, claude-opus-5-5, high effort

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

Review

Rewritten by quest review after each round.

  • Head: 25c7b07

  • Base: origin/main at 6488ce6

  • Risk: high, The billing changes could affect customer charges, and reverting the code would not necessarily undo charges already incurred.

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1e489-fb42-7fd0-a179-992796dcdb59, round 1

  • Score: 9/10, No actionable defects were found; the acknowledgment and billing changes are consistent with the transport implementation, but builds and tests were not run as instructed.

  • Raised the risk to high: The billing changes could affect customer charges, and reverting the code would not necessarily undo charges already incurred.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The transport acknowledgment API adds Acked::Sent for bytes accepted by a reliable link without confirmation of peer receipt. qmux now reports this state using its send offset, and tests cover sent bytes, stalled writes, and resets. Billing treats Acked::Sent like Acked::Delivered. Documentation updates describe the acknowledgment and billing behavior, and distinguish qmux from the browser transport.

Merge Risk: Minimal · up to e73d7

qmux egress billing now counts only bytes the reliable link has taken, so bytes later discarded by a reset or lost session are no longer billed. The documentation describes this accurately. The one remaining concern, premature billing on a stalled or failed transport, was checked against the driver path and does not occur. No merge-blocking risk remains.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: qmux egress billing now counts only bytes the link took. The title is specific and matches the summary, objectives, and code changes in Acked, the qmux SendStream offs…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description is specific and on-topic. It explains that qmux egress was billed for bytes the stream accepted, including bytes discarded by a reset or lost session. It describes the new Acked::Sent …
Full details: Title check

Explanation

The title names the main change: qmux egress billing now counts only bytes the link took. The title is specific and matches the summary, objectives, and code changes in Acked, the qmux SendStream offset, and the Billing logic. It is 53 characters and uses the conventional fix(qmux)! form for a breaking change.

Full details: Description check

Explanation

The description is specific and on-topic. It explains that qmux egress was billed for bytes the stream accepted, including bytes discarded by a reset or lost session. It describes the new Acked::Sent variant, the qmux SendStream Offset change, the Billing writer change, and the validation performed. This matches the changeset in quest/m3/qos/lag-histogram.md, quest/m4/quic-ack-hook.md, coding/writer.rs, stats.rs, transport.rs, qmux session.rs, and the qmux session tests.

Finishing Touches

Simplify code

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2876c0ad-3c6f-4ec3-a1b3-6887edd26348
Commits

Reviewing files that changed from the base of the PR and between 6488ce6 and e73d768.

Files selected for processing: 7

  • quest/m3/qos/lag-histogram.md
  • quest/m4/quic-ack-hook.md
  • rs/moq-net/src/coding/writer.rs
  • rs/moq-net/src/stats.rs
  • rs/moq-net/src/transport.rs
  • rs/qmux/src/session.rs
  • rs/qmux/src/session/tests.rs

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No code issues found. One quest doc still describes the old qmux behavior.

Reviewed changes

I reviewed the new Acked::Sent answer, qmux's poll_acked built on its mux's transmit bookkeeping, and how coding::Writer billing uses it.

  • Acked::Sent: a new variant, appended last, that means "a reliable link took the bytes" and is distinct from Delivered. The trait and stats docs explain how it differs.
  • qmux SendStream: now mints moq_quic::send::Offset and answers poll_acked from the mux. The mux marks stream data as acknowledged in Connection::transmitted(), which wakes Wait::Acked. The acked_len() shortcut keeps the prefix that went out after a reset, a stop, or a release, matching the QUIC adapters.
  • Billing: Billing::poll bills Sent the same as Delivered. Writer::reset and Drop settle billing before resetting, so the prefix that went out is still billed and the tail the reset discarded is not.
  • Tests: a new qmux test covers each version through the sent, stalled, and reset cases. The writer billing test now also runs against a stream that answers Sent.

ℹ️ quest/m4/quic-ack-hook.md still says qmux answers Unsupported

The PR updates quest/m3/qos/lag-histogram.md, but quest/m4/quic-ack-hook.md:19-22 still says "qmux and the browser transport stay on the trait's default Acked::Unsupported… writing into TCP is not delivery". After this PR, only the browser stays on Unsupported. A later quest run that reads this file would plan against the old behavior.

Technical details
# Stale qmux acknowledgment claim in quic-ack-hook quest

## Affected sites
- quest/m4/quic-ack-hook.md:19-22 — says qmux stays on `Acked::Unsupported`

## Required outcome
- The quest says qmux answers `Acked::Sent` once its link takes the bytes. `Sent` is not an acknowledgment, so the 2026-10-22 reasoning, that writing into TCP is not delivery, still holds for QoS. Only the browser transport keeps the default `Unsupported`.

Pullfrog  | Fix it ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 25c7b072f41e09633fa490134321a9a67ed83f74.

I support separating Sent from Delivered. Billing can retire the prefix QMux has handed off, including after reset, without treating it as a peer acknowledgment for latency metrics. I found no implementation blocker for fixing the mux-retained tail that was previously billed.

Narrow the stats contract: it still says “never what … a lost session discarded.” The QMux driver advances the offset on transport acceptance, before that batch flushes. A broken link can still leave billed bytes in that batch, as the PR body notes. Say the fix excludes bytes discarded before transport acceptance, not all bytes lost on session failure. That distinction should carry into the release note and any metering assertions.

Static review only; I did not independently reproduce the metering result or run the tests. COMMENT only, not approval.

@kixelated

Copy link
Copy Markdown
Collaborator Author

Maintainer decision (2026-11-17): ✅ accept the distinct Acked::Sent answer, revisiting the 2026-11-15 qmux choice in quest/m2/api-ack-one-primitive.md (rejected: qmux answers Delivered; bill unsupported streams only on finish; keep billing accepted bytes). ✅ /quest-merge this PR.

(Written by Claude Opus 5.5 with high)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Answering #8930 (comment): head 25c7b072f already implements the accepted Acked::Sent answer, and the PR body now records the decision as confirmed with its ## Waiting removed. No code change follows.

(Written by Claude Opus 5.5 with high)

@kixelated
kixelated enabled auto-merge November 17, 2026 06:51
…upported

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Fixed Pullfrog's note in e73d768: quest/m4/quic-ack-hook.md now says qmux answers Acked::Sent, which billing counts and acknowledgment consumers ignore, and only the browser transport stays on Acked::Unsupported.

(Written by Claude Opus 5.5 with high)

@kixelated
kixelated disabled auto-merge November 17, 2026 07:26
@kixelated
kixelated enabled auto-merge November 17, 2026 07:27

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the delta since the prior Pullfrog review (25c7b07 → e73d768).

  • Updated quest/m4/quic-ack-hook.md to say qmux answers Acked::Sent and the browser transport stays on Acked::Unsupported. This resolves the stale-doc note from the earlier review. No code changed.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit b719f78 Nov 17, 2026
7 checks passed
@kixelated
kixelated deleted the fix/qmux-bills-what-the-link-took branch November 17, 2026 07:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Assignees

Couldn't load assignees.