drpcstream: gate flow control at the message boundary and overdraft - #92
Open
suj-krishnan wants to merge 2 commits into
Open
drpcstream: gate flow control at the message boundary and overdraft#92suj-krishnan wants to merge 2 commits into
suj-krishnan wants to merge 2 commits into
Conversation
suj-krishnan
requested review from
5hubh4m,
Nukitt,
cthumuluru-crdb and
shubhamdhama
and removed request for
5hubh4m
August 3, 2026 05:36
…option Add drpcopts.FlowControl (Enabled, StreamWindow, GrantThreshold) and install the per-stream windows at stream creation when it is set: the send window seeded with StreamWindow credit and watching the send signal for termination, and the receive window returning consumed-byte credit coalesced at GrantThreshold. Enablement is internal-only for merge safety: the option lives in internal/drpcopts, not on the public drpcstream.Options, so a consumer bumping the dependency cannot enable flow control before the deliberate promotion to a public option. The installation site normalizes the configuration rather than failing. The frame size follows SplitData: zero uses the 64 KiB default, and a negative SplitSize means unbounded frames, which carry no per-frame bound. A valid config needs positive sizes and, for bounded frames, GrantThreshold plus one frame fitting in StreamWindow -- so credit withheld by coalescing cannot strand the sender below the next frame's cost. Anything that cannot make progress resorts to FlowControl.SetDefaults and logs the override, so a misconfiguration degrades to a working stream instead of crashing a process that embeds drpc. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Per-frame credit gating deadlocks under grant-on-consume: credit is returned only when a complete message is consumed, so a multi-frame message that runs out of credit mid-way strands the receiver with an incomplete message it cannot consume, and no grant is ever issued. Even a message that fits the window can deadlock once coalesced credit is withheld below the grant threshold. Gate at the message boundary instead. The first frame of a message acquires credit (parking until it arrives, or the window closes on termination); once committed, later frames overdraft -- debit without parking, letting the balance go negative -- so a started message always completes on the wire. The receiver then consumes the whole message and returns the credit, and applyGrant repays the deficit before any credit accrues. Backpressure is preserved at message granularity: a new message parks on its first frame until the overdraft is repaid. The overdraft is bounded by MaxMessageSize, a new FlowControl field. A send larger than the bound fails fast, and the receiver's PacketAssembler rejects an assembling message that exceeds it before buffering it whole. Only the first frame need fit the window, so MaxMessageSize may be smaller than StreamWindow (a stricter, safer bound); an unset bound defaults to 64 MiB independently, so omitting it does not force the window and threshold to their defaults. Because a message may overdraft past the window up to MaxMessageSize, peak per-stream memory is roughly StreamWindow + MaxMessageSize rather than StreamWindow. An oversized message surfaces as a dedicated MessageSizeError class, not a protocol fault: ToRPCErr maps it to codes.ResourceExhausted, as gRPC does. An oversized receive fails only its own stream -- HandleFrame sends the peer an abortive terminal error so a credit-gated send there wakes instead of hanging, then returns without tearing down the multiplexed connection (the framing faults that do tear it down rely on the transport close to notify the peer). Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
suj-krishnan
force-pushed
the
sujatha/flow-control-overdraft
branch
from
August 4, 2026 16:08
7d8c784 to
bb1d255
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes grant-on-consume flow control deadlock-free for multi-frame and larger-than-window messages via a message-boundary gate with overdraft, and bounds a single message with
MaxMessageSize.The deadlock
Per-frame credit gating deadlocks under grant-on-consume: credit is returned only when a complete message is consumed, so a multi-frame message that runs out of credit mid-way strands the receiver with an incomplete message it cannot consume — and no grant is ever issued. Even a message that fits the window can deadlock once coalesced credit is withheld below the threshold.
Message-boundary gate + overdraft
The first frame of a message acquires credit (parking until it arrives, or the window closes on termination); once committed, later frames overdraft — debit without parking, letting the balance go negative — so a started message always completes on the wire. The receiver then consumes the whole message and returns the credit (
applyGrantrepays the deficit first). Backpressure is preserved at message granularity: a new message parks on its first frame until the overdraft is repaid.Bounded by
MaxMessageSize(newFlowControlfield)PacketAssemblerrejects an assembling message that exceeds it before buffering it whole;MessageSizeError→codes.ResourceExhausted(matching gRPC) and fails only that stream —HandleFramesends the peer an abortive terminal error so a credit-gated send there wakes instead of hanging, then returns without tearing down the multiplexed connection.Because a message may overdraft past the window up to
MaxMessageSize, peak per-stream memory is roughlyStreamWindow + MaxMessageSizerather thanStreamWindow.