Skip to content

fix(rtc_engine): keep a retry window for reliable resume replay - #1341

Open
tarsyang wants to merge 1 commit into
livekit:mainfrom
tarsyang:fix/reliable-retry-window
Open

fix(rtc_engine): keep a retry window for reliable resume replay#1341
tarsyang wants to merge 1 commit into
livekit:mainfrom
tarsyang:fix/reliable-retry-window

Conversation

@tarsyang

Copy link
Copy Markdown

Summary

data_channel_task keeps recently sent reliable packets in a TxQueue so _enqueue_for_retry_from can replay them after a resume, starting from the server-provided last received sequence. TxQueue::trim(target) discards from the front until the retained size is at or below target, and the send loop passes sent (the byte count flushed by the current OnBufferedAmountChange event) as that target. The queue therefore retains roughly the last flushed burst, typically one packet: a resume that needs anything older replays nothing, logs "Wrong packet sequence while retrying: ... packets missing", and those reliable packets are permanently lost even though the retry mechanism exists to recover exactly them.

This mechanism was introduced by #688 as an implementation of the reliability improvements from livekit/client-sdk-js#1546 and livekit/client-sdk-swift#737. The Swift implementation sizes the same buffer as the flushed bytes plus a floor:

// If rtc drains its buffer to 0, keep at least this amount of data for retry.
// Should be >= the full backpressure amount to avoid losing packets.
private static let reliableRetryAmount: UInt64 = .init(Double(reliableLowThreshold) * 1.25)

(RetryBuffer.trim(toAmount:) keeps toAmount + minAmount.) The floor did not carry over. Its rationale applies unchanged here: everything below the send-gate threshold can be sitting in the transport undelivered when a connection dies, so the retained window must cover at least that amount.

The fix trims to sent + threshold + threshold / 4, using the live threshold: the 2 MiB default gives the same 2.5 MiB floor as Swift's constant, and the window scales if the threshold is adjusted at runtime.

Verification

cargo test -p livekit --lib (76 tests) and the data channel e2e suite against a local livekit-server --dev (including test_reliable_retry) pass; cargo fmt --check is clean. Note that the existing e2e cannot discriminate the window size: SignalReconnect keeps the SCTP association alive, so replay never needs more than the last burst there. The evidence for the fix is the reference semantics above and TxQueue::trim's documented keep-at-most contract.

TxQueue::trim() takes a target size to keep, but the send loop passed
the byte count flushed by the current OnBufferedAmountChange event, so
the retry queue retained roughly the last flushed burst (typically one
packet). A resume that needed anything older replayed nothing and logged
"Wrong packet sequence while retrying: ... packets missing", permanently
losing reliable packets that the retry mechanism exists to recover.

Trim to the flushed bytes plus a floor of 1.25x the low threshold
instead. This restores the retry-buffer floor from client-sdk-swift#737,
which this mechanism implements (livekit#688): the retained window must cover
at least the full backpressure amount, since everything below the
send-gate threshold can be sitting undelivered in the transport when a
connection dies.
@tarsyang
tarsyang requested a review from ladvoc as a code owner August 19, 2026 15:45

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant