Skip to content

v3: CycleBuffer recycles a segment without resetting StartTrimCount, corrupting another connection's reply stream (request to backport de301373 to 3.4.x) #3302

Closed

Description

@scottt732

Summary

In 3.x, RESPite.Buffers.CycleBuffer.Segment.Recycle() returns a segment to a single process-wide static Segment? _spare after resetting Next, Memory, RunningIndex and the trim flags, but not StartTrimCount. Segment.Create() / Init() then hand that segment to the next buffer that needs one without resetting it either. Because CycleBuffer computes read/write offsets as StartTrimCount + committed, the connection that inherits the segment reads and writes at the wrong offset.

This looks like the issue the stress test in #3251 found; that PR was closed in favour of v4, and the fix landed on the v4 line in de30137 (4.0.69-alpha and later). It is still present on main and in 3.4.20 (src/RESPite/Buffers/CycleBuffer.cs, Segment.Recycle / Create / Init).

What we see in production

  • StackExchange.Redis 3.3.1 and 3.4.20, .NET 10, RESP2, TLS, a single Valkey primary; several multiplexers in one process (a DI singleton plus Microsoft.Extensions.Caching.StackExchangeRedis and the SignalR backplane).
  • Shortly after process start, and then repeatedly, connections fail with InternalFailure … RESP protocol failure around offset 0-0: Unexpected protocol prefix: N where N is a byte from the middle of an earlier payload (10 = LF, or ASCII letters, ", /, digits), with small in-pipe values (≈10–40 KB).
  • Without HighIntegrity, connections instead silently match replies to the wrong commands (Unexpected response to GET: Array: 0 items, … to UNLINK: SimpleString: OK, Invalid format parsing Integer as Boolean) and stay that way until the process restarts.
  • With HighIntegrity on, each fault is detected and the connection is torn down, but the faults recur in waves across every connection in the process and some processes enter reconnect loops (40–117 faults in 2 minutes). Restarting the process does not help: the new process faults within minutes. This is consistent with each teardown recycling segments into the shared slot and seeding the next corruption.
  • We could not reproduce with a single multiplexer under synthetic load (large values up to 8 MB, TLS, fragmenting proxies, many databases, Lua, transactions), which also fits a cross-connection shared-state bug rather than a framing bug.
  • Pinning 2.13.17 (Pipelines.Sockets.Unofficial, no CycleBuffer) is our mitigation.

Request

Could de30137 (resetting StartTrimCount when a segment is recycled/initialised, or not sharing recycled segments across buffers) be backported to the 3.4.x line? We're happy to test a build.

Activity

  1. mgravell commented on Oct 27, 2026

    @mgravell
    Collaborator

    Thanks - confirmed, and the diagnosis is spot on. Beyond the stale field, I can reproduce actual byte corruption on main/3.4.20: when a buffer is drained while a write lease is open (the outbound copy-out loop racing the writer), Commit copies from an offset that includes the inherited trim, and the "new" buffer emits bytes from the previous owner's payload. It doesn't need a teardown to get there, either: the multi-segment discard path recycles trimmed segments in steady state.

    The fix from v4 backports cleanly; it's in #3303, along with a test that checks the bytes rather than the field.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions