fix(joiner): read into len(buffer), not cap(buffer) - #5619
Open
martinconic wants to merge 1 commit into
Open
martinconic wants to merge 1 commit into
martinconic wants to merge 1 commit into
Conversation
martinconic
requested review from
acud,
akrem-chabchoub,
aloknerurkar,
janos and
sbackend123
September 16, 2026 08:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checklist
Description
joiner.ReadAtsized its read fromcap(buffer)instead oflen(buffer):Reading into a
backing[:10]slice whose capacity is 32768 gives three distinct symptoms:Readreturnsn=32768forlen(p)=10, violating theio.Readercontract (n <= len(p)).len(p), into capacity that belongs to the caller.io.LimitedReader.Ngoes negative (−32758), so the following read reports EOF andio.CopyN(dst, joiner, 10)delivers 512 bytes.The last one is what bites in practice.
http.ServeContentserves everyRangerequest throughio.CopyN, so a ranged download returns a short body whileContent-LengthandContent-Rangestill advertise the full range — a silent truncation rather than an error.This is reachable from the API today.
GET /bytes/{ref}andGET /bzz/{ref}pass the joiner tohttp.ServeContentunwrapped whenSwarm-Lookahead-Buffer-Size: 0is set (pkg/api/bzz.go), so a ranged request on that path loses its tail. Measured against an 18 MB object, every large range stopped at exactly 753,664 bytes (23 × 32 KiB), and 80 of 119 requests came back short:It stayed hidden because every other caller passes a slice with
len == cap:file.JoinReadAllreads into a full 4096-byte buffer, and langos wraps the joiner in abufio.Readerthat fillsb.buf[b.w:]. The default download path therefore masks it entirely, and no existing test exercised a short slice with spare capacity.The fix is
cap→len. The recursion inreadAtOffsetis bounded entirely byreadLen, so capping it at the buffer length is sufficient to keep every write insidelen(buffer).TestJoinerReadBufferLengthcovers all three symptoms withRead,ReadAtandCopyNsubtests: it asserts the returned count equals the requested length, and that sentinel bytes pastlen(p)are untouched. All three subtests reproduce the bug when the fix is reverted.This is long-standing, not a recent regression:
capdates to #667 ("seekjoiner: fill read buffer, integrate langos"), and #2481 only renamedcap(b)tocap(buffer).Motivation and Context (Optional)
Found while testing whether a Parquet client can query a file directly over
/bytesusing HTTP range requests. The bug is not specific to that use case — it affects any consumer that reads into a sub-sliced buffer, which includeshttp.ServeContentfor every ranged download taken with lookahead buffering disabled.AI Disclosure