fix: don't silently drop data on short reads in concurrent WriteTo - #660
Open
ChrisJr404 wants to merge 1 commit into
Open
fix: don't silently drop data on short reads in concurrent WriteTo#660ChrisJr404 wants to merge 1 commit into
ChrisJr404 wants to merge 1 commit into
Conversation
The concurrent File.WriteTo path (which io.Copy uses) dispatches reads at offsets spaced chunkSize apart and advances the offset unconditionally. A server is allowed to return fewer bytes than requested, which it does whenever the client max packet size is larger than the server's, so every chunk kept only its first short read and the rest was skipped. The copy then returned a truncated file with a nil error. Fill the remainder of a short chunk in the worker before handing it on, the same way the sequential path already does via readChunkAt, so the pre-computed offsets stay aligned. A genuine EOF ends the fill and is carried through as the chunk's error. The full-read path is unchanged. Fixes pkg#658
puellanivis
reviewed
Aug 26, 2026
| // chunk's error. | ||
| if n < chunkSize { | ||
| var m int | ||
| m, err = f.readChunkAt(readWork.res, b[n:], readWork.off+int64(n)) |
Collaborator
There was a problem hiding this comment.
The problem is this is that we’ve already dispatched reads to later offsets, catch up reads like this result in non-sequential access, which has been a result of various bugs here and there.
Namely, some ssh implementation automatically delete a file once the whole file has been read the first time, this backfilling catch up would break under this situation.
Also, backing up and backfilling can cause significant performance degradation as servers are tuned for sequential read access, not really for random access.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #658.
The concurrent
File.WriteTopath (whatio.Copyuses) dispatches reads at offsets spacedchunkSizeapart and advances the offset unconditionally. Since a server is free to return a short read, and does whenever the client max packet size is larger than the server's, every chunk kept only its first short read and the rest of that chunk was skipped. The copy came back truncated with a nil error, so it was silent corruption rather than an error.The sequential paths already handle this by looping in
readChunkAt. This does the same thing in the worker: when the pre-dispatched read comes back short, fill the rest of the chunk before handing it on, so the pre-computed offsets stay aligned. A genuine EOF ends the fill and is carried through as the chunk's error. The full-read path (server returns the whole chunk) is unchanged.Added a regression test that opens a file through an in-memory server with a smaller max packet size than the client. It fails on master (WriteTo returns ~1/4 of the file) and passes with this change.
go test,go test -raceandgo vetare green on the package.