Skip to content

fix: don't silently drop data on short reads in concurrent WriteTo - #660

Open
ChrisJr404 wants to merge 1 commit into
pkg:masterfrom
ChrisJr404:fix-writeto-short-reads
Open

fix: don't silently drop data on short reads in concurrent WriteTo#660
ChrisJr404 wants to merge 1 commit into
pkg:masterfrom
ChrisJr404:fix-writeto-short-reads

Conversation

@ChrisJr404

Copy link
Copy Markdown

Fixes #658.

The concurrent File.WriteTo path (what io.Copy uses) dispatches reads at offsets spaced chunkSize apart and advances the offset unconditionally. Since a server is free to return a short read, and does whenever the client max packet size is larger than the server's, every chunk kept only its first short read and the rest of that chunk was skipped. The copy came back truncated with a nil error, so it was silent corruption rather than an error.

The sequential paths already handle this by looping in readChunkAt. This does the same thing in the worker: when the pre-dispatched read comes back short, fill the rest of the chunk before handing it on, so the pre-computed offsets stay aligned. A genuine EOF ends the fill and is carried through as the chunk's error. The full-read path (server returns the whole chunk) is unchanged.

Added a regression test that opens a file through an in-memory server with a smaller max packet size than the client. It fails on master (WriteTo returns ~1/4 of the file) and passes with this change. go test, go test -race and go vet are green on the package.

The concurrent File.WriteTo path (which io.Copy uses) dispatches reads at
offsets spaced chunkSize apart and advances the offset unconditionally. A
server is allowed to return fewer bytes than requested, which it does whenever
the client max packet size is larger than the server's, so every chunk kept
only its first short read and the rest was skipped. The copy then returned a
truncated file with a nil error.

Fill the remainder of a short chunk in the worker before handing it on, the
same way the sequential path already does via readChunkAt, so the pre-computed
offsets stay aligned. A genuine EOF ends the fill and is carried through as the
chunk's error. The full-read path is unchanged.

Fixes pkg#658
Comment thread client.go
// chunk's error.
if n < chunkSize {
var m int
m, err = f.readChunkAt(readWork.res, b[n:], readWork.off+int64(n))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is this is that we’ve already dispatched reads to later offsets, catch up reads like this result in non-sequential access, which has been a result of various bugs here and there.

Namely, some ssh implementation automatically delete a file once the whole file has been read the first time, this backfilling catch up would break under this situation.

Also, backing up and backfilling can cause significant performance degradation as servers are tuned for sequential read access, not really for random access.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

File.WriteTo silently drops data on short reads when the server's max packet size is smaller than the client's

2 participants