feat: download mode for the chunk stream endpoint - #5618
martinconic wants to merge 1 commit into
Conversation
| s.metrics.ChunkStreamOpenConnections.WithLabelValues("download").Inc() | ||
| defer s.metrics.ChunkStreamOpenConnections.WithLabelValues("download").Dec() | ||
|
|
||
| ctx, cancel := context.WithCancel(context.Background()) |
There was a problem hiding this comment.
you probably also want to have another goroutine that selects on s.quit. if the node gets shut down, this context doesn't cancel. ideally you cancel the context once if there's a shutdown or if there's an error, then you save selecting on s.quit later on in the method
| return | ||
| } | ||
|
|
||
| if len(msg) < 1+swarm.HashSize || (len(msg)-1)%swarm.HashSize != 0 { |
There was a problem hiding this comment.
some comment as to why is this off by one needed?
| // because ReadMessage allocates a fresh buffer per message; a pooled or | ||
| // reused read buffer would corrupt addresses across concurrent workers. | ||
| addrs := make([]swarm.Address, 0, batchCount) | ||
| for i := 0; i < len(payload); i += swarm.HashSize { |
There was a problem hiding this comment.
what about if there's an encrypted chunk address here? won't work no..?
| func (s *Service) fetchAndSendChunk( | ||
| streamCtx context.Context, | ||
| logger log.Logger, | ||
| loggerV1 log.Logger, |
| s.metrics.ChunkStreamDeliveryCount.WithLabelValues("success").Inc() | ||
|
|
||
| chunkData := chunk.Data() | ||
| resp := make([]byte, 1+swarm.HashSize+len(chunkData)) |
There was a problem hiding this comment.
not sure about this custom serialization by hand thing... it is just specced out in the openapi spec and assumed that implementers should implement it by hand. it is fragile and breakable. why not use some sort of standard serialization format to both decode the request and encode the response?
2 participants
Checklist
Description
Adds a download mode to
GET /chunks/stream. A client opens one websocket and pulls many chunks over it, instead of paying for an HTTP request per chunk.Closes #5417, closes #5599.
Protocol
Mode is fixed at handshake — via
Sec-WebSocket-Protocol: swarm-chunk-download, or?mode=downloadfor browser clients that cannot set headers. A connection is an upload stream or a download stream, never both.Request frame:
['D'][32-byte address]..., up to 256 addresses.'D'is the command byte from #5417; batching sits under it so the two compose, and it leaves room to add commands later without breaking clients.Reply frame:
[status][32-byte address][payload]—0x00success,0x01not found,0x02error. Replies arrive out of order as retrievals complete, so clients match on the address. Exactly one reply per requested address.swarm-cacheis honoured on the download path, matchingGET /chunks/{address}.From #5599
Batched requests (256/frame), a bounded worker pool rather than an uncontrolled request storm, streamed partial results, per-address success/failure, and relief from the browser's six-connections-per-host limit. Implemented over websocket — one of the response options that issue listed — rather than
POST /chunks/batch.The video-streaming motivation in #5599 is not served by this endpoint and has been split out. Playback wants HTTP Range and server-side read-ahead on
/bzz; this endpoint has no cancellation and strict FIFO delivery, so a seek would mean dropping the connection. It suits random-access and bulk workloads: manifest traversal, SQLite/Parquet-style page reads, bulk sync, pinning.Notes for review
topology.ErrNotFoundmaps to0x01, matching howbzz.gotreats the same error from the samestorer.Downloadcall.getter.DefaultFetchTimeout, the constant the joiner already uses, so one unreachable chunk can't park a worker indefinitely.WriteControlis safe concurrently, and holding the mutex would let a slow delivery delay the close frame.ReadMessagereturns, instead ofapi.Close()timing out against a 15-minute read deadline.AI Disclosure