perf(storer): filter reserve chunks by proximity before unmarshaling in sampler - #5616
Open
gacevicljubisa wants to merge 2 commits into
Open
gacevicljubisa wants to merge 2 commits into
gacevicljubisa wants to merge 2 commits into
Conversation
Sampling is about to start reading chunks into a buffer that each worker reuses. Nothing today checks that the bytes handed back in a SampleItem are still the bytes of that chunk, so a reused buffer would silently hand the redistribution proof the contents of some later chunk. assertValidSample now checks two things for every item: that ChunkData still reproduces ChunkAddress, and that no two items share a backing array. Every existing sample test picks both up. Also add the rulers for the work that follows. BenchmarkReserveSample1k keeps its name and behaviour so the recorded baseline stays comparable; its body moves to a helper that BenchmarkReserveSample10k reuses over a ten times larger reserve. BenchmarkChunkStoreGet measures a single chunk read, split into a variant that builds the ChunkStore handle per call as the sampler does today and one that hoists it, so the cost of the handle alone is visible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KFUseFQ8rhp6N9X6YS7pbq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checklist
Description
This PR introduces an early proximity filter for the reserve chunk iterator during
ReserveSample:ChunkBinItemserializes the chunk address at a fixed offset (byte 9, right after the 1-byte bin and 8-byte bin ID),reserve.ProximityFilterextracts the 32-byte chunk address directly from the raw LevelDB value buffer.< committedDepth, the iterator skips it immediately. This completely avoids allocating a value copy, instantiating&reserve.ChunkBinItem{}, and unmarshaling all fields (Address,BatchID,StampHash, etc.) for non-matching chunks.committedDepth > storageRadius), the reserve contains chunks in binscommittedDepthfor the sample round. For these nodes, this eliminates large amounts of heap allocations and unmarshaling work during sampling.Open API Spec Version Changes (if applicable)
N/A
Motivation and Context (Optional)
Part of reserve sampling performance optimizations.
Related Issue (Optional)
#5174
Screenshots (if appropriate):
AI Disclosure