Repository navigation
Cut a partition stream by its exact framing when it holds - #187
Merged
Merged
Conversation
The reader searched every partition stream for gzip headers. A stream whose framing holds is now cut into its segments, each one gzip member checked by its CRC and length; the search with its recovery of damaged members stays for a stream whose framing does not hold, and the diagnostics name the reason. On 52 project files the segments are the chunks of the search in all 434 partition streams, and the output of 14 files is identical. Refs #47 Refs #48
…read # Conflicts: # CHANGELOG.md # docs/test-fixtures.md
This was referenced Nov 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The reader cut every partition stream by a search for gzip headers, with a ladder of recoveries for damaged members, although the exact framing of the stream (
PartitionFraming: content blocks, segments, gzip members with their CRC and length, record lengths, the owners of each block) was decoded and used byrvtio coverage. Now:RvtFileReader.CutPartition).This is the first step of the read path onto the framing. The readers behind the output still see the concatenated chunks and find their records in them as before; handing them the records of the framing (and telling the documents of loaded families from the project's own block) is the next step, and it will change the output.
Measured
Refs #47, refs #48: both are about the byte scan that finds the blocks in the concatenated chunks; this change does not remove it yet, it gives the next step the framing to read the blocks from.
Not covered
Verification
ValidatedTests2,ApiTests.Proxy289128_*3,ElementHeaderBoundingBoxTests3) that fail the same way on main without this change. After the merge: the tests of the read path, the report, the framing, the census, the walker and rebar pass (211 passed, 17 skipped).PartitionCutTests): a stream whose framing holds is cut into its segments with every member counted as decoded; a stream whose terminal marker is damaged is searched for gzip members, with the reason; one channel written as a chain of two segments gives the output of each member on its own.CutPartitionand its tests), the test of the segments could not catch a wrong slice of a chain (now the chain test). Confirmed and answered by a measurement instead of a change: the bytes after the terminal marker are not checked; requiring none would leave 432 of the 434 streams to the search.