Skip to content

Cut a partition stream by its exact framing when it holds - #187

Merged
mahach666 merged 2 commits into
mainfrom
feat/exact-partition-read
Nov 11, 2026
Merged

mahach666 merged 2 commits into
mainfrom
feat/exact-partition-read

Conversation

@mahach666

Copy link
Copy Markdown
Collaborator

Summary

The reader cut every partition stream by a search for gzip headers, with a ladder of recoveries for damaged members, although the exact framing of the stream (PartitionFraming: content blocks, segments, gzip members with their CRC and length, record lengths, the owners of each block) was decoded and used by rvtio coverage. Now:

  • A partition stream whose framing holds is cut into its segments, each the output of one gzip member checked by its CRC and length (RvtFileReader.CutPartition).
  • A stream whose framing does not hold, or a file whose class table does not name the framing classes, is searched for gzip members as before, with the recovery of damaged members; the diagnostics name the reason the framing does not hold.
  • The export report counts the members of a stream read by its framing as decoded.

This is the first step of the read path onto the framing. The readers behind the output still see the concatenated chunks and find their records in them as before; handing them the records of the framing (and telling the documents of loaded families from the project's own block) is the next step, and it will change the output.

Measured

  • On 52 project files of the releases 2020 to 2027, the segments of the framing are the chunks the search found, byte for byte, in all 434 partition streams.
  • The IFC, the compact JSON and the export report of 14 of the files, one or more of every release, are identical to the output of the main branch; the export takes as long as before (within a few percent either way).
  • 432 of the 434 streams have bytes after the terminal marker of the framing. They are no part of the stream: the search found no gzip member in them on any file, and the framing leaves them out.

Refs #47, refs #48: both are about the byte scan that finds the blocks in the concatenated chunks; this change does not remove it yet, it gives the next step the framing to read the blocks from.

Not covered

  • The concatenated chunks and the byte scans of the readers behind them (the element index, the 2022 layout, the rewriter) are unchanged.
  • A file whose framing does not hold is read as before; none of the 52 is one.

Verification

  • Full test suite on net8.0, before main with Walk every record of the partition streams to its last byte: long texts and the entity schemas of family documents #185 was merged into this branch: 10805 tests, 10656 passed, 141 skipped, 8 failed. The 8 failures are bounding-box tests (ValidatedTests 2, ApiTests.Proxy289128_* 3, ElementHeaderBoundingBoxTests 3) that fail the same way on main without this change. After the merge: the tests of the read path, the report, the framing, the census, the walker and rebar pass (211 passed, 17 skipped).
  • 3 new tests (PartitionCutTests): a stream whose framing holds is cut into its segments with every member counted as decoded; a stream whose terminal marker is damaged is searched for gzip members, with the reason; one channel written as a chain of two segments gives the output of each member on its own.
  • Exports: 14 files with main and with this branch (IFC with its report, compact JSON), and 6 of them again with the final build: identical.
  • A review with a second check of every finding. Confirmed and fixed: the choice of the path was not tested (now CutPartition and its tests), the test of the segments could not catch a wrong slice of a chain (now the chain test). Confirmed and answered by a measurement instead of a change: the bytes after the terminal marker are not checked; requiring none would leave 432 of the 434 streams to the search.
  • net10.0 was not built or tested.

The reader searched every partition stream for gzip headers. A stream whose framing holds is now cut into its segments, each one gzip member checked by its CRC and length; the search with its recovery of damaged members stays for a stream whose framing does not hold, and the diagnostics name the reason. On 52 project files the segments are the chunks of the search in all 434 partition streams, and the output of 14 files is identical.

Refs #47

Refs #48
…read

# Conflicts:
#	CHANGELOG.md
#	docs/test-fixtures.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant