Repository navigation
Walk every record of the partition streams to its last byte: long texts and the entity schemas of family documents - #185
Merged
Conversation
The field walk stopped at a text longer than 100,000 characters. A text is now read as long as its count says; one that does not fit the rest of the record stops the walk with the reason of any read past its end, and twice the count is computed in 64 bits.
An entity in the block of a loaded family can name a schema that Global/Latest does not hold. Up to release 2024 the family's own ADocument in Global/ContentDocuments stores it; from 2025 Global/Latest holds it in m_schemaUsageMap. The census and the trace now read a keyed block by the schemas of its own document and those of Global/Latest.
# Conflicts: # CHANGELOG.md # docs/test-fixtures.md
This was referenced Nov 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With this change
rvtio coveragewalks every record of every partition stream of the 52 project files measured to its last byte: 29,794,710 records of the releases 2020 to 2027, none left. The 117 records that were left after #182 and #183 had two causes:end of payload, as any read past the end does. Twice the count is computed in 64 bits, so a count near 2^31 or above cannot wrap.Global/Latestdoes not hold. Up to release 2024 the family keeps its schemas in its ownADocument, whichGlobal/ContentDocumentsholds per content document under the key of the content block (aContentMarkerlaid out as a content block of a partition stream, the length, the document, the length again, and a closing marker at the last byte). From release 2025 those documents hold no schema andGlobal/Latestholds them inm_schemaUsageMap, whosem_contentDocsKeyslists the families that use each. The census and the trace now read a family block by the schemas of its own document and those ofGlobal/Latest.Refs #15
Measured
rvtio coverageon the 52 files, main against this branch:Items and payload bytes of every class row are unchanged.
Global/ContentDocumentson the 52 files: the stream frames to its last byte in every file and every embeddedADocumentwalks without an error. Documents and documents with schemas per release: 2020 907 / 8, 2021 4,920 / 30, 2022 6,688 / 94, 2023 3,145 / 7, 2024 1,548 / 3, 2025 4,153 / 0, 2026 121 / 0, 2027 52 / 0. A schema held both there and inGlobal/Latesthas the same fields in all 75 such cases. Every one of the 72 entities names a schema of the document whose key is the key of its block.The text change is in the walk the readers behind the output use as well. Ten files that hold such a text, exported with main and with this branch:
type-elementandno-project-phase-reference).Not covered
Global/Latest.Global/Latest; whetherm_contentDocsKeysnames its key is not checked.Verification
ValidatedTests2,ApiTests.Proxy289128_*3,ElementHeaderBoundingBoxTests3) that fail the same way on main without this change. 7 new test cases, all synthetic: a text longer than the old limit is read, and one character more than the record holds, a count of 2^31 and a count of all ones stop; a family block is read by the schemas of its own document and not the project block; a family block is read by the schemas ofGlobal/Latest(the release 2025 case); a family block stops when its document has another key, the stream is not closed, or it does not decode.Global/ContentDocumentswas walked again in a separate script on a file of release 2022 and one of 2025 (closed at the last byte), and the census was run again on 16 files with a build of its own: the same rows, 72 records gained, none lost.