Skip to content

Walk every record of the partition streams to its last byte: long texts and the entity schemas of family documents - #185

Merged
mahach666 merged 4 commits into
mainfrom
feat/census-every-record
Nov 11, 2026
Merged

mahach666 merged 4 commits into
mainfrom
feat/census-every-record

Conversation

@mahach666

@mahach666 mahach666 commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

With this change rvtio coverage walks every record of every partition stream of the 52 project files measured to its last byte: 29,794,710 records of the releases 2020 to 2027, none left. The 117 records that were left after #182 and #183 had two causes:

  • A long text (45 records). The field walk stopped at any text longer than 100,000 characters, which real files hold (a load report as HTML, long strings of a parameter, a long value of an entity). A text is now read as long as its count says; one longer than the rest of the record stops the walk with end of payload, as any read past the end does. Twice the count is computed in 64 bits, so a count near 2^31 or above cannot wrap.
  • An entity of a loaded family (72 records). An entity in the block of a loaded family can name a schema that the project's Global/Latest does not hold. Up to release 2024 the family keeps its schemas in its own ADocument, which Global/ContentDocuments holds per content document under the key of the content block (a ContentMarker laid out as a content block of a partition stream, the length, the document, the length again, and a closing marker at the last byte). From release 2025 those documents hold no schema and Global/Latest holds them in m_schemaUsageMap, whose m_contentDocsKeys lists the families that use each. The census and the trace now read a family block by the schemas of its own document and those of Global/Latest.

Refs #15

Measured

rvtio coverage on the 52 files, main against this branch:

main this branch
Records not exact, of 29,794,710 117 0
Class rows with more exact records 32
Class rows with fewer 0

Items and payload bytes of every class row are unchanged.

Global/ContentDocuments on the 52 files: the stream frames to its last byte in every file and every embedded ADocument walks without an error. Documents and documents with schemas per release: 2020 907 / 8, 2021 4,920 / 30, 2022 6,688 / 94, 2023 3,145 / 7, 2024 1,548 / 3, 2025 4,153 / 0, 2026 121 / 0, 2027 52 / 0. A schema held both there and in Global/Latest has the same fields in all 75 such cases. Every one of the 72 entities names a schema of the document whose key is the key of its block.

The text change is in the walk the readers behind the output use as well. Ten files that hold such a text, exported with main and with this branch:

  • The IFC of all ten is identical.
  • In the compact JSON of two MEP files, 8 family types of lighting fixtures get their parameter list (their record is now read to its end).
  • In the export report of three files of one MEP project, one view type and one sun study, which are not written, change the reason they are left out for (type-element and no-project-phase-reference).

Not covered

  • The family document schemas are read by the census and the trace only; the readers behind the output still read a family block by the schemas of Global/Latest.
  • The reader of the Revit 2022 layout and the rewriter to that layout keep their own text limits (10,000 and 100,000 characters); neither is changed here.
  • From release 2025 a family block is given all schemas of Global/Latest; whether m_contentDocsKeys names its key is not checked.

Verification

  • Full test suite on net8.0: 10808 tests, 10659 passed, 141 skipped, 8 failed. The 8 failures are bounding-box tests (ValidatedTests 2, ApiTests.Proxy289128_* 3, ElementHeaderBoundingBoxTests 3) that fail the same way on main without this change. 7 new test cases, all synthetic: a text longer than the old limit is read, and one character more than the record holds, a count of 2^31 and a count of all ones stop; a family block is read by the schemas of its own document and not the project block; a family block is read by the schemas of Global/Latest (the release 2025 case); a family block stops when its document has another key, the stream is not closed, or it does not decode.
  • The census on all 52 files with main and with this branch, compared class row by class row; the exports of the ten files above with main and with this branch.
  • The family document rule was checked by a second, independent run: the framing of Global/ContentDocuments was walked again in a separate script on a file of release 2022 and one of 2025 (closed at the last byte), and the census was run again on 16 files with a build of its own: the same rows, 72 records gained, none lost.
  • Two independent reviews (the text change and the readers that share the walk; the family document reader, its tests and the documentation) with a second check of every finding: no defect in the code; the documentation gave the count and the key of the content marker in the wrong order, fixed here.
  • After Write the bars of a rebar system that holds an extensible storage entity #184 was merged, main was merged into this branch (the changelog keeps both entries, the test totals are summed: 9680 of 10098); on the merged tree the walker, census, rebar and parameter tests pass (183 passed, 15 skipped) and the census of the 52 files is unchanged (0 records not exact). The full suite was run before the merge.
  • net10.0 was not built or tested.

The field walk stopped at a text longer than 100,000 characters. A text is now read as long as its count says; one that does not fit the rest of the record stops the walk with the reason of any read past its end, and twice the count is computed in 64 bits.
An entity in the block of a loaded family can name a schema that Global/Latest
does not hold. Up to release 2024 the family's own ADocument in
Global/ContentDocuments stores it; from 2025 Global/Latest holds it in
m_schemaUsageMap. The census and the trace now read a keyed block by the
schemas of its own document and those of Global/Latest.
# Conflicts:
#	CHANGELOG.md
#	docs/test-fixtures.md
@mahach666
mahach666 merged commit 6e902e9 into main Nov 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant