Skip to content

Read the element index from the records of the partition framing - #189

Merged
mahach666 merged 1 commit into
mainfrom
feat/framed-element-index
Nov 11, 2026
Merged

mahach666 merged 1 commit into
mainfrom
feat/framed-element-index

Conversation

@mahach666

@mahach666 mahach666 commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Second step of the read path onto the partition framing; #187 was the first.

The element index of the files with their own class schema (WideElementRecords.BuildIndex: the header category, design option and outline of each element, the class of its root object, and the field walk of that object) and the reinforcement reader scanned every partition stream position by position for byte runs that looked like an element header or a root object. The scan also took the records of the keyed content blocks, the documents of loaded families, whose element ids repeat project ids; where such a copy differed, the value was dropped as a conflict. Now:

  • When the framing of a stream holds, the cut (RvtFileReader.CutPartition) also lists the records of the project's own block (the null-key map): channel, owner and the place of the whole item in the inflated stream (PartitionFraming.Framed). Chain terminators and the records of keyed blocks are not listed.
  • The records that record routing marks as not the current copy are left out (RecordRouting.Drop), as the chunks already were.
  • The index reads such a stream by its records: the channel-101 records are the element headers and the channel-102 records the root objects. Measured on 52 files: every project channel-101 record is an ElementHeader, and channel 103 holds only GElement and SerializedDummy.
  • A stream whose framing does not hold, or whose framing was read with another id width than the schema's, is scanned as before.
  • RecordRouting.Prune keeps a chunk that no stale range reaches as the same array, so the framing and the chunks share it.

Refs #47, refs #48: both are about byte scans taking items that are not records of the project. This PR moves the index and the reinforcement reader off the scan; the block scan of issue 48 and the class counts of issue 47 are in other readers (the block parser and the stream walk) and are not changed here.

Measured

scan (base) framing (this PR)
Header-like items found, 29 files with the index 5,987,630 848,325
Block-like items found 6,640,008 848,325

52 project files of the releases 2020 to 2027, exported with the base branch and with this branch (IFC with its report, compact JSON):

  • IFC: identical in all 52.
  • Export report: identical in all 52.
  • Compact JSON: identical in 51. In one Revit 2021 file one element, the pen width table (id 2), gets its name and properties; before, a copy of id 2 in a family document made its root class a conflict. The other files write that element the same way.
  • Time of all exports: 5,555 s with the base, 5,643 s with this branch (+1.6%). Per file (29 files of 40 s or more) from -18% to +33%, 10 of them faster; the two builds ran one after the other on a shared machine and the timing was not repeated.

Not covered

  • The other readers (the record parser, the block parser, the stream walk, the 2022 layout rewriter, the string and parameter scans) still read the concatenated chunks.
  • In 6 files the project block holds more header records than the element table has rows (1 to 377); those ids are not wanted and are skipped, as before. Why they are there was not looked into.
  • The rebar dump of the CLI still scans every stream; it is a diagnostic tool and lists family blocks too.

Verification

  • Full test suite on net8.0: 10814 tests, 10665 passed, 141 skipped, 8 failed. The 8 failures are bounding-box tests (ValidatedTests 2, ApiTests.Proxy289128_* 3, ElementHeaderBoundingBoxTests 3) that fail the same way on main without this change.
  • New tests: the index reads a framed stream by its project records only (the copy in a family block is not read, the root block is handed over at its place, a framing of another id width is not used); the cut lists the records of the project block with their places and leaves out the family block and the chain terminators; the records that are not the current copy are dropped; Prune keeps an untouched chunk as the same array.
  • A review with a second check of every finding: no defect; the stale filter and the terminator filter had no test (added), and the unpruned segments of a pruned stream doubled its memory (now shared).
  • The 52-file comparison ran on a build before the last two of those fixes. Six files, five of them with stale copies removed by record routing and the file whose JSON changes, were exported again with the final build: IFC and compact JSON identical to the earlier export of this branch.
  • net10.0 was not built or tested.

The element index of the files with their own class schema and the reinforcement reader scanned each partition stream for byte runs that looked like an element header or a root object, which also took the records of the documents of loaded families, whose element ids repeat project ids. A stream whose framing holds is now read by its records: the channel-101 records of the project's own block are the headers and its channel-102 records the root objects, one of each per element, without the copies that record routing marks as not current. A stream whose framing does not hold is scanned as before.

On 29 files the scan found 5,987,630 header-like and 6,640,008 block-like items; the framing gives 848,325 of each. On 52 files the IFC and the export report are identical to before; in the compact JSON of one file one element gets its name and properties.

Refs #47
Refs #48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant