Read the element index from the records of the partition framing

CHANGELOG.md

@@ -56,6 +56,7 @@ API between minor releases.
 
 ### Changed
 
+- Reading: the element index of the files with their own class schema (the header category, design option and outline of each element, the class of its root object and the field walk of that object) and the reinforcement reader read a partition stream whose framing holds by its records: the channel-101 records of the project's own block are the element headers and its channel-102 records the root objects, one of each per element. Before, a scan of the stream took any byte run that looked like an item, including the records of the documents of loaded families, whose element ids repeat project ids; a value on which such a copy differed was dropped. On 29 files the scan found 5,987,630 header-like and 6,640,008 block-like items, the framing gives 848,325 of each. A stream whose framing does not hold is scanned as before. On 52 project files of the releases 2020 to 2027 the IFC and the export report are identical to the output of the main branch; in the compact JSON of one Revit 2021 file one element (the pen width table) now gets its name and properties, as in the other files.
 - Reading: a partition stream whose framing holds (content blocks, segments, gzip members with their CRC and length, record lengths, the owners of each block) is cut into its segments by that framing; the search for gzip members and its recovery of damaged members are kept for a stream whose framing does not hold, and the diagnostics say which way a stream was read. On 52 project files of the releases 2020 to 2027 the segments are the chunks the search found, byte for byte, in all 434 partition streams; the IFC, the compact JSON and the export report of 14 of them, one or more of every release, are identical to the output of the main branch.
 - Field walk by the class schema of the file: a text is read as long as its count says, where the walk stopped at a text longer than 100,000 characters; a text longer than the rest of the record stops it as any read past the end does. `rvtio coverage` also reads an extensible storage entity in the block of a loaded family by the schemas of that family's own document: up to release 2024 a family keeps them in its `ADocument` in `Global/ContentDocuments` (one document per content block, under the key of the block), from release 2025 `Global/Latest` holds them. On 52 project files of the releases 2020 to 2027 every record of every partition stream is now walked to its last byte (29,794,710 records; 117 were not: 45 at a long text, 72 at an entity of a family document); no class row has fewer exact records. Output: of ten files that hold such a text, eight family types of lighting fixtures in two MEP files get their parameter list in the compact JSON, and in three files of one MEP project one view type and one sun study change the reason they are left out of the IFC for; the IFC of the ten files does not change.
 - Default output, IFC and JSON: the readers behind the output read the values of an extensible storage entity by the schema of that entity, as `rvtio coverage` does, so the objects behind those values are read in their place instead of out of step, and the record of an element that keeps an entity is read to its end. An entity whose schema the file does not hold, or with a field type that is not read, is passed as before. The parameters that a store behind an entity would list as unset are still left out, as before; what that rule would give now is not measured. The bars of a rebar system that holds an entity are read now as well. A RebarInSystem that is the one RebarInSystem of its system takes the `GlobalId` of the system, as in the reference; the bar of a system with more keeps its own sub-element `GlobalId` (the number of its bars plus 3). Measured on 28 project files of the releases 2020 to 2026 against the output of the main branch, and for the IFC also against the reference: no IFC value became worse. On eight MEP files the in/out direction and the name of 887 distribution ports are now those of the reference (331, 331, 83, 83, 25, 25, 8 and 1), and on ten MEP files 3,221 assignments of a port or an element to a system are added that the reference has; 48 added assignments are not in the reference at that system (in four files; not checked in Revit). 6 systems the reference has are written that were missing; 26 pipe fitting types that the reference does not have are no longer written, and 9 others it does not have either are (types of a single fitting each, whose `GlobalId` changes with the parameters now read). In three files of one architectural project (releases 2021, 2023 and 2024) 42, 48 and 48 reinforcing bars that were left out are written, all with the `GlobalId` of the reference (in the file of release 2024, 26 of the 42 bars of its systems are the one bar of their system); the three files now write 159, 168 and 168 of the 169 bars of the reference (117, 120 and 120 before). The one bar of a system is not put into a group of its system, which it stands for. In the compact JSON 1,214 family instances of that project (three files) and 3,086 of another (one file) get the parameter list that the other instances of their categories already had, and 53 family types of one MEP file get theirs. Against Revit, on 172 of the 1,214 in the file of release 2021, every value the compact JSON writes as Revit shows it (type, mark, comments, phases, heights, levels, offsets and others, 1,561 values) is equal, and the fields the compact JSON writes in its own form (family, type and host as names, the design option as `None`, justifications as numbers, an empty level as `None`) are written as for 170 instances of the same categories that had their list before. The IFC of 16 of the 28 files and the JSON of 23 do not change.

docs/test-fixtures.md

@@ -45,7 +45,7 @@ folder (`TestFiles.Scratch`); no test uses a fixed absolute path.
 
 ## What runs without the private files
 
-With no private file at all, 9683 of the 10101 test cases run and pass and 418 are skipped: 412 need private files (listed
+With no private file at all, 9685 of the 10103 test cases run and pass and 418 are skipped: 412 need private files (listed
 below; 17 of them only `bim_dump.txt` or `formats_latest_decompressed.bin`) and 6 need other inputs (two need symbolic links, three
 need `RVTIO_FREEFORM_FULL`, `RVTIO_CYL_FULL` or `RVTIO_MESH_FULL`, one is a known gap). The tests that run use synthetic data built
 in the tests and the small fixtures committed under `tests/RvtIO.Tests/Fixtures`. With the full private fixture folder the

src/RvtIO/Parsing/PartitionFraming.cs

@@ -31,6 +31,15 @@ internal static class PartitionFraming
     /// that is the concatenation of every segment's output in file order; the item is <c>Item.Length</c> bytes there.</summary>
     internal readonly record struct Record(int Block, string Key, int Channel, ulong Owner, uint Extra, ReadOnlyMemory<byte> Body, ReadOnlyMemory<byte> Item, long Offset);
 
+    /// <summary>Where a record of the null-key map lies: its channel, owner, and the whole item (owner to closing size) at <see cref="Offset"/>
+    /// for <see cref="Length"/> bytes in the stream's inflated output.</summary>
+    internal readonly record struct ProjectItem(int Channel, ulong Owner, long Offset, int Length);
+
+    /// <summary>A partition stream read by its framing: the output of every segment in file order, and the records of the project's own block
+    /// (the null-key map) in file order, without the chain terminators. The records of the keyed blocks (loaded family documents) are not listed.
+    /// <see cref="IdWidth"/> is the byte width of the owner the walk read.</summary>
+    internal sealed record Framed(IReadOnlyList<byte[]> Segments, List<ProjectItem> Items, int IdWidth);
+
     internal static Tags? TagsOf(FormatsSchema schema)
     {
         ushort? Id(string name) => schema.IdOf(name) is int id and <= ushort.MaxValue ? (ushort)id : null;

src/RvtIO/Parsing/RebarReader.cs

@@ -123,7 +123,8 @@ internal static bool HeldLayout(FileSchema schema) =>
 
     internal static ReinforcementData Read(
         IReadOnlyList<IReadOnlyList<byte[]>> streams, FileSchema schema, IReadOnlySet<uint> wanted,
-        IReadOnlyDictionary<Guid, SchemaField[]?>? entitySchemas = null, bool passUnknownEntities = false)
+        IReadOnlyDictionary<Guid, SchemaField[]?>? entitySchemas = null, bool passUnknownEntities = false,
+        IReadOnlyList<PartitionFraming.Framed?>? framed = null)
     {
         var result = new Dictionary<uint, RebarBars>();
         var joists = new Dictionary<uint, JoistSystemData>();
@@ -220,15 +221,15 @@ internal static ReinforcementData Read(
                     foreach (var ((layer, index), bar) in ReadHeld(walk.Captures)) held.TryAdd((id, layer, index), bar);
             }
             else if (ReadMember(walker, walk, byQueue) is { } member) members.TryAdd(id, member);
-        });
+        }, framed);
         RebarBars? Solve(PendingBar bar) =>
             diameters.TryGetValue(bar.BarTypeId, out var barDiameters) && barKinds.TryGetValue(bar.BarTypeId, out var kind)
             && shapeDefinitions.TryGetValue(bar.ShapeId, out var definition) ? SolveShapeDriven(bar, definition, hookKinds, kind, barDiameters.Model) : null;
         foreach (var (id, bar) in pending)
             if (Solve(bar) is { } solved) result.TryAdd(id, solved);
         // the items of a container that can be solved; their bar types come with the pass, like those of the other bars. They are in the
         // coordinates of the host instance, so a container whose host has no frame is not written.
-        var hostFrames = ReadHostFrames(walker, streams, schema, containers.Values.Select(c => (uint)c.HostId).ToHashSet());
+        var hostFrames = ReadHostFrames(walker, streams, framed, schema, containers.Values.Select(c => (uint)c.HostId).ToHashSet());
         foreach (var (id, (typeId, hostId, inAssembly, items)) in containers)
         {
             if (!hostFrames.TryGetValue((uint)hostId, out var hostFrame)) continue;
@@ -299,7 +300,8 @@ private static (int TypeId, int HostId, bool InAssembly, List<PendingBar> Items)
     /// <summary>The frame (row-major 3x3 of the columns x direction (m_RefDir), its cross product with the axis, axis, then the origin: 12 numbers) of
     /// each of the FamilyInstance elements <paramref name="hosts"/> (FamilyInstance.m_instOrigin, m_RefDir and m_zAxis); a host that is no
     /// FamilyInstance or whose values are not a frame has none.</summary>
-    private static Dictionary<uint, double[]> ReadHostFrames(WideBlockWalker walker, IReadOnlyList<IReadOnlyList<byte[]>> streams, FileSchema schema, IReadOnlySet<uint> hosts)
+    private static Dictionary<uint, double[]> ReadHostFrames(WideBlockWalker walker, IReadOnlyList<IReadOnlyList<byte[]>> streams,
+        IReadOnlyList<PartitionFraming.Framed?>? framed, FileSchema schema, IReadOnlySet<uint> hosts)
     {
         var frames = new Dictionary<uint, double[]>();
         if (hosts.Count == 0) return frames;
@@ -320,7 +322,7 @@ private static Dictionary<uint, double[]> ReadHostFrames(WideBlockWalker walker,
             if (Math.Abs(Dot(x, x) - 1) > 1e-9 || Math.Abs(Dot(z, z) - 1) > 1e-9 || Math.Abs(Dot(x, z)) > 1e-9) return;
             double[] y = [z[1] * x[2] - z[2] * x[1], z[2] * x[0] - z[0] * x[2], z[0] * x[1] - z[1] * x[0]];
             frames.TryAdd(id, [x[0], y[0], z[0], x[1], y[1], z[1], x[2], y[2], z[2], origin[0], origin[1], origin[2]]);
-        });
+        }, framed);
         return frames;
     }
 

src/RvtIO/Parsing/RecordRouting.cs

@@ -120,6 +120,14 @@ internal static Dictionary<ulong, int> Current(Census census, IReadOnlyList<Elem
         return ranges;
     }
 
+    /// <summary>The records of the framing of one partition stream without those the given ranges (<see cref="Stale"/>) mark as not
+    /// current: both walks of the stream place an item at the same offset.</summary>
+    internal static void Drop(List<PartitionFraming.ProjectItem> items, IReadOnlyList<(long Start, int Length)> stale)
+    {
+        var staleAt = stale.Select(s => s.Start).ToHashSet();
+        items.RemoveAll(i => staleAt.Contains(i.Offset));
+    }
+
     /// <summary>The chunks of one partition stream without the given byte ranges of its inflated stream. Null when the chunks are
     /// not the segments of the framing (same count, same lengths), so a range could not be placed exactly.</summary>
     internal static List<byte[]>? Prune(IReadOnlyList<byte[]> chunks, IReadOnlyList<int> segmentLengths, IReadOnlyList<(long Start, int Length)> ranges)
@@ -133,6 +141,13 @@ internal static Dictionary<ulong, int> Current(Census census, IReadOnlyList<Elem
         foreach (var chunk in chunks)
         {
             long chunkEnd = chunkStart + chunk.Length;
+            // a chunk no range reaches is kept as it is (the framing of the stream holds on to it as well)
+            if (next >= ranges.Count || ranges[next].Start >= chunkEnd)
+            {
+                result.Add(chunk);
+                chunkStart = chunkEnd;
+                continue;
+            }
             var kept = new MemoryStream(chunk.Length);
             long position = chunkStart;
             while (next < ranges.Count && ranges[next].Start < chunkEnd)