Cut a partition stream by its exact framing when it holds

CHANGELOG.md

@@ -56,6 +56,7 @@ API between minor releases.
 
 ### Changed
 
+- Reading: a partition stream whose framing holds (content blocks, segments, gzip members with their CRC and length, record lengths, the owners of each block) is cut into its segments by that framing; the search for gzip members and its recovery of damaged members are kept for a stream whose framing does not hold, and the diagnostics say which way a stream was read. On 52 project files of the releases 2020 to 2027 the segments are the chunks the search found, byte for byte, in all 434 partition streams; the IFC, the compact JSON and the export report of 14 of them, one or more of every release, are identical to the output of the main branch.
 - Field walk by the class schema of the file: a text is read as long as its count says, where the walk stopped at a text longer than 100,000 characters; a text longer than the rest of the record stops it as any read past the end does. `rvtio coverage` also reads an extensible storage entity in the block of a loaded family by the schemas of that family's own document: up to release 2024 a family keeps them in its `ADocument` in `Global/ContentDocuments` (one document per content block, under the key of the block), from release 2025 `Global/Latest` holds them. On 52 project files of the releases 2020 to 2027 every record of every partition stream is now walked to its last byte (29,794,710 records; 117 were not: 45 at a long text, 72 at an entity of a family document); no class row has fewer exact records. Output: of ten files that hold such a text, eight family types of lighting fixtures in two MEP files get their parameter list in the compact JSON, and in three files of one MEP project one view type and one sun study change the reason they are left out of the IFC for; the IFC of the ten files does not change.
 - Default output, IFC and JSON: the readers behind the output read the values of an extensible storage entity by the schema of that entity, as `rvtio coverage` does, so the objects behind those values are read in their place instead of out of step, and the record of an element that keeps an entity is read to its end. An entity whose schema the file does not hold, or with a field type that is not read, is passed as before. The parameters that a store behind an entity would list as unset are still left out, as before; what that rule would give now is not measured. The bars of a rebar system that holds an entity are read now as well. A RebarInSystem that is the one RebarInSystem of its system takes the `GlobalId` of the system, as in the reference; the bar of a system with more keeps its own sub-element `GlobalId` (the number of its bars plus 3). Measured on 28 project files of the releases 2020 to 2026 against the output of the main branch, and for the IFC also against the reference: no IFC value became worse. On eight MEP files the in/out direction and the name of 887 distribution ports are now those of the reference (331, 331, 83, 83, 25, 25, 8 and 1), and on ten MEP files 3,221 assignments of a port or an element to a system are added that the reference has; 48 added assignments are not in the reference at that system (in four files; not checked in Revit). 6 systems the reference has are written that were missing; 26 pipe fitting types that the reference does not have are no longer written, and 9 others it does not have either are (types of a single fitting each, whose `GlobalId` changes with the parameters now read). In three files of one architectural project (releases 2021, 2023 and 2024) 42, 48 and 48 reinforcing bars that were left out are written, all with the `GlobalId` of the reference (in the file of release 2024, 26 of the 42 bars of its systems are the one bar of their system); the three files now write 159, 168 and 168 of the 169 bars of the reference (117, 120 and 120 before). The one bar of a system is not put into a group of its system, which it stands for. In the compact JSON 1,214 family instances of that project (three files) and 3,086 of another (one file) get the parameter list that the other instances of their categories already had, and 53 family types of one MEP file get theirs. Against Revit, on 172 of the 1,214 in the file of release 2021, every value the compact JSON writes as Revit shows it (type, mark, comments, phases, heights, levels, offsets and others, 1,561 values) is equal, and the fields the compact JSON writes in its own form (family, type and host as names, the design option as `None`, justifications as numbers, an empty level as `None`) are written as for 170 instances of the same categories that had their list before. The IFC of 16 of the 28 files and the JSON of 23 do not change.
 - The repository moved to the IOExtend organization. The project and repository URLs of the packages, the links in the documentation and the address of the documentation site (<https://ioextend.github.io/RvtIO/>) name the new place. The old repository address redirects to the new one; the old address of the documentation site does not.

docs/test-fixtures.md

@@ -45,7 +45,7 @@ folder (`TestFiles.Scratch`); no test uses a fixed absolute path.
 
 ## What runs without the private files
 
-With no private file at all, 9680 of the 10098 test cases run and pass and 418 are skipped: 412 need private files (listed
+With no private file at all, 9683 of the 10101 test cases run and pass and 418 are skipped: 412 need private files (listed
 below; 17 of them only `bim_dump.txt` or `formats_latest_decompressed.bin`) and 6 need other inputs (two need symbolic links, three
 need `RVTIO_FREEFORM_FULL`, `RVTIO_CYL_FULL` or `RVTIO_MESH_FULL`, one is a known gap). The tests that run use synthetic data built
 in the tests and the small fixtures committed under `tests/RvtIO.Tests/Fixtures`. With the full private fixture folder the

src/RvtIO/Parsing/PartitionFraming.cs

@@ -42,7 +42,7 @@ internal static class PartitionFraming
     /// <summary>Walks the stream; <paramref name="onRecord"/> sees every record of every block in file order. Returns false
     /// with the reason in <paramref name="failure"/> when the framing does not hold. <paramref name="idWidth"/> is 4 or 8.</summary>
     internal static bool Walk(byte[] raw, Tags tags, int idWidth, Action<Record> onRecord, out string? failure, out int blocks,
-        List<int>? segmentLengths = null)
+        List<int>? segmentLengths = null, List<byte[]>? segmentBytes = null)
     {
         failure = null;
         blocks = 0;
@@ -65,6 +65,7 @@ internal static bool Walk(byte[] raw, Tags tags, int idWidth, Action<Record> onR
                 if (expectedOwners == uint.MaxValue)
                 {
                     Require(pointer == 0 && blocks > 0, "terminal-marker");
+                    // what follows the terminal is no part of the stream (measured: 432 of 434 partition streams of 52 files have bytes after it)
                     return true;
                 }
                 Require((blocks != 0 || pointer == 0) && keys.Add(key), "map-key");
@@ -94,6 +95,7 @@ internal static bool Walk(byte[] raw, Tags tags, int idWidth, Action<Record> onR
                     int before = (int)chain.Length;
                     Inflate(payload, chain);
                     segmentLengths?.Add((int)chain.Length - before);
+                    segmentBytes?.Add(chain.GetBuffer().AsSpan(before, (int)chain.Length - before).ToArray());
                     countSum += count;
                     sizeSum += unc;
                     if (mode is not (0 or 1)) continue;

src/RvtIO/RvtFileReader.cs

@@ -1120,6 +1120,10 @@ private List<byte[]> DecompressPartitions(RootStorage root, TextWriter? diag,
         partitionStreams = new List<IReadOnlyList<byte[]>>();
         ownPartitionWallParts = null;
         var routing = _options.ResolveCurrentCopies && partEntries.Count > 1 ? PlanRecordRouting(root, partStor, partEntries.Select(e => e.Name).ToList(), diag) : null;
+        // the exact framing of the partition streams (PartitionFraming), when the class table names its marker classes and the id width is known
+        var framing = FormatsSchema.TryRead(root) is { } formats && PartitionFraming.TagsOf(formats) is { } framingTags
+            && SchemaRelease.IdWidth(SchemaRelease.Effective(SchemaRelease.Saved(BasicFileInfoReader.Read(root).GetValueOrDefault("RevitVersion")),
+                SchemaRelease.FromSchema(formats))) is int framingIdWidth ? (Tags: framingTags, IdWidth: framingIdWidth) : ((PartitionFraming.Tags Tags, int IdWidth)?)null;
 
         foreach (var partEntry in partEntries)
         {
@@ -1140,11 +1144,10 @@ private List<byte[]> DecompressPartitions(RootStorage root, TextWriter? diag,
                 diag?.WriteLine($"  Own partition wall parts: {ownPartitionWallParts?.Count.ToString() ?? "unavailable"} ({failure ?? "exact"})");
             }
 
-            var chunks = decompressor.DecompressAll(raw);
-            var (members, decodedMembers, failedMembers, recoveredMembers, unverifiedMembers) = decompressor.LastStats;
-            RecordStream("Partitions/" + partEntry.Name, raw.Length, members, decodedMembers,
-                failedMembers, recoveredMembers, unverifiedMembers);
-            diag?.WriteLine($"  Chunks: {chunks.Count} decompressed");
+            var chunks = CutPartition(raw, framing, decompressor, out var stats, out string? framingFailure);
+            RecordStream("Partitions/" + partEntry.Name, raw.Length, stats.Members, stats.Decoded, stats.Failed, stats.Recovered, stats.Unverified);
+            diag?.WriteLine(framingFailure == null ? $"  Chunks: {chunks.Count} segments of the exact framing"
+                : $"  Chunks: {chunks.Count} decompressed by a search for gzip members (framing: {framingFailure})");
             if (routing != null && int.TryParse(partEntry.Name, out int partitionNumber)
                 && RecordRouting.Stale(routing.Census, routing.Current, partitionNumber) is { Count: > 0 } stale)
             {
@@ -1166,6 +1169,24 @@ private List<byte[]> DecompressPartitions(RootStorage root, TextWriter? diag,
 
     private static Dictionary<uint, OwnPartitionParts.WallParts>? Fail(string reason, out string? failure) { failure = reason; return null; }
 
+    /// <summary>The chunks of a partition stream: when its framing holds (<see cref="PartitionFraming"/>), the output of each segment, every one a
+    /// gzip member checked by its CRC and length, and <paramref name="framingFailure"/> null; otherwise the members a search for gzip headers
+    /// finds, with the recovery of damaged ones (<see cref="ChunkDecompressor"/>), and the reason the framing does not hold.</summary>
+    internal static List<byte[]> CutPartition(byte[] raw, (PartitionFraming.Tags Tags, int IdWidth)? framing, ChunkDecompressor decompressor,
+        out (int Members, int Decoded, int Failed, int Recovered, int Unverified) stats, out string? framingFailure)
+    {
+        var segments = new List<byte[]>();
+        framingFailure = "the class table names no framing classes or the id width is not known";
+        if (framing is var (tags, idWidth) && PartitionFraming.Walk(raw, tags, idWidth, _ => { }, out framingFailure, out _, null, segments))
+        {
+            stats = (segments.Count, segments.Count, 0, 0, 0);
+            return segments;
+        }
+        var chunks = decompressor.DecompressAll(raw);
+        stats = decompressor.LastStats;
+        return chunks;
+    }
+
     private sealed record RoutingPlan(RecordRouting.Census Census, Dictionary<ulong, int> Current);
 
     /// <summary>The current copy of every element that several partition streams hold (<see cref="RecordRouting"/>), or null with the

tests/RvtIO.Tests/PartitionCutTests.cs

@@ -0,0 +1,106 @@
+using RvtIO.Parsing;
+using RvtIO.Writing;
+using Xunit;
+using static RvtIO.Tests.CoverageCensusTests;
+
+namespace RvtIO.Tests;
+
+/// <summary>How the reader cuts a partition stream into chunks (<see cref="RvtFileReader.CutPartition"/>): by its segments when the framing
+/// holds, else by a search for gzip members.</summary>
+public class PartitionCutTests
+{
+    private static (PartitionFraming.Tags Tags, int IdWidth) Framing => (PartitionFraming.TagsOf(new FormatsSchema(Schema))!.Value, 4);
+
+    private static byte[] TwoBlocks() => Partition(Block(false, R(1, Body("Plain", 5, 0, 0, 0))), Block(true, R(2, Body("Plain", 5, 0, 0, 0))));
+
+    [Fact]
+    public void AStreamWhoseFramingHolds_IsCutIntoItsSegments()
+    {
+        byte[] raw = TwoBlocks();
+
+        var chunks = RvtFileReader.CutPartition(raw, Framing, new ChunkDecompressor(), out var stats, out string? failure);
+
+        Assert.Null(failure);
+        Assert.Equal((6, 6, 0, 0, 0), stats);
+        Assert.Equal(new ChunkDecompressor().DecompressAll(raw), chunks);
+    }
+
+    [Fact]
+    public void AStreamWhoseFramingDoesNotHold_IsSearchedForGzipMembers()
+    {
+        byte[] raw = TwoBlocks();
+        raw[^10] ^= 0xFF;   // the class tag of the terminal marker: the framing does not end, the members are intact
+
+        var search = new ChunkDecompressor();
+        var chunks = RvtFileReader.CutPartition(raw, Framing, search, out var stats, out string? failure);
+
+        Assert.NotNull(failure);
+        Assert.Equal(search.LastStats, stats);
+        Assert.Equal(new ChunkDecompressor().DecompressAll(raw), chunks);
+        // without the framing classes the stream is searched as well, and the reason says so
+        RvtFileReader.CutPartition(TwoBlocks(), null, new ChunkDecompressor(), out _, out failure);
+        Assert.Equal("the class table names no framing classes or the id width is not known", failure);
+    }
+
+    [Fact]
+    public void TheSegmentsOfAChain_AreTheOutputOfEachOfItsMembers()
+    {
+        // one block whose channel 101 is a chain of two segments (the first continues into the second), channels 102 and 103 one segment each
+        byte[] first = Records(101, R(1, Body("Plain", 5, 0, 0, 0))), second = Records(101, R(2, Body("Plain", 5, 0, 0, 0)));
+        using var stream = new MemoryStream();
+        using (var w = new BinaryWriter(stream, System.Text.Encoding.UTF8, leaveOpen: true))
+        {
+            w.Write(9ul);
+            w.Write(Id("ContentMarker")); w.Write(0u); w.Write(2u);
+            Segment(w, 101, 2u, first, 1);
+            Segment(w, 101, 1u, second, 1);
+            Segment(w, 102, 0u, Records(102, R(1, Body("Plain", 5, 0, 0, 0)), R(2, Body("Plain", 5, 0, 0, 0))), 2);
+            Segment(w, 103, 0u, Records(103, R(1, Body("Plain", 5, 0, 0, 0)), R(2, Body("Plain", 5, 0, 0, 0))), 2);
+            w.Write(Id("SegmentMarker")); w.Write(new byte[16]);
+            w.Write(Id("SignatureMarker")); w.Write(0u); w.Write(0u);
+            w.Write(Id("ContentMarker")); w.Write(0u); w.Write(uint.MaxValue);
+        }
+        byte[] raw = stream.ToArray();
+
+        var chunks = RvtFileReader.CutPartition(raw, Framing, new ChunkDecompressor(), out _, out string? failure);
+
+        Assert.Null(failure);
+        Assert.Equal(4, chunks.Count);
+        Assert.Equal(first, chunks[0]);
+        Assert.Equal(second, chunks[1]);
+        Assert.Equal(new ChunkDecompressor().DecompressAll(raw), chunks);
+    }
+
+    // the items of records as a segment holds them inflated: owner, the checksum on channels 102 and 103, size, body, size
+    private static byte[] Records(uint channel, params ElementCensusTests.Rec[] records)
+    {
+        using var items = new MemoryStream();
+        using var item = new BinaryWriter(items);
+        foreach (var record in records)
+        {
+            item.Write(record.Owner);
+            if (channel != 101) item.Write(record.Extra);
+            item.Write((uint)record.Body.Length);
+            item.Write(record.Body);
+            item.Write((uint)record.Body.Length);
+        }
+        item.Flush();
+        return items.ToArray();
+    }
+
+    private static void Segment(BinaryWriter w, uint channel, uint flags, byte[] records, uint count)
+    {
+        byte[] payload = GzipMember.Write(records);
+        uint size = (uint)payload.Length + 8;
+        w.Write(Id("SegmentMarker"));
+        w.Write(flags);
+        w.Write(count);
+        w.Write(size);
+        w.Write((uint)(records.Length - count * (channel == 101 ? 12u : 16u)));
+        w.Write(channel);
+        w.Write(19u);
+        w.Write(payload);
+        w.Write(Id("SegmentCheckback"));
+        w.Write(size);
+    }
+}