Walk every record of the partition streams to its last byte: long texts and the entity schemas of family documents

CHANGELOG.md

@@ -56,6 +56,7 @@ API between minor releases.
 
 ### Changed
 
+- Field walk by the class schema of the file: a text is read as long as its count says, where the walk stopped at a text longer than 100,000 characters; a text longer than the rest of the record stops it as any read past the end does. `rvtio coverage` also reads an extensible storage entity in the block of a loaded family by the schemas of that family's own document: up to release 2024 a family keeps them in its `ADocument` in `Global/ContentDocuments` (one document per content block, under the key of the block), from release 2025 `Global/Latest` holds them. On 52 project files of the releases 2020 to 2027 every record of every partition stream is now walked to its last byte (29,794,710 records; 117 were not: 45 at a long text, 72 at an entity of a family document); no class row has fewer exact records. Output: of ten files that hold such a text, eight family types of lighting fixtures in two MEP files get their parameter list in the compact JSON, and in three files of one MEP project one view type and one sun study change the reason they are left out of the IFC for; the IFC of the ten files does not change.
 - Default output, IFC and JSON: the readers behind the output read the values of an extensible storage entity by the schema of that entity, as `rvtio coverage` does, so the objects behind those values are read in their place instead of out of step, and the record of an element that keeps an entity is read to its end. An entity whose schema the file does not hold, or with a field type that is not read, is passed as before. The parameters that a store behind an entity would list as unset are still left out, as before; what that rule would give now is not measured. The bars of a rebar system that holds an entity are read now as well. A RebarInSystem that is the one RebarInSystem of its system takes the `GlobalId` of the system, as in the reference; the bar of a system with more keeps its own sub-element `GlobalId` (the number of its bars plus 3). Measured on 28 project files of the releases 2020 to 2026 against the output of the main branch, and for the IFC also against the reference: no IFC value became worse. On eight MEP files the in/out direction and the name of 887 distribution ports are now those of the reference (331, 331, 83, 83, 25, 25, 8 and 1), and on ten MEP files 3,221 assignments of a port or an element to a system are added that the reference has; 48 added assignments are not in the reference at that system (in four files; not checked in Revit). 6 systems the reference has are written that were missing; 26 pipe fitting types that the reference does not have are no longer written, and 9 others it does not have either are (types of a single fitting each, whose `GlobalId` changes with the parameters now read). In three files of one architectural project (releases 2021, 2023 and 2024) 42, 48 and 48 reinforcing bars that were left out are written, all with the `GlobalId` of the reference (in the file of release 2024, 26 of the 42 bars of its systems are the one bar of their system); the three files now write 159, 168 and 168 of the 169 bars of the reference (117, 120 and 120 before). The one bar of a system is not put into a group of its system, which it stands for. In the compact JSON 1,214 family instances of that project (three files) and 3,086 of another (one file) get the parameter list that the other instances of their categories already had, and 53 family types of one MEP file get theirs. Against Revit, on 172 of the 1,214 in the file of release 2021, every value the compact JSON writes as Revit shows it (type, mark, comments, phases, heights, levels, offsets and others, 1,561 values) is equal, and the fields the compact JSON writes in its own form (family, type and host as names, the design option as `None`, justifications as numbers, an empty level as `None`) are written as for 170 instances of the same categories that had their list before. The IFC of 16 of the 28 files and the JSON of 23 do not change.
 - The repository moved to the IOExtend organization. The project and repository URLs of the packages, the links in the documentation and the address of the documentation site (<https://ioextend.github.io/RvtIO/>) name the new place. The old repository address redirects to the new one; the old address of the documentation site does not.
 - `rvtio coverage` reads the values of an extensible storage entity by the schema of that entity. `ESEntity.m_blob` is an owned pointer whose class tag is not written: a handle of 0 is no entity and nothing follows; behind 0xFFFFFFFF stands the 16-byte GUID of the entity's schema where the class tag would be, and the values are queued like any owned object, so they lie behind every object queued earlier and in front of the objects queued later. The values are the fields of that schema in the order of `ESField.m_entryIndex` (the schema lists them by name), each one value, a count and the values of an array, or a count and the pairs of a map, of the type `ESField.m_fieldTypeName` names: a text, a scalar, or a class of the class schema read inline, among them an entity, which queues its own values. The entity schemas are read from `Global/Latest`, an `ADocument` that the same walk reads: `ESSchemaStorage.m_storedSchemas` in the class schemas up to release 2024, `ESSchemaStorage.m_schemaUsageMap` from release 2025. Before, the walk passed an entity as one without values, read the objects behind the values out of step and named the stop after whatever class came next. On 52 project files of the releases 2020 to 2027 the records that are not exact go from 27,847 to 117 of 29,794,710, and every `FamilyInstance` record of the project blocks is exact (24,044 of 131,261 were not); 154 class rows have more exact records, in files of every release but 2026, whose one file gains nothing, and no class row of any channel of any file has fewer. Left are 72 records in the documents of loaded families whose entity names a schema that `Global/Latest` of the project does not hold (the new reason `entity schema not stored`) and 45 records that stop at the 100,000 character limit of a text (4 of them inside the values of an entity). When the entity schemas of a file cannot be read, every entity with values stops its record with `entity schema not stored` instead of being passed; a handle other than 0 and 0xFFFFFFFF stops with `entity handle`, and a schema with a field the walk does not read with `entity field type` (neither occurs in the 52 files). Some kinds of values are read by the same grammar although no record of those files holds one (`short`, `int64_t`, a single `char`, arrays of `bool`, `float` and `GUIDvalue`, and maps that occur only empty). The readers of the export are not given the entity schemas and read an entity as before; their output was not compared. `rvtio coverage <file> --trace <class|owner,...>` is new: a developer aid that prints the walk of single records (the queued objects with their offsets, the fields of the class the walk stopped in, the bytes around the stop). See [the coverage report](docs/coverage-report.md).

docs/coverage-report.md

@@ -88,6 +88,16 @@ its result is used only when it ends without an error. When `Global/Latest` is m
 error, no entity schema is known, and every entity that has values stops the walk of its record with `entity schema not stored`. An
 entity is never passed by the look of the bytes around it.
 
+An entity in the block of a loaded family (a keyed block) can name a schema that only the family's own document holds. Up to release
+2024 that document stores its schemas itself: `Global/ContentDocuments` holds, per content document, a `ContentMarker` object (the
+handle `0xFFFFFFFF`, the `ContentKey` tag, a 4-byte count, then `m_guidKey` of the `ContentKey`, 16 bytes, which is the key of the
+keyed block; the same layout as a content block of a partition stream), a 4-byte length, the document's own `ADocument` of that length, and the length again; a `ContentMarker` with the handle `0`,
+its count and a length of `0` closes the stream at its last byte. Each of those `ADocument`s is read like `Global/Latest`. From
+release 2025 the documents of `Global/ContentDocuments` store no schema: `m_schemaUsageMap` of `Global/Latest` holds them, and
+`SchemaUsageInfo.m_contentDocsKeys` lists the keys of the content documents that use each one. So a record of a keyed block is read by
+the schemas of its own document and those of `Global/Latest`; a GUID names one schema, so one held in both is the same. A stream that
+does not close as described gives no family document a schema.
+
 A stop inside the values of an entity ends with `, in ESBlob`. The field is `ESBlob.<field>` when the stop is in a value itself: a
 name from the entity schema stored in the file, which the add-in that wrote the entity defined, not from the class schema. When the
 stop is inside a value of a class type, the field is that of the inline class (for example `second` of a pair class).
@@ -106,10 +116,9 @@ being read, and with `, in <Class>` when that field belongs to a queued object a
 | `end of payload` | A field needs more bytes than the body has left. |
 | `class tag N` | A pointer names class N, which the schema does not declare. |
 | `entity handle` | The handle of an extensible storage entity (`ESEntity.m_blob`) is neither `0` nor `0xFFFFFFFF`. |
-| `entity schema not stored` | An extensible storage entity names a schema that `Global/Latest` of the file does not hold. An entity in the document of a loaded family can name a schema that only that document uses. Also every entity with values of a file whose entity schemas cannot be read. |
+| `entity schema not stored` | An extensible storage entity names a schema that `Global/Latest` of the file does not hold, nor, for a record of a keyed block, the family's own document in `Global/ContentDocuments`. Also every entity with values of a file whose entity schemas cannot be read. |
 | `entity field type` | The schema of an extensible storage entity has a field the walk does not read: its type name is neither `TCHAR` (a text), one of the scalars `bool`, `char`, `short`, `int`, `float`, `double` and `int64_t`, nor a class of the class schema, or it is an array of texts, or its container type is not 0, 1 or 2. The same reason stands for a schema whose fields are not numbered from 0 without a gap or a repeat. |
 | `count` | The count of an array is above 5,000,000. The count itself is left out, so that all such records share one reason. |
-| `text length` | A text is longer than 100,000 characters. |
 | `type N` | The field has a type code the walk does not read. |
 | `template field` | A template field without exactly one argument of a fixed count. |
 | `depth` | Inline structures nest deeper than 40 levels. |
@@ -235,7 +244,8 @@ count uses, with the entity schemas of the file.
 ## What it does not measure
 
 - The content of the `Global`, `Formats` and plain streams. A paged stream is only inflated; what it holds (the element table, the
-  history, the class table itself) is not walked. `Global/Latest` is walked for the entity schemas only and has no row in the counts.
+  history, the class table itself) is not walked. `Global/Latest` and the documents of `Global/ContentDocuments` are walked for the
+  entity schemas only and have no row in the counts.
 - The meaning of fields. An exact record is fully assigned to the fields of its class; whether a reader then uses those fields is
   another question, as is whether an element is exported. The export report (`--report` of an export, see
   [the export report](export-report.md)) answers that one.

docs/test-fixtures.md

@@ -45,7 +45,7 @@ folder (`TestFiles.Scratch`); no test uses a fixed absolute path.
 
 ## What runs without the private files
 
-With no private file at all, 9673 of the 10091 test cases run and pass and 418 are skipped: 412 need private files (listed
+With no private file at all, 9680 of the 10098 test cases run and pass and 418 are skipped: 412 need private files (listed
 below; 17 of them only `bim_dump.txt` or `formats_latest_decompressed.bin`) and 6 need other inputs (two need symbolic links, three
 need `RVTIO_FREEFORM_FULL`, `RVTIO_CYL_FULL` or `RVTIO_MESH_FULL`, one is a known gap). The tests that run use synthetic data built
 in the tests and the small fixtures committed under `tests/RvtIO.Tests/Fixtures`. With the full private fixture folder the

src/RvtIO/Diagnostics/CoverageCensus.cs

@@ -89,9 +89,10 @@ internal static Result Run(RootStorage root, SchemaDocument document, int? saved
         var streams = new List<StreamRow>();
         var total = new Dictionary<(bool Family, int Channel, string Class), Tally>();
         var walker = failure == null ? new WideBlockWalker(document, (_, _) => null, []) : null;
-        // the values of an extensible storage entity are read by the entity schemas of the file; when those cannot be read the set
-        // is empty and an entity with values stops its record: the count never passes one by the bytes around it
-        walker?.UseEntitySchemas(EntitySchemaReader.Read(root, document));
+        // the values of an extensible storage entity are read by the entity schemas of the file, in a family block also by those of its own
+        // document; when those cannot be read the set is empty and an entity with values stops its record: the count never passes one
+        // by the bytes around it
+        var entitySchemas = walker == null ? null : new BlockEntitySchemas(root, document, walker);
         var adler = new Adler32();
         ulong terminator = width == 8 ? ulong.MaxValue : uint.MaxValue;
         int partitions = 0;
@@ -125,6 +126,7 @@ internal static Result Run(RootStorage root, SchemaDocument document, int? saved
             long records = 0;
             bool framed = PartitionFraming.Walk(raw, tags!.Value, width!.Value, record =>
             {
+                entitySchemas!.Use(record.Key);
                 if (Count(record, walker!, terminator, adler, local)) records++;
             }, out string? reason, out int blocks, segments);
             if (!framed)
@@ -154,6 +156,31 @@ [.. pair.Value.Stops.OrderByDescending(s => s.Value.Count).ThenBy(s => s.Key, St
         return new Result(savedRelease, schemaRelease, width, document.Classes.Count, streams, failure, Sum(groups.Select(g => g.Sums)), groups, classes);
     }
 
+    /// <summary>Gives the walk the entity schemas of the block a record is in: those of Global/Latest in the project's block; in a keyed
+    /// block those of the family's own document in Global/ContentDocuments (<see cref="EntitySchemaReader.ReadContentDocuments"/>; it
+    /// holds them up to Revit 2024) and those of Global/Latest (from Revit 2025 its m_schemaUsageMap holds them and lists the keys of the
+    /// content documents that use each one). A GUID names one schema, so a schema held in both is the same.</summary>
+    internal sealed class BlockEntitySchemas(RootStorage root, SchemaDocument document, WideBlockWalker walker)
+    {
+        private readonly Dictionary<Guid, SchemaField[]?> _project = EntitySchemaReader.Read(root, document);
+        private readonly Dictionary<string, Dictionary<Guid, SchemaField[]?>> _families = EntitySchemaReader.ReadContentDocuments(root, document);
+        private string? _key;
+
+        internal void Use(string key)
+        {
+            if (key == _key) return;
+            _key = key;
+            if (key.Length == 0 || !_families.TryGetValue(key, out var own) || own.Count == 0)
+            {
+                walker.UseEntitySchemas(_project);
+                return;
+            }
+            var both = new Dictionary<Guid, SchemaField[]?>(own);
+            foreach (var (guid, fields) in _project) both.TryAdd(guid, fields);
+            walker.UseEntitySchemas(both);
+        }
+    }
+
     private static Sums Sum(IEnumerable<Sums> parts)
     {
         var sum = new Sums();

src/RvtIO/Diagnostics/CoverageTrace.cs

@@ -29,7 +29,7 @@ internal static void Write(string path, Selection selection, TextWriter output)
             || !root.TryOpenStorage("Partitions", out var storage))
             throw new RvtInputException("truncated-or-damaged", "The records of this file cannot be counted.");
         var walker = new WideBlockWalker(document, (_, _) => null, []);
-        walker.UseEntitySchemas(EntitySchemaReader.Read(root, document));
+        var entitySchemas = new CoverageCensus.BlockEntitySchemas(root, document, walker);
         var classes = document.ById;
         int printed = 0;
         foreach (var entry in storage.EnumerateEntries().Where(e => e.Type == EntryType.Stream).OrderBy(e => e.Name.Length).ThenBy(e => e.Name, StringComparer.Ordinal))
@@ -42,6 +42,7 @@ internal static void Write(string path, Selection selection, TextWriter output)
                 if (printed >= selection.Max || record.Channel != selection.Channel || (record.Key.Length != 0) != selection.Family || record.Body.Length < 2) return;
                 int tag = BinaryPrimitives.ReadUInt16LittleEndian(record.Body.Span);
                 if (walker.Class(tag) is not { } rootClass) return;
+                entitySchemas.Use(record.Key);
                 if (selection.Owners != null ? !selection.Owners.Contains(record.Owner) : rootClass.Name != selection.Class) return;
                 if (!MemoryMarshal.TryGetArray(record.Body, out var bytes)) bytes = record.Body.ToArray();
                 walker.StepBudget = 16L * bytes.Count + 65536;