diff --git a/CHANGELOG.md b/CHANGELOG.md index 61148dc..87b09a2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -56,6 +56,7 @@ API between minor releases. ### Changed +- Reading: in a file without its own class schema (Revit 2020 and 2022) whose partition streams are read by their framing, the root object of each element, from which the reader takes the class of the element and the stair and railing facts it keeps per root, is the channel-102 record of the element in the project's own block, in its current copy. Before, the stream walk counted every block it met for the id, including copies of the id in the documents of loaded families, and missed roots it did not reach; an element with two counted roots or none had no class. On 23 such files 59 elements in 7 files now get their class (a schedule, a view, a wall, a rebar, a sketch plane and others): in the compact JSON they carry the name of their class and their usual properties instead of an empty name, and the export report names their class instead of `unknown`; the IFC of all 23 files is unchanged. Refs #47. - Reading: the element index of the files with their own class schema (the header category, design option and outline of each element, the class of its root object and the field walk of that object) and the reinforcement reader read a partition stream whose framing holds by its records: the channel-101 records of the project's own block are the element headers and its channel-102 records the root objects, one of each per element. Before, a scan of the stream took any byte run that looked like an item, including the records of the documents of loaded families, whose element ids repeat project ids; a value on which such a copy differed was dropped. On 29 files the scan found 5,987,630 header-like and 6,640,008 block-like items, the framing gives 848,325 of each. A stream whose framing does not hold is scanned as before. On 52 project files of the releases 2020 to 2027 the IFC and the export report are identical to the output of the main branch; in the compact JSON of one Revit 2021 file one element (the pen width table) now gets its name and properties, as in the other files. - Reading: a partition stream whose framing holds (content blocks, segments, gzip members with their CRC and length, record lengths, the owners of each block) is cut into its segments by that framing; the search for gzip members and its recovery of damaged members are kept for a stream whose framing does not hold, and the diagnostics say which way a stream was read. On 52 project files of the releases 2020 to 2027 the segments are the chunks the search found, byte for byte, in all 434 partition streams; the IFC, the compact JSON and the export report of 14 of them, one or more of every release, are identical to the output of the main branch. - Field walk by the class schema of the file: a text is read as long as its count says, where the walk stopped at a text longer than 100,000 characters; a text longer than the rest of the record stops it as any read past the end does. `rvtio coverage` also reads an extensible storage entity in the block of a loaded family by the schemas of that family's own document: up to release 2024 a family keeps them in its `ADocument` in `Global/ContentDocuments` (one document per content block, under the key of the block), from release 2025 `Global/Latest` holds them. On 52 project files of the releases 2020 to 2027 every record of every partition stream is now walked to its last byte (29,794,710 records; 117 were not: 45 at a long text, 72 at an entity of a family document); no class row has fewer exact records. Output: of ten files that hold such a text, eight family types of lighting fixtures in two MEP files get their parameter list in the compact JSON, and in three files of one MEP project one view type and one sun study change the reason they are left out of the IFC for; the IFC of the ten files does not change. diff --git a/docs/test-fixtures.md b/docs/test-fixtures.md index aecd9d3..7dc2511 100644 --- a/docs/test-fixtures.md +++ b/docs/test-fixtures.md @@ -45,7 +45,7 @@ folder (`TestFiles.Scratch`); no test uses a fixed absolute path. ## What runs without the private files -With no private file at all, 9685 of the 10103 test cases run and pass and 418 are skipped: 412 need private files (listed +With no private file at all, 9686 of the 10105 test cases run and pass and 419 are skipped: 413 need private files (listed below; 17 of them only `bim_dump.txt` or `formats_latest_decompressed.bin`) and 6 need other inputs (two need symbolic links, three need `RVTIO_FREEFORM_FULL`, `RVTIO_CYL_FULL` or `RVTIO_MESH_FULL`, one is a known gap). The tests that run use synthetic data built in the tests and the small fixtures committed under `tests/RvtIO.Tests/Fixtures`. With the full private fixture folder the @@ -55,7 +55,7 @@ Run with the private files: `RVTIO_TEST_FILES= dotnet test` (PowerShell: ## Tests that need the private files -281 test methods (413 cases, theory rows counted) are skipped without the files: +282 test methods (414 cases, theory rows counted) are skipped without the files: | Class | Methods | Cases | Files | |---|---|---|---| @@ -94,6 +94,7 @@ Run with the private files: `RVTIO_TEST_FILES= dotnet test` (PowerShell: | `IfcExporterPlainWallBodyCurrentTests` | 4 | 4 | TestData blocks | | `IfcExporterSkipReasonTests` | 1 | 1 | TESTFILE.rvt | | `IfcExporterSlabQuantitiesSampleTests` | 2 | 2 | rstbasicsampleproject.rvt, struct25.rvt | +| `OwnElementClassTests` | 1 | 1 | TESTFILE.rvt | | `PageCodecSampleTests` | 2 | 2 | any `.rvt` project files | | `PartitionFramingFuzzTests` | 1 | 1 | TESTFILE.rvt | | `ProfileRoofAreaTests` | 3 | 11 | TestData blocks | diff --git a/src/RvtIO/Internal/ParsingState.cs b/src/RvtIO/Internal/ParsingState.cs index a6467d3..ad0300b 100644 --- a/src/RvtIO/Internal/ParsingState.cs +++ b/src/RvtIO/Internal/ParsingState.cs @@ -113,6 +113,10 @@ public Ifc.IfcNumericUnitPlan? OwnDocumentIfcNumericUnitPlan /// and the stop would hide it. public bool StaleCopiesRemoved { get; set; } + /// Each partition stream read by its framing, or null where the framing does not hold (): + /// its segments and the records of the project's own block, without the copies record routing marks as not current. + public IReadOnlyList? FramedPartitions { get; set; } + /// The own name of an element of a file, read from its root block by the schema-driven walk. public Dictionary WideNames { get; } = new(); diff --git a/src/RvtIO/RvtFileReader.cs b/src/RvtIO/RvtFileReader.cs index d8567d4..098ae47 100644 --- a/src/RvtIO/RvtFileReader.cs +++ b/src/RvtIO/RvtFileReader.cs @@ -103,6 +103,7 @@ public RevitFileInfo Read(string rvtPath) ElemRecOwningElementIds = elemRecOwners, R22View = r22View, StaleCopiesRemoved = staleCopiesRemoved, + FramedPartitions = framedStreams, }; ReadWideElementIndex(root, state, partitionStreams, framedStreams, wideSchema, diag); ReadRebarBars(root, state, partitionStreams, framedStreams, wideSchema); @@ -870,6 +871,7 @@ internal static void MergeElemScanStringsInChunkOrder( ElemRecOwningElementIds = elemRecOwners, R22View = r22View, StaleCopiesRemoved = staleCopiesRemoved, + FramedPartitions = framedStreams, }; ReadWideElementIndex(root, state, partitionStreams, framedStreams, wideSchema, diag); ReadRebarBars(root, state, partitionStreams, framedStreams, wideSchema); @@ -2129,6 +2131,45 @@ internal static void AddDriveCurveFromRewrittenBlock(TfBlockDeserializer.NativeP if (key.StartsWith("87:m_pCrv.", StringComparison.Ordinal)) target.DoubleFields.TryAdd(key, value); } + /// Hands the channel-102 records of the project's own block, one root object per + /// element in its current copy, instead of every block the stream walk meets: the walk also meets the copies of an id in the documents + /// of loaded families and byte runs inside other items, and each one counted as a second root of the element. The location line of a + /// wall () is read from its root record once every root is counted. + internal static void RecordFramedRoots(ParsingState state) + { + var walls = new List<(byte[] Item, ushort Class, uint Owner)>(); + foreach (var frame in state.FramedPartitions!) + { + var starts = new long[frame!.Segments.Count]; + for (int i = 1; i < starts.Length; i++) starts[i] = starts[i - 1] + frame.Segments[i - 1].Length; + foreach (var record in frame.Items) + { + if (record.Channel != 102 || record.Length < 18 || record.Owner > uint.MaxValue) continue; + // the item [owner:4][checksum:4][size:4][class:2 ...][size:4] is the block layout the walk hands over + var item = ItemBytes(frame.Segments, starts, record.Offset, record.Length); + ushort classId = BitConverter.ToUInt16(item, 12); + StairComponentSource.Record(state, item, 0, (uint)(item.Length - 16), classId, (uint)record.Owner); + if (classId is 3524 or 361) walls.Add((item, classId, (uint)record.Owner)); + } + } + foreach (var (item, classId, owner) in walls) RecordWallLocationLine(state, item, 0, (uint)(item.Length - 16), classId, owner); + } + + /// The bytes at of the concatenated segments (an item can run into the next segment of its chain). + private static byte[] ItemBytes(IReadOnlyList segments, long[] starts, long offset, int length) + { + var item = new byte[length]; + int s = Array.BinarySearch(starts, offset); + if (s < 0) s = ~s - 1; + for (int done = 0; done < length; s++) + { + int from = (int)(offset + done - starts[s]), n = Math.Min(length - done, segments[s].Length - from); + Buffer.BlockCopy(segments[s], from, item, done, n); + done += n; + } + return item; + } + /// Continuous-stream anchored block walk (see TfStreamWalker): native param /// stores + full-coverage re-run of block-driven signals (curve owners, node owners, /// registry categories, sketch/detail curves). @@ -2143,6 +2184,9 @@ private static void RunTfStreamWalk(ParsingState state) foreach (var part in dumpGrepEnv.Split(',')) if (uint.TryParse(part, out uint dge)) dumpGrepEids.Add(dge); var walkGReps = new List<(byte[] Segment, int Start, uint Size, uint Eid)>(); + // the roots the class and stair sources count: the records of the project's block when every stream is read by its framing + bool framedRoots = state.WideSchema == null && state.FramedPartitions is { Count: > 0 } framed && framed.All(f => f is { IdWidth: 4 }); + if (framedRoots) RecordFramedRoots(state); TfStreamWalker.Walk(state.Chunks, state.AllKnownIds, (seg, bs, ds, cls, eid) => { uint fullClassIdx = BitConverter.ToUInt32(seg, bs + 12); @@ -2170,14 +2214,14 @@ private static void RunTfStreamWalk(ParsingState state) } } if (!state.ElemClassIdx.ContainsKey(eid)) state.ElemClassIdx[eid] = cls; - StairComponentSource.Record(state, seg, bs, (uint)ds, cls, eid); + if (!framedRoots) StairComponentSource.Record(state, seg, bs, (uint)ds, cls, eid); if (cls is 485 or 492) { var prior = state.RailingQuantityInputs.GetValueOrDefault(eid); state.RailingQuantityInputs[eid] = (prior.BlockCount + 1, TfBlockDeserializer.ExtractOwnRailingQuantityInput(seg, bs, (uint)ds, cls)); } - RecordWallLocationLine(state, seg, bs, (uint)ds, cls, eid); + if (!framedRoots) RecordWallLocationLine(state, seg, bs, (uint)ds, cls, eid); RecordCurtainPanelOrder(state, seg, bs, (uint)ds, cls, eid); if (TfBlockDeserializer.IsSubclassOf(cls, 1794)) RecordCurtainFamilyHost(state, seg, bs, ds, cls, eid); diff --git a/tests/RvtIO.Tests/OwnElementClassTests.cs b/tests/RvtIO.Tests/OwnElementClassTests.cs index a4b7a9b..4245016 100644 --- a/tests/RvtIO.Tests/OwnElementClassTests.cs +++ b/tests/RvtIO.Tests/OwnElementClassTests.cs @@ -1,3 +1,4 @@ +using RvtIO.Internal; using RvtIO.Parsing; using Xunit; @@ -94,6 +95,46 @@ public void ValidSubspan_DoesNotRequireTheOuterBufferEnd() Assert.Equal(123, owner); } + [Fact] + public void TheRootOfAnElement_IsItsRecordInTheProjectBlock_NotACopyOfItsIdInAFamilyDocument() + { + // the stream holds the element's root and, behind it, a block of the same id in the document of a loaded family, which the + // framing does not list as a record of the project + byte[] root = Frame(3708), copy = Frame(157); + byte[] stream = [.. root, .. copy]; + var state = new ParsingState + { + ElemRecOwningElementIds = new() { [123] = -1 }, + FramedPartitions = [new PartitionFraming.Framed([stream[..10], stream[10..]], [new(102, 123, 0, root.Length)], 4)], + }; + + RvtFileReader.RecordFramedRoots(state); + + Assert.Equal(new StairComponentSource.ElementClassRoot(1, 3708), state.OwnedElementClassRoots[123]); + Assert.Equal(new StairComponentSource.ElementClassRoot(1, 3708), state.WalkedElementClassRoots[123]); + // counted per block met, as the stream walk does, the copy makes the root a duplicate and its class unknown + var walked = new ParsingState { ElemRecOwningElementIds = new() { [123] = -1 } }; + StairComponentSource.Record(walked, stream, 0, Size(root), 3708, 123); + StairComponentSource.Record(walked, stream, root.Length, Size(copy), 157, 123); + Assert.Equal(new StairComponentSource.ElementClassRoot(2, null), walked.OwnedElementClassRoots[123]); + } + + // TESTFILE.rvt is saved in five increments; reading it with and without the removal of the superseded copies changes the records around + // the elements that one partition holds, but not their roots. + [SkippableFact] + public void TheClassOfAnElementHeldOnce_DoesNotDependOnTheRemovalOfOtherCopies() + { + string path = TestFiles.Find("TESTFILE.rvt"); + var (_, resolved) = new RvtFileReader(new RvtReaderOptions { ResolveCurrentCopies = true }).ReadWithState(path); + var (_, everyCopy) = new RvtFileReader(new RvtReaderOptions { ResolveCurrentCopies = false }).ReadWithState(path); + Assert.True(resolved.StaleCopiesRemoved); + var heldOnce = everyCopy.WalkedElementClassRoots.Where(r => r.Value.BlockCount == 1).ToList(); + Assert.True(heldOnce.Count > 8000); + Assert.All(heldOnce, r => Assert.Equal(r.Value, resolved.WalkedElementClassRoots[r.Key])); + // the root of this arc wall is a record of the project's block that the stream walk does not reach + Assert.Equal(new StairComponentSource.ElementClassRoot(1, 361), resolved.WalkedElementClassRoots[290112]); + } + private static uint Size(byte[] bytes) => BitConverter.ToUInt32(bytes, 8); private static void Reject(byte[] bytes, ushort cls)