[Pipe] Fix historical TsFile schema recovery race - #18314
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #18314 +/- ##
============================================
+ Coverage 43.10% 43.41% +0.31%
Complexity 374 374
============================================
Files 5362 5367 +5
Lines 381676 383057 +1381
Branches 49568 49827 +259
============================================
+ Hits 164504 166288 +1784
+ Misses 217172 216769 -403 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| private final boolean isAutoCreateSchemaAllowed; | ||
| private final boolean isAutoCreateSchema; |
There was a problem hiding this comment.
Too confusing, give better names.
There was a problem hiding this comment.
Renamed the two states in 4ee2fbe: isAutoCreateSchemaRequested is the statement/attribute request, while isAutoCreateSchemaEnabled is the effective local setting after applying the server config. The related accessors and tests now use the same distinction.
| default ISchemaTree fetchSchemaList( | ||
| final List<PartialPath> devicePaths, | ||
| final List<String[]> measurementsList, | ||
| final MPPQueryContext context) { | ||
| DataNodeSchemaLockManager.getInstance() | ||
| .takeReadLock(context, SchemaLockType.VALIDATE_VS_DELETION_TREE); | ||
| final PathPatternTree patternTree = new PathPatternTree(); | ||
| for (int i = 0; i < devicePaths.size(); i++) { | ||
| for (final String measurement : measurementsList.get(i)) { | ||
| patternTree.appendFullPath(devicePaths.get(i), measurement); | ||
| } | ||
| } | ||
| return fetchSchema(patternTree, true, context, true); | ||
| } |
There was a problem hiding this comment.
Where is the lock released?
Add a comment.
There was a problem hiding this comment.
Added the lifecycle comment in 4ee2fbe. The lock is recorded in MPPQueryContext and intentionally remains held through execution; Coordinator releases the recorded schema read locks from its finally block. The focused LoadTsFileAnalyzerTest suite passes (7 tests).
|
2 participants
Description
Problem and observed behavior
Historical Pipe events from ConfigRegion, SchemaRegion, and DataRegion are transferred independently. A historical TsFile can therefore reach the receiver before the schema and template events that describe it, making the result dependent on event arrival order.
The timing-dependent failure is as follows:
SHOW PATHS SET DEVICE TEMPLATE aligned_templatedoes not return the expectedroot.sg_aligned.device_aligned.aligned_templateonroot.sg_aligned.device_alignedfailed because ordinary timeseries had already been created under that path.The early TsFile load auto-creates ordinary timeseries. When the historical template-set event arrives later, it conflicts with those timeseries and the receiver cannot restore the original template binding.
Fix
This avoids imposing cross-region transfer ordering: data loading simply waits and retries until the schema history has been applied.
Compatibility
The new behavior is opt-in through an internal system parameter and defaults to disabled on the wire. Existing behavior is preserved for external sources, incomplete schema inclusion, general-write requests,
mark-as-pipe-request=false, legacy requests, and ordinaryLOAD TSFILEoperations. The existing default that permits per-load schema auto-creation also remains unchanged unless the receiver gets the new marker.Tests
IoTDBPipeAutoConflictITto assert thatroot.sg_aligned.device_alignedis bound toaligned_templateon the receiver.OK (76 tests)across 6 test classes.template-nodeartifact.This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
PipeDataNodeTaskBuilder: determines whether a Pipe has complete schema history and injects the internal wait marker.PipeTransferTsFileSealWithModReqand DataRegion sinks: propagate the marker across all V2 transfer paths.IoTDBDataNodeReceiver: configures marked sync and async loads to verify without auto-creating schema.LoadTsFileAnalyzer,TreeSchemaAutoCreatorAndVerifier, andSchemaValidator: support per-load schema auto-creation control and retryable missing-schema handling.