feat(load): reject managed TsFile directories - #18271
Conversation
| private static void validateNotLoadingInternalTsFile(final File file) | ||
| throws FileNotFoundException { | ||
| final Path sourcePath = canonicalPath(file); | ||
| for (final String dataDir : IoTDBDescriptor.getInstance().getConfig().getLocalDataDirs()) { |
There was a problem hiding this comment.
[P2] Hoist managed-directory canonicalization out of the recursive traversal
findAllTsFile calls this method for every file and directory it encounters, and every call re-canonicalizes both sequence and unsequence under every local data directory. Directory loading therefore performs 1 + 2 * dataDirCount canonical-path resolutions per entry, even though the managed roots are invariant during one traversal.
On Windows/JDK 17, traversing 5,000 TsFiles with this PR took about 3.24 s with 1 local data directory, 9.68 s with 4, and 37.31 s with 16, before LOAD analysis or execution began. This makes large directory LOADs increasingly expensive for the supported multi-directory configuration.
Please compute the canonical managed roots once at the start of processTsFile and pass that snapshot through the recursion, or cache them alongside the data-directory configuration.
0141c40 to
8baaef0
Compare
8baaef0 to
e777e04
Compare
| ? org.apache.iotdb.db.queryengine.plan.statement.crud.LoadTsFileStatement | ||
| .processTsFile(new File(filePath), validateSourcePath) | ||
| : org.apache.iotdb.db.queryengine.plan.statement.crud.LoadTsFileStatement |
There was a problem hiding this comment.
Thanks for the reminder. processTsFileForPipe intentionally bypasses the managed-directory check because Pipe receiver/hardlink sources can be placed under IoTDB-managed paths. We have restricted this branch to createForPipe() call sites guarded by an existing isGeneratedByPipe context; normal table-model LOAD still uses processTsFile(...) and keeps the managed-directory validation.
Description
as the title said
This PR has:
for an unfamiliar reader.
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR