Repository navigation
Conversation
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #17666 +/- ##
============================================
+ Coverage 46.03% 49.13% +3.09%
- Complexity 712 715 +3
============================================
Files 5509 5645 +136
Lines 400878 415809 +14931
Branches 52375 54909 +2534
============================================
+ Hits 184536 204287 +19751
+ Misses 216342 211522 -4820 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/consensus/statemachine/schemaregion/SchemaExecutionVisitor.java # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/consensus/statemachine/schemaregion/SchemaRegionStateMachine.java # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/protocol/thrift/async/IoTDBDataRegionAsyncSink.java # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/protocol/thrift/sync/IoTDBDataRegionSyncSink.java
|
|
Preserve pipe metric registrations across metric service restarts while removing operator state on deregistration. Keep master synchronization and prevent late events from recreating dropped pipe metrics.
|
|
@hongzhi-gao Fixed in 5f71bb0d0e6. The missing-committer path now uses a read-only Added regression coverage for single and multiple commit IDs with DEBUG enabled and disabled, plus stale/current committers across a task restart. The four late-commit cases fail on the previous implementation and pass with this fix. Validation: all 9 targeted node-commons tests pass, and full-reactor |
|
Review round of Commit Validation: clean reactor test for |
|
CI follow-up for the dependency-analysis failure.
No additional update is pending. |
Independent re-reviewReviewed SHA: Validation: Spotless passed. A clean 28-module reactor compiled, including all DataNode sources, but The PR remains draft pending a fresh independent review of the updated head. |
Independent re-reviewReviewed PR head
Added deterministic unit regressions for these lifecycle cases. No cluster IT was added because the internal metric-map ordering cannot be controlled or observed deterministically from a cluster test without adding production test hooks; the latch-based unit test exercises the exact race. Validation: Final local SHA: The PR remains draft because this round found behavioral/test-logic issues. Please perform a fresh independent re-review of the updated full diff. |
|
Independent re-review completed for reviewed head I inspected the full 22-file PR diff and related metadata, metric, event-callback, and cleanup paths, with particular attention to Two P2 behavior issues were found and fixed in
The previously fixed late metric-creation/deregistration race is closed in the reviewed paths: the exact pipe-instance liveness check and operator insertion occur under the same metrics monitor used by deregistration, with metadata removal before deregistration. I found no additional lock-order inversion, deadlock, or obvious performance regression in those paths. The deterministic tests exercise the relevant lifecycle boundaries; no IT was added because ordinary cluster activity cannot reliably force these precise callback orders without production test hooks. Validation:
Separate pre-existing follow-up, left unchanged: the remaining-event operator retains closed schema source instances in Final local commit, confirmed |
Independent re-reviewReviewed the full current diff at head Verification: Final commit and confirmed source branch/PR head: |
|
Round 2 re-review: Apache IoTDB PR #17666 Reviewed head: 06f675c This review found and fixed Pipe drop lifecycle races:
Validation: Spotless passed. The clean offline 23-module reactor completed, but its method-level selector ran zero tests for the parameterized PipeEventCommitManagerTest; the selector was corrected to run the full class, and all 10 tests passed, including the new race test under INFO and DEBUG. The previously reported PipeTaskAgentTest#testPipeDropClearsSharedRuntimeState and 37 offline targeted tests passed. forbiddenapis was skipped because the existing dependency archive is corrupt. Remote reactor validation was stopped because JitPack was unreachable and Apache snapshots timed out. PR CI was not awaited. Because this round found and fixed behavior issues, the PR remains draft for a fresh-session re-review. Do not repeat the full review in this session or mark the PR ready. |
|
Independent re-review: one complete round Reviewed head: Found and fixed one P2 behavior issue in The fix closes and clears the old queue when the snapshot has no listener file, removes an unreferenced listener and its metrics, and preserves a referenced listener in the closed state until its final local reference is released. Two deterministic regressions exercise the actual schema-region state-machine snapshot entry points for both reference-count cases, without production test hooks. Validation:
Limitations: verification ran locally after remote dependency downloads timed out during TLS handshakes. Final local commit: The PR remains draft because this round found a behavior issue. Passing the repaired tests does not make this an issue-free review; a fresh session must independently review the complete updated PR before it can be marked ready. |
|







Description
This PR fixes several potential memory/resource leaks in Pipe runtime lifecycle cleanup.
Previously, some passive paths could create or retain per-schema-region or per-pipe runtime state during leader notification, snapshot handling, queue cleanup, metric updates, pipe drop, and schema-region deletion. This could leave stale listeners, metrics, committers, rate limiters, aggregate runtime state, or floating memory counters after a pipe/schema region was closed, dropped, altered, or deleted.
Main changes:
Avoid creating schema-region pipe listeners from passive paths.
schemaListenerIfPresent(...)and use it in snapshot/load/write/delete and queue cleanup paths.SchemaRegionListeningQueueand its metrics.Cleanup stale per-pipe runtime state on pipe drop/discard.
PipeEventCommitManagercommitter and restart-time state when a pipe is dropped.Make floating memory accounting safe for late event releases.
pipeName_creationTimeinstead of relying only on livePipeMeta.Fix aggregate processor shared-state cleanup.
Tests
Added unit tests for:
PipeSchemaRegionListenerManagerTest: verifies leader state does not create listeners and unused closed listeners are cleaned up.PipeTaskAgentTest: verifies floating memory usage remains tracked after pipe drop until late release, and zero counters are cleaned after drop.This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
PipeSchemaRegionListenerManagerPipeDataNodeRuntimeAgentSchemaExecutionVisitorSchemaRegionStateMachineDataNodeRegionManagerPipeDataNodeTaskAgentPipeEventCommitManagerIoTDBSinkPipeDataNodeSinglePipeMetricsPipeTsFileToTabletsMetricsPipeTaskAgentAggregateProcessor