Skip to content

Pipe: Fix potential resource leaks in pipe lifecycle cleanup - #17666

Draft
Caideyipi wants to merge 18 commits into
masterfrom
potential-leak
Draft

Caideyipi wants to merge 18 commits into
masterfrom
potential-leak

Conversation

@Caideyipi

@Caideyipi Caideyipi commented May 14, 2026 •

Copy link
Copy Markdown
Collaborator

Description

This PR fixes several potential memory/resource leaks in Pipe runtime lifecycle cleanup.

Previously, some passive paths could create or retain per-schema-region or per-pipe runtime state during leader notification, snapshot handling, queue cleanup, metric updates, pipe drop, and schema-region deletion. This could leave stale listeners, metrics, committers, rate limiters, aggregate runtime state, or floating memory counters after a pipe/schema region was closed, dropped, altered, or deleted.

Main changes:

  • Avoid creating schema-region pipe listeners from passive paths.

    • Add schemaListenerIfPresent(...) and use it in snapshot/load/write/delete and queue cleanup paths.
    • Keep schema-region leader-ready state separate from listener state, so leader notifications no longer instantiate SchemaRegionListeningQueue and its metrics.
    • Cleanup unused closed schema listeners and deregister listener metrics.
    • Clear listener and leader-ready state when deleting a schema region.
  • Cleanup stale per-pipe runtime state on pipe drop/discard.

    • Clear PipeEventCommitManager committer and restart-time state when a pipe is dropped.
    • Remove endpoint rate limiter entries when data-region sinks discard events of a dropped pipe.
    • Ensure pipe metrics deregistration removes internal maps/gauges/counters, including when the metric service is not bound.
    • Avoid recreating remaining-event metric operators for stale pipe instances during metric updates.
  • Make floating memory accounting safe for late event releases.

    • Track floating memory by pipeName_creationTime instead of relying only on live PipeMeta.
    • Keep counters after pipe drop/alter while late releases may still arrive.
    • Cleanup counters only after usage returns to zero and the matching pipe instance is no longer alive.
  • Fix aggregate processor shared-state cleanup.

    • Decrease per-pipe reference count safely.
    • Remove and clear per-timeseries runtime state only when the last processor reference closes.

Tests

Added unit tests for:

  • PipeSchemaRegionListenerManagerTest: verifies leader state does not create listeners and unused closed listeners are cleaned up.
  • PipeTaskAgentTest: verifies floating memory usage remains tracked after pipe drop until late release, and zero counters are cleaned after drop.

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the why and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • PipeSchemaRegionListenerManager
  • PipeDataNodeRuntimeAgent
  • SchemaExecutionVisitor
  • SchemaRegionStateMachine
  • DataNodeRegionManager
  • PipeDataNodeTaskAgent
  • PipeEventCommitManager
  • IoTDBSink
  • PipeDataNodeSinglePipeMetrics
  • PipeTsFileToTabletsMetrics
  • PipeTaskAgent
  • AggregateProcessor

@sonarqubecloud

Copy link
Copy Markdown

@codecov

codecov Bot commented May 14, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.10117% with 46 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.13%. Comparing base (f8135bb) to head (2654d8b).
⚠️ Report is 265 commits behind head on master.

Files with missing lines Patch % Lines
...machine/schemaregion/SchemaRegionStateMachine.java 45.00% 11 Missing ⚠️
...temachine/schemaregion/SchemaExecutionVisitor.java 0.00% 9 Missing ⚠️
...b/pipe/processor/aggregate/AggregateProcessor.java 0.00% 6 Missing ⚠️
...pe/metric/overview/PipeTsFileToTabletsMetrics.java 54.54% 5 Missing ⚠️
...metric/overview/PipeDataNodeSinglePipeMetrics.java 95.58% 3 Missing ⚠️
...pe/agent/task/progress/PipeEventCommitManager.java 90.00% 3 Missing ⚠️
...b/pipe/agent/runtime/PipeDataNodeRuntimeAgent.java 66.66% 2 Missing ⚠️
...e/iotdb/commons/pipe/agent/task/PipeTaskAgent.java 94.87% 2 Missing ⚠️
...agent/runtime/PipeSchemaRegionListenerManager.java 97.14% 1 Missing ⚠️
...otdb/db/pipe/agent/task/PipeDataNodeTaskAgent.java 83.33% 1 Missing ⚠️
... and 3 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #17666      +/-   ##
============================================
+ Coverage     46.03%   49.13%   +3.09%     
- Complexity      712      715       +3     
============================================
  Files          5509     5645     +136     
  Lines        400878   415809   +14931     
  Branches      52375    54909    +2534     
============================================
+ Hits         184536   204287   +19751     
+ Misses       216342   211522    -4820     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Caideyipi Caideyipi changed the title Pipe: Fixed multiple potential memory leaks Pipe: Fix potential resource leaks in pipe lifecycle cleanup Jun 8, 2026
# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/consensus/statemachine/schemaregion/SchemaExecutionVisitor.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/consensus/statemachine/schemaregion/SchemaRegionStateMachine.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/protocol/thrift/async/IoTDBDataRegionAsyncSink.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/protocol/thrift/sync/IoTDBDataRegionSyncSink.java
@sonarqubecloud

Copy link
Copy Markdown

@sonarqubecloud

sonarqubecloud Bot commented Aug 7, 2026

Copy link
Copy Markdown

Preserve pipe metric registrations across metric service restarts while removing operator state on deregistration. Keep master synchronization and prevent late events from recreating dropped pipe metrics.
@hongzhi-gao

Copy link
Copy Markdown
Contributor
  1. iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/agent/task/progress/PipeEventCommitManager.java:80
    • Trigger: An event with a valid commit ID commits after clear(pipeName, creationTime) finishes during pipe drop.
    • Potential issue: clear removes the committer and restart-time entries, but commitSingleId recreates the restart-time entry with computeIfAbsent when the committer is missing, even when debug logging is disabled. A late commit therefore restores state for the dropped pipe, with no subsequent cleanup shown. The new cleanup does not prevent the lifecycle leak it targets.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

@hongzhi-gao Fixed in 5f71bb0d0e6. The missing-committer path now uses a read-only getOrDefault inside the DEBUG logging guard. Late commits after clear(pipeName, creationTime) therefore no longer recreate restart-time entries.

Added regression coverage for single and multiple commit IDs with DEBUG enabled and disabled, plus stale/current committers across a task restart. The four late-commit cases fail on the previous implementation and pass with this fix.

Validation: all 9 targeted node-commons tests pass, and full-reactor test-compile -DskipTests passes with both the default locale and with-zh-locale.

@Caideyipi
Caideyipi marked this pull request as draft November 13, 2026 02:37
@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Review round of 5f71bb0d0e6 found one remaining lifecycle leak: SHOW PIPE can fall back to DataNode metrics after receiving a pipe row from ConfigNode. If the pipe is dropped between those steps, getRemainingEventAndTime() previously used computeIfAbsent and recreated a per-pipe operator after deregistration.

Commit df8d30d4215 changes the query to read existing operator state only and return zero when absent. I added the stale-query assertion to testSinglePipeMetricsIgnoreLateEventsAfterDeregister.

Validation: clean reactor test for PipeMetricsRestartTest passed (28 modules, 11 tests). Checks on the updated head were in progress when reviewed; CI was not awaited.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

CI follow-up for the dependency-analysis failure.

  • Finding: PipeEventCommitManagerTest imports ch.qos.logback.classic, but node-commons did not declare logback-classic directly.
  • Fix: Added logback-classic to iotdb-core/node-commons/pom.xml with test scope; the version remains managed by the root POM.
  • Review: Commit 33e41a31cf344bdba5be549069a446071138258b; the source branch and PR head both match this SHA.
  • Validation: TsFile source commit 8d487b753c441aa14c61dffed7d7709001f3039c built successfully. The offline IoTDB reactor check mvn -T 2 -B -ntp -o -Ddevelocity.off=true -P with-integration-tests -pl iotdb-core/node-commons -am clean test-compile dependency:analyze-only passed.

No additional update is pending.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Independent re-review

Reviewed SHA: df8d30d4215d9d662831801925a982c023c74c93. I found one remaining lifecycle race: an event update could pass the Pipe-alive check, then insert a remaining-event metric operator after drop had removed the Pipe and deregistered its metrics. Commit d7e2c2e00e9fc249a9036848ade31fa43f10be8f serializes the missing-operator creation path with deregistration.

Validation: Spotless passed. A clean 28-module reactor compiled, including all DataNode sources, but forbiddenapis failed while reading a dependency archive (ZipFile invalid LOC header) before tests ran, so the updated test result is unverified. On the preceding SHA, PipeMetricsRestartTest passed all 11 tests. CI for the final SHA was pending when checked and was not awaited.

The PR remains draft pending a fresh independent review of the updated head.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Independent re-review

Reviewed PR head 33e41a31cf344bdba5be549069a446071138258b against target base 673434b3335f68f0e4ef00de7323472a3121e136, focusing on d7e2c2e00e9fc249a9036848ade31fa43f10be8f. This round found and fixed behavioral and test-logic issues:

  • A late metric update could pass the liveness check and recreate its operator after deregistration. Creation now rechecks the exact pipe instance while serialized with deregistration; a deterministic concurrent regression test covers metadata deletion and deregistration interleaving.
  • Endpoint rate limiter state is now cleared only when the whole pipe is dropped.
  • Floating-memory counter updates and cleanup are serialized per pipe instance so a concurrent drop cannot lose a late increment.
  • Late event enrichment no longer recreates committer restart state after pipe cleanup.

Added deterministic unit regressions for these lifecycle cases. No cluster IT was added because the internal metric-map ordering cannot be controlled or observed deterministically from a cluster test without adding production test hooks; the latch-based unit test exercises the exact race.

Validation: mvn spotless:apply -P with-integration-tests passed. The focused offline clean reactor test run passed all 31 tests (12 node-commons and 19 DataNode) with -Dforbiddenapis.skip=true. A separate clean 28-module compile reached DataNode sources but forbiddenapis failed before tests with ZipFile invalid LOC header in a dependency archive. A remote rerun could not reach Maven Central/JitPack. The final GitHub check snapshot had 39 checks: 38 queued and 1 in progress; CI was not awaited.

Final local SHA: 6f2cc670214385796d95b94d27db34384b78534e.
Confirmed source branch apache/iotdb:potential-leak SHA and PR head SHA: 6f2cc670214385796d95b94d27db34384b78534e.

The PR remains draft because this round found behavioral/test-logic issues. Please perform a fresh independent re-review of the updated full diff.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Independent re-review completed for reviewed head 6f2cc670214385796d95b94d27db34384b78534e.

I inspected the full 22-file PR diff and related metadata, metric, event-callback, and cleanup paths, with particular attention to d7e2c2e00e9fc249a9036848ade31fa43f10be8f and the subsequent fixes. This session contains one complete review round.

Two P2 behavior issues were found and fixed in ca5d222e2265078b9fcd0ffc48b23b4ea308b8c9:

  • Late floating-memory acquisition was omitted while its release was still charged. An in-flight shared assigner can retain a matched source after its pipe metadata is removed. The liveness guard in PipeTaskAgent.addFloatingMemoryUsageInByte skipped that acquisition, but the corresponding release could subtract from outstanding usage, undercount it, and leave a negative/stale counter. Every actual acquisition is now recorded, including after drop; the existing per-key cleanup removes the counter after the matching releases reach zero. Two deterministic regressions cover both an outstanding counter and a counter that was already cleaned up.
  • A passive schema metric callback could recreate a removed listener. IoTDBNonDataRegionSource.getUnTransferredEventCount() used the creating queue accessor, so a callback after schema-region cleanup could reinsert listener state. The schema source now reads only an existing listener, returns zero after closure, and computes from one captured queue. A deterministic regression covers metric reads after region cleanup and after source closure.

The previously fixed late metric-creation/deregistration race is closed in the reviewed paths: the exact pipe-instance liveness check and operator insertion occur under the same metrics monitor used by deregistration, with metadata removal before deregistration. I found no additional lock-order inversion, deadlock, or obvious performance regression in those paths. The deterministic tests exercise the relevant lifecycle boundaries; no IT was added because ordinary cluster activity cannot reliably force these precise callback orders without production test hooks.

Validation:

  • Spotless apply passed for both affected modules with the integration-test profile.
  • The offline clean 28-module reactor completed successfully with all 482 goals executed and the build cache disabled. All 37 selected unit tests passed, with zero failures, errors, or skips. All three newly added regressions ran locally. Required source dependencies were rebuilt before the reactor run.
  • forbiddenapis was skipped using -Dforbiddenapis.skip=true because of the previously reported corrupt dependency archive; that check is not claimed as passing. The previously reported remote Maven Central/JitPack access failure remains a verification limitation. Remote CI was not awaited and is not claimed as passing.

Separate pre-existing follow-up, left unchanged: the remaining-event operator retains closed schema source instances in schemaRegionSources until the entire pipe is deregistered. Repeated leader changes can therefore retain old source objects. This round prevents their late metric reads from recreating listeners.

Final local commit, confirmed apache/iotdb:potential-leak head, and confirmed PR head all match ca5d222e2265078b9fcd0ffc48b23b4ea308b8c9. The PR remains draft because this round found behavior issues; the updated full diff requires a fresh independent review before marking ready. No second review round was run in this session.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Independent re-review

Reviewed the full current diff at head ca5d222e2265078b9fcd0ffc48b23b4ea308b8c9 against master. Follow-up finding: the previous whole-pipe cleanup was still limited to PipeDataNodeTaskAgent overrides. ConfigNode drops use the shared PipeTaskAgent.dropPipe overloads, so they could retain event committer and endpoint rate-limiter entries. Cleanup now runs from both shared overloads, and the DataNode duplicates were removed (common drop path, cleanup helper, DataNode path). The added unit regression covers both drop overloads and checks the shared maps (test).

Verification: mvn spotless:apply -P with-integration-tests passed. The offline clean reactor passed with the new regression test (1 test, 0 failures/errors/skips, 23 reactor modules); -Dforbiddenapis.skip=true was used because of the known damaged dependency archive. The remote attempt was stopped after the build host could not reach JitPack and timed out against Apache snapshots for the TsFile SNAPSHOT. No cluster IT was added because this shared cleanup is deterministic to cover in a unit test without production test hooks. PR CI was not awaited.

Final commit and confirmed source branch/PR head: 06f675ca98f7daa36d666dae74abf4e31cf6f6f1. The PR remains draft because this round found a behavior issue and requires a fresh independent re-review.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Round 2 re-review: Apache IoTDB PR #17666

Reviewed head: 06f675c
Updated head: bd13b0c
PR state: OPEN draft; base master; head potential-leak

This review found and fixed Pipe drop lifecycle races:

  • PipeTaskAgent.cleanupFloatingMemoryUsageCounterIfUnused now removes unused floating-memory accounting state without scanning Pipe metadata on event release. Tests cover balanced acquisition and release after drop.
  • Drop-aware liveness prevents late metric updates and endpoint rate-limiter creation from recreating per-Pipe state after deregistration.
  • A late PipeEventCommitManager.isCurrentCommitterKey lookup could repopulate restart-generation state through computeIfAbsent after concurrent clear. It now performs a read-only restart lookup and verifies committer identity. A latch-controlled test covers the clear/lookup interleaving.

Validation: Spotless passed. The clean offline 23-module reactor completed, but its method-level selector ran zero tests for the parameterized PipeEventCommitManagerTest; the selector was corrected to run the full class, and all 10 tests passed, including the new race test under INFO and DEBUG. The previously reported PipeTaskAgentTest#testPipeDropClearsSharedRuntimeState and 37 offline targeted tests passed. forbiddenapis was skipped because the existing dependency archive is corrupt. Remote reactor validation was stopped because JitPack was unreachable and Apache snapshots timed out. PR CI was not awaited.

Because this round found and fixed behavior issues, the PR remains draft for a fresh-session re-review. Do not repeat the full review in this session or mark the PR ready.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Independent re-review: one complete round

Reviewed head: bd13b0cd21b431224daa4d97b9b3eaa9032f2ece. I inspected the full current 25-file PR diff against master and relevant context, including d7e2c2e00e9fc249a9036848ade31fa43f10be8f, the earlier fixes, and the shared PipeTaskAgent.dropPipe cleanup. The review covered late memory acquisition/release, passive schema metrics, late metric updates versus deregistration, metadata registration/removal, callbacks, lock ordering, deadlock/performance risks, and deterministic concurrency tests.

Found and fixed one P2 behavior issue in listenerForSnapshotLoad: installing a snapshot without pipe_schema_region_listening_queue.bin left a lagging follower's existing schema listener queue open and retained its old events and metrics. A follower can reach this state when it has not applied the queue-close log entry before snapshot installation. The previous load path returned the existing listener, and deserialization returned immediately for the missing file.

The fix closes and clears the old queue when the snapshot has no listener file, removes an unreferenced listener and its metrics, and preserves a referenced listener in the closed state until its final local reference is released. Two deterministic regressions exercise the actual schema-region state-machine snapshot entry points for both reference-count cases, without production test hooks.

Validation:

  • Both new regressions failed before the production fix (2 failures, 0 errors, 0 skips).
  • After the fix, the focused lifecycle, metrics, committer, rate-limiter, schema-listener, and queue suite passed: 42 tests, 0 failures, 0 errors, 0 skips, using fresh JUnit reports with build caches disabled. All newly added tests were run locally.
  • The 28-module dependency closure was first compiled from clean source; the fixed test run reused its unchanged reactor dependencies and recompiled the changed DataNode sources. Chinese-locale clean test-compile of that closure: passed (460 goals executed; no test execution claimed for this compile check).
  • The required TsFile dependency was clean-built from public commit 8d487b753c441aa14c61dffed7d7709001f3039c (2.4.1-261009-SNAPSHOT); source and installed-artifact hashes were checked. Spotless and git diff --check passed.

Limitations: verification ran locally after remote dependency downloads timed out during TLS handshakes. forbiddenapis was skipped because of the previously reported damaged dependency archive. No cluster IT was added: reliably forcing a replica to miss the queue-close entry and install this particular snapshot would require additional test controls. CI was not awaited.

Final local commit: 2654d8b5776d842584e41fbdc29f9a3511a60972.
Confirmed remote branch and PR head: 2654d8b5776d842584e41fbdc29f9a3511a60972.

The PR remains draft because this round found a behavior issue. Passing the repaired tests does not make this an issue-free review; a fresh session must independently review the complete updated PR before it can be marked ready.

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants