Skip to content

Fix table session pool lifecycle leaks - #19247

Open
shuwenwei wants to merge 9 commits into
masterfrom
fix/table-session-pool-lifecycle
Open

shuwenwei wants to merge 9 commits into
masterfrom
fix/table-session-pool-lifecycle

Conversation

@shuwenwei

@shuwenwei shuwenwei commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Problem

TableSessionPool.getSession() borrowed a raw Session from the underlying SessionPool, but did not register it in occupied. Consequently, pool.close() could not see active table sessions, and returning one after the pool closed could publish the connection back into the closed queue.

Example:

ITableSessionPool pool = new TableSessionPoolBuilder()
    .nodeUrls(nodeUrls)
    .user(user)
    .password(password)
    .maxSize(1)
    .build();

ITableSession session = pool.getSession();
pool.close();
session.close();

Before this change:

  • pool.close() did not close the borrowed table session.
  • session.close() returned it to an already-closed pool.
  • Closing the wrapper after a connection failure could throw an NPE.
  • A failed SQL dialect/database reset could permanently consume pool capacity.
  • pool.close() could hold the pool monitor while waiting for an in-flight RPC, blocking unrelated borrow and release operations.

Tree-model result queries had a related acquisition window. They called occupy(session) only after the query RPC completed. If pool.close() ran in between, the Session could be registered in a pool that had already completed cleanup.

Corrected behavior

  • Borrowed table sessions are tracked in occupied until their wrapper is closed.
  • Releasing into a closed pool closes the Session directly instead of putting it back into the queue.
  • pool.close() snapshots and clears pool state under the monitor, then closes connections outside the monitor.
  • Borrowed sessions are force-closed through their transport so pool shutdown does not wait behind an in-flight statement RPC.
  • Tree-model query completion checks closed before adding the Session to occupied; a concurrent pool close causes the Session to be closed instead.
  • Failed dialect/database resets and connection errors release the unusable Session and restore pool capacity with a replacement when possible.
  • Table-session operations remain serialized, and old tree-model result-set wrappers cannot release a Session that has since been re-borrowed.
  • sortTablet() performs its in-memory sort without borrowing a pooled Session.
  • Reusing a table-session wrapper after a connection failure now raises IoTDBConnectionException instead of an NPE.

Changes

  • Add table-session ownership tracking through occupied.
  • Introduce a shared release path for successful reuse, failed sessions, ResultSets, and table-session wrappers.
  • Add lock-free-of-RPC pool shutdown and Session transport force close.
  • Reject tree-model occupied registration after pool close.
  • Preserve protected API compatibility for SessionPool subclasses.
  • Reconcile the implementation with the pooled-session cleanup and ResultSet detachment fixes already merged into master.
  • Add deterministic latch-based tests for borrow/close and release/close interleavings.
  • Add capacity tests for dialect/database reset and connection failure recovery.
  • Add tests for wrapper reuse after failure, pool close without monitor blocking, tree-query close races, and sortTablet() behavior.

Validation

  • git diff --check passed.
  • ./mvnw -o spotless:apply -P with-integration-tests completed successfully for all modules.
  • Tests were added but were not executed locally. CI will run the unit and integration tests.

Track borrowed table sessions until release and close them when the pool closes. Make wrapper close idempotent and preserve pool capacity on failure paths.
Return the borrowed session when sortTablet succeeds or throws, and remove an unused connection-exception cleanup overload.
@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 65.00000% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.04%. Comparing base (7b407ec) to head (665c926).
⚠️ Report is 13 commits behind head on master.

Files with missing lines Patch % Lines
...apache/iotdb/session/pool/TableSessionWrapper.java 41.02% 23 Missing ⚠️
...ava/org/apache/iotdb/session/pool/SessionPool.java 80.32% 12 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19247      +/-   ##
============================================
+ Coverage     48.89%   49.04%   +0.14%     
  Complexity      689      689              
============================================
  Files          5588     5590       +2     
  Lines        413444   414011     +567     
  Branches      54537    54645     +108     
============================================
+ Hits         202154   203052     +898     
+ Misses       211290   210959     -331     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Coordinate pool close with session borrowing and release to verify that sessions are not returned to a closed pool.
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lifecycle problems described in this PR are real. I checked the fixes with a fake Thrift server probe (a mock IClientRPCService.Iface behind a TThreadPoolServer, plus mocked Sessions), running the same probes against the merge base and against this head:

Scenario merge base this PR
pool.close() while a table session is borrowed session left open session closed
sql_dialect restore fails, maxSize=1 size=1 and empty queue, so the pool is exhausted for good capacity kept
tree-model result set closed after pool.close() session pushed into the closed pool's queue session closed
ITableSession.close() after a connection failure NPE returns normally

There is one new problem, though. pool.close() now waits for in-flight RPCs on borrowed table sessions, and it holds the pool monitor while it waits (see the comment on close()). On this head, with one table session running a statement that the server holds for about 4.5 s, close() took 4528 ms, and a concurrent getSession() was blocked on the monitor for 1.5 s before it failed with Session pool is closed. On the merge base, close() returned in 0 ms. Because connectionTimeoutInMs defaults to 0 (no socket read timeout), one stuck RPC can hang close(), and every other borrow and return, indefinitely. I think this should be fixed before merging.

The other comments are smaller:

  • The tree-model query path still has the close race that this PR fixes for table sessions.
  • The wrapper still throws an NPE if it is reused after a connection failure.
  • Two of the capacity fixes have no test.
  • A few nits, marked as such.

CI: the only failing IT job is IoTDBLoadTsFileClusterIT.testRestartedDataNodeIsSyncedWithTheLoadedFile (tree model, 1C3D), which looks unrelated. The new unit tests and table-model ITs ran and passed.

// do nothing
LOGGER.warn(CLOSE_THE_SESSION_FAILED, e);
}
closeSession(session);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Borrowed table sessions are now tracked in occupied, so this loop closes sessions that other threads may be using at that moment, and it holds the pool monitor while doing so. Session.close() calls SessionConnection.close(), which sends closeSession through the synchronized client proxy (SynchronizedHandler locks on the client). The call therefore waits until the other thread's in-flight RPC returns.

Probe: a fake server, a table pool with maxSize=2, and one borrowed session running a statement that the server holds for about 4.5 s.

  • This PR: close() returned after 4528 ms. A concurrent getSession() waited on the pool monitor for 1.5 s and then got Session pool is closed.
  • Merge base: close() returned in 0 ms, because borrowed table sessions were not touched here.

connectionTimeoutInMs defaults to 0, which means no socket read timeout. One stuck RPC can therefore hang close() indefinitely. Everything that needs the monitor hangs with it: getSession(), putBack(), releaseSession(), and TableSessionWrapper.close() (through isPoolClosed()).

Suggestion: under the lock, set closed = true and take and clear snapshots of queue and occupied. Close the snapshots after leaving the monitor. For sessions that are still in use, consider closing the transport directly instead of queueing a closeSession RPC behind the in-flight call. SessionConnection already has forceCloseTransport().

closeSession(session);
throw new IoTDBConnectionException(SESSION_POOL_IS_CLOSED);
}
occupied.put(session, session);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes the borrow/close race for table sessions, but the tree-model query path still has it. The 12 query methods call occupy(session) after the RPC returns, and occupy() does not check closed. If close() runs while the query RPC is in flight, the session goes into occupied after close() has already cleared it.

Probe: executeQueryStatement blocked inside the RPC, then pool.close(), then the RPC returns. The wrapper is still returned, session.close() was called 0 times, and currentOccupiedSize() is 1 on the closed pool. The new releaseSession() does close the session once the caller closes the result set. If the caller relies on pool.close() for cleanup, for example by abandoning result sets at shutdown, the connection stays open.

Could occupy() do the same check under the lock as this code, closing the session and throwing if the pool is closed? Another option is to register every borrowed session in getSession() itself, under the lock that already checks closed. That would also remove the window between the two lock acquisitions here.

@Override
public void insert(Tablet tablet) throws StatementExecutionException, IoTDBConnectionException {
try {
session.insertRelationalTablet(tablet);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

releaseOnConnectionFailure() and close() both set session to null. If the same wrapper is called again, this line dereferences null and throws NullPointerException instead of an IoTDBConnectionException. Both executeQueryStatement overloads and executeNonQueryStatement behave the same way.

Probe: the first executeNonQueryStatement fails with IoTDBConnectionException. The second throws NullPointerException: Cannot invoke "org.apache.iotdb.session.Session.executeNonQueryStatement(String)" because "this.session" is null. The merge base behaves the same, so this is not a regression. Still, a caller that retries on the same ITableSession after a connection error will hit it, and this PR already makes close() safe after a connection failure. A guard at the top of these methods that throws IoTDBConnectionException, with the message as an i18n constant, would make the behavior consistent.

}

sessionPool.putBack(session);
sessionPool.releaseSession(sessionToRelease, reusable);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two of the fixes listed in the description have no test:

  1. Restoring sql_dialect/database fails: the session is closed and pool capacity is kept (this line with reusable = false).
  2. A connection failure in insert/execute*: releaseOnConnectionFailure() closes the session and replenishes the pool.

If either path went back to a bare session.close(), every test in this PR would still pass.

I checked (1) with a probe: maxSize=1, a getSqlDialect() that differs from the pool's, and a restore that throws StatementExecutionException. The merge base ends with size=1 and an empty queue, so the pool is exhausted for good. This PR ends with size=0: in the probe the replacement could not connect, so the slot was freed. In SessionPoolTest, mocking a failing executeNonQueryStatement and then asserting size, currentOccupiedSize() and currentAvailableSize() would protect both paths.

}
}

protected void cleanSessionAndMayThrowConnectionException(ISession session) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SessionPool is public and not final, and this protected overload has existed since 2.0.1-beta. Removing it breaks any subclass that calls it. Recompiling against the new jar fails, and running an old build against the new jar throws NoSuchMethodError. If compatibility matters here, a @Deprecated overload that delegates to discardSession(session) would keep those subclasses working. Also note that releaseSession and isPoolClosed become new protected API.

}
if (reusable && !closed) {
queue.push(session);
this.notifyAll();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: only one session is pushed here, but notifyAll() wakes every waiting borrower. putBack() and tryConstructNewSession() use notify(). With many waiters, each release wakes all of them; all but one get null from poll(), and each of those calls notify() again. notify() is enough here and matches putBack().

ISession session, int times, IoTDBConnectionException e) throws IoTDBConnectionException {
closeSession(session);
tryConstructNewSession();
if (!releaseSession(session, false)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: every caller of this method is a tree-model operation, and its session is not yet in occupied when the exception happens. releaseSession(session, false) therefore always returns false here, and discardSession() does the actual work. discardSession() also duplicates the close-then-replenish tail of releaseSession(). A single private discard(session) that removes the session from occupied, closes it, and replenishes if the pool is open could serve both callers, so the two paths cannot drift apart.

session = null;
}

if (sessionToRelease == null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: closed = true is always set together with session = null. sessionToRelease can only be null here if closed was already true, so this branch is unreachable. Either use session == null as the closed state, or keep closed and drop this branch. The this.closed = false in the constructor is also redundant.

sessionPool.releaseSession(sessionToRelease, reusable);
}

private IoTDBConnectionException releaseOnConnectionFailure(IoTDBConnectionException exception) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit (stale comments): the class Javadoc still says the wrapper calls cleanSessionAndMayThrowConnectionException on connection errors, but that overload is gone and this method releases through releaseSession. Likewise, the comment on occupied in SessionPool (// for session whose resultSet is not released.) is no longer accurate, because borrowed table sessions are tracked there now. That also changes what currentOccupiedSize() reports.

// the borrowed session is registered in occupied.
while (!pool.isPoolClosed()) {
try {
pool.wait(5_000);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: both race tests depend on implementation details, and if those change, the failures hide the cause.

  • This pool.wait() works only because getSession() calls poll() while holding the pool monitor. If poll() moves out of the lock, pool.wait() throws IllegalMonitorStateException inside the borrow. The test then fails on assertTrue(e.getCause() instanceof IoTDBConnectionException) with no message, so the real exception is never shown.
  • testTableSessionReleaseRacingWithPoolClose depends on getDatabase() being evaluated before sessionPool.database != null. If someone swaps that condition, which is a natural micro-optimization, the latch never counts down. The test then fails at assertTrue(aboutToRelease.await(...)) after 5 s, again with no message.

Close borrowed sessions without holding the pool monitor, guard reuse after connection failure, and cover capacity recovery paths.
Prevent tree-model result queries from registering a session in occupied after the pool has closed.
Reconcile pooled session synchronization and result-set detach fixes from master with occupied tracking, force close, and tree query lease checks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants