Skip to content

fix(pool): bound return_to_pool so a dead connection can't leak its permit - #4350

Open
rslowinski wants to merge 1 commit into
transact-rs:mainfrom
rslowinski:fix/pool-permit-leak-unresponsive-connection
Open

rslowinski wants to merge 1 commit into
transact-rs:mainfrom
rslowinski:fix/pool-permit-leak-unresponsive-connection

Conversation

@rslowinski

@rslowinski rslowinski commented Jul 23, 2026

Copy link
Copy Markdown

Fixes #4349.

Problem

When a PoolConnection is dropped, the spawned return-to-pool task holds the connection's permit while it runs the after_release hook, pings, or closes the connection. Against a peer that vanished silently (no RST/FIN: NAT/firewall flow expiry, server restart mid-query, abandoned tokio::time::timeout) these awaits never complete. The permit is never released, and after max_connections such drops every acquire() fails with PoolTimedOut until the process restarts. Nothing logs, because nothing errors.

Fix

Per review, the whole Floating::return_to_pool call is wrapped in a 5s timeout (sibling of CLOSE_ON_DROP_TIMEOUT), which also covers the after_release and max_lifetime/pool-closed close() paths raised in the thread. On timeout the Floating is dropped: its DecrementSizeGuard releases the permit and the socket closes locally, so nothing inside (including close_hard) can pin the permit.

Test

tests/postgres/pool.rs runs a real PgPool against an in-process fake server that finishes the startup handshake and then goes silent. It drops a checked-out connection and asserts the next acquire() opens a fresh connection instead of hitting PoolTimedOut. Needs no DATABASE_URL; gated to runtime-tokio. Fails on main (PoolTimedOut at 20s), passes with the fix (~5s).

@adriangb

adriangb commented Jul 24, 2026

Copy link
Copy Markdown

Sorry about the noise, agent went a bit wild, I reigned it in and rewrote / went over this comment by hand.

We hit the same failure in production: a server restart during a query execution causes the query to hang forever. We noticed this because we have a ~ hot loop that is constantly hitting postgres, so we have a high chance of having an in-flight query where the timing of an upstream restart causes it to hang.

A gap found working on #4354 (now closed in favor of this PR): return_to_pool has three unbounded awaits, not one. The ping is the common case, but the same permit is held across:

  • the after_release hook, which is arbitrary user code and, more to the point, usually runs a query on the same dead socket;
  • self.close().await on the pool-closed and beyond-max_lifetime paths. close() is a graceful close: it writes Terminate and waits for the server to drop the connection, which on a silent peer never happens.

The max_lifetime one is not hypothetical for us: we set it, so every aged-out connection takes that path, and an aged-out connection on a dead socket leaks its permit exactly like the ping case.

The fix is mechanical: give each the same bound and fall back to close_hard(), which does no I/O. Cancelling close() also drops the connection and releases the permit, so a bounded close is safe:

async fn close_bounded(self) {
    let _ = crate::rt::timeout(RETURN_TO_POOL_PING_TIMEOUT, self.close()).await;
}

adriangb added a commit to adriangb/sqlx that referenced this pull request Jul 24, 2026
Follow-up to the previous commit (transact-rs#4350), which bounds the
on-release `ping()`. `return_to_pool` holds the pool permit across two more
unbounded awaits on the same connection:

- the `after_release` hook, which typically runs a query on the socket that
  may be dead;
- `close()` on the pool-closed and beyond-`max_lifetime` paths. `close()`
  writes `Terminate` and waits for the peer to drop the connection, which a
  silent peer never does.

We set `max_lifetime`, so the second path runs for every aged-out connection
and leaks its permit exactly like the ping case.

Offered upstream in
transact-rs#4350 (comment).
adriangb added a commit to adriangb/sqlx that referenced this pull request Jul 24, 2026
Follow-up to the previous commit (transact-rs#4350), which bounds the
on-release `ping()`. `return_to_pool` holds the pool permit across two more
unbounded awaits on the same connection:

- the `after_release` hook, which typically runs a query on the socket that
  may be dead;
- `close()` on the pool-closed and beyond-`max_lifetime` paths. `close()`
  writes `Terminate` and waits for the peer to drop the connection, which a
  silent peer never does.

We set `max_lifetime`, so the second path runs for every aged-out connection
and would leak its permit exactly like the ping case.

Offered upstream in
transact-rs#4350 (comment).

@abonander abonander left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to wrap the entire return_to_pool call in the timeout just to insure against stuff like close_hard deadlocking (though it really shouldn't).

I already have this fix in #3582 but that won't be merged until 0.10 at the earliest.

…ermit

When a PoolConnection is dropped, the spawned return-to-pool task holds the
connection's permit while it runs the after_release hook, pings, or closes
the connection. Against a peer that vanished silently (no RST/FIN) these
awaits never complete, so the permit is never released; after
max_connections such drops every acquire() fails with PoolTimedOut.

Bound the whole return_to_pool call. On timeout the Floating is dropped,
which releases the permit and closes the socket locally.

Fixes transact-rs#4349.
@rslowinski
rslowinski force-pushed the fix/pool-permit-leak-unresponsive-connection branch from 449f729 to 60772eb Compare September 21, 2026 11:10
@rslowinski rslowinski changed the title fix(pool): bound the on-release ping so a dead connection can't leak its permit fix(pool): bound return_to_pool so a dead connection can't leak its permit Sep 21, 2026
@rslowinski

Copy link
Copy Markdown
Author

Done: the timeout now wraps the whole return_to_pool call in PoolConnection::return_to_pool, so a timeout drops the Floating and releases the permit regardless of which await hung (ping, after_release, or close() on the max-lifetime / pool-closed paths). Rebased on main and squashed to one commit; test verified failing on main and passing with the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dropping a PoolConnection whose connection is silently dead leaks the pool permit forever (return_to_pool cleanup never completes)

3 participants