Skip to content

fix: honour request cancellation on PostgreSQL, and apply max_rows to comment-prefixed and CTE queries - #399

Closed
Elrendio wants to merge 2 commits into
bytebase:mainfrom
Elrendio:fix/postgres-cancellation-and-max-rows
Closed

fix: honour request cancellation on PostgreSQL, and apply max_rows to comment-prefixed and CTE queries#399
Elrendio wants to merge 2 commits into
bytebase:mainfrom
Elrendio:fix/postgres-cancellation-and-max-rows

Conversation

@Elrendio

Copy link
Copy Markdown
Contributor

Two independent bugs in one PR because they were found together and share a test run. They are two clean commits — happy to split into fix: apply max_rows to comment-prefixed and CTE queries and fix(postgres): cancel the running query when the request is aborted if you prefer one concern per PR.

Two independent bugs, both of which let a query keep consuming database resources that nobody
asked it to consume. We hit both running DBHub against ~24 production PostgreSQL read replicas
with agents driving execute_sql; between them they contributed to replication lag of up to
10.9 h and one production incident caused by a single agent query that ran for 1 h 30.

Each is its own commit and each stands alone.


1. max_rows silently does not apply to many queries — d3694d0

SQLRowLimiter.isSelectQuery decided whether to cap a statement with:

static isSelectQuery(sql: string): boolean {
  const trimmed = sql.trim().toLowerCase();
  return trimmed.startsWith('select');
}

so anything not opening with a bare SELECT got no LIMIT appended at all. Not a wrong
limit — no limit, and nothing anywhere says so. Verified by executing the shipped v1.2.0
module (applyMaxRows(sql, 100)):

input v1.2.0 output
-- reporting job\nSELECT * FROM t returned unchanged
/* tag: report */ SELECT * FROM t returned unchanged
WITH x AS (SELECT * FROM orders) SELECT * FROM x returned unchanged
(SELECT id FROM a) UNION (SELECT id FROM b) returned unchanged
SELECT * FROM (SELECT * FROM t LIMIT 5) s returned unchanged
SELECT * FROM t SELECT * FROM t\nLIMIT 100

This is not an exotic corner. Leading comments are how queries get attributed
(-- app=…, /* trace-id */), and a CTE is the ordinary shape of an analytical query — which
is exactly the kind of query a row cap exists to contain. In practice max_rows covered far
less than it appeared to.

The existing suite pinned the gap rather than catching it: three integration tests were named
should not apply maxRows to CTE queries (WITH clause). This PR inverts them.

To be clear about what those tests were pinning: git log -S shows the startsWith('select')
guard and those three tests both arrive in the same commit — 4fa886e feat: support max-row limit, the commit that introduced max_rows. The CTE exclusion was never a separate, deliberate
carve-out; it is a consequence of that one-line classifier, and the tests recorded whatever it
happened to do. (One of the assertions is commented "not limited anymore", which reads as though a
decision had been reversed — the history shows there was no earlier behaviour to reverse.) If the
exclusion is wanted, it deserves to be an explicit rule rather than a side effect of the guard,
and this PR should be redirected to documenting it instead.

Fix. Classify on the comment/string-blanked text, and treat a leading WITH as
row-returning. The SQL sent to the server is never rewritten by the classification, so an
attribution comment survives verbatim — only the classifier sees the blanked form.

A data-modifying CTE (WITH x AS (DELETE … RETURNING *) SELECT …) is deliberately still
not limited: a LIMIT would cap the rows handed back while the write ran in full, which reads
as a cap that isn't one. That check reuses the read-only classifier's own keyword heuristic,
extracted as hasMutatingKeyword, so both layers agree on what counts as a write hidden inside
a WITH. A false positive there only means the statement keeps today's behaviour.

Enabling CTEs exposed a second bug, which is why the diff is larger than "add with to a
list". Every LIMIT/TOP helper matched the first clause found textually, which on a CTE is the
inner one:

WITH recent AS (SELECT * FROM orders LIMIT 5)
SELECT * FROM recent JOIN big ON true

Naively enabling CTEs would have rewritten the CTE's own LIMIT 5 and left the statement — the
part that can return millions of rows — uncapped. So the helpers now scan with parenthesis-depth
tracking (the mechanism already in this file for hasSetOperator and the ORDER BY hoist) and
reason only about the statement's own clause; a nested LIMIT/TOP caps only its branch, so the
statement still gets a cap appended. This also fixes the same pre-existing hole for plain
subqueries (SELECT * FROM (SELECT * FROM t LIMIT 5) s, last row of the table above).

On SQL Server, TOP now lands on the statement's own SELECT rather than the first one
textually (for a CTE, the final SELECT), and a leading CTE is kept outside the derived table
when a set operation forces the #387 wrap — T-SQL has no SELECT … FROM (WITH …) AS subq form.

Explicitly preserved: SELECT … UNION ALL SELECT … starts with select, still gets a
trailing LIMIT, and on PostgreSQL that binds to the whole set operation. Pinned by a
regression test.


2. MCP cancellation is ignored — the query keeps running — db26b0b

src/tools/execute-sql.ts receives extra, which carries the MCP SDK's AbortSignal, and uses
it only for trackToolRequest(...). There was no AbortSignal/AbortController anywhere in
src/, and ExecuteOptions had no field to carry one. Cancelling a tool call therefore only
stopped us waiting for the answer: the backend went on producing a result set nobody would
ever read.

This is the same class of bug as #384 ("MySQL queries can continue running on the server
after query timeout"
) and this change mirrors the fix accepted for it in #386. MySQL's
trigger is its own client-side timeout; here the trigger is the client cancelling. Both end the
same way: kill the statement server-side over a second connection, and don't hand the connection
back to the pool in a state the next borrower inherits. PostgreSQL was simply left behind.

It matters most where it hurts most — read replicas serving agents, where an abandoned analytical
query keeps consuming IO for its full runtime and drives replication lag long after the agent has
given up and moved on.

Reproduction, as the added integration test Query cancellation (options.signal): start
SELECT pg_sleep(30), abort the signal, then watch pg_stat_activity.

  • Before: the test fails after 30,042 ms with expected null to be an instance of Error
    — the query resolved. The abort changed nothing and the backend slept the full 30 s.
  • After: the statement rejects with SQLSTATE 57014 (query_canceled) within
    milliseconds, and the backend is gone from pg_stat_activity.

Fix.

  • ExecuteOptions gains an optional signal; execute_sql passes the request's own. Connectors
    that ignore it are unaffected — MySQL/MariaDB/SQLite/SQL Server behaviour is untouched.
  • PostgreSQL cancellation is out-of-band by design: the request must arrive on a different
    connection, since the one running the query is blocked waiting for it. cancelBackend opens a
    dedicated short-lived connection rather than borrowing from the pool — a pool saturated with the
    very long-running queries being cancelled has nothing left to lend, which is exactly the
    situation this runs in.
  • The connection is destroyed instead of released after a cancellation. A CancelRequest races
    the statement it targets, so it can instead land on the ROLLBACK issued during cleanup and
    leave the session sitting in an aborted transaction. Mirrors MySQL's isConnectionPoisoned.
    Covered by a test asserting the connector still works after a cancellation.
  • The abort listener is removed in finally, so listeners don't accumulate across the statements
    of one request.
  • An already-aborted signal is refused before a connection is taken, and re-checked once one is
    held: an AbortSignal dispatches abort exactly once, so a signal that fired while we waited
    for the pool would never reach a listener registered afterwards.

Cancellation stays advisory — the backend acts on it at an interrupt point, and a statement that
already finished ignores it — so cancelBackend is best-effort and logs rather than throwing.


Testing

  • pnpm run test:unit983 passed (958 on main; +25 new).
  • pnpm run test:integration339 passed across PostgreSQL, MySQL, MariaDB, SQL Server and
    SQLite containers.
  • pnpm run build:backend — clean.
  • Both commits verified green independently (commit 1 alone: 981 unit + 98 integration).
  • The cancellation fix is backed by a negative control: reverting only the connector change makes
    the new tests fail after 30 s, as shown above.

New coverage: leading --//* *//mixed comments; CTE; CTE with an inner LIMIT; CTE with its own
LIMIT; CTE with a parameterized LIMIT; data-modifying CTE (all three of INSERT/UPDATE/DELETE);
parenthesised set operation; nested-subquery LIMIT; UNION ALL regression; SQL Server CTE/TOP
cases; signal forwarding from the tool handler; and four PostgreSQL cancellation integration
tests (plain, inside BEGIN READ ONLY, pool reuse after cancellation, already-aborted signal).

Notes for reviewers

  • explain_sql also runs user SQL and could take the same signal — one line — but is left out
    to keep this diff to the reported bugs. Happy to add it.
  • Cancellation is implemented for PostgreSQL only. The same treatment for SQLite/SQL Server would
    be a natural follow-up; MySQL already has the timeout-triggered half from fix(mysql): kill server-side query on timeout instead of leaking it into the pool #386.
  • npx tsc --noEmit reports 132 pre-existing errors on main; this branch reports the same 132,
    none in the changed files.

Elrendio and others added 2 commits August 9, 2026 19:17
`SQLRowLimiter.isSelectQuery` classified a statement with
`sql.trim().toLowerCase().startsWith("select")`, so `max_rows` was
silently a no-op for every query that does not open with a bare
`SELECT`. No LIMIT was appended at all — the cap simply did not exist
for these, and nothing surfaced that:

  -- reporting job\nSELECT * FROM t     -> returned unchanged
  /* tag */ SELECT * FROM t             -> returned unchanged
  WITH x AS (SELECT ...) SELECT * FROM x-> returned unchanged
  (SELECT a) UNION (SELECT b)           -> returned unchanged

Leading comments are common (query tags/attribution) and CTEs are the
normal shape of an analytical query, so in practice the row cap covered
much less than it appeared to.

Classify on the comment/string-blanked text instead, and accept `WITH`
as row-returning. The SQL handed to the server is never rewritten by the
classification, so an attribution comment survives verbatim. A
data-modifying CTE (`WITH x AS (DELETE ... RETURNING *) SELECT ...`) is
deliberately still not limited: a LIMIT would cap the rows handed back
while the write ran in full, which reads as a cap that isn't one. That
check reuses the read-only classifier's keyword heuristic, now shared as
`hasMutatingKeyword`.

Enabling CTEs exposed a second problem: the LIMIT/TOP helpers matched
the first clause found textually, which on a CTE is the *inner* one.
`WITH x AS (SELECT ... LIMIT 5) SELECT * FROM x JOIN big ON true` would
have had the CTE's own cap rewritten while the statement stayed
uncapped. Every helper now scans with parenthesis-depth tracking (the
mechanism already used for `hasSetOperator` and the ORDER BY hoist) and
reasons only about the statement's own clause; a nested LIMIT/TOP caps
only its branch, so the statement still gets a cap appended. This also
fixes the same pre-existing hole for plain subqueries
(`SELECT * FROM (SELECT * FROM t LIMIT 5) s`).

On SQL Server, TOP now lands on the statement's own SELECT rather than
the first one textually (for a CTE, the final SELECT), and a leading CTE
is kept outside the derived table when a set operation forces a wrap —
T-SQL has no `SELECT ... FROM (WITH ...) AS subq` form.

`SELECT ... UNION ALL SELECT ...` keeps its existing behaviour: it
starts with `select`, gets a trailing LIMIT, and on PostgreSQL that
binds to the whole set operation. Covered by a regression test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The MCP request's AbortSignal reached `execute_sql` (as `extra`) but was
only ever read for telemetry. Nothing in `src/` referenced an
AbortSignal, and `ExecuteOptions` had no field to carry one, so
cancelling a tool call merely stopped us waiting for the answer: the
backend went on producing a result set nobody would ever read.

That is the same class of bug as bytebase#384 on MySQL — a query outliving the
client that asked for it — and this mirrors the fix accepted for it in
bytebase#386. MySQL's trigger is its own client-side timeout; here the trigger
is the client cancelling. Both end the same way: kill the statement
server-side over a second connection, and don't hand the connection
back to the pool in a state the next borrower would inherit.

It matters most exactly where it hurts most: on read replicas serving
agents, where an abandoned analytical query keeps consuming IO for its
full runtime and drives replication lag long after the agent has gone.

Reproduction (integration test `Query cancellation (options.signal)`):
start `SELECT pg_sleep(30)`, abort the signal, then watch
`pg_stat_activity`. Before this change the test fails after 30,042ms
with the query having *resolved* — the abort changed nothing and the
backend slept the full 30s. After it, the statement rejects with
SQLSTATE 57014 (query_canceled) within milliseconds and the backend is
gone from `pg_stat_activity`.

- `ExecuteOptions` gains an optional `signal`; `execute_sql` passes the
  request's own. Connectors that ignore it are unaffected.
- PostgreSQL cancellation is out-of-band: the request must arrive on a
  different connection, since the one running the query is blocked
  waiting for it. `cancelBackend` opens a dedicated short-lived
  connection rather than borrowing from the pool — a pool saturated
  with the very queries being cancelled has nothing left to lend.
  `pg`'s own `Client.cancel` is not reusable here: it needs the
  internal Query object, which the promise-based query API never
  exposes.
- The connection is destroyed instead of released after a cancellation.
  A CancelRequest races the statement it targets, so it can instead
  land on the ROLLBACK issued during cleanup and leave the session in
  an aborted transaction. Mirrors MySQL's `isConnectionPoisoned`.
- The abort listener is removed in `finally`, so listeners don't
  accumulate across the statements of one request.
- An already-aborted signal is refused before a connection is taken,
  and re-checked once one is held: an AbortSignal dispatches "abort"
  exactly once, so a signal that fired while we waited for the pool
  would never reach a listener registered afterwards.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Elrendio added a commit to Elrendio/dbhub that referenced this pull request Aug 10, 2026
Integration branch: for_stockly_main carries every Stockly-facing change on top of
upstream main, one merge per upstream PR, so each PR branch stays reviewable on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Elrendio

Copy link
Copy Markdown
Contributor Author

Split into one PR per bug, as they are independent and easier to review separately:

Each branch is now a single commit on main and was re-verified standalone (unit + full integration suite + build). Closing this one; no content was dropped.

@Elrendio Elrendio closed this Aug 10, 2026
@Elrendio
Elrendio deleted the fix/postgres-cancellation-and-max-rows branch August 10, 2026 12:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant