feat(qwp)!: stop flushing pooled senders on close, add a commit window and a lease-scoped reset

[jovfer]

Why

Applications that borrow a pooled Sender per message paid one QWP frame and one server-side WAL commit per message, because the pooled close() flushed. A customer benchmark of that shape took 54 s for 1M rows against 6.5 s for the batched HTTP sender, and the server's commit rate was the ceiling. The quick start taught exactly that shape.

What changes

Close releases, flush ships, drain waits. close() on a borrowed sender now returns the slot and ships nothing by itself. Rows go out when an auto-flush threshold trips on a later append, or when the pool's housekeeper settles the idle slot on the auto_flush_interval cadence (100 ms by default). Borrowing per message therefore batches exactly like a sender held for the whole run. flush() on a lease still ships now, drain() still waits for the server.

• flush_on_close=on (builder flushOnClose(true)) restores the pre-1.4 behaviour: one frame and one commit per close.
• The housekeeper gains a settle step on its own cadence; reaping stays on housekeeper_interval_ms. A slot nobody borrows again no longer keeps its last rows until the handle closes.
• The free list is LIFO, so per-message borrows concentrate on hot slots and batches fill.
• A partial row at close() is cancelled and the slot kept (the close still throws); a dead delegate is retired by an explicit probe instead of a throwing flush.
• Close-time errors are wrapped: a LineSenderServerException keeps its type, other errors arrive as a LineSenderException whose cause is the delegate's error. A lease body that already threw keeps its exception as the primary one (previously try-with-resources failed with "Self-suppression not permitted").
• Sender gains eight default methods used by the pool: checkError, hasRowInProgress, pendingRowCount, rowsAppended, flushDeferred, hasDeferredFrames, markSavepoint, resetToSavepoint.

Commit window. With flush_on_close=on, commit_rows and commit_interval_ms (builder commitRows, commitIntervalMillis; setting one defaults the other to 1000 rows / 100 ms) make each close ship a frame flagged defer-commit and the pool commit once per window: the release that fills the window ships the committing frame, and the settle commits an idle open group. Rows are in the store-and-forward ring the moment close() returns, and the server commits once per window instead of once per close. Acks arrive at the commit, so a drain() should follow a flush(). The builder injects transaction=on for this mode and rejects a window with flush_on_close=off (designed, gated until measured) or with transaction=off.

Wording trap. In this client "flush" means published to the store-and-forward ring, not acknowledged by the server. flush_on_close=on does not mean committed when close() returns.

Review round 1 fixes

• Lease-scoped reset(). borrow() marks a savepoint on the delegate and the pooled reset() rolls back to it, so a borrower following the BatchTooLargeForCapException remedy no longer discards rows that earlier leases left in the slot. The WebSocket sender rolls every table buffer back to the mark, restores the pending counters and the batch symbol watermark, reclaims the dictionary ids the dropped rows registered (delta mode), and drops column definitions the server has not seen. A pooled sender ignores a borrower's markSavepoint(); its resetToSavepoint() is the lease-scoped reset().
• Byte trigger bounded by the server cap. With auto_flush_bytes unset, the WebSocket sender now flushes at 90 percent of the batch cap the server advertises; an explicit off keeps byte flushing off, and a node that advertises no cap gets no byte trigger. This changes the standalone default too: a batch the server would reject ships earlier instead. Rows appended before the first handshake (async connect) carry no byte trigger; the first append after it flushes the backlog.
• Over-cap batch. The sender remembers a cap rejection until reset(), a rollback or a later flush that passes the pre-flight check, and reports it through checkError(). At release the pool first rolls the lease back to its borrow mark and retries the flush: when the rows earlier leases left ship, the slot stays in service and the borrower gets a wrapper naming its own dropped rows. Only an inherited batch the cap still rejects retires the slot. The settle step retires such a slot directly. Both retirements log at ERROR with the dropped row count, and the pool's messages no longer repeat the standalone "call reset() and keep this sender" remedy.
• Housekeeper. A failed startup recovery step is retried once a second instead of on the settle cadence. Deadline arithmetic saturates, so housekeeper_interval_ms=Long.MAX_VALUE disables reaping and nothing else.
• Docs. The README no longer promises replay after a rejection. With the default one frame carries several leases' rows; the README, Javadoc and examples say so and name flush() before close() and flush_on_close=on as the isolation tools. Example and README comments that said close() flushes are corrected; license headers added to two test classes.
• Tests. Five tests that fail when their target line is mutated, wire tests for each item above, and a wire test that pins the documented fate-sharing behaviour.

Trade-off

With the default, a terminal rejection of one lease's row discards the other leases' rows in the same frame, after their close() calls returned, and the error surfaces on a later call on that slot, possibly to a lease that wrote none of the rejected rows. This is the HTTP sender's failure span moved across threads. The alternative, flush_on_close=on as the default, keeps one frame and one commit per message, which is the regression this PR removes.

The byte trigger measures raw column bytes after the row that crosses it is committed, so a row larger than the remaining 10 percent can still push a batch over the cap. The release path above limits the loss to that lease's rows; the settle path, where no lease owns the batch, still retires the slot.

One existing test with a 150-byte advertised cap needed auto_flush_bytes=off to keep its explicit-flush control flow. Downstream tests that call setAdvertisedMaxBatchSize with small values may need the same.

Benchmark

Manual benchmark (PoolBorrowPerRowBenchmark, @Ignored), 1M rows against the in-repo ack-all test server, which has no WAL:

configuration	time	data frames
held lease, default	271 ms	1000
borrow per row, default	98 ms	1000
borrow per row, flush_on_close=on (previous behaviour)	8462 ms	1,000,000
borrow per row, flush_on_close=on + window 1000/100	8335 ms	1,000,000

The frame count dominates on the client and the wire. The window's commit saving needs a real server to show; the test server commits nothing.

Release note (1.4.0)

close() on a pooled sender no longer flushes; set flush_on_close=on to restore it. Rows left in a slot ship within auto_flush_interval. Close-time errors are wrapped with the delegate's error as cause. New keys commit_rows and commit_interval_ms. reset() on a pooled sender discards only the rows appended through that lease. With auto_flush_bytes unset, the WebSocket sender flushes at 90 percent of the server's advertised batch cap.

Follow-ups

• Settle-failure WARN flag clears only on a settle-side success.
• Server-side QWP commit interval and per-connection schema caching (OSS), and the fourth combination gate, per the spec.
• Documentation repository: quick start, pool and migration sections.
• OSS tandem: QuestDBFacadeE2ETest#testTwoSendersPerTableWhileReaderThreadStreamsTwoQueries relies on close-time flushing with a one-hour auto_flush_interval; it needs flush() before close or flush_on_close=on when the submodule pin moves to this release.
• Enterprise: four files use borrowSender() (QwpRoleBounceChaosLosslessTest, SqlFailoverQwpDeferredCloseExactlyOnceTest, QwpSidecarMain, QwpSidecarMainTest) and need the same check.
• A slot retired with an over-cap batch logs at ERROR but does not reach SenderErrorHandler; routing client-side drops there needs a client-side error category.
• Known narrow gaps left from review:
  • In full-dictionary mode a symbol value larger than the cap stays in the dictionary after the rollback (id reclaim is delta-mode only), so the release path keeps the slot and later leases that add a new symbol fail with the "dropped N row(s)" wrapper. Retiring the slot when the rejection came from the dictionary would restart the id space.
  • In the full-dictionary chunk fallback, a non-cap failure can leave an older cap rejection latched.
  • No test drives the release-time rollback and retry through close's own flush with flush_on_close=on, with or without a commit window.
  • A node that advertises no cap gets no byte bound on the config-string path, while the programmatic path keeps 8 MiB.
  • The housekeeper wakes once a second for the life of the handle, and a settle drop logs two lines.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ENEBF3hLEx585DdJfKoUSG

[mtopolnik]

[PR Coverage check]

😍 pass : 420 / 449 (93.54%)

file detail

	path	covered line	new line	coverage
🔵	io/questdb/client/Sender.java	7	9	77.78%
🔵	io/questdb/client/impl/PoolHousekeeper.java	30	38	78.95%
🔵	io/questdb/client/impl/PooledSender.java	81	88	92.05%
🔵	io/questdb/client/impl/SenderPool.java	103	111	92.79%
🔵	io/questdb/client/QuestDBBuilder.java	51	53	96.23%
🔵	io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java	94	96	97.92%
🔵	io/questdb/client/impl/ConfigSchema.java	3	3	100.00%
🔵	io/questdb/client/impl/SenderSlot.java	13	13	100.00%
🔵	io/questdb/client/cutlass/qwp/client/BatchTooLargeForCapException.java	5	5	100.00%
🔵	io/questdb/client/impl/QuestDBImpl.java	4	4	100.00%
🔵	io/questdb/client/cutlass/qwp/protocol/QwpTableBuffer.java	29	29	100.00%