[c++] Add write callbacks with batch completion dispatch - #4328
naivedogger wants to merge 17 commits into
Conversation
db30250 to
e0d7dd6
Compare
|
@fresh-borzoni @loserwang1024 @leekeiabstraction, Appreciate a review here, thanks! 🙏 |
fresh-borzoni
left a comment
fresh-borzoni
left a comment
There was a problem hiding this comment.
@naivedogger Thank you, left a comment, PTAL
|
@fresh-borzoni thanks for the earlier review. I pushed three commits on top. enqueue_timeout now bounds the whole callback submission, both callback capacity and buffer backpressure, in the max.block.ms style, and a zero timeout makes submission non-blocking. The callback executor thread count is now overridable through an advanced FLUSS_CALLBACK_WORKERS variable, with the default of four unchanged. I also expanded the docs and example around callback behavior: callbacks run on a small shared pool and should stay short and non-blocking, failures should be recorded and handled outside the callback with dedup by id, and crash recovery should come from a replayable source that advances only after Flush. When you have time, another look would be appreciated. |
77113f3 to
ac3775e
Compare
Make WriteCallbackOptions::enqueue_timeout cover the whole submission, both callback capacity and buffer backpressure, following the Kafka max.block.ms model. The callback path passes a submit budget through the FFI so the Rust buffer-memory wait is bounded by the remaining budget, while the public overloads keep the writer's configured buffer wait timeout. A zero timeout makes submission non-blocking.
Read the process-wide callback executor thread count from the advanced FLUSS_CALLBACK_WORKERS environment variable, falling back to the default when it is unset, invalid, or zero. This is a rarely needed escape hatch; the default of four workers is unchanged.
Explain that callbacks run on a small shared executor pool and must stay short and non-blocking, and note the advanced FLUSS_CALLBACK_WORKERS knob. Clarify failure handling: record the outcome and either stop or retry outside the callback, deduplicating by identifier, and drive crash recovery from a replayable source that advances only after Flush. Update the enqueue_timeout wording and the example comments to match.
ac3775e to
f332fe3
Compare
I do think client.writer.buffer.wait-timeout is enough, no need top. enqueue_timeout anymore |
Remove the WriteCallbackOptions enqueue_timeout field and route the whole callback submit through the connection's client.writer.buffer.wait-timeout. The capacity reservation and the buffer-backpressure wait now share one deadline sourced from that setting, so a submit returns within a single timeout instead of two. UINT64_MAX keeps the default unbounded and a zero timeout makes submission non-blocking.
…bmit Set writer_buffer_wait_timeout_ms in the example config so its role is visible: it bounds both the write-buffer wait and the whole callback submission. Add an end-to-end test that fills a writer's capacity with a blocking callback and asserts the next submit returns with a timeout error after the configured budget, not after the callback finally releases the slot.
|
@loserwang1024 I think you’re right. I’ve updated it to use |
3 participants
Purpose
Linked issue: close #4276
Add C++ write callbacks so applications can handle write completion without maintaining a separate WriteResult waiting queue.
Brief change log
Tests
API and Format
Documentation
Update the C++ API reference with callback examples, compatibility notes, partial-failure semantics, and application responsibilities for memory limits and shutdown.