jw/backfill api poc - #4703
Draft
josephwoodward wants to merge 11 commits into
Draft
Conversation
josephwoodward
commented
Aug 17, 2026
josephwoodward
commented
Contributor
- oracledb_cdc: track in-flight snapshot batch acks in the publisher
- oracledb_cdc: add flushCurrent to publish partial batches without stopping the publisher
- oracledb_cdc: gate post-snapshot checkpoint on downstream acks
- oracledb_cdc: adversarial crash test for the snapshot ack barrier
- oracledb_cdc: make batch tracking atomic with batch flushing
- oracledb_cdc: fail the snapshot gate on nack, drop orphaned publishBatch
- oracledb_cdc: log downstream batch rejections
- oracledb_cdc: log downstream snapshot rejections at error level
- oracledb_cdc: terminal nacks restart with a fresh tracker
- oracledb_cdc: nacks resolve checkpoints (auto_replay_nacks off is an opt-in drop)
- oracledb_cdc: poc of dedicated backfill api
…pping the publisher
Track order defines the ordered checkpoint sequence, but Track was called after releasing the batcher mutex, so the count-triggered flush (Publish) and the timed-flush loop could register batches out of order and persist a regressing SCN on ack. Track now happens under the same lock as the flush. Also guards the loop's UntilNext call, which read batcher state concurrently mutated by Publish (a data race confirmed by the new stress test under -race). Same fixes as the mssqlserver batcher, which shares this lifted pattern.
Mirrors the mssqlserver review fixes (#4677): a nacked batch no longer resolves its checkpoint slot, and a nacked snapshot batch fails waitSnapshotAcks so the post-snapshot SCN is not persisted over undelivered rows (auto_replay_nacks is user-toggleable, so a nack can be terminal). publishBatch had no production callers left after the flush/track refactor; deleted, with the batcher tests rewritten to drive the production Publish/flushCurrent paths.
A terminal nack (auto_replay_nacks disabled) deliberately pins the checkpoint and eventually stalls the input behind checkpoint_limit, but that consequence was invisible: nothing was logged anywhere on the nack path. Emit an error identifying the batch's checkpoint SCN, whether it was a snapshot batch, and the pinned-checkpoint consequence so operators can connect a stalled input to the downstream rejection.
The snapshot ack gate collapses soft-stop cancellation and downstream rejection into one Info line whose wording only describes the former. A rejection discards the post-snapshot SCN and re-runs the whole snapshot - an unexpected, data-affecting outcome - so it now logs at error level with wording that names it, mirroring the cancellation/error split used by the surrounding branches.
Aligns with the mssqlserver review outcome (#4677): a terminal nack (auto_replay_nacks disabled) pinned the ordered checkpoint tracker permanently — the publisher and tracker were built once and reused across Connect retries, so after one nack no SCN could ever be persisted again and the input eventually wedged behind checkpoint_limit. A nack now triggers a restart, and Connect rebuilds the publisher (batcher + tracker) per attempt, sealing the old one so late acks from the previous session can neither persist stale positions nor trigger spurious restarts. The restart resumes from the last durable SCN and redelivers.
…opt-in drop)
Unwinds the nack-pinning and terminal-nack-restart changes from the review
rounds. Per the framework's documented contract for auto_replay_nacks
("If set to false these messages will instead be deleted"), disabling
replay is an explicit opt-in to drop rejected messages - typically because
failures are routed to a DLQ, which acks. Pinning the checkpoint (or
restarting to force redelivery) contradicted that contract: pinning
produced permanent backpressure once checkpoint_limit filled, and the
restart variant turned a persistently-failing message into an infinite
redelivery loop. The snapshot ack gate still guards the crash window; a
nack now simply settles its slot and the stream continues.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.