Repository navigation
Conversation
Capture preserved COMMIT and ABORT progress when terminal-history migration fails, including before lazy manager creation. Add snapshot and restore coverage, and exercise permanent COMMIT exceptions in the state-machine test.
Implemented-draft re-reviewReviewed
Both fixes are in Validation at the final revision:
The linked lifecycle IT failure in #19051 does not exercise this retry path: it uses SimpleConsensus and fails before the intentional crash. Its global |
Implemented-draft re-reviewReviewed the complete PR diff and relevant context at Fixed by returning the readable legacy terminal when publishing its migrated copy fails, while preserving the staging record so migration can be retried. Added Validation: Reviewed SHA: The PR remains draft because the documented TODOs are still incomplete. |
Independent re-review of the implemented draftCompleted one independent re-review of the full 13-file PR diff and relevant context at The review covered retryable versus permanent PREPARE/COMMIT failures, persisted COMMIT/ABORT identity and late replicated commands, snapshot selection/copy boundaries, legacy migration failures, cleaner/retention ordering, lazy manager deletion fencing, lock ordering, and regression assertions. In particular, the earlier fixes remain intact:
Validation: this round passed The PR and source branch were refreshed and reconfirmed. The recorded PR base is
The PR remains draft. The documented TODOs are still incomplete and were neither expanded nor treated as resolved by this review. |
|
Would it make sense to split this into two dependent PRs, with the recovery guarantees landing before the retry behavior?
The first PR would need to be independently safe and testable. If there is no safe intermediate implementation, keeping one PR with clearly separated commit groups would be preferable to forcing the split. As a follow-up, it may also be worth exploring targeted retransmission of missing PIECEs: let the receiver identify missing pieces and resend only those, rather than retransmitting the original consensus batch. This would require stable piece identities, retained retransmission sources, and idempotent duplicate handling. I would keep it outside this fix; it would not replace the recovery and WAL guarantees above. |
Description
A replicated LOAD PREPARE or COMMIT can arrive before a missing PIECE is retried after reconnect. Returning a permanent LOAD error lets consensus retire the command before the replica has finished the task. This draft retries only temporarily incomplete replicated commands and preserves completed task identities across retries and restarts.
This is an incomplete fix and is not ready to merge. The remaining recovery requirements below belong to the same LOAD retry path. Some underlying behaviors predate this patch, but the retry guarantee still depends on addressing them.
Changes currently implemented
LoadFileNotReadyExceptionand map it toWRITE_PROCESS_REJECTfor replicated PREPARE/COMMIT only. Permanent failures and local command failures retain their existing error classification.TODO before marking ready, and how to adapt
Preserve acknowledged, unfinished staging. Adapt
LoadTsFileManager.stop,TsFileWriterManager.close, andLoadTsFileCleaner.reclaimIfIdleto distinguish coordinator temporary files from consensus-owned accepted state. Ordinary shutdown must close handles while preserving recoverable payload and progress. Idle time alone must not delete accepted replicated data: remote requests use a negative local search index, so the existing watermark guard does not protect them. Reclamation needs an explicit terminal decision, a proven safe replay frontier, or an implemented payload reconstruction protocol. Otherwise, a sender that already retired the PIECE batch can only retry PREPARE/COMMIT against missing data indefinitely.Make COMMIT import recoverable and idempotent. Adapt
LoadTsFileManager.loadAll,TsFileWriterManager.loadAll, and theDataRegionimport path. Persist an import plan/COMMITTING state with stable per-task, per-partition installation identities and enough information to recognize published files after restart. Recovery must finish missing installations without reinstalling completed ones or repeating their publication effects. The current sequence imports files before persisting COMMIT, while ordinary import allocates a new version/name each time. Keeping the staged source does not by itself make replay safe. Moving the final COMMIT record before import would leave a different missing-data window and is insufficient.Coordinate the terminal decision with replication WAL durability. Adapt the terminal-hit paths and WAL logging in
LoadTsFileManager.loadAll/deleteAll. Currently the terminal record can become durable before the command reaches WAL, and a subsequent terminal-hit return skips WAL logging. Define a replayable decision/intent and local application-completion protocol so successful retries also satisfy outstanding replication obligations. An in-memory dispatcher re-offer is insufficient across restart or queue overflow. The design must reconcile stable decision identity, WAL replay, and application progress; a local terminal file alone must not authorize success when durable replication information is missing.Apply one state/ownership model to snapshots and lifecycle operations. A bounded direction is a region-local task protocol such as RECEIVING -> PREPARED -> COMMITTING -> COMMITTED, with explicit permitted ABORTED transitions. Define the durability and acknowledgement conditions for every transition before implementation. Capture data files, active task/import state, terminal outcomes, and replay position from a consistent boundary. Preserve the existing staging-copy boundary, migration-before-listener ordering, and consistent lock order. Separate ordinary stop from region deletion. Terminal-history garbage collection needs a proven replay frontier; arbitrary TTL or an in-memory receive index is insufficient. Measure terminal-record growth and staging/snapshot copy latency. This does not require adding cross-region atomic LOAD or rollback semantics.
Validate the recovery boundaries above. Add failure-oriented coverage for acknowledged PIECE -> normal stop/restart -> later PREPARE/COMMIT; acknowledged PIECE -> idle reclamation attempt -> later command; partial/full import -> process termination before terminal publication -> restart/retry; terminal publication -> WAL failure/restart or dispatch queue overflow -> replica catch-up; and snapshots spanning these transitions. Assert installed-file identity/count, relevant publication effects, recovered task state, and replica catch-up rather than only successful status codes. Re-run the focused tests and the restart ClusterIT on the completed implementation, then perform a complete independent review before marking ready.
Validation and revision limits
aa6b6acd01da446363f2e337c0163aaeeec0f4d6.git diff --checkpassed.6e7448d9bebd644259b695dfc8774d3d9f91252e. The refreshed targetmasterwas54bff3d9a421f9ba148fda3a26626df570422f54when this draft was prepared. The candidate was not rebased, and an integration merge with that newer target has not been validated.This PR has:
Key changed classes
DataExecutionVisitor,DataRegionStateMachine,LoadFileNotReadyException,DataRegion,SnapshotTaker,LoadTsFileManager,TsFileWriterManager,LoadTsFileSnapshot, andLoadTsFileCleaner, plus regression tests.