Skip to content

[Load] Retry incomplete replicated LOAD commands - #19241

Draft
Caideyipi wants to merge 10 commits into
apache:masterfrom
Caideyipi:fix/load-consensus-command-retry
Draft

Caideyipi wants to merge 10 commits into
apache:masterfrom
Caideyipi:fix/load-consensus-command-retry

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

A replicated LOAD PREPARE or COMMIT can arrive before a missing PIECE is retried after reconnect. Returning a permanent LOAD error lets consensus retire the command before the replica has finished the task. This draft retries only temporarily incomplete replicated commands and preserves completed task identities across retries and restarts.

This is an incomplete fix and is not ready to merge. The remaining recovery requirements below belong to the same LOAD retry path. Some underlying behaviors predate this patch, but the retry guarantee still depends on addressing them.

Changes currently implemented

  • Introduce LoadFileNotReadyException and map it to WRITE_PROCESS_REJECT for replicated PREPARE/COMMIT only. Permanent failures and local command failures retain their existing error classification.
  • Persist COMMIT/ABORT outcomes outside task staging, include them in LOAD snapshots, and use the bounded in-memory history as a cache. Record empty ABORT outcomes and ignore late replicated PIECEs for completed tasks.
  • Coordinate staging snapshot copies with COMMIT/ABORT. Migrate legacy terminal records before retention listener registration, preserve staging when migration fails, and fence lazy manager initialization against region deletion.
  • Correct the lazy-initialization lock order relative to snapshot flushing.
  • Include preserved legacy COMMIT/ABORT records in snapshots when terminal-history migration cannot publish, including before lazy manager initialization.

TODO before marking ready, and how to adapt

  1. Preserve acknowledged, unfinished staging. Adapt LoadTsFileManager.stop, TsFileWriterManager.close, and LoadTsFileCleaner.reclaimIfIdle to distinguish coordinator temporary files from consensus-owned accepted state. Ordinary shutdown must close handles while preserving recoverable payload and progress. Idle time alone must not delete accepted replicated data: remote requests use a negative local search index, so the existing watermark guard does not protect them. Reclamation needs an explicit terminal decision, a proven safe replay frontier, or an implemented payload reconstruction protocol. Otherwise, a sender that already retired the PIECE batch can only retry PREPARE/COMMIT against missing data indefinitely.

  2. Make COMMIT import recoverable and idempotent. Adapt LoadTsFileManager.loadAll, TsFileWriterManager.loadAll, and the DataRegion import path. Persist an import plan/COMMITTING state with stable per-task, per-partition installation identities and enough information to recognize published files after restart. Recovery must finish missing installations without reinstalling completed ones or repeating their publication effects. The current sequence imports files before persisting COMMIT, while ordinary import allocates a new version/name each time. Keeping the staged source does not by itself make replay safe. Moving the final COMMIT record before import would leave a different missing-data window and is insufficient.

  3. Coordinate the terminal decision with replication WAL durability. Adapt the terminal-hit paths and WAL logging in LoadTsFileManager.loadAll/deleteAll. Currently the terminal record can become durable before the command reaches WAL, and a subsequent terminal-hit return skips WAL logging. Define a replayable decision/intent and local application-completion protocol so successful retries also satisfy outstanding replication obligations. An in-memory dispatcher re-offer is insufficient across restart or queue overflow. The design must reconcile stable decision identity, WAL replay, and application progress; a local terminal file alone must not authorize success when durable replication information is missing.

  4. Apply one state/ownership model to snapshots and lifecycle operations. A bounded direction is a region-local task protocol such as RECEIVING -> PREPARED -> COMMITTING -> COMMITTED, with explicit permitted ABORTED transitions. Define the durability and acknowledgement conditions for every transition before implementation. Capture data files, active task/import state, terminal outcomes, and replay position from a consistent boundary. Preserve the existing staging-copy boundary, migration-before-listener ordering, and consistent lock order. Separate ordinary stop from region deletion. Terminal-history garbage collection needs a proven replay frontier; arbitrary TTL or an in-memory receive index is insufficient. Measure terminal-record growth and staging/snapshot copy latency. This does not require adding cross-region atomic LOAD or rollback semantics.

  5. Validate the recovery boundaries above. Add failure-oriented coverage for acknowledged PIECE -> normal stop/restart -> later PREPARE/COMMIT; acknowledged PIECE -> idle reclamation attempt -> later command; partial/full import -> process termination before terminal publication -> restart/retry; terminal publication -> WAL failure/restart or dispatch queue overflow -> replica catch-up; and snapshots spanning these transitions. Assert installed-file identity/count, relevant publication effects, recovered task state, and replica catch-up rather than only successful status codes. Re-run the focused tests and the restart ClusterIT on the completed implementation, then perform a complete independent review before marking ready.

Validation and revision limits

  • Current candidate: aa6b6acd01da446363f2e337c0163aaeeec0f4d6.
  • Standard focused clean local Maven reactor (build cache disabled): 81 tests passed, with zero failures, errors, or skips. Coverage includes state-machine classification, LOAD manager/snapshot/cleaner behavior, selected DataRegion lifecycle tests, and selected IoTDB snapshot tests. All 15 test methods added in this diff were run locally at this revision.
  • Spotless, Checkstyle, forbidden-API checks, and git diff --check passed.
  • These results do not establish crash safety for the unfinished TODOs above. No current-candidate ClusterIT, Chinese-locale compilation, process-kill crash test, or performance benchmark has been run. Full CI results for this revision have not been evaluated.
  • The candidate was developed and tested on recorded baseline 6e7448d9bebd644259b695dfc8774d3d9f91252e. The refreshed target master was 54bff3d9a421f9ba148fda3a26626df570422f54 when this draft was prepared. The candidate was not rebased, and an integration merge with that newer target has not been validated.
  • Review findings remain outstanding at the design level; this draft has not passed an issue-free independent review.

This PR has:

  • Added regression unit tests and run every newly added test locally.
  • Completed the recovery protocol TODOs above.
  • Validated the completed candidate with the restart ClusterIT.
  • Passed a complete independent review without unresolved behavioral or test-logic findings.

Key changed classes

DataExecutionVisitor, DataRegionStateMachine, LoadFileNotReadyException, DataRegion, SnapshotTaker, LoadTsFileManager, TsFileWriterManager, LoadTsFileSnapshot, and LoadTsFileCleaner, plus regression tests.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Implemented-draft re-review

Reviewed a7d199b65162c9d62bad1c4e64883e33472fc58f against the full target diff and relevant context, excluding the documented TODOs. This round found two actionable issues:

  1. [P1] Snapshots omitted legacy terminal decisions when migration could not publish. Recovery deliberately preserves legacy COMMIT/ABORT progress in staging after a failed .finished migration, but those tasks are outside the active manager map. The original history capture only included published history, allowing a successful snapshot to lose the only terminal identity. Fixed by capturing unmanaged legacy terminal progress under its task ID, including when no lazy manager exists. Added snapshot/restore coverage for both outcomes, failed publication, absent/present manager, and mismatched legacy progress filenames.
  2. [P2] The permanent COMMIT exception test did not inject a COMMIT exception. The original test stubbed only PREPARE; COMMIT returned Mockito's default false, leaving its exception-classification path untested. Fixed by injecting the same permanent COMMIT exception and checking its returned message.

Both fixes are in aa6b6acd01da446363f2e337c0163aaeeec0f4d6. Final local HEAD, confirmed source-branch HEAD, and confirmed PR HEAD all match that revision. The PR remains draft: the existing TODOs are unchanged, and this round's behavioral/test findings require a fresh independent re-review after the update.

Validation at the final revision:

  • Clean 28-module local Maven reactor with JDK 17, Maven 3.8.8, the integration-test profile, and build cache disabled: 81 tests passed; 0 failures, 0 errors, 0 skips. This covers the four focused LOAD/state-machine suites plus the selected DataRegion lifecycle and IoTDB snapshot cases. All 15 newly added test methods in the full PR diff ran locally at this revision.
  • Spotless apply, the reactor's Checkstyle/forbidden-API checks, and git diff --check passed.
  • The remote attempt was stopped before compilation because public dependency-repository connections timed out; its result is unverified. The passing result above is from the local fallback.
  • No current-revision ClusterIT, Chinese-locale build, process-kill crash test, or performance benchmark was run. Full CI results for this revision have not been evaluated. These checks do not establish the unfinished TODO recovery guarantees.

The linked lifecycle IT failure in #19051 does not exercise this retry path: it uses SimpleConsensus and fails before the intentional crash. Its global transferFile breakpoint also suspends the canary's local loader while holding the DataRegion write lock. The log shows read-lock retries for a minute, followed by immediate successful loading when debugger cleanup releases it. The conditional test transfer was therefore not applied; #19051 is unchanged.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Implemented-draft re-review

Reviewed the complete PR diff and relevant context at aa6b6acd01da446363f2e337c0163aaeeec0f4d6, excluding the documented TODOs. This round found one actionable issue: when a legacy COMMIT/ABORT record was readable in staging but migration into .finished failed, LoadTsFileManager.getTerminalOp() propagated the migration IOException. Replicated retries for an already terminal task then returned permanent LOAD_FILE_ERROR instead of honoring the persisted decision.

Fixed by returning the readable legacy terminal when publishing its migrated copy fails, while preserving the staging record so migration can be retried. Added testReplicatedRetriesUseLegacyTerminalWhenMigrationFails, covering remote PIECE, PREPARE, COMMIT, and ABORT against a legacy ABORT with an unavailable history directory.

Validation: mvn spotless:apply -P with-integration-tests passed. The focused clean 28-module reactor command mvn clean test -pl iotdb-core/datanode -am -P with-integration-tests -Dtest=LoadTsFileManagerTest -Dsurefire.failIfNoSpecifiedTests=false passed with 40 tests, 0 failures, 0 errors, and 0 skips. Datanode main and test sources were recompiled; 42 reactor goals were served from the Maven build cache. Checkstyle, Spotless, forbidden-API, and git diff --check passed. Restart ClusterIT, Chinese-locale build, process-kill recovery test, performance test, and full CI were not evaluated; no CI was awaited.

Reviewed SHA: aa6b6acd01da446363f2e337c0163aaeeec0f4d6.
Final local SHA: a730680121d64c526ea2c41efc8961f9b6eced6b.
Confirmed source branch and PR HEAD: a730680121d64c526ea2c41efc8961f9b6eced6b.

The PR remains draft because the documented TODOs are still incomplete.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Independent re-review of the implemented draft

Completed one independent re-review of the full 13-file PR diff and relevant context at a730680121d64c526ea2c41efc8961f9b6eced6b, excluding the TODOs explicitly documented in the PR description. No new actionable source-behavior or test-logic findings were confirmed within that scope; no fixes or branch updates were needed. This is not approval of the unfinished recovery protocol.

The review covered retryable versus permanent PREPARE/COMMIT failures, persisted COMMIT/ABORT identity and late replicated commands, snapshot selection/copy boundaries, legacy migration failures, cleaner/retention ordering, lazy manager deletion fencing, lock ordering, and regression assertions. In particular, the earlier fixes remain intact:

  • Preserved legacy COMMIT/ABORT decisions are included in snapshot history under their task IDs when migration cannot publish, with and without an initialized manager.
  • The permanent COMMIT exception test actually injects the COMMIT exception and checks the returned message.
  • A readable legacy decision remains authoritative when migration fails. The failure leaves staging intact and does not cache an unsuccessful migration, allowing later terminal lookups to retry publication.

Validation: this round passed git diff --check for the full PR diff and confirmed a clean review worktree. No Maven tests were rerun because this round changed no source or tests. The previously reported evidence remains revision-specific: at a730680, the clean focused 28-module reactor ran LoadTsFileManagerTest with 40 tests passed, 0 failures/errors/skips, recompiled datanode main and test sources, and reused 42 reactor goals from the Maven build cache; Spotless, Checkstyle, forbidden-API checks, and git diff --check passed. The earlier 81-test focused run and 15 added tests were run at aa6b6acd, not rerun here. All added tests have previously been run locally. ClusterIT, Chinese-locale compilation, process-kill recovery testing, performance testing, and full CI were not evaluated in this round; no CI was awaited or polled.

The PR and source branch were refreshed and reconfirmed. The recorded PR base is 54bff3d9a421f9ba148fda3a26626df570422f54; the merge base of the reviewed diff is 6e7448d9bebd644259b695dfc8774d3d9f91252e; refreshed target master is d47472156d793fdaa13b8f87051f363222eddd50. Integration with the newer target was not built or tested.

  • Reviewed SHA: a730680121d64c526ea2c41efc8961f9b6eced6b.
  • Final local SHA: a730680121d64c526ea2c41efc8961f9b6eced6b.
  • Confirmed source-branch SHA: a730680121d64c526ea2c41efc8961f9b6eced6b.
  • Confirmed PR HEAD: a730680121d64c526ea2c41efc8961f9b6eced6b.

The PR remains draft. The documented TODOs are still incomplete and were neither expanded nor treated as resolved by this review.

@hongzhi-gao

Copy link
Copy Markdown
Contributor

Would it make sense to split this into two dependent PRs, with the recovery guarantees landing before the retry behavior?

  1. Recovery and idempotency foundations: preserve acknowledged unfinished staging across shutdown and cleanup; make partial imports recoverable without duplicate installation or publication effects; coordinate terminal outcomes with replication WAL durability; and keep snapshots and lifecycle operations consistent with that protocol. The related lock-order and initialization/deletion fixes could belong here.
  2. Retry behavior: distinguish temporarily incomplete replicated PREPARE/COMMIT commands from permanent failures, return the retryable status, and validate eventual completion through disconnect/reconnect and restart ClusterITs.

The first PR would need to be independently safe and testable. If there is no safe intermediate implementation, keeping one PR with clearly separated commit groups would be preferable to forcing the split.

As a follow-up, it may also be worth exploring targeted retransmission of missing PIECEs: let the receiver identify missing pieces and resend only those, rather than retransmitting the original consensus batch. This would require stable piece identities, retained retransmission sources, and idempotent duplicate handling. I would keep it outside this fix; it would not replace the recovery and WAL guarantees above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants