Skip to content

[WAL] Clean up failed WAL creation and make checkpoint rolls atomic - #19287

Open
HTHou wants to merge 4 commits into
apache:masterfrom
HTHou:codex/fix-wal-init-checkpoint
Open

HTHou wants to merge 4 commits into
apache:masterfrom
HTHou:codex/fix-wal-init-checkpoint

Conversation

@HTHou

@HTHou HTHou commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Description

WAL initialization

If a V2/V3 header write or an extension provider fails after opening a new WAL, the writer closes the file but can leave the empty file or partial header on disk. For example, if initialization writes two bytes to _1-1-1.wal and then throws, the next roll rejects that nonempty successor.

The writer now uses exclusive creation to identify files it owns. On failure it closes the channel and format resources, removes a file created by that attempt, and propagates the original exception with any cleanup errors suppressed. Retrying can create _1-1-1.wal again. Existing files retain their append behavior and are kept on failure.

Checkpoint rolling and recovery

If writing GLOBAL_MEMORY_TABLE_INFO fails after writing the maximum memtable ID to _1.checkpoint, the old roll path can still force the incomplete file and delete _0.checkpoint. Recovery can then skip an already forced insert at timestamp 10 for memtable 1 because it lost the mapping from _0.checkpoint.

The manager writes and forces the complete recovery snapshot under a temporary name. It then renames the file and synchronizes the parent directory before switching writers and retiring the old checkpoint. On Windows, the rename uses a write-through file handle. If synchronizing publication fails, the manager attempts a durable rename back to the temporary name, keeping _0.checkpoint active. If the rollback fails, it keeps the previous checkpoint on disk and rejects further WAL writes until recovery. An oversized snapshot waits for an incremental checkpoint before another roll.

The WAL batch also depends on the active checkpoint's writes and force completing. For example, if forcing the create record for memtable 2 fails, the batch returns the original I/O exception before appending the insert to the WAL. The prior durable checkpoint can still replay earlier acknowledged inserts.

Legacy files can contain complete create and flush records even if their initial snapshot write failed. For example, when _1.checkpoint contains a create record for memtable 2 and its WAL insert was forced, recovery applies that record and replays the insert both with and without _0.checkpoint. Recovery applies records in version order within the newest contiguous run of readable versions, starting at its latest complete snapshot when available. For example, when _1.checkpoint is missing, recovery retains the create record in _2.checkpoint without reviving a stale memtable mapping from _0.checkpoint. Recovery ignores temporary checkpoint candidates and removes them after successful replay, before reusing an IoTConsensus WAL directory.

A retained _0.checkpoint can identify a flushing memtable 1, and the next legacy checkpoint can identify memtable 2 for the same unsealed TsFile. With one forced insert per memtable at timestamps 10 and 20, recovery previously reported success with just one row. Both IDs now share one recovery performer, and the recovered TsFile contains both timestamps. A performer initialization failure keeps the checkpoints and WAL available for retry.

Validation

  • DataNode Spotless passed.

  • On macOS with JDK 17, DataNode WAL tests reported 186 passed and 2 existing performance tests skipped. This includes the WALFormatExtensionTest wrapper (20 isolated cases), 14 checkpoint rolling and replay tests, and the two-memtable retry regression that verifies the recovered TsFile contents.

  • Full reactor mvn test-compile -DskipTests passed with the default locale and with -P with-zh-locale.

  • Self-reviewed the patch.

  • Added regression tests for the new code paths.

@HTHou HTHou left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The checkpoint-roll and WAL format extension tests passed in a focused local run. Two additional legacy recovery cases replay on the base recovery code and skip the write on this PR. I'd fix that regression and make checkpoint publication durable before merging.

@HTHou HTHou left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review of 6636a16. The checkpoint, WAL recovery, and WAL format extension suites passed in a focused local run. Two additional replay cases passed with the base implementations and skipped the insert on this revision. Details are inline.

@HTHou HTHou left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review of 87d053f. The active checkpoint sync failure and version-gap regression cases pass. The inline finding concerns two valid memtables that share an unsealed TsFile.

}
continue;
}
readers.addFirst(reader);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Replay all active memtables for the same TsFile. A retained _0.checkpoint can contain memtable 1 while the consecutive _1.checkpoint contains a readable create record for memtable 2 after a legacy snapshot write failed. Both memtables can target the same unsealed TsFile while the older one is still flushing. Merging the checkpoints then returns both IDs. WALNodeRecoverTask removes the recovery performer by TsFile path on its first lookup, so the second ID has no performer and its WAL entries are skipped. In a local test with one forced insert for each ID, this revision returned SUCCESS and replayed only memtable 1. With the base checkpoint reader, memtable 2 was replayed. Please route the valid memtable IDs for each TsFile to one performer and run its recovery lifecycle once per file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant