Repository navigation
Conversation
HTHou
left a comment
There was a problem hiding this comment.
The checkpoint-roll and WAL format extension tests passed in a focused local run. Two additional legacy recovery cases replay on the base recovery code and skip the write on this PR. I'd fix that regression and make checkpoint publication durable before merging.
HTHou
left a comment
There was a problem hiding this comment.
Follow-up review of 6636a16. The checkpoint, WAL recovery, and WAL format extension suites passed in a focused local run. Two additional replay cases passed with the base implementations and skipped the insert on this revision. Details are inline.
HTHou
left a comment
There was a problem hiding this comment.
Follow-up review of 87d053f. The active checkpoint sync failure and version-gap regression cases pass. The inline finding concerns two valid memtables that share an unsealed TsFile.
| } | ||
| continue; | ||
| } | ||
| readers.addFirst(reader); |
There was a problem hiding this comment.
[P2] Replay all active memtables for the same TsFile. A retained _0.checkpoint can contain memtable 1 while the consecutive _1.checkpoint contains a readable create record for memtable 2 after a legacy snapshot write failed. Both memtables can target the same unsealed TsFile while the older one is still flushing. Merging the checkpoints then returns both IDs. WALNodeRecoverTask removes the recovery performer by TsFile path on its first lookup, so the second ID has no performer and its WAL entries are skipped. In a local test with one forced insert for each ID, this revision returned SUCCESS and replayed only memtable 1. With the base checkpoint reader, memtable 2 was replayed. Please route the valid memtable IDs for each TsFile to one performer and run its recovery lifecycle once per file.
Description
WAL initialization
If a V2/V3 header write or an extension provider fails after opening a new WAL, the writer closes the file but can leave the empty file or partial header on disk. For example, if initialization writes two bytes to
_1-1-1.waland then throws, the next roll rejects that nonempty successor.The writer now uses exclusive creation to identify files it owns. On failure it closes the channel and format resources, removes a file created by that attempt, and propagates the original exception with any cleanup errors suppressed. Retrying can create
_1-1-1.walagain. Existing files retain their append behavior and are kept on failure.Checkpoint rolling and recovery
If writing
GLOBAL_MEMORY_TABLE_INFOfails after writing the maximum memtable ID to_1.checkpoint, the old roll path can still force the incomplete file and delete_0.checkpoint. Recovery can then skip an already forced insert at timestamp 10 for memtable 1 because it lost the mapping from_0.checkpoint.The manager writes and forces the complete recovery snapshot under a temporary name. It then renames the file and synchronizes the parent directory before switching writers and retiring the old checkpoint. On Windows, the rename uses a write-through file handle. If synchronizing publication fails, the manager attempts a durable rename back to the temporary name, keeping
_0.checkpointactive. If the rollback fails, it keeps the previous checkpoint on disk and rejects further WAL writes until recovery. An oversized snapshot waits for an incremental checkpoint before another roll.The WAL batch also depends on the active checkpoint's writes and force completing. For example, if forcing the create record for memtable 2 fails, the batch returns the original I/O exception before appending the insert to the WAL. The prior durable checkpoint can still replay earlier acknowledged inserts.
Legacy files can contain complete create and flush records even if their initial snapshot write failed. For example, when
_1.checkpointcontains a create record for memtable 2 and its WAL insert was forced, recovery applies that record and replays the insert both with and without_0.checkpoint. Recovery applies records in version order within the newest contiguous run of readable versions, starting at its latest complete snapshot when available. For example, when_1.checkpointis missing, recovery retains the create record in_2.checkpointwithout reviving a stale memtable mapping from_0.checkpoint. Recovery ignores temporary checkpoint candidates and removes them after successful replay, before reusing an IoTConsensus WAL directory.A retained
_0.checkpointcan identify a flushing memtable 1, and the next legacy checkpoint can identify memtable 2 for the same unsealed TsFile. With one forced insert per memtable at timestamps 10 and 20, recovery previously reported success with just one row. Both IDs now share one recovery performer, and the recovered TsFile contains both timestamps. A performer initialization failure keeps the checkpoints and WAL available for retry.Validation
DataNode Spotless passed.
On macOS with JDK 17, DataNode WAL tests reported 186 passed and 2 existing performance tests skipped. This includes the
WALFormatExtensionTestwrapper (20 isolated cases), 14 checkpoint rolling and replay tests, and the two-memtable retry regression that verifies the recovered TsFile contents.Full reactor
mvn test-compile -DskipTestspassed with the default locale and with-P with-zh-locale.Self-reviewed the patch.
Added regression tests for the new code paths.