[Epic] P1 — Durability Correctness & Crash Safety (v0.5)

[habibtalib]

Epic — v0.5 production-hardening, phase P1: Durability Correctness & Crash Safety

Goal: a crash at any point — mid-checkpoint, mid-flush, with a slow core — never loses an acknowledged write, and recovery replays exactly the committed suffix.

Critical (filed)

• Checkpoint truncates the WAL at the min LSN of only the responding cores — a slow core loses acknowledged writes #160 — checkpoint truncates WAL at the min LSN of only responding cores → slow core loses acknowledged writes

High

• O_DIRECT replay armed to corrupt — the WAL reader skips no inter-batch alignment padding; an O_DIRECT segment with >1 flush batch replays only the first batch. Masked only because bootstrap/wal_init.rs:25 hardcodes use_direct_io=false. nodedb-wal/src/reader.rs:83-101, writer.rs:388-429.

Medium

• DWB torn-write recovery over-advances the read offset (double-adds payload.len()) → discards records after a mid-segment recovery. nodedb-wal/src/reader.rs:118-129.
• Mid-file WAL corruption is indistinguishable from a torn tail → silently truncates the committed suffix. recovery.rs:63-88.
• ENOSPC on WAL append: retained buffer + already-advanced LSN; retry/torn-tail interaction untested. writer.rs:388-423.

Test coverage

• No nodedb-wal failpoints — add crash injection between checkpoint-marker and truncate, between DWB and WAL fsync, mid-truncate_segments.
• No multi-batch O_DIRECT replay test.

Note: verify #149 (document_strict data loss on restart) is not a shared root cause with the above during planning.

Tracked in .planning/v1-production-readiness/AUDIT.md.