Checkpoint truncates the WAL at the min LSN of only the responding cores — a slow core loses acknowledged writes

[habibtalib]

Source: static engine audit, 2026-07-05 (durability lens), verified against main @ d92ede42
Severity: Critical — silent loss of acknowledged writes on a routine checkpoint.

Summary

The checkpoint routine truncates the WAL at the minimum flush LSN of only the cores that responded in time. A core that is alive but slow (misses the 30 s core_timeout) is pushed to failed_cores and excluded from the truncation LSN — but truncation still proceeds. That core's acknowledged-but-unflushed WAL records, which sit below the responding cores' min LSN, are deleted and become unrecoverable on restart.

Location

• nodedb/src/control/checkpoint_manager.rs:167-220

Failure scenario

1. 4 Data-Plane cores. Core 3 hits a GC/scheduler pause and misses its checkpoint-response deadline.
2. failed_cores = [3]; the code logs warn!("… using partial results") (line 167-173) and continues.
3. global_lsn = checkpoint_lsns.iter().min() over cores 0,1,2 only (line 181).
4. wal.truncate_before(global_lsn) deletes sealed segments below that LSN (line 220) — including records core 3 acknowledged to clients but had not yet flushed past.
5. On the next restart, core 3's committed writes are gone. No error is surfaced.

Fix direction

Do not truncate when !failed_cores.is_empty() — skip WAL truncation for this cycle and retry next cycle once all cores respond. Add a fault-injection test (a slow/timed-out core) asserting its acknowledged writes survive a restart.

Part of the v1.0 production-hardening workstream (.planning/v1-production-readiness/), phase P1.