Raft HardState (current_term/voted_for) is never persisted — split-brain on restart

[habibtalib]

Source: static engine audit, 2026-07-05 (cluster/consensus lens), verified against main @ d92ede42
Severity: Critical — split-brain / committed-entry divergence after a restart.

Summary

Raft's HardState (current_term, voted_for) is never durably persisted. persist_hard_state only stages it into Ready.hard_state, and no consumer ever writes it: save_hard_state has zero production callers (verified by full-repo grep — only trait def, impls, and #[cfg(test)] roundtrips). The multi-raft tick loop reads only .messages/.committed_entries/.snapshots_needed from Ready and drops .hard_state.

Compounding it: even the staged state would be flushed on the next tick, but vote/append RPC responses are returned on the wire before that — violating Raft's "persist to stable storage before replying" rule.

Location

• nodedb-raft/src/node/internal.rs:366 (stages only)
• nodedb-cluster/src/raft_loop/tick.rs:55-75 (consumer ignores .hard_state)
• nodedb-cluster/src/raft_loop/handle_rpc.rs:70-79 (replies before persist)
• nodedb-raft/src/storage.rs:28 (save_hard_state — no prod caller)

Failure scenario

1. Node votes for candidate A in term 5 → voted_for=A, current_term=5 held in memory only.
2. Node crashes and restarts → load_hard_state returns HardState::default() (term 0, voted_for 0).
3. Candidate B requests a vote in term 5 → node has no record of having voted → grants it.
4. Two leaders in term 5 → divergent committed entries. Term also regresses to 0 on every restart (monotonicity violation), only re-learned from an incoming leader.

Fix direction

Persist HardState to stable storage in the RPC path, before returning any vote/append response (drain Ready.hard_state → save_hard_state synchronously). Add a test: "vote A, crash, restart, must NOT grant a vote to B in the same term."

v1.0 production-hardening, phase P2 (cluster consensus safety).