Data-group InstallSnapshot advances the Raft index without transferring state-machine data — silent divergence

[habibtalib]

Source: static engine audit, 2026-07-05 (cluster/consensus lens), verified against main @ d92ede42
Severity: Critical — silent state-machine divergence on a lagging follower.

Summary

Data-group InstallSnapshot fast-forwards Raft's commit_index/last_applied to last_included_index and compacts the log, but transfers no state-machine data — the sender hardcodes snapshot_bytes: &[] and the receiver ignores req.data. The assembled .snap file is written but never read back (the only .snap reference is the write site). The engine never produces real snapshot bytes, so the whole snapshot-install path is a non-functional stub for data groups. The code comments the hole directly.

Location

• nodedb-cluster/src/raft_loop/tick.rs:290 (snapshot_bytes: &[])
• nodedb-raft/src/node/rpc/install_snapshot.rs:34-52 (ignores req.data)
• nodedb-cluster/src/raft_loop/handle_rpc.rs:227-237 (comment: "does NOT restore the data-plane state machine … silent data-loss-shaped bug")

Failure scenario

1. A follower falls behind the leader's log-retention/compaction window.
2. Leader sends InstallSnapshot; follower's Raft last_applied jumps to N.
3. Follower's data plane still reflects only pre-snapshot state → permanent silent divergence between Raft index and actual data. Queries wake and read missing state.

Fix direction

Either (a) transfer + restore real state-machine bytes for data groups, or (b) refuse to advance last_applied when no data payload is present (fail loud instead of diverging). Add a test that compacts past a follower and verifies the follower's data matches the leader.

v1.0 production-hardening, phase P2.