Skip to content

Conversation

@cjen1-msft
Copy link
Contributor

@cjen1-msft cjen1-msft commented Aug 15, 2025

This PR is the reification of: #7003

The idea is that if a service has fully crashed, it should be able to heal itself so long as it isn't too damaged.
Specifically, the restarting nodes should gossip the knowledge they have locally to try and elect the replica with the best local state.

The result is that after the self-healing-open one replica is chosen to recover and open, while all others restart to then join it.

The protocol can be roughly surmised as:

  • Start up.
  • Gossip your state (claimed length of ledger) and authenticate other replicas attestation and identities.
  • Once you have heard from everyone you expect to hear from, vote for the node with the longest ledger (ties broken by identity).
  • If you receive votes from a majority of your expected cluster, transition_to_open and broadcast IAmOpen to the other nodes.
  • If you receive IAmOpen from a trusted node, restart and join it.

This still requires the submission of ledger recovery shares, however if local sealing is available those can be used instead.

UPDATE:
To be specific on when and where things happen.

  • After public ledger recovery, during create rpc it will reset the self-healing-open state and once that state is committed start several timers, one to resend gossips/votes/iamopen messages, and one to advance the failover state-machine
  • All messages include an attestation that is validated on receipt, and the handlers all make a change to the KV state and then try to advance the protocol.
  • Finally once the protocol is complete either:
    • The chosen replica runs transition-to-open and on the next timer tick stops the timers.
    • The others shutdown, to later be restarted by the host's orchestrator where they will try and join the network, landing on the chosen replica.

@achamayou achamayou enabled auto-merge (squash) December 28, 2025 13:57
@achamayou achamayou merged commit 337d41d into microsoft:main Dec 28, 2025
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-long-test Run Long Test job

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants