State-sync bootstrap is structurally broken on ceremony chains: empty genesis validator set crashes seid (nil proposer) and wedges the node

[bdchatham]

Summary

A SeiNode with spec.fullNode.snapshot.stateSync on a ceremony chain (harbor ephemeral) crash-loops deterministically on first boot and can never recover — on any image. Reproduced on eng-yiming/yiming-migration-mock-4-rpc-1 (harbor), including a clean-room re-provision on a fresh PVC that failed on the very first boot before any snapshot discovery.

Error: error starting node: cannot load state: fromProto: validatorSet proposer error: nil validator

and on every subsequent restart, earlier:

Error: error creating node: LoadStateFromDBOrGenesisDocProvider(): fromProto: validatorSet proposer error: nil validator

Root cause (sei-tendermint)

1. With state-sync enabled, OnStart skips the handshake/InitChain (sei-tendermint/node/node.go OnStart: "state sync will cover initialization the chain").
2. A ceremony chain's assembled genesis has an empty tendermint validators: [] — validators materialize from gentxs at InitChain, which state-sync just skipped. Node construction has already saved the genesis-derived state (empty valset, nil proposer) to the state store.
3. OnStart then reloads state (node.go ~L451: n.stateStore.Load()); the proto round-trip calls ValidatorFromProto(vp.GetProposer()) (types/validator_set.go ~L883) → nil proposer → fatal.
4. Restarts fail earlier at LoadStateFromDBOrGenesisDocProvider (node.go ~L732) because the state store is now non-empty — i.e. a single failed first boot poisons the volume; recovery requires deleting the SeiNode + PVC.

Canonical chains are immune: their genesis carries the validator set (pacific-1 embeds 25 validators). Only ceremony chains (empty validators until InitChain) hit this — which is exactly the population the new spec.fullNode.snapshot.rpcServers field (spec-declared witnesses, replacing the canonical-syncer registry) exists to serve. As shipped, the feature green-lights a bootstrap that cannot succeed.

Timeline of the field case

• Node requested stateSync; controller correctly gated on NoSyncersConfigured (chain has no registry entry).
• Witnesses declared via rpcServers (harbor-engineering-workspace docs: LLD for SeiNode import-existing-storage feature #112): gate cleared, plan ran all tasks Complete, sidecar state-sync configure fetched a valid trust point (height 74523, hash verified) from the chain's own validators.
• seid then crash-looped as above. Full teardown (SeiNode + cascaded PVC) + Flux re-provision reproduced the identical crash on boot 1 of a pristine volume with zero state-sync progress lines.
• Workaround: drop stateSync, block-sync from genesis (harbor-engineering-workspace feat: SeiNode import existing PVC + ensure-data-pvc create-path fix #113).

Suggested fixes (layered)

1. sei-tendermint: when state-sync will bootstrap the chain, don't persist (or don't reload via the strict fromProto path) the genesis-derived empty-valset state — or make an empty validator set survive the proto round-trip until state-sync installs the real one. This is the root fix.
2. Controller: the StateSyncReady gate should refuse (or warn in the condition) when the target chain's genesis carries an empty validator set — i.e. any ceremony chain — until (1) lands. Today the gate validates witness count, then hands the node a guaranteed wedge.
3. Secondary bug: a first-boot state-sync failure of any kind leaves an unloadable state DB with no auto-recovery; the sidecar could detect this exact error signature and reset the tendermint state rather than requiring PVC deletion.
4. Runbook: troubleshooting-seinode.md (harbor-dev skill) has no entry for this signature; add symptom → cause → recovery once the behavior is settled.

Also observed

The pod existed (Pending on the missing PVC) for the entire 3.5h the plan was gated — the running harbor controller applies the STS before the gate clears. origin/main's "hold STS while gate blocks plan" (#456 era) fixes this; noting for deploy prioritization.

🤖 Generated with Claude Code

[bdchatham]

Update from review discussion (h/t Yiming): Sei's own launch tooling has always populated the top-level validators array in genesis.json — both sei-infra genesis.sh and localnode step3_add_validator_to_genesis.sh run a jq loop deriving each validator's pubkey + power (delegation / 10^6) from the gentxs before collect-gentxs.

The harbor ceremony (seictl assemble-genesis) follows the vanilla cosmos genutil shape instead — GenAppStateFromConfig only, validators: [] left empty for InitChain to fill. That shape is valid upstream (every block-sync node works), but it means Sei-launched chains never produced an empty-validators genesis in the wild — which is why the sei-tendermint round-trip bug went unnoticed until harbor's ceremony chains met state-sync.

This adds a third fix layer, arguably the fastest unblock: teach seictl's assemble task to populate genesis.validators from the collected gentxs (typed Go via MsgCreateValidator — pubkey + tokens/PowerReduction — not jq), matching Sei launch convention. That makes ceremony chains state-syncable today, with existing seid images, independent of the sei-chain#3724 release train, and it also gives followers a meaningful trust anchor shape identical to canonical chains. The seid fix (#3724) remains correct on its own terms — a node shouldn't wedge on a genesis shape upstream considers valid — and the controller gate (layer 2) can then be scoped to images older than the fix rather than all ceremony chains.

[bdchatham]

Systems-engineer validation of the layered plan completed (all claims code-verified). Refinements that change the layer-3 (seictl ceremony) implementation contract:

Confirmed: the ceremony fix unblocks pre-fix seid images. The state-sync path is genesis-valset-agnostic (zero genDoc.Validators references in internal/statesync/; trust anchor is config-driven TrustHeight/TrustHash; Bootstrap overwrites at snapshot height; handshake skipped). The nil-proposer load was the only ceremony-specific blocker, so populating genesis.validators makes existing images state-sync cleanly.

Implementation contract (proto-equality, not approximation). The handshaker consistency-checks a populated genesis valset against the InitChain response — sorted, proto.Equal per update, hard error on mismatch (sei-tendermint/internal/consensus/replay.go:212-224). It never merges or replaces. So the assemble-task population must byte-match ApplyAndReturnValidatorSetUpdates output:

• power = floor(self_delegation / PowerReduction) — take PowerReduction from the staking keeper/params path, do not hardcode 1e6
• mirror bonded-set semantics: cap at MaxValidators (top-N by power) and exclude power-0 validators (sub-PowerReduction self-delegation), not a raw all-gentxs loop — either divergence trips the handshake length check
• block-1 ValidatorsHash derives from the InitChain response either way, so a correct population changes no consensus-observable value

Co-requisite: the genesis-hash wiring (originally parked as a separate gap). Populating the array changes the assembled genesis bytes. Today the controller sends a zero-valued ConfigureGenesisTask{} (internal/planner/planner.go:645) and never stamps status.genesisHash/genesisS3URI, so nothing fences a mixed old/new genesis for the same chain-id — a follower could fetch bytes that don't match what the validators assembled. The ExpectedGenesisHash plumbing (assemble result → status → follower ConfigureGenesis) must land with the ceremony change, and the population must apply to new ceremonies only (key off the assembly idempotency marker).

Layer-2 gate scope shrinks accordingly: only chains whose already-assembled genesis has an empty valset need the refusal; new ceremonies and post-#3724 images don't.

Placement verified: population slots after applyOverrides (which rewrites only AppState — genDoc.Validators survives) and before uploadGenesis (hashes final bytes), sidecar/tasks/assemble_genesis.go:169-173.