fix(nightly): unblock the chain-upgrade suite on a pre-giga image - #506
Conversation
Two independent fixes to TestNightlyChainUpgrade. Only the first addresses
the failure the suite shows today.
Pin giga_executor.enabled=false in upgradeConfig.
sei-config v0.0.25 added GigaExecutor{Enabled: true} to its base defaults and
renders it into every app.toml, overriding the binary's own default. The
sidecar carries that version and applies it to every chain it provisions. But
sei-chain only began copying libevmone into the runtime image on 2026-06-23
(b8776ed2d); before that the library exists in the source tree and never
reaches the final layer. A pre-2026-06-23 binary therefore reads
giga_executor.enabled=true, calls InitEvmoneVM, and panics inside app.New at
app.go:753 before it binds the RPC port.
The upgrade suite is the only one that runs such a binary. Its
SEID_UPGRADE_FROM_IMAGE is pinned by design, because a minor-version upgrade
test needs a pre-upgrade binary; every other suite runs a fresh nightly whose
libevmone loads. All four validators crash-looped at height 0, nothing served
/status, and WaitReady consumed the full 60-minute context — four consecutive
nights from 2026-08-21, each failing at exactly 3600s.
The pin is a compatibility declaration with an expiry, not a preference. Drop
it once SEID_UPGRADE_FROM_IMAGE is v6.6.0 or later.
Gate the first block before submitting the upgrade proposal.
This is a separate, older defect and it is NOT what fails today. Genesis
accounts become queryable only once block 1 commits, and the proposal task
looks the proposer account up before it signs. WaitReady gates on the
SeiNetwork phase plus a single /status probe, and a chain still at height 0
satisfies both. It bit once, on 2026-08-14: the run read "current height 0"
and the task failed with "account sei1zmaq0e... not found: key not found".
Six runs that read height 1 or above all passed. pollHeightAtLeast already
existed in this file.
Verified: gofmt clean, go vet -tags integration clean, the integration test
binary compiles. The suite itself was not run; it needs a cluster.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PR SummaryLow Risk Overview Separately, the test waits for height ≥ 1 via Reviewed by Cursor Bugbot for commit 80b76cb. Bugbot is set up for automated code reviews on this repo. Configure here. |
Why
TestNightlyChainUpgradehas failed every night since 2026-08-21, each run atexactly 3600.0s. Before that it passed at 309s, 331s and 319s. The failure is
net.WaitReadynever getting an answer:The chain never serves, because
seidnever starts. All four validatorscrash-loop at height 0:
The mechanism
sei-configv0.0.25 addedGigaExecutor{Enabled: true}to its base defaults.Its own comment states the field renders into every app.toml and overrides
the binary's own default. The sidecar links that version and renders config for
every chain it provisions — the controller has no rendering path of its own.
Meanwhile
sei-chainonly began copying libevmone into the runtime image on2026-06-23 (
b8776ed2d). Before that the library sits in the source tree andnever reaches the final
ubuntu:24.04layer, which is why the panic quotes a/go/src/…builder path that does not exist at runtime.app/app.gogates the load on config, so a pre-giga binary that is told gigais on has no way to satisfy it:
Why only this suite
The upgrade suite is the only one that runs a pre-giga binary, and it does so by
design — a minor-version upgrade test needs a binary from before the upgrade.
Every other suite runs a fresh nightly whose libevmone loads. On the failing
night, sixteen chains ran and exactly one panicked: the upgrade suite's.
What changes
giga_executor.enabled: "false"inupgradeConfig. A compatibilitydeclaration with a stated expiry, not a preference — drop it once
SEID_UPGRADE_FROM_IMAGEis v6.6.0 or later.A
pollHeightAtLeast(…, 1)gate before the proposal. This is a separate,older defect and not the current failure. Genesis accounts are queryable only
once block 1 commits, and the proposal task looks the proposer up before signing.
WaitReadygates on the SeiNetwork phase plus one/statusprobe, both of whicha chain at height 0 satisfies. It bit once, on 2026-08-14 — that run logged
current height 0and failed withaccount sei1zmaq0e… not found: key not found, while six runs that read height ≥ 1 all passed.Follow-up in
sei-protocol/platformThis PR alone does not green the nightly. Once it merges and the harness image
builds,
clusters/harbor/nightly/harness/cronjobs.yamlneeds four values —moving the upgrade window forward one minor version, since the current pins are
broken at both ends (v6.5.0 also predates libevmone):
v6.5.0'sapp/tagsends atv6.5, so it lacks thev6.6handler;v6.6.0has it. That is a real minor-version transition. The protocol only supports
upgrade proposals at the minor level, so a patch-to-patch pairing would not
exercise anything.
Verification
gofmt -lclean.go vet -tags integration ./test/integration/...clean. Theintegration test binary compiles. The suite was not run — it needs a cluster.
Investigation
Root-caused across two blinded specialist lenses plus Loki, which retained the
crash-looping chain's logs after
cleanupChainremoved the pods. Falsified alongthe way: the FROM image bytes changing (ECR says pushed 2026-05-05), harness and
controller version skew (10 insertions of doc comment between them), and a
consensus stall (
seidpanics before binding, so nothing ever listens).🤖 Generated with Claude Code