Skip to content

fix(nightly): unblock the chain-upgrade suite on a pre-giga image - #506

Merged
bdchatham merged 1 commit into
mainfrom
fix/nightly-upgrade-suite-pre-giga
Aug 25, 2026
Merged

fix(nightly): unblock the chain-upgrade suite on a pre-giga image#506
bdchatham merged 1 commit into
mainfrom
fix/nightly-upgrade-suite-pre-giga

Conversation

@bdchatham

Copy link
Copy Markdown
Collaborator

Why

TestNightlyChainUpgrade has failed every night since 2026-08-21, each run at
exactly 3600.0s. Before that it passed at 309s, 331s and 319s. The failure is
net.WaitReady never getting an answer:

upgrade_test.go:139: network "nightly-dkx3hvkb4n3t" ready:
  SeiNetwork nightly/nightly-dkx3hvkb4n3t: TM /status serve-probe: context deadline exceeded

The chain never serves, because seid never starts. All four validators
crash-loop at height 0:

sidecar ready, starting seid
panic: failed to load evmone: evmc.Load("/go/src/sei-chain/giga/executor/lib/libevmone.0.12.0_linux_amd64.so"):
  EVMC loading error
  github.com/sei-protocol/sei-chain/app.New(…) /go/src/sei-chain/app/app.go:753

The mechanism

sei-config v0.0.25 added GigaExecutor{Enabled: true} to its base defaults.
Its own comment states the field renders into every app.toml and overrides
the binary's own default. The sidecar links that version and renders config for
every chain it provisions — the controller has no rendering path of its own.

Meanwhile sei-chain only began copying libevmone into the runtime image on
2026-06-23 (b8776ed2d). Before that the library sits in the source tree and
never reaches the final ubuntu:24.04 layer, which is why the panic quotes a
/go/src/… builder path that does not exist at runtime.

app/app.go gates the load on config, so a pre-giga binary that is told giga
is on has no way to satisfy it:

if gigaExecutorConfig.Enabled {
    evmoneVM, err := gigalib.InitEvmoneVM()
    if err != nil {
        panic(fmt.Sprintf("failed to load evmone: %s", err))
    }

Why only this suite

The upgrade suite is the only one that runs a pre-giga binary, and it does so by
design — a minor-version upgrade test needs a binary from before the upgrade.
Every other suite runs a fresh nightly whose libevmone loads. On the failing
night, sixteen chains ran and exactly one panicked: the upgrade suite's.

What changes

giga_executor.enabled: "false" in upgradeConfig. A compatibility
declaration with a stated expiry, not a preference — drop it once
SEID_UPGRADE_FROM_IMAGE is v6.6.0 or later.

A pollHeightAtLeast(…, 1) gate before the proposal. This is a separate,
older defect and not the current failure.
Genesis accounts are queryable only
once block 1 commits, and the proposal task looks the proposer up before signing.
WaitReady gates on the SeiNetwork phase plus one /status probe, both of which
a chain at height 0 satisfies. It bit once, on 2026-08-14 — that run logged
current height 0 and failed with account sei1zmaq0e… not found: key not found, while six runs that read height ≥ 1 all passed.

Follow-up in sei-protocol/platform

This PR alone does not green the nightly. Once it merges and the harness image
builds, clusters/harbor/nightly/harness/cronjobs.yaml needs four values —
moving the upgrade window forward one minor version, since the current pins are
broken at both ends (v6.5.0 also predates libevmone):

SEID_UPGRADE_FROM_IMAGE  → sei-chain:fbc0d9342ca28887958013170e4020d93cacdbfa  (v6.5.0)
SEID_UPGRADE_TO_IMAGE    → sei-chain:082fbe7549e6137fa4c2e597199a13c62f720b08  (v6.6.0)
SEI_UPGRADE_NAME         → "v6.6"
integration-harness      → this PR's merge SHA   (line 41, no $imagepolicy marker)

v6.5.0's app/tags ends at v6.5, so it lacks the v6.6 handler; v6.6.0
has it. That is a real minor-version transition. The protocol only supports
upgrade proposals at the minor level, so a patch-to-patch pairing would not
exercise anything.

Verification

gofmt -l clean. go vet -tags integration ./test/integration/... clean. The
integration test binary compiles. The suite was not run — it needs a cluster.

Investigation

Root-caused across two blinded specialist lenses plus Loki, which retained the
crash-looping chain's logs after cleanupChain removed the pods. Falsified along
the way: the FROM image bytes changing (ECR says pushed 2026-05-05), harness and
controller version skew (10 insertions of doc comment between them), and a
consensus stall (seid panics before binding, so nothing ever listens).

🤖 Generated with Claude Code

Two independent fixes to TestNightlyChainUpgrade. Only the first addresses
the failure the suite shows today.

Pin giga_executor.enabled=false in upgradeConfig.

sei-config v0.0.25 added GigaExecutor{Enabled: true} to its base defaults and
renders it into every app.toml, overriding the binary's own default. The
sidecar carries that version and applies it to every chain it provisions. But
sei-chain only began copying libevmone into the runtime image on 2026-06-23
(b8776ed2d); before that the library exists in the source tree and never
reaches the final layer. A pre-2026-06-23 binary therefore reads
giga_executor.enabled=true, calls InitEvmoneVM, and panics inside app.New at
app.go:753 before it binds the RPC port.

The upgrade suite is the only one that runs such a binary. Its
SEID_UPGRADE_FROM_IMAGE is pinned by design, because a minor-version upgrade
test needs a pre-upgrade binary; every other suite runs a fresh nightly whose
libevmone loads. All four validators crash-looped at height 0, nothing served
/status, and WaitReady consumed the full 60-minute context — four consecutive
nights from 2026-08-21, each failing at exactly 3600s.

The pin is a compatibility declaration with an expiry, not a preference. Drop
it once SEID_UPGRADE_FROM_IMAGE is v6.6.0 or later.

Gate the first block before submitting the upgrade proposal.

This is a separate, older defect and it is NOT what fails today. Genesis
accounts become queryable only once block 1 commits, and the proposal task
looks the proposer account up before it signs. WaitReady gates on the
SeiNetwork phase plus a single /status probe, and a chain still at height 0
satisfies both. It bit once, on 2026-08-14: the run read "current height 0"
and the task failed with "account sei1zmaq0e... not found: key not found".
Six runs that read height 1 or above all passed. pollHeightAtLeast already
existed in this file.

Verified: gofmt clean, go vet -tags integration clean, the integration test
binary compiles. The suite itself was not run; it needs a cluster.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cursor

cursor Bot commented Aug 25, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Changes only affect integration test harness config and sequencing; no production or controller runtime behavior.

Overview
Unblocks TestNightlyChainUpgrade when the pre-upgrade seid image is older than the libevmone runtime layer. The harness now sets giga_executor.enabled: "false" in upgradeConfig so sidecar-rendered app.toml does not force giga on a binary that panics in app.New before RPC comes up. Comments document removing that override once SEID_UPGRADE_FROM_IMAGE is v6.6.0+.

Separately, the test waits for height ≥ 1 via pollHeightAtLeast after WaitReady before submitting the gov upgrade proposal, because genesis accounts are not queryable at height 0 and the propose task looks up the proposer first—WaitReady alone can pass while still at height 0.

Reviewed by Cursor Bugbot for commit 80b76cb. Bugbot is set up for automated code reviews on this repo. Configure here.

@bdchatham
bdchatham merged commit b0870ca into main Aug 25, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant