ci: Rust Integration Tests job is failing across branches — validators_and_full_nodes_reach_height_3 killed by SIGKILL

[0xrlawrence]

Summary

The Rust Integration Tests job is currently red on unrelated branches, including Circle's own sync/v0_8_0. In each case the same test is killed by the OOM killer:

SIGKILL [9.729s] (11/13) arc-test-integration::basic validators_and_full_nodes_reach_height_3
(test aborted with signal 9: SIGKILL)
Summary [65.769s] 11/13 tests run: 10 passed, 1 failed, 0 skipped
warning: 2/13 tests were not run due to signal
error: test run failed

This does not look branch-specific, so I wanted to flag it rather than assume it is known.

Affected runs

Branch	Run	Result
sync/v0_8_0 (internal release sync)	33160093215	SIGKILL, same test
fix/executor-cumulative-gas-overflow	33372562134	Rust Integration Tests failed
fix/key-file-permissions (mine, #288)	33388046314	SIGKILL, same test

The sync/v0_8_0 failure is dated 2026-08-28 and has the identical 10 passed, 1 failed split, so this predates the PRs above.

Notes on the failing test

crates/test/integration/tests/basic.rs:65. From the run logs it brings up seven nodes (node-0 through node-6) in-process. The job runs on ubuntu-latest and already passes --test-threads 1 (ci.yml:133), so test-level parallelism is not the cause; a single test appears to exceed the runner's memory.

For what it is worth, I saw this same binary reach roughly 7.3 GB RSS when running cargo test --workspace on a 7 GB VM, where the kernel OOM killer reported:

Out of memory: Killed process (basic-5d309b0f0) total-vm:21601360kB, anon-rss:7349572kB

I have not been able to determine whether the failure is deterministic or marginal, so I cannot say whether it is a genuine memory regression or a test that has simply grown past the runner budget.

Possibly related: the nextest ci profile is never used

.config/nextest.toml defines a [profile.ci] (line 8) with fail-fast = false (line 11) and retries (line 15):

[profile.ci]
fail-fast = false
retries = { backoff = "exponential", count = 2, delay = "1s", max-delay = "10s" }

None of the three CI invocations pass --profile ci:

• ci.yml:91 — cargo nextest run --locked --workspace --exclude arc-test-integration
• ci.yml:133 — cargo nextest run --locked -p arc-test-integration --test-threads 1
• ci.yml:139 — cargo nextest run --locked --workspace --exclude arc-test-integration --features integration

So CI runs under [profile.default], which sets fail-fast = true and no retries. That matches the observed 2/13 tests were not run due to signal: the run aborts rather than completing, and the failing test is never retried.

To be clear about what this is and is not: wiring up --profile ci would give fuller failure reporting and would retry a flaky test, but if the OOM is deterministic it will not fix the underlying problem. I am flagging it as a likely-unintended gap, not as a fix.

Happy to open a PR for the --profile ci wiring if that is useful. The memory issue itself looks like it needs someone with more context on the integration harness.

🤖 Generated with Claude Code

[osr21]

Good report — I verified the repo-side claims and can add two things: an answer to the deterministic-vs-marginal question, and a concrete v0.8.0 suspect for why this test's memory grew.

Claims verified

• validators_and_full_nodes_reach_height(5, 5, 2, 3) = 5 validators to height 5 plus 2 full nodes to height 3 — 7 in-process nodes, the largest node count of any test in basic.rs (the others run 5). Consistent with it being the one that dies.
• All three ci.yml nextest invocations run under [profile.default]; nothing sets NEXTEST_PROFILE either, so the [profile.ci] block is indeed dead config. fail-fast = true matches the observed 2/13 tests were not run due to signal.

It is marginal, not deterministic

Job-level results for every completed Public CI run that reached the Rust Integration Tests job since the v0.8.0 sync landed:

Run	Branch / sha	Rust Integration Tests
33164470081	main @ 66ad2d5 (the v0.8.0 merge itself, Aug 28)	✅ pass
33160093215	sync/v0_8_0 @ 42cf057	❌ fail
33372562134	fix/executor-cumulative-gas-overflow @ 2a31a4f	❌ fail
33388046314	fix/key-file-permissions @ f8334fa	❌ fail

The trees of 42cf057 and 66ad2d5 are not byte-identical overall, but the test harness (crates/test/framework/src/lib.rs) is identical between them, and the two failing PR branches are small deltas on top of 66ad2d5 that don't touch the harness or node internals. So effectively the same code has gone 1 pass / 3 fails: the test sits right at the runner's memory boundary and the outcome is a coin flip weighted toward failure.

One calibration note on the numbers: ubuntu-latest for public repos is 4 vCPU / 16 GB RAM (per GitHub's hosted-runner docs), so what CI is exceeding is ~16 GB — your 7.3 GB RSS kill on a 7 GB VM is just where the smaller box gave out, not the peak the binary reaches when given room.

A concrete v0.8.0 suspect: the harness now keeps every node alive until the whole test finishes

git diff v0.7.3..v0.8.0 -- crates/test/framework/src/lib.rs shows a node-handle lifetime refactor:

• v0.7.3: run_node returned only a NodeOutcome, and the node's handle was dropped when its own step sequence completed — dropping the handle shuts the node down. Nodes were torn down incrementally: in this test, the 2 full nodes died at height 3 and each validator died as it reached height 5, freeing memory as the test progressed.
• v0.8.0: run_node returns (NodeOutcome, Option<Handle>), and check_results collects the handles into live_handles, dropping them only after every task has joined. The new doc comment is explicit: "handles are released only once all tasks complete."

Net effect on this test: all 7 nodes now stay resident — still producing blocks, gossiping, and growing their caches — until the slowest node finishes. Peak memory therefore scales with how long the slowest node takes, and consensus timing varies run to run (missed rounds, rebroadcast timers). That gives you both halves of the observed behavior in one mechanism: higher peak than v0.7.3, and nondeterministic OOM.

The keep-alive change is well-motivated for the crash/restart scenarios (a restarted node needs live peers to sync from), but in this scenario it buys nothing: full nodes never provide sync support (they're not in the validator registry), and validator availability for the full nodes was already guaranteed by the val_height > full_height assertion in the scenario builder. A targeted fix could drop handles for nodes no remaining peer depends on, rather than holding all of them to the end.

Cheap A/B for anyone with a big enough machine to confirm: cargo nextest run --locked -p arc-test-integration -E 'test(validators_and_full_nodes_reach_height_3)' at v0.7.3 vs v0.8.0 while recording peak RSS (/usr/bin/time -v, or /sys/fs/cgroup/memory.peak in a scoped cgroup).

On wiring up --profile ci

Agreed it looks like an unintended gap, with one caveat given the failure is marginal rather than deterministic: retries = 2 would likely turn most of these red runs green, which unblocks PRs but also silently hides the memory regression. If it gets wired up, it's worth pairing with a visibility step (dump peak memory after the nextest invocation) so the drift stays measurable instead of being retried away. fail-fast = false alone would at least let the remaining 2 tests run and give full reports in the meantime.