Skip to content

Absorb a re-raised interrupt still in flight when a --load run ends (#2973) - #2978

Merged
cliffhall merged 1 commit into
v2/mainfrom
v2/fix/2973-load-sigterm-race
Nov 12, 2026
Merged

cliffhall merged 1 commit into
v2/mainfrom
v2/fix/2973-load-sigterm-race

Conversation

@cliffhall

Copy link
Copy Markdown
Member

Closes #2973

Root cause: a production bug, not a test race

The test was right. The CLI could lose its exit status.

when-exit (pulled in by atomically) hooks SIGINT/SIGTERM with process.once, runs its cleanups, and then re-raises the signal with process.kill(process.pid, …). withLoadInterrupt already ignores that copy when it arrives while the run is pending (#2834). It did not cover the case where the copy has not arrived by the time the run ends.

A signal a process sends to itself is handled by whichever thread the kernel picks. On a loaded machine that thread can wait several milliseconds to be scheduled, which is longer than the in-flight 50ms calls take to settle. The run's finally then removed the last SIGINT/SIGTERM listener, so Node restored the default disposition. The pending copy then landed and killed the process by the signal: the partial summary had already been printed, but the process exited with code null instead of 130. A real user hitting Ctrl+C or a CI job sending SIGTERM on a busy host gets the same wrong status.

I confirmed this by instrumenting the built CLI under load. Each failing child logged the first signal, then finally 2–3ms later, and never logged the re-raise. Each passing child logged the re-raise about 0.3–1ms after the first signal, before finally. The summary was on stdout in every failure.

Fix (clients/cli/src/handlers/load-interrupt.ts)

  • Record when the re-raise has been seen (echoed).
  • If the run ends after an interrupt and the copy has not been seen yet, install a one-shot (process.once) no-op listener for the first signal. It is added before the run's listeners are removed, so the signal never has zero listeners. A synchronous gap would not be safe, because the copy is delivered on another thread.
  • That listener is removed when the copy actually arrives, not by a timer. That makes the fix deterministic however late the copy lands; nothing here is headroom (test-stability: eliminate timeout flakiness in the web test suite (zero-flake requirement) #1596).

The one cost is a caller with no re-raising dependency that invokes the listener directly, such as the in-process tests. It is left with one no-op once listener that swallows one later signal and then removes itself. The header comment documents this.

Tests (clients/cli/__tests__/load-summary.test.ts)

  • New case: the run settles before the re-raise. It asserts that exactly one SIGINT listener remains and no SIGTERM listener, that the late copy is swallowed without exiting, and that the count then returns to baseline.
  • The existing in-process cases now deliver the re-raise the way a real process receives it, or drain the absorber, so every case still leaves no stray listener.
  • signalLoadRun now calls the raw listener. process.listeners() returns the function a once wrapper wraps, and calling that directly would leave the wrapper registered.

load-interrupt-e2e.test.ts is unchanged; it is the test that catches this.

Reproduction, before and after

I ran load-interrupt-e2e.test.ts in 4 parallel lanes of 15 runs each, alongside 30 busy-loop CPU burners on an 18-core machine. Load average was about 30–40 with other agent gates also running.

Build File runs Failed Failures
origin/v2/main 60 4 3 × SIGINT, 1 × SIGTERM, all expected null to be 130
this PR 60 0 —

An earlier 32-run pass on the original code under the same load failed 2 times (both SIGINT; this is where the instrumentation above came from). Both signals share the race. The issue happened to observe the SIGTERM case.

npm run local:gate is green.

🤖 Generated with Claude Code

…2973)

when-exit (via atomically) re-raises the first SIGINT/SIGTERM with
process.kill(process.pid, ...). A self-sent signal is handed to whichever
thread the kernel picks, and on a loaded machine that delivery can lag the
in-flight calls settling. withLoadInterrupt then removed its last listener,
restoring the default disposition, and the pending copy killed the CLI by
the signal after the partial summary was printed: exit code null, not 130.

When the run ends without the re-raise having been seen, leave a one-shot
listener that swallows it, added before the run's listeners come off so the
signal never has zero listeners. Its removal is driven by the copy arriving,
not by a timer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: cliffhall <cliff@futurescale.com>
@cliffhall cliffhall added the v2 Issues and PRs for v2 label Nov 12, 2026
@cliffhall
cliffhall requested a balanced review from Copilot November 12, 2026 08:21

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the CLI shutdown-race fix.

No open issues 🎉

What changed in this PR

Fixes a CLI --load shutdown race where a delayed re-raised interrupt could terminate the process by signal instead of preserving exit code 130 after printing the partial summary.

Changes:

  • Tracks whether the re-raised signal arrived and installs a one-shot absorber before removing run listeners when it has not.
  • Adds regression coverage for late signal delivery and updates test helpers to invoke raw listeners so one-shot cleanup behaves correctly.
File Description
clients/​cli/​src/​handlers/​load-interrupt.ts Absorbs a pending re-raised interrupt after the run ends and documents listener lifetime.
clients/​cli/​__tests__/​load-summary.test.ts Tests late re-raise handling and updates simulated signal delivery and cleanup.

Review effort: Balanced

@cliffhall

Copy link
Copy Markdown
Member Author

Copilot review loop closed: round 1 (review 5871309902) was clean — no inline comments, nothing in the headline, no suppressed comments. Exit: clean round.

@cliffhall
cliffhall merged commit d330df9 into v2/main Nov 12, 2026
12 checks passed
@cliffhall
cliffhall deleted the v2/fix/2973-load-sigterm-race branch November 12, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v2 Issues and PRs for v2

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CLI --load interrupt e2e: SIGTERM case exits by signal (code null) under load

2 participants