Clean up live ephemeral runners after terminal jobs

[robinbraemer]

Closes #4603.

What this changes

ARC normally relies on the ephemeral runner process exiting after its one job. If that process remains alive after GitHub has already made the job terminal, the current HandleJobCompleted discards the terminal event and ARC can retain the Pod, registration, and capacity slot indefinitely.

This patch:

• records JobCompleted on the exact EphemeralRunner status;
• requires runner ID, runner name, job ID, and workflow-run ID to match before acting;
• gives normal runner-driven shutdown a 30-second grace period;
• then issues deletion through the existing controller finalizer path;
• treats late NotFound, duplicate, and mismatched events as harmless;
• generates the API deepcopy and chart/base CRD changes.

The normal path remains unchanged. In a controlled cancellation on the reported ARC/Kata-Firecracker setup, the runner exited and ARC cleaned everything within seven seconds, well before the fallback grace period.

TDD evidence

RED commit: robinbraemer@ec1febe

The regression test fails on the old implementation because HandleJobCompleted acknowledges the event without making any Kubernetes request.

GREEN commit: robinbraemer@86a5bed

Added coverage for:

• listener persistence of the exact terminal event;
• fail-closed rejection of runner/job/workflow identity mismatches;
• full envtest lifecycle from a still-running Pod to ER deletion and existing finalizer cleanup.

Validation

• go test ./cmd/ghalistener/scaler ./controllers/actions.github.com -count=1 (full controller suite: 180s, pass)
• focused envtest regression (pass)
• go vet ./cmd/ghalistener/scaler ./controllers/actions.github.com (pass)
• make generate and make manifests (generated output included)
• git diff --check (pass)

Runtime investigation

The original runner process hang remains unexplained because its Pod logs were gone after emergency cleanup. It was not a generally broken cancellation configuration: same ARC 0.14.2, official runner 2.336.0, /home/runner/run.sh, Pod template, and Kata Firecracker RuntimeClass handled a controlled cancellation normally. The runner received cancellation, completed service cleanup, exited 0, and ARC deleted the Pod/ER/registration. This patch provides bounded ARC self-healing independently of the rare underlying runner/runtime trigger.

[Copilot]

Pull request overview

This PR adds a fallback cleanup path for live EphemeralRunner resources when GitHub reports a job is terminal but the runner process fails to exit, by persisting JobCompleted details onto the EphemeralRunner status and letting the controller delete the runner after a short grace period.

Changes:

• Add status.jobCompletion to EphemeralRunner (API types, deepcopy, CRDs/charts) and have the listener patch it on JobCompleted.
• Update the EphemeralRunner controller to wait a 30s grace period after a matching terminal event, then delete the runner via the existing finalizer path.
• Add unit + envtest coverage for identity matching and the live-runner deletion behavior.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
controllers/actions.github.com/ephemeralrunner_job_completion_test.go	Unit test for jobCompletionMatchesRunner identity matching.
controllers/actions.github.com/ephemeralrunner_controller.go	Controller logic to requeue for grace period then delete runner after matching terminal job event.
controllers/actions.github.com/ephemeralrunner_controller_test.go	Envtest regression ensuring a live runner is deleted after terminal job event is recorded.
config/crd/bases/actions.github.com_ephemeralrunners.yaml	Adds status.jobCompletion schema to base CRD.
cmd/ghalistener/scaler/scaler.go	Records JobCompleted by patching status.jobCompletion on the exact EphemeralRunner.
cmd/ghalistener/scaler/scaler_test.go	New test verifying HandleJobCompleted performs the expected status patch request.
charts/gha-runner-scale-set-controller/crds/actions.github.com_ephemeralrunners.yaml	Propagates CRD schema update into Helm chart CRDs.
charts/gha-runner-scale-set-controller-experimental/crds/actions.github.com_ephemeralrunners.yaml	Propagates CRD schema update into experimental Helm chart CRDs.
apis/actions.github.com/v1alpha1/zz_generated.deepcopy.go	Generated deepcopy support for the new EphemeralRunnerJobCompletion type and pointer field.
apis/actions.github.com/v1alpha1/ephemeralrunner_types.go	Adds JobCompletion to status and defines EphemeralRunnerJobCompletion type.

Files not reviewed (1)

• apis/actions.github.com/v1alpha1/zz_generated.deepcopy.go: Generated file

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[robinbraemer]

Live validation against the latest stable release (gha-runner-scale-set-0.14.2) is now complete on a real Kubernetes + Kata/Firecracker ARC scale set.

I backported this change to robinbraemer:akua/0.14.2-jobcompleted (246fa0632ea9) and ran a controlled end-to-end reproduction where the Actions runner completed its job normally but the container wrapper deliberately remained alive (/home/runner/run.sh; sleep infinity).

Evidence from the live run akua-dev/cnap#31789400724:

• GitHub job completed successfully at 09:46:04Z.
• Listener received JobCompleted and persisted result, runner ID, job ID, workflow run ID, and finish time to EphemeralRunner.status.jobCompletion.
• The runner Pod remained 2/2 Running, proving the ordinary runner-exit cleanup did not mask the test.
• The controller logged a ~29.36s grace-period requeue.
• At 09:46:34Z, it logged Job is terminal but runner is still present; issuing delete, removed the GitHub runner registration, Pod, JIT secret, and finalizers.
• The scale set returned to zero.

The normal Docker/Buildx/isolation canary 31788504613 also passed on the patched controller.

Targeted backport tests pass. The release-tag full suite currently has an unrelated time-dependent failure because its checked-in test CA expired on 2026-07-13; the new focused controller regression and scaler tests pass.