feat(scale-down): opt-in idle confirmation window before terminating not-busy runners

[jensenbox]

Problem

Closes the failure mode reported in #5085, including the harder variant discussed in its comments: GitHub's runner busy flag is not a reliable input for termination decisions. We traced 14 terminated-mid-job runners across one incident window (org-level, non-ephemeral, on-demand, module v7.10.0, scale-down on the default 5-minute schedule) and every one was killed through removeRunner's normal path on a busy: false reading, in two distinct shapes:

Variant A — assignment lag. The flag reads false for 25–60+ seconds after a job is already running on the runner:

job started 23:44:18 ... Busy: false at 23:45:18  → terminated → job killed
job started 23:44:23 ... Busy: false at 23:45:15  → terminated → job killed

Variant B — stale flag mid-job. The flag flips to false on a runner that has been continuously executing one job for many minutes:

03:05:10  Runner 'i-0a014...' - Busy: true   → correctly skipped
03:10:12  Runner 'i-0a014...' - Busy: false  → terminated; job died at 98% complete, all tests passing

Another case read false 12m15s into a running job. Because the DELETE call also succeeds in these cases (GitHub's server-side view is consistent with the wrong flag), no re-check-after-deregister sequencing can prevent Variant B — the module needs evidence across time instead of a single reading. With scale-to-zero config (idle_config = []) every instance past minimum_running_time_in_minutes is evaluated on every tick, so one stale-flag window kills several runners in the same invocation — we measured roughly one multi-runner burst per hour under CI load, each costing a full matrix rerun.

Change

Adds an opt-in confirmation window, default off (scale_down_idle_confirmation_seconds = 0 keeps today's single-reading behaviour bit-for-bit):

• On a not-busy reading, scale-down tags the instance ghr:idle_detected_at=<ISO8601> and defers termination.
• It terminates only when not-busy readings span at least the configured window (set it to one scale-down schedule interval to require two consecutive not-busy evaluations).
• Any busy reading clears the tag and resets the window — this is what saves the Variant B runners, which read true on the tick before their fatal stale reading.
• An unparsable tag value restarts the window rather than blocking or terminating.
• Each invocation logs a census (evaluated N runner(s): busy=… idle-deferred=… terminated=…) so "ran and found nothing" is distinguishable from "never ran".

No new IAM: the scale-down role already carries scoped ec2:CreateTags/DeleteTags for ghr:orphan. Threaded through the root module and multi-runner (runner_config.scale_down_idle_confirmation_seconds). Existing behaviours unchanged: bypass-removal, orphan handling, and the "only terminate EC2 after successful de-registration" guard.

Cost trade-off: a genuinely idle runner lives one extra evaluation interval before reap (~5 minutes on the default schedule).

Validation

• Unit tests: 5 new cases (first-reading deferral, in-window deferral, window-elapsed termination, busy-reading reset, unparsable-tag restart); full control-plane suite green on main (542 tests), eslint + prettier clean.
• Production: we have run this patch on our fleet (both pools, 300s window, 5-minute schedule) since 2026-07-28 05:32 UTC. Prior baseline: ~1 kill burst/hour. Since deploy: zero mid-job terminations in 14 hours of CI load, ~35 terminations all via the confirmed-idle path, zero de-registration failures. We directly observed the mechanism catching a would-have-been kill: a runner deferred on a false reading reported busy: true five minutes later and survived.

Happy to adjust naming/defaults or split the census logging out if you'd prefer a narrower diff. cc @npwolf / @ben-smyth from #5085 — this covers Variant B from your reports, which the deregister-then-recheck proposal cannot.