From 20d8d9595843abd70edd192c7293dd263be56e29 Mon Sep 17 00:00:00 2001 From: Christian Jensen Date: Thu, 30 Jul 2026 21:51:12 -0700 Subject: [PATCH] fix(runners): guard SEGMENT against set -u in start-runner.sh MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `start-runner.sh` assigns `SEGMENT` only inside the X-Ray branch (line 191) but consumes it unconditionally at line 263 and in the `error_handler` trap (line 98). With `enable_ssm_on_runners`-style tracing disabled — the default — the branch never runs and `SEGMENT` is never assigned. That is harmless while user-data runs without `set -u`. It becomes fatal as soon as anything enables `-u` earlier in the same shell, and user-data is one concatenated script: `${post_install}` is spliced inline at `modules/runners/templates/user-data.sh:63`. A consumer whose `post_install` hook opens with `set -euo pipefail` therefore aborts user-data here — after the runner has registered with GitHub, but before its listener starts. The failure mode is unusually quiet: the runner appears healthy in the GitHub UI (it registered and holds an agent ID), but never claims a job, so scale-down reaps it as idle and the scaler launches an identically-broken replacement. We lost every job on one runner pool for ~19h before tracing it to: √ Runner successfully added Tagging instance with GitHub runner agent ID: 49534 /var/lib/cloud/instance/scripts/part-001: line 703: SEGMENT: unbound variable ERROR: runner-start-failed with exit code 1 occurred on 1 FAILED Failed to start cloud-final.service - Cloud-init: Final Stage. Use `"$${SEGMENT:-}"` at both unguarded call sites. This restores the script's existing intent rather than changing behaviour — both helpers already open with local SEGMENT_DOC="$1" if [ -z "$SEGMENT_DOC" ]; then echo "No segment doc provided" return fi so they were written to tolerate an absent segment; the call sites simply never expressed it in a `-u`-safe way. Line 192's `echo "$SEGMENT"` is deliberately left alone: it sits inside the branch that assigns the variable. Note the `$$` — this template is itself rendered through `templatefile()`, so a bare `${SEGMENT:-}` is parsed as a Terraform interpolation and fails the plan with "Template interpolation doesn't expect a colon at this location". Existing precedent in the same file: `$${extra_flags}` and `$${config}`. Verified by rendering the template with `templatefile()` and running `bash -n` over the output. --- modules/runners/templates/start-runner.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/runners/templates/start-runner.sh b/modules/runners/templates/start-runner.sh index 7f2c0f82c5..b2c6efed1a 100644 --- a/modules/runners/templates/start-runner.sh +++ b/modules/runners/templates/start-runner.sh @@ -95,7 +95,7 @@ cleanup() { if [ "$exit_code" -ne 0 ]; then echo "ERROR: runner-start-failed with exit code $exit_code occurred on $error_location" - create_xray_error_segment "$SEGMENT" "runner-start-failed with exit code $exit_code occurred on $error_location - $error_lineno" + create_xray_error_segment "$${SEGMENT:-}" "runner-start-failed with exit code $exit_code occurred on $error_location - $error_lineno" fi # allows to flush the cloud watch logs and traces sleep 10 @@ -260,7 +260,7 @@ if [[ "$enable_jit_config" == "false" || $agent_mode != "ephemeral" ]]; then tag_instance_with_runner_id fi -create_xray_success_segment "$SEGMENT" +create_xray_success_segment "$${SEGMENT:-}" if [[ $agent_mode = "ephemeral" ]]; then echo "Starting the runner in ephemeral mode"