Skip to content

fix(runners): guard SEGMENT against set -u in start-runner.sh - #5233

Open
jensenbox wants to merge 1 commit into
github-aws-runners:mainfrom
closient:fix/segment-unbound-under-set-u
Open

fix(runners): guard SEGMENT against set -u in start-runner.sh#5233
jensenbox wants to merge 1 commit into
github-aws-runners:mainfrom
closient:fix/segment-unbound-under-set-u

Conversation

@jensenbox

Copy link
Copy Markdown
Contributor

Problem

modules/runners/templates/start-runner.sh assigns SEGMENT only inside the X-Ray branch (line 191) but consumes it unconditionally at line 263 and in the error_handler trap (line 98). With tracing disabled — the default — the branch never executes and SEGMENT is never assigned.

That is harmless while user-data runs without set -u. It becomes fatal the moment anything enables -u earlier in the same shell, and user-data is one concatenated script: ${post_install} is spliced inline at modules/runners/templates/user-data.sh:63. So a consumer whose post_install hook opens with set -euo pipefail — a perfectly ordinary thing for a hook to do — silently arms this latent bug for the module's own script.

Why it is worth fixing rather than documenting

The failure mode is unusually quiet. User-data dies after the runner registers with GitHub but before its listener starts, so:

  • the runner appears healthy in the GitHub UI (registered, holds an agent ID);
  • it never claims a job;
  • scale-down reaps it as idle after the configured window;
  • the scaler launches an identically-broken replacement, forever.

From the outside it looks like a capacity or scaling problem, not a script error. We lost every job on one runner pool for ~19 hours before finding it in the instance console:

√ Runner successfully added
√ Settings Saved.
Tagging instance with GitHub runner agent ID: 49534
/var/lib/cloud/instance/scripts/part-001: line 703: SEGMENT: unbound variable
ERROR: runner-start-failed with exit code 1 occurred on 1
FAILED Failed to start cloud-final.service - Cloud-init: Final Stage.

Fix

"$${SEGMENT:-}" at both unguarded call sites.

This restores the script's existing intent rather than changing behaviour. Both helpers already begin with:

local SEGMENT_DOC="$1"
if [ -z "$SEGMENT_DOC" ]; then
  echo "No segment doc provided"
  return
fi

so they were written to tolerate an absent segment doc — the call sites simply never expressed it in a -u-safe way. With tracing enabled nothing changes; with tracing disabled the helpers now hit the early return they already had.

Line 192's echo "$SEGMENT" is deliberately untouched: it sits inside the branch that assigns the variable.

Note the doubled $ — this template is itself rendered through templatefile(), so a bare ${SEGMENT:-} is parsed as a Terraform interpolation and fails the plan with "Extra characters after interpolation expression; Template interpolation doesn't expect a colon at this location". Existing precedent in the same file: $${extra_flags} and $${config} (lines 257, 269).

Verification

  • Rendered the template through templatefile() and confirmed the output contains "${SEGMENT:-}" at both sites.
  • bash -n over the rendered script.
  • Deployed to a production runner pool: instances that previously died at SEGMENT: unbound variable now reach a running listener, claim jobs, and complete them.

Scope

Two characters at two call sites. No behaviour change for tracing-enabled consumers.

`start-runner.sh` assigns `SEGMENT` only inside the X-Ray branch (line 191) but
consumes it unconditionally at line 263 and in the `error_handler` trap (line
98). With `enable_ssm_on_runners`-style tracing disabled — the default — the
branch never runs and `SEGMENT` is never assigned.

That is harmless while user-data runs without `set -u`. It becomes fatal as soon
as anything enables `-u` earlier in the same shell, and user-data is one
concatenated script: `${post_install}` is spliced inline at
`modules/runners/templates/user-data.sh:63`. A consumer whose `post_install`
hook opens with `set -euo pipefail` therefore aborts user-data here — after the
runner has registered with GitHub, but before its listener starts.

The failure mode is unusually quiet: the runner appears healthy in the GitHub
UI (it registered and holds an agent ID), but never claims a job, so scale-down
reaps it as idle and the scaler launches an identically-broken replacement. We
lost every job on one runner pool for ~19h before tracing it to:

  √ Runner successfully added
  Tagging instance with GitHub runner agent ID: 49534
  /var/lib/cloud/instance/scripts/part-001: line 703: SEGMENT: unbound variable
  ERROR: runner-start-failed with exit code 1 occurred on 1
  FAILED Failed to start cloud-final.service - Cloud-init: Final Stage.

Use `"$${SEGMENT:-}"` at both unguarded call sites. This restores the script's
existing intent rather than changing behaviour — both helpers already open with

    local SEGMENT_DOC="$1"
    if [ -z "$SEGMENT_DOC" ]; then
      echo "No segment doc provided"
      return
    fi

so they were written to tolerate an absent segment; the call sites simply never
expressed it in a `-u`-safe way. Line 192's `echo "$SEGMENT"` is deliberately
left alone: it sits inside the branch that assigns the variable.

Note the `$$` — this template is itself rendered through `templatefile()`, so a
bare `${SEGMENT:-}` is parsed as a Terraform interpolation and fails the plan
with "Template interpolation doesn't expect a colon at this location". Existing
precedent in the same file: `$${extra_flags}` and `$${config}`.

Verified by rendering the template with `templatefile()` and running `bash -n`
over the output.
@jensenbox
jensenbox requested a review from a team as a code owner July 31, 2026 04:51
jensenbox added a commit to closient/terraform-aws-github-runner that referenced this pull request Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant