Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
id: caching-artifacts-153
title: 'actions/cache key found via REST API but "Cache not found" when crossing GitHub-hosted/self-hosted runner boundary'
category: caching-artifacts
severity: silent-failure
tags:
- cache
- self-hosted
- github-hosted
- cache-version
- runner-type
- cache-miss
patterns:
- regex: 'Cache not found for input keys'
flags: 'i'
- regex: 'cache.*self.?hosted.*not found|self.?hosted.*cache.*miss'
flags: 'i'
error_messages:
- "Cache not found for input keys: Linux-3.12.3-b7dbd3ea392a6a23acbfac7fae36dded9622e49c3fa5824b0b168cfa64a66cc1"
- "Cache not found for input keys: <key>"
root_cause: |
The `actions/cache` action computes a cache `version` by hashing both the
cache `path` input AND the `RUNNER_TOOL_CACHE` environment variable. This
version is used as a secondary lookup key alongside the user-supplied cache key.

GitHub-hosted runners set `RUNNER_TOOL_CACHE` to a fixed OS-specific path:
- Linux: /opt/hostedtoolcache
- Windows: C:\hostedtoolcache
- macOS: /Users/runner/hostedtoolcache

Self-hosted runners use whatever path the administrator configured, which is
almost never the same as the GitHub-hosted default. As a result, a cache
entry saved by a GitHub-hosted runner has a different `version` hash than the
one the self-hosted runner computes, causing a guaranteed 100% cache miss even
when the cache key matches exactly.

Critically, the cache entry IS visible via the REST API (it exists in the
cache store with the matching key) but the `actions/cache` action cannot find
it because the version mismatch causes the lookup to return no results.

This affects any workflow that mixes runner types — for example, a "seed
cache" job on a GitHub-hosted runner followed by consuming jobs on
self-hosted runners.

Tracked in: https://github.com/actions/cache/issues/1595
fix: |
Option 1 — Standardize RUNNER_TOOL_CACHE on self-hosted runners.
Set RUNNER_TOOL_CACHE to the same path as GitHub-hosted runners so the
version hash matches:

On Linux self-hosted runners, set the env var before starting the runner:
export RUNNER_TOOL_CACHE=/opt/hostedtoolcache

Option 2 — Include runner type in the cache key.
Scope the cache key so each runner type writes and reads its own cache entry,
eliminating cross-type lookups entirely.

Option 3 — Use restore-keys for cross-runner sharing.
Add a broad restore-keys prefix that matches entries from any runner type.
This still misses due to the version difference, so this option does NOT work.
Use option 1 or 2 instead.
fix_code:
- language: yaml
label: 'Option 1 — Pin RUNNER_TOOL_CACHE on self-hosted runner (at OS level, not workflow)'
code: |
# On Linux self-hosted runner host, before starting the runner service:
# export RUNNER_TOOL_CACHE=/opt/hostedtoolcache
# Or add to /etc/environment for persistence.

# Your workflow cache step remains unchanged:
- name: Cache dependencies
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
- language: yaml
label: 'Option 2 — Include runner type in cache key to scope per runner'
code: |
- name: Cache dependencies (runner-type scoped)
uses: actions/cache@v4
with:
path: ~/.cache/pip
# Add runner.environment or a custom env var to the key
key: ${{ runner.os }}-${{ env.RUNNER_TYPE }}-pip-${{ hashFiles('**/requirements.txt') }}
env:
RUNNER_TYPE: ${{ contains(runner.name, 'self') && 'self-hosted' || 'github-hosted' }}
- language: yaml
label: 'Option 3 — Separate seed job on same runner type as consuming jobs'
code: |
jobs:
seed-cache:
# Run seed job on the SAME runner type that will consume the cache
runs-on: self-hosted # Changed from ubuntu-latest to self-hosted
steps:
- uses: actions/checkout@v4
- name: Seed cache
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
- run: pip install -r requirements.txt

build:
needs: seed-cache
runs-on: self-hosted # Same runner type as seed-cache
steps:
- uses: actions/checkout@v4
- name: Restore cache
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
prevention:
- "Never mix GitHub-hosted and self-hosted runners in workflows that share a cache entry unless RUNNER_TOOL_CACHE is standardized."
- "When using self-hosted runners, always set RUNNER_TOOL_CACHE to the same value as GitHub-hosted runners for the OS."
- "If you suspect a version mismatch, use the Cache REST API to verify the entry exists and check the version field."
- "Include runner.environment or a custom RUNNER_TYPE marker in cache keys when mixing runner types is unavoidable."
docs:
- url: 'https://github.com/actions/cache/issues/1595'
label: 'actions/cache#1595: GitHub-hosted runner-generated cache entry not found from self-hosted runner'
- url: 'https://docs.github.com/en/actions/writing-workflows/choosing-where-your-workflow-runs/about-self-hosted-runners'
label: 'GitHub Docs: About self-hosted runners'
- url: 'https://github.com/actions/cache/blob/main/tips-and-workarounds.md'
label: 'actions/cache: Tips and workarounds'
109 changes: 109 additions & 0 deletions errors/runner-environment/ubuntu-24-mono-msbuild-nuget-removed.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
id: runner-environment-501
title: 'ubuntu-24.04 — Mono, MSBuild, and NuGet removed — "no installation candidate"'
category: runner-environment
severity: error
tags:
- ubuntu-24.04
- ubuntu-latest
- mono
- msbuild
- nuget
- migration
- dotnet
patterns:
- regex: 'Package .mono-complete. has no installation candidate'
flags: 'i'
- regex: 'Unable to locate package (mono|msbuild|nuget)'
flags: 'i'
- regex: 'mono.*command not found|msbuild.*command not found'
flags: 'i'
- regex: 'E: Package .msbuild. has no installation candidate'
flags: 'i'
error_messages:
- "E: Package 'mono-complete' has no installation candidate"
- "E: Package 'msbuild' has no installation candidate"
- "E: Package 'nuget' has no installation candidate"
- "Unable to locate package mono-complete"
- "/bin/sh: 1: mono: not found"
- "msbuild: command not found"
root_cause: |
Mono, the open-source .NET framework (including MSBuild and NuGet via Mono),
is not available in Ubuntu 24.04 (Noble Numbat) apt repositories. The Mono
Project does not provide packages for Noble, only up to Focal (20.04) and
Jammy (22.04).

When GitHub changed `ubuntu-latest` to point to Ubuntu 24.04 in January 2025
(actions/runner-images#10636, rolled out Dec 2024 – Jan 2025), any workflow
that installed Mono, MSBuild, or NuGet via apt — or relied on them being
pre-installed — broke with "no installation candidate" errors.

Affected workflows include:
- Building .NET Framework projects with MSBuild
- Running NuGet package restore or publish via the Mono-based CLI
- Any step using `apt-get install mono-complete`, `mono-devel`, `msbuild`,
`nuget`, or `libmono-*` packages
- Scripts that call `mono`, `msbuild`, or `nuget` directly

This is distinct from `actions/setup-dotnet` which installs .NET (Core/5+)
and is not affected. Only Mono-based tooling is impacted.

Tracked in: https://github.com/actions/runner-images/issues/10636
fix: |
For .NET 5+ / .NET 6+ projects:
Switch from Mono-based MSBuild to the `dotnet` CLI via `actions/setup-dotnet`.
Use `dotnet build` and `dotnet restore` instead of `msbuild` and `nuget`.

For legacy .NET Framework projects that require Mono:
Pin the workflow to ubuntu-22.04 where Mono packages are still available.
Note: Ubuntu 22.04 will reach end-of-standard-support in April 2027.

For NuGet specifically:
The `dotnet` CLI includes `dotnet nuget` commands for package management.
fix_code:
- language: yaml
label: 'Option 1 — Migrate to dotnet CLI (recommended for .NET Core/.NET 5+)'
code: |
jobs:
build:
runs-on: ubuntu-latest # Ubuntu 24.04
steps:
- uses: actions/checkout@v4
- name: Setup .NET
uses: actions/setup-dotnet@v4
with:
dotnet-version: '8.x'
- name: Restore packages
run: dotnet restore # replaces: nuget restore
- name: Build
run: dotnet build --no-restore # replaces: msbuild /p:Configuration=Release
- name: Test
run: dotnet test --no-build
- language: yaml
label: 'Option 2 — Pin to ubuntu-22.04 for legacy Mono-dependent workflows'
code: |
jobs:
build:
# Pin to Ubuntu 22.04 where Mono is still available via apt
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- name: Install Mono
run: |
sudo apt-get update
sudo apt-get install -y mono-complete msbuild nuget
- name: Build with MSBuild
run: msbuild MyProject.sln /p:Configuration=Release
prevention:
- "Audit workflows for `apt-get install mono-*`, `msbuild`, or `nuget` before migrating to ubuntu-24.04."
- "Prefer `actions/setup-dotnet` and the `dotnet` CLI for new projects — it works on all runner versions."
- "If you must use Mono, explicitly pin `runs-on: ubuntu-22.04` rather than relying on `ubuntu-latest`."
- "Check https://packages.ubuntu.com before assuming a package is available on a new Ubuntu release."
docs:
- url: 'https://github.com/actions/runner-images/issues/10636'
label: 'actions/runner-images#10636: ubuntu-latest → Ubuntu 24.04 — software differences and removals'
- url: 'https://www.mono-project.com/download/stable/'
label: 'Mono Project download page (check supported distros)'
- url: 'https://github.com/actions/setup-dotnet'
label: 'actions/setup-dotnet — install .NET SDK in GitHub Actions'
- url: 'https://docs.github.com/en/actions/use-cases-and-examples/building-and-testing/building-and-testing-net'
label: 'GitHub Docs: Building and testing .NET'
100 changes: 100 additions & 0 deletions errors/runner-environment/ubuntu-24-terraform-cli-removed.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
id: runner-environment-502
title: 'ubuntu-24.04 — Terraform CLI removed from runner image — "terraform: command not found"'
category: runner-environment
severity: error
tags:
- ubuntu-24.04
- ubuntu-latest
- terraform
- runner-image
- migration
- iac
- command-not-found
patterns:
- regex: 'terraform.*command not found|command not found.*terraform'
flags: 'i'
- regex: '\/usr\/bin\/terraform.*not found|terraform.*No such file'
flags: 'i'
- regex: 'Error: Terraform is not installed\b'
flags: 'i'
- regex: 'Run terraform --version.*Process completed with exit code 127'
flags: 'i'
error_messages:
- "/bin/sh: 1: terraform: not found"
- "terraform: command not found"
- "Process completed with exit code 127"
- "Error: Terraform is not installed or not in PATH"
root_cause: |
Terraform CLI was pre-installed on the Ubuntu 22.04 runner image but was
intentionally removed from Ubuntu 24.04 to reduce image size. When GitHub
changed `ubuntu-latest` to point to Ubuntu 24.04 in January 2025
(actions/runner-images#10636, rolled out Dec 2024 – Jan 2025), any workflow
that ran `terraform` commands without first installing the CLI broke with
exit code 127 ("command not found").

Common failure patterns:
- `run: terraform init` fails in the first step that calls terraform
- `run: terraform --version` in a pre-check step fails
- Third-party actions that assume terraform is pre-installed (e.g., some
older community actions that wrap terraform without their own installer)
- Scripts using `which terraform` for version checks fail silently

This is a silent failure pattern when `which terraform` is used in
conditional logic (it returns exit code 1, which some scripts ignore).

Tracked in: https://github.com/actions/runner-images/issues/10636
fix: |
Install Terraform explicitly using the official HashiCorp setup action
(`hashicorp/setup-terraform`) before any terraform step. This works on all
runner versions and pins an explicit Terraform version.

Alternatively, use the HashiCorp apt repository to install a specific version
directly via apt.
fix_code:
- language: yaml
label: 'Recommended — Install Terraform via hashicorp/setup-terraform'
code: |
jobs:
terraform:
runs-on: ubuntu-latest # Works on Ubuntu 24.04
steps:
- uses: actions/checkout@v4
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_version: '1.9.x' # Pin to a specific version
- name: Terraform Init
run: terraform init
- name: Terraform Plan
run: terraform plan
- language: yaml
label: 'Alternative — Install Terraform via HashiCorp apt repository'
code: |
- name: Install Terraform
run: |
wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install -y terraform=1.9.*
- language: yaml
label: 'Fallback — Pin to ubuntu-22.04 where Terraform is pre-installed'
code: |
jobs:
terraform:
# Terraform is still pre-installed on ubuntu-22.04
# Note: ubuntu-22.04 EOL April 2027 — migrate to explicit install
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- run: terraform init
prevention:
- "Never rely on tools being pre-installed on runner images — always install them explicitly with a setup action."
- "Use `hashicorp/setup-terraform` to install Terraform with a pinned version for reproducible builds."
- "Audit workflows for bare `terraform` commands before migrating from ubuntu-22.04 to ubuntu-latest (24.04)."
- "Check the runner-images software comparison table when bumping ubuntu-latest for your workflows."
docs:
- url: 'https://github.com/actions/runner-images/issues/10636'
label: 'actions/runner-images#10636: ubuntu-latest → Ubuntu 24.04 — software differences and removals'
- url: 'https://github.com/hashicorp/setup-terraform'
label: 'hashicorp/setup-terraform — official GitHub Action for Terraform'
- url: 'https://developer.hashicorp.com/terraform/install'
label: 'HashiCorp Terraform installation docs'
Loading