Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 119 additions & 0 deletions errors/caching-artifacts/ca-154.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
id: ca-154
title: 'CodeQL TRAP caching fills GitHub Actions cache quota — other workflow caches evicted'
category: caching-artifacts
severity: warning
tags:
- codeql
- trap-caching
- cache-limit
- cache-eviction
- code-scanning
- monorepo
- performance
patterns:
- regex: 'Caching TRAP files for|TRAP cache.*saved|Restoring TRAP cache'
flags: 'i'
- regex: 'cache.*storage.*quota|Cache capacity reached.*Evicting'
flags: 'i'
- regex: 'Unable to restore.*TRAP.*cache|TRAP.*cache.*miss'
flags: 'i'
error_messages:
- 'Caching TRAP files for org/repo (Language: javascript-typescript)'
- 'Restoring TRAP cache from org/repo...'
- 'Cache capacity reached. Evicting old cache entries.'
root_cause: |
The `github/codeql-action` (init and analyze steps) uses `actions/cache`
internally to persist TRAP (Text Representation of Analyzed Programs) files
between runs. TRAP files are the compiled intermediate representation of source
code that CodeQL analyzes.

In high-frequency repositories (active monorepos with dozens of merges per
day), each CI run generates a new TRAP cache entry keyed to the exact code
state. Because the key changes on every commit, cache restoration is rare
and a new entry is written each time. GitHub Actions enforces a 10 GB
per-repository cache limit. When hundreds of TRAP cache entries accumulate,
they exhaust this quota within one to a few days.

Once the 10 GB limit is reached, GitHub begins evicting the
least-recently-used entries. This evicts OTHER workflow caches — npm
dependencies, Go modules, Docker layer caches, pip wheels — that were
actively providing value, causing build times to increase across all
workflows in the repository.

The TRAP caching feature was designed to accelerate incremental CodeQL
re-analysis. For repos where most source lines change between runs, the
cache hit rate is near zero, making TRAP caching net-negative: it consumes
cache space without improving CodeQL speed.

Source: github/codeql-action#2030 (May 2025, confirmed by GitHub staff).
fix: |
Disable TRAP caching in the CodeQL init action by setting `trap-caching: false`.
This prevents TRAP files from being written to the GitHub Actions cache and
frees the full 10 GB budget for other workflow caches.

Use `trap-caching: false` when any of the following apply:
- Repository has high commit velocity (10+ merges per day)
- The Actions Cache tab shows many entries with "TRAP" in the key name
- Non-CodeQL workflows are experiencing more frequent cache misses
- Total repository cache usage is near the 10 GB limit

Alternatively, reduce CodeQL scan frequency (e.g., scan only on push to the
default branch rather than every PR) to reduce the number of unique TRAP
cache entries created per day.
fix_code:
- language: yaml
label: 'Disable TRAP caching — prevents cache churn in high-frequency repos'
code: |
jobs:
analyze:
name: Analyze code
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write
steps:
- uses: actions/checkout@v4

- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: javascript-typescript
trap-caching: false # prevents TRAP files from filling 10 GB cache quota

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
- language: yaml
label: 'Limit CodeQL scans to default branch to reduce TRAP cache entry growth'
code: |
on:
push:
branches: [main] # analyze on merge to main only — not every PR
schedule:
- cron: '0 6 * * 1' # weekly full-codebase scan

jobs:
analyze:
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write
steps:
- uses: actions/checkout@v4
- uses: github/codeql-action/init@v3
with:
languages: javascript-typescript
- uses: github/codeql-action/analyze@v3
prevention:
- 'Set `trap-caching: false` in the CodeQL init step for repos with more than ~10 merges per day.'
- 'Inspect the Actions Cache tab regularly — many entries with "TRAP" in the key name signal cache churn.'
- 'Limit CodeQL analysis to pushes on protected branches rather than every PR to reduce cache entry growth.'
- 'Review total cache usage periodically; if near 10 GB, CodeQL TRAP caches are a common culprit.'
docs:
- url: 'https://github.com/github/codeql-action/issues/2030'
label: 'codeql-action#2030 — TRAP caching causes GitHub Actions Cache to churn (May 2025)'
- url: 'https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages#caching-trap-files'
label: 'GitHub Docs — CodeQL TRAP caching (advanced setup)'
- url: 'https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy'
label: 'GitHub Docs — Cache usage limits and eviction policy (10 GB per repository)'
103 changes: 103 additions & 0 deletions errors/runner-environment/re-503.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
id: re-503
title: 'Ubuntu 26.04 public preview runner: DNS resolution failures for third-party package repositories'
category: runner-environment
severity: error
tags:
- ubuntu-26.04
- runner-image
- dns
- network
- preview
- curl-error-6
- hostname-resolution
- third-party-repos
patterns:
- regex: 'Curl error \(6\).*Could not resolve host|curl.*6.*resolve.*hostname'
flags: 'i'
- regex: 'Could not resolve host.*ubuntu-26|ubuntu-26.*Could not resolve host'
flags: 'i'
- regex: 'getaddrinfo.*Name or service not known.*26\.04|EAI_NONAME.*ubuntu-26'
flags: 'i'
error_messages:
- 'Curl error (6): Could not resolve hostname for https://fedoraproject-updates-archive.fedoraproject.org'
- 'Could not resolve host: dl.fedoraproject.org'
- 'curl: (6) Could not resolve host: packages.example-repo.org'
- 'getaddrinfo: Name or service not known'
root_cause: |
The Ubuntu 26.04 GitHub Actions runner (available as a public preview from
June 11, 2026, as `ubuntu-26.04` and `ubuntu-26.04-arm`) has intermittent
DNS resolution failures for some third-party package repository hostnames.

The Ubuntu 26.04 image switches from the traditional `resolv.conf` / `systemd-
resolved` DNS configuration used in Ubuntu 24.04 to a newer resolver stack.
Some third-party repository mirrors that rely on external DNS servers not
in the runner's configured resolver list may fail to resolve, producing
`Curl error (6): Could not resolve hostname`.

This issue is specific to the preview image. Ubuntu 24.04 runners using the
same workflows on the same network resolve the same hostnames successfully.
The failure is non-deterministic — the same hostname may resolve on some runs
but not others due to timing or caching differences in the new DNS stack.

Known affected scenarios include:
- Installing packages from Fedora/EPEL mirrors via `dnf`/`yum` within
container build steps
- `curl` or `wget` to third-party APT/YUM/pip repository hosts
- Custom Docker image builds that `apt-get` from third-party PPAs during
the build phase

Source: actions/runner-images#14230 (filed June 13, 2026, Ubuntu 26.04
public preview — network related issues).
fix: |
**Short-term workaround while the preview image issue is investigated:**

1. **Pin to ubuntu-24.04** — Use an explicit runner version that does not
have the DNS issue. Ubuntu 24.04 runners are stable and resolve the
same hostnames correctly.

2. **Use `ubuntu-latest`** — `ubuntu-latest` currently aliases ubuntu-24.04
and is unaffected.

3. **Retry logic** — For intermittent failures, add a retry mechanism around
the failing step using `--retry` flags in `curl` or a retry composite action.

4. **Check runner-images for updates** — The Ubuntu 26.04 preview image is
actively updated. Monitor runner-images#14230 for a fix or workaround from
GitHub's runner team.
fix_code:
- language: yaml
label: 'Pin to ubuntu-24.04 until ubuntu-26.04 DNS issues are resolved'
code: |
jobs:
build:
# ubuntu-26.04 has intermittent DNS resolution failures (June 2026 preview)
# Use ubuntu-24.04 or ubuntu-latest (currently aliases ubuntu-24.04) as workaround
runs-on: ubuntu-24.04 # stable DNS, no hostname resolution issues
steps:
- uses: actions/checkout@v4
- name: Install packages from third-party repository
run: |
curl -fsSL https://packages.example-repo.org/install.sh | sh
- language: yaml
label: 'Add retry logic for curl calls on ubuntu-26.04 preview'
code: |
jobs:
build:
runs-on: ubuntu-26.04
steps:
- uses: actions/checkout@v4
- name: Download with retry (workaround for ubuntu-26.04 DNS issues)
run: |
# curl --retry retries on transient failures including DNS (Curl error 6)
curl --retry 3 --retry-delay 5 --retry-all-errors \
-fsSL https://packages.example-repo.org/install.sh | sh
prevention:
- 'Do not migrate CI workflows to ubuntu-26.04 during the public preview phase without testing for DNS resolution issues.'
- 'If using ubuntu-26.04 is required, add `--retry 3 --retry-all-errors` flags to curl/wget calls as a safety net.'
- 'Monitor actions/runner-images for a patch to the ubuntu-26.04 DNS configuration before adopting it in production workflows.'
- 'Use explicit runner version labels (ubuntu-24.04, ubuntu-26.04) rather than ubuntu-latest to control when new image versions take effect.'
docs:
- url: 'https://github.com/actions/runner-images/issues/14230'
label: 'runner-images#14230 — [Ubuntu 26.04] network related issues (June 13, 2026)'
- url: 'https://github.com/actions/runner-images/issues/14226'
label: 'runner-images#14226 — Ubuntu 26.04 and Ubuntu 26.04 Arm public preview announcement (June 11, 2026)'
129 changes: 129 additions & 0 deletions errors/runner-environment/re-504.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
id: re-504
title: 'Ubuntu 26.04: setup-python only pre-caches Python 3.14 — older versions always trigger a live download'
category: runner-environment
severity: warning
tags:
- ubuntu-26.04
- setup-python
- toolcache
- python
- slow-ci
- cache-miss
- runner-image
- migration
patterns:
- regex: 'Downloading Python (3\.10|3\.11|3\.12|3\.13)\.[0-9]+ - ubuntu-26'
flags: 'i'
- regex: 'Python (3\.10|3\.11|3\.12|3\.13).*not.*toolcache.*ubuntu-26|ubuntu-26.*Python.*(3\.10|3\.11|3\.12|3\.13).*not.*cached'
flags: 'i'
- regex: 'No pre-built Python (3\.10|3\.11|3\.12|3\.13) found|Downloading cpython.*3\.(10|11|12|13)'
flags: 'i'
error_messages:
- 'Downloading Python 3.12.x - ubuntu-26.04-x64'
- 'Python 3.11 not found in toolcache; downloading from python.org'
- 'No pre-cached Python 3.10 found for ubuntu-26.04; installing from source'
root_cause: |
The Ubuntu 26.04 GitHub Actions runner image (public preview, June 11 2026)
ships with only Python 3.14 pre-installed in the toolcache. The following
Python versions are NOT pre-cached and trigger a live network download when
requested via `actions/setup-python`:
- Python 3.10.x
- Python 3.11.x
- Python 3.12.x
- Python 3.13.x

By contrast, the ubuntu-24.04 image pre-caches Python 3.10, 3.11, 3.12
(default), 3.13, and 3.14. Workflows that pin `python-version: '3.12'` or
any older version via `setup-python` on ubuntu-26.04 will trigger a
download from GitHub Toolcache servers on every job run.

The download itself typically succeeds and produces the correct Python version,
but the impact is:
- 30–120 seconds of additional download time per job, depending on the
Python version and runner network conditions
- Increased sensitivity to GitHub Toolcache server availability
- Higher CI costs on paid runners proportional to download time

This is expected to be resolved as the ubuntu-26.04 image matures beyond
public preview and GitHub populates the toolcache with additional Python
versions.

Source: actions/runner-images#14226 (Ubuntu 26.04 public preview
announcement, June 11, 2026) — Python row shows "3.14.* (default)" only.
fix: |
**Option 1 — Stay on ubuntu-24.04** (recommended for stable Python toolcache):
Ubuntu 24.04 pre-caches Python 3.10–3.14. Workflows requiring specific
Python versions incur no download overhead.

**Option 2 — Pin python-version: '3.14'** on ubuntu-26.04:
Python 3.14 is the only pre-cached version. If your workflow can target
Python 3.14, there is no download delay.

**Option 3 — Use pip caching to amortize download cost**:
If you must use an older Python version on ubuntu-26.04, add
`cache: 'pip'` to `setup-python` to cache installed packages and reduce
total CI time, even though the Python binary itself downloads each run.
fix_code:
- language: yaml
label: 'Pin to ubuntu-24.04 for consistent Python toolcache (no download for 3.10–3.13)'
code: |
jobs:
test:
# ubuntu-26.04 only pre-caches Python 3.14 — use ubuntu-24.04 for 3.10/3.11/3.12/3.13
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12' # pre-cached on ubuntu-24.04, instant
cache: 'pip'
- run: pip install -r requirements.txt
- run: pytest
- language: yaml
label: 'Use Python 3.14 on ubuntu-26.04 to avoid toolcache download'
code: |
jobs:
test:
runs-on: ubuntu-26.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.14' # only pre-cached version on ubuntu-26.04 at preview launch
cache: 'pip'
- run: pip install -r requirements.txt
- run: pytest
- language: yaml
label: 'Matrix test across Python versions — pin runner per version for speed'
code: |
jobs:
test:
strategy:
matrix:
include:
- python-version: '3.12'
runs-on: ubuntu-24.04 # pre-cached — no download
- python-version: '3.13'
runs-on: ubuntu-24.04 # pre-cached — no download
- python-version: '3.14'
runs-on: ubuntu-26.04 # pre-cached on ubuntu-26.04
runs-on: ${{ matrix.runs-on }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- run: pytest
prevention:
- 'On ubuntu-26.04, only Python 3.14 is pre-cached in the toolcache at public preview launch (June 2026); use ubuntu-24.04 for older versions.'
- 'Add `cache: pip` to setup-python to reduce total job time even when the Python binary must be downloaded.'
- 'Monitor runner-images announcements for when ubuntu-26.04 toolcache is updated to include Python 3.10–3.13.'
- 'Use explicit ubuntu-24.04 or ubuntu-26.04 labels instead of ubuntu-latest to control which toolcache versions are available.'
docs:
- url: 'https://github.com/actions/runner-images/issues/14226'
label: 'runner-images#14226 — Ubuntu 26.04 public preview announcement — Python toolcache row'
- url: 'https://github.com/actions/setup-python'
label: 'actions/setup-python — Python version management in GitHub Actions'
- url: 'https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#caching-dependencies'
label: 'GitHub Docs — Caching Python dependencies'
Loading