CI performance dashboard #1182

Maintained by the CI profiler routine, which runs every 4h. Each run replaces this body and carries the history rows over.

Headline (run 2026-10-15 08:16 UTC)

Window: 2026-10-14 08:16 → 2026-10-15 08:16 (run list reaches back to 12:24). 23 PRs merged. All 19 merge_group CI runs in the sample succeeded, with 0 evictions, 0 failures, 0 cancellations and 0 re-runs. 0 non-success jobs in the 1,723 sampled. Entries still arrive in pairs at about :48 each hour; only 2 merges (#1559, #1560 at 05:49) since the last run.

• The 10-15 nightly was green. Run 38688589166 (05:46 → 06:11, 25 min) passed every leg, including the old-uv legs that failed on 10-14. #1525 delivered.
• Depot hosts stayed fast overnight. Slow-start share of Depot Linux jobs: 00h 6% (9/161), 01h 0% (0/158), 02h 17%, 03h 0%, 05h 21% (53/254, 48 of them nightly jobs), 06h 0% (0/155), 08h 0% (0/29). The 9 entries enqueued 23:48–05:49 took 8.07–9.75 min enqueue → merge, against 11.37–12.42 for the 6 entries at 16:48–20:48. #1512 stays open, with no comment (last fresh numbers 10-15 00:20, one a day): it relapsed once after a quiet night, so it waits for a clean busy day (08h–20h) before closing.
• 6 full-matrix CI dispatches now in the window on agent/fix-ci-*-envelope-asserts branches (Tracking: assert stable codes instead of human sentences in CLI tests, and fold the covgap suites #1089 test refactors), triggered from the maintainer account: 23:03, 00:09, 01:19, 03:17, 05:03, 06:18. At ~380 job-min each (240 Linux / 78 Windows / 62 macOS) that is ~1.44k / 0.47k / 0.37k job-min, more than all merge_group runs combined and all of the macOS minutes apart from the nightly. macOS queue wait stayed at 0.13 / 0.17 min, so there is no runner pressure. Still not filed: it is a manual choice, and since the refactors change e2e assertions that only the full tier runs, the full matrix may be intended. If it is not, a scope choice input on workflow_dispatch in ci.yml (next to hosted_e2e; today every full-tier if: keys on github.event_name == 'workflow_dispatch') would let a dispatch run only test + the PR tier.
• No new issues. Nothing closed. Nothing new cleared the bar. #1248 gets no comment, because no batch went above 2 entries.
• CI: scan mod tests assert codes, not sentences (#1089) #1560 (ci-perf label) merged at 05:57. Open ci-perf PRs: CI: vex tests assert codes, not sentences (#1089) #1563, CI: cli suite tests assert codes, not sentences (#1089) #1565 (Tracking: assert stable codes instead of human sentences in CLI tests, and fold the covgap suites #1089 test refactors, no CI-time effect) and CI: fold last top-level covgap test binaries (#1089) #1568 "fold last top-level covgap test binaries", which may shorten Compile the CLI and every CLI test target / coverage link time; it will be measured after it merges.

Sample: 300 of 369 runs in 24h. The run list is capped at 3 pages, which reach back to 10-14 12:24. Job detail (filter=all, first page) was fetched for 73 runs:

• all 19 CI merge_group runs
• all 13 push runs
• the nightly and the 6 CI dispatches (first 100 jobs each; dispatch cost carried from the full 245-job count of the last run)
• every third CI pull_request run (34)

That is 1,723 non-skipped jobs. Merge-queue timelines were read for the 15 most recent merges. No job logs were read, because the step timings were enough. The run used about 100 REST requests.

Top jobs by sampled minutes (PR + merge_group + push):

1. coverage: 240 min, p50 5.92 / p90 8.87
2. Gradle e2e leg: 152 min, p50 4.66 / p90 6.88
3. e2e_vex_build pip: 84 min, p50 2.71
4. e2e_redirect_npm_build: 83 min, p50 2.60
5. hosted-e2e: 82 min, p50 1.55 / p90 2.98
6. clippy: 69 min, p50 1.45 / p90 2.23
7. e2e-build: 68 min, p50 1.35
8. yarn-classic: 55 min
9. yarn-berry: 54 min
10. windows-compile-check: 53 min, p50 2.65
11. node-addon: 49 min

Top steps: Run e2e tests 376 min (510 jobs), Run tests with coverage 187 (43), Checkout 62, Run hosted-mode production e2e 46, Set up job 44, Install Rust 41 (324 jobs), Compile the CLI and every CLI test target 39.

Critical path (unchanged): clippy → coverage → ci-ok, in parallel with e2e-build → Gradle e2e leg → ci-ok. On fast hosts the two paths finish within about 1 min of each other.

Other items watched, not filed (below the bar):

• Full-matrix CI dispatches (~380 job-min each, 6 in the window, see above): a manual choice
• Install Rust (~0.12k/day spread over 324 sampled jobs)
• Sharding coverage or splitting the Gradle leg (~1 min, because the two paths finish within a minute of each other on fast hosts)
• The hosted fail-fast watcher (no cost)

Merge-queue settings (ruleset 24668460, read 08:20): unchanged.

• max_entries_to_build 5 and max_entries_to_merge 5
• min_entries_to_merge_wait_minutes 5
• ALLGREEN, timeout 90, SQUASH

History

Open ci-perf issues by ROI

ROI on one scale: (thousands of job-min/day weighted Linux ×1, Windows ×2, macOS ×3, plus 1 per critical-path or PR-latency minute) × confidence ÷ effort (S=1, M=2, L=3).

Open ci-perf / ci-janitor PRs: #1568 fold last top-level covgap test binaries (#1089; may cut coverage compile/link time, to be measured after merge), #1565 and #1563 (#1089 test refactors, no CI-time effect).

Merged ci-perf / ci-janitor PRs (last 7 days) and measured effect

Method notes

• GraphQL is blocked in this environment, so merge-queue state comes from REST timelines of merged PRs. The timelines use the last enqueue before the merge.
• Jobs are fetched with filter=all. Queue wait = job started_at − created_at; skipped jobs are excluded. "Start-up gap" = job started_at → first step started_at; over 15 s counts as a slow host.
• Critical path = merge_group run created_at → ci-ok completed_at.
• Daily job-min = per-run mean × 24h run count per event. The PR count is scaled from the 3 listed pages: this run, 101 PR runs over ~19.9 h, about 122 per day. Merges per day come from the closed-PR list.

Generated by Claude Code