Repository navigation
CI perf (settings): slow Depot 16-core hosts since 10-13 15:20 add ~1.5–2.5 min to the merge-queue critical path (~0.6k Linux job-min/day) #1512
Description
Filed by the CI profiler routine. This needs a human: the cause is in the Depot runner fleet, not in the repo, so the implementer routine shouldn't pick it up as a code change.
Measurement
Window: 2026-10-13 13:11 → 2026-10-14 00:00 UTC. Sample: jobs from all 52 CI merge_group runs, 41 CI pull_request runs and 15 push runs (2,253 jobs, filter=all).
Two speeds of host on the same runner class. A job is counted as being on a "slow" host when the gap from job start to its first step is over 15 s. Healthy hosts start in about 6 s.
| period (UTC) | depot-ubuntu-24.04-16: share of slow-start jobs |
depot-ubuntu-24.04-4: share of slow-start jobs |
|---|---|---|
| 13:00–15:20 | 21% (n=126, p50 gap 6 s) | 23% (n=999, p50 6 s) |
| 15:20–22:00 | 59% (n=90, p50 23.5 s) | 70% (n=637, p50 23 s) |
| 22:00–00:00 | 57% (n=42, p50 19.5 s) | 24% (n=286, p50 6 s): recovered |
The 4-core class recovered at about 22:00. The 16-core class, which runs clippy and coverage (both on the merge-queue critical path), did not.
coverage → Run tests with coverage step time (successful jobs):
| fast host | slow host | |
|---|---|---|
| 13:00–15:20 | p50 332 s (n=29) | – |
| 15:20–22:00 | p50 324 s (n=13) | p50 424 s (n=17) |
| 22:00+ | 320 s (n=1) | p50 406 s (n=7 of 8) |
The inputs are the same, so the difference is the host. In the 23:49 batch the CI inputs differ by one PR each, and the host coverage TOTAL lines match within 0.02% (350,679 vs 350,622):
- job 115602304980 on
depot-s9bzhj8q5x: start gap 5 s, test step 320 s, job 5.7 min. - job 115602276746 on
depot-xb3zx5n43w: start gap 22 s, test step 466 s, job 8.7 min.
clippy (also 16-core, and coverage waits on it) takes 1.3–1.4 min on fast hosts and 2.1–2.2 min on slow ones, for example 38486575290 compared with 38480343655.
Effect on landing PRs:
- merge_group created →
ci-ok: 7.4–9.1 min before 15:20 (n=14 non-evicted runs at 13:11–13:35). Since then it has been p50 9.5, max 12.2 (n=8 since 20:16).coveragewas the last job beforeci-okin 7 of 8 of the runs since 22:49. - Enqueue → merge: p50 11.3 / p90 12.0 min (n=15). The 10-12 profiles had 8.0–8.8 at the same settings.
- PR created →
ci-ok: p50 9.9 / p90 12.1 (n=19 full runs). It was 7.4–7.6 on 10-12.
Where the time goes
Each slow-host coverage job loses about 15–20 s of start-up and 7–9 s of Checkout/Cache cargo/Install Rust (2–3 s each on fast hosts), plus ~80–145 s in the test step. Each slow-host clippy job loses about 50 s. When both land on slow hosts, which happened in 4 of the 8 runs since 22:49, the critical path grows by ~2–2.5 min.
Root cause
Part of the Depot 16-core pool is running ~25–45% slower on CPU and I/O than the rest. That started at about 15:20 UTC on 10-13. It isn't load from this repo: at 13:00 the repo ran ~700 jobs/hour with a 6 s start gap, and at 21:00 it ran ~33 jobs/hour with a 26 s gap. The 4-core pool recovered at 22:00 and the 16-core pool hasn't.
Proposed fix
- Human, S: check the Depot status page and the org's Depot dashboard for
depot-ubuntu-24.04-16host health since 2026-10-13 15:20 UTC. If nothing is posted, open a Depot support ticket with the two job links above, the runner namesdepot-xb3zx5n43w,depot-c59k7s930p,depot-f5768wcmwc,depot-jctrbg38bj,depot-cg4tbh0c9f,depot-g024vxvvxzanddepot-n4hvg6kjz3, and the start-gap numbers. - Only if Depot can't fix it within a day (repo side, S): in
.github/workflows/ci.yml, movecoverageandclippyfromdepot-ubuntu-24.04-16to another class whose start gap is healthy (for exampledepot-ubuntu-24.04-32, or the 4-core class, which recovered). Measure one batch before committing to it. Don't rename either job:clippyis a required check, andcoveragefeedsci-ok.
Expected saving
- Merge-queue critical path: ~1.5 min p50, 9.5 → ~8. Up to 2.5 min when both jobs land on slow hosts.
- PR feedback (created →
ci-ok): ~1.5 min. - Job-minutes: ~2.5 min per
coveragejob and ~0.8 perclippyjob on slow hosts, at ~57% slow placement. That is about 1.9 min per run, which is ~0.1k per day over 52 merge-queue runs and ~0.3k per day over ~150 PR runs. Adding the other 16-core jobs gives ~0.4–0.6k Linux job-min/day.
Coverage and risk
No test coverage changes. Option 1 doesn't touch the repo. Option 2 changes only runs-on labels, and keeps job names and ci-ok/clippy the same. Risk: another class may be slow too, which is why it should be measured first.
Effort
S. Option 1 is a ticket. Option 2 is a two-line runs-on change.
ROI
(0.5k job-min → 0.5, + 1.5 critical-path min + 1.5 PR-latency min) × confidence 0.5 ÷ effort 1 ≈ 1.75
Activity
mikolalysenko commentedon Oct 14, 2026
Fresh numbers (profiler run 2026-10-14 04:16 UTC): the slowdown looks over since about 01:00 UTC. The sample is small, though.
Share of jobs whose first step started more than 15 s after the job did (a slow-start host). The sample is 118 CI runs from the last 24h (
filter=all):Since 01:00:
Run tests with coveragetook 306–363 s (n=6, p50 315). Between 22:00 and 01:00 it took 401–466 s on slow hosts.clippyp50 was 1.37 min, against 1.47 for 22:00–01:00 and 2.17 for 15:20–22:00.ci-okwas 7.12 / 7.77 / 7.98 / 8.55 min (runs at 01:48, 01:49, 03:49, 03:49). The 8 runs at 22:49–23:49 took 9.1–11.1 min.That is 4 merge_group runs and 27 16-core jobs. I'll keep this open until the next 1–2 profiler runs confirm it on a weekday load. If those show the 16-core slow share under ~25%, I'll close this as resolved externally. No runner-class change is needed.
Generated by Claude Code
mikolalysenko commentedon Oct 14, 2026
Closing: the Depot slowdown has been over for about 7 hours, and the 4h profile at 08:16 UTC confirms it. No repo change was needed. The hosts recovered on their own.
Slow-start share is the share of jobs whose first step started more than 15 s after the job did. Sample: 110 CI runs from 10-13 13:11 to 10-14 08:16, 2,366 jobs,
filter=all.depot-ubuntu-24.04-16depot-ubuntu-24.04-4Effect on landing PRs:
ci-ok: the 5 runs since 04:16 took 7.33, 7.43, 7.60, 7.78 and 7.85 min. The p50 of the 9 runs since 01:00 is 7.77 min. During the slowdown the p50 was 9.5 min, with a max of 12.2. Before 15:20 on 10-13 it was 7.4–9.1 min.If the slow hosts come back, the profiler will reopen this or file a new issue with fresh numbers.
Generated by Claude Code
mikolalysenko commentedon Oct 14, 2026
Reopening: the slow Depot hosts came back in three spells in the last 24h (profiler run 2026-10-14 16:16 UTC). Last profile's rule was to reopen this if the slow-start share stayed above 20% for 2+ hours. It did, at 12:00–14:00 and again at 21:00–02:00.
A job counts as slow-start when its first step starts more than 15 s after the job does. Healthy hosts take about 6 s. Sample: 99 runs from 10-13 21:27 to 10-14 16:16, 1,543 jobs,
filter=all.depot-ubuntu-24.04-16depot-ubuntu-24.04-4By hour across all Linux jobs: 12:xx 36% (33/92), 13:xx 51% (72/142), 15:xx 58% (18/31).
The same inputs run slower on slow hosts. Here is
coverage→Run tests with coveragein successful jobs:clippytakes p50 1.35–1.40 min on fast hosts and 1.72–2.18 min on slow ones.Effect on the merge queue (all 25 merge_group
CIruns since 22:49):coveragelanded on a slow host in 10 of 25 runs. Those runs went from created toci-okin p50 9.64 min (range 9.13–12.93). The other 15 runs took p50 7.80 (range 7.12–9.20). That is about 1.85 min added to 40% of merges.What's still needed is the same as in the body: a human should raise this with Depot. Send the job links above, the hours above, and the fact that both the 4-core and 16-core classes slow down together. Because both classes are affected, moving
coverage/clippyto another class (option 2) is less likely to help than it looked on 10-13. Only try it after Depot confirms that a specific pool is healthy.ROI is unchanged at ~1.75: about 1.85 critical-path minutes on 40% of merges, plus a similar hit to PR feedback, at confidence 0.5 and effort S.
Generated by Claude Code
mikolalysenko commentedon Oct 15, 2026
Fresh numbers (profiler run 2026-10-15 00:16 UTC): slow Depot hosts kept hitting CI through the evening of 10-14. The two merges at 23:48 were the first in 7 hours that ran on fast hosts.
A job counts as slow-start when its first step started more than 15 s after the job did (median gap on slow hosts is 25 s, p90 43 s). Sample: 93 CI runs from 10-14 04:04 → 10-15 00:10, 1,553 Linux jobs (
filter=all).By hour: 17h 84%, 19h 57%, 20h 69%, 21h 56%, 22h 15%, 23h 57%.
Effect on landing PRs (merge-queue REST timelines, enqueue → merge):
ci-ok: 10.6–11.75 min on the 6 slow runs (16:48–20:48), 7.3–8.15 min on fast runs.coveragep50 8.3 min on a slow host vs 5.96 on a fast one (n=15 / 22), and the Gradlee2eleg 6.08 vs 4.55 (n=13 / 28).ci-okon full runs: p50 9.65 / p90 11.37 min (n=16). The 5 with no slow host took 7.3–8.2.The slowdown is still on Depot's side and needs a human to raise it with them. Next update after 10-16 00:00 unless it clears.
Generated by Claude Code