ci: improve runner disk space usage situation#213
Conversation
d4cee4c to
c9dabc8
Compare
| egress-policy: audit | ||
|
|
||
| - name: report disk space (before) | ||
| run: df -h / /mnt 2>/dev/null || df -h / |
There was a problem hiding this comment.
Can we just use https://github.com/ublue-os/remove-unwanted-software as-is here, ex: https://github.com/bleggett/drelbsos-kernel/actions/runs/29326079638/job/87062603519#step:2:1
so we don't DIY this when there's an off-the-shelf equivalent.
There was a problem hiding this comment.
I was thinking of making an edera-dev/actions repo for some of these common tasks which we'll need for all build workflows, but wanted to prove this out first. The remove-unwanted-software one is good but not quite as thorough as the one I am trying out here.
There was a problem hiding this comment.
Ok - we have https://github.com/edera-dev/actions already and use actions from it across several repos, so that would be the place to add this if we want it centralized.
There was a problem hiding this comment.
Oh, it already exists, I didn't even check. 😆
Yeah, I want to prove out these things in linux-kernel-oci, then move the stuff we can use commonly across our build workflows into there.
c9dabc8 to
a57ca0b
Compare
| # Some runner generations carry a secondary temp disk at /mnt; its | ||
| # size varies by VM SKU and is not reliably documented, so route the | ||
| # build trees there only when it measurably has more free space than | ||
| # the root disk. sudo because /mnt is root-owned while docker.sh and | ||
| # the in-container build user both run unprivileged. |
There was a problem hiding this comment.
We probably don't even need this really, the "delete crap" step should free dozens of gigs and that will be enough.
a57ca0b to
999c2cb
Compare
|
Interesting discovery during testing this. Here's the "before" usage reported by one of the jobs: Before cleaning up we already had 109GB free, it's shocking we managed to exhaust that space in one of the kernel builds... |
Agreed it's suspicious, I'm not sure how we could have legitimately accumulated that much disk usage in a single run, but whatever - all of this is Stuff We Need anyway so if it turns out to be something else causing the issue, I'd still want this in place. |
A recent zone-amdgpu build died with ENOSPC and the logs offered no way to see how close to the limit builds normally run, or what the disk layout of a given runner class even is (relevant since the edera-large runner's specs are only visible to org admins). Print df for / and /mnt before docker setup and after the build; the after step runs on failure too, so an ENOSPC post-mortem shows where the space went. Signed-off-by: Steven Noonan <steven@edera.dev>
firmware.sh extracted the entire linux-firmware tarball (5194 members, several GB) and then, for the zone-amdgpu flavor - the only flavor that receives a FIRMWARE_URL at all - immediately deleted everything except the amdgpu/ directory. The full extraction existed only to be rm -rf'd, and its peak disk usage took out a zone-amdgpu builder with ENOSPC (run 29428894669: "unxz: /build/override-firmware.tar: Write error: No space left on device" with 36MB left on the runner). Extract just the amdgpu subtree instead: GNU tar member selection with --wildcards 'linux-firmware-*/amdgpu/*' pulls exactly the 675 amdgpu members (verified against linux-firmware-20260622: 110MB extracted, one top-level directory, nothing left to delete). Flavors other than zone-amdgpu keep the full extraction, should any ever start passing a FIRMWARE_URL. Signed-off-by: Steven Noonan <steven@edera.dev>
The extracted kernel source and object trees are the dominant disk consumers of a build, and they live in the compile container's writable layer - i.e. on the host root disk, which GitHub-hosted runners keep mostly full (the preinstalled toolcache leaves ~20GB free) while a mostly empty ~65GB temp disk sits unused at /mnt. That imbalance recently cost us a zone-amdgpu build (ENOSPC in run 29428894669). Teach the docker-script generator an optional KERNEL_SCRATCH_DIR: when set, /build/src and /build/obj are bind-mounted from directories under it instead of living in the container layer. CI sets it to /mnt/kernel-scratch (with a sudo'd prepare step, since /mnt is root-owned and both docker.sh and the in-container build user run unprivileged); local builds leave it unset and behave exactly as before, though a developer can point it at any big disk the same way. Signed-off-by: Steven Noonan <steven@edera.dev>
The GitHub runner image preinstalls tooling that kernel builds never touch - the Android SDK, .NET, CodeQL, Haskell, and Swift - which together eat tens of GB of the root disk. Even with the source/obj trees routed to the /mnt scratch disk, the root disk still holds the docker/containerd image store and the workspace, and a zone-amdgpu build recently exhausted it (run 29428894669). Delete the unused tooling at the start of every build job. The rm paths are no-ops where an image doesn't ship them, so this is safe across runner classes (ubuntu-latest, ubuntu-24.04-arm, edera-large all use the same image family). Costs well under a minute and the surrounding disk-space report steps show exactly what it buys on each class. Signed-off-by: Steven Noonan <steven@edera.dev>
999c2cb to
fc935d7
Compare
2 participants
A zone-amdgpu build recently died mid-build with ENOSPC (
unxz: /build/override-firmware.tar: Write error: No space left on device, with the runner reporting 36MB free). Investigating turned up three compounding problems: builds had no disk-usage visibility at all, the firmware step extracted the entire linux-firmware tree only to delete everything butamdgpu/two lines later, and everything a build writes - the extracted kernel source and object trees, the docker/containerd image store, the workspace - lands on the runner's root disk, which the preinstalled toolcache keeps ~70% full on current x64 images.This PR addresses each of those. Every build job now reports
dffor/and/mntat three points: on arrival, after reclaiming space, and after the build (that last one runs on failure too, so the next ENOSPC comes with a post-mortem). Beyond debugging, this gives us empirical per-class disk numbers, including for theedera-largerunner whose specs are only visible to org admins.Each build job then deletes the preinstalled tooling kernel builds never touch - language runtimes (Java, .NET, Swift, Haskell, Julia, Rust), the Android SDK, browsers, PowerShell, the Azure and Google Cloud CLIs, and the entire hosted toolcache - reclaiming tens of GB on the root disk. The step runs before any tool-installing action so that installers (e.g. cosign-installer, which places cosign in the toolcache) re-create what they need afterwards; the merge job, which serves Python from the toolcache via setup-python, is untouched.
The firmware step now extracts only the
linux-firmware-*/amdgpu/*members it actually keeps: 675 of 5194 members, 110MB instead of multiple GB at peak, with nothing left over to delete.Finally, the docker-script generator gains an optional
KERNEL_SCRATCH_DIRthat bind-mounts the source and object trees onto a host directory instead of the container's writable layer. Some runner generations carry a secondary temp disk at/mnt, but its size varies by VM SKU and the public documentation is poor (community references range from a ~14GB mostly-swap disk to nothing at all), so CI makes no assumptions: a prepare step compares free space at runtime and only routes the build trees to/mntwhen it measurably beats the root disk, logging the comparison either way. Local builds are unaffected - the variable is simply unset outside CI, though a developer with a big disk can opt in the same way.