Fix CI flakes from ccache saturation and stale schedule estimates - #11098
Open
Frauschi wants to merge 1 commit into
Open
Fix CI flakes from ccache saturation and stale schedule estimates#11098Frauschi wants to merge 1 commit into
Frauschi wants to merge 1 commit into
Conversation
pq-all and trackmemory restore a seed ccache that already sits at its cap, so every object a run compiles evicts a seeded one that is still needed. Both reported a cache 100% full with over 200 cleanups and a 43-44% hit rate, and their configs took 3-5x their warm time. That left pq-all shards finishing at 23-26 min against a 30 min limit: a healthy shard passed with 5.6 min to spare, so any hiccup ahead of the build took the job down. One did, twice, when the prebuilt deb image came up short and the dependency step fell back to apt for 9.7 min - the only two jobs out of 47 whose install ran past 2 min are the only two that timed out. Size both caches to hold the seeded objects and a run's own alongside them, and widen the limits so a slow install is survivable. Raise the trackmemory limit as well, for a different reason: its wolfEntropy configs have overrun it outright, once with six configs done inside 8 min while all-wolfentropy ran on for another 32. That is a stall rather than a slow run - 91 jobs have a 17.2 min median and a 17.9 min p90, with nothing between 18 min and the cap - so the higher limit is headroom, not a fix. It costs nothing on a healthy run. Recalibrate "minutes", which drives longest-first scheduling and shard balancing. pq-all and fips-dev-no-post had drifted several-fold in both directions: two frodokem cross-builds were declared at 4 min but take 0.3, so the pool kept scheduling them first as its longest work. pq-all is calibrated from runs whose ccache actually hit, which the resized cache makes the normal case; a value read off a cold run makes every warm run warn instead. os-check-linux was already calibrated cold and correct, so only its three genuine outliers change. Both config lists are re-sorted to restore the documented descending order, so most of their diff is a mechanical move rather than a value change. Shard totals come out at 29.4/29.6 for pq-all and 76.9-77.7 across the four os-check shards.
dgarske
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two unrelated CI flakes trace back to the same place, the
parallel-make-check.pyworkflows, so they are fixed together here. No library code is touched - this is workflow and config JSON only.1. ccache saturation in
pq-allandtrackmemoryBoth workflows restore a seed ccache that is already at its cap, so every object a PR run compiles evicts a seeded one that is still needed. The tell is in each job's
ccache statsstep: a cache 100% full with a non-zero cleanup count in the same run.max-sizepq-alltrackmemoryfips-dev-no-post(healthy control)For
pq-allthat left shards finishing at 23-26 min against a 30 min limit. A healthy shard passed with 5.6 min to spare, so anything slow ahead of the build took the job down - and something did, twice, when the prebuilt deb image came up short and the dependency step fell back to apt for 9.7 min. Out of 47 jobs, the only two whose install step ran past 2 min are exactly the two that timed out.The fix sizes each cache to hold the seeded objects and a run's own alongside them (
350M -> 700M,250M -> 450M), and widens the timeouts so a slow install step is survivable rather than fatal.2.
trackmemorytimeout headroomRaised
40 -> 60min for a different reason: thewolfEntropyconfigs gather entropy from CPU timing jitter and have overrun the limit outright - once with six configs finished inside 8 min whileall-wolfentropyran on for another 32. That is a stall, not a slow run: across 91 jobs the median is 17.2 min and p90 is 17.9 min, with nothing at all between 18 min and the cap. The higher limit is headroom for the stall, not a fix for it, and it costs nothing on a healthy run.3. Recalibrated
minutesminutesis a scheduling weight only -parallel-make-check.pyruns configs longest-first and--shard K/Nbalances shards by it. A stale value never fails a run, it just packs the schedule worse and prints a non-fatal warning when the measured time lands outside +/-50%.pq-allandfips-dev-no-posthad drifted several-fold in both directions. The clearest case: two frodokem cross-builds were declared at 4 min but actually take 0.3, so the pool kept scheduling them first as its longest work.pq-allis calibrated from runs whose ccache actually hit, which the resized cache makes the normal case; a figure read off a cold run would make every warm run warn instead.os-check-linuxwas already calibrated cold and correct, so only its three genuine outliers change:user-settings-all-compatall-asn-templatetsp-verifierChanges
.github/workflows/pq-all.ymlmax-size350M -> 700M,timeout-minutes30 -> 45, plus comments recording the measurement and the warm-calibration rule.github/workflows/trackmemory.ymlmax-size250M -> 450M,timeout-minutes40 -> 60.github/workflows/fips-dev-no-post.ymltimeout-minutes30 -> 40 (set bykernel-settings-all-pqc-asm, which alone measures 23 min cold against a 23 min job total), and recalibratedminutesfor all seven inline configs.github/configs/pq-all.jsonminutes, list re-sorted.github/configs/os-check-linux.jsonminutescorrections, list re-sortedBoth config lists are kept sorted by
minutesdescending (cosmetic - the scheduler sorts by value, not list order), so most of their diff is a mechanical move of unchanged entries rather than a value change.Note on the Actions cache budget
The repository's total Actions cache is capped at 10 GB and wolfSSL currently sits near it, so raising a
max-sizeaccelerates LRU eviction of other workflows' caches. The two raises here are the smallest that clear the eviction threshold for their working sets; both are still well under the 1.5 GB thatsmoke-ccache-masteroccupies.