Skip to content

Fix CI flakes from ccache saturation and stale schedule estimates - #11098

Open
Frauschi wants to merge 1 commit into
wolfSSL:masterfrom
Frauschi:script_fix
Open

Fix CI flakes from ccache saturation and stale schedule estimates#11098
Frauschi wants to merge 1 commit into
wolfSSL:masterfrom
Frauschi:script_fix

Conversation

@Frauschi

@Frauschi Frauschi commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Description

Two unrelated CI flakes trace back to the same place, the parallel-make-check.py workflows, so they are fixed together here. No library code is touched - this is workflow and config JSON only.

1. ccache saturation in pq-all and trackmemory

Both workflows restore a seed ccache that is already at its cap, so every object a PR run compiles evicts a seeded one that is still needed. The tell is in each job's ccache stats step: a cache 100% full with a non-zero cleanup count in the same run.

Workflow old max-size cleanups hit rate effect
pq-all 350M 223 43% configs at 3-5x their warm time
trackmemory 250M 232 44% configs at 2-3x their warm time
fips-dev-no-post (healthy control) 500M 0 (21% full) 89% -

For pq-all that left shards finishing at 23-26 min against a 30 min limit. A healthy shard passed with 5.6 min to spare, so anything slow ahead of the build took the job down - and something did, twice, when the prebuilt deb image came up short and the dependency step fell back to apt for 9.7 min. Out of 47 jobs, the only two whose install step ran past 2 min are exactly the two that timed out.

The fix sizes each cache to hold the seeded objects and a run's own alongside them (350M -> 700M, 250M -> 450M), and widens the timeouts so a slow install step is survivable rather than fatal.

2. trackmemory timeout headroom

Raised 40 -> 60 min for a different reason: the wolfEntropy configs gather entropy from CPU timing jitter and have overrun the limit outright - once with six configs finished inside 8 min while all-wolfentropy ran on for another 32. That is a stall, not a slow run: across 91 jobs the median is 17.2 min and p90 is 17.9 min, with nothing at all between 18 min and the cap. The higher limit is headroom for the stall, not a fix for it, and it costs nothing on a healthy run.

3. Recalibrated minutes

minutes is a scheduling weight only - parallel-make-check.py runs configs longest-first and --shard K/N balances shards by it. A stale value never fails a run, it just packs the schedule worse and prints a non-fatal warning when the measured time lands outside +/-50%.

pq-all and fips-dev-no-post had drifted several-fold in both directions. The clearest case: two frodokem cross-builds were declared at 4 min but actually take 0.3, so the pool kept scheduling them first as its longest work. pq-all is calibrated from runs whose ccache actually hit, which the resized cache makes the normal case; a figure read off a cold run would make every warm run warn instead.

os-check-linux was already calibrated cold and correct, so only its three genuine outliers change:

Config before after
user-settings-all-compat 3.0 9.5
all-asn-template 8.5 3.5
tsp-verifier 1.3 3

Changes

File Change
.github/workflows/pq-all.yml max-size 350M -> 700M, timeout-minutes 30 -> 45, plus comments recording the measurement and the warm-calibration rule
.github/workflows/trackmemory.yml max-size 250M -> 450M, timeout-minutes 40 -> 60
.github/workflows/fips-dev-no-post.yml timeout-minutes 30 -> 40 (set by kernel-settings-all-pqc-asm, which alone measures 23 min cold against a 23 min job total), and recalibrated minutes for all seven inline configs
.github/configs/pq-all.json recalibrated minutes, list re-sorted
.github/configs/os-check-linux.json three minutes corrections, list re-sorted

Both config lists are kept sorted by minutes descending (cosmetic - the scheduler sorts by value, not list order), so most of their diff is a mechanical move of unchanged entries rather than a value change.

Note on the Actions cache budget

The repository's total Actions cache is capped at 10 GB and wolfSSL currently sits near it, so raising a max-size accelerates LRU eviction of other workflows' caches. The two raises here are the smallest that clear the eviction threshold for their working sets; both are still well under the 1.5 GB that smoke-ccache-master occupies.

pq-all and trackmemory restore a seed ccache that already sits at its
cap, so every object a run compiles evicts a seeded one that is still
needed. Both reported a cache 100% full with over 200 cleanups and a
43-44% hit rate, and their configs took 3-5x their warm time. That left
pq-all shards finishing at 23-26 min against a 30 min limit: a healthy
shard passed with 5.6 min to spare, so any hiccup ahead of the build
took the job down. One did, twice, when the prebuilt deb image came up
short and the dependency step fell back to apt for 9.7 min - the only
two jobs out of 47 whose install ran past 2 min are the only two that
timed out. Size both caches to hold the seeded objects and a run's own
alongside them, and widen the limits so a slow install is survivable.

Raise the trackmemory limit as well, for a different reason: its
wolfEntropy configs have overrun it outright, once with six configs
done inside 8 min while all-wolfentropy ran on for another 32. That is
a stall rather than a slow run - 91 jobs have a 17.2 min median and a
17.9 min p90, with nothing between 18 min and the cap - so the higher
limit is headroom, not a fix. It costs nothing on a healthy run.

Recalibrate "minutes", which drives longest-first scheduling and shard
balancing. pq-all and fips-dev-no-post had drifted several-fold in both
directions: two frodokem cross-builds were declared at 4 min but take
0.3, so the pool kept scheduling them first as its longest work. pq-all
is calibrated from runs whose ccache actually hit, which the resized
cache makes the normal case; a value read off a cold run makes every
warm run warn instead. os-check-linux was already calibrated cold and
correct, so only its three genuine outliers change.

Both config lists are re-sorted to restore the documented descending
order, so most of their diff is a mechanical move rather than a value
change. Shard totals come out at 29.4/29.6 for pq-all and 76.9-77.7
across the four os-check shards.
@Frauschi Frauschi self-assigned this Aug 6, 2026
@Frauschi Frauschi assigned wolfSSL-Bot and unassigned Frauschi Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants