Skip to content

Speedup Windows ROCm builds - #34

Merged
sreeram-11 merged 1 commit into
lemonadefrom
sreeram/fix-windows-rocm-ccache-misses
Oct 24, 2026
Merged

sreeram-11 merged 1 commit into
lemonadefrom
sreeram/fix-windows-rocm-ccache-misses

Conversation

@sreeram-11

@sreeram-11 sreeram-11 commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Windows ROCm builds randomly took 2-4h even when the source hadn't changed.

Problem

  • Whenever the cached ROCm install got evicted, ROCm was reinstalled.
  • pip gives clang.exe a new modified time on every install, and with ccache's default compiler_check=mtime that makes every cache lookup miss.
  • Ubuntu got this fix in August (11cded4) but Windows never did.

Fix

  1. Set ccache to compiler_check=content (+ Ubuntu's sloppiness flags) in windows-rocm, matching ubuntu-24-rocm.
  2. Removed the rocm-wheels cache and now install ROCm every run, like Ubuntu. Costs ~1 min per job but frees ~3.3 GB of the repo's 10 GB cache quota, which we're currently over.
  3. Bumped ggml-org/ccache-action from v1.2.16 to v1.2.21 (ccache 4.9 to 4.13), so each run's Summary page shows ccache hit/miss stats. The build logs are ~1.5 GB and get truncated, so the stats never show up there.

Notes

  • The first run on this PR will be a full cold build (~3.5-4h per Windows ROCm job), because the ccache hash method and version both changed. When re-run after the first run finishes, the Windows ROCm builds should then take ~10 min, with near-100% hits under "CCache Statistics".
  • This doesn't fix runs where upstream touches core headers (ggml.h, common.cuh, etc.). Those force a full rebuild on both Windows and Ubuntu and will still take 2.5-4h.
  • The old rocm-wheels-*-Windows-v2 caches are no longer used and will age out on their own (or can be deleted manually).

@sreeram-11
sreeram-11 requested a review from superm1 October 23, 2026 19:52

@superm1 superm1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please send same upstream

@sreeram-11
sreeram-11 merged commit fc47223 into lemonade Oct 24, 2026
50 of 52 checks passed
@sreeram-11
sreeram-11 deleted the sreeram/fix-windows-rocm-ccache-misses branch October 24, 2026 18:43
@sreeram-11

Copy link
Copy Markdown
Collaborator Author

@superm1,

Bumping ccache-action from v1.2.16 to v1.2.21 in this PR is what actually fixed the Windows ROCm build jobs. Starting from v1.2.21, the action sets compiler_check=content on Windows by itself. The explicit "Tune ccache" step added here is redundant but harmless.

This change isn't needed upstream. Upstream already uses ggml-org/ccache-action@v1.2.24 (which sets compiler_check=content on Windows) and doesn't have a ROCm wheel cache. Its Windows ROCm builds take 3 to 6 min every run, with ccache stats of 154 hits out of 155.

@superm1

superm1 commented Oct 27, 2026

Copy link
Copy Markdown
Member

IMO I would just mirror what upstream does. Redundancy is tech debt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants