Skip to content

Use ccache depend mode for ROCm builds - #35

Merged
sreeram-11 merged 1 commit into
lemonadefrom
sreeram/fix-rocm-stale-ccache-hits
Oct 25, 2026
Merged

sreeram-11 merged 1 commit into
lemonadefrom
sreeram/fix-rocm-stale-ccache-hits

Conversation

@sreeram-11

Copy link
Copy Markdown
Collaborator

Problem

ccache could reuse old GPU code on ROCm 10.1.

  • When ccache's quick check misses, it falls back to a second check that compares the preprocessed source.
  • On ROCm 10.1 that preprocessed output only contains the CPU code, so a change that only touches GPU code (for example inside #if defined(RDNA3)) looks unchanged, and an object with the old GPU code is reused.
  • Saw this on PR Speedup Windows ROCm builds #34's re-run: upstream changed GPU-only code in mma.cuh, and 10.1 reused 69 cached objects in 5 min while 7.14 and 10.0 rebuilt them.
    • The first PR run run #1 was slow and that was expected. Both the hash setting and the ccache version changed, so nothing old can match.
    • However, in the second PR run (run #2), upstream moved from 1a24df16 (b11846) to edc35384 (b11851), and changed GPU code. ROCm 7.14 and 10.0 recompiled correctly but 10.1 did not (it re-used stale GPU code).

FIX
Turn on ccache depend_mode in ubuntu-24-rocm and windows-rocm for all ROCm versions. ccache then only reuses a result when the source file and every file the compiler says it read are unchanged, and never uses the second check.


Notes

  • 7.14 and 10.0 aren't affected today. This change keeps them safe once their compiler picks up the same behaviour.
  • Small speed cost: in recent runs the second check usually saved 0-7 files, occasionally ~80-90.
  • To confirm it's active, every ROCm job's ccache stats should show Preprocessed: 0 (Ubuntu: Post ccache step log; Windows: "CCache Statistics" on this PR's run Summary page).

@sreeram-11
sreeram-11 requested a review from superm1 October 24, 2026 19:59
@sreeram-11

Copy link
Copy Markdown
Collaborator Author

This PR and #34 are meant to go together.

  • Speedup Windows ROCm builds #34 makes Windows ROCm builds fast again. Reinstalling ROCm no longer throws away the whole ccache. In the re-run of this PR, Windows ROCm reinstalled from scratch and still built in 9m 1s / 8m 51s / 11m 9s (7.14/10.0/10.1), down from 2-4h.
  • This PR makes sure the speed doesn't come from skipping work. With depend_mode, ccache only reuses a result when the source and every file it includes are unchanged. All six ROCm jobs show Preprocessed: 0, so the check that earlier let 10.1 reuse old GPU code is no longer used.

@superm1

superm1 commented Oct 25, 2026

Copy link
Copy Markdown
Member

Make sure to send upstream too.

@sreeram-11
sreeram-11 merged commit 9032949 into lemonade Oct 25, 2026
50 of 52 checks passed
@sreeram-11
sreeram-11 deleted the sreeram/fix-rocm-stale-ccache-hits branch October 25, 2026 18:24
@sreeram-11

sreeram-11 commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator Author

Make sure to send upstream too.

Upstream PR submitted and merged: ggml-org/llama.cpp#31231

Upstream is still on ROCm 10.0. PR that bumps ROCm to 10.1 is still open ggml-org/llama.cpp#30652.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants