ci: name the kernel behind a GPU memory fault, and stop CI condemning healthy nodes - #1800

Merged

sbryngelson merged 36 commits into

ci/gpu-fault-diagnostics

Sep 4, 2026

Commits on Sep 2, 2026

test: diagnose a GPU memory fault on the retry that follows it
sbryngelson
committed
7cdbb5e
test: bound the diagnostic retry's output
sbryngelson
committed
95ecca7
DO NOT MERGE: inject a GPU fault to exercise the retry diagnostics in CI
sbryngelson
committed
3ef4c28
DO NOT MERGE: confine the injected fault to GPU builds
sbryngelson
committed
cc4b52e
DO NOT MERGE: enlarge the injected offset so it actually faults
sbryngelson
committed
c2d0579
fix: make the GPU-fault marker one the retry actually reads
sbryngelson
committed
462a42b
fix: serialize kernel dispatch so the fault trace names the right kernel
sbryngelson
committed
45ec609
docs: record that CCE does not honour the HIP serialization variables
sbryngelson
committed
b9a0585
fix: keep only the offload diagnostic that adds information
sbryngelson
committed
ce6c05f
feat: make the first failure informative instead of retrying for it
sbryngelson
committed
d67d291
DO NOT MERGE: measure whether auto_async_none fixes CCE fault attribution
sbryngelson
committed
7e21c29
Revert "DO NOT MERGE: measure whether auto_async_none fixes CCE fault attribution"
sbryngelson
committed
bac1376
Revert the injected GPU fault; the experiment is finished
sbryngelson
committed
0c85e5a
fix: address review findings on the fault diagnostics
sbryngelson
committed
72ecd30
docs: CCE faults CAN be attributed; correct the claim that they cannot
sbryngelson
committed
4f98317
feat: give CCE a faulting kernel via the ROCm debug agent
sbryngelson
committed
282e0bd
fix: CCE OpenMP attribution is expected, not measured
sbryngelson
committed
9b54efa
fix: the agent summarizer returned nothing on ROCm 7.2.0
sbryngelson
committed
d544695
docs: all four GPU lanes measured; symbol form follows the compiler
sbryngelson
committed
788892d
feat: diagnose GPU faults in bench and case-opt too, and say when the summary is missing
sbryngelson
committed
8ac4682
fix: do not hijack a developer's own GPU debugging session
sbryngelson
committed
dc73590
docs: record the measured cost of the fault diagnostics
sbryngelson
committed
cddc6d8
fix: remove offload variables that made every GPU test 67x slower
sbryngelson
committed
876ea4d
refactor: cut the diagnostics patch down to what earns its place
sbryngelson
committed
c33d346
Merge branch 'master' into ci/gpu-fault-diagnostics
sbryngelson
authored
e5769e9
DO NOT MERGE: re-inject the GPU fault to exercise the diagnostics in CI
sbryngelson
committed
79c5e2e

Commits on Sep 3, 2026

Commits on Sep 4, 2026