GHA: Add NumPy SIMD dispatch diagnostic for flaky Weber_BMC2015 (#3078) - #3231
Open
dweindl wants to merge 1 commit into
Open
GHA: Add NumPy SIMD dispatch diagnostic for flaky Weber_BMC2015 (#3078)#3231dweindl wants to merge 1 commit into
dweindl wants to merge 1 commit into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3231 +/- ##
==========================================
- Coverage 78.55% 77.97% -0.58%
==========================================
Files 318 318
Lines 21106 21106
Branches 1488 1488
==========================================
- Hits 16579 16458 -121
- Misses 4519 4640 +121
Partials 8 8
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
…I-dev#3078) Root cause of the AVX-512-correlated flakiness turned out to be NumPy's own runtime SIMD dispatch for transcendental ufuncs (np.power/np.exp), used when unscaling log10/log-scaled PEtab parameters -- not AMICI's own code, SUNDIALS, SuiteSparse, glibc, or OpenBLAS, all of which were directly tested and refuted. See AMICI-dev#3078 (comment) for the full writeup. Add a step that prints NumPy's actual runtime dispatch state (numpy._core._multiarray_umath.__cpu_features__/__cpu_dispatch__) right after NumPy becomes available, so future occurrences show the real variable directly instead of requiring it to be inferred from raw CPU feature flags. Generic hardware diagnostics (dweindl/gha-runner-diagnostics@v1) can't provide this, since they run before Python/NumPy even exist in the job. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dweindl
force-pushed
the
avx512-causal-probe
branch
from
August 10, 2026 18:52
247200c to
4634219
Compare
dweindl
marked this pull request as ready for review
August 10, 2026 18:55
dweindl
enabled auto-merge
August 10, 2026 18:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #3078 and #3229. The AVX-512-presence correlation reported
there was real but indirect: the actual root cause is NumPy's own runtime
SIMD dispatch for transcendental ufuncs (
np.power,np.exp), used whenAMICI unscales
log10/log-scaled PEtab parameters back to linear scale.Weber_BMC2015has all 40 of its parameters onlog10scale, so this pathis exercised heavily. NumPy's optimized kernels at different SIMD widths
(
X86_V3vs.X86_V4/AVX512_ICL/AVX512_SPR) aren't required to agreebit-for-bit -- documented, accepted NumPy behavior -- and it silently picks
the widest tier the host CPU supports. AVX-512 presence only predicted the
outcome because it gates which tier NumPy even has the option to pick; it
is not a causal pathway through AMICI's own C++ core, SUNDIALS, SuiteSparse,
glibc, or OpenBLAS, all of which were directly tested (via causal probes on
real AVX-512 CI hardware) and refuted. Full writeup:
#3078 (comment)
This PR previously carried the exploratory causal-probe steps
(
GLIBC_TUNABLES,OPENBLAS_CORETYPE=HASWELL, a full state-trajectorydump) used to reach that conclusion. Now that the mechanism is confirmed,
those are removed, leaving only the one diagnostic that's actually useful
going forward.
What this adds
A step that prints NumPy's actual runtime SIMD dispatch state
(
numpy._core._multiarray_umath.__cpu_baseline__/__cpu_dispatch__/__cpu_features__) right after NumPy becomes available in the job. Generichardware diagnostics (
dweindl/gha-runner-diagnostics@v1, already present,merged in #3229) run before Python/NumPy even exist and can only show raw
CPU feature flags -- not which dispatch tier NumPy actually selected. This
shows the real variable directly, rather than requiring it to be inferred
indirectly from CPU flags as this investigation had to do.
Test plan
fields locally.
extract_subexpressionsmatrixlegs.
🤖 Generated with Claude Code