Skip to content

GHA: Add NumPy SIMD dispatch diagnostic for flaky Weber_BMC2015 (#3078) - #3231

Open
dweindl wants to merge 1 commit into
AMICI-dev:mainfrom
dweindl:avx512-causal-probe
Open

GHA: Add NumPy SIMD dispatch diagnostic for flaky Weber_BMC2015 (#3078)#3231
dweindl wants to merge 1 commit into
AMICI-dev:mainfrom
dweindl:avx512-causal-probe

Conversation

@dweindl

@dweindl dweindl commented Aug 10, 2026

Copy link
Copy Markdown
Member

Summary

Follow-up to #3078 and #3229. The AVX-512-presence correlation reported
there was real but indirect: the actual root cause is NumPy's own runtime
SIMD dispatch for transcendental ufuncs (np.power, np.exp), used when
AMICI unscales log10/log-scaled PEtab parameters back to linear scale.
Weber_BMC2015 has all 40 of its parameters on log10 scale, so this path
is exercised heavily. NumPy's optimized kernels at different SIMD widths
(X86_V3 vs. X86_V4/AVX512_ICL/AVX512_SPR) aren't required to agree
bit-for-bit -- documented, accepted NumPy behavior -- and it silently picks
the widest tier the host CPU supports. AVX-512 presence only predicted the
outcome because it gates which tier NumPy even has the option to pick; it
is not a causal pathway through AMICI's own C++ core, SUNDIALS, SuiteSparse,
glibc, or OpenBLAS, all of which were directly tested (via causal probes on
real AVX-512 CI hardware) and refuted. Full writeup:
#3078 (comment)

This PR previously carried the exploratory causal-probe steps
(GLIBC_TUNABLES, OPENBLAS_CORETYPE=HASWELL, a full state-trajectory
dump) used to reach that conclusion. Now that the mechanism is confirmed,
those are removed, leaving only the one diagnostic that's actually useful
going forward.

What this adds

A step that prints NumPy's actual runtime SIMD dispatch state
(numpy._core._multiarray_umath.__cpu_baseline__/__cpu_dispatch__/
__cpu_features__) right after NumPy becomes available in the job. Generic
hardware diagnostics (dweindl/gha-runner-diagnostics@v1, already present,
merged in #3229) run before Python/NumPy even exist and can only show raw
CPU feature flags -- not which dispatch tier NumPy actually selected. This
shows the real variable directly, rather than requiring it to be inferred
indirectly from CPU flags as this investigation had to do.

Test plan

  • Verified the exact snippet runs correctly and reports the expected
    fields locally.
  • YAML validated.
  • Confirm it runs cleanly in CI on both extract_subexpressions matrix
    legs.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.97%. Comparing base (be70933) to head (4634219).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3231      +/-   ##
==========================================
- Coverage   78.55%   77.97%   -0.58%     
==========================================
  Files         318      318              
  Lines       21106    21106              
  Branches     1488     1488              
==========================================
- Hits        16579    16458     -121     
- Misses       4519     4640     +121     
  Partials        8        8              
Flag Coverage Δ
cpp 72.00% <ø> (+0.04%) ⬆️
cpp_python 36.56% <ø> (ø)
petab 48.29% <ø> (ø)
petab_sciml 16.28% <ø> (ø)
petab_sciml_benchmarks 14.85% <ø> (ø)
python 70.37% <ø> (+0.04%) ⬆️
sbmlsuite-jax ?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…I-dev#3078)

Root cause of the AVX-512-correlated flakiness turned out to be NumPy's
own runtime SIMD dispatch for transcendental ufuncs (np.power/np.exp),
used when unscaling log10/log-scaled PEtab parameters -- not AMICI's own
code, SUNDIALS, SuiteSparse, glibc, or OpenBLAS, all of which were
directly tested and refuted. See
AMICI-dev#3078 (comment)
for the full writeup.

Add a step that prints NumPy's actual runtime dispatch state
(numpy._core._multiarray_umath.__cpu_features__/__cpu_dispatch__) right
after NumPy becomes available, so future occurrences show the real
variable directly instead of requiring it to be inferred from raw CPU
feature flags. Generic hardware diagnostics
(dweindl/gha-runner-diagnostics@v1) can't provide this, since they run
before Python/NumPy even exist in the job.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dweindl
dweindl force-pushed the avx512-causal-probe branch from 247200c to 4634219 Compare August 10, 2026 18:52
@dweindl dweindl changed the title GHA: Add AVX-512 causal probe for flaky Weber_BMC2015 (#3078) GHA: Add NumPy SIMD dispatch diagnostic for flaky Weber_BMC2015 (#3078) Aug 10, 2026
@dweindl
dweindl marked this pull request as ready for review August 10, 2026 18:55
@dweindl
dweindl requested a review from a team as a code owner August 10, 2026 18:55
@dweindl
dweindl enabled auto-merge August 10, 2026 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant