Skip to content

Fix flaky Weber_BMC2015 gradient check with AMICI-native RobustConsistency - #3227

Open
dweindl wants to merge 7 commits into
AMICI-dev:mainfrom
dweindl:worktree-gh-3078
Open

Fix flaky Weber_BMC2015 gradient check with AMICI-native RobustConsistency#3227
dweindl wants to merge 7 commits into
AMICI-dev:mainfrom
dweindl:worktree-gh-3078

Conversation

@dweindl

@dweindl dweindl commented Aug 7, 2026

Copy link
Copy Markdown
Member

Summary

  • Fixes the long-standing intermittent test_benchmark_gradient[Weber_BMC2015-*-unscaled] failures (Check flaky test_benchmark_gradient / Weber_BMC2015 #3078): fiddy's Consistency success checker only verifies that forward/backward/central methods agree with each other at a given step size ("self-consistent"), then blends every self-consistent size into the final value. A step size can become small enough that all methods sample points within the target function's floating-point noise floor and become spuriously self-consistent while biased away from the true derivative -- this hit parameter a32, which is several orders of magnitude smaller than Weber_BMC2015's other free parameters.
  • Adds RobustConsistency to amici.adapters.fiddy: a subclass of fiddy's Consistency that additionally rejects step sizes that are self-consistent but inconsistent with the majority of other step sizes, via iterative, order-independent, MAD-based outlier rejection.
  • Switches every remaining Consistency usage in AMICI to RobustConsistency, in both tests/benchmark_models/test_petab_benchmark.py and python/tests/adapters/test_fiddy.py -- there's no reason to keep using the plain checker anywhere now, since RobustConsistency only depends on fiddy's released Consistency base class.
  • Adds unit test coverage for the new class in python/tests/adapters/test_fiddy.py (12 tests, including a regression test reproducing the flaky-check mechanism from a minimal synthetic function).
  • Unskips test_benchmark_gradient's unscaled (linear-scale) parametrization for 5 benchmark models -- Elowitz_Nature2000, Borghans_BiophysChem1997, Sneyd_PNAS2002, Bertozzi_PNAS2020, Zheng_PNAS2012 -- that were previously skipped because their finite-difference gradient checks only failed on linear scale, the same failure mode RobustConsistency guards against (step sizes spuriously self-consistent but biased, more likely when parameter magnitudes vary widely on linear scale). Verified locally that all 5 now pass reliably (forward and adjoint). Brannmark_JBC2010 remains skipped there -- it still fails independently of this fix. test_nominal_parameters_llh_v2's skip list is left untouched.

Background

This was originally proposed upstream as ICB-DCM/fiddy#77. After review feedback (the fiddy maintainer preferred not to add a parameter-magnitude-calibrating statistic to a general-purpose library, and suggested per-parameter/relative-step-size tuning instead), that PR was closed in favor of implementing the fix directly in AMICI. This PR supersedes the earlier approach of pinning CI to an unmerged fiddy branch -- that pin has been reverted, since RobustConsistency now lives in AMICI and only depends on fiddy's released Consistency base class.

Test plan

  • pytest python/tests/adapters/test_fiddy.py -v -- 15/15 passed (3 pre-existing + 12 new RobustConsistency/_reject_outliers tests)
  • pytest tests/benchmark_models/test_petab_benchmark.py -k "Weber_BMC2015 and unscaled" -- previously-flaky cases now pass reliably
  • Spot-checked other benchmark models (Boehm_JProteomeRes2014, Fujita_SciSignal2010, both scales/sensitivity methods) for regressions -- all pass
  • Verified the 5 newly-unskipped models pass their unscaled gradient checks (forward and adjoint) locally
  • Benchmark Collection CPP GHA job passes on this branch

🤖 Generated with Claude Code

…#3078)

Pin fiddy to ICB-DCM/fiddy#77, which addresses the root cause of the
flaky test_benchmark_gradient[Weber_BMC2015-*-unscaled] failures: fiddy's
Consistency checker could silently accept a catastrophically-cancelled
finite-difference estimate for parameters with much smaller magnitude
than the rest of the model.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.08197% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.02%. Comparing base (3aebbe8) to head (dbfa0e6).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
python/sdist/amici/adapters/fiddy.py 95.08% 3 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3227      +/-   ##
==========================================
- Coverage   78.62%   78.02%   -0.60%     
==========================================
  Files         318      318              
  Lines       21102    21167      +65     
  Branches     1487     1487              
==========================================
- Hits        16591    16516      -75     
- Misses       4503     4643     +140     
  Partials        8        8              
Flag Coverage Δ
cpp 72.06% <95.08%> (+0.04%) ⬆️
cpp_python 36.45% <0.00%> (-0.12%) ⬇️
petab 48.13% <0.00%> (-0.17%) ⬇️
petab_sciml 16.22% <0.00%> (-0.05%) ⬇️
petab_sciml_benchmarks 14.80% <0.00%> (-0.04%) ⬇️
python 70.44% <95.08%> (+0.04%) ⬆️
sbmlsuite-jax ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
python/sdist/amici/adapters/fiddy.py 91.47% <95.08%> (+1.91%) ⬆️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dweindl and others added 2 commits August 10, 2026 19:32
…5 gradient check

fiddy's `Consistency` checker only verifies that forward/backward/central
methods agree with each other at a given step size, then blends every
self-consistent size into the final value. A step size can become small
enough that all methods sample points within the target function's
floating-point noise floor and become spuriously self-consistent while
biased away from the true derivative -- the confirmed root cause of the
intermittent test_benchmark_gradient[Weber_BMC2015-*-unscaled] failures
(AMICI-dev#3078), affecting parameter a32, which is orders of magnitude smaller
than the model's other free parameters.

This was originally proposed upstream as ICB-DCM/fiddy#77, which added a
`RobustConsistency` subclass performing iterative, order-independent,
MAD-based outlier rejection across self-consistent step sizes. After
review feedback, that PR was closed in favor of implementing it directly
in AMICI, since fiddy's maintainer preferred not to add a
parameter-magnitude-calibrating statistic to a general-purpose library.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ICB-DCM/fiddy#77 was closed after review feedback; the fix for the
flaky Weber_BMC2015 gradient check (AMICI-dev#3078) is now implemented directly
in AMICI (RobustConsistency in amici.adapters.fiddy) instead, so the
benchmark CI job no longer needs to depend on an unmerged fiddy branch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dweindl dweindl changed the title GHA: Test fiddy fix for flaky Weber_BMC2015 gradient check (#3078) Fix flaky Weber_BMC2015 gradient check with AMICI-native RobustConsistency Aug 10, 2026
dweindl and others added 2 commits August 10, 2026 20:22
…tConsistency

Elowitz_Nature2000, Borghans_BiophysChem1997, Sneyd_PNAS2002,
Bertozzi_PNAS2020, and Zheng_PNAS2012 were previously skipped for
test_benchmark_gradient's unscaled (linear-scale) parametrization because
their finite-difference gradient checks only failed on linear scale --
the same failure mode RobustConsistency now guards against (step sizes
that are spuriously self-consistent but biased, more likely when
parameter magnitudes vary widely on linear scale). Verified locally that
all 5 now pass reliably (forward and adjoint) with RobustConsistency.

Brannmark_JBC2010 remains skipped -- it still fails independently of
this fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tests

The two other call sites in test_fiddy.py (test_run_amici_simulation_to_functions,
test_simulate_petab_to_functions) were still constructing fiddy's plain
Consistency checker directly. Switch them to amici's own RobustConsistency
for consistency with the rest of the test suite -- it only depends on
fiddy's released Consistency base class, so there's no reason to keep
using the plain version anywhere in AMICI now.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dweindl
dweindl marked this pull request as ready for review August 10, 2026 19:04
@dweindl
dweindl requested a review from a team as a code owner August 10, 2026 19:04
@dweindl dweindl linked an issue Aug 10, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Check flaky test_benchmark_gradient / Weber_BMC2015

1 participant