Skip to content

Add calibrated global min-PTE threshold for many null tests - #370

Merged
cailmdaley merged 1 commit into
developfrom
feat/calibrated-min-pte
Oct 1, 2026
Merged

cailmdaley merged 1 commit into
developfrom
feat/calibrated-min-pte

Conversation

@cailmdaley

Copy link
Copy Markdown
Collaborator

Problem. A null test that runs many correlated statistics (E/B estimators, bin pairs, scale cuts) cannot use a fixed per-test PTE cut. The global false-positive rate is unknown because it depends on how the statistics correlate and how well each PTE is calibrated.

Change. Adds calibrate_min_pte to sp_validation.statistics. It takes an (n_mocks, n_stats) PTE matrix from noise-only realisations and returns:

  • the α-quantile of the per-mock minimum PTE, which is the global threshold, with an order-statistic interval that makes no distributional assumption;
  • the effective number of independent tests k_eff, from 1 − (1 − t)^k = α;
  • .global_pte(data_ptes): the fraction of mocks whose minimum PTE is ≤ the data's, with a Wilson interval.

It works with any statistic and any number of tomographic bins. An optional two-sided mode uses 2 min(p, 1−p). It needs only numpy and scipy.

Verification. Five new tests in src/sp_validation/tests/test_statistics.py:

  • independent uniform PTEs recover the Šidák threshold and k_eff = k;
  • the same holds for two-sided PTEs;
  • perfectly correlated statistics give k_eff = 1;
  • the global p-value matches a hand-worked example;
  • the global p-value is uniform under a correlated null.

🤖 Generated with Claude Code

calibrate_min_pte takes a (n_mocks, n_stats) PTE matrix from noise-only
realisations and returns the alpha-quantile of the per-mock minimum PTE
(with a distribution-free order-statistic interval), the implied effective
number of independent tests k_eff from 1 - (1 - t)^k = alpha, and a
global_pte method giving the data's global p-value with a Wilson interval.
Statistic-agnostic: columns can be any statistics, bin pairs or scale cuts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley
cailmdaley merged commit df4f250 into develop Oct 1, 2026
4 checks passed
@cailmdaley
cailmdaley deleted the feat/calibrated-min-pte branch October 1, 2026 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant