Skip to content

Breakdown grid v2: paired six-arm estimator, honest bootstrap errors, degeneracy guards#804

Open
cailmdaley wants to merge 6 commits into
developfrom
feat/breakdown-rigor
Open

Breakdown grid v2: paired six-arm estimator, honest bootstrap errors, degeneracy guards#804
cailmdaley wants to merge 6 commits into
developfrom
feat/breakdown-rigor

Conversation

@cailmdaley

@cailmdaley cailmdaley commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Replaces the v1 breakdown-grid harness (16 unpaired seeds/cell, no uncertainties) with a statistically sound v2:

  • Paired ±γ arms on identical noise realizations — six arms per seed (g1±, g2±, psf±) sharing the RNG stream, so measurement noise cancels in the branch difference (same statistical move as Seed fix based on position for Pujol noise cancellation for image simulations #796 at survey scale).
  • Joint seed-resample bootstrap on every reported quantity — one index array per replicate feeds numerators, R denominators, and the legacy estimator, so the pairing survives the resampling.
  • Degeneracy guard: cells where |R̄| < 5·se(R̄) report null + degenerate: true instead of dividing (the v1 grid printed m2 = −58.6 at the point-source/low-S/N corner from exactly this division).
  • Independent seeds per cell (v1 recycled one seed set across all 24 cells, correlating the apparent "trends"), per-band seed counts {high 100, moderate 400, low 1600}, per-cell npz of raw per-(seed,arm) records so reanalysis never refits, --true-noise ablation knob (routes prepare_ngmix_weights onto the per-pixel true-inverse-variance path via stamp.bkg_rms).
  • Plotter: error bars on every point, degenerate cells as open markers, ±5e-3 tolerance band.

Why it matters

The v1 panel showed alarming S/N-dependent m (up to +0.31 at S/N≈50) and c trends. The v2 grid (24 cells, ~100k metacal fits, SLURM) shows they were statistical artifacts of the v1 harness: every resolved cell in every band is consistent with m = 0 (precision ~1e-3 high band, ~1e-2 moderate, ~1.5e-2 low), |c1| < 1e-3 and leakage α ≤ 0.2% everywhere, and R11 ≈ R22 on resolved rungs. The live do_ngmix_metacal path is healthy at survey-relevant S/N; the earlier panel was measuring its own noise floor.

Test plan

  • Pairing identity: noise draw + sub-pixel shift bit-identical across all six arms of a seed
  • Bootstrap validated against direct per-seed error decomposition (numerator 0.0087 ⊕ R̄-noise 0.0264, corr +0.42 → observed 0.0243 at the 80-seed pilot cell)
  • Degeneracy guard fires at (r=0.15, low) instead of printing an exploded number
  • Full 24-cell production run completed (job 808255); results in the twin results tree
  • No src/ changes — harness-side only (scripts/python/)

— Claude Fable on behalf of Cail

🤖 Generated with Claude Code

https://claude.ai/code/session_01J9qEVRCWEekT5ACNB3rjkq

cailmdaley and others added 2 commits July 10, 2026 01:53
@cailmdaley @claude
breakdown-grid v2: paired six-arm estimator + honest error bars
f369b5c 
Rebuild the resolution x noise breakdown harness for statistical rigor. v1
reported bare m = <g>/<R>/gamma - 1 with no uncertainty; at moderate/low S/N
the per-seed shear noise gives sigma_m ~ 0.5, so its S/N trends were unmeasured.

Producer (run_breakdown_grid.py):
- Six signed arms per seed -- (g1+,g1-), (g2+,g2-), (psf+,psf-) -- sharing a
  bit-identical noise realisation and sub-pixel shift (same RandomState into
  make_data; galsim .shear() draws no rng). The +/- difference cancels the
  shared shape noise that dominated v1.
- Paired m1/m2 = (<g[+]> - <g[-]>)/(2 gamma Rbar) - 1; additive c1/c2 and
  leakage alpha from the PSF-shear pair; legacy ratio-of-averages kept for v1
  continuity. Bootstrap-over-seeds errors (B=1000) on every estimator.
- Degeneracy guard: |Rbar| < 5 se(Rbar) -> m null with degenerate:true rather
  than dividing by a response consistent with zero.
- Seeds independent per cell (10_000_000 + cell_index*100_000 + s), never
  recycled. Per-band seed counts default {high:100, moderate:400, low:1600}.
- --true-noise ablation: per-pixel true-inverse-variance weight path via
  stamp.bkg_rms. Diagnostics: mean_s2n, mean_T_ratio, sigma_mad_est/noise_true
  contamination ratio.
- ProcessPoolExecutor over cells (--jobs); per-cell .npz of raw per-(seed,arm)
  arrays for refit-free re-analysis; provenance (ngmix/galsim/git/wall time).
- Emits breakdown_grid_v2.json and mirrors breakdown_grid.json so the status
  page reader (build_status.py loads that name) stays green unchanged.

Plotter (plot_breakdown_grid.py):
- m1/m2/|c1| vs resolution, hue=band, error bars on every point, degenerate
  cells as open hatched markers at 0, +/-5e-3 tolerance band. Same PNG name.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J9qEVRCWEekT5ACNB3rjkq
@cailmdaley @claude
breakdown-grid v2: joint seed-resample bootstrap — pairing must survi…
a59bed9 
…ve the resampling

The paired estimators' bootstrap resampled each arm independently, which
destroys the +/-gamma pairing inside every replicate and inflates the
reported error to the unpaired level (read 0.23 at a moderate cell where
the per-seed pairs give a 0.009 numerator term). One index array per
replicate now feeds every quantity: numerator pairs, R denominators
(replacing the analytic se(R) that treated the two arms' correlated R
values as independent), and the legacy ratio. Verified on the r=0.37
moderate cell: m1 = -0.0005 +/- 0.0243, decomposing as 0.0087 (paired
numerator) + 0.0264 (R-bar noise) with corr(dg, R_i) = +0.42 — the
honest error is response-noise dominated at moderate S/N.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J9qEVRCWEekT5ACNB3rjkq
@cailmdaley cailmdaley mentioned this pull request Jul 10, 2026
Open
cailmdaley and others added 4 commits July 10, 2026 02:51
@cailmdaley @claude
breakdown-grid: S4 production-parity knobs (wcs centroid, WCS jacobia…
fb745d1 
…n, epochs)

Three CLI knobs turn the clean-room v2 grid into production-settings
ablations, each defaulting to the bit-identical clean baseline:

- --centroid-source {hsm,wcs}: "wcs" is the production default since
  31ae736. The harness fabricates the astrometry truth the path consumes:
  a TAN WCS whose CD matrix IS the drawing jacobian, ra/dec evaluated
  through that same WCS at the object's true pixel position (the toy
  analogue of coadd (x,y) -> (ra,dec) -> epoch pixel).
- --wcs-g1/--wcs-g2/--wcs-theta-deg: shear/rotate the drawing WCS away
  from a pure pixel scale (the esheldon/ngmix#72 sensitivity axis).
- --n-epochs (pre-existing) completes the parity set.

make_data grows wcs= and return_centers= kwargs, both default-off and
backward-compatible. Under a non-trivial WCS the sub-pixel shift is drawn
in PIXEL space (uniform +-0.5/axis): a world-space +-scale/2 box maps
through a sheared jacobian to a parallelogram exceeding +-0.5 pixel,
which silently breaks round-to-nearest-pixel centroid logic with
off-by-one-pixel centres. The legacy path keeps the exact legacy draw
lines: smoke-verified bit-identical (78/78 arrays) against the v2
production baseline npz.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VjLSiytceGpbMfzZe6wqGa
@cailmdaley @claude
breakdown-grid: guard wcs-centroid mode to odd img_size (review F1)
0b2b576 
Even stamps put the galsim centre on a half-integer; np.round() in the
ngmix "wcs" centroid path then lands one pixel off for every object -- a
systematic half-pixel origin error that would masquerade as a production
pathology. Production VIGNETs are odd; assert it. Also soften the
"exact by construction" round-trip claim (TAN residual ~5e-9 px).

Fresh-eyes review of fb745d1: otherwise clean -- coordinate chain,
rng-pairing contract, matrix orientation, pickle boundary, and all 14
make_data call sites verified.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VjLSiytceGpbMfzZe6wqGa
@cailmdaley @claude
breakdown-grid: survive per-fit bootstrap failures as flagged records
35da920 
One BootPSFFailure raise out of ~1e5 fits killed the whole jac-realistic
ablation grid (job 808259). Production treats bootstrap failures as
flagged objects, so the harness now does the same: record flags=-1 /
finite=False, let the pair-drop policy drop the seed, and let fail_frac
record the rate -- a systematic failure mode still surfaces loudly there
instead of fatally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VjLSiytceGpbMfzZe6wqGa
@cailmdaley @claude
breakdown-grid: S4 ablation comparison figure + paired delta-m analysis
1fb3924 
plot_s4_ablations.py overlays ablation grids on the baseline (3 bands x
m1/m2/c1, tolerance bands, dodged error bars) and prints per-arm
max|m|/max|delta| significance tables. --paired recomputes delta-m1 per
cell from the raw npz records with a shared bootstrap resample for
bit-identical-image arms (the wcs-centroid ablation), cancelling shape
noise to ~1e-6-level errors at high S/N.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VjLSiytceGpbMfzZe6wqGa
@cailmdaley

cailmdaley commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Update: S4 production-parity ablations — the v2 PASS survives production settings.

Three new parity knobs on the harness (--centroid-source hsm|wcs, --wcs-g1/--wcs-g2/--wcs-theta-deg, existing --n-epochs), five full-grid ablation arms with the same seeds as the baseline (jobs 808257–808262):

arm result
wcs centroid (production default since 31ae736c) paired per-seed Δm₁ vs hsm: few ×1e-6 at S/N≈1000; |Δm₁| < ~1e-3 at every resolved rung down to S/N≈15 — the centroid source is not an m-lever (relevant to #767)
WCS jacobian shear g1=0.1 (the esheldon/ngmix#72 config that produced m≈−0.2) no effect: |Δm| < 1.5e-3 (1σ, high band) — the ngmix#72 pathology is absent from our stack, at ~5× any realistic MegaCam distortion
realistic jacobian (g1=0.02, g2=0.01, θ=10°) no effect (max |Δ|/σ = 0.57)
n_epochs=2 the old single-seed CI ladder's −2e-3 does not reappear: every resolved high-band cell within ~1e-3 of zero
production combo (wcs centroid + realistic jacobian + 2 epochs + true-noise weights) all cells within tolerance; max resolved high-band |m₁| = 1.0e-3 ± 0.5e-3

No |Δ|/σ ≥ 2 anywhere across ~300 arm×band×cell comparisons (the 1.7–1.9σ cluster in the two 2-epoch arms traces to a shared baseline fluctuation — both arms sit closer to zero than the baseline at that cell). Comparison figure + per-arm significance tables: plot_s4_ablations.py, with a --paired mode that cancels shape noise for bit-identical-image arms (that's what resolves the centroid effect at 1e-6).

Two notes on the diff since the original description: the test-plan line "no src/ changes" predates these commits — simulate.py gained two backward-compatible kwargs (all 14 make_data call sites verified unaffected; legacy path smoke-verified bit-identical against the baseline npz). And a robustness fix: a single BootPSFFailure in ~1e5 fits used to kill the whole grid; it's now recorded as a flagged object (production behaviour), with the rate surfaced in fail_frac.

— Claude Fable on behalf of Cail

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@cailmdaley