Image-simulation m-bias: extracted core + snakemake orchestration

[cailmdaley]

Closes #224
Closes #226

Image-simulation m-bias: the sp_validation half

This is the sp_validation-side half of the image-simulation validation story (umbrella: UNIONS-WL/MultiBand_ImSim#1). It pairs with the ShapePipe-side module enablement in CosmoStat/shapepipe#766: that PR makes ShapePipe able to run on the SKiLLS sims; this one makes the whole chain runnable end-to-end and produces the shear-bias result, on sp_validation's native snakemake/container framework.

The split follows the dependency arrow — ShapePipe produces catalogues, sp_validation consumes/validates them — and puts the m-bias measurement where it belongs.

What's here

1. A self-contained m/c-bias core (src/sp_validation/image_sims.py + scripts/compute_m_bias_image_sims.py)

• ImageSimMBias reads the five calibrated grid catalogues (reference 1z2z + the ±-shear pairs 1p2z/1m2z and 1z2p/1z2m), RA/Dec-matches them, and returns m1/m2/c1/c2 with bootstrap errors. Only internal dependency: the new match_catalogs_radec in catalog.py.
• Paired "pool" estimator (4e38326): +g and −g catalogues are matched object-by-object to each other and differenced per object, so shape noise cancels before averaging; bootstrap resamples pairs. A guard test (test_mbias_pool_cancels_shape_noise) locks the cancellation — it fails on the naive unpaired estimator.
• config/calibration/mask_v1.X.{4,9}_im_sim.yaml — image-sim calibration configs.
• Synthetic recovery tests: inject known m/c, assert recovery.

2. Snakemake orchestration — the full DAG (init → pipeline → merge → extract → calibrate → m_bias) wiring both containers: ShapePipe SIF for pipeline + merge, sp_validation SIF for extract + calibrate + m-bias. Everything is parameterised through one config file (container paths, repo roots, data roots, PSF dict, tile list, sim/calibration knobs) — no hard-coded clone layout; this is the "runnable by someone other than Fabian" fix.

• im_init now stages the raw SKiLLS inputs (input_tiles/input_exp symlinks) so im_pipeline genuinely runs from raw simulated images, not from pre-staged intermediates (58756eb).
• The downstream (extract → calibrate → m-bias) runs on the migrated ngmix-v2 column grammar end-to-end (00ed2c4) — this flushed out a 6-defect cascade of old-grammar assumptions, all fixed on this branch.

Status: the chain runs end-to-end; the m-bias gate caught two real bugs

The full chain — raw SKiLLS images → ShapePipe → merge → extract → calibrate → m-bias — now runs on candide with a single consistent new-grammar vintage (both containers built from the current #766 / this branch). That was the point of the exercise, and the gate then did its job: the first trustworthy-conditions run returned m ≈ −1, and chasing that number down surfaced two genuine bugs in the shared calibration path, both now filed with full diagnosis:

• Additive-bias subtraction in calibration zeroes the signal on constant-shear sims (m = −1) #226 — the additive-bias subtraction zeroes the signal on constant-shear sims. get_calibrated_m_c subtracts each catalogue's own mean shear; on a constant-shear sim that mean is the injected signal, so ⟨e⟩ = 0 per sim by construction and m = −1 exactly. Correct for data, fatal for sims. A hand-diagnostic with multiplicative-only calibration recovers a sane m₁ = +0.14 ± 0.10, m₂ = +0.27 ± 0.11 (1 tile).
• Gatti/DES weights pathological on image sims: N_eff ~1–5% of N, inflates σ(m) ~12× #227 — compute_weights_gatti weights are pathological on the sims (N_eff ~1–5% of N), inflating σ(m) ~12× and skewing the central value. w_iv is healthy, isolating the problem to the Gatti weight construction.

Where this leaves the PR: the orchestration and estimator halves are done and verified; the trustworthy m-bias number itself is blocked on #226 (a design call on the shared calibration path — options laid out in the issue) and #227. Martin — the ball is in your court on those two: once the calibration route is chosen, re-running im_calibrate → im_mbias on the existing intermediates is cheap, and then we scale to the 5-grid ensemble.

— Fable, on behalf of Cail

[cailmdaley]

Status update (PR description rewritten to match): the chain now runs end-to-end from raw SKiLLS images on a single consistent ngmix-v2 grammar vintage, and the m-bias estimator is fixed to a paired, shape-noise-cancelling form with a guard test.

The gate then earned its keep: the first clean run returned m ≈ −1, and the diagnosis landed on two real bugs in the shared calibration path, now filed with full mechanism and evidence:

• Additive-bias subtraction in calibration zeroes the signal on constant-shear sims (m = −1) #226 — the additive-bias subtraction in get_calibrated_m_c zeroes the signal on constant-shear sims (correct for data, fatal for sims; m = −1 exactly, by construction). Three fix options laid out in the issue — this one's a design call on shared calibration code, so it's yours to choose.
• Gatti/DES weights pathological on image sims: N_eff ~1–5% of N, inflates σ(m) ~12× #227 — compute_weights_gatti weights are pathological on the sims (N_eff ~1–5% of N), inflating σ(m) ~12×.

A multiplicative-only hand-diagnostic gives m₁ = +0.14 ± 0.10, m₂ = +0.27 ± 0.11 (grid_1, 1 tile), so the pipeline itself looks healthy. Once the calibration route is chosen, re-running im_calibrate → im_mbias on the existing intermediates is cheap, and we scale to the 5-grid ensemble for the real number. Ball's in your court, Martin.

— Fable, on behalf of Cail

[cailmdaley]

Three commits pushed to this branch, all exercised end-to-end on tile 233.293 (grid_1):

• additive_correction mirrored into this branch's layout (64ec1d7). Martin — your Additive-bias subtraction in calibration zeroes the signal on constant-shear sims (m = −1) #226 fix (e38ace9) sits on image_sims_val, which still has calibration under notebooks/; this branch moved it to scripts/calibration/, so the keyword wasn't reaching the workflow here. Same semantics as yours: default True leaves the data path unchanged; the calibrate script reads the flag from the mask config's metacal: section; both *_im_sim.yaml configs set False.
• w_col: null → unit weights in ImageSimMBias (d0984f2), wiring the Gatti/DES weights pathological on image sims: N_eff ~1–5% of N, inflates σ(m) ~12× #227 verdict (no shape weights for sim m-bias) through the workflow config.
• global_R_weight: null in both im_sim mask configs (04464e1) — the second, less visible place weights entered: the calibration's global response was w_des-weighted, and on this tile that R is measured on N_eff ≈ 21 objects and differs from the unweighted mean by 17% (0.780 vs 0.669). Gatti/DES weights pathological on image sims: N_eff ~1–5% of N, inflates σ(m) ~12× #227 applies here too.

With all three active, plus shapepipe#803 (position-seeded fixnoise, now also on the im_sims branch with SEED_FROM_POSITION = True in the sims ngmix config), the chain runs green from raw images and gives, on one tile:

m1 = +0.274 ± 0.028,  m2 = +0.231 ± 0.022,  c1 = −0.003 ± 0.008,  c2 = −0.003 ± 0.008

The σ_m is ~5–6× smaller per tile than without the noise pairing (measured same-tile A/B; details on shapepipe#803). Percent-level σ_m now needs ~4–7 tiles.

Note the central value: the m ≈ +0.2 is now ~8–10σ from zero on a single tile. It is not estimator noise. The leading suspect is metacal's measured response: the true sample response (Δe across paired branches / 2γ) is ≈0.80 while metacal's R̄ ≈ 0.64–0.67 — a ~20% under-response. The in-memory twin passes at |m|<5e-3 with the same metacal code and an exact PSF, and its measured PSF-size-error coefficient (m ≈ +1.8·δ_size) would put a +0.2 bias at ≈ +11% PSF size mismatch — so fake_psf input fidelity is where we'll look first, then selection response.

One reconciliation heads-up for the eventual merge: image_sims_val carries Martin's parallel m-bias/diagnostics work (ddb95f0, 003058c) that this branch's scripts/ versions overlap with — worth deciding which surface is canonical before merging both.

— Claude (Fable) on behalf of Cail

[github-actions]

🔴 ruff found lint / format issues

@cailmdaley — these block the merge into develop. Full list below (also surfaced as annotations in the CI run):

ruff check .

scripts/diagnostics_image_sims.py:16:1: I001 [*] Import block is un-sorted or un-formatted
scripts/diagnostics_image_sims.py:22:1: I001 [*] Import block is un-sorted or un-formatted
Found 2 errors.
[*] 2 fixable with the `--fix` option.

ruff format --check .

Would reformat: scripts/diagnostics_image_sims.py
1 file would be reformatted, 230 files already formatted

---

CI run · Updates on every push and turns green when ruff passes — nothing else to do.