ngmix: position-seeded per-object RNG for Pujol noise cancellation (#796)

[cailmdaley]

Summary

Adds an opt-in SEED_FROM_POSITION knob (default False) to the ngmix module. When enabled, the per-object RNG — and the prior it feeds — are seeded from the object's sky position instead of once per tile, so metacal's fixnoise counter-noise and the fit's initial guesses are identical for the same object across Pujol image-simulation shear branches and cancel in the branch difference, shrinking σ(m). For image simulations only; no benefit on real data, so off by default.

Design

• Seed = position_seed(ra, dec, ccd). Box math is Seed fix based on position for Pujol noise cancellation for image simulations #796's exactly (floor(ra*3600/3)+(ccd+1), floor(dec*3600/3)+(ccd+2), first-epoch coordinates); the 3-arcsec boxes make the seed robust to detection-order and sub-box centroid changes across branches.
• Deviation from Seed fix based on position for Pujol noise cancellation for image simulations #796: the issue combines the boxes as box_x + box_y, which collides along anti-diagonals (e.g. boxes (10,20) and (11,19) both → 30, giving two distinct objects the same noise stream). Replaced with a Cantor pairing (bijection on non-negative integers) after a zig-zag fold for negative Dec, reduced mod 2^32 into the valid RandomState range. Box math untouched.
• The prior is rebuilt from the per-object RNG, not just the runner RNG — this turned out to be load-bearing: the guesser's initial guess comes from prior.sample(), which draws from the RNG the prior was constructed with (ngmix 2.4.0 guessers.py/priors.py). A per-object RNG alone leaves the guess on the shared tile stream and silently breaks the cancellation. Anyone wiring Seed fix based on position for Pujol noise cancellation for image simulations #796 by hand would hit this without an error.

Verification (in-container, dev image)

• Determinism: same object, knob ON, under two different tile-RNG contexts + a processing-order perturbation → metacal g byte-identical for all five types (noshear/1p/1m/2p/2m).
• Sub-box jitter: ±1″ centroid jitter within the 3″ box → same seed, byte-identical fit.
• Regression guard: knob OFF is byte-identical to unmodified develop for all five metacal types on a fixed tile seed.
• Seed properties: hypothesis tests for range validity, jitter invariance, adjacent-box distinctness, and the resolved anti-diagonal collision; 200k-sample sky sweep, zero real collisions.
• Full fast suite: 326 passed.

Coordination notes

• Expects to rebase after Image simulations workflow #766 — that PR touches ngmix.py/ngmix_runner.py too and should land first; this diff is small and localized, the rebase is ours to do.
• The knob is documented in example/cfis/config_tile_Ng_template.ini; the image-sims configs (example/cfis_image_sims/, on the Image simulations workflow #766 branch) should set SEED_FROM_POSITION = True once both are in — happy to add that there after Image simulations workflow #766 merges.

Design and box math: @fabianhervaspeters (#796).

— Claude Fable on behalf of Cail

🤖 Generated with Claude Code

https://claude.ai/code/session_01J9qEVRCWEekT5ACNB3rjkq

[cailmdaley]

Empirical validation at survey scale: same-tile A/B on the SKiLLS sims (tile 233.293, grid_1).

Three measurements on existing data first, then an A/B of this branch:

1. The sim branches share their sky-noise realization — background pixel correlation between shear branches is 0.9996–0.99999 at both tile and exposure level (all 5 branches carry identical 291-exposure lists; independent noise would give diff-std/img-std ≈ √2, we measure ~1e-4). The Seed fix based on position for Pujol noise cancellation for image simulations #796 design premise holds by construction.

2. The current tile-seeded fixnoise is the one thing breaking cross-branch cancellation. Matching objects across ±γ branches in the existing cut catalogues: corr(e⁺,e⁻)=0.81 — but the control against the γ=0 branch gives the same 0.82, i.e. that correlation is entirely shared intrinsic shape; the noise is uncorrelated across branches. With no weights (PLplot Removed - [merged] #227) the paired estimator gives per-tile m1=+0.13±0.12, m2=+0.29±0.12 — the entire σ_m budget is measurement noise this PR targets.

3. The in-memory twin measures the full-pairing ceiling: ~11× in σ_m at SKiLLS-like S/N (from the v2 breakdown grid's per-seed records: same-seed ±γ arms vs independently-resampled arms).

The A/B (identical chain end-to-end; only SEED_FROM_POSITION = True differs; this PR cherry-picked onto im_sims — conflict-free, 21/21 ngmix module tests pass; engagement confirmed in the module log):

	tile-seeded (current)	position-seeded (this PR)
corr(e⁺,e⁻)	0.811 / 0.819	0.9930 / 0.9954
per-tile σ_m1, σ_m2	0.119, 0.122	0.0247, 0.0190

Measured gain: 4.8× (m1) / 6.4× (m2) in σ_m per tile (Var(Δe) down 26–41×; implied cross-branch noise correlation ρ ≈ 0.96–0.98). Matched-pair count even rose (1560→1645) — shared noise makes marginal detections consistent across branches.

The chain's own m-bias output for the position-seeded arm — the first end-to-end number with every repair active (unit object weights AND unweighted global response per sp_validation#227, additive_correction: False per #226, noise-paired):

m1 = +0.274 ± 0.028,  m2 = +0.231 ± 0.022,  c1 = −0.003 ± 0.008,  c2 = −0.003 ± 0.008   (1 tile)

Sizing consequence: percent-level σ_m needs ~4–7 tiles with this PR (vs ~140 without); 15 tiles → σ_m ≈ 0.005–0.006.

And a science consequence: the m ≈ +0.25 offset is now ~10σ from zero on a single tile, isotropic (m1≈m2), with clean additive terms — it is not estimator noise. The direct statement of the anomaly: the true sample response measured across paired branches is Δe/2γ ≈ 0.80, while metacal's finite-difference R̄ ≈ 0.64 — metacal under-measures the response by ~20% on these sims. The in-memory twin passes at |m|<5e-3 with the same metacal code and an exact PSF, and its measured PSF-size-error coefficient (m ≈ +1.8·δ_size) would place a +0.25 bias at ≈ +14% PSF size mismatch — so fake_psf input fidelity is the leading suspect, with selection response secondary. Mechanism hunt is the next phase; this PR is what makes it statistically affordable.

Where the code landed: this branch is now also on im_sims (with SEED_FROM_POSITION = True in the sims ngmix config, alongside the #802 crash fix), so the sims campaign runs with it while the develop merge waits for the complete picture.

— Claude (Fable) on behalf of Cail