Skip to content

Seed the jackknife k-means and drop the patch-centre file - #367

Open
cailmdaley wants to merge 5 commits into
developfrom
feat/jackknife-patch-centres
Open

cailmdaley wants to merge 5 commits into
developfrom
feat/jackknife-patch-centres

Conversation

@cailmdaley

@cailmdaley cailmdaley commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

calculate_2pcf builds the jackknife patches for ξ± with TreeCorr's k-means, which picks a fresh random seed on every unseeded call. Develop hides this by drawing the centres once per output tree and reusing {ver}_patches_npatch=N.dat, but a fresh tree, a rerun elsewhere, or two jobs racing to write the file all split the sky differently, so the ξ± jackknife covariance changed from run to run. It is the same class of bug that moved ρ/τ statistics by up to ~3σ between reruns (fixed in shear_psf_leakage #43).

A seed alone is not enough: TreeCorr's automatic k-means starts from a ball tree whose depth follows the machine's OpenMP thread count, so for non-power-of-two patch counts (e.g. 100) a seeded Catalog(npatch=..., rng=...) still splits the sky differently on 12-, 24- and 48-CPU nodes. This PR takes the centres from a seeded k-means on a tree of pinned depth (statistics.jackknife_patch_centers: min_top=6), the same fix as shear_psf_leakage #47. The patches are then a pure function of the catalogue's positions and weights, so the centre file has no job left and goes too.

What changes

  • calculate_2pcf and calculate_aperture_mass_dispersion build their catalogue through one helper that passes the pinned-depth centres as patch_centers; the patch-centre file, its reuse and its write are removed.
  • The per-patch ρ/τ covariance path in rho_tau.py takes its centres from the same helper.
  • The thread-count test becomes a reproducibility test: two fresh output trees with 100 patches on simulated 4- and 16-CPU machines give identical patch labels, per-patch-pair counts, ξ± and jackknife variance.

Effect on results

  • ξ± values: essentially unchanged. Patches enter the covariance, and otherwise only decide which cell pairs TreeCorr's bin_slop approximates; not measured against develop's draws.
  • ξ± jackknife covariance: reproducible across trees, reruns and thread counts. It differs from any earlier unseeded draw.
  • Existing _patches_npatch=*.dat files are no longer read. They can be deleted.
  • A variant that reweights the catalogue gets its own patches. One that keeps positions and weights shares them.

Cost

The k-means runs on every measurement. On 16 threads it takes about 1 s per 10⁶ objects (≈10–15 s at 10⁷ for 20–200 patches), which is small next to the pair counting.

Verified

  • Reproducibility test: fails on the seed-only commit (2f70c93; 59% of objects change patch), passes on the pinned-depth helper.
  • Container suite (-m "not slow"), with develop merged in: 307 passed, 1 xfailed.

— Claude (Opus) on behalf of Cail.

🤖 Generated with Claude Code

cailmdaley and others added 2 commits September 29, 2026 16:33
…tre file

TreeCorr seeds its k-means afresh on every unseeded call, so the jackknife
patches, and with them the ξ± jackknife covariance, differed between runs
that did not share a {ver}_patches_npatch=N.dat. The k-means is now seeded
with a fixed generator, which makes the patches a pure function of the
catalogue's positions and weights on any thread count; the centre file and
its reuse logic go. The aperture-mass ξ± takes the same seed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cailmdaley and others added 3 commits October 1, 2026 05:27
statistics.jackknife_patch_centers runs the seeded k-means on an NField with
min_top=6, so the initial centres no longer depend on the OpenMP thread count
TreeCorr reads when it sizes the tree. calculate_2pcf and
calculate_aperture_mass_dispersion build their shear catalogue through
RealSpaceMixin._shear_catalog, which passes those centres as patch_centers;
rho_tau.get_jackknife_cov takes its per-patch centres from the same helper.

The reproducibility test now runs 100 patches and simulates 4- and 16-CPU
machines, and asserts identical patch labels, per-patch-pair counts, xi+- and
jackknife variance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UzeJqdtGgoeWizre32L5fD
…tch-centres

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UzeJqdtGgoeWizre32L5fD
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cailmdaley added a commit that referenced this pull request Oct 2, 2026
…r go

blinding.py now holds the theory (shear, fiducial, cosmology, and
no_signal for statistics without cosmological signal) and blind_of, which
reads an already-resolved catalogue entry's `blind:` and refuses a missing
one or one shear file read under two blinds. Its scientific imports live
inside the functions, so workflow/common.py loads it by path on the host.
custody.py and theory.py are deleted.

Catalogue and seed resolution are develop's again (_materialize_seed_path,
catalogue_entry, get_shear_catalog); CosmologyValidation.blind and
common.blind_of call blinding.blind_of on the resolved entry. The per-launch
[blind] banner is gone; configure() still refuses a run catalogue that
declares no blind before any job is built. Patch centres are written as on
develop (PR #367 owns them).

Tests keep what guards blinding: ξ±/Cℓ_EE shift and BB/EB do not; pure-E/B
under a blind moves E and not B (develop's operator); seal refuses a
double stamp; one file under two blinds is refused; a flip reruns the
part; an undeclared catalogue stops the launch. _synthetic.py,
test_blinding_bmodes.py, test_custody.py and test_toy_run.py go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@sachaguer sachaguer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow a subtle thing found here. I validate the strategy and jackknife seeding but hint again at the conflict between the parallel PRs that we have to figure first I believe (see PR #366)

@cailmdaley

Copy link
Copy Markdown
Collaborator Author

Wow indeed, Claude will stop at nothing to vanquish irreproducibilities... yes pausing this until we figure out the API conflicts

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants