Seed the jackknife k-means and drop the patch-centre file - #367
Open
cailmdaley wants to merge 5 commits into
Open
cailmdaley wants to merge 5 commits into
cailmdaley wants to merge 5 commits into
Conversation
…tre file
TreeCorr seeds its k-means afresh on every unseeded call, so the jackknife
patches, and with them the ξ± jackknife covariance, differed between runs
that did not share a {ver}_patches_npatch=N.dat. The k-means is now seeded
with a fixed generator, which makes the patches a pure function of the
catalogue's positions and weights on any thread count; the centre file and
its reuse logic go. The aperture-mass ξ± takes the same seed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UzeJqdtGgoeWizre32L5fD
statistics.jackknife_patch_centers runs the seeded k-means on an NField with min_top=6, so the initial centres no longer depend on the OpenMP thread count TreeCorr reads when it sizes the tree. calculate_2pcf and calculate_aperture_mass_dispersion build their shear catalogue through RealSpaceMixin._shear_catalog, which passes those centres as patch_centers; rho_tau.get_jackknife_cov takes its per-patch centres from the same helper. The reproducibility test now runs 100 patches and simulates 4- and 16-CPU machines, and asserts identical patch labels, per-patch-pair counts, xi+- and jackknife variance. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UzeJqdtGgoeWizre32L5fD
…tch-centres Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UzeJqdtGgoeWizre32L5fD
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cailmdaley
added a commit
that referenced
this pull request
Oct 2, 2026
…r go blinding.py now holds the theory (shear, fiducial, cosmology, and no_signal for statistics without cosmological signal) and blind_of, which reads an already-resolved catalogue entry's `blind:` and refuses a missing one or one shear file read under two blinds. Its scientific imports live inside the functions, so workflow/common.py loads it by path on the host. custody.py and theory.py are deleted. Catalogue and seed resolution are develop's again (_materialize_seed_path, catalogue_entry, get_shear_catalog); CosmologyValidation.blind and common.blind_of call blinding.blind_of on the resolved entry. The per-launch [blind] banner is gone; configure() still refuses a run catalogue that declares no blind before any job is built. Patch centres are written as on develop (PR #367 owns them). Tests keep what guards blinding: ξ±/Cℓ_EE shift and BB/EB do not; pure-E/B under a blind moves E and not B (develop's operator); seal refuses a double stamp; one file under two blinds is refused; a flip reruns the part; an undeclared catalogue stops the launch. _synthetic.py, test_blinding_bmodes.py, test_custody.py and test_toy_run.py go. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Wow indeed, Claude will stop at nothing to vanquish irreproducibilities... yes pausing this until we figure out the API conflicts |
This was referenced Oct 2, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
calculate_2pcfbuilds the jackknife patches for ξ± with TreeCorr's k-means, which picks a fresh random seed on every unseeded call. Develop hides this by drawing the centres once per output tree and reusing{ver}_patches_npatch=N.dat, but a fresh tree, a rerun elsewhere, or two jobs racing to write the file all split the sky differently, so the ξ± jackknife covariance changed from run to run. It is the same class of bug that moved ρ/τ statistics by up to ~3σ between reruns (fixed in shear_psf_leakage #43).A seed alone is not enough: TreeCorr's automatic k-means starts from a ball tree whose depth follows the machine's OpenMP thread count, so for non-power-of-two patch counts (e.g. 100) a seeded
Catalog(npatch=..., rng=...)still splits the sky differently on 12-, 24- and 48-CPU nodes. This PR takes the centres from a seeded k-means on a tree of pinned depth (statistics.jackknife_patch_centers:min_top=6), the same fix as shear_psf_leakage #47. The patches are then a pure function of the catalogue's positions and weights, so the centre file has no job left and goes too.What changes
calculate_2pcfandcalculate_aperture_mass_dispersionbuild their catalogue through one helper that passes the pinned-depth centres aspatch_centers; the patch-centre file, its reuse and its write are removed.rho_tau.pytakes its centres from the same helper.Effect on results
bin_slopapproximates; not measured against develop's draws._patches_npatch=*.datfiles are no longer read. They can be deleted.Cost
The k-means runs on every measurement. On 16 threads it takes about 1 s per 10⁶ objects (≈10–15 s at 10⁷ for 20–200 patches), which is small next to the pair counting.
Verified
-m "not slow"), with develop merged in: 307 passed, 1 xfailed.— Claude (Opus) on behalf of Cail.
🤖 Generated with Claude Code