cosmo_val + Snakemake migration to the SACC writers (PRD #241 row 4)#251
Draft
cailmdaley wants to merge 19 commits into
Draft
cosmo_val + Snakemake migration to the SACC writers (PRD #241 row 4)#251cailmdaley wants to merge 19 commits into
cailmdaley wants to merge 19 commits into
Conversation
Add sacc_writers.py: pure per-statistic *_to_sacc functions (xi coarse/fine,
pseudo-Cl, COSEBIs, pure-E/B, rho/tau) that turn computed arrays into
single-statistic SACC "parts", plus assemble_analysis_sacc, which rebuilds the
single {version}.sacc analysis file with a FullCovariance assembled block-
diagonally in canonical order (per the SACC layout contract — not
concatenate_data_sets, whose BlockDiagonalCovariance the contract rules out).
Add bandpower_window_from_workspace to pseudo_cl.py: the NEW plumbing that
threads NaMaster's get_bandpower_windows() into sacc_io.add_pseudo_cl (the
EE/BB/EB diagonal window is verified identical).
Tests: 12 fast tests, each writer round-tripped and checked against the
contract; a real small-nside NaMaster window round-trip; analysis assembly
proven to cover every point with aligned blocks and zero cross-blocks, incl.
the reload-from-disk DAG path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The theoretical tau covariance is not (6·nbin)² over [τ+; τ−]. Read from the
write-side (shear_psf_leakage.rho_tau_cov.CovTauTh.build_cov): it is a
(3·nbin)² k-major matrix over {τ0, τ2, τ5} with plus/minus folded into one
component per index — exactly the flavor today's CosmoSIS chain consumes via
covdat_to_fits. rho_tau_to_sacc now scatters that plus-only block into the
τ-plus rows/columns of the 6·nbin τ block (per-k [+;−] insertion order),
leaves τ-minus a vartau diagonal, and keeps plus↔minus cross zero. The kwarg
is renamed tau_cov -> tau_cov_th to name the flavor. tau_cov_th=None keeps a
compact DiagonalCovariance placeholder.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…CC part
calculate_pseudo_cl_map/_catalog already call pseudo_cl_to_sacc_part (from the
prior commit); this defines it — building the EE/BB/EB part via
pseudo_cl_to_sacc(nz, meta, ell_eff, cl_all, wsp) and saving it as the native
pseudo_cl_{ver}.sacc product. Drops the legacy save_pseudo_cl FITS writer (no
external callers) and the unused _NMT_* constants.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tics mixins
Thin *_to_sacc_part methods that turn each already-computed diagnostic into its
single-statistic SACC part via the born-as-SACC writers:
- cosebis.py: cosebis_to_sacc_part picks the fiducial scale cut's {En,Bn,cov}
(find_conservative_scale_cut_key, else the widest cut — mirroring plot_cosebis)
and writes it; the multi-cut .npz sidecar (the PTE scan) is untouched.
- pure_eb.py: pure_eb_to_sacc_part writes the six PURE_KEYS blocks at
gg.meanr with the results['cov'] block.
- psf_systematics.py: calculate_rho_tau_stats now writes a rho_tau_{base}.sacc
part per version from the handler tables, passing cov_tau_{base}_th.npy as
tau_cov_th when present (the tau-plus CovTauTh inference block), else a
loud diagonal-placeholder fallback.
nz/metadata come from the shared core helpers sacc_nz / sacc_metadata.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mble)
- run_2pcf.py: after calculate_2pcf, write {ver}_xi_coarse.sacc via
xi_to_sacc(grid='coarse', theta_nom=rnom, npairs, weight) — no covariance
(added at assembly). Drops the save_fits threading (mixin no longer writes
DES-style xi FITS).
- run_2pcf_highres.py: on rank 0, write the terminal {version}_xi_fine.sacc via
xi_to_sacc(grid='fine', variances=[varxip; varxim]) as a DiagonalCovariance.
Deletes write_xi_fits (DES-FITS); keeps the .txt. sacc_io import works on the
bare-host MPI path (no healpy), n(z) read from shear.redshift_path.
- generate_pseudo_cl.py: native product is now pseudo_cl_{ver}.sacc (the mixin
writes it); report reads it back via sacc_io.get_pseudo_cl.
- assemble_sacc.py (new, dual-mode): loads the per-statistic parts in canonical
order and calls assemble_analysis_sacc. Injects the xi-coarse block from the
CosmoCov .txt (already [xi+;xi-]-ordered) and the pseudo-Cl block from the
NaMaster iNKA cov FITS as block-diagonal [EE_EE;BB_BB;EB_EB] (cross-spectrum
blocks dropped, TODO flagged). --allow-placeholder attaches a documented
diagonal for cov-less parts so DAG dry-runs / tests produce a valid
FullCovariance; production requires the real cov inputs (fail-fast otherwise).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t_config The migrated pseudo-Cl SACC part calls get_redshift(), which reads cc[version]['shear']['redshift_path']; the synthetic fixture wrote the dndz file but never pointed that key at it (KeyError at core.py:422). Verified: the KeyError layer is resolved; the end-to-end test now fails one layer deeper (test asserts the legacy ELL/EE/BB/EB FITS columns while the migrated writer emits the SACC part - the open wiring seam). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ko89uZF84ez6HEDcTWbJe
…-flight Working-tree state of the wiring subagent when it died on the session limit (its final turn also went unreviewed by the safety classifier). Wires cosmo_val.smk/twopoint.smk rules and cv_* scripts to the SACC parts + assemble_sacc rule. NOT dry-run-tested (no snakemake in the shared venv), NOT reviewed; test_calculate_pseudo_cl_catalog_end_to_end still fails at the writer/consumer seam (test reads legacy ELL columns). Next session: verify this diff against the DAG design in the PR-4 plan, reconcile the pseudo-Cl end-to-end test with the born-as-SACC output, add writer-path tests, run the suite, then de-WIP. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ko89uZF84ez6HEDcTWbJe
… test calculate_pseudo_cl_catalog is born-as-SACC (pseudo_cl_to_sacc_part writes EE/BB/EB + a shared bandpower window), so the end-to-end test can no longer read legacy ELL/EE/EB/BB FITS columns. Round-trip through sacc_io.get_pseudo_cl instead and assert the bandpower window rides the part per the layout contract. The pinned golden spectra are unchanged (identical computation, SACC serialization); they held bitwise through the round-trip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HPfbe2XVTGzbnPn4g7BdN6
… parts
The born-as-SACC parts existed but nothing assembled them. Wire the DAG:
per-statistic parts (xi_coarse, pseudo_cl, cosebis, pure_eb, rho_tau) →
assemble_sacc rule → terminal {version}.sacc.
- rule xi: give the coarse ξ± .sacc output the same reporting-binning wildcards
as the .txt (Snakemake requires one wildcard set per rule; the bare
{version}_xi_coarse.sacc name left the binning wildcards unbound and broke DAG
resolution). run_2pcf.py writes the part to the declared output path.
- rule rho_tau_stats: declare the rho_tau .sacc part (already written by
calculate_rho_tau_stats) as a real output; run_rho_tau.py verifies it.
- new rule assemble_sacc + assemble_sacc_all: load the five parts in canonical
order via assemble_sacc.py, injecting a documented diagonal placeholder for
the cov-less ξ/pseudo-Cℓ blocks (real CosmoCov/NaMaster covariance plugs into
the same --xi-cov/--pseudo-cl-cov seam later — PR-3 converter territory).
- cosmo_val_all: request the terminal {version}.sacc per version.
- cv_pseudo_cl.py docstring: born-as-SACC, no longer FITS.
Dry-runs clean for assemble_sacc_all, cosmo_val_all, bmode_summary.json, and
the fine-grid xi_fine target.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPfbe2XVTGzbnPn4g7BdN6
… seam
The pure assembler is covered in test_sacc_writers; this pins the DAG-facing
script that loads per-statistic .sacc part *files* and rebuilds {version}.sacc:
canonical block order across all five statistics, the diagonal placeholder for
the cov-less ξ± part, real CosmoCov ξ covariance .txt injection, the fail-loud
path when a required covariance is absent, and the pseudo_cl config toggle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPfbe2XVTGzbnPn4g7BdN6
The MPI xi_highres rule shells out to run_2pcf_highres.py by absolute path through the deprecated pure_eb/ compat symlink (it can't use Snakemake's script: directive — mpiexec wraps the python call). Anchor the path on common.py's own location (WORKFLOW_SCRIPTS), which resolves to the generic workflow/scripts of whatever checkout parses the DAG, regardless of which paper composes it — workflow.basedir is unreliable here (under module composition it reflects the composing paper, verified: it resolved to papers/cosmo_val/scripts). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HPfbe2XVTGzbnPn4g7BdN6
… real cov
The analysis {version}.sacc must be byte-comparable against today's
cosmosis_fitting.py assembly (the SACC layout contract's stated requirement for
PR-3's converter). cosmosis consumes the tagged, blinded pseudo-Cl product
(blind=A, powspace, nbins=32 per config harmonic.fiducial), not the untagged
cv_pseudo_cl diagnostic — so assemble_sacc now inputs the tagged part and
injects its real NaMaster covariance from the matching pseudo_cl_cov FITS
(COVAR_EE_EE/BB_BB/EB_EB → block-diagonal). The untagged cv_pseudo_cl part stays
the cv_summarize_bmodes B-mode diagnostic, unchanged.
The ξ± coarse block keeps a documented diagonal placeholder: its real CosmoCov
covariance is blind/gaussian/mask-keyed in the inference tree, and wiring it
couples cosmo_val to the whole inference covariance DAG — that sourcing is PR-3
converter territory, ready at the --xi-cov seam.
test_assemble_sacc: add the pseudo-Cl COVAR-FITS injection path (the live
default) alongside the existing ξ .txt and placeholder paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPfbe2XVTGzbnPn4g7BdN6
…OFILE test_bmodes_workflow_dry_runs invoked a bare python3.12 (resolves off PATH to intel-python without snakemake) and inherited the login shell's slurm SNAKEMAKE_PROFILE (forces an executor plugin the test env need not have). Use sys.executable (the interpreter pytest/snakemake live in) and drop the profile for the dry-run. The guard now passes and exercises the migrated rule graph (rule xi's coarse-sacc output, the WORKFLOW_SCRIPTS path). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HPfbe2XVTGzbnPn4g7BdN6
…red to PR 7
The SACC migration removed the xi_plus/xi_minus FITS and switched pseudo_cl to
.sacc, so inference_prep's DAG no longer resolves. It is not reachable from the
cosmo_val suite (cosmo_val_all never requests it), so the cosmo_val DAG stays
clean. Per the PR-4 scope this subsystem is left dormant — a comment block at
the rule head names the stale inputs and states PR 7 (native-SACC inference
consumption) rewires it to consume the assembled {version}.sacc directly,
retiring cosmosis_fitting.py's per-product FITS assembly. Comment-only; no
behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPfbe2XVTGzbnPn4g7BdN6
…collision
Both pseudo-Cl rules call calculate_pseudo_cl, which hardcoded the untagged
native pseudo_cl_{ver}.sacc; generate_pseudo_cl.py then renamed it to the tagged
name. In one cosmo_val_all DAG both fire: if the untagged cv_pseudo_cl (blind=
None) ran first, the tagged rule's (blind=A) skip-if-exists silently adopted the
blind=None file and the rename deleted cv_pseudo_cl's declared output — rebuild
loops, and wrong-blind n(z) stamped into the terminal {version}.sacc's pseudo-Cl
part.
Thread out_path through calculate_pseudo_cl so each part is born directly at its
final declared name (tagged for the producer, untagged for the diagnostic). No
shared native basename, no rename; skip-if-exists keys on the declared path, so
the two rules' paths are provably disjoint and the blind that computes each file
matches its name. Multi-version + out_path now raises. generate_pseudo_cl.py and
its CLI updated to pass the declared/native out_path directly.
Regression tests: out_path is honoured (native name untouched) and the
multi-version guard fires.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPfbe2XVTGzbnPn4g7BdN6
…covariance
{version}.sacc is the terminal science file; shipping it with a var=1.0 diagonal
as the LEADING (ξ±) covariance block — ~20 orders off the real variance — is a
silent catastrophic χ²/PTE for any consumer. Drop the unconditional
placeholder_var=1.0. Default: no real ξ-cov wired → assemble_sacc.py's existing
ValueError fires. The placeholder is now gated behind an explicit opt-in
(cosmo_val.allow_placeholder_cov: true), for dry-run / test configs only. The
pseudo-Cℓ block stays real (pseudo_cl_cov input); the integration tests pass
placeholder_var explicitly and are unaffected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPfbe2XVTGzbnPn4g7BdN6
test_bmodes_workflow_dry_runs uses papers/bmodes, whose config has no cosmo_val block, so cosmo_val.smk (the born-as-SACC + assemble rules) is never included — the guard couldn't catch a break in them. Add test_cosmo_val_workflow_assemble_ dry_runs targeting assemble_sacc_all in papers/cosmo_val (the only paper that includes cosmo_val.smk): asserts the DAG resolves and each assemble_sacc job pulls the tagged, blinded pseudo-Cl part + its NaMaster cov (not the untagged diagnostic) plus all five per-statistic parts. Shared _dry_run helper factored out of the bmodes guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HPfbe2XVTGzbnPn4g7BdN6
…f call
run_xi_sweep.py called run_2pcf(..., save_fits=True) — a kwarg the SACC
migration removed, so every invocation TypeErrors. The sweep consumes only the
.txt dump; drop the kwarg. run_2pcf is born-as-SACC, so give its coarse part a
grid-qualified sacc_out (the default {ver}_xi_coarse.sacc carries no binning, so
the reporting + integration grids would collide per version). Stale "+ ξ+/ξ-
FITS" docstring dropped.
test_cli_seams: bind the sweep's exact run_2pcf call against the live signature
(and assert save_fits no longer binds) so this CLI seam can't silently rot —
the compute is cluster-only and never exercised by the fast suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPfbe2XVTGzbnPn4g7BdN6
_from_snakemake built part_paths with a hasattr filter, so a typo'd input
keyword (cosebi for cosebis) silently omitted that statistic from the terminal
{version}.sacc — the exact silent-truncation class this series has been bitten
by. assemble_sacc now takes an `expected` list (the statistics the caller wired,
from its config toggles) and raises if any is missing from part_paths or names a
non-CANONICAL statistic. The rule derives expected from cv_assemble_inputs so it
tracks the include_pseudo_cl toggle. CLI path is already typo-safe (argparse
rejects unknown flags), so it passes expected=None.
Tests: typo'd input key raises; typo in the expected list itself raises.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPfbe2XVTGzbnPn4g7BdN6
This was referenced Jul 10, 2026
Draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #247. Row 4 of the PRD #241 §7 table: cosmo_val data products are born as SACC. Stacked on #245 (
sacc_io, base branchfeat/sacc-2-sacc-io).What this does
cosmo_val/sacc_writers.py+ mixin*_to_sacc_partmethods): coarse/fine ξ± (TreeCorr →xi_to_sacc), pseudo-Cℓ EE/BB/EB with the NaMasterBandpowerWindow(new plumbing — nothing saved windows before), COSEBIs at the fiducial scale cut, pure-E/B, ρ/τ (τ theory covariance from theCovTauThwrite-side).assemble_saccrule loads the parts in canonical order, injects the externally-sourced covariance blocks (NaMaster FITS for pseudo-Cℓ; CosmoCov--xi-covseam for ξ±, ready for the Converters: SACC→2pt-FITS (byte-compared) + SACC↔OneCovariance glue #249 converter round-trip), validates the expected-parts set, and writes the terminal{version}.saccanalysis file.rule xi_highreswrites the terminal{version}_xi_fine.sacc(DiagonalCovariance fromvarxip/varxim) for COSEBIs/pure-E/B integration.Flags for reviewers
60b74c6); the dry-run guard not covering the new rules (6a5df5b); silent part-drop on a misnamed rule input (149a50c); plus asave_fitsTypeError inrun_xi_sweep.py(3561b36).e0519c8): with no real ξ-covariance wired, assembly fails loudly by default.cosmo_val.allow_placeholder_cov: trueexists for dry-runs/tests only — the placeholder is flagged and never a science covariance.{version}.saccis the tagged legacy inference product (blind=A_powspace_nbins=32). The measured spectra are blind-independent (catalogue shear, no n(z) dependence) and the iNKA covariance is analytic, so the tag is provenance-naming, not wrong data; blind-label reconciliation is PR 6's scope (Smokescreen).{version}.sacc,{version}_xi_fine.sacc) have contract-stable names.dd474b3's "UNVERIFIED WIP" message is historical — it preserved an interrupted session's state; the verification it says is missing is what the subsequent commits did (wildcard fix, assemble rule, tests, review fixes).inference.smk(dormant) still names pre-SACC inputs; migration to consuming{version}.saccis the native-sacc PR (row 7).xi_highresnow resolves its script path from the running checkout (9afaeaa) — replaces a hardcoded path through the deprecatedpure_eb/compat symlink.— Claude (Fable) on behalf of Cail.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HPfbe2XVTGzbnPn4g7BdN6