Skip to content

Add tomographic and 3x2pt covariance realspace measurement #256

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LisaGoh wants to merge 4 commits into
base: develop
Choose a base branch
from
+371 −181
Open

Add tomographic and 3x2pt covariance realspace measurement #256

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ee0b5d1
implement cosmocov
LisaGoh Jul 10, 2026
e8e0462
ruff check
LisaGoh Jul 10, 2026
0d4a6a6
ruff format
LisaGoh Jul 10, 2026
00a0f6b
ruff format
LisaGoh Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions scratch/lgoh/cov_config_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
fiducial:
version: SP_v1.4.6
blind: A
min_sep: 1.0
max_sep: 100.0
nbins: 20
npatch: 30
mock_version: SP_v1.4.6
gaussian: g

probe_3x2pt: wl
glass_mocks:
version: "v0"
seed_range: [1, 350]

covariance:
default_masked: True

tools:
python_executable: /n23data1/n06data/lgoh/scratch/my_env_new/bin/python
onecov_executable: /n23data1/n06data/lgoh/scratch/UNIONS/OneCovariance/covariance.py

SP_v1.4.6:
cov_th:
sigma_e: 0.28, 0.28
n_e: 5.0, 5.0
A: 1000
104 changes: 82 additions & 22 deletions workflow/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,23 +9,27 @@
# fresh tree without clobbering (or silently reusing) prior products.
COSMO_VAL = Path(
os.environ.get(
"COSMO_VAL", "/n17data/cdaley/unions/code/sp_validation/cosmo_val/output"
"COSMO_VAL",
"/n23data1/n06data/lgoh/scratch/UNIONS/sp_validation/results/cosmo_val",

@cailmdaley cailmdaley Jul 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than these defaults carrying anyone's personal tree (the old values were Cail's, so this is a fix-the-pattern note, not a revert request): suggest requiring COSMO_VAL / COSMO_INFERENCE / CAT_CONFIG explicitly — env var or config key, loud error if unset — so each of us sets our own and the repo carries nobody's paths. Fine as a small follow-up if you'd rather keep this PR focused.

)
)
COSMO_INFERENCE = Path(
os.environ.get(
"COSMO_INFERENCE", "/n17data/cdaley/unions/code/sp_validation/cosmo_inference"
"COSMO_INFERENCE",
"/n23data1/n06data/lgoh/scratch/UNIONS/sp_validation/cosmo_inference",
)
)
CAT_CONFIG = "/n17data/cdaley/unions/code/sp_validation/cosmo_val/cat_config.yaml"
CAT_CONFIG = (
"/n23data1/n06data/lgoh/scratch/UNIONS/sp_validation/cosmo_val/cat_config.yaml"
)
BLINDS = ["A", "B", "C"]
BLOCK_PAIRS = [("++", "1"), ("--", "2"), ("+-", "3")]

# Fiducial cosmology: Planck 2018 (astropy Planck18, Table 2 + BAO)
# Source of truth: cs_util.cosmo.PLANCK18
# Regenerate with: snakemake results/cosmology/planck18.json
# Resolved relative to the run directory at configure() time.
COSMOLOGY_PARAMS = "results/cosmology/planck18.json"
COSMOLOGY_PARAMS = "/n23data1/n06data/lgoh/scratch/UNIONS/sp_validation/results/cosmology/planck18.json"

@cailmdaley cailmdaley Jul 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one does need to stay relative (independent of the defaults question above): the cosmology_params rule generates this file at the relative path results/cosmology/planck18.json in the run directory, so pointing the constant at an absolute path decouples the reader from the generator — the rule would write one file and everything else would read another.


# Wildcard constraints shared by every Snakefile that composes these rules.
# Patterns must match all expected values; overly restrictive patterns cause
Expand All @@ -37,6 +41,7 @@
"min_sep": r"[0-9.]+",
"max_sep": r"[0-9.]+",
"gaussian": r"(g|ng)",
"probe": r"(wl|ggl|3x2pt)",
"block_pm": r"(\+\+|--|\+-)",
"block_i": r"[123]",
"mask_suffix": r"(_masked)?",
Expand All @@ -46,15 +51,17 @@

FIDUCIAL = None
DEFAULT_MASK_SUFFIX = ""
DEFAULT_PROBE = None
CATALOG_CONFIG = None
PLANCK18 = None


def configure(workflow_config):
"""Install config-derived values after Snakemake has loaded configfiles."""
global CATALOG_CONFIG, DEFAULT_MASK_SUFFIX, FIDUCIAL, PLANCK18
global CATALOG_CONFIG, DEFAULT_MASK_SUFFIX, DEFAULT_PROBE, FIDUCIAL, PLANCK18
CATALOG_CONFIG = workflow_config
FIDUCIAL = workflow_config["fiducial"]
DEFAULT_PROBE = "wl"
DEFAULT_MASK_SUFFIX = (
"_masked" if workflow_config["covariance"].get("default_masked", False) else ""
)
Expand All @@ -76,44 +83,78 @@ def resolve_covariance_version(version):
return version


# Single source of truth for the covariance naming scheme. Rule outputs use
# this with wildcards left in braces; covariance_base() fills concrete values.
# Keep the two in sync by construction: covariance_base() formats this string.
COV_BASE_TEMPLATE = (
"covariance_{version}_{blind}_{gaussian}"
"_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}_{probe}{mask_suffix}"
)


def cov_output(suffix):
"""Wildcard-bearing output path under the covariance tree.

E.g. cov_output(".ini") ->
<COSMO_INFERENCE>/data/covariance/<BASE>/<BASE>.ini
with {version}, {blind}, ... left as Snakemake wildcards.
"""
return str(
COSMO_INFERENCE
/ f"data/covariance/{COV_BASE_TEMPLATE}/{COV_BASE_TEMPLATE}{suffix}"
)


def cov_output_dirfile(filename):
"""Wildcard-bearing path to a fixed-name file inside the covariance dir."""
return str(COSMO_INFERENCE / f"data/covariance/{COV_BASE_TEMPLATE}/{filename}")


def covariance_base(
version,
blind,
gaussian="ng",
gaussian=None,
min_sep=None,
max_sep=None,
nbins=None,
probe=None,
mask_suffix=None,
resolve_version=True,
fiducial=None,
default_mask_suffix=None,
):
"""Construct covariance base name."""
"""Construct covariance base name (concrete values, no wildcards)."""
fiducial = fiducial or FIDUCIAL
gaussian = gaussian if gaussian is not None else fiducial["gaussian"]
min_sep = min_sep if min_sep is not None else fiducial["min_sep"]
max_sep = max_sep if max_sep is not None else fiducial["max_sep"]
nbins = nbins if nbins is not None else fiducial["nbins"]
mask_suffix = (
mask_suffix
if mask_suffix is not None
else (
probe = probe if probe is not None else DEFAULT_PROBE
if mask_suffix is None:
mask_suffix = (
DEFAULT_MASK_SUFFIX if default_mask_suffix is None else default_mask_suffix
)
)
cov_version = resolve_covariance_version(version) if resolve_version else version
return (
f"covariance_{cov_version}_{blind}_{gaussian}"
f"_minsep={min_sep}_maxsep={max_sep}_nbins={nbins}{mask_suffix}"
return COV_BASE_TEMPLATE.format(
version=cov_version,
blind=blind,
gaussian=gaussian,
min_sep=min_sep,
max_sep=max_sep,
nbins=nbins,
probe=probe,
mask_suffix=mask_suffix,
)


def covariance_dir(
version,
blind,
gaussian="ng",
gaussian=None,
min_sep=None,
max_sep=None,
nbins=None,
probe=None,
mask_suffix=None,
resolve_version=True,
):
Expand All @@ -125,6 +166,7 @@ def covariance_dir(
min_sep,
max_sep,
nbins,
probe,
mask_suffix,
resolve_version=resolve_version,
)
Expand All @@ -134,12 +176,13 @@ def covariance_dir(
def covariance_path(
version,
blind,
gaussian="ng",
gaussian=None,
min_sep=None,
max_sep=None,
nbins=None,
probe=None,
mask_suffix=None,
suffix="_processed.txt",
suffix=".dat",
resolve_version=True,
):
"""Construct covariance file path."""
Expand All @@ -150,22 +193,39 @@ def covariance_path(
min_sep,
max_sep,
nbins,
probe,
mask_suffix,
resolve_version=resolve_version,
)
return str(COSMO_INFERENCE / f"data/covariance/{base}/{base}{suffix}")


def build_redshift_path(version, blind):
def build_redshift_dir(version):
"""Construct n(z) filepath for given catalog version and blind."""
base_version = re.sub(r"_leak_corr$", "", version)
base_version = re.sub(r"_ecut\d+", "", base_version)
if "v1.4.11" in base_version:
base_version = "SP_v1.4.6"
version_dir = base_version.replace("SP_", "")
return (
f"/n17data/sguerrini/UNIONS/WL/nz/{version_dir}/nz_{base_version}_{blind}.txt"
)
return f"/n17data/sguerrini/UNIONS/WL/nz/{version_dir}/"


def build_redshift_path_lens(version, blind):
"""Construct n(z) filepath for given catalog version and blind."""
base_version = re.sub(r"_leak_corr$", "", version)
base_version = re.sub(r"_ecut\d+", "", base_version)
if "v1.4.11" in base_version:
base_version = "SP_v1.4.6"
return f"nz_{base_version}_{blind}.txt"


def build_redshift_path_source(version, blind):
"""Construct n(z) filepath for given catalog version and blind."""
base_version = re.sub(r"_leak_corr$", "", version)
base_version = re.sub(r"_ecut\d+", "", base_version)
if "v1.4.11" in base_version:
base_version = "SP_v1.4.6"
return f"nz_{base_version}_{blind}_source.txt" # TO DO: check lens and source file format

@cailmdaley cailmdaley Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three copies of the same version-normalization regex block now — worth one _nz_base_version(version) helper that the three functions share.

Also a naming/semantics check to resolve with the TO-DO: in the ini, zlens_file gets build_redshift_path_lens (the existing source-galaxy n(z)) and zclust_file gets ..._source.txt. OneCov's zlens is the lensing source sample and zclust the clustering lens sample — so a file suffixed _source feeding zclust reads inverted. If the intent is _lens.txt for the clustering sample, let's fix the suffix before anyone generates files against this convention.

Minor: build_redshift_dir returns a trailing-slash dir which the rule then passes through os.path.dirname, and the _lens/_source helpers return bare filenames that go through os.path.basename. Both round-trips can go — return dir and filenames and use them directly.



def get_shear_catalog(wildcards):
Expand Down
Loading
Loading