Image simulations workflow#766
Open
martinkilbinger wants to merge 33 commits into
Open
Conversation
added 25 commits
May 17, 2026 19:39
Closed
6 tasks
…preview # Conflicts: # example/cfis/config_tile_Sx_nomask.ini # example/cfis_simu/readme.txt # scripts/sh/job_sp_canfar_v2.0.bash # src/shapepipe/modules/ngmix_package/ngmix.py # src/shapepipe/modules/ngmix_runner.py # uv.lock
Keep only genuine sims-enablement; revert shared-module changes that were side-effects of the stale ngmix_v2.0 base and collide with in-flight work: - make_cat: restore HSM_G*_PSF grammar (fix/ngmix-psf-columns) + spread_model (714847c, split-761); drops the input_file_List NameError - vignetmaker: drop broad try/except churn (_w_log AttributeError -> test fail) - mask: restore CDSCLIENT_PATH knob (live in 4 example configs) - restore pipeline_tutorial.md (toc.rst ref -> Sphinx build) + run_scratch_local.sh Verification branch only; not for push. Grounds the #766 review.
… -> sp_validation The image-sims workflow splits by concern along the repo dependency arrow (shapepipe produces catalogues; sp_validation consumes + validates them): - ShapePipe (this PR): only the pipeline-module changes needed to run on background-free sims -- the fake_psf module, ngmix BKG_SUB path, numeric sim-exposure-ID handling (exp_utils/sextractor/merge_headers), the cfis_image_sims module configs, create_final_cat's image_sims mode, and make_data_pujol. - sp_validation (separate PR): the snakemake orchestration + the merge/calibrate/m-bias half, rebuilt on sp_validation's native container workflow. The orchestration files removed here are preserved in git history (origin/im_sims) and move there. Incidental stale-base edits to shared modules/standard configs/tests reverted to ngmix_v2.0 (which already carries the fixes: 662fad4 HSM g-cols, 714847c spread_model). Retargeted onto ngmix_v2.0; uv.lock regenerated.
Contributor
|
@martinkilbinger I split off the snakemake parts to sp_validation and kept only the shapepipe changes here (please see updated description above). |
The ngmix_v2.0 retarget slimmed run_job_sp_canfar_v2.0.bash and job_sp_canfar_v2.0.bash of their image-sims code paths, leaving #766 unable to run the image-sims pipeline at all: run_job had no -t/--type dispatch (so the sp_validation workflow's `-t image_sims` was silently swallowed and it ran standard-DATA mode), and its job-1024 completeness check referenced ngmix_interp_runner, a runner that exists nowhere in the codebase. job_sp_canfar rejected `-r symlink` ("vos only for v2.0"), named the Gie and make_cat configs for the data path only, and still invoked shapepipe_run.py (the console entry point is `shapepipe_run`, no extension, since 797eeff). run_job: restored the canonical image-sims runner (full -t/--type dispatch — cfis_image_sims config dir, symlink retrieve, tile_det=sx, faked Uz weight-uncompress, fake_psf under job 512, job-2048 make_cat, and the correct ngmix_runner check). Made it relocatable per #766's goal: a self-derived $SP_ROOT replaces ~/shapepipe for sourcing functions.sh/job_list_help.bash and resolving config dirs, and job_sp_canfar is invoked by explicit repo path (it is not on the container PATH). run_tile_job now re-anchors cwd to the tile work dir on entry: SP_RUN/SP_CONFIG derive from cwd, and run_exp_job leaves cwd in the last exposure dir, so when run_job is called once for the whole pipeline (-j 4095, as the workflow does) the post-exposure tile jobs would otherwise execute in the stale exposure dir. job_sp_canfar: kept the ngmix_v2.0 base — the #746 NUMBER_LIST mechanism that matches the new-grammar container (the canonical runner still uses the retired shapepipe_run -e flag and would crash) — and grafted the image-sims bits: accept `-r symlink`, config_exp_Gie_$retrieve, config_tile_Mc_$psf (the config present in cfis_image_sims and matching run_job's job-2048 check), the bare `shapepipe_run` entry point, and a self-derived $SP_ROOT. Verified on candide against the post-#761 container: `-t image_sims` selects cfis_image_sims + symlink + sx and runs Git -> faked-Uz -> Fe -> Gie(7/7) -> Sp -> Ma -> Mh_exp, each in its correct work dir with new-grammar shapepipe_run. The two scripts run from the bind-mounted repo, so the container supplies only the Python stack. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Jd85D8xNZZ43FgesabSnk
The image-sims make_cat config lacked SM_DO_CLASSIFICATION, which the newer make_cat_runner reads unconditionally (NoOptionError -> job 2048 crashed, so the per-tile final_cat was never built). The image-sims pipeline has no spread_model_runner stage (make_cat inputs are sextractor + fake_psf + ngmix only), so there is nothing to classify: set it False. With this, a single tile completes end-to-end (Git -> ... -> ngmix -> make_cat), producing a 2345-object final_cat-233-293.fits against the post-#761 container. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Jd85D8xNZZ43FgesabSnk
create_final_cat selects this column list from the per-tile make_cat output
into final_cat_{sim}.hdf5; the list was still old-grammar (NGMIX_ELL_*,
NGMIX_Tpsf_*, NGMIX_T_PSFo_*, NGMIX_MOM_FAIL) while the post-#761 make_cat
writes NGMIX_G{1,2}_*, NGMIX_T_PSF_ORIG/RECONV_*, so the merge died on
'NGMIX_Tpsf_2M' is not in list.
Rewrite to the v2 grammar, mirroring example/cfis/final_cat.param minus the
two columns the simulation pipeline does not emit (IMAFLAGS_ISO: no
imaging-flag masking stage; NGMIX_MOM_FAIL: superseded by NGMIX_MCAL_TYPES_FAIL,
which is included). Verified: all 65 columns present in the per-tile cat, merge
-> extract -> calibrate runs clean on 1z2z_grid_1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D9atA9VPrg7VMC7oBArny9
Contributor
|
Status update (description rewritten to match): pushed the runner restoration ( The end-to-end m-bias run this enabled surfaced two bugs, but both live downstream in the shared sp_validation calibration path — sp_validation#226 and sp_validation#227 — not in anything here. From the ShapePipe side this PR is, as far as we can tell, complete; the trustworthy m-bias number waits on the calibration design call in #226. — Fable, on behalf of Cail |
With HALO_MAKE and SPIKE_MAKE both False (the image-sims mask config), find_stars never runs, so _CDS_stdout is never set and the unconditional read at the end of make_mask raised AttributeError for every CCD. The flag files were already written, so each mask stage 'failed' with 40 bookkeeping errors and exit 1, relying on the runner's continue-on-error to keep the pipeline alive. Guard the CDS block with hasattr, matching the WeightWatcher/rm blocks below it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TWURLMoixhuMqVrPVdRmCp
This was referenced Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Image simulations — ShapePipe module enablement
Part of [Tracking] Image simulations runnable by Martin & others (UNIONS-WL/MultiBand_ImSim#1).
Running ShapePipe on image simulations splits along the repo dependency arrow — ShapePipe produces catalogues, sp_validation consumes and validates them:
What's here
fake_psfmodule — synthetic per-galaxy PSF stamps from a precomputed dict, drop-in forpsfex_interp's downstream format.BKG_SUBngmix path — skip background subtraction when a sim has no background image (shorter input list); composes additively withngmix_v2.0'sBKG_RMS_VIGNET_PATHandCENTROID_SOURCE.exp_utils,sextractor_script, andmerge_headers'WARN_MISSING_EXPoption.cfis_image_sims/module configs andcreate_final_cat'simage_simsmode.make_data_pujol— Pujol denoising shear-bias estimator intesting/simulate.py.run_job/job_sp_canfarhad lost the-t image_simsdispatch path in the retarget (plus a symlink-retrieval rejection, stale config names, and a tile-vs-exposure cwd bug) — restored and de-hardcoded, sojob_sp_canfar -t image_simsdrives the full chain again. Plus new-grammarfinal_cat.paramand theSM_DO_CLASSIFICATIONconfig fix for the image-sims configs.Base
Retargeted from
developontongmix_v2.0, so the diff shows only the sims enablement (the branch had merged a stalengmix_v2.0snapshot; the retarget also fixes theuv.lockthat was breaking CI). Incidental edits to shared modules were dropped in favour ofngmix_v2.0's versions, which already carry the relevant fixes.Status: verified end-to-end; the remaining blockers are downstream in sp_validation
This half is done and verified: the full ShapePipe chain (raw SKiLLS tile → per-tile catalogues →
final_cat_{sim}.hdf5) runs per-stage on candide against the post-#761 container built from this branch, on the new ngmix column grammar. The end-to-end m-bias run this enabled then surfaced two real bugs in the shared calibration path — sp_validation#226 (additive-bias subtraction zeroes the signal on constant-shear sims → m = −1) and sp_validation#227 (pathological Gatti weights on the sims) — which gate the trustworthy m-bias number but don't touch anything in this PR. Details and the design fork live in those issues.— Fable, on behalf of Cail