Name every tile_shape store per campaign, not only for image sims - #945
Open
cailmdaley wants to merge 1 commit into
Open
cailmdaley wants to merge 1 commit into
cailmdaley wants to merge 1 commit into
Conversation
LOCAL_TAG, the run-dir hash in /local/scratch/sp-{LOCAL_TAG}{tile}, was empty
for data campaigns, so two concurrent data campaigns over the same tiles
(two-arm A/B runs) share one node-local store whenever their same-tile
tile_shape jobs land on one node. Make the tag unconditional.
The params pin moves for tile_vignets, tile_ngmix and tile_make_cat; the pin
normalises the run-dir hash so it stays root-independent and still catches an
empty tag. New test: two campaigns get distinct stores per tile, and every
tile_shape member of a tile names the same store.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dj4HnkNqqPkVCeDeUGEGuz
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every
tile_shapenode-local store is now named per campaign:LOCAL_TAG(a hash of the run dir) is unconditional, not only set forimage_sims.Two concurrent data campaigns over the same tiles, such as A/B arms, can no longer share
/local/scratch/sp-<tile>when their same-tile jobs land on one node.Closes #944, which has the mechanism and our forensics (0 of 895
tile_shapejobs on nibi ever collided).Changes
workflow/rules/tile.smk:LOCAL_TAG = sha1(RUN_DIR)[:8] + "-"for every input type; thetile_local()docstring describes the per-campaign naming.tests/workflow/test_dag.py: newtest_tile_store_is_unique_per_campaign, for bothdata+psfexandimage_sims+fake. Two campaigns at different roots must give each tile different stores, and within a campaigntile_vignets, alltile_ngmixchunks andtile_make_catmust name the same store.tests/workflow/params.py: the pin replaces the run-dir hash with<RUN_DIR_SHA1>before hashing, computed from the fixture's run dir rather than read fromLOCAL_TAG. The pin stays root-independent, and an empty tag still moves it.tests/workflow/params_pin.json: regenerated. Onlytile_vignets,tile_ngmixandtile_make_catchange.tests/workflow/README.md,profiles/nibi/config.yaml: the normalisation, mutation probes and store-name comment are updated to match.Resuming a data campaign across this change
The store path is in
params.pre, andparamsis a rerun trigger in both profiles.So the first
sp runof an existing data campaign after this change reschedulestile_vignets, the eighttile_ngmixchunks andtile_make_catfor every finished tile.As the warning block above
LOCAL_TAGintile.smkexplains, that rerun cannot succeed once the tile's exposure stores are reclaimed, and the failed group deletes the finishedfinal_cat.Running campaigns are unaffected (they use their launch snapshot).
Land this between campaigns. If an existing data campaign must be resumed, run that invocation once with
--rerun-triggers mtime code software-env.Image-sim campaigns are unaffected: their store names do not change.
Tests
Run on nibi in an allocation, in
shapepipe-develop-240b37e4.sifwith the checkout'ssrc/onPYTHONPATH:tests -m "not slow": 936 passed, 4 skipped, 2 deselected.tests/workflow(snakemake 9.23.1, params pins forced withSHAPEPIPE_ON_CANDIDE=1): 19 passed, after--update-params-pinand again without it.image_sims-only tag failstest_tile_store_is_unique_per_campaign[data+psfex]andtest_unit_pre_changes_at_campaign_boundary(2 failed, 17 passed).test_unit_pre_changes_at_campaign_boundaryandtest_params_pin_ignores_fixture_root(2 failed, 17 passed).The sweep-ordering hazard noted in #944 is out of scope here.
— Claude (Opus) on behalf of Cail
🤖 Generated with Claude Code
https://claude.ai/code/session_01Dj4HnkNqqPkVCeDeUGEGuz