Skip to content

Name every tile_shape store per campaign, not only for image sims - #945

Open
cailmdaley wants to merge 1 commit into
developfrom
fix/local-tag-data
Open

cailmdaley wants to merge 1 commit into
developfrom
fix/local-tag-data

Conversation

@cailmdaley

Copy link
Copy Markdown
Contributor

Every tile_shape node-local store is now named per campaign: LOCAL_TAG (a hash of the run dir) is unconditional, not only set for image_sims.
Two concurrent data campaigns over the same tiles, such as A/B arms, can no longer share /local/scratch/sp-<tile> when their same-tile jobs land on one node.
Closes #944, which has the mechanism and our forensics (0 of 895 tile_shape jobs on nibi ever collided).

Changes

  • workflow/rules/tile.smk: LOCAL_TAG = sha1(RUN_DIR)[:8] + "-" for every input type; the tile_local() docstring describes the per-campaign naming.
  • tests/workflow/test_dag.py: new test_tile_store_is_unique_per_campaign, for both data+psfex and image_sims+fake. Two campaigns at different roots must give each tile different stores, and within a campaign tile_vignets, all tile_ngmix chunks and tile_make_cat must name the same store.
  • tests/workflow/params.py: the pin replaces the run-dir hash with <RUN_DIR_SHA1> before hashing, computed from the fixture's run dir rather than read from LOCAL_TAG. The pin stays root-independent, and an empty tag still moves it.
  • tests/workflow/params_pin.json: regenerated. Only tile_vignets, tile_ngmix and tile_make_cat change.
  • tests/workflow/README.md, profiles/nibi/config.yaml: the normalisation, mutation probes and store-name comment are updated to match.

Resuming a data campaign across this change

The store path is in params.pre, and params is a rerun trigger in both profiles.
So the first sp run of an existing data campaign after this change reschedules tile_vignets, the eight tile_ngmix chunks and tile_make_cat for every finished tile.
As the warning block above LOCAL_TAG in tile.smk explains, that rerun cannot succeed once the tile's exposure stores are reclaimed, and the failed group deletes the finished final_cat.
Running campaigns are unaffected (they use their launch snapshot).
Land this between campaigns. If an existing data campaign must be resumed, run that invocation once with --rerun-triggers mtime code software-env.
Image-sim campaigns are unaffected: their store names do not change.

Tests

Run on nibi in an allocation, in shapepipe-develop-240b37e4.sif with the checkout's src/ on PYTHONPATH:

  • tests -m "not slow": 936 passed, 4 skipped, 2 deselected.
  • tests/workflow (snakemake 9.23.1, params pins forced with SHAPEPIPE_ON_CANDIDE=1): 19 passed, after --update-params-pin and again without it.
  • Mutation probes, each in a disposable copy:
    • Restoring the image_sims-only tag fails test_tile_store_is_unique_per_campaign[data+psfex] and test_unit_pre_changes_at_campaign_boundary (2 failed, 17 passed).
    • Removing the run-dir-hash normalisation fails test_unit_pre_changes_at_campaign_boundary and test_params_pin_ignores_fixture_root (2 failed, 17 passed).

The sweep-ordering hazard noted in #944 is out of scope here.

— Claude (Opus) on behalf of Cail

🤖 Generated with Claude Code

https://claude.ai/code/session_01Dj4HnkNqqPkVCeDeUGEGuz

LOCAL_TAG, the run-dir hash in /local/scratch/sp-{LOCAL_TAG}{tile}, was empty
for data campaigns, so two concurrent data campaigns over the same tiles
(two-arm A/B runs) share one node-local store whenever their same-tile
tile_shape jobs land on one node. Make the tag unconditional.

The params pin moves for tile_vignets, tile_ngmix and tile_make_cat; the pin
normalises the run-dir hash so it stays root-independent and still catches an
empty tag. New test: two campaigns get distinct stores per tile, and every
tile_shape member of a tile names the same store.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dj4HnkNqqPkVCeDeUGEGuz

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent data campaigns over the same tiles can share a node-local tile_shape store

1 participant