Image simulations workflow

.felt/blank-input-tiles-grid-4-b9a61358.md

@@ -0,0 +1,9 @@
+---
+title: Blank input tiles grid_4
+depends-on:
+    - snakemake-crash-runtimeerror-20c2564f
+created-at: 2026-07-08T17:09:48.720667191+02:00
+outcome: '12 of 37 input tiles in /n09data/hervas/skills_out/*_grid_4/images/SP_tiles are all-zero (image AND weight), identical set in all 4 shear sims: 276.280 276.281 277.282 280.280 280.282 281.281 282.281 284.279 285.279 285.280 286.279 286.280. Upstream sim-generation problem, needs regeneration by Fabian (hervas). Workaround: new tile_IDs_exclude config key (Snakefile + info.py), active in run config.yaml. Also added: log directive err_job_{tile}_{bit}.txt with tee so errors survive failure; config.yaml no longer tracked input of rule init (edits cascaded into full re-run); standard command now uses --rerun-triggers=mtime and fixed -j 4.'
+---
+
+Snakemake pipeline_all failed at job 512 (vignetmaker) for tile 281.281 in two sims: SExtractor (job 256) succeeds but finds 0 objects, writes header-only sexcat, vignetmaker raises ValueError empty/corrupt catalogue. Snakemake deletes outputs of failed jobs, so the shapepipe log vanished; real error was in tiles/<xxx>/<tile>/output/run_sp_tile_ViVi_*/vignetmaker_runner_run_1/logs/process-*.log

.felt/snakemake-crash-runtimeerror-20c2564f.md

@@ -0,0 +1,9 @@
+---
+title: 'Snakemake crash: RuntimeError can''t start new thread during pipeline_all'
+depends-on:
+    - shapepipe
+created-at: 2026-07-08T09:32:38.198900721+02:00
+outcome: 'Root cause: user thread/process limit exhaustion. ulimit -u soft=1200 hard=1280 on candide. Snakemake''s scheduler Timer thread was just the victim; the real consumers are the N parallel apptainer ShapePipe jobs. apptainer_noslurm.sh strips only SLURM/PMI/PMIX/OMPI vars, so host OMP_NUM_THREADS leaks into every container job: -j $OMP_NUM_THREADS gives N jobs x N BLAS/OpenMP threads each ~ N^2 threads. Fix (agreed, NOT yet applied): in scripts/image_sims_pipeline/Snakefile add --env OMP_NUM_THREADS=1,OPENBLAS_NUM_THREADS=1,MKL_NUM_THREADS=1,NUMEXPR_NUM_THREADS=1 to APP_CMD and SPV_CMD, and run with a fixed conservative -j 4or5 instead of $OMP_NUM_THREADS. The completed job''s log file is valid, so rerunning snakemake resumes where it stopped.'
+---
+
+Run: snakemake -s Snakefile --configfile config.yaml -j $OMP_NUM_THREADS pipeline_all in ~/v2.0/image_sims/grids on candide (node c02). Crash in snakemake 9.19.0 job_scheduler._schedule_reevalutation -> threading.Timer().start() right after tile 288.279 job 16 completed.

auxdir/CFIS/im_sims_202606/tile_numbers_1.txt

@@ -0,0 +1,40 @@
+233.293
+237.292
+238.292
+239.291
+239.292
+239.293
+240.291
+240.292
+240.293
+241.293
+242.290
+242.291
+242.292
+242.293
+243.290
+243.291
+243.292
+243.293
+244.293
+245.290
+245.291
+245.292
+246.290
+246.291
+247.289
+247.291
+247.292
+247.293
+248.288
+248.289
+248.290
+248.291
+249.289
+249.290
+249.291
+249.292
+250.289
+250.290
+250.292
+251.287

auxdir/CFIS/im_sims_202606/tile_numbers_2.txt

@@ -0,0 +1,43 @@
+251.288
+251.289
+251.291
+252.287
+252.288
+252.289
+252.291
+253.287
+253.288
+253.291
+254.286
+254.287
+254.288
+255.286
+255.288
+255.290
+256.286
+256.287
+257.285
+257.286
+257.287
+257.288
+258.287
+258.289
+259.285
+259.286
+259.287
+259.288
+259.289
+260.284
+260.285
+260.286
+260.287
+260.288
+260.289
+261.284
+261.285
+261.286
+261.287
+261.288
+262.285
+262.286
+262.287

auxdir/CFIS/im_sims_202606/tile_numbers_3.txt

@@ -0,0 +1,46 @@
+262.288
+263.283
+263.284
+263.285
+263.286
+264.283
+264.284
+264.286
+264.287
+265.284
+265.285
+265.286
+265.287
+266.282
+266.283
+266.285
+266.286
+267.283
+267.284
+267.285
+267.286
+268.282
+268.283
+268.284
+268.285
+269.281
+269.282
+269.283
+269.284
+269.286
+270.281
+270.282
+270.283
+271.281
+271.282
+271.283
+271.284
+271.285
+272.281
+272.282
+272.283
+272.285
+273.280
+273.282
+273.283
+273.284

auxdir/CFIS/im_sims_202606/tile_numbers_4.txt

@@ -0,0 +1,37 @@
+274.279
+274.280
+274.282
+274.283
+275.279
+275.280
+275.282
+275.283
+276.279
+276.280
+276.281
+276.282
+276.283
+277.279
+277.281
+277.282
+278.279
+278.281
+278.282
+279.281
+279.282
+280.280
+280.281
+280.282
+281.280
+281.281
+282.280
+282.281
+283.280
+284.279
+285.279
+285.280
+286.279
+286.280
+287.279
+288.279
+289.279

bin/shapepipe_run.py

@@ -1,3 +1,5 @@
 #!/usr/bin/env python
 from shapepipe.shapepipe_run import main
+print("MKDEBUG start")
 main()
+print("MKDEBUG end")