diff --git a/.github/workflows/deploy-image.yml b/.github/workflows/deploy-image.yml index 6e8eb1206..9e420fd2b 100644 --- a/.github/workflows/deploy-image.yml +++ b/.github/workflows/deploy-image.yml @@ -120,9 +120,10 @@ jobs: docker run --rm -e HYPOTHESIS_PROFILE=ci "$IMAGE" \ pytest -rX -m "not slow" --no-cov tests/science - # The actual test suite, run inside the shipped image — replacing the + # The rest of the test suite, run inside the shipped image — replacing the # retired conda-based suite. pytest exercises the same wheels, binaries, # and Python (3.12) that production runs on, not a parallel environment. + # tests/science ran in the step above, so it is left out here. # pyproject's addopts add `--cov=shapepipe`; COVERAGE_FILE is set to /tmp # in the image so it works on read-only filesystems too. The Hypothesis # profile is explicit here so CI always uses the deterministic, capped @@ -130,7 +131,7 @@ jobs: - name: Test — pytest suite run: | IMAGE=$(echo "${{ steps.meta.outputs.tags }}" | head -n1) - docker run --rm -e HYPOTHESIS_PROFILE=ci -e SHAPEPIPE_ON_CANDIDE=0 "$IMAGE" pytest -rX + docker run --rm -e HYPOTHESIS_PROFILE=ci -e SHAPEPIPE_ON_CANDIDE=0 "$IMAGE" pytest -rX --ignore=tests/science # tests/workflow drives the Snakefile through snakemake's API. Snakemake # is a host tool and stays out of the image (it wraps each job in the diff --git a/astra.yaml b/astra.yaml index 382669d62..bb5d7b914 100644 --- a/astra.yaml +++ b/astra.yaml @@ -239,13 +239,14 @@ analyses: On exposures SExtractor reads the split flag image, producing IMAFLAGS_ISO, which the PSF star selection requires to be zero. The multi-epoch vignet run cuts flag stamps from the same image; ngmix - gives flagged pixels weight 0 and drops epochs more than a third + gives flagged pixels weight 0 and drops epochs more than 10% flagged (shape_measurement.defect_fill, shape_measurement.epoch_masked_fraction_cut). Tiles have no flag image, and ShapePipe's tile SExtractor run requests no IMAFLAGS_ISO (detection.detection_source_mode). Neighbour pixels are masked separately, from the -1e30 markers SExtractor writes into the tile - VIGNET (shape_measurement.blend_handling). Sky-fixed masks never touch + VIGNET and the coadd segmentation map + (shape_measurement.blend_handling). Sky-fixed masks never touch pixels: a star halo changes no stamp. Values: config_exp_psfex.ini#SEXTRACTOR_RUNNER.FLAG_IMAGE = True; @@ -1251,7 +1252,8 @@ analyses: metacal_scheme, centroid_source, epoch_flux_rescaling, psf_epoch_averaging, galaxy_pixel_weights, psf_likelihood_noise, megacam_ccd_flip, defect_fill, blend_handling, - epoch_masked_fraction_cut, central_defect_veto] + defect_weighting, epoch_masked_fraction_cut, + central_defect_veto] decisions: ngmix_seed_mode: label: Per-object RNG seeded from sky position @@ -1449,49 +1451,68 @@ analyses: defect_fill: label: Image content of defect pixels before metacal rationale: >- - A defect pixel (nonzero instrument flag, zero exposure weight or - invalid background RMS) gets weight 0 in prepare_ngmix_weights. Its - image value still matters: ngmix's metacal deconvolves, shears and - reconvolves an InterpolatedImage of the whole image and copies the - weights through, so a zero-weight pixel's content spreads into the - weighted pixels within about a PSF width. DES's ngmixer fills for - that reason: "it may be important for codes that take moments or use - FFTs". In the committed default (`BLEND_HANDLING = noisefill`), - masked pixels are replaced with independent noise at the per-pixel - background RMS when supplied, or the stamp's robust noise scale - otherwise. With `BLEND_HANDLING = uberseg`, the image is left - untouched and weights are zeroed on defects and neighbour-side - pixels. [LINT] the prepare_ngmix_weights docstring says noisefill - keeps the weight of filled pixels (the code zeroes it), and the - ngmix_runner comment says noisefill fills neighbour pixels (it fills - flagged pixels and leaves neighbours untouched). The committed fill - uses the unsymmetrized defect set: DES symmetrized its masks, but - four-fold symmetrization quadruples m and still leaves an additive c1 - (symmetrized_4fold_noise). The cost of not symmetrizing, a hole in - the galaxy light, is bounded by central_defect_veto. Noise stays the - default until a survey A/B against interpolation. + [HARDCODED] A defect is a pixel with no usable data: a nonzero + instrument flag, zero exposure weight, an invalid background RMS, + or off the tile. prepare_ngmix_weights gives it weight 0, and this + decision sets the image value it holds. That value matters because + metacal deconvolves, shears and reconvolves the whole stamp, which + spreads a zero-weight pixel's content into its weighted neighbours + within about a PSF width; DES fills defects for the same reason. + Short defect runs are interpolated from the surrounding good + pixels and the rest are filled with noise, under every + BLEND_HANDLING. Interpolation is used because an epoch can then + keep a defect as close as 7 px to the object, whereas a 0.7 arcsec + galaxy through a 0.9 arcsec PSF needs noise-filled defects at least + 13 px away. Only the defect pixels are filled, not a four-fold + symmetrized mask as in DES, since symmetrizing quadruples m and + leaves an additive c1 (symmetrized_4fold_noise); the hole this + leaves near the galaxy is bounded by central_defect_veto. In the + tile VIGNET, SExtractor writes -1e30 beyond the tile edge and over + every above-threshold pixel outside the object's pre-CLEAN + footprint. Rows and columns that are entirely -1e30 and start at + the stamp border are off-tile, hence defects (the epoch has the + object's light there); the other -1e30 pixels on another object's + segmentation footprint are neighbours, handled by blend_handling. + The rest (sub-MINAREA noise peaks, the object's own CLEAN-merged + fragments) are not defects either: the epoch stamps hold real + light there, so they keep their weight and image. Values: VIGNETMAKER_RUNNER_RUN_1.MASKING = False; VIGNETMAKER_RUNNER_RUN_2.MASKING = False. - default: noise + default: interpolate options: - noise: - label: Independent noise on the unsymmetrized defect set - description: >- - Masked pixels get independent noise using the per-pixel - background RMS when supplied, or the stamp's robust noise scale - otherwise; their inverse-variance weights stay zero. Metacal's - fixnoise noise image covers every pixel. - insights: [mask_metacal_acts_on_whole_stamp, mask_bad_column_symmetrize] interpolate: label: Interpolate short bounded runs; noise-fill the rest description: >- - Not implemented. PR #916 measures c1 = -1.3e-3 for a column - 8 px from a 0.5 arcsec galaxy and +2e-6 when the weight is also - zeroed on its quarter-turn orbit. For a 3-px bleed 6 px from that - galaxy, PR #916 measures m11 = +0.89% when the fill is - symmetrized, against +0.19% otherwise. + Row or column runs of at most MAX_INTERPOLATED_RUN = 3 defect + pixels whose neighbours beyond both ends are kept pixels (on + the stamp, not defects, and not noisefill neighbours) are + Clough-Tocher interpolated from the kept pixels within + SUPPORT_RADIUS = 4 px, with one triangulation for the image + and the metacal noise image. Other defects + (wider holes, edge bands, runs ending on a noisefill neighbour) + get independent noise at the per-pixel background RMS when + supplied, or the stamp's robust noise scale otherwise. The + interpolated pixels are weighted by defect_weighting. For a + 3-px bleed 6 px from a 0.5 arcsec galaxy, PR #916 measures + m11 = +0.89% when the fill is symmetrized, against +0.19% + otherwise. insights: [mask_interpolate_with_noise, mask_sharp_edges_ring] + noise: + label: Independent noise on every defect + description: >- + Every defect pixel gets independent noise; their + inverse-variance weights stay zero. + excluded: true + excluded_reason: >- + A noise-filled defect leaves a hole in the object's light that + metacal shears and the sky does not. Columns, 3-px bleeds and + single pixels then need a 10 px veto on 0.3 and 0.5 arcsec + galaxies through a 0.7 arcsec PSF, and 13 px on a 0.7 arcsec + galaxy through a 0.9 arcsec PSF (a 3-px bleed at 10 px gives + m11 = -6.5%). Interpolated, the same defects pass from 7 px on + all three (PR #922 bias scan). + insights: [mask_metacal_acts_on_whole_stamp, mask_bad_column_symmetrize] symmetrized_4fold_noise: label: Four-fold-symmetrized defect set (M | rot90 | rot180 | rot270), then noise fill description: Not implemented. @@ -1504,11 +1525,11 @@ analyses: with symmetrization against 0.7e-4 without it. insights: [mask_bad_column_symmetrize, mask_des_defect_practice, mask_fixed_orientation] raw: - label: "No fill: raw defect values (BLEND_HANDLING = uberseg)" + label: "No fill: raw defect values" description: >- - With BLEND_HANDLING = uberseg, defect pixels keep weight 0 but - their image values stay untouched in the image that metacal - deconvolves, shears and reconvolves. + Defect pixels keep weight 0 but their image values stay + untouched in the image that metacal deconvolves, shears and + reconvolves. excluded: true excluded_reason: >- Metacal acts on every pixel regardless of weight, so raw defects @@ -1518,12 +1539,15 @@ analyses: blend_handling: label: Neighbour treatment before metacal rationale: >- - Covers only pixels shared with a neighbour; defect_fill is coupled to - it through BLEND_HANDLING. noisefill (the default; the committed - config sets no key) leaves neighbours fully weighted and untouched. - uberseg zeroes the weight of pixels nearer a neighbour's coadd - segmentation footprint than the target's (DILATE_NEIGHBOUR, default 1) - and leaves the image untouched, as official uberseg does: + Covers only pixels shared with a neighbour; defect_fill does not + depend on it: defects are filled the same way under every + BLEND_HANDLING, and the epoch cuts never count neighbour pixels. + The committed choice is uberseg (the workflow's default + blend_handling), which ignores the -1e30 neighbour markers in the + tile VIGNET and zeroes the weight of pixels nearer a neighbour's + coadd segmentation footprint than the target's (DILATE_NEIGHBOUR, + default 1), leaving the neighbour side's image untouched, as + official uberseg does: esheldon/meds get_uberseg returns a weight map (a nearest-segment-pixel Voronoi split). DES Y1's fiducial metacal and the last Y3 config ran on uberseg-weighted stamps with raw neighbour @@ -1540,24 +1564,39 @@ analyses: with simulations (mask_des_y1_uberseg_only, mask_blend_bias_detection). Noise-filling the neighbour side would instead cut the target's own light along an unsheared boundary, a - sharp edge that rings in the FFTs. The recommended comparison arm is - uberseg (weight-only), with defect_fill held equal across arms. - default: none + sharp edge that rings in the FFTs. The alternative, noisefill + (ngmix's own default, and the workflow's blend_handling: + noisefill), gives weight 0 to the pixels marked -1e30 in the tile + VIGNET (about 93% of neighbour-footprint pixels on a simulated + tile), excluding the off-tile rows and columns, which are defects, + and replaces them with noise; unmarked neighbour pixels keep their + weight and light. Under both handlings the neighbours are defined + by the coadd segmentation map: a marker counts only on a footprint + labelled neither 0 nor the target. SExtractor's MASK_TYPE CORRECT + also marks sub-MINAREA sky blobs and fragments CLEAN merged into + the target, which the seg labels as sky or as the target; noise- + filling those removed the target's own light (15% of + NEIGHBOUR_FLAG = 0 objects differed between the arms on 25 tiles). + tile_detect therefore writes SEG_VIGNET under both handlings. + Uberseg reads no marker, so its output is unchanged. It is + the comparison arm, with defect_fill held equal across arms. + default: uberseg options: - none: - label: No neighbour treatment (BLEND_HANDLING = noisefill) + noisefill: + label: Noise-fill the marked neighbour pixels (BLEND_HANDLING = noisefill) description: >- - Neighbour pixels keep their full weight and image values, so the - fit sees all neighbour light, which biases shapes toward - neighbours (Jarvis et al. 2016); this masks less than even their - plain segmentation map. A candidate cause of the FLAGS=2 B-modes + Pixels marked -1e30 in the tile VIGNET get weight 0 and noise. + The fill stops at the marked footprint, so unmarked neighbour + pixels and the neighbour's wings keep their weight and light, + which biases shapes toward neighbours (Jarvis et al. 2016). A + candidate cause of the FLAGS=2 B-modes investigated in #814. insights: [mask_uberseg_neighbour_bias] uberseg: label: UberSeg, weight-only description: >- - As in DES Y1/Y3. Needs the coadd segmentation stamp - (SEG_VIGNET_PATH). DILATE_NEIGHBOUR absorbs the coadd-vs-epoch + As in DES Y1/Y3. Needs the coadd segmentation stamp (the tile + catalogue's SEG_VIGNET). DILATE_NEIGHBOUR absorbs the coadd-vs-epoch overlay offset: ShapePipe reuses one coadd seg stamp for every epoch where MEDS reprojects it. insights: [mask_uberseg_weight_only, mask_uberseg_neighbour_bias, mask_des_y1_uberseg_only, mask_blend_bias_detection] @@ -1578,34 +1617,117 @@ analyses: the ~2% uberseg-only bias in DES Y1 simulations, although Sheldon et al. 2020 found similar blend biases with and without MOF. insights: [mask_des_y1_uberseg_only, mask_blend_bias_detection] + defect_weighting: + label: Weighting of interpolated defects (DEFECT_WEIGHTING) + rationale: >- + ngmix fits a Gaussian to a profile that is not one, so a zero-weight + hole on one side of the object pulls the fit (a spin-2 term), while + interpolated light at full weight is a smooth guess the fit trusts + as data. DEFECT_WEIGHTING (prepare_ngmix_weights, + defect_weighting_masks; the workflow's `defect_weighting`) chooses + between them for the interpolated defects only: noise-filled pixels + and noisefill neighbours have no light to weight and keep weight 0, + and the uberseg neighbour side keeps weight 0 under every option. + Under every option the metacal noise image is interpolated in + fixnoise's quarter-turned frame (ngmix turns it k=1 before shearing + and k=3 after), by the operator that interpolates the science image. + Interpolated in the detector frame, its interpolated pixels sit a + quarter turn from the science image's, and the correlated noise of + the interpolant survives fixnoise as an additive bias that grows as + S/N falls: a 3-px bleed 7 px from a 0.7 arcsec galaxy through a 0.9 + arcsec PSF gives c1 = 2.7e-3 +/- 0.5e-3 at S/N 30 under + fourfold_zero, and 4.6e-4 or less on every galaxy with the noise + image turned. Measured with paired, antithetic-noise seeds on 51-px + stamps, galaxies of half-light radius 0.3 and 0.5 arcsec through a + 0.7 arcsec PSF and 0.7 arcsec through a 0.9 arcsec PSF, round and + (0.05, 0.02) PSFs, defects 7-12 px from the centre (sym-verify + harness, PR #922). The rotation is about the stamp centre, where the + extractor places the object to within half a pixel per axis. + Values: + ngmix.py::DEFECT_WEIGHTING = "fourfold_zero". + default: fourfold_zero + options: + fourfold_zero: + label: Interpolate; zero the weight on the defect and its three quarter turns + description: >- + The quarter-turn copies keep their light. An interpolated column + 8 px from the galaxy gives c1 = -1.3e-3 with a one-sided hole and + +2e-6 symmetrized (PR #916, noiseless). + insights: [mask_bad_column_symmetrize] + des_y6: + label: "DES Y6 (pizza-cutter): mask OR its quarter turn, interpolated at full weight" + description: >- + The defect mask is ORed with its np.rot90 (k=1) copy about the + stamp centre and the union is interpolated, as pizza-cutter + symmetrizes the bit mask and weight of each SE slice before + interpolating; every interpolated pixel keeps full weight. + Departures forced by per-object stamps: the rotation is about + the object, not a cell centre; there is no mfrac cut, the epoch + cuts (10% masked fraction, central veto) stand in for it; the + noise image is interpolated in fixnoise's turned frame rather + than in the science frame; and copies that land on a + noise-filled pixel stay noise-filled. Without the turned noise + image it gives c1 = -5.5e-3 +/- 1.8e-3 for the bleed above at + S/N 30. + insights: [mask_des_defect_practice, mask_bad_column_symmetrize] + hole: + label: Interpolate; zero the weight on the defect only + description: >- + The one-sided hole pulls the fit: c1 = -5.7e-3 for the bleed + above, noiseless. + full: + label: Interpolate at full weight, no symmetrization + description: >- + c1 = 6.8e-4 for the bleed above through the elliptical PSF, + noiseless; 9.5e-3 at S/N 30 without the turned noise image. central_defect_veto: label: Per-epoch veto on a defect near the stamp centre rationale: >- - Not on develop; implemented on feat/defect-fill-veto (7777181b). - There an epoch is dropped when a defect pixel lies strictly closer - to the stamp centre than its fill's radius, beside the - masked-fraction cut in the epoch loop. The veto reads only the - defect mask, so it selects on nothing shear-responsive; for the same - reason the radii are fixed rather than scaled by galaxy size: 10 px - for noise-filled pixels (EPOCH_CENTRAL_DEFECT_RADIUS) and 7 px for - interpolated ones (EPOCH_INTERPOLATED_DEFECT_RADIUS, - feat/defect-interpolation). Calibrated on 51-px stamps at known RMS - and high S/N, through a round and a (0.05, 0.02) elliptical 0.7 - arcsec PSF, on galaxies of half-light radius 0.3 and 0.5 arcsec, and - for interpolation also 0.7 and 0.9 arcsec. Known limits: at 10 px, - wide defects through the elliptical PSF sit at the 1% bound (m11 = - -0.98%; -0.24% at 11 px), and noise fill needs 14 px for 0.7 and 0.9 - arcsec galaxies. Measured on feat/defect-fill-veto and - feat/defect-interpolation. - default: disabled + [HARDCODED] An epoch is dropped when a defect pixel lies strictly + closer to the stamp centre than the radius of its fill: 13 px for + noise-filled pixels (EPOCH_CENTRAL_DEFECT_RADIUS) and 7 px for + interpolated ones (EPOCH_INTERPOLATED_DEFECT_RADIUS). The veto + reads the same interpolated set as the fill, including the runs + that end on a noisefill neighbour and so are noise-filled. The tile + VIGNET's -1e30 neighbour markers are not defects: all epochs share + the tile VIGNET, so a neighbour inside the radius would drop every + epoch. Off-tile pixels are defects, so an object near the tile edge + is vetoed. The veto reads only masks, so it selects on nothing + shear-responsive; for the same reason the radii are fixed rather + than scaled by galaxy size. Each radius is the smallest at which + the defects it keeps give |m11|, |m22| < 1% and |c1|, |c2| < 5e-4, + on 51-px stamps at known RMS and high S/N, through a round and a + (0.05, 0.02) elliptical 0.7 arcsec PSF, on galaxies of half-light + radius 0.3 and 0.5 arcsec, and 0.7 arcsec through a 0.9 arcsec + PSF. That largest galaxy sets both radii: 3-px bleeds the 7 px, + noise-filled holes the 13 px (a 4-column cluster at 13 px gives + m11 = -0.67% +/- 0.05%, c1 = 7e-5). Guarded by + tests/science/test_defect_recovery.py. + Values: + EPOCH_CENTRAL_DEFECT_RADIUS = 13; + EPOCH_INTERPOLATED_DEFECT_RADIUS = 7. + default: fixed_radii options: - disabled: - label: No central veto (committed code) fixed_radii: - label: Fixed radii, 10 px for noise-filled and 7 px for interpolated defects - description: >- - Implemented on feat/defect-fill-veto and feat/defect-interpolation, - not on develop. + label: Fixed radii, 13 px for noise-filled and 7 px for interpolated defects + noise_fill_10px: + label: 10 px for noise-filled defects + excluded: true + excluded_reason: >- + Enough for 0.3 and 0.5 arcsec galaxies through a 0.7 arcsec + PSF, not for a 0.7 arcsec galaxy through a 0.9 arcsec PSF. A + noise-filled 4-column cluster (8% of the stamp, so the 10% + masked-fraction cut keeps it) gives m11 = -6.6% +/- 0.3% and + c1 = -4.3e-3 at 10 px, -3.6% and -1.6e-3 at 11 px, -1.7% and + -3.7e-4 at 12 px, and first passes at 13 px (PR #922, 6 seeds). + disabled: + label: No central veto + excluded: true + excluded_reason: >- + A noise-filled column 8 px from a 0.5 arcsec galaxy gives + m11 = -3.7% and c1 = -3.6e-3; a 3-px bleed + three pixels inside the 7 px radius, interpolated, gives + |c| > 1e-3 on a 0.3 arcsec galaxy. size_scaled_radius: label: Veto radius scaled by galaxy size excluded: true @@ -1615,37 +1737,59 @@ analyses: epoch_masked_fraction_cut: label: Per-epoch masked-fraction cut rationale: >- - [HARDCODED] on develop, an epoch whose stamp has more than 1/3 of its - pixels flagged (any nonzero flag bit, including the tile-coverage bit - 2**10 set where the tile vignet is off-image) is dropped from the - multi-epoch fit; an object with no surviving epoch has no shape. - Zero-weight and invalid-RMS pixels are not counted. On - feat/defect-fill-veto the cut counts the raw, unsymmetrized defect - set (flagged, zero-weight and invalid-RMS pixels) against - EPOCH_MASKED_FRACTION_CUT, default 1/3. Before the cut, an epoch is - dropped silently if its galaxy stamp is all zeros or its - background-subtracted sigma_mad is not positive. DES was stricter: - Y1 rejected any epoch with a masked or zero-weight pixel, or with - its central 4-pixel region masked; Y3 cut at 10% raw zero-weight - pixels; Y6 drops images more than 10% missing and keeps objects at - mfrac < 0.1 (mask_multi_epoch_drop). UNIONS has fewer epochs than - DES, so the cost in effective number density has to be measured, - not assumed. A defect near the centre is handled separately - (central_defect_veto). - default: one_third + [HARDCODED] An epoch whose stamp has more than + EPOCH_MASKED_FRACTION_CUT = 10% of its pixels in the raw, + unsymmetrized defect set is dropped from the multi-epoch fit: + flagged pixels (any nonzero exposure flag bit), zero-weight pixels + (off-tile pixels included) and invalid-RMS pixels, the set + defect_fill fills. On a 51-px stamp an object within about 20 px + of the tile edge fails the cut. Neighbour footprints are not + counted: every epoch shares the tile VIGNET, so a large neighbour + would drop them all. An object with no surviving epoch has no + shape. Before the cut, an epoch is dropped silently if its galaxy + stamp is all zeros or its background-subtracted sigma_mad is not + positive. After it, an epoch whose ngmix observation raises while + being built (for example a non-finite defect interpolant) is + dropped, logged with its tile, object and epoch, and counted in + NGMIX_N_EPOCH_FAILED; the object is fitted on its other epochs. The cut is not a bias control; the central veto is + (central_defect_veto). Defects the veto keeps, covering up to 58% + of the stamp (two-sided bands, corner Ls and frames 12 px or more + from the centre), give |m11|, |m22| <= 0.05% and |c| < 7e-5 on 0.3 + and 0.5 arcsec galaxies through a round and an elliptical 0.7 + arcsec PSF (PR #922). A one-sided edge band reaching the 13 px + radius (25%) gives |m| <= 0.01% on those galaxies through the + round PSF and m11 = -0.64% on a 0.7 arcsec galaxy through a 0.9 arcsec PSF. The 10% cut is chosen for comparability with DES + and as a margin against real-data effects the simulations do not + model, and its measured cost is small. On real tile 202.301 + (interior shape-proxy objects), it drops 5.08% of epochs, leaves + 0.55% of objects with no epoch and costs 13.7% of objects at least + one epoch. Every epoch it removes beyond a 1/3 cut is a CCD-edge or + off-tile band whose nearest defect is 10-20 px from the object; + bad-column clusters cover 2-8% of a stamp, and only 4 epochs + exceed 10% because of them. DES Y1 rejected any epoch with a + masked or zero-weight pixel, or with its central 4-pixel region + masked; Y3 cut at 10% raw zero-weight pixels; Y6 drops images more + than 10% missing and keeps objects at mfrac < 0.1 + (mask_multi_epoch_drop). + default: ten_percent options: - one_third: - label: 1/3 of the stamp in the defect set - description: >- - Drop an epoch only if more than 1/3 of the stamp pixels are - defects (on develop, flagged pixels). ten_percent: label: 10% (DES Y3 / Y6) description: >- - Not a default; EPOCH_MASKED_FRACTION_CUT = 0.1 on - feat/defect-fill-veto. Matches DES Y3 max_zero_weight_frac and - Y6 max_masked_fraction. + Drop an epoch if more than 10% of the stamp pixels are defects, + matching DES Y3 max_zero_weight_frac and Y6 + max_masked_fraction. insights: [mask_multi_epoch_drop] + one_third: + label: 1/3 of the stamp in the defect set + excluded: true + excluded_reason: >- + Equally unbiased in simulations, since far-field defects bias + nothing measured up to 58% of the stamp, and it keeps more + epochs (3.38% dropped on tile 202.301 against 5.08% at 10%). + But the extra epochs it keeps are CCD-edge and off-tile bands + 10-20 px from the object, where effects the simulations do not + model are likeliest, and it departs from the DES convention. any_masked: label: Any masked pixel (DES Y1) description: >- @@ -1906,7 +2050,7 @@ analyses: (Sheldon & Huff 2017). DES Y6 drops input images more than 10% missing and cuts objects at masked fraction mfrac < 0.1, which its image simulations show avoids shear calibration bias. ShapePipe's - cut is 1/3. + cut is also 10%. created_at: "2026-09-26T00:00:00Z" derived: true evidence: diff --git a/scripts/validation/masked_pixels/bias_vs_distance.py b/scripts/validation/masked_pixels/bias_vs_distance.py new file mode 100644 index 000000000..9d64abc39 --- /dev/null +++ b/scripts/validation/masked_pixels/bias_vs_distance.py @@ -0,0 +1,221 @@ +"""Shear bias from one detector defect against its distance from the object, +under the production defect treatment and under noise fill. + +Production interpolates the defects ``interpolable_defects`` selects +(columns, 3-px bleeds, finite bleeds, single pixels) and noise-fills the +rest (4-column clusters, edge bands). The counterfactual noise-fills every defect, by making +``interpolated_defects`` select none. The recovery is the full-matrix +metacal measurement of ``tests/helpers/defect_response``; the figure plots +the worse axis of |m| and |c| with the veto radii and the bounds +|m| < 1%, |c| < 5e-4. + +Run from the shapepipe checkout root inside the shapepipe container: + PYTHONPATH=src:. python scripts/validation/masked_pixels/bias_vs_distance.py \ + scan OUT.json NPROC [KIND,KIND...] + PYTHONPATH=src:scripts/validation/masked_pixels \ + python scripts/validation/masked_pixels/bias_vs_distance.py \ + plot OUT.png IN.json [IN.json ...] +""" +import json +import sys + +import numpy as np + +N, CENTRE = 51, 25 +DISTANCES = range(3, 17) +SEEDS = range(6) +GALAXIES = [(0.3, 0.7), (0.5, 0.7), (0.7, 0.9)] # (hlr, PSF FWHM) arcsec +INTERPOLATED = ("column", "bleed", "finite_bleed", "pixel") +KINDS = INTERPOLATED + ("cluster4", "edge") +BOUND_M, BOUND_C = 0.01, 5e-4 + + +def geometry(kind, distance): + """A defect whose nearest pixel is ``distance`` px from the stamp + centre (the geometries of tests/science/test_defect_recovery.py).""" + bad = np.zeros((N, N), dtype=bool) + near = CENTRE + distance + if kind == "pixel": + bad[CENTRE, near] = True + elif kind == "column": + bad[:, near] = True + elif kind == "bleed": + bad[:, near:near + 3] = True + elif kind == "finite_bleed": + bad[CENTRE - 5:CENTRE + 6, near:near + 3] = True + elif kind == "cluster4": + bad[:, near:near + 4] = True + elif kind == "edge": + bad[:, near:] = True + return bad + + +def one(job): + kind, fill, distance, hlr, psf = job + from shapepipe.modules.ngmix_package import ngmix as ngm + from tests.helpers.defect_response import defect_response + + if fill == "noise": + ngm.interpolated_defects = ( + lambda defect, neighbour, blend: np.zeros_like(defect)) + row = dict(kind=kind, fill=fill, d=distance, hlr=hlr, psf=psf) + try: + r = defect_response(geometry(kind, distance), hlr=hlr, psf=psf, + seeds=SEEDS) + row.update(m=r["m"], m_err=r["m_err"], c=r["c"], c_err=r["c_err"]) + except Exception as e: # noqa: BLE001 + row["error"] = repr(e) + return row + + +def scan(out_path, nproc, kinds=KINDS): + from multiprocessing import get_context + + jobs = [(k, f, d, h, p) for h, p in GALAXIES for d in DISTANCES + for k in kinds + for f in (("production", "noise") if k in INTERPOLATED + else ("production",))] + # spawn: each worker imports ngmix afresh, so the noise-fill patch + # cannot leak into a production job. + with get_context("spawn").Pool(nproc, maxtasksperchild=1) as pool: + rows = pool.map(one, jobs, chunksize=1) + with open(out_path, "w") as f: + json.dump(rows, f, indent=1) + + +def worst(row, q): + return max(abs(x) for x in row[q]) + + +def smallest_passing(rows): + """Smallest distance from which every larger distance passes both + bounds (None if the largest fails).""" + rows = sorted(rows, key=lambda r: r["d"]) + passing = None + for r in reversed(rows): + if "error" in r or worst(r, "m") >= BOUND_M or worst(r, "c") >= BOUND_C: + break + passing = r["d"] + return passing + + +def plot(out_path, *in_paths): + import matplotlib + + matplotlib.use("Agg") + import matplotlib.pyplot as plt + from matplotlib.lines import Line2D + + from figstyle import AQUA, BLUE, INK, INK2, ORANGE, VIOLET + from shapepipe.modules.ngmix_package.ngmix import ( + EPOCH_CENTRAL_DEFECT_RADIUS as RN, + EPOCH_INTERPOLATED_DEFECT_RADIUS as RI, + EPOCH_MASKED_FRACTION_CUT as FRAC_CUT, + ) + + rows = [r for path in in_paths for r in json.load(open(path))] + grid, fail, bad_ink = "#e4e3df", "#f6e3dc", "#a8462a" + GOLD = "#c49a1c" + series = { + "column": ("bad column", BLUE), + "bleed": ("3-px bleed", ORANGE), + "finite_bleed": ("finite 3-px bleed (11 rows)", VIOLET), + "pixel": ("single pixel", AQUA), + "edge": ("edge band (noise-filled)", INK), + "cluster4": ("4-column cluster (noise-filled)", GOLD), + } + plt.rcParams.update({"font.size": 10.5, "axes.edgecolor": INK2, + "xtick.color": INK2, "ytick.color": INK2, + "axes.labelcolor": INK}) + fig, axs = plt.subplots( + 2, len(GALAXIES), figsize=(13, 7.4), dpi=150, sharex=True, + sharey="row", gridspec_kw=dict(hspace=0.1, wspace=0.05)) + for j, (hlr, psf) in enumerate(GALAXIES): + for i, (q, bound, scale, ylabel) in enumerate([ + ("m", BOUND_M, 100, "worse-axis |m| [%]"), + ("c", BOUND_C, 1, "worse-axis |c|")]): + ax = axs[i, j] + b = bound * scale + ax.axhspan(b, 1e4, color=fail, zorder=0, lw=0) + ax.axhline(b, color=bad_ink, lw=0.9, zorder=1) + for r, txt, ha, dx in ( + (RI, f"interpolated\nveto {RI:g} px", "right", -0.2), + (RN, f"noise-fill\nveto {RN:g} px", "left", 0.2)): + ax.axvline(r, color=INK2, ls=":", lw=1.3, zorder=1) + if i == 0: + ax.text(r + dx, 8e2, txt, color=INK2, fontsize=8.5, + va="top", ha=ha, linespacing=1.1) + for kind, (label, col) in series.items(): + for fill, ls, lw, alpha in (("production", "-", 2.0, 1.0), + ("noise", (0, (3, 2)), 1.4, 0.8)): + sel = sorted( + (r for r in rows if r["kind"] == kind + and r["fill"] == fill and r["hlr"] == hlr + and r["psf"] == psf and "error" not in r), + key=lambda r: r["d"]) + if not sel: + continue + d = np.array([r["d"] for r in sel]) + v = np.array([worst(r, q) * scale for r in sel]) + z = 3 if fill == "production" else 2 + ax.plot(d, v, color=col, ls=ls, lw=lw, alpha=alpha, + zorder=z) + if fill != "production": + continue + cut = np.array([geometry(kind, x).mean() > FRAC_CUT + for x in d]) + ax.plot(d[~cut], v[~cut], "o", color=col, ms=3.5, + zorder=z) + ax.plot(d[cut], v[cut], "o", mfc="white", mec=col, + mew=1.1, ms=4, zorder=z) + ax.set_yscale("log") + ax.set_ylim((1e-3, 1e3) if q == "m" else (1e-6, 0.5)) + ax.grid(axis="y", color=grid, lw=0.6, which="major") + for s in ("top", "right"): + ax.spines[s].set_visible(False) + ax.set_xticks(range(4, 17, 2)) + if j == 0: + ax.set_ylabel(ylabel) + ax.text(16.5, b * 1.3, "|m| > 1%" if q == "m" else + "|c| > 5e-4", color=bad_ink, fontsize=9, + ha="right", va="bottom") + if i == 0: + ax.set_title(f'{hlr}″ galaxy, {psf}″ PSF', color=INK, + fontsize=11, loc="left") + fig.supxlabel("distance of the defect's nearest pixel from the object " + "centre [px]", fontsize=10.5, color=INK, y=0.045) + fig.suptitle("Shear bias from one defect: production treatment (solid) " + "vs noise fill (dashed); 6 seeds, worse of axes 1, 2", + color=INK2, fontsize=10, y=0.965) + handles = [Line2D([], [], color=c, lw=2, marker="o", ms=3.5, label=l) + for l, c in series.values()] + handles += [ + Line2D([], [], color=INK2, lw=2, marker="o", ms=3.5, + label="production (narrow defects interpolated)"), + Line2D([], [], color=INK2, lw=1.4, ls=(0, (3, 2)), + label="noise fill (not used for narrow defects)"), + Line2D([], [], color=INK2, lw=0, marker="o", mfc="white", mec=INK2, + ms=4, label=f"epoch dropped by the {FRAC_CUT:.0%} masked-fraction cut"), + ] + fig.legend(handles=handles, frameon=False, fontsize=9.5, ncol=3, + loc="lower center", bbox_to_anchor=(0.5, -0.075)) + fig.savefig(out_path, bbox_inches="tight") + + +if __name__ == "__main__": + if sys.argv[1] == "scan": + scan(sys.argv[2], int(sys.argv[3]), + *([tuple(sys.argv[4].split(","))] if len(sys.argv) > 4 else [])) + elif sys.argv[1] == "plot": + plot(sys.argv[2], *sys.argv[3:]) + elif sys.argv[1] == "summary": + rows = [r for path in sys.argv[2:] for r in json.load(open(path))] + for hlr, psf in GALAXIES: + for kind in KINDS: + for fill in ("production", "noise"): + sel = [r for r in rows if r["kind"] == kind + and r["fill"] == fill and r["hlr"] == hlr + and r["psf"] == psf] + if sel: + print(f"{hlr}/{psf} {kind:13s} {fill:10s} " + f"passes from {smallest_passing(sel)} px") diff --git a/scripts/validation/masked_pixels/defect_gallery.py b/scripts/validation/masked_pixels/defect_gallery.py new file mode 100644 index 000000000..f894d5c06 --- /dev/null +++ b/scripts/validation/masked_pixels/defect_gallery.py @@ -0,0 +1,75 @@ +"""One real DR6 epoch per defect type: the raw epoch stamp, then the image +and weights ngmix receives (BLEND_HANDLING = noisefill, the default). + +Each epoch is an npz written by ``dr6_epochs.py cut``. Run from the +shapepipe checkout root inside the shapepipe container: + PYTHONPATH=src:scripts/validation/masked_pixels \ + python scripts/validation/masked_pixels/defect_gallery.py OUT.png \ + "bad column=ep_4202.npz" "3-column cluster=ep_28647.npz" ... +""" +import sys + +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np +from matplotlib.lines import Line2D + +from figstyle import (AQUA, BLUE, INK, ORANGE, WEIGHT_LEGEND, clean, outline, + show_image, show_weight) +from shapepipe.modules.ngmix_package.ngmix import ( + defect_mask, + interpolated_defects, + prepare_ngmix_weights, +) + +COLS = ["raw stamp", "image ngmix sees", "weight"] + + +def main(out_path, rows): + plt.rcParams.update({"font.size": 10}) + fig, axs = plt.subplots(len(rows), 3, figsize=(7.4, 2.35 * len(rows)), + dpi=150, + gridspec_kw=dict(hspace=0.06, wspace=0.05)) + for i, (kind, path) in enumerate(rows): + ep = dict(np.load(path, allow_pickle=True)) + rms = float(np.median(ep["bkg_rms"])) + stretch = dict(vmax=120.0 * rms, soft=3.0 * rms) + defect = defect_mask(ep["weight"], ep["flag"], ep["bkg_rms"]) + interp = interpolated_defects(defect, ep["neighbour"], "noisefill") + img, w, _ = prepare_ngmix_weights( + ep["gal"], ep["weight"], ep["flag"], np.random.RandomState(1), + bkg_rms=ep["bkg_rms"], neighbour=ep["neighbour"], + ) + show_image(axs[i, 0], ep["gal"], **stretch) + outline(axs[i, 0], ep["neighbour"], BLUE, lw=0.9) + outline(axs[i, 0], defect, INK, lw=0.9) + show_image(axs[i, 1], img, **stretch) + outline(axs[i, 1], interp, AQUA, lw=0.9) + outline(axs[i, 1], defect & ~interp, ORANGE, lw=0.9) + show_weight(axs[i, 2], w, interp, defect | ep["neighbour"]) + axs[i, 0].set_ylabel( + f"{kind}\n{ep['label']}\n{ep['source']}", color=INK, fontsize=8, + linespacing=1.15, + ) + for j, t in enumerate(COLS): + axs[0, j].set_title(t, color=INK, fontsize=10) + for ax in axs.flat: + clean(ax) + + handles = [ + Line2D([], [], color=INK, lw=1.6, label="defect"), + Line2D([], [], color=BLUE, lw=1.6, + label="neighbour (noise-filled, zero weight)"), + Line2D([], [], color=AQUA, lw=1.6, label="defect, interpolated"), + Line2D([], [], color=ORANGE, lw=1.6, label="defect, noise-filled"), + ] + WEIGHT_LEGEND + fig.legend(handles=handles, loc="upper center", ncol=2, frameon=False, + fontsize=8.5, handlelength=1.4, + bbox_to_anchor=(0.5, axs[-1, 0].get_position().y0 - 0.005)) + fig.savefig(out_path, bbox_inches="tight") + + +if __name__ == "__main__": + main(sys.argv[1], [a.split("=", 1) for a in sys.argv[2:]]) diff --git a/scripts/validation/masked_pixels/dr6_epochs.py b/scripts/validation/masked_pixels/dr6_epochs.py new file mode 100644 index 000000000..6f1ecb068 --- /dev/null +++ b/scripts/validation/masked_pixels/dr6_epochs.py @@ -0,0 +1,185 @@ +"""Cut real DR6 epoch stamps into the arrays prepare_ngmix_weights takes, +and find epochs with each kind of defect. + +Inputs: + * one MegaCam exposure with its weight and flag maps (vos:cfis/pitcairn, + vos:cfis/weights, vos:cfis/flags), e.g. 2086786p.fits.fz; + * the DR6 tile catalogue with VIGNET (-1e30 neighbour and off-tile + markers) and SEG_VIGNET columns, as read_ext_sexcat writes it from + CFIS..r.cat and CFIS..r.seg.fits.fz + (sexcat--.fits). + +The epoch stamps are cut as vignetmaker does: 51x51 around the rounded +pixel of the object's (XWIN_WORLD, YWIN_WORLD) in the CCD's own WCS, CCDs +numbered as split_exp names them (HDU index - 1). The tile VIGNET and seg +stamp are MegaCam-flipped onto the epoch and split into neighbour and +off-tile pixels (split_tile_markers); the off-tile pixels get zero weight, +as prepare_postage_stamps gives them. The background is a sigma-clipped +median over the clean pixels of a 257x257 box around the object, and the +background RMS is constant over the stamp; the pipeline takes both from +SExtractor's maps instead. Only the survey flag and weight maps mark +defects here: the pipeline's mask step adds star halos and borders. + +Run from the shapepipe checkout root inside the shapepipe container: + PYTHONPATH=src python scripts/validation/masked_pixels/dr6_epochs.py \ + scan EXP WEIGHT FLAG SEXCAT > scan.txt # one line per epoch + PYTHONPATH=src python scripts/validation/masked_pixels/dr6_epochs.py \ + cut EXP WEIGHT FLAG SEXCAT NUMBER OUT.npz +""" +import re +import sys + +import fitsio +import numpy as np +from astropy.io import fits +from astropy.stats import sigma_clipped_stats +from astropy.wcs import WCS +from scipy.ndimage import find_objects, label + +from shapepipe.modules.ngmix_package.ngmix import ( + Ngmix, + defect_mask, + split_tile_markers, +) +from shapepipe.modules.vignetmaker_package.vignetmaker import get_stamps + +RAD = 25 +N = 2 * RAD + 1 +_rr = np.hypot(*(np.indices((N, N)) - RAD)) +BOX = 128 + + +def read_tile(sexcat): + with fitsio.FITS(sexcat) as f: + hdu = [h for h in f if h.get_extname() == "LDAC_OBJECTS"][0] + return hdu.read(columns=["NUMBER", "XWIN_WORLD", "YWIN_WORLD", + "MAG_AUTO", "FLUX_RADIUS", "VIGNET", + "SEG_VIGNET"]) + + +def tile_name(sexcat): + return ".".join(re.search(r"(\d{3})-(\d{3})", sexcat).groups()) + + +def ccd_positions(hdus, ra, dec): + """For each CCD extension, the tile rows whose stamps fit on it and their + 0-indexed [row, col] positions.""" + out = [] + for ext in range(1, len(hdus)): + h = hdus[ext].header + x, y = WCS(h).all_world2pix(ra, dec, 1) + inside = ((x - 1 >= RAD) & (x - 1 < h["NAXIS1"] - RAD) + & (y - 1 >= RAD) & (y - 1 < h["NAXIS2"] - RAD)) + rows = np.flatnonzero(inside) + out.append((ext, rows, np.column_stack([y[rows] - 1, x[rows] - 1]))) + return out + + +def tile_overlay(tile, i, ccd): + tile_vign = Ngmix.MegaCamFlip(np.copy(tile["VIGNET"][i]), ccd) + seg = Ngmix.MegaCamFlip(np.copy(tile["SEG_VIGNET"][i]), ccd) + neighbour, off_tile = split_tile_markers(tile_vign, (N, N)) + return seg.astype(np.int32), neighbour, off_tile + + +def components(defect, flag, off_tile): + """Defect components outside the off-tile band, nearest first: (kind, + distance, height, width, size, flag values).""" + labels, _ = label(defect & ~off_tile, structure=np.ones((3, 3))) + found = [] + for k, sl in enumerate(find_objects(labels), start=1): + part = labels == k + h, w = sl[0].stop - sl[0].start, sl[1].stop - sl[1].start + size = int(part.sum()) + if h == N: + kind = {1: "column"}.get(w, f"columns{w}" if w <= 4 else "wide") + elif size <= 6: + kind = "pixel" + elif w <= 4 and h >= 8: + kind = "trail" + else: + kind = "blob" + values = sorted(set(np.unique(flag[part]).tolist()) - {0}) + found.append((kind, float(_rr[part].min()), h, w, size, values)) + return sorted(found, key=lambda c: c[1]) + + +def cut(hdus, ext, pos, tile, i): + """One epoch of tile row ``i`` from CCD extension ``ext``.""" + img, wgt, flg = (h[ext].data for h in hdus) + ccd = ext - 1 + stamps = {} + for key, a, dtype in (("gal", img, float), ("weight", wgt, float), + ("flag", flg, np.int32)): + s, int_pos, offset = get_stamps(a, pos[None, :], RAD) + stamps[key] = s[0].astype(dtype) + r0, c0 = int_pos[0] + box = (slice(max(r0 - BOX, 0), r0 + BOX + 1), + slice(max(c0 - BOX, 0), c0 + BOX + 1)) + good = (wgt[box] > 0) & (flg[box] == 0) + _, bkg, rms = sigma_clipped_stats(img[box][good], sigma=3.0) + stamps["gal"] = stamps["gal"] - bkg + seg, neighbour, off_tile = tile_overlay(tile, i, ccd) + stamps["weight"][off_tile] = 0 + stamps.update( + bkg_rms=np.full((N, N), rms), seg=seg, neighbour=neighbour, + off_tile=off_tile, + object_number=int(tile["NUMBER"][i]), ccd=ccd, + offset=offset[0], int_pos=int_pos[0], + ) + return stamps + + +def scan(hdus, tile): + """Print one line per epoch with any defect pixel.""" + for ext, rows, pos in ccd_positions(hdus[0], tile["XWIN_WORLD"], + tile["YWIN_WORLD"]): + if not len(rows): + continue + wgt = get_stamps(hdus[1][ext].data, pos, RAD)[0] + flg = get_stamps(hdus[2][ext].data, pos, RAD)[0].astype(np.int32) + for i, w, f in zip(rows, wgt, flg): + seg, neighbour, off_tile = tile_overlay(tile, i, ext - 1) + w[off_tile] = 0 + defect = defect_mask(w, f) + if not defect.any(): + continue + d_nb = _rr[neighbour].min() if neighbour.any() else np.inf + parts = ";".join( + f"{k}@{d:.1f}:{h}x{wd}:{n}:{'/'.join(map(str, v))}" + for k, d, h, wd, n, v in components(defect, f, off_tile) + ) + print(f"{tile['NUMBER'][i]} {tile['MAG_AUTO'][i]:.2f} " + f"{tile['FLUX_RADIUS'][i]:.2f} {ext - 1} " + f"{int(defect.sum())} {int(off_tile.sum())} {d_nb:.1f} " + f"{parts or '-'}", flush=True) + + +def main(): + mode, exp, wgt, flg, sexcat = sys.argv[1:6] + tile = read_tile(sexcat) + hdus = [fits.open(p) for p in (exp, wgt, flg)] + if mode == "scan": + print("NUMBER MAG_AUTO FLUX_RADIUS CCD N_DEFECT N_OFF_TILE " + "D_NEIGHBOUR COMPONENTS(kind@dist:hxw:size:flags)") + scan(hdus, tile) + return + number, out = int(sys.argv[6]), sys.argv[7] + i = int(np.flatnonzero(tile["NUMBER"] == number)[0]) + ext, rows, pos = next( + (e, r, p) for e, r, p in ccd_positions( + hdus[0], tile["XWIN_WORLD"][[i]], tile["YWIN_WORLD"][[i]]) + if len(r) + ) + ep = cut(hdus, ext, pos[0], tile, i) + expname = exp.split("/")[-1].split(".")[0] + ep["label"] = f"tile {tile_name(sexcat)}, object {number}" + ep["source"] = f"exposure {expname}, CCD {ep['ccd']}" + ep.update(ra=tile["XWIN_WORLD"][i], dec=tile["YWIN_WORLD"][i], + mag=tile["MAG_AUTO"][i]) + np.savez(out, **ep) + print(ep["label"], ep["source"]) + + +if __name__ == "__main__": + main() diff --git a/scripts/validation/masked_pixels/figstyle.py b/scripts/validation/masked_pixels/figstyle.py new file mode 100644 index 000000000..977f12795 --- /dev/null +++ b/scripts/validation/masked_pixels/figstyle.py @@ -0,0 +1,83 @@ +"""Shared look of the masked-pixel figures (image stretch, weight classes).""" +import numpy as np +from matplotlib.colors import ListedColormap +from matplotlib.patches import Patch + +from shapepipe.modules.ngmix_package.defect_interpolation import fourfold + +INK, INK2 = "#0b0b0b", "#52514e" +PAPER, DARK = "#f4f3ef", "#3a3a38" +BLUE, ORANGE, AQUA, VIOLET = "#2a78d6", "#eb6834", "#1baf7a", "#8e5bd0" +TARGET = "#c3c2b7" + + +def stretch(a, soft=3.0): + return np.arcsinh(np.asarray(a, dtype=float) / soft) + + +def show_image(ax, img, vmax=120.0, soft=3.0): + ax.imshow(stretch(img, soft), origin="lower", cmap="gray_r", + vmin=stretch(-3 * soft / 3.0, soft), vmax=stretch(vmax, soft), + interpolation="nearest") + + +# Weight panel: 0 weighted, 1 zero weight, 2 zero weight on a rotated copy +# of an interpolated pixel (its light is kept). +WEIGHT_CMAP = ListedColormap([PAPER, DARK, VIOLET]) + + +def weight_classes(w, interpolated, zeroed=None): + """0 = weighted, 1 = zero weight, 2 = zero weight only as a quarter-turn + copy of an interpolated pixel. ``zeroed`` marks the pixels that have + zero weight for another reason (defects, neighbours); they stay 1.""" + cls = np.where(w > 0, 0, 1) + copies = fourfold(interpolated) & ~interpolated + if zeroed is not None: + copies &= ~zeroed + cls[(w == 0) & copies] = 2 + return cls + + +def show_weight(ax, w, interpolated=None, zeroed=None): + if interpolated is None: + interpolated = np.zeros(w.shape, bool) + ax.imshow(weight_classes(w, interpolated, zeroed), origin="lower", + cmap=WEIGHT_CMAP, vmin=-0.5, vmax=2.5, interpolation="nearest") + + +WEIGHT_LEGEND = [ + Patch(color=PAPER, ec=INK2, lw=0.5, label="weighted"), + Patch(color=DARK, label="zero weight"), + Patch(color=VIOLET, label="zero weight, light kept: rotated copy of an " + "interpolated pixel"), +] + + +def outline(ax, mask, color, lw=1.0): + """Draw the pixel-edge outline of a boolean mask.""" + mask = np.asarray(mask, bool) + ny, nx = mask.shape + pad = np.pad(mask, 1) + segs = [] + for r in range(ny): + for c in range(nx): + if not mask[r, c]: + continue + if not pad[r, c + 1]: + segs.append(((c - .5, c + .5), (r - .5, r - .5))) + if not pad[r + 2, c + 1]: + segs.append(((c - .5, c + .5), (r + .5, r + .5))) + if not pad[r + 1, c]: + segs.append(((c - .5, c - .5), (r - .5, r + .5))) + if not pad[r + 1, c + 2]: + segs.append(((c + .5, c + .5), (r - .5, r + .5))) + for xs, ys in segs: + ax.plot(xs, ys, color=color, lw=lw, solid_capstyle="butt") + + +def clean(ax): + ax.set_xticks([]) + ax.set_yticks([]) + for s in ax.spines.values(): + s.set_color(INK2) + s.set_linewidth(0.6) diff --git a/scripts/validation/masked_pixels/pixel_classes.py b/scripts/validation/masked_pixels/pixel_classes.py new file mode 100644 index 000000000..0d4d377d3 --- /dev/null +++ b/scripts/validation/masked_pixels/pixel_classes.py @@ -0,0 +1,187 @@ +"""Pixel classes of one epoch stamp and what ngmix receives: a synthetic +blend under BLEND_HANDLING = noisefill and uberseg, then a real DR6 blend +under uberseg. + +Run from the shapepipe checkout root inside the shapepipe container: + PYTHONPATH=src:scripts/validation/masked_pixels \ + python scripts/validation/masked_pixels/pixel_classes.py REAL.npz OUT.png + +REAL.npz is one real epoch written by dr6_epochs.py. +""" +import sys + +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np +from matplotlib.colors import ListedColormap +from matplotlib.lines import Line2D +from matplotlib.patches import Circle, Patch + +from figstyle import (AQUA, BLUE, INK, INK2, ORANGE, PAPER, TARGET, + WEIGHT_LEGEND, clean, show_image, show_weight) +from shapepipe.modules.ngmix_package.ngmix import ( + EPOCH_CENTRAL_DEFECT_RADIUS, + EPOCH_INTERPOLATED_DEFECT_RADIUS, + defect_mask, + interpolated_defects, + prepare_ngmix_weights, + uberseg_mask, +) + + +def synthetic(): + """The synthetic epoch: a target, a neighbour 15 px away, a bad column, + a hot pixel and an off-tile band.""" + N, C = 51, 25 + rng = np.random.RandomState(4) + yy, xx = np.indices((N, N)) + + def blob(r0, c0, sigma, flux, q=1.0, theta=0.0): + dy, dx = yy - r0, xx - c0 + ct, st = np.cos(theta), np.sin(theta) + u, v = ct * dx + st * dy, -st * dx + ct * dy + r = np.sqrt(u ** 2 + (v / q) ** 2) + return flux * np.exp(-r / sigma) / (2 * np.pi * sigma ** 2 * q) + + rms = 1.0 + target = blob(C, C, 2.2, 3000, q=0.7, theta=0.5) + nbr = blob(C - 10, C - 11, 2.0, 4000, q=0.8, theta=-0.3) + gal = target + nbr + rng.normal(0, rms, (N, N)) + # Segmentation: each pixel above 2 sigma joins the brighter profile. + above = (target + nbr) > 2 * rms + seg = np.where(above, np.where(target >= nbr, 1, 2), 0).astype(np.int32) + flag = np.zeros((N, N), np.int32) + flag[:, C + 12] = 1 # bad column, 12 px right of centre + flag[C + 9, C - 9] = 1 # hot pixel, 12.7 px from centre + off_tile = np.zeros((N, N), bool) + off_tile[N - 5:, :] = True # off-tile band at the stamp's edge + gal[:, C + 12] = 400.0 + gal[C + 9, C - 9] = 400.0 + return dict(gal=gal, weight=np.where(off_tile, 0.0, 1.0), flag=flag, + bkg_rms=np.full((N, N), rms), seg=seg, object_number=1, + neighbour=seg == 2, off_tile=off_tile) + + +def run(ep, blend_handling): + kw = {} + if blend_handling == "uberseg": + kw.update(seg=ep["seg"], object_number=int(ep["object_number"])) + img, w, _ = prepare_ngmix_weights( + ep["gal"], ep["weight"], ep["flag"], np.random.RandomState(1), + bkg_rms=ep["bkg_rms"], neighbour=ep["neighbour"], + blend_handling=blend_handling, **kw, + ) + defect = defect_mask(ep["weight"], ep["flag"], ep["bkg_rms"]) + interp = interpolated_defects(defect, ep["neighbour"], blend_handling) + neighbour = (ep["neighbour"] if blend_handling == "noisefill" else + uberseg_mask(ep["seg"], int(ep["object_number"]))) + return img, w, interp, defect | neighbour + + +CLASS_CMAP = ListedColormap([PAPER, BLUE, ORANGE, AQUA, TARGET]) + + +def classes(ep): + cls = np.zeros(ep["gal"].shape, int) + target = ep["seg"] == int(ep["object_number"]) + cls[target] = 4 + cls[ep["neighbour"]] = 1 + defect = defect_mask(ep["weight"], ep["flag"], ep["bkg_rms"]) + cls[defect] = 2 + cls[ep["off_tile"]] = 3 + return cls + + +def class_panel(ax, ep, radii=True): + ax.imshow(classes(ep), origin="lower", cmap=CLASS_CMAP, vmin=-0.5, + vmax=4.5, interpolation="nearest") + if radii: + n = ep["gal"].shape[0] + c = (n - 1) / 2 + for r, ls in ((EPOCH_CENTRAL_DEFECT_RADIUS, "--"), + (EPOCH_INTERPOLATED_DEFECT_RADIUS, ":")): + ax.add_patch(Circle((c, c), r, fill=False, ec=INK, ls=ls, + lw=1.0)) + + +def main(real_path, out_path): + syn = synthetic() + real = dict(np.load(real_path, allow_pickle=True)) + # The synthetic stamp has unit noise; scale the real one's stretch by + # its background RMS so both read alike. + rms = float(np.median(real["bkg_rms"])) + vmax_real, soft_real = 120.0 * rms, 3.0 * rms + + plt.rcParams.update({"font.size": 10}) + fig, axs = plt.subplots( + 2, 5, figsize=(13.6, 6.1), dpi=150, + gridspec_kw=dict(hspace=0.08, wspace=0.05, + width_ratios=[1, 1, 1, 1, 1]), + ) + # Open a gap between the synthetic and the real columns. + for ax in axs[:, 3:].flat: + p = ax.get_position() + ax.set_position([p.x0 + 0.025, p.y0, p.width, p.height]) + + # (a) synthetic epoch and its pixel classes + show_image(axs[0, 0], syn["gal"]) + class_panel(axs[1, 0], syn) + axs[0, 0].set_title("(a) one epoch's stamp", color=INK, fontsize=10) + axs[0, 0].set_ylabel("image ngmix sees", color=INK, fontsize=11) + axs[1, 0].set_ylabel("pixel class / weight", color=INK, fontsize=11) + c = 25 + rn = EPOCH_CENTRAL_DEFECT_RADIUS + axs[1, 0].text(c, c + rn + 0.8, f"{rn:g} px", ha="center", va="bottom", + fontsize=8, color=INK) + axs[1, 0].text(c, c - 6.3, "7 px", ha="center", va="bottom", + fontsize=8, color=INK) + + for j, (blend, title) in enumerate( + (("noisefill", "(b) noisefill (default)"), + ("uberseg", "(c) uberseg")), start=1): + img, w, interp, zeroed = run(syn, blend) + show_image(axs[0, j], img) + show_weight(axs[1, j], w, interp, zeroed) + axs[0, j].set_title(title, color=INK, fontsize=10) + + # (d) the real DR6 epoch and its classes, (e) under uberseg + show_image(axs[0, 3], real["gal"], vmax=vmax_real, soft=soft_real) + class_panel(axs[1, 3], real) + axs[0, 3].set_title(f"(d) {real['label']}\n{real['source']}", + color=INK, fontsize=10) + img, w, interp, zeroed = run(real, "uberseg") + show_image(axs[0, 4], img, vmax=vmax_real, soft=soft_real) + show_weight(axs[1, 4], w, interp, zeroed) + axs[0, 4].set_title("(e) uberseg", color=INK, fontsize=10) + for ax in axs.flat: + clean(ax) + + y = axs[0, 0].get_position().y1 + 0.075 + for (a, b), t in (((0, 2), "synthetic blend"), + ((3, 4), "real DR6 blend")): + x0 = axs[0, a].get_position().x0 + x1 = axs[0, b].get_position().x1 + fig.text((x0 + x1) / 2, y, t, ha="center", va="bottom", color=INK, + fontsize=11) + fig.add_artist(Line2D([x0 + 0.004, x1 - 0.004], [y - 0.006] * 2, + color=INK2, lw=0.8)) + + handles = [ + Patch(color=TARGET, label="target footprint"), + Patch(color=BLUE, label="neighbour (−1e30 in tile)"), + Patch(color=ORANGE, label="defect (flag / weight / RMS)"), + Patch(color=AQUA, label="off-tile (a defect)"), + Line2D([], [], color=INK, ls="--", lw=1, label=f"{EPOCH_CENTRAL_DEFECT_RADIUS:g} px veto"), + Line2D([], [], color=INK, ls=":", lw=1, + label="7 px veto (interpolated)"), + ] + WEIGHT_LEGEND + fig.legend(handles=handles, loc="upper center", ncol=5, frameon=False, + fontsize=9, handlelength=1.4, + bbox_to_anchor=(0.5, axs[1, 0].get_position().y0 - 0.01)) + fig.savefig(out_path, bbox_inches="tight") + + +if __name__ == "__main__": + main(sys.argv[1], sys.argv[2]) diff --git a/src/shapepipe/modules/make_cat_package/make_cat.py b/src/shapepipe/modules/make_cat_package/make_cat.py index 8a6f9548a..e18b9cb4e 100644 --- a/src/shapepipe/modules/make_cat_package/make_cat.py +++ b/src/shapepipe/modules/make_cat_package/make_cat.py @@ -113,7 +113,8 @@ def save_sextractor_data(final_cat_file, sexcat_path, remove_vignet=True): sexcat_path : str Path to SExtractor catalogue to save remove_vignet : bool - If ``True`` will not save the ``VIGNET`` field into the final catalogue + If ``True`` will not save the ``VIGNET`` and ``SEG_VIGNET`` stamp + fields into the final catalogue Returns ------- @@ -127,6 +128,7 @@ def save_sextractor_data(final_cat_file, sexcat_path, remove_vignet=True): data = np.copy(sexcat_file.get_data()) if remove_vignet: data = remove_field_name(data, "VIGNET") + data = remove_field_name(data, "SEG_VIGNET") cat_size = len(data) tile_name = os.path.basename(sexcat_path) @@ -334,11 +336,16 @@ def _save_ngmix_data(self, ngmix_cat_path, moments=False): Save the NGMIX catalogue into the final one. Column grammar: ``NGMIX[m]_[_ERR][_]_``, - plus four OBJECT/SHEAR-less per-object metadata columns + plus OBJECT/SHEAR-less per-object metadata columns (``NGMIX[m]_MCAL_FLAGS``, ``NGMIX_N_EPOCH``, - ``NGMIX_MCAL_TYPES_FAIL``, ``NGMIX_NEIGHBOUR_FLAG`` — the last a blend + ``NGMIX_N_EPOCH_FAILED`` — epochs dropped because building their + observation raised, see ``ngmix.do_ngmix_metacal`` — + ``NGMIX_MCAL_TYPES_FAIL``, ``NGMIX_NEIGHBOUR_FLAG`` — a blend flag set when the coadd seg stamp held a non-central footprint, - shapepipe#776). The galaxy is the implicit default object + shapepipe#776 — and the defect diagnostics ``NGMIX_N_EPOCH_INTERP``, + ``NGMIX_MIN_DIST_INTERP``, ``NGMIX_MIN_DIST_NOISEFILL``, described + by ``Postage_stamp.record_defects``; -1 is their absent distance, + and a never-fit row carries 0 and -1). The galaxy is the implicit default object and carries NO ``OBJECT`` token (``NGMIX_G1_NOSHEAR``, dropping the ``GAL`` segment carried by the pre-#761 names). The explicit PSF objects are ``PSF_ORIG`` @@ -391,6 +398,18 @@ def _save_ngmix_data(self, ngmix_cat_path, moments=False): # Per-object blend flag (shapepipe#776): the seg stamp held a # non-central footprint. Galaxy-only (non-moments), like N_EPOCH. ngmix_neighbour_flag = ngmix_cat_file.get_data()["neighbour_flag"] + # Failed-epoch count (ngmix.do_ngmix_metacal) and defect diagnostics + # (ngmix Postage_stamp.record_defects). + defect_columns = { + "NGMIX_N_EPOCH_FAILED": "n_epoch_failed", + "NGMIX_N_EPOCH_INTERP": "n_epoch_interp", + "NGMIX_MIN_DIST_INTERP": "min_dist_interp", + "NGMIX_MIN_DIST_NOISEFILL": "min_dist_noisefill", + } + ngmix_defects = { + out: ngmix_cat_file.get_data()[col] + for out, col in defect_columns.items() + } # Needed in both moments and non-moments modes (used unconditionally # below), so read them outside the branch. ngmix_mcal_flags = ngmix_cat_file.get_data()["mcal_flags"] @@ -413,6 +432,10 @@ def _save_ngmix_data(self, ngmix_cat_path, moments=False): np.full(n_obj, get_mcal_types_fail(never_fit), dtype=float), ) self._add2dict("NGMIX_NEIGHBOUR_FLAG", np.zeros(n_obj)) + self._add2dict("NGMIX_N_EPOCH_FAILED", np.zeros(n_obj)) + self._add2dict("NGMIX_N_EPOCH_INTERP", np.zeros(n_obj)) + self._add2dict("NGMIX_MIN_DIST_INTERP", np.full(n_obj, -1.0)) + self._add2dict("NGMIX_MIN_DIST_NOISEFILL", np.full(n_obj, -1.0)) prefix = f"NGMIX{m}" @@ -576,6 +599,8 @@ def _save_ngmix_data(self, ngmix_cat_path, moments=False): ngmix_neighbour_flag[ind[0]], idx, ) + for out, values in ngmix_defects.items(): + self._add2dict(out, values[ind[0]], idx) ngmix_cat_file.close() diff --git a/src/shapepipe/modules/ngmix_package/defect_interpolation.py b/src/shapepipe/modules/ngmix_package/defect_interpolation.py new file mode 100644 index 000000000..5e539b437 --- /dev/null +++ b/src/shapepipe/modules/ngmix_package/defect_interpolation.py @@ -0,0 +1,160 @@ +"""DEFECT INTERPOLATION. + +Clough-Tocher interpolation of short defect runs before metacal (see +:func:`shapepipe.modules.ngmix_package.ngmix.prepare_ngmix_weights`). + +""" + +import numpy as np +from scipy.interpolate import CloughTocher2DInterpolator +from scipy.ndimage import binary_dilation, label +from scipy.spatial import QhullError + +# Longest row or column run of defect pixels that is interpolated. +MAX_INTERPOLATED_RUN = 3 + +# Clean pixels within this Chebyshev distance (pixels) of the interpolated +# pixels support the interpolant. +SUPPORT_RADIUS = 4 + +_ROW_RUNS = np.array([[0, 0, 0], [1, 1, 1], [0, 0, 0]]) + + +def _short_row_runs(run, blocked, max_run): + """Pixels of ``run`` in row runs of at most ``max_run`` pixels whose + neighbours beyond both ends lie on the stamp and outside ``blocked``.""" + labels, n_runs = label(run, structure=_ROW_RUNS) + if n_runs == 0: + return np.zeros_like(run) + short = np.bincount(labels.ravel(), minlength=n_runs + 1) <= max_run + short[0] = False + # A run pixel's left (right) neighbour must be in the run or open. + in_run = np.pad(run, ((0, 0), (1, 1))) + is_open = np.pad(~blocked, ((0, 0), (1, 1))) + closed_end = run & ~( + (in_run[:, :-2] | is_open[:, :-2]) & (in_run[:, 2:] | is_open[:, 2:]) + ) + short[labels[closed_end]] = False + return short[labels] + + +def interpolable_defects(defect, removed=None, max_run=MAX_INTERPOLATED_RUN): + """Defect pixels that :func:`interpolate_defects` fills. + + @sc [decision:shape_measurement.defect_fill] interpolable-defects + A defect pixel is interpolated when its row or its column run of defect + pixels is at most ``max_run`` (3) long and the pixels beyond both ends + support the interpolant: on the stamp, not a defect, and not + ``removed``. That covers columns, 3-px bleeds and isolated pixels. + Wider holes, edge bands and runs that end on a removed pixel are + noise-filled and vetoed at the noise-fill radius + (:func:`~shapepipe.modules.ngmix_package.ngmix.central_defect_vetoes`). + Each interpolated pixel lies between two support pixels, so the + interpolant reaches it. The rule reads only the masks and commutes with + quarter turns of the stamp. + + Parameters + ---------- + defect : numpy.ndarray of bool + Defect mask of one epoch stamp. + removed : numpy.ndarray of bool, optional + Pixels whose light the image replaces (noisefill neighbours). They + neither support the interpolant nor are interpolated. + max_run : int, optional + Longest interpolated run; the default is ``MAX_INTERPOLATED_RUN``. + + Returns + ------- + numpy.ndarray of bool + ``True`` on the defect pixels to interpolate. + """ + defect = np.asarray(defect, dtype=bool) + removed = ( + np.zeros_like(defect) if removed is None + else np.asarray(removed, dtype=bool) + ) + run, blocked = defect & ~removed, defect | removed + return ( + _short_row_runs(run, blocked, max_run) + | _short_row_runs(run.T, blocked.T, max_run).T + ) + + +def fourfold(mask): + """Union of a square stamp mask with its quarter turns. + + Parameters + ---------- + mask : numpy.ndarray of bool + Square mask. + + Returns + ------- + numpy.ndarray of bool + ``mask`` ORed with its rotations by 90, 180 and 270 degrees about + the stamp centre. + + Raises + ------ + ValueError + If ``mask`` is not square. + """ + mask = np.asarray(mask, dtype=bool) + if mask.ndim != 2 or mask.shape[0] != mask.shape[1]: + raise ValueError( + f"A quarter-turn orbit needs a square stamp, not {mask.shape}" + ) + return mask | np.rot90(mask) | np.rot90(mask, 2) | np.rot90(mask, 3) + + +def interpolate_defects(planes, excluded, target): + """Replace the ``target`` pixels of every plane by a Clough-Tocher + interpolant of the kept pixels around them. + + @sc [decision:shape_measurement.defect_fill] shared-interpolant + The support is the pixels within ``SUPPORT_RADIUS`` (4 px) of the target + outside ``excluded``; no excluded pixel enters it, so their values are + never read. One Delaunay triangulation of the support serves every + plane, so the science image and the metacal noise image see the same + linear operator and fixnoise mirrors the science image's interpolated + noise. + + Parameters + ---------- + planes : array_like + Stamp planes, shape ``(n, ny, nx)``. + excluded : numpy.ndarray of bool + Pixels that never support the interpolant, shape ``(ny, nx)``: every + defect, and any pixel whose light the image does not keep. + target : numpy.ndarray of bool + Defect pixels to interpolate (:func:`interpolable_defects`). + + Returns + ------- + numpy.ndarray + A copy of ``planes`` with ``target`` pixels interpolated; NaN at a + target pixel the support cannot reach. + """ + planes = np.asarray(planes, dtype=float) + excluded = np.asarray(excluded, dtype=bool) + target = np.asarray(target, dtype=bool) + out = planes.copy() + if not target.any(): + return out + out[:, target] = np.nan + support = binary_dilation( + target, structure=np.ones((3, 3), dtype=bool), + iterations=SUPPORT_RADIUS, + ) & ~excluded + points = np.argwhere(support).astype(float) + if len(points) < 3: + return out + query = np.argwhere(target) + try: + interpolant = CloughTocher2DInterpolator( + points, planes[:, support].T, fill_value=np.nan, + ) + except QhullError: + return out + out[:, query[:, 0], query[:, 1]] = interpolant(query.astype(float)).T + return out diff --git a/src/shapepipe/modules/ngmix_package/ngmix.py b/src/shapepipe/modules/ngmix_package/ngmix.py index 12b192fd8..49fd9e19f 100644 --- a/src/shapepipe/modules/ngmix_package/ngmix.py +++ b/src/shapepipe/modules/ngmix_package/ngmix.py @@ -8,6 +8,7 @@ import os import re +from collections import Counter from typing import NamedTuple import ngmix @@ -21,11 +22,41 @@ from scipy.spatial import cKDTree from sqlitedict import SqliteDict +from shapepipe.modules.ngmix_package.defect_interpolation import ( + fourfold, + interpolable_defects, + interpolate_defects, +) from shapepipe.pipeline import file_io # Neighbour treatments selectable with the BLEND_HANDLING option. BLEND_HANDLINGS = ("noisefill", "uberseg") +# How interpolated defects enter the image and the weight map, selectable +# with the DEFECT_WEIGHTING option (see :func:`defect_weighting_masks`), and +# the value an absent or empty option takes. +DEFECT_WEIGHTINGS = ("des_y6", "fourfold_zero", "hole", "full") +# @sc [decision:shape_measurement.defect_weighting] +DEFECT_WEIGHTING = "fourfold_zero" + +# The epoch cuts (see :func:`prepare_postage_stamps`). Calibration outputs, +# not options: an epoch is dropped when more than EPOCH_MASKED_FRACTION_CUT +# of its stamp is in :func:`defect_mask`, or when a noise-filled +# (interpolated) defect pixel lies closer than EPOCH_CENTRAL_DEFECT_RADIUS +# (EPOCH_INTERPOLATED_DEFECT_RADIUS) pixels to the stamp centre (see +# :func:`central_defect_vetoes`). The central veto is the bias control: defects +# outside it bias nothing measured up to 58% of the stamp. The noise-fill +# radius is set by the largest galaxy tested (0.7" through a 0.9" PSF), the +# interpolated radius by 3-px bleeds on the same galaxy. The 10% fraction +# cut matches DES Y3/Y6 and guards against real-data effects the simulations +# do not model, at the cost of about 5% of epochs on a real tile. +# @sc [decision:shape_measurement.epoch_masked_fraction_cut] +EPOCH_MASKED_FRACTION_CUT = 0.10 +# @sc [decision:shape_measurement.central_defect_veto] +EPOCH_CENTRAL_DEFECT_RADIUS = 13 +# @sc [decision:shape_measurement.central_defect_veto] +EPOCH_INTERPOLATED_DEFECT_RADIUS = 7 + # @sc [decision:shape_measurement.metacal_scheme] METACAL_TYPES = ('noshear', '1p', '1m', '2p', '2m') @@ -226,8 +257,15 @@ def empty_metacal_output(): names2 = [ 'id', 'n_epoch_model', + # epochs dropped because building their observation raised + # (failed-epoch-dropped) + 'n_epoch_failed', 'mcal_types_fail', 'neighbour_flag', + # defect diagnostics (Postage_stamp.record_defects) + 'n_epoch_interp', + 'min_dist_interp', + 'min_dist_noisefill', 'nfev_fit', # galaxy 'g1', @@ -500,30 +538,68 @@ def position_seed(ra, dec, ccd): return cantor % (2 ** 32) +class ChunkStamps(): + """One stamp column of the tile catalogue, held for one chunk's rows. + + Indexed by tile-catalogue row like the full column it stands in for, so + ``stamps[i_tile]`` is the stamp of row ``i_tile``; rows outside the chunk + raise ``IndexError``. + + Parameters + ---------- + column : numpy.ndarray + Full stamp column, one stamp per catalogue row. A memory-mapped + column is read only over ``rows``. + rows : range + 0-based catalogue rows to hold (see :func:`chunk_rows`). + + """ + def __init__(self, column, rows): + self.rows = rows + self._stamps = np.array(column[rows.start:rows.stop]) + + def __getitem__(self, i_tile): + if not self.rows.start <= i_tile < self.rows.stop: + raise IndexError( + f"tile-catalogue row {i_tile} is outside this chunk's rows" + + f" {self.rows.start}..{self.rows.stop - 1}" + ) + return self._stamps[i_tile - self.rows.start] + + class Tile_cat(): """Tile_cat. catalog measured on a tile + The per-object columns (``obj_id``, ``ra``, ``dec``, ``flux``) are held + for every row. The stamp columns (``vign``, ``seg``) are held only for + the chunk's rows (``self.rows``) as :class:`ChunkStamps`, still indexed + by tile-catalogue row: the catalogues are memory-mapped, so a chunk reads + only its own stamps. + Parameters ---------- cat_path : str - Path to the tile SExtractor catalogue. - seg_cat_path : str, optional - Path to the coadd-frame segmentation VIGNET catalogue (a CLASSIC-mode - vignetmaker output cut from the tile ``SEGMENTATION`` check image), - row-aligned to ``cat_path``. When given, ``self.seg`` holds one integer - seg stamp per object for the ``"uberseg"`` blend handling; ``None`` - leaves ``self.seg`` unset (the noise-fill path is unaffected). + Path to the tile SExtractor catalogue. Its optional ``SEG_VIGNET`` + column, one integer coadd segmentation stamp per object on the grid + of its ``VIGNET``, becomes ``self.seg``, which defines the neighbours + under both blend handlings; without it ``self.seg`` is ``None``. + row_min, row_max : int, optional + First and last catalogue row of the chunk (1-based, inclusive; see + :func:`chunk_rows`). The default, ``-1``, is unbounded, so the whole + catalogue. """ def __init__( self, cat_path, - seg_cat_path=None, + row_min=-1, + row_max=-1, ): self.cat_path = cat_path - self.seg_cat_path = seg_cat_path + self.row_min = row_min + self.row_max = row_max if cat_path: self.get_data(cat_path) @@ -531,6 +607,7 @@ def get_data(self, cat_path): tile_cat = file_io.FITSCatalogue( cat_path, SEx_catalogue=True, + memmap=True, ) tile_cat.open() data = tile_cat.get_data() @@ -539,45 +616,27 @@ def get_data(self, cat_path): self.obj_id = np.copy(data['NUMBER']) self.ra = np.copy(data['XWIN_WORLD']) self.dec = np.copy(data['YWIN_WORLD']) + self.rows = chunk_rows(len(self.obj_id), self.row_min, self.row_max) # Optional columns — may be absent in external (non-SExtractor) catalogs self.flux = np.copy(data['FLUX_AUTO']) if 'FLUX_AUTO' in cols else None - self.vign = np.copy(data['VIGNET']) if 'VIGNET' in cols else None + self.vign = ( + ChunkStamps(data['VIGNET'], self.rows) + if 'VIGNET' in cols + else None + ) - tile_cat.close() + # Coadd-frame segmentation stamp (integer labels, the catalogue's + # NUMBER), one per object on the grid of its VIGNET, overlaid + # unchanged on every epoch; it defines the neighbours (shapepipe#776, + # split_tile_markers, uberseg_mask). + self.seg = ( + ChunkStamps(data['SEG_VIGNET'], self.rows) + if 'SEG_VIGNET' in cols + else None + ) - # Coadd-frame SExtractor segmentation stamp (integer labels), one per - # object and row-aligned to the tile catalogue, overlaid unchanged on - # every epoch for uberseg neighbour masking (shapepipe#776). None -> - # uberseg unavailable; the noise-fill path is unaffected. - self.seg = None - if self.seg_cat_path: - seg_cat = file_io.FITSCatalogue( - self.seg_cat_path, - SEx_catalogue=True, - ) - seg_cat.open() - seg_data = seg_cat.get_data() - # The seg VIGNETs are indexed by tile-catalogue row (self.seg[i]), - # so the two catalogues MUST be row-aligned. Fail loud at load if - # they are not — a silent length/order mismatch would hand every - # object the wrong footprint and quietly corrupt every mask. - if len(seg_data) != len(self.obj_id): - raise ValueError( - f"SEG_VIGNET_PATH '{self.seg_cat_path}' has" - + f" {len(seg_data)} rows but the tile catalogue has" - + f" {len(self.obj_id)}; the segmentation vignets must be" - + " row-aligned to the tile catalogue." - ) - if 'NUMBER' in seg_data.dtype.names: - if not np.array_equal(seg_data['NUMBER'], self.obj_id): - raise ValueError( - f"SEG_VIGNET_PATH '{self.seg_cat_path}' NUMBER column" - + " does not match the tile catalogue NUMBER; the" - + " segmentation vignets are misaligned or reordered." - ) - self.seg = np.copy(seg_data['VIGNET']) - seg_cat.close() + tile_cat.close() class Postage_stamp(): """Galaxy Postage Stamp. @@ -602,9 +661,15 @@ def __init__( self.psfs = [] self.weights = [] self.flags = [] + # Neighbour masks, one per epoch: the pixels marked -1e30 in the tile + # VIGNET on other detections' footprints (off-tile markers get zero + # weight instead; see split_tile_markers), MegaCam-flipped to the + # epoch. noisefill zero-weights and noise-fills them, and the epoch + # cuts do not count them (see prepare_ngmix_weights). + self.neighbours = [] self.bkg_rms = [] # Segmentation stamps, one per epoch, used only by the "uberseg" blend - # handling; empty for the default noise-fill path. All epochs carry the + # handling; empty under the default "noisefill". All epochs carry the # SAME coadd-frame seg stamp (shapepipe#776: one coadd seg per object, # no per-epoch reprojection), each MegaCam-flipped to match its galaxy # stamp so the overlay stays registered. @@ -621,9 +686,68 @@ def __init__( # CCD number of the first epoch, used only to build the per-object # position seed (see :func:`position_seed`). self.ccd = None + self.epoch_cuts = Counter( + considered=0, masked_fraction=0, central_veto=0 + ) + # The exposure-CCD name of each kept epoch, for the log. + self.epoch_names = [] + # ``(epoch name, exception)`` for each kept epoch whose observation + # raised while being built; :func:`do_ngmix_metacal` drops the epoch + # and records it here (failed-epoch-dropped). + self.epoch_failures = [] + # Defect diagnostics over the epochs kept (see + # :meth:`record_defects`). + self.n_epoch_interp = 0 + self.min_dist_interp = np.inf + self.min_dist_noisefill = np.inf self.bkg_sub = bkg_sub self.megacam_flip = megacam_flip + def record_defects(self, defect, interpolated): + """Add one kept epoch to the defect diagnostics. + + @sc [label:schema] defect-diagnostic-columns + Per object, over the epochs handed to the fit: ``n_epoch_interp`` + counts those with an interpolated defect pixel, and + ``min_dist_interp`` (``min_dist_noisefill``) is the smallest distance + in pixels from the stamp centre to an interpolated (noise-filled) + defect pixel (:func:`centre_distance`). A distance is -1 when no + kept epoch has such a pixel. They read only masks, so splitting a + catalogue on them selects on nothing shear-responsive. + + Parameters + ---------- + defect : numpy.ndarray of bool + The epoch's defect mask (:func:`defect_mask`). + interpolated : numpy.ndarray of bool + Its interpolated defect pixels (:func:`interpolated_defects`). + """ + distance = centre_distance(defect.shape) + noisefill = defect & ~interpolated + if interpolated.any(): + self.n_epoch_interp += 1 + self.min_dist_interp = min( + self.min_dist_interp, distance[interpolated].min() + ) + if noisefill.any(): + self.min_dist_noisefill = min( + self.min_dist_noisefill, distance[noisefill].min() + ) + + def defect_diagnostics(self): + """The defect diagnostic columns, -1 for an absent distance.""" + return { + "n_epoch_interp": self.n_epoch_interp, + "min_dist_interp": ( + float(self.min_dist_interp) + if np.isfinite(self.min_dist_interp) else -1.0 + ), + "min_dist_noisefill": ( + float(self.min_dist_noisefill) + if np.isfinite(self.min_dist_noisefill) else -1.0 + ), + } + class Vignet(): """Vignet. @@ -708,15 +832,22 @@ class Ngmix(object): adaptive-moment centroid measured from the stamp pixels. See :func:`make_ngmix_observation`. blend_handling : {"noisefill", "uberseg"}, optional - Neighbour treatment; ``"noisefill"`` (default) is the historical - noise-fill, ``"uberseg"`` hard-masks neighbour-side pixels from the - coadd segmentation map and requires ``seg_cat_path``. - seg_cat_path : str, optional - Path to the coadd-frame segmentation VIGNET catalogue (see - :class:`Tile_cat`). Required when ``blend_handling="uberseg"``. + Neighbour treatment. ``"noisefill"`` (default) zero-weights and + noise-fills the pixels marked -1e30 in the tile VIGNET on other + detections' footprints; ``"uberseg"`` ignores those markers and + zeroes the weight of neighbour-side pixels from the coadd + segmentation stamps, the tile catalogue's ``SEG_VIGNET`` column (see + :class:`Tile_cat`), which it requires. Defect pixels are filled + under both (see :func:`prepare_ngmix_weights`). dilate_neighbour : int, optional Neighbour-mask dilation iterations for ``"uberseg"`` (see - :func:`uberseg_weight`); the default is ``1``. + :func:`uberseg_mask`); the default is ``1``. + metacal_psf : str, optional + Metacal reconvolution-kernel scheme (see :func:`do_ngmix_metacal`). + defect_weighting : str, optional + How interpolated defects are weighted, one of ``DEFECT_WEIGHTINGS`` + (see :func:`defect_weighting_masks`); the default is + ``DEFECT_WEIGHTING``. Notes ----- @@ -728,8 +859,7 @@ class Ngmix(object): IndexError If the length of the input file list is incorrect ValueError - If ``blend_handling`` is unknown, or ``"uberseg"`` is selected without - ``seg_cat_path``. + If ``blend_handling`` or ``defect_weighting`` is unknown. """ @@ -747,9 +877,9 @@ def __init__( bkg_sub=True, centroid_source="wcs", blend_handling="noisefill", - seg_cat_path=None, dilate_neighbour=1, metacal_psf="fitgauss", + defect_weighting=DEFECT_WEIGHTING, ): # Base count = catalogue + vignets, excluding the f_wcs headers (passed @@ -769,12 +899,10 @@ def __init__( + f" {BLEND_HANDLINGS}" ) - # Fail fast at construction (not deep in the per-epoch loop) when - # uberseg is requested without its segmentation input (shapepipe#776). - if blend_handling == "uberseg" and seg_cat_path is None: + if defect_weighting not in DEFECT_WEIGHTINGS: raise ValueError( - "blend_handling='uberseg' requires SEG_VIGNET_PATH (the coadd" - + " SExtractor segmentation vignets); none configured." + f"Unknown DEFECT_WEIGHTING '{defect_weighting}'; expected one" + + f" of {DEFECT_WEIGHTINGS}" ) self._tile_cat_path = input_file_list[0] @@ -816,9 +944,9 @@ def __init__( self._bkg_sub = bkg_sub self._centroid_source = centroid_source self._blend_handling = blend_handling - self._seg_cat_path = seg_cat_path self._dilate_neighbour = dilate_neighbour self._metacal_psf = metacal_psf + self._defect_weighting = defect_weighting self._w_log = w_log @@ -922,12 +1050,21 @@ def compile_results(self, results): output_dict[name]["n_epoch_model"].append( results[idx]["n_epoch_model"] ) + output_dict[name]["n_epoch_failed"].append( + results[idx]["n_epoch_failed"] + ) output_dict[name]["mcal_types_fail"].append(mcal_types_fail) # Per-object blend flag (see process()); replicated across all # shear types like id / n_epoch_model / mcal_types_fail. output_dict[name]["neighbour_flag"].append( results[idx]["neighbour_flag"] ) + # Defect diagnostics (Postage_stamp.record_defects), likewise + # object-level and replicated. + for key in ( + "n_epoch_interp", "min_dist_interp", "min_dist_noisefill" + ): + output_dict[name][key].append(results[idx][key]) # ngmix 2.x reports the solver's function-evaluation count # (nfev, ~tens-hundreds; -1 on some failures), not the v1 # 1-5 retry count, so the column is named accordingly. Fits @@ -1200,9 +1337,29 @@ def process(self): @sc [decision:shape_measurement.fit_initialisation,decision:shape_measurement.ngmix_seed_mode] """ - tile_cat = Tile_cat(self._tile_cat_path, self._seg_cat_path) + tile_cat = Tile_cat( + self._tile_cat_path, + self._id_obj_min, + self._id_obj_max, + ) vignet_cat = self._vignet_cat + # Fail before the per-object loop, whose try/except would otherwise + # drop every object one by one (shapepipe#776). + if self._blend_handling == "uberseg" and tile_cat.seg is None: + raise ValueError( + "BLEND_HANDLING = uberseg needs the tile catalogue's" + + f" SEG_VIGNET column, which {self._tile_cat_path} lacks;" + + " write it at tile detection (SEG_VIGNET = True)." + ) + if tile_cat.seg is None and getattr(tile_cat, "vign", None) is not None: + self._w_log.warning( + f"{self._tile_cat_path} has no SEG_VIGNET column: every" + + " interior -1e30 marker is taken as a neighbour," + + " including the object's own CLEAN-merged fragments" + + " (neighbours-are-other-footprints)." + ) + check_wcs_centroid_offset( self._centroid_source, tile_cat, vignet_cat.gal_vign_cat ) @@ -1218,6 +1375,10 @@ def process(self): id_first = -1 id_last = -1 count_batch = 0 + epoch_cuts = Counter( + considered=0, masked_fraction=0, central_veto=0, failed=0 + ) + n_emptied = 0 saved_batch_cumul = 0 rows = chunk_rows( @@ -1251,10 +1412,13 @@ def process(self): self._bkg_sub, psf_obj, gal_obj, + blend_handling=self._blend_handling, ) + epoch_cuts.update(stamp.epoch_cuts) if len(stamp.gals) == 0: n_no_epoch += 1 + n_emptied += stamp.epoch_cuts["considered"] > 0 continue # Per-object RNG, seeded from (ra, dec, ccd) — see @@ -1296,6 +1460,7 @@ def process(self): object_number=obj_id, dilate_neighbour=self._dilate_neighbour, metacal_psf=self._metacal_psf, + defect_weighting=self._defect_weighting, ) except Exception as ee: self._w_log.info( @@ -1303,11 +1468,20 @@ def process(self): ) n_ngmix_fail += 1 continue + finally: + # Epochs do_ngmix_metacal dropped (failed-epoch-dropped). + epoch_cuts["failed"] += len(stamp.epoch_failures) + for epoch_name, err in stamp.epoch_failures: + self._w_log.warning( + f"ngmix: tile {self._file_number_string} object" + + f" {obj_id} epoch {epoch_name} dropped:" + + f" {type(err).__name__}: {err}" + ) res['obj_id'] = obj_id # Neighbour flag: does the coadd seg stamp hold any non-central, # non-zero label? Systematics-test hook (shapepipe#776); computed - # from the raw coadd seg, 0 when uberseg is off / seg absent. + # from the raw coadd seg, 0 when the seg is absent. res['neighbour_flag'] = int( seg_has_neighbour(tile_cat.seg[i_tile], obj_id) if getattr(tile_cat, "seg", None) is not None @@ -1316,6 +1490,8 @@ def process(self): # epochs that survived the PSF fit and entered the model, # not the number of epochs submitted (v1 contract) res['n_epoch_model'] = psf_res['n_epoch'] + res['n_epoch_failed'] = len(stamp.epoch_failures) + res.update(stamp.defect_diagnostics()) # The mcal flag columns are derived from the per-type results in # compile_results; here they only feed the run-health count. if get_mcal_flags(res) != 0: @@ -1364,6 +1540,14 @@ def process(self): + f" {n_fitted} fitted" ) + self._w_log.info( + "epoch cuts:" + + f" considered={epoch_cuts['considered']}" + + f" masked_fraction={epoch_cuts['masked_fraction']}" + + f" central_veto={epoch_cuts['central_veto']}" + + f" failed={epoch_cuts['failed']}" + + f" objects_emptied={n_emptied}" + ) log_run_health(self._w_log, count, n_fitted, n_flagged) vignet_cat.close() @@ -1385,11 +1569,54 @@ def prepare_postage_stamps( bkg_sub=True, psf_obj=None, gal_obj=None, + blend_handling="noisefill", ): - """Prepare the per-object lists of exposures passed to ngmix. + """Gather one object's epoch stamps, dropping epochs its defects spoil. + + @sc [decision:shape_measurement.central_defect_veto,decision:shape_measurement.epoch_masked_fraction_cut] epoch-cut-on-defect-mask + An epoch is dropped when more than ``EPOCH_MASKED_FRACTION_CUT`` of its + stamp lies in :func:`defect_mask`, the set :func:`prepare_ngmix_weights` + zero-weights and fills, or when :func:`central_defect_vetoes` finds a + defect near the object. The quarter-turn copies of interpolated pixels + that ``DEFECT_WEIGHTING`` zero-weights or interpolates are not counted. + + @sc [decision:shape_measurement.central_defect_veto,decision:shape_measurement.epoch_masked_fraction_cut,decision:shape_measurement.blend_handling] neighbour-markers-are-not-defects + The tile VIGNET's -1e30 markers on other objects' segmentation + footprints (:func:`split_tile_markers`) are the epoch's neighbour mask + (``stamp.neighbours``), not defects: every epoch shares the tile VIGNET, + so counting them would drop every epoch of a blended object. + + @sc [decision:shape_measurement.central_defect_veto,decision:shape_measurement.epoch_masked_fraction_cut,decision:shape_measurement.defect_fill] off-tile-pixels-are-defects + Beyond the tile's edge the epoch holds the object's own light, cut off by + the tile, so those pixels get zero exposure weight and join the defect + set under every ``blend_handling``. On a 51-px stamp an object within + about 8.5 px of the tile edge fails the masked-fraction cut. - @sc [decision:shape_measurement.central_defect_veto,decision:shape_measurement.epoch_masked_fraction_cut] + Parameters + ---------- + vignet : Vignet + Per-object vignet stores. + obj_id : int + Object ID (SExtractor ``NUMBER``). + i_tile : int + Row of the object in ``tile_cat``. + tile_cat : Tile_cat + Tile catalogue. + bkg_sub : bool, optional + Subtract the background vignet; the default is ``True``. + psf_obj, gal_obj : dict, optional + The object's PSF and galaxy vignet dicts, if already read. + blend_handling : {"noisefill", "uberseg"}, optional + The neighbour treatment :func:`prepare_ngmix_weights` will apply, + which decides which defects it can interpolate and so their veto + radius; the default is ``"noisefill"``. + + Returns + ------- + Postage_stamp + The surviving epochs' stamps. """ + # define per-object lists of individual exposures to go into ngmix stamp = Postage_stamp(bkg_sub=bkg_sub) # Read each store's per-object dict ONCE: every sqlitedict access # unpickles the object's whole all-epoch dict, so keeping these out of @@ -1456,19 +1683,27 @@ def prepare_postage_stamps( tile_seg = Ngmix.MegaCamFlip(tile_seg, int(ccd_n)) flag_vign = flag_obj[expccd_name]['VIGNET'] - if tile_vign is not None: - flag_vign[np.where(tile_vign == -1e30)] = 2**10 - v_flag_tmp = flag_vign.ravel() - # remove objects that are more than 1/3 masked - if len(np.where(v_flag_tmp != 0)[0]) / v_flag_tmp.size > 1 / 3.0: - continue - - weight_vign = weight_obj[expccd_name]['VIGNET'] + # Off-tile pixels are defects (off-tile-pixels-are-defects); the + # other -1e30 markers are neighbours (neighbour-markers-are-not-defects). + neighbour, off_tile = split_tile_markers( + tile_vign, np.shape(gal_vign), tile_seg, obj_id + ) + weight_vign = np.where(off_tile, 0, weight_obj[expccd_name]['VIGNET']) bkg_rms_vign = ( bkg_rms_obj[expccd_name]['VIGNET'] if bkg_rms_obj is not None else None ) + defect = defect_mask(weight_vign, flag_vign, bkg_rms_vign) + stamp.epoch_cuts["considered"] += 1 + if defect.mean() > EPOCH_MASKED_FRACTION_CUT: + stamp.epoch_cuts["masked_fraction"] += 1 + continue + interpolated = interpolated_defects(defect, neighbour, blend_handling) + if central_defect_vetoes(defect, interpolated): + stamp.epoch_cuts["central_veto"] += 1 + continue + stamp.record_defects(defect, interpolated) # One unpickle per exposure (all CCDs), reused across this object's # epochs; the cache is per-call, so bounded by the object's exposures @@ -1502,7 +1737,9 @@ def prepare_postage_stamps( stamp.psfs.append(psf_obj[expccd_name]['VIGNET']) stamp.weights.append(weight_vign_scaled) stamp.flags.append(flag_vign) + stamp.neighbours.append(neighbour) stamp.bkg_rms.append(bkg_rms_vign_scaled) + stamp.epoch_names.append(expccd_name) if tile_seg is not None: stamp.segs.append(tile_seg) stamp.jacobs.append(jacob) @@ -1511,9 +1748,7 @@ def prepare_postage_stamps( # make_ngmix_observation), which raises if it is missing; the "hsm" # path ignores it, so read it leniently rather than coupling hsm to a # field it never uses. - stamp.offsets.append( - vignet.gal_vign_cat[str(obj_id)][expccd_name].get('OFFSET') - ) + stamp.offsets.append(gal_obj[expccd_name].get('OFFSET')) stamp.ra.append(tile_cat.ra[i_tile]) stamp.dec.append(tile_cat.dec[i_tile]) # CCD of the first surviving epoch — Fabian's coord_list[0] convention @@ -1524,6 +1759,67 @@ def prepare_postage_stamps( return stamp +def split_tile_markers(tile_vign, shape, seg=None, object_number=None): + """Split the tile VIGNET's -1e30 markers into neighbour and off-tile. + + @sc [decision:shape_measurement.blend_handling,decision:shape_measurement.defect_fill] off-tile-is-marked-border-rows-and-columns + SExtractor writes -1e30 into the tile VIGNET on stamp pixels beyond the + tile's edge and, under MASK_TYPE CORRECT, on every above-threshold pixel + that is not the object's own pre-CLEAN footprint. A stamp clipped by the + tile's rectangle loses whole rows and whole columns from its border, so + the off-tile pixels are the union of the runs of entirely -1e30 rows and + columns that start at a stamp border. + + @sc [decision:shape_measurement.blend_handling] neighbours-are-other-footprints + The other markers also fall on pixels no other object owns: detections + smaller than DETECT_MINAREA (noise peaks, catalogued nowhere) and + fragments CLEAN merged into the object itself, which the segmentation + map labels as the object. With ``seg``, the neighbour pixels are the + markers on another label's footprint (any label but 0 and + ``object_number``; the catalogue join's -1 marks a detection that left + the catalogue). The markers left over are nothing: -1e30 is a flag in the + tile VIGNET only, and the epoch stamps hold real light there, the + object's own or sky. Without ``seg``, every interior marker is a + neighbour. + + Parameters + ---------- + tile_vign : numpy.ndarray or None + Tile VIGNET stamp, oriented like the epoch; ``None`` marks nothing. + shape : tuple of int + Stamp shape, used when ``tile_vign`` is ``None``. + seg : numpy.ndarray, optional + The object's SEG_VIGNET stamp, oriented like ``tile_vign``. + object_number : int, optional + The object's label in ``seg``; required with ``seg``. + + Returns + ------- + numpy.ndarray of bool + Neighbour pixels. + numpy.ndarray of bool + Off-tile pixels. + """ + if tile_vign is None: + return np.zeros(shape, dtype=bool), np.zeros(shape, dtype=bool) + marker = tile_vign == -1e30 + + def border_runs(full): + # Lines in the unbroken run of marked lines from either border. + lead = np.logical_and.accumulate(full) + trail = np.logical_and.accumulate(full[::-1])[::-1] + return lead | trail + + off_tile = ( + border_runs(marker.all(axis=1))[:, None] + | border_runs(marker.all(axis=0))[None, :] + ) + neighbour = marker & ~off_tile + if seg is not None: + neighbour &= (seg != 0) & (seg != object_number) + return neighbour, off_tile + + def background_subtract(gal,bkg): """background subtraction. @@ -1740,16 +2036,16 @@ def seg_has_neighbour(seg, object_number): return bool(labels.size and np.any(labels != object_number)) -def uberseg_weight(weight, seg, object_number, dilate_neighbour=0): - """Zero a stamp's weight on neighbour-side pixels — the UberSeg treatment. +def uberseg_mask(seg, object_number, dilate_neighbour=0): + """Neighbour-side pixels of a stamp: the UberSeg mask. UberSeg is Erin Sheldon's MEDS/ngmix neighbour mask (``esheldon/meds``, https://github.com/esheldon/meds — ``MEDS.get_uberseg`` / ``meds._uberseg.uberseg_tree``, in MEDS itself "adapted from Niall MacCrann and Joe Zuntz", and used in the DES shear pipeline). Each stamp pixel is assigned to the object whose segmentation footprint it lies nearest to — a - nearest-segment Voronoi partition — and only pixels assigned to the central - object keep their weight. + nearest-segment Voronoi partition — and the pixels assigned to a neighbour + are masked. This function **reimplements** the partition rather than depending on ``meds``: it is a five-line ``scipy.spatial.cKDTree`` nearest-neighbour @@ -1767,97 +2063,307 @@ def uberseg_weight(weight, seg, object_number, dilate_neighbour=0): Because the partition is by distance to the nearest footprint, the pixels surviving around a compact central object form a single connected, roughly circular core; the "circularisation" is emergent geometry, not a - separate aperture. Unlike the noise-fill treatment, masked pixels are - handed to ngmix as a hard mask (weight = 0), never replaced by a noise - realisation. + separate aperture. :func:`prepare_ngmix_weights` zeroes the weight of the + masked pixels and keeps their image values, so metacal shears the + neighbour's light along with the target's. Parameters ---------- - weight : numpy.ndarray - Per-pixel weight (inverse variance) map for the stamp. seg : numpy.ndarray - Segmentation map on the same grid as ``weight``: 0 for sky, the - SExtractor object number for each detected object's footprint. + Segmentation stamp: 0 for sky, the SExtractor object number for each + detected object's footprint. object_number : int Segmentation label of the central object — its SExtractor ``NUMBER`` (``obj_id``), authoritative because seg labels are the NUMBERs of the same SE run. Not the centre-pixel label, which a few-pixel coadd-vs- epoch offset can steal for a neighbour and so invert this mask. dilate_neighbour : int, optional - Enlarge the neighbour mask by this many binary-dilation iterations - (4-connected, ~one pixel per iteration) on top of the base Voronoi - partition, to absorb the few-pixel coadd-vs-epoch registration offset - the coadd-seg overlay accepts (shapepipe#776, decision on seg source). - ``0`` (the default) recovers the pure Sheldon UberSeg mask, byte-for- - byte. The dilation is additive: it only ever zeros more pixels, never - restores a base-masked one, so over-masking a boundary pixel costs a - little central-object S/N but never leaks neighbour flux into the fit. + Enlarge the mask by this many binary-dilation iterations of the + neighbour footprints (4-connected, ~one pixel per iteration) on top of + the Voronoi partition, to absorb the few-pixel coadd-vs-epoch + registration offset the coadd-seg overlay accepts (shapepipe#776, + decision on seg source). ``0`` (the default) is the pure Sheldon + UberSeg mask. The dilation only ever adds pixels, so over-masking a + boundary pixel costs a little central-object S/N but never leaks + neighbour flux into the fit. Returns ------- - numpy.ndarray - Copy of ``weight`` with neighbour-side pixels zeroed. + numpy.ndarray of bool + ``True`` on pixels nearer a neighbour's footprint than the central + object's; all ``False`` when the stamp holds no neighbour. @sc [decision:shape_measurement.blend_handling] """ - weight = np.copy(weight) + seg = np.asarray(seg) + masked = np.zeros(seg.shape, dtype=bool) obj_pix = np.argwhere(seg != 0) - # No detected footprint, or only sky plus the central object: there is - # no neighbour to mask, so the weight is unchanged (cf. MEDS' early - # ``len(np.unique(seg)) == 2`` return). - if obj_pix.shape[0] == 0: - return weight labels = seg[seg != 0] + # No neighbour footprint on the stamp: nothing to mask (cf. MEDS' early + # ``len(np.unique(seg)) == 2`` return). if np.all(labels == object_number): - return weight + return masked - # Nearest segmentation pixel for every stamp pixel; zero the weight - # wherever that nearest footprint belongs to a neighbour rather than to - # the central object. + # Nearest segmentation pixel for every stamp pixel; mask wherever that + # nearest footprint belongs to a neighbour rather than to the central + # object. grid = np.indices(seg.shape).reshape(2, -1).T _, nearest = cKDTree(obj_pix).query(grid) - nearest_label = labels[nearest].reshape(seg.shape) - weight[nearest_label != object_number] = 0.0 + masked |= labels[nearest].reshape(seg.shape) != object_number - # Enlarge the neighbour mask to absorb the coadd-vs-epoch offset: zero any - # pixel within ``dilate_neighbour`` of a neighbour footprint. Purely - # additive over the base Voronoi result above. if dilate_neighbour > 0: - neighbour_mask = binary_dilation( + masked |= binary_dilation( (seg != 0) & (seg != object_number), iterations=dilate_neighbour, ) - weight[neighbour_mask] = 0.0 - return weight + return masked + + +def defect_mask(weight, flag, bkg_rms=None): + """Defect pixels of one epoch stamp. + + @sc [decision:shape_measurement.defect_fill,label:physics] defect-set-unsymmetrized + A defect is a pixel with zero exposure weight (off-tile pixels + included), a nonzero exposure flag, or, when a background RMS map is + given, a non-finite or non-positive RMS. This one set is zero-weighted + and filled by :func:`prepare_ngmix_weights` and counted by the epoch + cuts. It is not ORed with its rotations: a four-fold fill quadruples the + filled area near the object and with it m (astra option + ``defect_fill.symmetrized_4fold_noise``). + + Parameters + ---------- + weight : numpy.ndarray + Exposure weight stamp. + flag : numpy.ndarray + Exposure flag stamp. + bkg_rms : numpy.ndarray, optional + Background RMS stamp. + + Returns + ------- + numpy.ndarray of bool + ``True`` on defect pixels. + """ + defect = (weight == 0) | (flag != 0) + if bkg_rms is not None: + defect |= ~(np.isfinite(bkg_rms) & (bkg_rms > 0)) + return defect + + +def interpolated_defects(defect, neighbour, blend_handling): + """The defect pixels :func:`prepare_ngmix_weights` interpolates. + + Under ``"noisefill"`` the neighbour pixels' light is replaced, so they + neither support the interpolant nor are interpolated + (:func:`interpolable_defects` with ``removed=neighbour``); under + ``"uberseg"`` their light stays and supports it. The epoch cuts + (:func:`central_defect_vetoes`) read the same set. + + Parameters + ---------- + defect : numpy.ndarray of bool + Defect mask of one epoch stamp (:func:`defect_mask`). + neighbour : numpy.ndarray of bool or None + The epoch's neighbour mask (``Postage_stamp.neighbours``); ``None`` + marks no pixel. + blend_handling : {"noisefill", "uberseg"} + Neighbour treatment. + + Returns + ------- + numpy.ndarray of bool + ``True`` on the defect pixels to interpolate; the other defect + pixels are noise-filled. + """ + removed = neighbour if blend_handling == "noisefill" else None + return interpolable_defects(defect, removed) + + +def central_defect_vetoes(defect, interpolated): + """Whether a defect near the stamp centre drops the epoch. + + @sc [decision:shape_measurement.central_defect_veto,decision:shape_measurement.defect_fill] veto-radius-follows-the-fill + The epoch is dropped when an interpolated defect pixel lies closer than + ``EPOCH_INTERPOLATED_DEFECT_RADIUS`` to the stamp centre, or a + noise-filled one closer than ``EPOCH_CENTRAL_DEFECT_RADIUS``: the + smallest radii at which the defects kept recover shear within + |m| < 1% and |c| < 5e-4. ``interpolated`` is the set + :func:`prepare_ngmix_weights` interpolates, so each pixel is vetoed at + the radius of the fill it gets. The veto reads only masks, so it selects + on nothing that responds to shear. Calibration: astra decision + ``shape_measurement.central_defect_veto``; guarded by + ``tests/science/test_defect_recovery.py``. + + Parameters + ---------- + defect : numpy.ndarray of bool + Defect mask of one epoch stamp (:func:`defect_mask`). + interpolated : numpy.ndarray of bool + The defect pixels that are interpolated + (:func:`interpolated_defects`); the rest are noise-filled. + + Returns + ------- + bool + ``True`` if the epoch should be dropped. + """ + radius = np.where( + interpolated, + EPOCH_INTERPOLATED_DEFECT_RADIUS, + EPOCH_CENTRAL_DEFECT_RADIUS, + ) + return bool(np.any(defect & (centre_distance(defect.shape) < radius))) + + +def centre_distance(shape): + """Distance of every stamp pixel from the stamp centre, in pixels.""" + rows, cols = np.indices(shape) + return np.hypot(rows - (shape[0] - 1) / 2, cols - (shape[1] - 1) / 2) + + +def defect_weighting_masks(interpolated, clean, defect_weighting): + """The pixels each DEFECT_WEIGHTING interpolates, re-weights and zeroes. + + @sc [decision:shape_measurement.defect_weighting,label:physics] one-place-decides-defect-weights + ``interpolated`` (:func:`interpolated_defects`) starts at weight 0, like + every defect. The options: + + * ``"fourfold_zero"``: interpolate it; the weight is also zeroed on its + three quarter-turn copies about the stamp centre, which keep their + light, so the likelihood hole has no spin-2 part. + * ``"des_y6"``: DES Y6's pizza-cutter symmetrization on a per-object + stamp. The mask is ORed with its quarter turn (``np.rot90``, k=1) + about the stamp centre, the union is interpolated, and every + interpolated pixel keeps its full weight. Copies that land on a pixel + whose light is already replaced (a noise-filled defect, a noisefill + neighbour) stay as they are, and so do copies the interpolant cannot + reach: a copy is interpolated only where it passes + :func:`interpolable_defects` among the pixels the image keeps, so a + copy wedged into a corner of the clean region (an edge band beside + the stamp edge) keeps its light rather than leaving the fill + non-finite. + * ``"hole"``: interpolate it and leave it at weight 0. + * ``"full"``: interpolate it at full weight. + + Parameters + ---------- + interpolated : numpy.ndarray of bool + The interpolable defect pixels of a square stamp. + clean : numpy.ndarray of bool + Pixels whose light the image keeps raw. + defect_weighting : str + One of ``DEFECT_WEIGHTINGS``. + + Returns + ------- + fill : numpy.ndarray of bool + Pixels to replace by the interpolant. + restore : numpy.ndarray of bool + Defect pixels that get their full inverse-variance weight back. + zero : numpy.ndarray of bool + Pixels whose weight is zeroed though their light stays. + + Raises + ------ + ValueError + If ``defect_weighting`` is unknown. + """ + none = np.zeros_like(interpolated) + if defect_weighting == "des_y6": + copy = interpolable_defects(np.rot90(interpolated) & clean, ~clean) + return interpolated | copy, interpolated, none + if defect_weighting == "fourfold_zero": + return interpolated, none, fourfold(interpolated) + if defect_weighting == "hole": + return interpolated, none, none + if defect_weighting == "full": + return interpolated, interpolated, none + raise ValueError( + f"Unknown DEFECT_WEIGHTING '{defect_weighting}'; expected one of" + + f" {DEFECT_WEIGHTINGS}" + ) + def prepare_ngmix_weights( gal, weight, flag, rng, bkg_rms=None, blend_handling="noisefill", seg=None, object_number=None, - dilate_neighbour=0, + dilate_neighbour=0, neighbour=None, defect_weighting=DEFECT_WEIGHTING, ): - """bookkeeping for ngmix weights. runs on a single galaxy and epoch - pixel scale and galaxy guess - TO DO: decide if we want galaxy guess stuff + """Build one epoch's image, weight map and noise image for ngmix. + + Every stamp pixel falls in one of four classes. A clean pixel keeps its + light and its weight. A pixel whose light is replaced by noise keeps + neither: a defect that is not interpolated, or under ``"noisefill"`` a + marked neighbour pixel. An interpolated pixel takes the interpolant, at + the weight ``defect_weighting`` gives it. A pixel whose light stays at + zero weight is a hole in the likelihood only: under ``"fourfold_zero"`` + the quarter-turn copies of the interpolated defects, under ``"uberseg"`` + a neighbour-side pixel. + + @sc [decision:shape_measurement.defect_fill,decision:shape_measurement.blend_handling] defects-filled-whatever-the-blend-handling + Every pixel of :func:`defect_mask` gets weight 0 and is filled the same + way under every ``blend_handling``. The short runs of + :func:`interpolated_defects` take a Clough-Tocher interpolant of the + pixels whose light the image keeps (:func:`interpolate_defects`), in the + image and the noise image alike; the other defects take noise at the + background RMS. Metacal shears the whole image whatever the weights, so + no raw defect value may reach it. + + @sc [decision:shape_measurement.blend_handling] noisefill-fills-markers + Under ``"noisefill"`` the ``neighbour`` pixels (the tile VIGNET's -1e30 + neighbour markers) get weight 0 and noise. Their light is gone, so they + neither support the interpolant nor receive it. + + @sc [decision:shape_measurement.blend_handling] uberseg-ignores-markers + Under ``"uberseg"`` the markers are ignored: the pixels of + :func:`uberseg_mask` get weight 0 and keep their light, which is real + sky that metacal shears with the target and which supports the + interpolant. + + @sc [decision:shape_measurement.defect_weighting] defect-weighting-option + ``defect_weighting`` decides which pixels are interpolated and how they + and their quarter-turn copies are weighted + (:func:`defect_weighting_masks`). Noise-filled pixels and the uberseg + neighbour side are never symmetrized, and an interpolated pixel on the + uberseg neighbour side stays at weight 0. Measurements: astra decision + ``shape_measurement.defect_weighting``; guarded by + ``tests/science/test_defect_recovery.py``. + + @sc [decision:shape_measurement.defect_weighting,label:physics] noise-image-in-the-fixnoise-frame + Metacal's ``fixnoise`` turns the noise image a quarter turn + (``np.rot90``, k=1) before shearing it and turns the result back (k=3) + before adding it to the sheared image. The noise image is therefore + interpolated in that turned frame, by the operator that interpolates the + science image: in the frame where both are sheared, the interpolated + pixels and their correlated noise coincide. Interpolating it in the + detector frame instead puts its interpolated pixels a quarter turn away + from the science image's, and the correlated noise of an interpolated + column then survives the cancellation as an additive bias. + ``tests/module/test_ngmix_defect_fill.py`` pins ngmix's k=1/k=3. Parameters ---------- gal : numpy.ndarray + Background-subtracted galaxy stamp. weight : numpy.ndarray + Exposure weight stamp; zero marks a defect. flag : numpy.ndarray + Exposure flag stamp; nonzero marks a defect. rng : numpy.random.RandomState Random state for the noise realisations (seeded per object; see :func:`position_seed`). bkg_rms : numpy.ndarray, optional - Per-pixel background RMS map. If supplied, unmasked pixels use - ``1 / bkg_rms**2`` as the ngmix inverse variance. + Per-pixel background RMS map. If supplied, clean pixels use + ``1 / bkg_rms**2`` as the ngmix inverse variance, and non-finite or + non-positive values mark defects. Otherwise every clean pixel gets + ``1 / sigma_mad(gal)**2``. blend_handling : {"noisefill", "uberseg"}, optional - How to treat pixels shared with a neighbour. ``"noisefill"`` (default) - replaces flagged pixels with a noise realisation and keeps their - inverse-variance weight — the historical behaviour. ``"uberseg"`` - instead hard-masks (weight = 0) every pixel closer to a neighbour's - segmentation footprint than to the central object's, leaving the - image untouched (see :func:`uberseg_weight`). + Neighbour treatment. ``"noisefill"`` (default) zero-weights and + noise-fills the ``neighbour`` pixels. ``"uberseg"`` ignores + ``neighbour``, zeroes the weight of the pixels of + :func:`uberseg_mask` and keeps their raw image values. seg : numpy.ndarray, optional Segmentation map on the stamp grid (object NUMBERs). Required for ``blend_handling="uberseg"``; ignored otherwise. @@ -1865,51 +2371,89 @@ def prepare_ngmix_weights( Central object's segmentation label. Required for ``blend_handling="uberseg"``; ignored otherwise. dilate_neighbour : int, optional - Neighbour-mask dilation iterations, passed to :func:`uberseg_weight` + Neighbour-mask dilation iterations, passed to :func:`uberseg_mask` under ``blend_handling="uberseg"``; ignored otherwise. + neighbour : numpy.ndarray of bool, optional + The epoch's neighbour mask (``Postage_stamp.neighbours``); read only + under ``blend_handling="noisefill"``. ``None`` marks no pixel. + defect_weighting : str, optional + One of ``DEFECT_WEIGHTINGS``; the default is ``DEFECT_WEIGHTING``. Returns ------- numpy.ndarray - Galaxy image. For ``"noisefill"`` masked pixels are replaced by noise; - for ``"uberseg"`` the image is returned untouched. + Galaxy image with defect pixels, and under noisefill marked + neighbour pixels, filled. numpy.ndarray - Variance map for NGMIX. + Inverse-variance weight map for ngmix. numpy.ndarray - Noise image. - @sc [decision:masking.pixel_mask_source,decision:shape_measurement.blend_handling,decision:shape_measurement.defect_fill,decision:shape_measurement.galaxy_pixel_weights] + Noise image: an independent realisation over the whole stamp, for + metacal's ``fixnoise``, interpolated in fixnoise's quarter-turned + frame where the galaxy image is interpolated. + + Raises + ------ + ValueError + If ``blend_handling`` or ``defect_weighting`` is unknown, or + ``"uberseg"`` lacks ``seg`` or ``object_number``. + RuntimeError + If the interpolant does not reach a pixel :func:`interpolated_defects` + selected (degenerate support, or non-finite image values in it). + @sc [decision:masking.pixel_mask_source,decision:shape_measurement.blend_handling,decision:shape_measurement.defect_fill,decision:shape_measurement.galaxy_pixel_weights,decision:shape_measurement.defect_weighting] """ if blend_handling not in BLEND_HANDLINGS: raise ValueError( f"Unknown blend_handling '{blend_handling}'; expected one of" + f" {BLEND_HANDLINGS}" ) + if defect_weighting not in DEFECT_WEIGHTINGS: + raise ValueError( + f"Unknown defect_weighting '{defect_weighting}'; expected one of" + + f" {DEFECT_WEIGHTINGS}" + ) + if blend_handling == "uberseg" and (seg is None or object_number is None): + raise ValueError( + "blend_handling='uberseg' requires a segmentation map and the" + + " central object_number; none reached prepare_ngmix_weights." + + " The tile catalogue's SEG_VIGNET column carries the map (see" + + " CosmoStat/shapepipe#776)." + ) - mask = np.copy(weight) != 0 - mask[flag != 0] = False + defect = defect_mask(weight, flag, bkg_rms) + no_pixel = np.zeros_like(defect) + # Neighbour pixels whose light noisefill replaces (noisefill-fills-markers) + # or whose weight uberseg zeroes (uberseg-ignores-markers). + removed_neighbour, neighbour_side = no_pixel, no_pixel + if blend_handling == "noisefill" and neighbour is not None: + removed_neighbour = np.asarray(neighbour, dtype=bool) + elif blend_handling == "uberseg": + neighbour_side = uberseg_mask(seg, object_number, dilate_neighbour) + # Pixels whose light the image keeps raw. + clean = ~(defect | removed_neighbour) + weighted = clean & ~neighbour_side if bkg_rms is None: sig_noise = sigma_mad(gal) # Guard the degenerate constant stamp (sigma_mad == 0): 0 * inf # would otherwise put NaN in a fully-masked weight map. weight_map = ( - mask.astype(float) / sig_noise ** 2 + weighted.astype(float) / sig_noise ** 2 if sig_noise > 0 else np.zeros_like(gal, dtype=float) ) else: - valid_rms = np.isfinite(bkg_rms) & (bkg_rms > 0) - mask &= valid_rms weight_map = np.zeros_like(gal, dtype=float) - weight_map[mask] = 1.0 / bkg_rms[mask] ** 2 + weight_map[weighted] = 1.0 / bkg_rms[weighted] ** 2 # Per-pixel noise sigma for the realisations below: metacal's # fixnoise bookkeeping (1/w + 1/w_noise) assumes the noise image # is a faithful realisation of the per-pixel variance the weights # claim; a scalar sigma there mis-reports errors and erodes the # inverse-variance advantage whenever the RMS map actually varies. + # Pixels without a valid RMS take the median over clean pixels. + valid_rms = np.isfinite(bkg_rms) & (bkg_rms > 0) sig_noise = ( - np.where(valid_rms, bkg_rms, np.median(bkg_rms[mask])) - if mask.any() + np.where(valid_rms, bkg_rms, np.median(bkg_rms[clean])) + if clean.any() else sigma_mad(gal) ) @@ -1922,34 +2466,38 @@ def prepare_ngmix_weights( noise_img = rng.standard_normal(gal.shape) * sig_noise noise_img_gal = rng.standard_normal(gal.shape) * sig_noise - - gal_masked = np.copy(gal) - if blend_handling == "uberseg": - # Hard-mask neighbour-side pixels (weight -> 0) from the segmentation - # geometry; the image is left untouched (the masked pixels carry no - # weight, so ngmix ignores them in the likelihood). Bad/flagged - # pixels already sit at weight 0 from the mask above. - if seg is None or object_number is None: - raise ValueError( - "blend_handling='uberseg' requires a segmentation map and the" - + " central object_number; none reached prepare_ngmix_weights." - + " Set SEG_VIGNET_PATH on the ngmix run (see" - + " CosmoStat/shapepipe#776)." - ) - weight_map = uberseg_weight( - weight_map, seg, object_number, dilate_neighbour=dilate_neighbour + gal_filled = np.where(clean, gal, noise_img_gal).astype(gal.dtype) + interpolated = interpolated_defects(defect, neighbour, blend_handling) + if interpolated.any(): + fill, restore, zero = defect_weighting_masks( + interpolated, clean, defect_weighting + ) + # One operator for the image and the noise image, the latter in + # fixnoise's quarter-turned frame (noise-image-in-the-fixnoise-frame). + filled = interpolate_defects( + [gal, np.rot90(noise_img)], ~clean | fill, fill ) - elif (~mask).any(): - # noisefill (default): replace masked pixels with a noise realisation. - gal_masked[~mask] = noise_img_gal[~mask] + if not np.all(np.isfinite(filled[:, fill])): + raise RuntimeError( + "The defect interpolant is not finite on pixels" + + " defect_weighting_masks selected." + ) + gal_filled[fill] = filled[0, fill] + noise_img = np.rot90(filled[1], -1) + restore = restore & ~neighbour_side + weight_map[restore] = np.broadcast_to( + 1.0 / np.asarray(sig_noise) ** 2, gal.shape + )[restore] + weight_map[zero] = 0.0 + + return gal_filled, weight_map, noise_img - return gal_masked, weight_map, noise_img def make_ngmix_observation( gal, weight, flag, psf, wcs, rng, bkg_rms=None, centroid_source="wcs", offset=None, blend_handling="noisefill", seg=None, object_number=None, - dilate_neighbour=0, + dilate_neighbour=0, neighbour=None, defect_weighting=DEFECT_WEIGHTING, ): """Build an ngmix Observation for a single galaxy epoch. @@ -1993,7 +2541,8 @@ def make_ngmix_observation( ``"hsm"``). blend_handling : {"noisefill", "uberseg"}, optional Neighbour treatment passed through to :func:`prepare_ngmix_weights`; - the default ``"noisefill"`` is the historical behaviour. + the default ``"noisefill"`` zero-weights and noise-fills the + ``neighbour`` pixels. seg : numpy.ndarray, optional Segmentation map on the stamp grid. Required for ``blend_handling="uberseg"`` (ignored otherwise). @@ -2003,6 +2552,11 @@ def make_ngmix_observation( dilate_neighbour : int, optional Neighbour-mask dilation iterations passed through to :func:`prepare_ngmix_weights` under ``blend_handling="uberseg"``. + neighbour : numpy.ndarray of bool, optional + Neighbour mask passed through to :func:`prepare_ngmix_weights`. + defect_weighting : str, optional + Interpolated-defect weighting passed through to + :func:`prepare_ngmix_weights`. Returns ------- @@ -2031,16 +2585,18 @@ def make_ngmix_observation( gal_masked, weight_map, noise_img = prepare_ngmix_weights( gal, weight, flag, rng, bkg_rms=bkg_rms, blend_handling=blend_handling, seg=seg, object_number=object_number, - dilate_neighbour=dilate_neighbour, + dilate_neighbour=dilate_neighbour, neighbour=neighbour, + defect_weighting=defect_weighting, ) if centroid_source == "hsm": # Re-center the Jacobian on the HSM adaptive-moment centroid (pixel - # offset from the stamp center); fall back to the stamp center if - # HSM fails. + # offset from the stamp center), measured on the filled image so no + # raw defect value pulls it; fall back to the stamp center if HSM + # fails. try: _hsm = galsim.hsm.FindAdaptiveMom( - galsim.Image(gal, scale=1.0), strict=False + galsim.Image(gal_masked, scale=1.0), strict=False ) if _hsm.error_message != "": raise galsim.hsm.GalSimHSMError(_hsm.error_message) @@ -2261,13 +2817,21 @@ def make_runners(prior, flux_guess, rng): def do_ngmix_metacal( stamp, prior, flux_guess, rng, centroid_source="wcs", blend_handling="noisefill", object_number=None, dilate_neighbour=0, - metacal_psf="fitgauss", + metacal_psf="fitgauss", defect_weighting=DEFECT_WEIGHTING, ): """Do Ngmix Metacal. Performs metacalibration on a single multi-epoch object and returns the joint shape measurement with NGMIX. + @sc [decision:shape_measurement.epoch_masked_fraction_cut,label:operations] failed-epoch-dropped + An epoch whose observation raises while being built (for example the + non-finite interpolant guard in :func:`prepare_ngmix_weights`) is + dropped and the object is fitted on its other epochs. The epoch's name + and exception are appended to ``stamp.epoch_failures``; the caller logs + them and writes their count as ``n_epoch_failed``. Errors outside + building the observations propagate. + Parameters ---------- stamp : Postage_stamp @@ -2286,9 +2850,9 @@ def do_ngmix_metacal( stamp pixels — see that function. blend_handling : {"noisefill", "uberseg"}, optional Neighbour treatment passed through to - :func:`make_ngmix_observation`; the default ``"noisefill"`` is the - historical behaviour. ``"uberseg"`` consumes ``stamp.segs`` and - ``object_number``. + :func:`make_ngmix_observation`; the default ``"noisefill"`` + zero-weights and noise-fills the pixels of ``stamp.neighbours``. + ``"uberseg"`` consumes ``stamp.segs`` and ``object_number``. object_number : int, optional Central object's segmentation label — its SExtractor ``NUMBER`` (``obj_id``), authoritative because seg labels are the NUMBERs of the @@ -2308,6 +2872,9 @@ def do_ngmix_metacal( round PSF that metacal reconvolves with after shearing, so it moves the metacal *response* (and therefore the recovered shear) but never reaches the deconvolution, which is by the PSF image. + defect_weighting : str, optional + Interpolated-defect weighting (one of ``DEFECT_WEIGHTINGS``) passed + through to :func:`make_ngmix_observation`. Returns ------- @@ -2318,7 +2885,12 @@ def do_ngmix_metacal( dict (:func:`average_original_psf`). The two PSF dicts share keys but describe different PSFs; the named fields guard against transposing them. Unpacks positionally as ``resdict, psf_res, psf_orig_res``. - @sc [decision:shape_measurement.defect_fill,decision:shape_measurement.metacal_scheme] + + Raises + ------ + ValueError + If the stamp has no epoch, or every epoch failed to build. + @sc [decision:shape_measurement.defect_fill,decision:shape_measurement.metacal_scheme,decision:shape_measurement.defect_weighting] """ n_epoch = len(stamp.gals) if n_epoch == 0: @@ -2327,22 +2899,41 @@ def do_ngmix_metacal( gal_obs_list = ObsList() for n_e in range(n_epoch): bkg_rms = stamp.bkg_rms[n_e] if len(stamp.bkg_rms) > n_e else None - gal_obs = make_ngmix_observation( - stamp.gals[n_e], - stamp.weights[n_e], - stamp.flags[n_e], - stamp.psfs[n_e], - stamp.jacobs[n_e], - rng, - bkg_rms=bkg_rms, - centroid_source=centroid_source, - offset=stamp.offsets[n_e] if n_e < len(stamp.offsets) else None, - blend_handling=blend_handling, - seg=stamp.segs[n_e] if n_e < len(stamp.segs) else None, - object_number=object_number, - dilate_neighbour=dilate_neighbour, - ) + try: + gal_obs = make_ngmix_observation( + stamp.gals[n_e], + stamp.weights[n_e], + stamp.flags[n_e], + stamp.psfs[n_e], + stamp.jacobs[n_e], + rng, + bkg_rms=bkg_rms, + centroid_source=centroid_source, + offset=( + stamp.offsets[n_e] if n_e < len(stamp.offsets) else None + ), + blend_handling=blend_handling, + seg=stamp.segs[n_e] if n_e < len(stamp.segs) else None, + object_number=object_number, + dilate_neighbour=dilate_neighbour, + neighbour=( + stamp.neighbours[n_e] + if n_e < len(stamp.neighbours) else None + ), + defect_weighting=defect_weighting, + ) + except Exception as err: + stamp.epoch_failures.append(( + stamp.epoch_names[n_e] + if n_e < len(stamp.epoch_names) else str(n_e), + err, + )) + continue gal_obs_list.append(gal_obs) + if len(gal_obs_list) == 0: + raise ValueError( + f"all {n_epoch} epochs failed to build an observation" + ) runner, psf_runner = make_runners(prior, flux_guess, rng) diff --git a/src/shapepipe/modules/ngmix_runner.py b/src/shapepipe/modules/ngmix_runner.py index 6d15905fa..0037db34d 100644 --- a/src/shapepipe/modules/ngmix_runner.py +++ b/src/shapepipe/modules/ngmix_runner.py @@ -11,7 +11,11 @@ from sqlitedict import SqliteDict from shapepipe.modules.module_decorator import module_runner -from shapepipe.modules.ngmix_package.ngmix import Ngmix, write_empty_tile_output +from shapepipe.modules.ngmix_package.ngmix import ( + DEFECT_WEIGHTING, + Ngmix, + write_empty_tile_output, +) @module_runner( @@ -44,7 +48,7 @@ def ngmix_runner( ): """Define The Ngmix Runner. - @sc [decision:shape_measurement.blend_handling,decision:shape_measurement.centroid_source,decision:shape_measurement.defect_fill,decision:shape_measurement.metacal_scheme,decision:shape_measurement.galaxy_pixel_weights] + @sc [decision:shape_measurement.blend_handling,decision:shape_measurement.centroid_source,decision:shape_measurement.defect_fill,decision:shape_measurement.metacal_scheme,decision:shape_measurement.galaxy_pixel_weights,decision:shape_measurement.defect_weighting] @sc [label:operations] empty-tile-product A tile whose PSF or galaxy vignette store is entirely empty never @@ -86,24 +90,6 @@ def ngmix_runner( else: input_file_list = input_file_list[:wcs_idx] - # SEG_VIGNET_PATH (optional): coadd-frame SExtractor segmentation vignets - # (a CLASSIC-mode vignetmaker output), row-aligned to the tile catalogue. - # Required for BLEND_HANDLING = uberseg; when set, the file must exist for - # every tile (missing file -> error). Read on Tile_cat, not via Vignet, so - # it is threaded to Ngmix as its own argument rather than into - # input_file_list. - if config.has_option(module_config_sec, "SEG_VIGNET_PATH"): - seg_vignet_path = config.getexpanded( - module_config_sec, - "SEG_VIGNET_PATH", - ).format(file_number_string=file_number_string) - if not os.path.exists(seg_vignet_path): - raise FileNotFoundError( - f"Segmentation vignet file not found: {seg_vignet_path}" - ) - else: - seg_vignet_path = None - # Batch save option if config.has_option(module_config_sec, "SAVE_BATCH"): save_batch = config.getint( @@ -130,15 +116,32 @@ def ngmix_runner( else: centroid_source = "wcs" - # Neighbour treatment: "noisefill" (default, historical) replaces a - # neighbour's pixels with a noise realisation; "uberseg" hard-masks - # (weight -> 0) every pixel closer to a neighbour than to the central - # object, from the segmentation map. See the ngmix module docstrings. + # Neighbour treatment: "noisefill" (default) zero-weights and noise-fills + # the pixels marked -1e30 in the tile VIGNET on other objects' + # segmentation footprints (split_tile_markers); "uberseg" ignores those + # markers, zeroes the weight of every + # pixel closer to a neighbour than to the central object, from the + # segmentation map, and leaves its image raw. Defect pixels (flagged, + # zero-weight, invalid-RMS or off-tile) are zero-weighted and filled the + # same way under both, and the epoch cuts are fixed; see + # prepare_postage_stamps and prepare_ngmix_weights. if config.has_option(module_config_sec, "BLEND_HANDLING"): - blend_handling = config.get(module_config_sec, "BLEND_HANDLING") + blend_handling = config.getexpanded(module_config_sec, "BLEND_HANDLING") else: blend_handling = "noisefill" + # DEFECT_WEIGHTING (optional, environment-expanded): how interpolated + # defects are weighted, one of DEFECT_WEIGHTINGS (see + # defect_weighting_masks). Absent or empty takes DEFECT_WEIGHTING; the + # workflow sets it from the run config's `defect_weighting`. + defect_weighting = DEFECT_WEIGHTING + if config.has_option(module_config_sec, "DEFECT_WEIGHTING"): + defect_weighting = ( + config.getexpanded(module_config_sec, "DEFECT_WEIGHTING").strip() + or DEFECT_WEIGHTING + ) + w_log.info(f"DEFECT_WEIGHTING = {defect_weighting}") + # DILATE_NEIGHBOUR (optional): binary-dilation iterations enlarging the # uberseg neighbour mask, to absorb the few-pixel coadd-vs-epoch seg-overlay # offset. Ignored unless BLEND_HANDLING = uberseg. Default 1 (~one pixel). @@ -204,9 +207,9 @@ def ngmix_runner( bkg_sub=bkg_sub, centroid_source=centroid_source, blend_handling=blend_handling, - seg_cat_path=seg_vignet_path, dilate_neighbour=dilate_neighbour, metacal_psf=metacal_psf, + defect_weighting=defect_weighting, ) # Process ngmix shape measurement and metacalibration diff --git a/src/shapepipe/modules/sextractor_package/match_catalogue.py b/src/shapepipe/modules/sextractor_package/match_catalogue.py index 68bca78f7..6c59d31e7 100644 --- a/src/shapepipe/modules/sextractor_package/match_catalogue.py +++ b/src/shapepipe/modules/sextractor_package/match_catalogue.py @@ -23,9 +23,13 @@ # The SEG_VIGNET label of a footprint whose SExtractor row has no partner in # the external catalogue, and so leaves the catalogue. Negative, so it never -# collides with a NUMBER; UberSeg only asks "self or not self". +# collides with a NUMBER (match_catalogue requires positive ones); UberSeg only +# asks "self or not self". UNMATCHED_LABEL = -1 +# The largest NUMBER the int32 NUMBER and SEG_VIGNET columns hold. +MAX_NUMBER = np.iinfo(np.int32).max + def mutual_nearest(x_a, y_a, x_b, y_b, radius): """One-to-one pairs of mutual nearest neighbours closer than ``radius``. @@ -100,9 +104,12 @@ def match_catalogue(cat_path, ext_cat_path, radius=1.0, min_fraction=0.98, external catalogue (``X_IMAGE``, ``Y_IMAGE``, same image grid) within ``radius`` pixels. Paired rows take the external ``NUMBER``; unpaired rows leave the catalogue. A ``SEG_VIGNET`` column, when present, is - relabelled to the new numbering, with the footprints of rows that left - marked ``UNMATCHED_LABEL``. The catalogue is rewritten in place; every - other HDU and column is kept. + relabelled through the whole map from old to new numbers, with the + footprints of rows that left marked ``UNMATCHED_LABEL``: as the external + numbers are unique integers in ``[1, MAX_NUMBER]``, each row's own + footprint carries its new ``NUMBER`` and no other footprint can, + whatever the two numberings share. + The catalogue is rewritten in place; every other HDU and column is kept. Both catalogues come from the same pixels, so pairs agree to ~1e-4 pixel, and on eight DR6 tiles at least 99.2% of each side pairs; the @@ -144,13 +151,25 @@ def match_catalogue(cat_path, ext_cat_path, radius=1.0, min_fraction=0.98, Raises ------ ValueError - If more than ``tolerated_unpaired`` rows of either side, and more - than ``1 - min_fraction`` of it, have no partner + If the external ``NUMBER`` repeats or is not an integer in + ``[1, MAX_NUMBER]``, or if more + than ``tolerated_unpaired`` rows of either side, and more than + ``1 - min_fraction`` of it, have no partner @sc [decision:detection.tile_detection] """ ext = asc.read(ext_cat_path, format="sextractor", include_names=["NUMBER", "X_IMAGE", "Y_IMAGE"]) + ext_number = np.asarray(ext["NUMBER"]) + if (not np.issubdtype(ext_number.dtype, np.integer) + or (ext_number <= 0).any() or (ext_number > MAX_NUMBER).any() + or len(np.unique(ext_number)) < len(ext)): + raise ValueError( + f"{ext_cat_path} has a NUMBER that is not an integer in" + + f" [1, {MAX_NUMBER}] or that repeats; the join needs unique" + + " numbers that fit the int32 NUMBER and SEG_VIGNET columns, so" + + " that no relabelled footprint takes another object's number." + ) with fits.open(cat_path) as hdul: hdus = [hdu.copy() for hdu in hdul] objects = next(h for h in hdus if h.name == "LDAC_OBJECTS") @@ -194,7 +213,7 @@ def too_many_unpaired(n_total): old_number = np.asarray(data["NUMBER"]) new_number = np.full(len(data), UNMATCHED_LABEL, np.int64) - new_number[i_sex] = np.asarray(ext["NUMBER"])[i_ext] + new_number[i_sex] = ext_number[i_ext] columns = [] for col in objects.columns: diff --git a/src/shapepipe/modules/sextractor_package/sextractor_script.py b/src/shapepipe/modules/sextractor_package/sextractor_script.py index 002bf14f7..8d05d7edb 100644 --- a/src/shapepipe/modules/sextractor_package/sextractor_script.py +++ b/src/shapepipe/modules/sextractor_package/sextractor_script.py @@ -16,6 +16,190 @@ from shapepipe.pipeline.sqlite_store import read_sqlitedict +def cut_stamps(array, col, row, stamp_size, fill): + """Cut one square stamp per object from a 2-D array. + + Each stamp is ``stamp_size`` pixels on a side with the 0-based pixel + (``row``, ``col``) at index ``stamp_size // 2`` on both axes; pixels off + the array take ``fill``. This is the window SExtractor cuts ``VIGNET`` + with, so stamps cut at the same centres from arrays on one pixel grid are + registered pixel for pixel. + + Parameters + ---------- + array : numpy.ndarray + 2-D array, shape ``(ny, nx)`` + col, row : array_like of int + 0-based column and row of each stamp's centre pixel + stamp_size : int + Side length of the stamps + fill : scalar + Value of the stamp pixels off the array + + Returns + ------- + numpy.ndarray + Stamps, shape ``(n_obj, stamp_size, stamp_size)``, dtype of ``array`` + + """ + ny, nx = array.shape + half = stamp_size // 2 + stamps = np.full((len(col), stamp_size, stamp_size), fill, array.dtype) + for i, (xi, yi) in enumerate(zip(col, row)): + x0, y0 = xi - half, yi - half + xc0, xc1 = max(0, x0), min(nx, x0 + stamp_size) + yc0, yc1 = max(0, y0), min(ny, y0 + stamp_size) + if xc0 >= xc1 or yc0 >= yc1: + continue + stamps[i, yc0 - y0:yc1 - y0, xc0 - x0:xc1 - x0] = ( + array[yc0:yc1, xc0:xc1] + ) + return stamps + + +def seg_vignet_column(seg_vignets): + """The ``SEG_VIGNET`` LDAC column: one int32 seg stamp per object. + + Parameters + ---------- + seg_vignets : numpy.ndarray + Segmentation stamps, shape ``(n_obj, stamp_size, stamp_size)``, cut + on ``VIGNET``'s grid (:func:`cut_stamps`) + + Returns + ------- + astropy.io.fits.Column + The column, laid out as ``VIGNET`` is + + """ + n_obj, ny, nx = seg_vignets.shape + return fits.Column( + name="SEG_VIGNET", + format=f"{ny * nx}J", + array=seg_vignets.astype(np.int32, copy=False).reshape(n_obj, -1), + dim=f"({nx},{ny})", + ) + + +# Double-precision positions add_seg_vignet centres SEG_VIGNET on. +DOUBLE_POSITIONS = ("X_IMAGE_DBL", "Y_IMAGE_DBL") + + +def vignet_centre(pos): + """The 0-based pixel SExtractor centres VIGNET on, along one axis. + + SExtractor 2.25.0 (``src/analyse.c``, ``ix=(int)(obj->mx+0.49999)``, + unchanged in Debian's 2.25.0+ds-3) truncates the 0-based barycentre plus + 0.49999. That is not round-to-nearest: a fractional part in + [0.5, 0.50001) goes down, where ``np.rint`` would go up (or to even). + + Parameters + ---------- + pos : array_like of float + 1-based double-precision position (``X_IMAGE_DBL`` or + ``Y_IMAGE_DBL``; SExtractor writes ``mx + 1``) + + Returns + ------- + numpy.ndarray + 0-based pixel index, int64 + + """ + mx = np.asarray(pos, np.float64) - 1.0 + return np.trunc(mx + 0.49999).astype(np.int64) + + +def seg_vignet_param_file(dot_param, output_path): + """Write a SExtractor parameter file that also asks for the double + positions. + + SExtractor centres VIGNET from its double-precision barycentre + (:func:`vignet_centre`), which the float32 ``X_IMAGE`` / ``Y_IMAGE`` + cannot resolve near half pixels; ``X_IMAGE_DBL`` / ``Y_IMAGE_DBL`` + carry it. + :func:`add_seg_vignet` reads them and drops them again. + + Parameters + ---------- + dot_param : str + Path to the configured parameter file + output_path : str + Path to write the extended parameter file to + + Returns + ------- + str + ``output_path`` + + """ + with open(dot_param) as f: + text = f.read() + if text and not text.endswith("\n"): + text += "\n" + with open(output_path, "w") as f: + f.write(text + "".join(f"{name}\n" for name in DOUBLE_POSITIONS)) + return output_path + + +def add_seg_vignet(cat_path, seg_path, w_log=None): + """Add the ``SEG_VIGNET`` column to a SExtractor catalogue. + + The SEGMENTATION check image, whose labels are the catalogue's + ``NUMBER``, is cut on the grid of each object's VIGNET, centred where + SExtractor centres VIGNET (:func:`vignet_centre` of ``X_IMAGE_DBL`` / + ``Y_IMAGE_DBL``, :func:`cut_stamps`), and written, int32 and 0 off the image, as + ``SEG_VIGNET`` in ``LDAC_OBJECTS``, which ngmix's UberSeg blend handling + reads. The double positions (:func:`seg_vignet_param_file`) are dropped; + every other HDU and column is kept. + + Parameters + ---------- + cat_path : str + Path to the SExtractor FITS-LDAC catalogue, rewritten in place + seg_path : str + Path to the SEGMENTATION check image + w_log : logging.Logger, optional + Pipeline logger + + Raises + ------ + ValueError + If the catalogue lacks the double positions + + """ + with fits.open(cat_path) as hdul: + hdus = [hdu.copy() for hdu in hdul] + objects = next(h for h in hdus if h.name == "LDAC_OBJECTS") + data = objects.data + missing = [name for name in DOUBLE_POSITIONS if name not in data.names] + if missing: + raise ValueError( + f"{cat_path} lacks {', '.join(missing)}, which SEG_VIGNET is" + + " centred on; run SExtractor with seg_vignet_param_file." + ) + col = vignet_centre(data["X_IMAGE_DBL"]) + row = vignet_centre(data["Y_IMAGE_DBL"]) + size = data["VIGNET"].shape[1] + seg_vignets = cut_stamps(fits.getdata(seg_path), col, row, size, 0) + kept = [c for c in objects.columns.columns + if c.name not in DOUBLE_POSITIONS] + new = fits.BinTableHDU.from_columns( + kept + [seg_vignet_column(seg_vignets)], + header=objects.header, + name="LDAC_OBJECTS", + ) + fits.HDUList( + [new if h is objects else h for h in hdus] + ).writeto(cat_path, overwrite=True) + if w_log: + centre = size // 2 + own = np.mean(seg_vignets[:, centre, centre] == data["NUMBER"]) + w_log.info( + f"SEG_VIGNET cut from {seg_path} for {len(col)} objects;" + + f" centre label = NUMBER for {own:.4f}" + ) + + def get_header_value(image_path, key): """Get Header Value. @@ -536,6 +720,7 @@ def get_check_image(self, check_image): if (len(check_image) == 1) & (check_image[0] == ""): check_type = ["NONE"] check_name = ["none"] + self.check_paths = {} else: check_type = [] check_name = [] @@ -549,6 +734,7 @@ def get_check_image(self, check_image): + self._num_str + ".fits" ) + self.check_paths = dict(zip(check_type, check_name)) self._cmd_line_extra += ( f' -CHECKIMAGE_TYPE {",".join(check_type)} ' diff --git a/src/shapepipe/modules/sextractor_runner.py b/src/shapepipe/modules/sextractor_runner.py index 65181712f..493bc5c96 100644 --- a/src/shapepipe/modules/sextractor_runner.py +++ b/src/shapepipe/modules/sextractor_runner.py @@ -79,6 +79,24 @@ def sextractor_runner( f_wcs_path = input_file_list[-1] input_file_list = list(input_file_list[:-1]) + # SEG_VIGNET (optional, environment-expanded boolean): add the + # SEGMENTATION check image's stamps, on each VIGNET's grid, as the + # SEG_VIGNET column ngmix's UberSeg blend handling reads. SExtractor then + # also writes the double-precision positions VIGNET is centred on, which + # add_seg_vignet reads and drops. + seg_vignet = config.has_option( + module_config_sec, "SEG_VIGNET" + ) and config.getexpandedboolean(module_config_sec, "SEG_VIGNET") + if seg_vignet: + if "SEGMENTATION" not in [key.upper() for key in check_image]: + raise ValueError( + "SEG_VIGNET needs the SEGMENTATION check image in CHECKIMAGE." + ) + dot_param = ss.seg_vignet_param_file( + dot_param, + f"{run_dirs['tmp']}/seg_vignet{file_number_string}.param", + ) + # Create sextractor caller class instance ss_inst = ss.SExtractorCaller( input_file_list, @@ -110,6 +128,15 @@ def sextractor_runner( # Parse SExtractor errors stdout, stderr = ss_inst.parse_errors(stderr, stdout) + # SEG_VIGNET is cut before the join, which relabels its stamps with the + # rows' new NUMBERs. + if seg_vignet: + ss.add_seg_vignet( + ss_inst.path_output_file, + ss_inst.check_paths["SEGMENTATION"], + w_log=w_log, + ) + # MATCH_CATALOGUE (optional, environment-expanded path; empty for none): # take membership and NUMBER from that external catalogue of the same # image, before the post-processing keys the epoch HDUs on NUMBER. diff --git a/src/shapepipe/pipeline/config.py b/src/shapepipe/pipeline/config.py index 2c294ae65..1f49fe043 100644 --- a/src/shapepipe/pipeline/config.py +++ b/src/shapepipe/pipeline/config.py @@ -97,6 +97,29 @@ def getexpanded(self, section, option, **kwargs): """ return self._get(section, _expandvars_strict, option, **kwargs) + def getexpandedboolean(self, section, option, **kwargs): + """Get Expanded Boolean. + + Expand enviroment variables in the value, then read it as a boolean + the way ``getboolean`` does. + + Parameters + ---------- + section : str + Configuration file section + option : str + Configuration file option + + Returns + ------- + bool + The expanded value as a boolean + + """ + return self._convert_to_boolean( + self.getexpanded(section, option, **kwargs) + ) + def getlist(self, section, option, delimiter=",", **kwargs): """Get List. diff --git a/tests/helpers/defect_response.py b/tests/helpers/defect_response.py new file mode 100644 index 000000000..3d10d1dbf --- /dev/null +++ b/tests/helpers/defect_response.py @@ -0,0 +1,142 @@ +"""Full-matrix metacal recovery for fixed detector defects and neighbours.""" + +import galsim +import numpy as np + +from shapepipe.modules.ngmix_package import ngmix as ngm +from shapepipe.testing.simulate import make_data +from tests.helpers.metacal_sim import build_stamp + + +N = 51 +PIXEL_SCALE = 0.1857 +FLUX = 1000.0 + + +def neighbour_scene(offset, shear, psf, psf_shear, hlr=0.5, flux_ratio=1.0, + seg_radius=3): + """A neighbour's image and the stamp's segmentation map. + + The neighbour is an exponential galaxy whose centre lies ``offset`` + (row, col) pixels from the stamp centre, sheared with the scene (its + profile and its position) and convolved with the detector-frame PSF. + The segmentation map labels a disk of ``seg_radius`` px around the + stamp centre 1 (the target) and one around the unsheared neighbour + position 2. + """ + row, col = offset + matrix = galsim.Shear(g1=shear[0], g2=shear[1]).getMatrix() + dx, dy = matrix @ np.array([col, row]) * PIXEL_SCALE + image = galsim.Convolve( + galsim.Moffat(beta=2.5, fwhm=psf).shear( + g1=psf_shear[0], g2=psf_shear[1]), + galsim.Exponential(half_light_radius=hlr, flux=FLUX * flux_ratio) + .shear(g1=shear[0], g2=shear[1]), + ).shift(dx, dy).drawImage(nx=N, ny=N, scale=PIXEL_SCALE).array + rows, cols = np.indices((N, N)) - N // 2 + seg = np.zeros((N, N), dtype=np.int32) + seg[rows ** 2 + cols ** 2 <= seg_radius ** 2] = 1 + seg[(rows - row) ** 2 + (cols - col) ** 2 <= seg_radius ** 2] = 2 + return image, seg + + +def defect_response(bad, hlr=0.5, psf=0.7, seeds=range(4), + psf_shear=(0.0, 0.0), options=None, known_rms=True, + defect_value=1e3, neighbour=None): + """Recover c and M = inverse(mean R) A - I with paired seed errors. + + Null pairs rotate the pixels by 90 degrees while pre-rotating the PSF + ellipticity oppositely, so the final PSF stays fixed in detector + coordinates. + This does NOT average away elliptical-PSF leakage. + + The flagged pixels ``bad`` hold ``defect_value`` (far above the galaxy's + peak) rather than sky, as a hot column or bleed would, so any defect value + that reaches metacal shows up as a bias. + + ``neighbour``, if given, holds the keyword arguments of + :func:`neighbour_scene` but ``shear``, ``psf`` and ``psf_shear``: a + neighbour fixed in detector coordinates like the defect. Its + segmentation footprint is the stamp's neighbour mask under + ``blend_handling="noisefill"``, and the segmentation map rides the stamp + for ``"uberseg"`` (pass ``object_number=1`` in ``options``). + + The result also holds ``s2n``, the mean over arms of the no-shear fit's + S/N, the price of the pixels a treatment drops. + """ + gamma = 0.02 + options = {} if options is None else options + samples, s2n = [], [] + for seed in seeds: + def arm(axis, sign, rotation=0): + shear = [0.0, 0.0] + if axis >= 0: + shear[axis] = sign * gamma + ps = tuple(v * (-1 if rotation else 1) for v in psf_shear) + data = list(make_data( + rng=np.random.RandomState(seed + 100), shear=shear, + psf_shear=ps, noise=1e-4, n_epochs=1, img_size=51, + gal_hlr=hlr, psf_fwhm=psf, return_centers=True, + )) + centre = data.pop()[0] + offset = np.array([centre.y - 26, centre.x - 26]) + if rotation: + data[0] = [np.rot90(a).copy() for a in data[0]] + data[1] = [np.rot90(a).copy() for a in data[1]] + offset = np.array([-offset[1], offset[0]]) + seg = None + if neighbour is not None: + light, seg = neighbour_scene( + shear=shear, psf=psf, psf_shear=psf_shear, **neighbour) + data[0] = [a + light for a in data[0]] + data[0] = [np.where(bad, defect_value, a) for a in data[0]] + data[4] = [bad.astype(np.int32)] + stamp = build_stamp(data) + stamp.offsets = [offset] + if seg is not None: + stamp.segs = [seg] + stamp.neighbours = [seg == 2] + if known_rms: + stamp.bkg_rms = [np.full((51, 51), 1e-4)] + rng = np.random.RandomState(seed) + result, _, _ = ngm.do_ngmix_metacal( + stamp, ngm.get_prior(0.1857, rng), 1.0, rng, + centroid_source="wcs", **options, + ) + assert all(result[t]["flags"] == 0 for t in ngm.METACAL_TYPES) + s2n.append(result["noshear"]["s2n"]) + e = np.asarray(result["noshear"]["g"]) + response = np.column_stack([ + (np.asarray(result[p]["g"]) - result[m]["g"]) / 0.02 + for p, m in (("1p", "1m"), ("2p", "2m")) + ]) + assert np.all(np.isfinite(e)) and np.all(np.isfinite(response)) + return e, response + null = [arm(-1, 0, k) for k in (0, 1)] + e0 = np.mean([a[0] for a in null], axis=0) + r0 = np.mean([a[1] for a in null], axis=0) + derivatives, responses = [], [] + for axis in (0, 1): + plus, minus = arm(axis, 1), arm(axis, -1) + derivatives.append((plus[0] - minus[0]) / (2 * gamma)) + responses.extend([plus[1], minus[1]]) + samples.append((e0, r0, np.column_stack(derivatives), + np.mean(responses, axis=0))) + e, r, a, rm = [np.array([s[i] for s in samples]) for i in range(4)] + assert np.linalg.svd(rm.mean(axis=0), compute_uv=False).min() > 0.1 + c = np.linalg.solve(r.mean(axis=0), e.mean(axis=0)) + matrix = np.linalg.solve(rm.mean(axis=0), a.mean(axis=0)) - np.eye(2) + draw = np.random.RandomState(91).randint(len(e), size=(1000, len(e))) + cb = np.linalg.solve(r[draw].mean(axis=1), + e[draw].mean(axis=1)[..., None])[..., 0] + mb = np.linalg.solve(rm[draw].mean(axis=1), a[draw].mean(axis=1)) + mb -= np.eye(2) + return dict(c=c.tolist(), m=np.diag(matrix).tolist(), + matrix=matrix.tolist(), + c_err=cb.std(axis=0).tolist(), + m_err=np.diag(mb.std(axis=0)).tolist(), + response=r.mean(axis=0).tolist(), + s2n=float(np.mean(s2n)), + seed_groups=[dict(e=s[0].tolist(), R=s[1].tolist(), + A=s[2].tolist(), Rm=s[3].tolist()) + for s in samples]) diff --git a/tests/module/test_defect_interpolation.py b/tests/module/test_defect_interpolation.py new file mode 100644 index 000000000..791bb1c15 --- /dev/null +++ b/tests/module/test_defect_interpolation.py @@ -0,0 +1,247 @@ +"""Defect interpolation: which pixels are interpolated, and the properties +of the interpolant. + +:func:`interpolable_defects` picks the defect pixels that lie in a short row +or column run with support pixels at both ends; :func:`interpolate_defects` +fills them from the nearby clean pixels with one Clough-Tocher interpolant +shared by every plane (science image and metacal noise image). +""" + +import numpy as np +import numpy.testing as npt +import pytest +from hypothesis import given, settings +from hypothesis import strategies as st + +from shapepipe.modules.ngmix_package.defect_interpolation import ( + MAX_INTERPOLATED_RUN, + fourfold, + interpolable_defects, + interpolate_defects, +) + + +# --- interpolable_defects --------------------------------------------------- + +def _oracle(defect, removed, max_run=MAX_INTERPOLATED_RUN): + """Brute force: walk each kept defect pixel's row and column run.""" + n0, n1 = defect.shape + blocked = defect | removed + defect = defect & ~removed + out = np.zeros_like(defect) + for i, j in zip(*np.nonzero(defect)): + for di, dj in ((0, 1), (1, 0)): + lo, hi = (i, j), (i, j) + while (0 <= lo[0] - di and 0 <= lo[1] - dj + and defect[lo[0] - di, lo[1] - dj]): + lo = (lo[0] - di, lo[1] - dj) + while (hi[0] + di < n0 and hi[1] + dj < n1 + and defect[hi[0] + di, hi[1] + dj]): + hi = (hi[0] + di, hi[1] + dj) + length = hi[0] - lo[0] + hi[1] - lo[1] + 1 + bounded = (lo[0] - di >= 0 and lo[1] - dj >= 0 + and hi[0] + di < n0 and hi[1] + dj < n1 + and not blocked[lo[0] - di, lo[1] - dj] + and not blocked[hi[0] + di, hi[1] + dj]) + if bounded and length <= max_run: + out[i, j] = True + return out + + +@given( + n=st.integers(5, 21), + density=st.floats(0.0, 0.6), + removed_density=st.sampled_from([0.0, 0.1, 0.4]), + seed=st.integers(0, 2**31 - 1), +) +@settings(max_examples=60, deadline=None) +def test_interpolable_defects_are_the_short_bounded_runs( + n, density, removed_density, seed, +): + """A defect pixel is interpolated exactly when its row or column run is + at most MAX_INTERPOLATED_RUN long and the pixels beyond both ends are on + the stamp and neither defects nor removed; the rule commutes with + quarter turns. + + Failure modes: a run touching the stamp border (edge band, corner), a + wide hole, or a run ending on a removed neighbour pixel is interpolated + from one side; a 3-px bleed is left to noise; a clean or removed pixel + is selected; one axis is ignored, so the rule has a preferred direction. + """ + rng = np.random.RandomState(seed) + defect = rng.uniform(size=(n, n)) < density + removed = rng.uniform(size=(n, n)) < removed_density + out = interpolable_defects(defect, removed) + npt.assert_array_equal(out, _oracle(defect, removed)) + assert not out[~defect | removed].any() + for k in range(1, 4): + npt.assert_array_equal( + interpolable_defects(np.rot90(defect, k), np.rot90(removed, k)), + np.rot90(out, k), + ) + + +@given( + n=st.integers(5, 31), + density=st.floats(0.0, 0.6), + removed_density=st.sampled_from([0.0, 0.1, 0.4]), + seed=st.integers(0, 2**31 - 1), +) +@settings(max_examples=60, deadline=None) +def test_the_interpolant_reaches_every_interpolable_pixel( + n, density, removed_density, seed, +): + """Each selected pixel lies between two support pixels, so the + interpolant is finite there; prepare_ngmix_weights relies on this to + veto and fill the same pixels. + + Failure mode: a selected pixel falls outside the support's hull and gets + NaN, so the fill and the central-defect veto disagree on it. + """ + rng = np.random.RandomState(seed) + defect = rng.uniform(size=(n, n)) < density + removed = rng.uniform(size=(n, n)) < removed_density + target = interpolable_defects(defect, removed) + planes = rng.normal(size=(2, n, n)) + out = interpolate_defects(planes, defect | removed, target) + assert np.all(np.isfinite(out[:, target])) + + +@pytest.mark.parametrize("kind,expected", [ + ("column", True), ("bleed3", True), ("finite_bleed3", True), + ("pixel", True), ("bleed4", False), ("blob5", False), + ("edge_band", False), ("corner", False), +]) +def test_calibrated_widths_are_interpolated_and_wider_holes_are_not( + kind, expected, +): + """Columns, 3-px bleeds and single pixels are interpolated; 4-px bleeds, + blobs, edge bands and corners are not.""" + n, c = 51, 25 + defect = np.zeros((n, n), dtype=bool) + region = { + "column": np.s_[:, c + 8], + "bleed3": np.s_[:, c + 8:c + 11], + "finite_bleed3": np.s_[c - 5:c + 6, c + 8:c + 11], + "pixel": np.s_[c, c + 8], + "bleed4": np.s_[:, c + 8:c + 12], + "blob5": np.s_[c - 2:c + 3, c + 8:c + 13], + "edge_band": np.s_[:, -3:], + "corner": np.s_[:2, :2], + }[kind] + defect[region] = True + out = interpolable_defects(defect) + assert out[defect].all() if expected else not out.any() + + +# --- fourfold --------------------------------------------------------------- + +@given(n=st.integers(2, 20), seed=st.integers(0, 2**31 - 1)) +@settings(max_examples=30, deadline=None) +def test_fourfold_is_the_quarter_turn_orbit(n, seed): + """The union contains the mask, is invariant under quarter turns, and is + the smallest such set (idempotent).""" + mask = np.random.RandomState(seed).uniform(size=(n, n)) < 0.1 + out = fourfold(mask) + assert out[mask].all() + for k in range(1, 4): + npt.assert_array_equal(np.rot90(out, k), out) + npt.assert_array_equal(fourfold(out), out) + npt.assert_array_equal( + out, mask | np.rot90(mask) | np.rot90(mask, 2) | np.rot90(mask, 3) + ) + + +def test_fourfold_rejects_rectangular_stamps(): + with pytest.raises(ValueError, match="square"): + fourfold(np.zeros((11, 12), dtype=bool)) + + +# --- interpolate_defects ---------------------------------------------------- + +def _mask(n=31): + """A column, a finite 3-px bleed and a single pixel, all bounded.""" + defect = np.zeros((n, n), dtype=bool) + defect[:, 19] = True + defect[4:12, 7:10] = True + defect[22, 11] = True + return defect + + +@given(seed=st.integers(0, 2**31 - 1)) +@settings(max_examples=20, deadline=None) +def test_interpolation_reproduces_planes_without_reading_defects(seed): + """Linear planes are reproduced at the interpolated pixels; clean pixels, + and defect pixels not selected, are returned untouched; the inputs are + not modified; and no defect value (NaN or a sentinel) is ever read. + + Failure modes: defect pixels enter the support; coordinates are + transposed or mis-rotated; the fill smooths clean pixels. + """ + n = 31 + rng = np.random.RandomState(seed) + rows, cols = np.indices((n, n)) + coeff = rng.uniform(-2, 2, (2, 3)) + planes = np.array([a + b * rows + c * cols for a, b, c in coeff]) + defect = _mask(n) + defect[:, -2:] = True # an edge band and a blob next to the column: + defect[13:18, 21:26] = True # defects that are not targets + target = interpolable_defects(defect) + assert target[15, 19] + assert not target[13:18, 21:26].any() and not target[:, -2:].any() + contaminated = planes.copy() + contaminated[:, defect] = np.nan + saved = contaminated.copy() + + out = interpolate_defects(contaminated, defect, target) + + npt.assert_allclose(out[:, target], planes[:, target], atol=1e-5) + npt.assert_array_equal(out[:, ~target], contaminated[:, ~target]) + npt.assert_array_equal(contaminated, saved) + contaminated[:, defect] = 1e30 + npt.assert_array_equal( + interpolate_defects(contaminated, defect, target)[:, target], + out[:, target], + ) + + +def test_every_plane_sees_the_same_operator(): + """The fill is one linear operator applied to every plane: filling + a * image + b * noise gives a * fill(image) + b * fill(noise), up to the + Clough-Tocher gradient solver's tolerance. + + Failure mode: the noise image is filled differently from the science + image (another support or triangulation), so metacal's + fixnoise no longer mirrors the science image's correlated noise. + """ + n = 31 + rng = np.random.RandomState(11) + image, noise = rng.normal(size=(2, n, n)) + defect = _mask(n) + target = interpolable_defects(defect) + both = interpolate_defects(np.array([image, noise]), defect, target) + mixed = interpolate_defects( + np.array([2.0 * image - 3.0 * noise]), defect, target + ) + npt.assert_allclose(mixed[0], 2.0 * both[0] - 3.0 * both[1], atol=1e-5) + alone = interpolate_defects(noise[None], defect, target) + npt.assert_allclose(alone[0], both[1], atol=1e-5) + + +def test_no_target_is_a_no_op(): + planes = np.random.RandomState(5).normal(size=(2, 15, 15)) + defect = np.zeros((15, 15), dtype=bool) + defect[:, -3:] = True + out = interpolate_defects(planes, defect, np.zeros_like(defect)) + npt.assert_array_equal(out, planes) + + +def test_degenerate_support_gives_nan_not_an_error(): + """With fewer than three non-collinear clean pixels the target is NaN, + which the caller replaces by noise.""" + defect = np.ones((5, 5), dtype=bool) + defect[2, 1] = defect[2, 3] = False + target = np.zeros_like(defect) + target[2, 2] = True + out = interpolate_defects(np.ones((1, 5, 5)), defect, target) + assert np.isnan(out[0, 2, 2]) diff --git a/tests/module/test_make_cat.py b/tests/module/test_make_cat.py index e29109d19..857f743eb 100644 --- a/tests/module/test_make_cat.py +++ b/tests/module/test_make_cat.py @@ -56,8 +56,10 @@ def warning(self, message, *_args, **_kwargs): NGMIX_KEYS = [ "id", "n_epoch_model", + "n_epoch_failed", "mcal_types_fail", "neighbour_flag", + "n_epoch_interp", "min_dist_interp", "min_dist_noisefill", "nfev_fit", "g1", "g1_err", "g2", "g2_err", "T", "T_err", @@ -85,8 +87,11 @@ def _ngmix_row(obj_id): return { "id": obj_id, "n_epoch_model": 3, + "n_epoch_failed": 1, "mcal_types_fail": 0, "neighbour_flag": 1, + "n_epoch_interp": 2, "min_dist_interp": 7.5, + "min_dist_noisefill": 14.25, "nfev_fit": 7, "g1": 0.10, "g1_err": 0.011, "g2": -0.20, "g2_err": 0.022, "T": 0.30, "T_err": 0.033, @@ -117,8 +122,10 @@ def _write_ngmix_cat(path, obj_ids): cols = [ fits.Column( name=key, - format="K" if key in ("id", "n_epoch_model", "mcal_types_fail", - "nfev_fit", "flags", "mcal_flags") else "D", + format="K" if key in ("id", "n_epoch_model", "n_epoch_failed", + "mcal_types_fail", + "n_epoch_interp", "nfev_fit", "flags", + "mcal_flags") else "D", array=np.array([row[key] for row in rows]), ) for key in NGMIX_KEYS @@ -176,7 +183,8 @@ def test_save_ngmix_data_uses_new_grammar_and_no_old_names(tmp_path): # Object-level metadata columns carry no OBJECT/SHEAR token. for col in ( "NGMIX_MCAL_FLAGS", "NGMIX_N_EPOCH", "NGMIX_MCAL_TYPES_FAIL", - "NGMIX_NEIGHBOUR_FLAG", + "NGMIX_NEIGHBOUR_FLAG", "NGMIX_N_EPOCH_FAILED", "NGMIX_N_EPOCH_INTERP", + "NGMIX_MIN_DIST_INTERP", "NGMIX_MIN_DIST_NOISEFILL", ): assert col in out, f"missing {col}" @@ -275,6 +283,16 @@ def test_save_ngmix_data_fills_sentinels_for_absent_objects(tmp_path): npt.assert_allclose(n_epoch[present], row["n_epoch_model"]) npt.assert_allclose(n_epoch[absent], [0.0, 0.0]) + for col, key, never_fit in ( + ("NGMIX_N_EPOCH_FAILED", "n_epoch_failed", 0.0), + ("NGMIX_N_EPOCH_INTERP", "n_epoch_interp", 0.0), + ("NGMIX_MIN_DIST_INTERP", "min_dist_interp", -1.0), + ("NGMIX_MIN_DIST_NOISEFILL", "min_dist_noisefill", -1.0), + ): + values = np.asarray(out[col]) + npt.assert_allclose(values[present], row[key]) + npt.assert_allclose(values[absent], [never_fit, never_fit]) + def test_low_match_fraction_warns_and_continues_with_sentinels(tmp_path): """A low match count warns while unmatched detections stay in the output.""" @@ -313,7 +331,11 @@ def _metacal_result(obj_id): res = { "obj_id": obj_id, "n_epoch_model": row["n_epoch_model"], + "n_epoch_failed": row["n_epoch_failed"], "neighbour_flag": row["neighbour_flag"], + "n_epoch_interp": row["n_epoch_interp"], + "min_dist_interp": row["min_dist_interp"], + "min_dist_noisefill": row["min_dist_noisefill"], } for key in NGMIX_KEYS: if key.endswith("_psf_orig") or key.endswith("_psf_reconv"): diff --git a/tests/module/test_ngmix.py b/tests/module/test_ngmix.py index 959d007f1..348b4909c 100644 --- a/tests/module/test_ngmix.py +++ b/tests/module/test_ngmix.py @@ -1,5 +1,7 @@ """UNIT TESTS FOR MODULE PACKAGE: NGMIX.""" +from collections import Counter + from astropy.io import fits from astropy.wcs import WCS import galsim @@ -326,8 +328,12 @@ def _fake_metacal_result(T, T_err, T_psf, T_psf_err): res = { "obj_id": 1, "n_epoch_model": 1, + "n_epoch_failed": 0, "mcal_types_fail": 0, "neighbour_flag": 0, + "n_epoch_interp": 0, + "min_dist_interp": -1.0, + "min_dist_noisefill": -1.0, # original image PSF (psfex/mccd) family "g1_psf_orig": ORIG_PSF_G[0], "g2_psf_orig": ORIG_PSF_G[1], @@ -629,7 +635,11 @@ def test_process_counts_flagged_fits_across_batches(tmp_path, monkeypatch, flags galaxies = {str(i): {"exp-1": {"OFFSET": [0., 0.]}} for i in tile.obj_id} stamp = SimpleNamespace( gals=[np.ones((5, 5))], ra=[42.], dec=[30.], ccd=20, + epoch_cuts=Counter(considered=1), epoch_failures=[], jacobs=[galsim.JacobianWCS(.186, 0., 0., .186)], + defect_diagnostics=lambda: dict( + n_epoch_interp=0, min_dist_interp=-1.0, min_dist_noisefill=-1.0, + ), ) psf = dict( n_epoch=1, g_psf=[.01, -.01], g_psf_err=[.001, .001], @@ -642,13 +652,14 @@ def test_process_counts_flagged_fits_across_batches(tmp_path, monkeypatch, flags results.append((result, psf, psf)) fits_to_return = iter(results) monkeypatch.setattr(module, "Tile_cat", lambda *args: tile) - monkeypatch.setattr(module, "prepare_postage_stamps", lambda *args: stamp) + monkeypatch.setattr( + module, "prepare_postage_stamps", lambda *args, **kwargs: stamp, + ) monkeypatch.setattr( module, "do_ngmix_metacal", lambda *args, **kwargs: next(fits_to_return), ) inst = object.__new__(Ngmix) inst._tile_cat_path = "in-memory-tile" - inst._seg_cat_path = None inst._vignet_cat = SimpleNamespace( gal_vign_cat=galaxies, psf_vign_cat=galaxies, close=lambda: None, ) @@ -658,6 +669,7 @@ def test_process_counts_flagged_fits_across_batches(tmp_path, monkeypatch, flags inst._blend_handling = "noisefill" inst._dilate_neighbour = 1 inst._metacal_psf = "fitgauss" + inst._defect_weighting = module.DEFECT_WEIGHTING inst._save_batch = 1 inst._zero_point = 30. inst._output_dir = str(tmp_path) @@ -725,6 +737,7 @@ def test_process_centroid_prior_is_each_objects_own_pixel_scale(monkeypatch): obj_id: SimpleNamespace( gals=[np.ones((5, 5))] * len(jacobs), jacobs=jacobs, ra=[10. * obj_id], dec=[30.], ccd=obj_id, + epoch_cuts=Counter(considered=len(jacobs)), epoch_failures=[], ) for obj_id, jacobs in epochs.items() } @@ -737,14 +750,13 @@ def capture(stamp, prior, flux_guess, rng, **kwargs): monkeypatch.setattr(module, "Tile_cat", lambda *args: tile) monkeypatch.setattr( module, "prepare_postage_stamps", - lambda vignet, obj_id, *args: stamps[obj_id], + lambda vignet, obj_id, *args, **kwargs: stamps[obj_id], ) monkeypatch.setattr(module, "do_ngmix_metacal", capture) monkeypatch.setattr(Ngmix, "save_results", lambda self, res: None) monkeypatch.setattr(Ngmix, "log_mean_ellipticity", lambda self: None) inst = object.__new__(Ngmix) inst._tile_cat_path = "in-memory-tile" - inst._seg_cat_path = None inst._vignet_cat = SimpleNamespace( gal_vign_cat=galaxies, psf_vign_cat=galaxies, close=lambda: None, ) @@ -754,6 +766,7 @@ def capture(stamp, prior, flux_guess, rng, **kwargs): inst._blend_handling = "noisefill" inst._dilate_neighbour = 1 inst._metacal_psf = "fitgauss" + inst._defect_weighting = module.DEFECT_WEIGHTING inst._save_batch = -1 inst._w_log = _RecordingLogger() @@ -1112,10 +1125,12 @@ def test_background_rms_builds_per_pixel_inverse_variance(): gal, weight, flag, np.random.RandomState(0), bkg_rms=bkg_rms ) + # The bad-RMS pixel (1, 2) is interpolated from (0, 2) and (2, 2), so + # its quarter turns (0, 1) and (1, 0) lose their weight too. expected = np.array( [ - [1.0, 0.25, 0.0625], - [4.0, 0.0, 0.0], + [1.0, 0.0, 0.0625], + [0.0, 0.0, 0.0], [0.0, 0.0, 1.0], ] ) @@ -1258,3 +1273,110 @@ def test_wcs_centroid_requires_offset(): gal, weight, flag, psf, jacob, np.random.RandomState(7), centroid_source="wcs", offset=None, ) + + +def test_failed_epoch_is_dropped_not_the_object(): + """Contract failed-epoch-dropped: one epoch whose observation raises is + dropped, and the object is fitted on its other epochs. + + The poisoned epoch has one flagged pixel, so it is interpolated, beside + NaN pixels that support the interpolant: the non-finite-interpolant guard + in prepare_ngmix_weights raises a RuntimeError for that epoch alone. + """ + from shapepipe.modules.ngmix_package.ngmix import ( + Postage_stamp, + do_ngmix_metacal, + get_prior, + ) + from shapepipe.testing.simulate import make_data + + rng = np.random.RandomState(5) + gals, psfs, _, weights, flags, jacobs = make_data( + rng=np.random.RandomState(123), shear=(0.02, 0.0), noise=1e-4, + n_epochs=3, img_size=51, + ) + gals = [np.array(g, dtype=float) for g in gals] + flags = [np.array(f) for f in flags] + flags[1][5, 5] = 1 + gals[1][4:7, 4:7] = np.nan + gals[1][5, 5] = 0.0 + stamp = Postage_stamp(bkg_sub=False, megacam_flip=False) + stamp.gals, stamp.psfs, stamp.weights, stamp.flags, stamp.jacobs = ( + gals, psfs, weights, flags, jacobs, + ) + stamp.bkg_rms = [np.full(g.shape, 1e-4) for g in gals] + stamp.epoch_names = ["2000001-1", "2000002-1", "2000003-1"] + + res, psf_res, _ = do_ngmix_metacal( + stamp, get_prior(0.1857, rng), 1.0, rng, centroid_source="hsm", + ) + + assert [name for name, _ in stamp.epoch_failures] == ["2000002-1"] + assert isinstance(stamp.epoch_failures[0][1], RuntimeError) + assert psf_res["n_epoch"] == 2 + assert res["noshear"]["flags"] == 0 + assert np.all(np.isfinite(res["noshear"]["g"])) + + +def test_process_logs_and_counts_failed_epochs(tmp_path, monkeypatch): + """process logs each dropped epoch with tile, object and epoch, and + writes the count as n_epoch_failed (contract failed-epoch-dropped).""" + from types import SimpleNamespace + from shapepipe.modules.ngmix_package import ngmix as module + + tile = SimpleNamespace(obj_id=[7], flux=None, seg=None) + galaxies = {"7": {"exp-1": {"OFFSET": [0., 0.]}}} + stamp = SimpleNamespace( + gals=[np.ones((5, 5))] * 2, ra=[42.], dec=[30.], ccd=20, + epoch_cuts=Counter(considered=3), + epoch_failures=[], + jacobs=[galsim.JacobianWCS(.186, 0., 0., .186)] * 2, + defect_diagnostics=lambda: dict( + n_epoch_interp=0, min_dist_interp=-1.0, min_dist_noisefill=-1.0, + ), + ) + psf = dict( + n_epoch=2, g_psf=[.01, -.01], g_psf_err=[.001, .001], + T_psf=.1, T_psf_err=.01, + ) + + def fit(stamp, *args, **kwargs): + stamp.epoch_failures.append(("2000002-1", RuntimeError("poisoned"))) + return _fake_metacal_result(.18, .02, .09, .001), psf, psf + + monkeypatch.setattr(module, "Tile_cat", lambda *args: tile) + monkeypatch.setattr( + module, "prepare_postage_stamps", lambda *args, **kwargs: stamp, + ) + monkeypatch.setattr(module, "do_ngmix_metacal", fit) + warnings = [] + log = _RecordingLogger() + log.warning = lambda msg, *a, **k: warnings.append(msg) + inst = object.__new__(Ngmix) + inst._tile_cat_path = "in-memory-tile" + inst._seg_cat_path = None + inst._vignet_cat = SimpleNamespace( + gal_vign_cat=galaxies, psf_vign_cat=galaxies, close=lambda: None, + ) + inst._centroid_source = "wcs" + inst._id_obj_min = inst._id_obj_max = -1 + inst._bkg_sub = True + inst._blend_handling = "noisefill" + inst._dilate_neighbour = 1 + inst._metacal_psf = "fitgauss" + inst._defect_weighting = module.DEFECT_WEIGHTING + inst._save_batch = -1 + inst._zero_point = 30. + inst._output_dir = str(tmp_path) + inst._file_number_string = "-001-001" + inst._w_log = log + + inst.process() + + dropped = [msg for msg in warnings if "dropped" in msg] + assert len(dropped) == 1 + for part in ("-001-001", "object 7", "epoch 2000002-1", "poisoned"): + assert part in dropped[0] + with fits.open(inst.get_output_path(str(tmp_path))) as hdul: + npt.assert_array_equal(hdul["NOSHEAR"].data["n_epoch_failed"], [1]) + npt.assert_array_equal(hdul["NOSHEAR"].data["n_epoch_model"], [2]) diff --git a/tests/module/test_ngmix_defect_fill.py b/tests/module/test_ngmix_defect_fill.py new file mode 100644 index 000000000..776df59f8 --- /dev/null +++ b/tests/module/test_ngmix_defect_fill.py @@ -0,0 +1,1360 @@ +"""Defect fill and the epoch cuts (ngmix module). + +A defect is a stamp pixel with a nonzero flag, zero exposure weight +(off-tile pixels included) or an invalid background RMS. Before metacal, +:func:`prepare_ngmix_weights` gives every defect weight 0 and fills it the +same way whatever ``BLEND_HANDLING`` is: short defect runs take an +interpolant of the kept pixels around them, and the rest take noise at the +background RMS. Under uberseg, pixels on the neighbour side only lose their +weight, and their image values stay raw. The per-epoch cuts in +:func:`prepare_postage_stamps` act on the same defect set: the +masked-fraction cut counts it, and the central-defect veto drops an epoch +with a defect near the stamp centre, at the radius of the fill that defect +gets. The tile VIGNET's -1e30 neighbour markers are not defects: noisefill +zero-weights and noise-fills them, uberseg ignores them, and the epoch cuts +never count them. +""" + +import re +from pathlib import Path +from types import SimpleNamespace + +import numpy as np +import numpy.testing as npt +import pytest +from astropy.io import fits +from astropy.wcs import WCS +from hypothesis import given, settings +from hypothesis import strategies as st +from sqlitedict import SqliteDict +from shapepipe.modules.ngmix_package import ngmix as ngmix_module + +from shapepipe.modules.ngmix_package.defect_interpolation import ( + fourfold, + interpolable_defects, + interpolate_defects, +) +from shapepipe.modules.ngmix_package.ngmix import ( + DEFECT_WEIGHTINGS, + EPOCH_CENTRAL_DEFECT_RADIUS, + EPOCH_INTERPOLATED_DEFECT_RADIUS, + EPOCH_MASKED_FRACTION_CUT, + Ngmix, + defect_mask, + make_ngmix_observation, + prepare_ngmix_weights, + prepare_postage_stamps, + split_tile_markers, + uberseg_mask, +) +from shapepipe.modules.sextractor_package.sextractor_script import cut_stamps + + +# --- prepare_ngmix_weights: the filled set is the defect set --------------- + +@st.composite +def defect_stamps(draw): + """Square stamp with random flagged, zero-weight and bad-RMS pixels.""" + n = draw(st.integers(min_value=5, max_value=21)) + pixels = st.lists( + st.tuples(st.integers(0, n - 1), st.integers(0, n - 1)), max_size=n + ) + weight = np.ones((n, n)) + flag = np.zeros((n, n), dtype=np.int32) + bkg_rms = np.ones((n, n)) + for i, j in draw(pixels): + weight[i, j] = 0.0 + for i, j in draw(pixels): + flag[i, j] = draw(st.sampled_from([1, 2, 2**10])) + for i, j in draw(pixels): + bkg_rms[i, j] = draw(st.sampled_from([0.0, -1.0, np.nan, np.inf])) + # Unit-noise sky with raw defect values far above it, so a filled pixel + # is unambiguous and a leaked raw value is obvious. + seed = draw(st.integers(0, 2**31 - 1)) + gal = np.random.RandomState(seed).normal(size=(n, n)) + valid_rms = np.isfinite(bkg_rms) & (bkg_rms > 0) + defect = (weight == 0) | (flag != 0) | ~valid_rms + gal[defect] = 1.0e6 + return gal, weight, flag, bkg_rms + + +def _uberseg_seg(n): + """Central object on the centre pixel, neighbour footprint in a corner.""" + seg = np.zeros((n, n), dtype=np.int32) + seg[n // 2, n // 2] = 1 + seg[:2, :2] = 2 + return seg + + +@given( + stamp=defect_stamps(), + blend_handling=st.sampled_from(["noisefill", "uberseg"]), + seed=st.integers(0, 2**31 - 1), +) +@settings(deadline=None) +def test_filled_set_is_the_defect_set(stamp, blend_handling, seed): + """Filled pixels are exactly the defects (flag, zero weight, bad RMS), + and no raw defect value survives. Every other pixel keeps its raw value, + under either BLEND_HANDLING. The weight is zero on the defects, on the + quarter-turn copies of the interpolated ones, and under uberseg on the + neighbour side; those copies and the neighbour side keep their light. + + Failure modes: + * a defect source (flag, zero weight, bad RMS) is left out of the fill; + * the filled set grows beyond the defects (for example symmetrized); + * the zero-weight set is not the defects plus the interpolated pixels' + quarter turns (and the uberseg neighbour side); + * the fill is skipped under uberseg; + * neighbour-side pixels are filled. + """ + gal, weight, flag, bkg_rms = stamp + n = gal.shape[0] + kwargs = ( + dict(seg=_uberseg_seg(n), object_number=1, dilate_neighbour=1) + if blend_handling == "uberseg" + else {} + ) + + gal_out, w_out, _ = prepare_ngmix_weights( + gal, weight, flag, np.random.RandomState(seed), bkg_rms=bkg_rms, + blend_handling=blend_handling, **kwargs, + ) + + defect = ( + (weight == 0) | (flag != 0) | ~(np.isfinite(bkg_rms) & (bkg_rms > 0)) + ) + if defect.all(): + return # fully masked: no clean pixel left to set the noise level + filled = gal_out != gal + npt.assert_array_equal(filled, defect, "filled set is not the defect set") + assert np.all(np.abs(gal_out[filled]) < 100.0), "a raw defect value leaks" + neighbour = ( + uberseg_mask(kwargs["seg"], 1, dilate_neighbour=1) + if blend_handling == "uberseg" + else np.zeros((n, n), dtype=bool) + ) + # Neighbour pixels that are not defects keep their raw values + # (filled-set equality above). + zero = defect | neighbour | fourfold(interpolable_defects(defect)) + npt.assert_array_equal(w_out == 0.0, zero) + npt.assert_array_equal(w_out[~zero], 1.0) + + +def test_uberseg_defect_in_neighbour_region_is_filled(): + """A defect pixel that also lies on the neighbour side is filled. + + Failure mode: the fill is restricted to pixels uberseg keeps, so a raw bad + pixel on the neighbour side still reaches metacal. + """ + n = 21 + gal = 1.0e3 + np.arange(n * n, dtype=float).reshape(n, n) + weight = np.ones((n, n)) + flag = np.zeros((n, n), dtype=np.int32) + flag[1, 1] = 1 # inside the neighbour footprint + gal[1, 1] = 1.0e6 + gal_out, w_out, _ = prepare_ngmix_weights( + gal, weight, flag, np.random.RandomState(0), bkg_rms=np.ones((n, n)), + blend_handling="uberseg", seg=_uberseg_seg(n), object_number=1, + ) + assert w_out[1, 1] == 0.0 and gal_out[1, 1] != gal[1, 1] + # A neighbour-side pixel that is not a defect: zero weight, raw value. + assert w_out[0, 3] == 0.0 and gal_out[0, 3] == gal[0, 3] + + +def test_committed_blend_handling_fills_defects(): + """The committed universe's blend_handling is the workflow's default, + and under it every defect is zero-weighted and filled exactly as under + noisefill: the fill does not depend on the blend handling. + + Failure mode: the committed blend handling skips or changes the defect + fill, so the record claims a fill the default campaign does not run + (raw defects, which astra excludes, reaching metacal). + """ + import yaml + + repo = Path(__file__).resolve().parents[2] + universe = yaml.safe_load((repo / "universes" / "committed.yaml").read_text()) + decisions = universe["analyses"]["shape_measurement"]["decisions"] + blend_handling = decisions["blend_handling"] + assert blend_handling in ngmix_module.BLEND_HANDLINGS + workflow = yaml.safe_load((repo / "workflow" / "config.yaml").read_text()) + assert workflow["blend_handling"] == blend_handling + + n = 21 + gal = 1.0e3 + np.random.default_rng(1).normal(0.0, 1.0, (n, n)) + weight = np.ones((n, n)) + flag = np.zeros((n, n), dtype=np.int32) + flag[3, 15] = 1 + flag[10, 2:9] = 2**10 + weight[17, 9] = 0.0 + defect = (weight == 0) | (flag != 0) + kwargs = ( + dict(seg=_uberseg_seg(n), object_number=1) + if blend_handling == "uberseg" + else {} + ) + gal_out, w_out, _ = prepare_ngmix_weights( + gal, weight, flag, np.random.RandomState(0), bkg_rms=np.ones((n, n)), + blend_handling=blend_handling, **kwargs, + ) + ref_out, ref_w, _ = prepare_ngmix_weights( + gal, weight, flag, np.random.RandomState(0), bkg_rms=np.ones((n, n)), + ) + assert np.all(w_out[defect] == 0.0) + assert np.all(gal_out[defect] != gal[defect]), "defects are left raw" + npt.assert_array_equal(gal_out[defect], ref_out[defect]) + + +# --- prepare_postage_stamps: the fraction cut counts the defect set -------- + +N_STAMP = 51 +RA, DEC = 150.0, 2.0 + + +def _fake_inputs(epochs): + """Minimal vignet / tile-catalogue stand-ins for prepare_postage_stamps. + + ``epochs`` maps ``"-"`` to ``(flag, weight)`` or + ``(flag, weight, bkg_rms)``. Returns ``(vignet, tile_cat, psf_obj, + gal_obj)``. + """ + rng = np.random.default_rng(1) + wcs = WCS(naxis=2) + wcs.wcs.ctype = ["RA---TAN", "DEC--TAN"] + wcs.wcs.crval = [RA, DEC] + wcs.wcs.crpix = [N_STAMP / 2, N_STAMP / 2] + wcs.wcs.cdelt = [-0.187 / 3600, 0.187 / 3600] + header = fits.Header({"FSCALE": 1.0}).tostring() + + def per_epoch(make): + return {k: {"VIGNET": make(k)} for k in epochs} + + with_rms = any(len(v) == 3 for v in epochs.values()) + psf_obj = per_epoch(lambda k: np.ones((N_STAMP, N_STAMP))) + gal_obj = per_epoch(lambda k: rng.normal(0.0, 1.0, (N_STAMP, N_STAMP))) + for epoch in gal_obj.values(): + epoch["OFFSET"] = np.zeros(2) + vignet = SimpleNamespace( + gal_vign_cat={"1": gal_obj}, + bkg_vign_cat=None, + bkg_rms_vign_cat=( + {"1": per_epoch( + lambda k: epochs[k][2] if len(epochs[k]) == 3 + else np.ones((N_STAMP, N_STAMP)) + )} + if with_rms + else None + ), + flag_vign_cat={"1": per_epoch(lambda k: epochs[k][0])}, + weight_vign_cat={"1": per_epoch(lambda k: epochs[k][1])}, + f_wcs_file={ + k.split("-")[0]: { + int(k.split("-")[1]): {"WCS": wcs, "header": header} + } + for k in epochs + }, + ) + tile_cat = SimpleNamespace( + vign=None, seg=None, ra=np.array([RA]), dec=np.array([DEC]) + ) + return vignet, tile_cat, psf_obj, gal_obj + + +def _two_sided_band(width): + """Mask of ``width`` columns on each side of the stamp, far from the + centre (at least 23 px for width 3).""" + band = np.zeros((N_STAMP, N_STAMP), dtype=bool) + band[:, :width] = True + band[:, -width:] = True + return band + + +def _epochs(): + """A clean epoch and four masked ones, all defects far from the centre. + + * band5: 5 flagged columns on one side, 9.8% (35% with its quarter turns); + * flag6: 2 x 3 flagged columns, 11.8%; + * dead6: the same columns at zero weight, no flags; + * rms6: the same columns with an invalid background RMS. + """ + clean = np.zeros((N_STAMP, N_STAMP), dtype=np.int32) + ones = np.ones((N_STAMP, N_STAMP)) + band5 = clean.copy() + band5[:, -5:] = 1 + two = _two_sided_band(3) + dead = ones.copy() + dead[two] = 0.0 + rms = ones.copy() + rms[two] = np.nan + return { + "2100001-10": (clean, ones), + "2100002-11": (band5, ones), + "2100003-12": (two.astype(np.int32), ones), + "2100004-13": (clean.copy(), dead), + "2100005-14": (clean.copy(), ones, rms), + } + + +def _surviving(epochs, **kwargs): + vignet, tile_cat, psf_obj, gal_obj = _fake_inputs(epochs) + stamp = prepare_postage_stamps( + vignet, 1, 0, tile_cat, bkg_sub=False, + psf_obj=psf_obj, gal_obj=gal_obj, **kwargs, + ) + names = {id(v[0]): name for name, v in epochs.items()} + return sorted(names[id(flag)] for flag in stamp.flags) + + +def test_epoch_cut_counts_the_defect_set(): + """At the 10% cut, the three 11.8% epochs are dropped, whichever defect + source masks them, and the 9.8% band survives. + + Failure modes: the cut counts flags only (keeps dead6 and rms6), omits + one defect source, or counts a symmetrized set (drops band5); in each + case the cut disagrees with the set that is zero-weighted and filled + (epoch-cut-on-defect-mask). + """ + epochs = _epochs() + fractions = [ + defect_mask(weight, flag, *rms).mean() + for flag, weight, *rms in epochs.values() + ] + assert fractions[1] <= EPOCH_MASKED_FRACTION_CUT < min(fractions[2:]) + assert _surviving(epochs) == ["2100001-10", "2100002-11"] + + +# --- prepare_postage_stamps: the central-defect veto ----------------------- + +def _veto_epochs(): + """A clean epoch and five with one defect each, all far below the + fraction cut: a column at EPOCH_INTERPOLATED_DEFECT_RADIUS, a column one + pixel inside it, a single pixel at 9 px (all interpolated), and a 5-px + bleed (noise-filled) at the interpolated radius and at + EPOCH_CENTRAL_DEFECT_RADIUS.""" + centre = N_STAMP // 2 + ri = int(EPOCH_INTERPOLATED_DEFECT_RADIUS) + rn = int(EPOCH_CENTRAL_DEFECT_RADIUS) + clean = np.zeros((N_STAMP, N_STAMP), dtype=np.int32) + ones = np.ones((N_STAMP, N_STAMP)) + column_at, column_inside, pixel, wide_at, wide_far = ( + clean.copy() for _ in range(5) + ) + column_at[:, centre + ri] = 1 + column_inside[:, centre + ri - 1] = 1 + pixel[centre, centre + 9] = 1 + wide_at[:, centre + ri:centre + ri + 5] = 1 + wide_far[:, centre + rn:centre + rn + 5] = 1 + return { + "2100001-10": (clean, ones), + "2100002-11": (column_at, ones), + "2100003-12": (column_inside, ones), + "2100004-13": (pixel, ones), + "2100005-14": (wide_at, ones), + "2100006-15": (wide_far, ones), + } + + +def test_the_veto_radius_follows_the_fill(): + """An interpolated defect drops the epoch inside + EPOCH_INTERPOLATED_DEFECT_RADIUS, a noise-filled one inside + EPOCH_CENTRAL_DEFECT_RADIUS; a defect at its radius is kept. + + Failure modes: the veto is skipped, so a filled hole in the object's + light enters the fit; the interpolated radius is applied to noise-filled + pixels (a wide hole next to the object survives), or not applied at all + (a 9-px pixel is dropped); the boundary is inclusive + (veto-radius-follows-the-fill). + """ + assert _surviving(_veto_epochs()) == [ + "2100001-10", "2100002-11", "2100004-13", "2100006-15" + ] + + +# --- prepare_postage_stamps: per-epoch OFFSET ------------------------------ + +def test_each_surviving_epoch_carries_its_own_offset(): + """``stamp.offsets`` holds each surviving epoch's vignette OFFSET, in the + order of ``stamp.flags``. + + Failure mode: the offset is dropped or read from another epoch, so the + default "wcs" centroid raises or puts the Jacobian origin off the object. + """ + clean = np.zeros((N_STAMP, N_STAMP), dtype=np.int32) + ones = np.ones((N_STAMP, N_STAMP)) + epochs = {f"210000{i}-1{i}": (clean.copy(), ones) for i in range(3)} + vignet, tile_cat, psf_obj, gal_obj = _fake_inputs(epochs) + del vignet.gal_vign_cat # OFFSET must come from the supplied gal_obj. + for i, name in enumerate(epochs): + gal_obj[name]["OFFSET"] = np.array([0.1 * i, -0.1 * i]) + stamp = prepare_postage_stamps( + vignet, 1, 0, tile_cat, bkg_sub=False, + psf_obj=psf_obj, gal_obj=gal_obj, + ) + names = {id(flag): name for name, (flag, _) in epochs.items()} + assert len(stamp.offsets) == len(stamp.flags) == 3 + for flag, offset in zip(stamp.flags, stamp.offsets): + npt.assert_array_equal(offset, gal_obj[names[id(flag)]]["OFFSET"]) + + +# --- Ngmix.process: the per-tile epoch-cut tally --------------------------- + +class _RecordingLogger: + def __init__(self): + self.messages = [] + + def info(self, msg, *_args, **_kwargs): + self.messages.append(msg) + + warning = error = info + + +def test_process_logs_the_epoch_cut_tally(tmp_path, monkeypatch): + """One tile, four objects; the end-of-tile line counts each cut's drops. + + * object 1: clean, 18-column edge band, defect 3 px from the centre -> one + epoch each for considered, masked_fraction, central_veto; survives. + * object 2: edge band and central defect -> both epochs dropped; emptied. + * object 3: one all-zero stamp, skipped before the cuts -> not considered, + and not emptied by the cuts. + * object 4: no PSF ('empty') -> never reaches the cuts. + + Failure modes: a cut's drops are not counted or land in the wrong + counter; epochs skipped before the cuts are counted as considered; an + object with no epoch at all is reported as emptied by the cuts; counts + from one object overwrite another's. + """ + centre = N_STAMP // 2 + clean = np.zeros((N_STAMP, N_STAMP), dtype=np.int32) + ones = np.ones((N_STAMP, N_STAMP)) + wide18, near = clean.copy(), clean.copy() + wide18[:, -18:] = 1 + near[centre, centre + 3] = 1 + objects = { + 1: {"2100001-10": (clean, ones), "2100002-11": (wide18, ones), + "2100003-12": (near, ones)}, + 2: {"2100004-13": (wide18.copy(), ones), + "2100005-14": (near.copy(), ones)}, + 3: {"2100006-15": (clean.copy(), ones)}, + } + stores = {} + for obj_id, epochs in objects.items(): + vignet, _, psf_obj, gal_obj = _fake_inputs(epochs) + if obj_id == 3: + gal_obj["2100006-15"]["VIGNET"] = np.zeros((N_STAMP, N_STAMP)) + stores[obj_id] = (vignet, psf_obj, gal_obj) + vignet = SimpleNamespace( + bkg_vign_cat=None, + bkg_rms_vign_cat=None, + psf_vign_cat={ + "4": "empty", **{str(i): s[1] for i, s in stores.items()} + }, + gal_vign_cat={ + "4": "empty", **{str(i): s[2] for i, s in stores.items()} + }, + flag_vign_cat={ + str(i): s[0].flag_vign_cat["1"] for i, s in stores.items() + }, + weight_vign_cat={ + str(i): s[0].weight_vign_cat["1"] for i, s in stores.items() + }, + f_wcs_file={ + k: v for s in stores.values() for k, v in s[0].f_wcs_file.items() + }, + close=lambda: None, + ) + tile_cat = SimpleNamespace( + obj_id=np.array([1, 2, 3, 4]), ra=np.full(4, RA), dec=np.full(4, DEC), + vign=None, seg=None, flux=None, + ) + + paths = [tmp_path / f"{name}.sqlite" for name in + ("gal", "psf", "weight", "flag", "headers")] + for path in paths: + SqliteDict(str(path)).close() + log = _RecordingLogger() + ngmix = Ngmix( + ["tile_cat.fits"] + [str(p) for p in paths[:4]], + str(tmp_path), "-001-001", 30.0, str(paths[4]), log, + bkg_sub=False, + ) + ngmix._vignet_cat.close() + ngmix._vignet_cat = vignet + monkeypatch.setattr(ngmix_module, "Tile_cat", lambda *a, **k: tile_cat) + + def no_fit(*_args, **_kwargs): + raise RuntimeError("metacal is not under test") + + monkeypatch.setattr(ngmix_module, "do_ngmix_metacal", no_fit) + for method in ("compile_results", "save_results", "log_mean_ellipticity"): + monkeypatch.setattr(Ngmix, method, lambda *_a, **_k: None) + + ngmix.process() + + lines = [m for m in log.messages if m.startswith("epoch cuts:")] + assert len(lines) == 1, log.messages + tally = dict( + (k, int(v)) for k, v in re.findall(r"(\w+)=(\d+)", lines[0]) + ) + assert tally == dict( + considered=5, masked_fraction=2, central_veto=2, failed=0, + objects_emptied=1, + ), lines[0] + + +# --- make_ngmix_observation: the HSM centroid reads the filled image ------- + +def test_hsm_centroid_ignores_raw_defect_values(): + """With ``centroid_source="hsm"``, the Jacobian origin lands on the + object even when a flagged column a few pixels away holds a raw value + ten times the object's peak. + + Failure mode: HSM measures the raw stamp, so the defect drags the centroid + off the object (or HSM fails and falls back to the stamp centre). + """ + import galsim + + n = 51 + centre = (n - 1) / 2 + d_row, d_col = 1.3, -0.7 + rows, cols = np.mgrid[:n, :n] + gal = np.exp( + -((rows - centre - d_row) ** 2 + (cols - centre - d_col) ** 2) + / (2 * 2.0 ** 2) + ) + flag = np.zeros((n, n), dtype=np.int32) + flag[:, n // 2 + 6] = 1 + gal[flag != 0] = 10.0 + psf = np.exp(-((rows - centre) ** 2 + (cols - centre) ** 2) / 2.0) + obs = make_ngmix_observation( + gal, np.ones((n, n)), flag, psf / psf.sum(), + galsim.PixelScale(0.1857).jacobian(), np.random.RandomState(0), + bkg_rms=np.full((n, n), 1e-3), centroid_source="hsm", + ) + row, col = obs.jacobian.get_cen() + npt.assert_allclose( + [row - centre, col - centre], [d_row, d_col], atol=0.05 + ) + + +# --- Interpolated defects --------------------------------------------------- + +def _hot_stamp(): + """A bright object with hot defects: a column and a finite 3-px bleed + (interpolated), an edge band and a 5x5 blob (noise-filled).""" + centre = N_STAMP // 2 + rows, cols = np.mgrid[:N_STAMP, :N_STAMP] + gal = 1e3 * np.exp( + -((rows - centre) ** 2 + (cols - centre) ** 2) / (2 * 3.0 ** 2) + ) + flag = np.zeros((N_STAMP, N_STAMP), dtype=np.int32) + flag[:, centre + 8] = 1 + flag[centre - 5:centre + 6, centre - 11:centre - 8] = 1 + flag[:, :4] = 1 + flag[centre + 9:centre + 14, centre + 12:centre + 17] = 1 + gal[flag != 0] = 5e4 + return gal, np.ones((N_STAMP, N_STAMP)), flag + + +def _expected_masks(target, defect, defect_weighting): + """The fill and the zero-weight set each DEFECT_WEIGHTING promises, + written out independently of ``defect_weighting_masks``.""" + noisefilled = defect & ~target + if defect_weighting == "des_y6": + copy = interpolable_defects(np.rot90(target) & ~defect, defect) + return target | copy, noisefilled + zero = { + "fourfold_zero": defect | fourfold(target), + "hole": defect, + "full": noisefilled, + }[defect_weighting] + return target, zero + + +@pytest.mark.parametrize("defect_weighting", DEFECT_WEIGHTINGS) +@pytest.mark.parametrize("blend_handling", ["noisefill", "uberseg"]) +def test_interpolated_fill_and_its_weights(blend_handling, defect_weighting): + """Short defect runs take the interpolant of the clean image; other + defects take noise. Under fourfold_zero the weight is zero on the + defects and on the quarter-turn orbit of the interpolated pixels, whose + light stays; under hole on the defects; under full and des_y6 only on + the noise-filled ones. des_y6 also interpolates the clean quarter-turn + (k=1) copies of the interpolated pixels, at full weight. The uberseg + neighbour side stays at weight 0 under every option. The metacal noise + image is interpolated by the same operator in fixnoise's quarter-turned + frame. + + Failure modes: the orbit is not zero-weighted under fourfold_zero (a + one-sided hole in the likelihood biases c), or its light is replaced; + the copies are not interpolated under des_y6, or a noise-filled pixel + is; an interpolated pixel on the neighbour side gets weight; wide + defects or edge bands are extrapolated; raw defect values leak; the + noise image is interpolated in the detector frame + (noise-image-in-the-fixnoise-frame). + """ + gal, weight, flag = _hot_stamp() + defect = flag != 0 + target = interpolable_defects(defect) + assert target.any() and (defect & ~target).any() + kwargs = ( + dict(seg=_uberseg_seg(N_STAMP), object_number=1, dilate_neighbour=1) + if blend_handling == "uberseg" + else {} + ) + gal_out, w_out, noise_out = prepare_ngmix_weights( + gal, weight, flag, np.random.RandomState(4), + bkg_rms=np.ones((N_STAMP, N_STAMP)), blend_handling=blend_handling, + defect_weighting=defect_weighting, **kwargs, + ) + + neighbour = ( + uberseg_mask(kwargs["seg"], 1, dilate_neighbour=1) + if kwargs else np.zeros_like(defect) + ) + fill, zero = _expected_masks(target, defect, defect_weighting) + if defect_weighting == "des_y6": + assert (fill & ~defect).any() + npt.assert_array_equal(w_out == 0.0, zero | neighbour) + npt.assert_array_equal(w_out[~(zero | neighbour)], 1.0) + kept = ~defect & ~fill + npt.assert_array_equal(gal_out[kept], gal[kept]) + expected = interpolate_defects(gal[None], defect | fill, fill)[0] + npt.assert_allclose(gal_out[fill], expected[fill], rtol=1e-5) + assert np.all(np.abs(gal_out[defect & ~target]) < 10.0) + turned = np.rot90(noise_out) + refilled = interpolate_defects(turned[None], defect | fill, fill)[0] + npt.assert_allclose(turned[fill], refilled[fill], atol=1e-5) + + +@pytest.mark.parametrize("band", [1, 2, 4]) +def test_des_y6_copy_in_a_corner_of_the_clean_region_keeps_its_light(band): + """An isolated defect whose quarter-turn copy lands where the top edge + band meets the stamp's right edge: under des_y6 the copy has no clean + pixel beyond the band or the stamp edge, so the interpolant cannot + reach it and it keeps its light; the defect itself is interpolated. + + Failure mode: the copy is interpolated anyway, the interpolant is NaN + there and prepare_ngmix_weights raises, dropping every epoch of the + object. + """ + n = N_STAMP + flag = np.zeros((n, n), dtype=np.int32) + flag[:band] = 1 + flag[n - 1, n - 1 - band] = 1 + gal = np.random.RandomState(1).normal(size=(n, n)) + gal_out, w_out, _ = prepare_ngmix_weights( + gal, np.ones((n, n)), flag, np.random.RandomState(4), + bkg_rms=np.ones((n, n)), defect_weighting="des_y6", + ) + corner = (band, n - 1) + assert np.rot90(flag != 0)[corner] and flag[corner] == 0 + assert gal_out[corner] == gal[corner] + assert w_out[corner] == 1.0 + assert np.all(np.isfinite(gal_out)) + assert gal_out[n - 1, n - 1 - band] != gal[n - 1, n - 1 - band] + + +def test_fixnoise_turns_the_noise_image_k1_then_k3(): + """ngmix's fixnoise adds np.rot90(sheared(np.rot90(noise, 1)), 3) to + each sheared image: with an asymmetric noise image, the fixnoise output + minus the plain metacal output is exactly that. + + Failure mode: an ngmix release changes the turn (k=3 first, or none), + and the noise image prepare_ngmix_weights interpolates in the k=1 frame + no longer lines up with the science image's interpolated pixels + (noise-image-in-the-fixnoise-frame). + """ + from ngmix import DiagonalJacobian, Observation + from ngmix.metacal import get_all_metacal + + n = 33 + rows, cols = np.indices((n, n)) - (n - 1) / 2 + psf_image = np.exp(-(rows ** 2 + cols ** 2) / (2 * 2.0 ** 2)) + psf_image /= psf_image.sum() + galaxy = 100 * np.exp(-(rows ** 2 + cols ** 2) / (2 * 3.0 ** 2)) + jacobian = DiagonalJacobian(row=(n - 1) / 2, col=(n - 1) / 2, scale=0.2) + noise = np.random.RandomState(3).normal(size=(n, n)) + noise[:, 20] += 5.0 + + def metacal(image, noise_image=None): + obs = Observation( + image, weight=np.ones((n, n)), jacobian=jacobian, + psf=Observation(psf_image, jacobian=jacobian), noise=noise_image, + ) + return get_all_metacal( + obs, psf="gauss", types=["noshear", "1p"], + fixnoise=noise_image is not None, use_noise_image=True, + rng=np.random.RandomState(0), + ) + + fixed = metacal(galaxy, noise) + plain = metacal(galaxy) + turned = metacal(np.rot90(noise, 1).copy()) + for t in ("noshear", "1p"): + added = fixed[t].image - plain[t].image + npt.assert_allclose(added, np.rot90(turned[t].image, 3), atol=1e-9) + assert not np.allclose(added, np.rot90(turned[t].image, 1)) + + +def test_defect_diagnostics_summarise_the_kept_epochs(): + """Over the epochs the veto keeps, ``n_epoch_interp`` counts those with + an interpolated pixel and the two distances are the nearest + interpolated and noise-filled defect pixels to the stamp centre; an + object without either has -1. + + Failure modes: a vetoed epoch is counted; noise-filled and interpolated + pixels are swapped; the sentinel is 0, a valid distance + (defect-diagnostic-columns). + """ + vignet, tile_cat, psf_obj, gal_obj = _fake_inputs(_veto_epochs()) + stamp = prepare_postage_stamps( + vignet, 1, 0, tile_cat, bkg_sub=False, + psf_obj=psf_obj, gal_obj=gal_obj, + ) + assert stamp.defect_diagnostics() == { + "n_epoch_interp": 2, + "min_dist_interp": float(EPOCH_INTERPOLATED_DEFECT_RADIUS), + "min_dist_noisefill": float(EPOCH_CENTRAL_DEFECT_RADIUS), + } + clean = {"2100001-10": _veto_epochs()["2100001-10"]} + vignet, tile_cat, psf_obj, gal_obj = _fake_inputs(clean) + stamp = prepare_postage_stamps( + vignet, 1, 0, tile_cat, bkg_sub=False, + psf_obj=psf_obj, gal_obj=gal_obj, + ) + assert stamp.defect_diagnostics() == { + "n_epoch_interp": 0, "min_dist_interp": -1.0, + "min_dist_noisefill": -1.0, + } + + +# --- The tile VIGNET's -1e30 neighbour markers are not defects ------------- +# +# The tile VIGNET carries -1e30 on the footprints of other detections. Every +# epoch shares that tile stamp, so a marker counted as a defect would drop +# every epoch of an object with a neighbour inside the veto radius. The +# markers are their own per-epoch mask (``stamp.neighbours``): noisefill +# zero-weights and noise-fills them, uberseg ignores them, and the epoch cuts +# never read them. + +_MARKER = -1.0e30 +_CENTRE = N_STAMP // 2 +# Flipped (ccd < 18) and unflipped (ccd >= 18) MegaCam CCDs. +_MARKER_EPOCH_NAMES = ["2100001-10", "2100002-20", "2100003-11"] + + +def _tile_with_neighbour(columns_from=_CENTRE + 3, rows=(_CENTRE - 2, + _CENTRE + 3)): + """Tile VIGNET with a -1e30 neighbour footprint whose nearest pixel is + 3 px from the stamp centre. The footprint is off-centre, so the MegaCam + flip moves it.""" + tile = np.random.default_rng(5).normal(0.0, 1.0, (N_STAMP, N_STAMP)) + tile[rows[0]:rows[1], columns_from:columns_from + 6] = _MARKER + return tile + + +def _marker_stamp(tile, epochs=None, **kwargs): + """Run prepare_postage_stamps on defect-free epochs under ``tile``.""" + clean = np.zeros((N_STAMP, N_STAMP), dtype=np.int32) + ones = np.ones((N_STAMP, N_STAMP)) + if epochs is None: + epochs = {name: (clean.copy(), ones) for name in _MARKER_EPOCH_NAMES} + vignet, tile_cat, psf_obj, gal_obj = _fake_inputs(epochs) + tile_cat.vign = tile[np.newaxis] + stamp = prepare_postage_stamps( + vignet, 1, 0, tile_cat, bkg_sub=False, + psf_obj=psf_obj, gal_obj=gal_obj, **kwargs, + ) + return stamp, epochs, gal_obj + + +def _expected_neighbours(tile, name): + return Ngmix.MegaCamFlip(tile, int(name.split("-")[1])) == _MARKER + + +def test_neighbour_markers_near_the_centre_keep_every_epoch(): + """A neighbour footprint 3 px from the centre, and one covering 41% of + the stamp, drop no epoch: the masked-fraction cut and the central veto + count no marker. + + Failure mode: the markers are written into the flag stamp and counted as + defects, so every epoch (they all share the tile VIGNET) is dropped by + the veto or the fraction cut and the object loses its shape + (neighbour-markers-are-not-defects). + """ + small = _tile_with_neighbour() + # Large, but short of the stamp border: no row or column is entirely + # marked, so it is a neighbour, not off-tile. + large = _tile_with_neighbour() + large[1:-1, _CENTRE + 3:-1] = _MARKER + assert (large == _MARKER).mean() > EPOCH_MASKED_FRACTION_CUT + for tile in (small, large): + stamp, _, _ = _marker_stamp(tile) + assert len(stamp.gals) == len(_MARKER_EPOCH_NAMES) + assert stamp.epoch_cuts["considered"] == len(_MARKER_EPOCH_NAMES) + assert stamp.epoch_cuts["masked_fraction"] == 0 + assert stamp.epoch_cuts["central_veto"] == 0 + + +def test_neighbour_markers_are_their_own_per_epoch_mask(): + """``stamp.neighbours`` holds the MegaCam-flipped marker mask of each + surviving epoch, and the flag stamps stay the exposure's own. + + Failure mode: the markers are merged into the flags, or the neighbour + mask is not flipped with its epoch and lands on the wrong pixels. + """ + tile = _tile_with_neighbour() + stamp, epochs, _ = _marker_stamp(tile) + names = {id(v[0]): name for name, v in epochs.items()} + assert len(stamp.neighbours) == len(stamp.flags) + for flag, neighbour in zip(stamp.flags, stamp.neighbours): + name = names[id(flag)] + npt.assert_array_equal(flag, 0) + npt.assert_array_equal(neighbour, _expected_neighbours(tile, name)) + assert not np.array_equal(stamp.neighbours[0], stamp.neighbours[1]) + + +def test_a_flagged_column_near_the_centre_is_still_vetoed(): + """With a neighbour footprint present, an epoch with a genuinely flagged + column 3 px from the centre is still dropped, and only that epoch. + + Failure mode: handling the markers apart also exempts real defects from + the central veto. + """ + clean = np.zeros((N_STAMP, N_STAMP), dtype=np.int32) + ones = np.ones((N_STAMP, N_STAMP)) + column = clean.copy() + column[:, _CENTRE - 3] = 1 + epochs = { + "2100001-10": (clean, ones), + "2100002-20": (column, ones), + "2100003-11": (clean.copy(), ones), + } + stamp, _, _ = _marker_stamp(_tile_with_neighbour(), epochs) + names = {id(v[0]): name for name, v in epochs.items()} + assert sorted(names[id(f)] for f in stamp.flags) == [ + "2100001-10", "2100003-11" + ] + assert stamp.epoch_cuts["central_veto"] == 1 + assert stamp.epoch_cuts["masked_fraction"] == 0 + + +def _stamp_epoch_weights(stamp, i, seed, **kwargs): + return prepare_ngmix_weights( + 1.0e3 + stamp.gals[i], stamp.weights[i], stamp.flags[i], + np.random.RandomState(seed), bkg_rms=stamp.bkg_rms[i], + neighbour=stamp.neighbours[i], **kwargs, + ) + + +def test_noisefill_fills_exactly_the_marked_pixels(): + """Under noisefill, a defect-free epoch has zero weight and noise + exactly on the marked neighbour pixels; every other pixel keeps its raw + value and its weight. + + Failure mode: the neighbour markers are dropped with the flags, so + noisefill no longer removes neighbour light (noisefill-fills-markers). + """ + tile = _tile_with_neighbour() + stamp, epochs, _ = _marker_stamp(tile) + assert len(stamp.gals) == len(_MARKER_EPOCH_NAMES) + for i in range(len(stamp.gals)): + gal = 1.0e3 + stamp.gals[i] + gal_out, w_out, _ = _stamp_epoch_weights( + stamp, i, seed=i, blend_handling="noisefill", + ) + neighbour = stamp.neighbours[i] + assert neighbour.any() + npt.assert_array_equal(gal_out != gal, neighbour) + npt.assert_array_equal(w_out == 0.0, neighbour) + assert np.all(np.abs(gal_out[neighbour]) < 10.0) + + +def test_uberseg_leaves_the_marked_pixels_raw(): + """Under uberseg, the markers mask nothing: with a seg map holding only + the central object, a defect-free epoch keeps every pixel raw and + weighted, marked or not. + + Failure mode: the markers reach the defect set or the fill, so uberseg + noise-fills the neighbour's light instead of leaving it to the seg-based + weight (uberseg-ignores-markers). + """ + tile = _tile_with_neighbour() + stamp, _, _ = _marker_stamp(tile) + seg = np.zeros((N_STAMP, N_STAMP), dtype=np.int32) + seg[_CENTRE - 1:_CENTRE + 2, _CENTRE - 1:_CENTRE + 2] = 1 + assert len(stamp.gals) == len(_MARKER_EPOCH_NAMES) + for i in range(len(stamp.gals)): + gal = 1.0e3 + stamp.gals[i] + gal_out, w_out, _ = _stamp_epoch_weights( + stamp, i, seed=i, blend_handling="uberseg", seg=seg, + object_number=1, + ) + npt.assert_array_equal(gal_out, gal) + assert np.all(w_out > 0.0) + + +def test_do_ngmix_metacal_threads_each_epochs_neighbour_mask(monkeypatch): + """Each epoch's neighbour mask reaches make_ngmix_observation. + + Failure mode: the mask is built but never used, so noisefill silently + stops filling neighbours. + """ + tile = _tile_with_neighbour() + stamp, _, _ = _marker_stamp(tile) + assert len(stamp.gals) == len(_MARKER_EPOCH_NAMES) + seen = [] + + class _Stop(Exception): + pass + + def fake_observation(*args, **kwargs): + seen.append(kwargs["neighbour"]) + return object() + + def stop(*args, **kwargs): + raise _Stop + + monkeypatch.setattr( + ngmix_module, "make_ngmix_observation", fake_observation, + ) + monkeypatch.setattr(ngmix_module, "make_runners", stop) + monkeypatch.setattr(ngmix_module, "ObsList", list) + with pytest.raises(_Stop): + ngmix_module.do_ngmix_metacal( + stamp, None, 1.0, np.random.RandomState(0), + ) + assert len(seen) == len(stamp.gals) + for got, want in zip(seen, stamp.neighbours): + assert got is want + + +# --- Neighbours are markers on another object's segmentation footprint ----- +# +# Under MASK_TYPE CORRECT SExtractor also writes -1e30 over sub-MINAREA sky +# blobs and over fragments CLEAN merged into the object itself. The seg map +# labels those 0 and the object's own NUMBER, so they are not neighbours, and +# the epoch stamps hold real light there: they keep weight and image. + +_OBJ = 1 # the object's NUMBER in _fake_inputs's stores + + +def _seg_marked_tile(): + """Tile VIGNET and SEG_VIGNET with four kinds of interior -1e30 marker. + + Returns ``(tile, seg, kinds)``: a neighbour footprint (label 7), a + detection that left the catalogue (label -1), a fragment of the object + itself (label ``_OBJ``) and a sky blob (label 0); ``kinds`` maps each + name to its marker mask. All of them are off-centre, so the MegaCam flip + moves them. + """ + c = _CENTRE + tile = _tile_with_neighbour() + seg = np.zeros_like(tile, dtype=np.int32) + yy, xx = np.mgrid[:N_STAMP, :N_STAMP] + seg[(yy - c) ** 2 + (xx - c) ** 2 <= 9] = _OBJ + kinds = {name: np.zeros_like(tile, dtype=bool) for name in + ("neighbour", "departed", "own_fragment", "sky")} + kinds["neighbour"][c - 2:c + 3, c + 3:c + 9] = True + kinds["departed"][c - 12:c - 9, c - 12:c - 9] = True + kinds["own_fragment"][c + 6:c + 9, c - 8:c - 5] = True + kinds["sky"][c - 10:c - 8, c + 10] = True + seg[kinds["neighbour"]] = 7 + seg[kinds["departed"]] = -1 + seg[kinds["own_fragment"]] = _OBJ + for mask in kinds.values(): + tile[mask] = _MARKER + return tile, seg, kinds + + +def test_split_tile_markers_keeps_only_other_footprints(): + """With the seg stamp, the neighbour mask is the markers on a label other + than 0 and the object's; the object's own fragments and sky blobs are + dropped, and the off-tile split is unchanged. Without it, every interior + marker is a neighbour. + + Failure mode: noisefill noise-fills the object's own CLEAN-merged light + and sky (neighbours-are-other-footprints). + """ + tile, seg, kinds = _seg_marked_tile() + tile[:, :4] = _MARKER # an off-tile band + neighbour, off_tile = split_tile_markers(tile, tile.shape, seg, _OBJ) + bare_neighbour, bare_off_tile = split_tile_markers(tile, tile.shape) + npt.assert_array_equal(off_tile, bare_off_tile) + assert off_tile[:, :4].all() and not off_tile[:, 4:].any() + npt.assert_array_equal( + neighbour, (kinds["neighbour"] | kinds["departed"]) & ~off_tile + ) + npt.assert_array_equal(bare_neighbour, (tile == _MARKER) & ~off_tile) + + +def _seg_marker_stamp(tile, seg, blend_handling="noisefill"): + clean = np.zeros((N_STAMP, N_STAMP), dtype=np.int32) + ones = np.ones((N_STAMP, N_STAMP)) + epochs = {name: (clean.copy(), ones) for name in _MARKER_EPOCH_NAMES} + vignet, tile_cat, psf_obj, gal_obj = _fake_inputs(epochs) + tile_cat.vign = tile[np.newaxis] + tile_cat.seg = seg[np.newaxis] + return prepare_postage_stamps( + vignet, _OBJ, 0, tile_cat, bkg_sub=False, + psf_obj=psf_obj, gal_obj=gal_obj, blend_handling=blend_handling, + ) + + +def test_noisefill_keeps_light_under_non_neighbour_markers(): + """Through prepare_postage_stamps, each epoch's neighbour mask is its + flipped other-footprint markers; noisefill zero-weights and noise-fills + those and keeps the weight and image of the object's own fragment and + the sky blob. + """ + tile, seg, kinds = _seg_marked_tile() + stamp = _seg_marker_stamp(tile, seg) + assert len(stamp.gals) == len(_MARKER_EPOCH_NAMES) + for i, name in enumerate(stamp.epoch_names): + ccd = int(name.split("-")[1]) + flip = lambda m: Ngmix.MegaCamFlip(m, ccd) + npt.assert_array_equal( + stamp.neighbours[i], + flip(kinds["neighbour"] | kinds["departed"]), + ) + gal, weight, _ = _stamp_epoch_weights(stamp, i, seed=3) + kept = flip(kinds["own_fragment"] | kinds["sky"]) + assert np.all(weight[kept] > 0) + npt.assert_array_equal(gal[kept], 1.0e3 + stamp.gals[i][kept]) + assert np.all(weight[stamp.neighbours[i]] == 0) + + +def test_uberseg_is_unchanged_by_seg_defined_neighbours(monkeypatch): + """Uberseg reads no marker, so defining neighbours by the seg map leaves + every epoch's observation bit-identical: the same epochs survive, with + the same image, weight and noise image. Noisefill does change. + """ + tile, seg, _ = _seg_marked_tile() + marker_split = split_tile_markers + + def observations(blend_handling): + stamp = _seg_marker_stamp(tile, seg, blend_handling) + out = [] + for i in range(len(stamp.gals)): + obs = make_ngmix_observation( + 1.0e3 + stamp.gals[i], stamp.weights[i], stamp.flags[i], + stamp.psfs[i], stamp.jacobs[i], np.random.RandomState(i), + bkg_rms=stamp.bkg_rms[i], offset=stamp.offsets[i], + blend_handling=blend_handling, seg=stamp.segs[i], + object_number=_OBJ, neighbour=stamp.neighbours[i], + ) + out.append((obs.image, obs.weight, obs.noise)) + return stamp.epoch_names, out + + new = {b: observations(b) for b in ("uberseg", "noisefill")} + monkeypatch.setattr( + ngmix_module, "split_tile_markers", + lambda t, shape, seg=None, object_number=None: marker_split(t, shape), + ) + old = {b: observations(b) for b in ("uberseg", "noisefill")} + + assert new["uberseg"][0] == old["uberseg"][0] + for got, want in zip(new["uberseg"][1], old["uberseg"][1]): + for a, b in zip(got, want): + npt.assert_array_equal(a, b) + assert any( + not np.array_equal(got[1], want[1]) + for got, want in zip(new["noisefill"][1], old["noisefill"][1]) + ) + + +# --- Off-tile pixels are defects ------------------------------------------- +# +# The tile VIGNET also holds -1e30 beyond the tile's edge, where the epoch +# holds the object's own light, cut off. Those pixels are the runs of +# entirely -1e30 stamp rows and columns that start at a stamp border (the +# off-image part of a rectangle clip); they join the epoch's defect set with +# zero exposure weight. The other markers are the neighbour mask. + + +def _off_tile_expected(tile, name): + flipped = Ngmix.MegaCamFlip(tile, int(name.split("-")[1])) == _MARKER + return flipped.all(axis=1)[:, None] | flipped.all(axis=0)[None, :] + + +def test_object_three_px_from_the_tile_edge_is_dropped(): + """With the tile edge 3 px from the object, every epoch is dropped: the + off-tile band counts toward the epoch cuts. + + Failure mode: off-tile pixels are treated as neighbour markers, so an + edge object is measured with a noise-filled band through its own light + (off-tile-pixels-are-defects). + """ + tile = np.random.default_rng(5).normal(0.0, 1.0, (N_STAMP, N_STAMP)) + tile[:, :_CENTRE - 2] = _MARKER + stamp, _, _ = _marker_stamp(tile) + assert len(stamp.gals) == 0 + assert stamp.epoch_cuts["considered"] == len(_MARKER_EPOCH_NAMES) + assert ( + stamp.epoch_cuts["masked_fraction"] + stamp.epoch_cuts["central_veto"] + == len(_MARKER_EPOCH_NAMES) + ) + + +def test_the_central_veto_sees_off_tile_pixels(monkeypatch): + """An off-tile band at EPOCH_CENTRAL_DEFECT_RADIUS from the object passes + the central veto; one 9 px away is noise-filled inside it and vetoed. + + Every off-tile band within reach of the veto also exceeds the 10% + fraction cut (this one is 17 columns, 1/3 of the stamp), so the fraction + cut is lifted here to isolate the veto, which is what keeps an edge + object's epochs safe if the fraction cut is ever relaxed. + + Failure mode: the central veto does not read the off-tile set. + """ + monkeypatch.setattr(ngmix_module, "EPOCH_MASKED_FRACTION_CUT", 1.0) + sky = np.random.default_rng(5).normal(0.0, 1.0, (N_STAMP, N_STAMP)) + far, near = sky.copy(), sky.copy() + rn = int(EPOCH_CENTRAL_DEFECT_RADIUS) + far[:, :_CENTRE - rn + 1] = _MARKER + near[:, :_CENTRE - 8] = _MARKER + assert (near == _MARKER).mean() > EPOCH_MASKED_FRACTION_CUT + kept, _, _ = _marker_stamp(far) + assert len(kept.gals) == len(_MARKER_EPOCH_NAMES) + vetoed, _, _ = _marker_stamp(near) + assert len(vetoed.gals) == 0 + assert vetoed.epoch_cuts["central_veto"] == len(_MARKER_EPOCH_NAMES) + + +def test_corner_off_tile_region_and_border_neighbour_are_classified(): + """At a tile corner, exactly the L-shaped off-tile region gets zero + weight, and a neighbour footprint touching the stamp border without + filling a row or column stays in the neighbour mask. + + Failure modes: off-tile pixels are classified by something other than + whole marked rows and columns (the L is missed or a border-touching + neighbour is swallowed); the classification ignores the MegaCam flip. + """ + tile = np.random.default_rng(5).normal(0.0, 1.0, (N_STAMP, N_STAMP)) + tile[:2, :] = _MARKER + tile[:, -2:] = _MARKER + tile[40:, :6] = _MARKER # neighbour on the bottom-left border + stamp, epochs, _ = _marker_stamp(tile) + names = {id(v[0]): name for name, v in epochs.items()} + assert len(stamp.gals) == len(_MARKER_EPOCH_NAMES) + for flag, weight, neighbour in zip( + stamp.flags, stamp.weights, stamp.neighbours + ): + name = names[id(flag)] + off_tile = _off_tile_expected(tile, name) + assert off_tile.sum() == 2 * N_STAMP * 2 - 4 + npt.assert_array_equal(weight == 0, off_tile) + npt.assert_array_equal(flag, 0) + npt.assert_array_equal( + neighbour, _expected_neighbours(tile, name) & ~off_tile + ) + assert neighbour.sum() == 11 * 6 + + +@pytest.mark.parametrize("blend_handling", ["noisefill", "uberseg"]) +def test_off_tile_pixels_are_zero_weighted_and_filled(blend_handling): + """Off-tile pixels are zero-weighted and noise-filled under either + BLEND_HANDLING; under uberseg the neighbour markers stay raw. + + Failure mode: under uberseg the off-tile band keeps its weight, or the + fill differs between blend handlings. + """ + tile = np.random.default_rng(5).normal(0.0, 1.0, (N_STAMP, N_STAMP)) + tile[:5, :] = _MARKER + tile[20:24, 35:40] = _MARKER + stamp, _, _ = _marker_stamp(tile) + seg = np.zeros((N_STAMP, N_STAMP), dtype=np.int32) + seg[_CENTRE - 1:_CENTRE + 2, _CENTRE - 1:_CENTRE + 2] = 1 + kwargs = ( + dict(seg=seg, object_number=1) if blend_handling == "uberseg" else {} + ) + for i in range(len(stamp.gals)): + gal = 1.0e3 + stamp.gals[i] + gal_out, w_out, _ = _stamp_epoch_weights( + stamp, i, seed=i, blend_handling=blend_handling, **kwargs, + ) + off_tile = stamp.weights[i] == 0 + assert off_tile.sum() == 5 * N_STAMP + removed = off_tile | ( + stamp.neighbours[i] if blend_handling == "noisefill" else False + ) + npt.assert_array_equal(gal_out != gal, removed) + npt.assert_array_equal(w_out == 0.0, removed) + + +# --- Tile VIGNETs marked as SExtractor marks them --------------------------- +# +# SExtractor writes -1e30 into the tile VIGNET off the image and on the +# segmentation footprints of other detections. The off-image pixels are the +# off-tile defects and the footprint pixels are the neighbour mask, exactly. + +DR6_PATCH = Path(__file__).parent / "data" / "dr6_202.301_seg_patch.fits" + + +def _marked_stamps(seg, x, y): + """Tile VIGNETs on a unit image, marked -1e30 off the image and on every + footprint but the one under the stamp's centre pixel, and each stamp's + off-image mask.""" + col = np.rint(np.asarray(x)).astype(np.int64) - 1 + row = np.rint(np.asarray(y)).astype(np.int64) - 1 + seg_stamps = cut_stamps(seg.astype(np.int64), col, row, N_STAMP, -1) + off_image = [s == -1 for s in seg_stamps] + own = seg[row, col] + vignets = np.ones(seg_stamps.shape, np.float32) + for vign, s, o in zip(vignets, seg_stamps, own): + vign[(s == -1) | ((s != 0) & (s != o))] = _MARKER + return vignets, off_image + + +def test_dr6_marked_stamps_split_into_off_image_and_neighbours(): + """On the real 202.301 segmentation patch, every marked stamp splits into + its off-image pixels (off-tile) and its other -1e30 pixels (neighbours). + + Failure modes: a float32 -1e30 is not recognised as a + marker; off-image pixels of an edge stamp land in the neighbour mask; + neighbour-footprint pixels become off-tile defects. + """ + with fits.open(DR6_PATCH) as hdul: + seg = hdul["SEG"].data + objects = hdul["OBJECTS"].data + x, y = np.array(objects["X_IMAGE"]), np.array(objects["Y_IMAGE"]) + vignets, off_image = _marked_stamps(seg, x, y) + assert vignets.dtype == np.float32 + n_edge = 0 + for vign, off in zip(vignets, off_image): + neighbour, off_tile = split_tile_markers(vign, vign.shape) + npt.assert_array_equal(off_tile, off) + npt.assert_array_equal(neighbour, (vign == _MARKER) & ~off) + n_edge += off.any() + assert n_edge >= 5 + assert sum( + split_tile_markers(v, v.shape)[0].sum() for v in vignets + ) > 1000 + + +def test_a_neighbour_completing_rows_beside_the_tile_edge_stays_a_neighbour(): + """An object 20 px from the tile's left edge, with a wide neighbour + footprint that runs from the tile edge across the stamp: in the rows of + that footprint every stamp pixel is -1e30, off the image or on the + neighbour. Only the off-image columns are off-tile; the footprint is the + neighbour mask, through prepare_postage_stamps. + + Failure mode: every entirely -1e30 row counts as off-tile, so the + neighbour's rows become defects that the epoch cuts count and the defect + fill interpolates (off-tile-is-marked-border-rows-and-columns). + """ + seg = np.zeros((80, 80), np.int32) + seg[20:24, 0:50] = 5 + seg[27:32, 19:24] = 1 + x, y = np.array([21.0, 30.0]), np.array([30.0, 22.0]) + vignets, off_image = _marked_stamps(seg, x, y) + tile, off = vignets[0], off_image[0] + footprint = (tile == _MARKER) & ~off + assert footprint.sum() == 4 * (N_STAMP - 5) + assert ((tile == _MARKER).all(axis=1) & ~off.all(axis=1)).sum() == 4 + + stamp, epochs, _ = _marker_stamp(tile) + names = {id(v[0]): name for name, v in epochs.items()} + assert len(stamp.flags) == len(_MARKER_EPOCH_NAMES) + for flag, weight, neighbour in zip( + stamp.flags, stamp.weights, stamp.neighbours + ): + ccd = int(names[id(flag)].split("-")[1]) + npt.assert_array_equal(weight == 0, Ngmix.MegaCamFlip(off, ccd)) + npt.assert_array_equal(neighbour, Ngmix.MegaCamFlip(footprint, ccd)) + + +# --- Interpolation beside a removed neighbour ------------------------------- + +def _defect_beside_neighbour(): + """A single flagged pixel 6 px right of the centre, with a bright marked + neighbour footprint starting on the next column.""" + n = 31 + c = n // 2 + gal = np.random.default_rng(3).normal(0.0, 1.0, (n, n)) + flag = np.zeros((n, n), dtype=np.int32) + flag[c, c + 6] = 1 + neighbour = np.zeros((n, n), dtype=bool) + neighbour[c - 2:c + 3, c + 7:c + 10] = True + gal[neighbour] += 1.0e4 + seg = np.zeros((n, n), dtype=np.int32) + seg[c - 1:c + 2, c - 1:c + 2] = 1 + return gal, flag, neighbour, seg, (c, c + 6) + + +def test_noisefill_interpolation_does_not_read_removed_neighbour_light(): + """Under noisefill, the interpolant of a defect beside a marked neighbour + is built from the pixels the image keeps: the neighbour's light, which + noisefill removes, is not in its support. Under uberseg the neighbour + light is raw in the image and supports the interpolant. + + Failure mode: the support includes the removed neighbour pixels, so the + defect is filled with light the image no longer contains + (interpolation-support-is-the-kept-image). + """ + gal, flag, neighbour, seg, pix = _defect_beside_neighbour() + weight = np.ones_like(gal) + target = np.zeros_like(neighbour) + target[pix] = True + + out, _, _ = prepare_ngmix_weights( + gal, weight, flag, np.random.RandomState(0), + blend_handling="noisefill", neighbour=neighbour, + ) + expected = interpolate_defects(gal[None], (flag != 0) | neighbour, target) + assert out[pix] == pytest.approx(expected[0][pix]) + assert abs(out[pix]) < 100.0 + + out, _, _ = prepare_ngmix_weights( + gal, weight, flag, np.random.RandomState(0), + blend_handling="uberseg", seg=seg, object_number=1, + neighbour=neighbour, + ) + expected = interpolate_defects(gal[None], flag != 0, target) + assert out[pix] == pytest.approx(expected[0][pix]) + assert out[pix] > 1000.0 + + +def test_a_column_beside_a_noisefill_neighbour_is_vetoed_as_noise_filled(): + """A column 8 px from the centre is interpolated, and kept, unless a + marked neighbour lies against it. Under noisefill the neighbour's light + is removed, so the column's rows that end on the neighbour cannot be + interpolated; the fill noise-fills them, and the veto drops the epoch at + the noise-fill radius. Under uberseg the neighbour's light stays and + supports the interpolant, so the epoch is kept. + + Failure mode: the veto decides which pixels are interpolated from the + defect mask alone, so it keeps the epoch at the 7-px radius while the + fill noise-fills pixels 8 px from the object, inside the radius that + noise fill needs. + """ + column = np.zeros((N_STAMP, N_STAMP), dtype=np.int32) + column[:, _CENTRE + 8] = 1 + # CCD 20 is not flipped, so tile and epoch share their orientation. + epochs = {"2100001-20": (column, np.ones((N_STAMP, N_STAMP)))} + sky = np.random.default_rng(5).normal(0.0, 1.0, (N_STAMP, N_STAMP)) + tile = sky.copy() + tile[_CENTRE - 3:_CENTRE + 4, _CENTRE + 9:_CENTRE + 14] = _MARKER + for tile_vign, blend_handling, kept in ( + (sky, "noisefill", 1), + (tile, "noisefill", 0), + (tile, "uberseg", 1), + ): + stamp, _, _ = _marker_stamp( + tile_vign, epochs, blend_handling=blend_handling, + ) + assert len(stamp.gals) == kept + assert stamp.epoch_cuts["central_veto"] == 1 - kept + + # The fill agrees: under noisefill only the rows clear of the neighbour + # are interpolated (their quarter turns lose weight too); the rows + # against it are noise-filled. + neighbour = tile == _MARKER + defect = column != 0 + interpolated = defect & ~(neighbour[:, _CENTRE + 9][:, None]) + _, weights, _ = prepare_ngmix_weights( + sky, np.ones_like(sky), column, np.random.RandomState(0), + bkg_rms=np.ones_like(sky), neighbour=neighbour, + ) + npt.assert_array_equal( + weights == 0.0, defect | neighbour | fourfold(interpolated) + ) diff --git a/tests/module/test_ngmix_tile_cat_chunk.py b/tests/module/test_ngmix_tile_cat_chunk.py new file mode 100644 index 000000000..49d3ed78c --- /dev/null +++ b/tests/module/test_ngmix_tile_cat_chunk.py @@ -0,0 +1,116 @@ +"""A chunk's Tile_cat holds the stamps of its own rows, and only those. + +Each ngmix chunk reads the tile catalogue for rows ``chunk_rows(n_obj, +row_min, row_max)``. The per-object columns stay full length; the stamp +columns (``VIGNET`` and ``SEG_VIGNET``) are held for the chunk's rows only, +indexed by tile-catalogue row exactly like the full column. +""" + +import numpy as np +import pytest +from astropy.io import fits + +from shapepipe.modules.ngmix_package.ngmix import ( + ChunkStamps, + Tile_cat, + chunk_rows, +) + +N_OBJ = 23 +STAMP = 7 + + +def _ldac(path, columns): + imhead = fits.BinTableHDU.from_columns( + [fits.Column(name="Field Header Card", format="1A", array=["x"])], + name="LDAC_IMHEAD", + ) + objects = fits.BinTableHDU.from_columns(columns, name="LDAC_OBJECTS") + fits.HDUList([fits.PrimaryHDU(), imhead, objects]).writeto(path) + return str(path) + + +@pytest.fixture +def catalogue(tmp_path): + """A tile catalogue with VIGNET and SEG_VIGNET stamp columns.""" + rng = np.random.default_rng(3) + number = rng.permutation(np.arange(1, 10 * N_OBJ, 10))[:N_OBJ] + vign = rng.normal(size=(N_OBJ, STAMP, STAMP)).astype(np.float32) + vign[::4, 0, :] = -1e30 + vign[1, 2, 2] = np.nan + seg = rng.integers(0, 5, size=(N_OBJ, STAMP, STAMP)).astype(np.int32) + dim = f"({STAMP}, {STAMP})" + cat = _ldac(tmp_path / "sexcat.fits", [ + fits.Column(name="NUMBER", format="J", array=number), + fits.Column(name="XWIN_WORLD", format="D", array=rng.uniform(size=N_OBJ)), + fits.Column(name="YWIN_WORLD", format="D", array=rng.uniform(size=N_OBJ)), + fits.Column(name="FLUX_AUTO", format="E", array=rng.uniform(size=N_OBJ)), + fits.Column( + name="VIGNET", format=f"{STAMP * STAMP}E", dim=dim, array=vign + ), + fits.Column( + name="SEG_VIGNET", format=f"{STAMP * STAMP}J", dim=dim, array=seg + ), + ]) + return cat + + +def _full_columns(path): + """Every column as an in-memory (not memory-mapped) read gives it.""" + with fits.open(path, memmap=False) as hdul: + data = hdul[2].data + return {name: np.copy(data[name]) for name in data.dtype.names} + + +@pytest.mark.parametrize( + "row_min, row_max", + [(-1, -1), (1, 6), (7, 15), (16, N_OBJ), (20, -1), (N_OBJ + 1, N_OBJ)], +) +def test_chunk_holds_exactly_its_rows(catalogue, row_min, row_max): + full = _full_columns(catalogue) + rows = chunk_rows(N_OBJ, row_min, row_max) + + tile = Tile_cat(catalogue, row_min=row_min, row_max=row_max) + + assert tile.rows == rows + # Per-object columns: full length, every row. + np.testing.assert_array_equal(tile.obj_id, full["NUMBER"]) + np.testing.assert_array_equal(tile.ra, full["XWIN_WORLD"]) + np.testing.assert_array_equal(tile.dec, full["YWIN_WORLD"]) + np.testing.assert_array_equal(tile.flux, full["FLUX_AUTO"]) + + for stamps, column in ((tile.vign, full["VIGNET"]), + (tile.seg, full["SEG_VIGNET"])): + # Same stamps, bit for bit, at the same tile-catalogue rows. + for i_tile in rows: + assert stamps[i_tile].dtype == column[i_tile].dtype + assert stamps[i_tile].tobytes() == column[i_tile].tobytes() + # Only the chunk's stamps are held, in memory of their own. + assert stamps._stamps.shape == (len(rows), STAMP, STAMP) + assert stamps._stamps.flags.owndata + # The other rows are absent, and touching one says so. + for i_tile in set(range(N_OBJ)) - set(rows): + with pytest.raises(IndexError, match="outside this chunk"): + stamps[i_tile] + + +def test_stamps_outlive_the_catalogue_file(catalogue): + """The held stamps are copies, not views into the memory-mapped file.""" + full = _full_columns(catalogue) + tile = Tile_cat(catalogue, row_min=5, row_max=9) + # Overwrite the file in place with different stamps. + with fits.open(catalogue, mode="update") as hdul: + hdul[2].data["VIGNET"][:] = 0 + hdul[2].data["SEG_VIGNET"][:] = 0 + assert tile.vign._stamps.tobytes() == full["VIGNET"][4:9].tobytes() + assert tile.seg._stamps.tobytes() == full["SEG_VIGNET"][4:9].tobytes() + + +def test_chunk_stamps_index_by_tile_row(): + column = np.arange(10 * 4).reshape(10, 2, 2) + stamps = ChunkStamps(column, range(3, 6)) + np.testing.assert_array_equal(stamps[np.int64(3)], column[3]) + np.testing.assert_array_equal(stamps[5], column[5]) + for i_tile in (2, 6, -1): + with pytest.raises(IndexError): + stamps[i_tile] diff --git a/tests/module/test_ngmix_uberseg.py b/tests/module/test_ngmix_uberseg.py index 7e7790c55..4be3ee697 100644 --- a/tests/module/test_ngmix_uberseg.py +++ b/tests/module/test_ngmix_uberseg.py @@ -2,14 +2,14 @@ Covers the ``BLEND_HANDLING = uberseg`` option added to the ngmix module: -* :func:`uberseg_weight` — the Sheldon/MEDS nearest-segment Voronoi mask. +* :func:`uberseg_mask` — the Sheldon/MEDS nearest-segment Voronoi mask. Geometry assertions on a synthetic two-object stamp: neighbour-side pixels - are zeroed, the surviving central core is a *single connected* region (the - emergent "circularisation"), and the neighbour footprint is fully removed. -* :func:`prepare_ngmix_weights` — the ``noisefill`` default is byte-for-byte - unchanged (asserted against an independent recomputation of the legacy - three-line noise-fill on a shared RNG), while ``uberseg`` hard-masks the - weight (weight -> 0) and leaves the image untouched. + are masked, the surviving central core is a *single connected* region (the + emergent "circularisation"), and the neighbour footprint is fully masked. +* :func:`prepare_ngmix_weights` under ``uberseg`` — neighbour-side pixels + lose their weight and keep their raw image values, while defect pixels are + filled as under any blend handling (the defect fill itself is covered in + ``test_ngmix_defect_fill.py``). * The error contract when ``uberseg`` is selected without a segmentation map (the seg-map source is plumbing-gated upstream). """ @@ -17,15 +17,17 @@ import numpy as np import numpy.testing as npt import pytest +from astropy.io import fits from scipy import ndimage from sqlitedict import SqliteDict from shapepipe.modules.ngmix_package.ngmix import ( Ngmix, + Tile_cat, central_seg_label, prepare_ngmix_weights, seg_has_neighbour, - uberseg_weight, + uberseg_mask, ) @@ -53,33 +55,24 @@ def two_object_seg(npix=41, sep=12, r_central=3, r_neighbour=3): return seg, centre, neigh -def test_uberseg_zeros_neighbour_side_keeps_centre(): - """Neighbour-side pixels lose their weight; the central pixel keeps it.""" +def test_uberseg_masks_neighbour_side_keeps_centre(): + """Neighbour-side pixels are masked; the central footprint is not.""" seg, centre, neigh = two_object_seg() - weight = np.ones_like(seg, dtype=float) - out = uberseg_weight(weight, seg, object_number=1) + masked = uberseg_mask(seg, object_number=1) - # Central object pixel kept; deep-neighbour pixel zeroed. - assert out[centre] == 1.0 - assert out[neigh] == 0.0 - # Every neighbour-footprint pixel is removed from the fit. - assert np.all(out[seg == 2] == 0.0) - # Every central-footprint pixel survives. - assert np.all(out[seg == 1] == 1.0) - # The input weight is not mutated in place. - assert np.all(weight == 1.0) + assert not masked[centre] + assert masked[neigh] + assert np.all(masked[seg == 2]) + assert not np.any(masked[seg == 1]) def test_uberseg_core_is_single_connected_region(): """The surviving (kept-weight) region is one connected component — the emergent circular core of the nearest-segment Voronoi partition.""" seg, _, _ = two_object_seg() - weight = np.ones_like(seg, dtype=float) - out = uberseg_weight(weight, seg, object_number=1) - - kept = out > 0 + kept = ~uberseg_mask(seg, object_number=1) _, n_components = ndimage.label(kept) assert n_components == 1 # The partition splits the stamp: some pixels survive, some are masked. @@ -91,28 +84,26 @@ def test_uberseg_partition_is_the_perpendicular_bisector(): half-plane beyond the footprint bisector: the left edge survives, the column past the neighbour is gone.""" seg, centre, neigh = two_object_seg(npix=41, sep=12) - weight = np.ones_like(seg, dtype=float) - out = uberseg_weight(weight, seg, object_number=1) + masked = uberseg_mask(seg, object_number=1) c_row, c_col = centre - assert out[c_row, 0] == 1.0 # far side from the neighbour: kept - assert out[c_row, -1] == 0.0 # neighbour side edge: masked + assert not masked[c_row, 0] # far side from the neighbour: kept + assert masked[c_row, -1] # neighbour side edge: masked def test_uberseg_no_neighbour_is_passthrough(): - """A stamp with only the central object (or empty seg) is unchanged.""" + """A stamp with only the central object (or empty seg) masks nothing.""" npix = 21 - weight = np.random.default_rng(0).random((npix, npix)) + 0.1 # Only the central object present. seg = np.zeros((npix, npix), dtype=np.int32) seg[8:13, 8:13] = 1 - npt.assert_array_equal(uberseg_weight(weight, seg, object_number=1), weight) + assert not uberseg_mask(seg, object_number=1, dilate_neighbour=2).any() # Wholly empty seg (no detections). empty = np.zeros((npix, npix), dtype=np.int32) - npt.assert_array_equal(uberseg_weight(weight, empty, object_number=1), weight) + assert not uberseg_mask(empty, object_number=1).any() # --- central_seg_label: centre-pixel identification (#776 decision 3) ------- @@ -162,7 +153,7 @@ def test_seg_has_neighbour(): # --- uberseg dilation: additive neighbour-mask enlargement (#776 dec. 2) ---- def test_uberseg_dilation_grows_neighbour_mask(): - """dilate_neighbour>0 zeros a strict superset of the base (dilate=0) mask, + """dilate_neighbour>0 masks a strict superset of the base (dilate=0) mask, and every base-masked pixel stays masked (additive-only). Geometric subtlety: for well-separated objects the base Voronoi cut (the @@ -171,62 +162,53 @@ def test_uberseg_dilation_grows_neighbour_mask(): crosses the bisector into the central Voronoi cell. A few iterations guarantee that crossing here (sep=8, r=3 -> ~2px footprint gap).""" seg, _, _ = two_object_seg(npix=41, sep=8) - weight = np.ones_like(seg, dtype=float) - out0 = uberseg_weight(weight, seg, object_number=1, dilate_neighbour=0) - out3 = uberseg_weight(weight, seg, object_number=1, dilate_neighbour=3) + out0 = uberseg_mask(seg, object_number=1, dilate_neighbour=0) + out3 = uberseg_mask(seg, object_number=1, dilate_neighbour=3) - # dilate=0 reproduces the validated no-dilation result byte-for-byte. - npt.assert_array_equal( - out0, uberseg_weight(weight, seg, object_number=1) - ) + # dilate=0 reproduces the validated no-dilation result. + npt.assert_array_equal(out0, uberseg_mask(seg, object_number=1)) # Every pixel masked at dilate=0 is still masked at dilate=3 (additive). - assert np.all(out3[out0 == 0.0] == 0.0) + assert np.all(out3[out0]) # And strictly more pixels are masked once the dilation crosses the # bisector into the central cell. - assert (out3 == 0.0).sum() > (out0 == 0.0).sum() + assert out3.sum() > out0.sum() def test_uberseg_dilation_zero_is_pure_sheldon(): """dilate_neighbour=0 is bit-identical to the default (no-kwarg) call and to the O(N^2) brute-force nearest-segment rule.""" seg, _, _ = two_object_seg(npix=25, sep=8) - weight = np.ones_like(seg, dtype=float) - out = uberseg_weight(weight, seg, object_number=1, dilate_neighbour=0) - npt.assert_array_equal(out, uberseg_weight(weight, seg, object_number=1)) - - obj = np.argwhere(seg != 0) - labels = seg[seg != 0] - brute = np.ones_like(weight) - for i in range(seg.shape[0]): - for j in range(seg.shape[1]): - d2 = (i - obj[:, 0]) ** 2 + (j - obj[:, 1]) ** 2 - if labels[np.argmin(d2)] != 1: - brute[i, j] = 0.0 - npt.assert_array_equal(out, brute) + out = uberseg_mask(seg, object_number=1, dilate_neighbour=0) + npt.assert_array_equal(out, uberseg_mask(seg, object_number=1)) + npt.assert_array_equal(out, _brute_force_uberseg(seg)) def test_uberseg_matches_bruteforce_nearest_segment(): """The cKDTree result equals the O(N^2) brute-force nearest-segment rule (Sheldon's non-C fallback) it stands in for.""" seg, _, _ = two_object_seg(npix=25, sep=8) - weight = np.ones_like(seg, dtype=float) - out = uberseg_weight(weight, seg, object_number=1) + npt.assert_array_equal( + uberseg_mask(seg, object_number=1), _brute_force_uberseg(seg) + ) + +def _brute_force_uberseg(seg): + """O(N^2) nearest-segment rule: True where the nearest footprint pixel + is not the central object's (label 1).""" obj = np.argwhere(seg != 0) labels = seg[seg != 0] - brute = np.ones_like(weight) + brute = np.zeros(seg.shape, dtype=bool) for i in range(seg.shape[0]): for j in range(seg.shape[1]): d2 = (i - obj[:, 0]) ** 2 + (j - obj[:, 1]) ** 2 - if labels[np.argmin(d2)] != 1: - brute[i, j] = 0.0 - npt.assert_array_equal(out, brute) + brute[i, j] = labels[np.argmin(d2)] != 1 + return brute -# --- prepare_ngmix_weights: default unchanged, uberseg hard-masks ---------- +# --- prepare_ngmix_weights: uberseg zeroes neighbour weights only --------- def _gal_flag_weight(npix=41, seed=7): rng = np.random.default_rng(seed) @@ -239,35 +221,9 @@ def _gal_flag_weight(npix=41, seed=7): return gal, flag, weight -def test_noisefill_default_is_byte_identical_to_legacy(): - """The default path reproduces the legacy three-line noise-fill exactly - (same RNG stream): masked pixels replaced by noise, weight 1/sigma^2.""" - gal, flag, weight = _gal_flag_weight() - - gal_out, w_out, noise_out = prepare_ngmix_weights( - gal, weight, flag, np.random.RandomState(123), - ) - - # Independent recomputation of the legacy algorithm on the same stream. - from modopt.math.stats import sigma_mad - rng = np.random.RandomState(123) - mask = np.copy(weight) != 0 - mask[flag != 0] = False - sig = sigma_mad(gal) - w_exp = mask.astype(float) / sig ** 2 - noise_exp = rng.standard_normal(gal.shape) * sig - noise_gal = rng.standard_normal(gal.shape) * sig - gal_exp = np.copy(gal) - gal_exp[~mask] = noise_gal[~mask] - - npt.assert_array_equal(gal_out, gal_exp) - npt.assert_array_equal(w_out, w_exp) - npt.assert_array_equal(noise_out, noise_exp) - - def test_noisefill_ignores_seg_and_dilate_kwargs(): - """Under noisefill, passing seg / dilate_neighbour changes nothing: the - result matches the plain default call on the same RNG stream.""" + """Under BLEND_HANDLING = noisefill, passing seg / dilate_neighbour changes + nothing: the result matches the plain default call on the same RNG stream.""" gal, flag, weight = _gal_flag_weight() seg, _, _ = two_object_seg(npix=gal.shape[0], sep=12) @@ -282,9 +238,16 @@ def test_noisefill_ignores_seg_and_dilate_kwargs(): npt.assert_array_equal(a, b) -def test_uberseg_hard_masks_weight_and_leaves_image_untouched(): - """uberseg: image returned untouched, weight zeroed on neighbour-side and - flagged pixels, positive on the central core.""" +def test_uberseg_fills_defects_and_leaves_neighbour_pixels_raw(): + """uberseg: flagged pixels are filled at weight 0; neighbour-side + pixels get weight 0 and keep their raw image values; the central core + keeps weight and image. + + Failure modes: the defect fill is skipped under uberseg, so raw bad + pixels reach metacal; or the neighbour side is noise-filled + (defects-filled-whatever-the-blend-handling, + uberseg-ignores-markers). + """ npix = 41 gal, flag, weight = _gal_flag_weight(npix=npix) seg, centre, neigh = two_object_seg(npix=npix, sep=12) @@ -294,13 +257,16 @@ def test_uberseg_hard_masks_weight_and_leaves_image_untouched(): blend_handling="uberseg", seg=seg, object_number=1, ) - # Image untouched under uberseg (no noise fill). - npt.assert_array_equal(gal_out, gal) - # Neighbour footprint hard-masked; central centre kept. + # Flagged pixels: zero weight, raw value replaced. + for pix in [(5, 5), (30, 12)]: + assert w_out[pix] == 0.0 + assert gal_out[pix] != gal[pix] + # Neighbour footprint: zero weight, raw image (never noise-filled). assert np.all(w_out[seg == 2] == 0.0) + npt.assert_array_equal(gal_out[seg == 2], gal[seg == 2]) + # Central core: weight and image untouched. assert w_out[centre] > 0.0 - # Flagged bad pixels remain at weight 0 (folded into the base mask). - assert w_out[5, 5] == 0.0 + assert gal_out[centre] == gal[centre] def test_uberseg_requires_seg_and_object_number(): @@ -366,59 +332,100 @@ def test_check_central_seg_label_missing_number_raises(tmp_path): ngmix._check_central_seg_label(seg, obj_id=99) -# --- FIX 3a: uberseg without seg_cat_path fails at construction ------------- - -def test_ngmix_init_uberseg_without_seg_cat_raises(tmp_path): - """blend_handling='uberseg' with seg_cat_path=None raises at __init__.""" +# --- the seg stamps ride the tile catalogue as SEG_VIGNET ------------------- + +def _write_tile_cat(path, seg_vignets=None): + """A three-object LDAC tile catalogue, with SEG_VIGNET when given.""" + n_obj, size = 3, 5 + cols = [ + fits.Column(name="NUMBER", format="J", array=np.array([4, 5, 9])), + fits.Column(name="XWIN_WORLD", format="D", array=np.zeros(n_obj)), + fits.Column(name="YWIN_WORLD", format="D", array=np.zeros(n_obj)), + fits.Column(name="VIGNET", format=f"{size * size}E", + array=np.ones((n_obj, size * size), np.float32), + dim=f"({size},{size})"), + ] + if seg_vignets is not None: + cols.append(fits.Column(name="SEG_VIGNET", format=f"{size * size}J", + array=seg_vignets.reshape(n_obj, -1), + dim=f"({size},{size})")) + imhead = fits.BinTableHDU.from_columns( + [fits.Column(name="Field Header Card", format="1A", array=["x"])], + name="LDAC_IMHEAD", + ) + fits.HDUList([ + fits.PrimaryHDU(), imhead, + fits.BinTableHDU.from_columns(cols, name="LDAC_OBJECTS"), + ]).writeto(path) + return path + + +def test_tile_cat_reads_seg_vignet(tmp_path): + """Tile_cat's seg stamps are the catalogue's SEG_VIGNET, row for row; + a catalogue without the column has none.""" + seg = np.arange(3 * 25, dtype=np.int32).reshape(3, 5, 5) + with_seg = Tile_cat(str(_write_tile_cat(tmp_path / "a.fits", seg))) + for i_tile in range(len(seg)): + npt.assert_array_equal(with_seg.seg[i_tile], seg[i_tile]) + assert with_seg.seg[i_tile].dtype.kind == "i" + assert Tile_cat(str(_write_tile_cat(tmp_path / "b.fits"))).seg is None + + +def test_uberseg_without_seg_vignet_fails_loudly(tmp_path): + """BLEND_HANDLING = uberseg on a catalogue without SEG_VIGNET raises + before any object is fitted, rather than dropping every object.""" + cat = _write_tile_cat(tmp_path / "tile_cat.fits") names = ("gal", "bkg", "psf", "weight", "flag", "headers") paths = [tmp_path / f"{name}.sqlite" for name in names] for path in paths: SqliteDict(str(path)).close() - with pytest.raises(ValueError, match="requires SEG_VIGNET_PATH"): - Ngmix( - ["tile_cat.fits"] + [str(p) for p in paths[:5]], - str(tmp_path), "-001-001", 30.0, str(paths[5]), - _RecordingLogger(), - blend_handling="uberseg", - seg_cat_path=None, - ) - - -# --- FIX 3b: runner raises when SEG_VIGNET_PATH is set but missing ---------- - -class _FakeConfig: - """Config stub: SEG_VIGNET_PATH is the only present option, and it resolves - to ``seg_path`` (a path the test leaves nonexistent).""" - - def __init__(self, seg_path): - self._seg_path = seg_path - - def getfloat(self, _sec, _key): - return 30.0 - - def getboolean(self, _sec, _key, fallback=False): - return fallback - - def has_option(self, _sec, key): - return key == "SEG_VIGNET_PATH" - - def getexpanded(self, _sec, _key): - return self._seg_path - - -def test_runner_missing_seg_vignet_file_raises(tmp_path): - """SEG_VIGNET_PATH configured but the resolved file is absent -> the runner - fails fast with FileNotFoundError, before constructing Ngmix.""" - from shapepipe.modules.ngmix_runner import ngmix_runner - - input_file_list = ["tile_cat.fits"] + [f"in{i}.sqlite" for i in range(6)] - seg_path = str(tmp_path / "does_not_exist.fits") - with pytest.raises(FileNotFoundError, match="Segmentation vignet file"): - ngmix_runner( - input_file_list, - {"output": str(tmp_path)}, - "-001-001", - _FakeConfig(seg_path), - "NGMIX_RUNNER", - _RecordingLogger(), - ) + ngmix = Ngmix( + [str(cat)] + [str(p) for p in paths[:5]], + str(tmp_path), "-001-001", 30.0, str(paths[5]), + _RecordingLogger(), blend_handling="uberseg", + ) + with pytest.raises(ValueError, match="SEG_VIGNET"): + ngmix.process() + + +def test_runner_reads_blend_handling_from_the_environment( + tmp_path, monkeypatch, +): + """The committed ini's ``${SP_BLEND_HANDLING:-noisefill}`` reaches Ngmix + as noisefill when the variable is unset and as its value when set.""" + from shapepipe.modules import ngmix_runner as runner_module + from shapepipe.pipeline.config import CustomParser + + names = ("cat", "gal", "bkg", "psf", "weight", "flag", "headers") + paths = [str(tmp_path / f"{name}.sqlite") for name in names] + for path in paths[1:]: + SqliteDict(path).close() + config = CustomParser() + config.add_section("NGMIX_RUNNER") + for key, value in {"MAG_ZP": "30.0", "ID_OBJ_MIN": "-1", + "ID_OBJ_MAX": "-1", + "BLEND_HANDLING": "${SP_BLEND_HANDLING:-noisefill}", + }.items(): + config.set("NGMIX_RUNNER", key, value) + + seen = [] + + class _Stop(Exception): + pass + + def fake_ngmix(*_args, **kwargs): + seen.append(kwargs["blend_handling"]) + raise _Stop + + monkeypatch.setattr(runner_module, "Ngmix", fake_ngmix) + runner = getattr(runner_module.ngmix_runner, "__wrapped__", + runner_module.ngmix_runner) + for env in (None, "uberseg"): + if env is None: + monkeypatch.delenv("SP_BLEND_HANDLING", raising=False) + else: + monkeypatch.setenv("SP_BLEND_HANDLING", env) + with pytest.raises(_Stop): + runner(paths, {"output": str(tmp_path)}, "-001-001", config, + "NGMIX_RUNNER", _RecordingLogger()) + assert seen == ["noisefill", "uberseg"] diff --git a/tests/module/test_pipeline.py b/tests/module/test_pipeline.py index 400ca2ec3..125f24dc6 100644 --- a/tests/module/test_pipeline.py +++ b/tests/module/test_pipeline.py @@ -134,6 +134,46 @@ def test_custom_parser_getexpanded_expands_fallback(monkeypatch): ) +def test_custom_parser_expands_a_default_only_when_the_variable_is_unset( + monkeypatch, +): + """``${VAR:-default}`` takes ``default`` when VAR is unset or empty, and + VAR's value otherwise; a bare unset ``$VAR`` still raises.""" + parser = config.CustomParser() + parser.add_section("S") + parser.set("S", "MODE", "${SP_TEST_MODE:-noisefill}") + parser.set("S", "FLAG", "${SP_TEST_FLAG:-False}") + parser.set("S", "BARE", "$SP_TEST_MODE") + + monkeypatch.delenv("SP_TEST_MODE", raising=False) + monkeypatch.delenv("SP_TEST_FLAG", raising=False) + assert parser.getexpanded("S", "MODE") == "noisefill" + assert parser.getexpandedboolean("S", "FLAG") is False + with pytest.raises(ValueError, match="SP_TEST_MODE"): + parser.getexpanded("S", "BARE") + + monkeypatch.setenv("SP_TEST_MODE", "") + assert parser.getexpanded("S", "MODE") == "noisefill" + + monkeypatch.setenv("SP_TEST_MODE", "uberseg") + monkeypatch.setenv("SP_TEST_FLAG", "True") + assert parser.getexpanded("S", "MODE") == "uberseg" + assert parser.getexpanded("S", "BARE") == "uberseg" + assert parser.getexpandedboolean("S", "FLAG") is True + + +def test_custom_parser_expands_a_variable_value_once(monkeypatch): + """A variable's value is inserted as is, never expanded again.""" + monkeypatch.setenv("SP_TEST_ROOT", "/data/$SP_TEST_OTHER") + monkeypatch.setenv("SP_TEST_OTHER", "expanded-again") + parser = config.CustomParser() + parser.add_section("S") + parser.set("S", "PATH", "${SP_TEST_ROOT:-/x}/a $SP_TEST_ROOT/b") + assert parser.getexpanded("S", "PATH") == ( + "/data/$SP_TEST_OTHER/a /data/$SP_TEST_OTHER/b" + ) + + def test_custom_parser_getlist_honours_custom_delimiter(): parser = config.CustomParser() diff --git a/tests/module/test_psf_averaging_properties.py b/tests/module/test_psf_averaging_properties.py index d22435719..d6799bc29 100644 --- a/tests/module/test_psf_averaging_properties.py +++ b/tests/module/test_psf_averaging_properties.py @@ -258,12 +258,18 @@ def test_all_epochs_failed_raises_zero_division(flagged_specs): "NGMIX_MCAL_FLAGS": LM_FUNC_NOTFINITE, "NGMIX_MCAL_TYPES_FAIL": len(METACAL_TYPES), "NGMIX_NEIGHBOUR_FLAG": 0.0, + "NGMIX_N_EPOCH_FAILED": 0.0, + "NGMIX_N_EPOCH_INTERP": 0.0, + "NGMIX_MIN_DIST_INTERP": -1.0, + "NGMIX_MIN_DIST_NOISEFILL": -1.0, } # Per-key write format and a measured value distinct from every sentinel, so a # matched row is unmistakably "overwritten" and an absent row unmistakably not. _NGMIX_KEYS = [ - "id", "n_epoch_model", "mcal_types_fail", "neighbour_flag", "nfev_fit", + "id", "n_epoch_model", "n_epoch_failed", "mcal_types_fail", + "neighbour_flag", + "n_epoch_interp", "min_dist_interp", "min_dist_noisefill", "nfev_fit", "g1", "g1_err", "g2", "g2_err", "T", "T_err", "flux", "flux_err", "s2n", "mag", "mag_err", "flags", "mcal_flags", "g1_psf_orig", "g2_psf_orig", "g1_err_psf_orig", "g2_err_psf_orig", @@ -272,8 +278,9 @@ def test_all_epochs_failed_raises_zero_division(flagged_specs): "T_psf_reconv", "T_err_psf_reconv", ] _INT_KEYS = { - "id", "n_epoch_model", "mcal_types_fail", "neighbour_flag", "nfev_fit", - "flags", "mcal_flags" + "id", "n_epoch_model", "n_epoch_failed", "mcal_types_fail", + "neighbour_flag", + "n_epoch_interp", "nfev_fit", "flags", "mcal_flags" } _SHEAR_EXTS = ["1M", "1P", "2M", "2P", "NOSHEAR"] diff --git a/tests/module/test_psf_grammar_properties.py b/tests/module/test_psf_grammar_properties.py index 461597557..4a6442f13 100644 --- a/tests/module/test_psf_grammar_properties.py +++ b/tests/module/test_psf_grammar_properties.py @@ -4,9 +4,10 @@ hypothesis to pin three invariants of the renamed grammar ``NGMIX[m]_[_ERR][_]_`` (shapepipe#749, #761): the galaxy is the implicit default object and carries NO ``OBJECT`` token -(``NGMIX_G1_NOSHEAR``, never ``NGMIX_G1_GAL_NOSHEAR``), plus the four +(``NGMIX_G1_NOSHEAR``, never ``NGMIX_G1_GAL_NOSHEAR``), plus the OBJECT/SHEAR-less metadata columns (``NGMIX[m]_MCAL_FLAGS``, ``NGMIX_N_EPOCH``, -``NGMIX_MCAL_TYPES_FAIL``, ``NGMIX_NEIGHBOUR_FLAG``), where the ORIGINAL image +``NGMIX_MCAL_TYPES_FAIL``, ``NGMIX_NEIGHBOUR_FLAG`` and the defect +diagnostics), where the ORIGINAL image PSF (``PSF_ORIG``) and the metacal RECONVOLUTION kernel (``PSF_RECONV``) are independent fits of *different* PSFs: @@ -63,11 +64,14 @@ def info(self, *_args, **_kwargs): # shear-type extension. Integer-typed keys carry FITS format "K", the rest # "D"; everything else mirrors test_make_cat._ngmix_row. INT_KEYS = { - "id", "n_epoch_model", "mcal_types_fail", "neighbour_flag", "nfev_fit", - "flags", "mcal_flags", + "id", "n_epoch_model", "n_epoch_failed", "mcal_types_fail", + "neighbour_flag", + "n_epoch_interp", "nfev_fit", "flags", "mcal_flags", } NGMIX_KEYS = [ - "id", "n_epoch_model", "mcal_types_fail", "neighbour_flag", "nfev_fit", + "id", "n_epoch_model", "n_epoch_failed", "mcal_types_fail", + "neighbour_flag", + "n_epoch_interp", "min_dist_interp", "min_dist_noisefill", "nfev_fit", "g1", "g1_err", "g2", "g2_err", "T", "T_err", "flux", "flux_err", "s2n", "mag", "mag_err", @@ -108,8 +112,9 @@ def _base_row(obj_id): """One object's per-key values; PSF keys overwritten by the caller.""" return { "id": obj_id, - "n_epoch_model": 3, "mcal_types_fail": 0, "neighbour_flag": 1, - "nfev_fit": 7, + "n_epoch_model": 3, "n_epoch_failed": 1, "mcal_types_fail": 0, "neighbour_flag": 1, + "n_epoch_interp": 2, "min_dist_interp": 7.5, + "min_dist_noisefill": 14.25, "nfev_fit": 7, "g1": 0.10, "g1_err": 0.011, "g2": -0.20, "g2_err": 0.022, "T": 0.30, "T_err": 0.033, "flux": 100.0, "flux_err": 1.0, "s2n": 55.0, @@ -178,7 +183,8 @@ def _run_save_ngmix(ngmix_path, obj_ids, cat_size_target=None): _SHEAR = "NOSHEAR|1P|1M|2P|2M" GRAMMAR_RE = re.compile( rf"^NGMIXm?_(?:{_COMPONENT})(?:_ERR)?(?:_(?:{_OBJECT}))?_(?:{_SHEAR})$" - rf"|^NGMIXm?_(?:MCAL_FLAGS|MCAL_TYPES_FAIL|N_EPOCH|NEIGHBOUR_FLAG)$" + rf"|^NGMIXm?_(?:MCAL_FLAGS|MCAL_TYPES_FAIL|N_EPOCH|NEIGHBOUR_FLAG" + rf"|N_EPOCH_FAILED|N_EPOCH_INTERP|MIN_DIST_INTERP|MIN_DIST_NOISEFILL)$" ) # The shipped final-catalogue param files, two levels up from tests/module/. @@ -328,7 +334,7 @@ def test_emitted_column_names_match_grammar(obj_ids, tmp_path_factory): """Every column _save_ngmix_data emits matches the grammar regex. The grammar is ``NGMIX[m]_[_ERR]__`` plus the - four metadata columns. The regex is anchored, so a legacy token (``_PSFo``, + metadata columns. The regex is anchored, so a legacy token (``_PSFo``, ``_Tpsf``, ``NGMIX_ELL_*``) or a malformed name would fail to match. Run over hypothesis-varied object sets to confirm the emitted name set is input-independent and always well-formed. @@ -417,7 +423,8 @@ def test_param_file_ngmix_tokens_are_producible(param_path, obj_ids): # ngmix: galaxy (no object token) or explicit PSF_ORIG/PSF_RECONV. r"NGMIXm?_(?:G1|G2|T|SNR|FLUX|MAG|FLAGS)(?:_ERR)?" r"(?:_(?:PSF_ORIG|PSF_RECONV))?_(?:NOSHEAR|1P|1M|2P|2M)" - r"|NGMIXm?_(?:MCAL_FLAGS|MCAL_TYPES_FAIL|N_EPOCH|NEIGHBOUR_FLAG)" + r"|NGMIXm?_(?:MCAL_FLAGS|MCAL_TYPES_FAIL|N_EPOCH|NEIGHBOUR_FLAG" + r"|N_EPOCH_FAILED|N_EPOCH_INTERP|MIN_DIST_INTERP|MIN_DIST_NOISEFILL)" # HSM: g-type, explicit PSF/STAR object, singular FLAG; the multi-epoch # sink in make_cat._save_psf_data appends a bare epoch index. r"|HSM_(?:G1|G2|T|M4_1|M4_2|RHO4)_(?:PSF|STAR)(?:_\d+)?" diff --git a/tests/module/test_sextractor_seg_vignet.py b/tests/module/test_sextractor_seg_vignet.py new file mode 100644 index 000000000..ad8299ae4 --- /dev/null +++ b/tests/module/test_sextractor_seg_vignet.py @@ -0,0 +1,452 @@ +"""UNIT TESTS FOR SEXTRACTOR MODE'S SEG_VIGNET. + +``sextractor_script.add_seg_vignet`` cuts the SEGMENTATION check image on the +grid SExtractor cut each object's VIGNET on and adds it to the sexcat as the +int32 ``SEG_VIGNET`` column. SExtractor 2.25.0 centres VIGNET on +``(int)(mx + 0.49999)`` of its 0-based double-precision barycentre +(``src/analyse.c``), which the catalogue's float32 X_IMAGE / Y_IMAGE cannot +resolve near half pixels, so the runner asks SExtractor for X_IMAGE_DBL / +Y_IMAGE_DBL too (``seg_vignet_param_file``), and ``add_seg_vignet`` centres +on them and then drops them. + +The expected centres below are written out by hand from that rule, and the +expected stamps are cut by plain slicing, not by the module's helpers. +""" + +import numpy as np +import numpy.testing as npt +import pytest +from astropy.io import fits + +from shapepipe.modules.sextractor_package import sextractor_script as ss + +NX, NY = 60, 50 +STAMP = 7 +BIG = np.float32(-1e30) + +# (NUMBER, X_IMAGE_DBL, Y_IMAGE_DBL, VIGNET centre col, row (0-based)). +# Object 3's x is an exact half with an odd floor (np.rint goes to 32 - 1), +# object 4's y has a fractional part in [0.5, 0.50001) (np.rint goes up), and +# object 6's x is just past 0.50001, where float32 falls back below it. +OBJECTS = [ + (1, 20.2, 15.7, 19, 15), + (2, 1.3, 48.9, 0, 48), + (3, 32.5, 30.1, 31, 29), + (4, 44.8, 22.5000099, 44, 21), + (5, 59.9, 1.2, 59, 0), + (6, 31.5000101, 10.2, 31, 9), +] +DBL = ["X_IMAGE_DBL", "Y_IMAGE_DBL"] + + +@pytest.mark.parametrize("pos, centre", [ + (32.5, 31), # exact half, odd floor: down, not to even + (31.5, 30), # exact half, even floor + (31.49999, 30), + (31.5000099, 30), # fraction 0.5000099 < 0.50001: down + (31.5000101, 31), # fraction 0.5000101 > 0.50001: up + (31.4999, 30), + (20.2, 19), + (20.8, 20), + (0.6, 0), # mx = -0.4 truncates to 0 +]) +def test_vignet_centre_is_sextractors_rule(pos, centre): + assert ss.vignet_centre([pos]).tolist() == [centre] + + +def _cut(array, col, row, fill): + """The STAMP stamp of ``array`` with 0-based (row, col) at its centre.""" + half = STAMP // 2 + padded = np.pad(array, half, constant_values=fill) + return padded[row:row + STAMP, col:col + STAMP] + + +def _scene(tmp_path, flat=False): + """A sexcat whose VIGNETs are cut on the hand-written centres. + + ``flat`` makes the image uniform, so no pixel comparison can tell a tie + object's two candidate centres apart. + """ + rng = np.random.default_rng(3) + image = (np.full((NY, NX), 100, np.float32) if flat + else rng.normal(100, 5, (NY, NX)).astype(np.float32)) + seg = np.zeros((NY, NX), np.int32) + for number, _, _, col, row in OBJECTS: + seg[max(row - 2, 0):row + 3, max(col - 2, 0):col + 3] = number + number = np.array([o[0] for o in OBJECTS]) + x = np.array([o[1] for o in OBJECTS]) + y = np.array([o[2] for o in OBJECTS]) + + vignets = np.array([_cut(image, o[3], o[4], BIG) for o in OBJECTS]) + seg_true = np.array([_cut(seg, o[3], o[4], 0) for o in OBJECTS]) + vignets[(seg_true != 0) & (seg_true != number[:, None, None])] = BIG + + paths = {name: str(tmp_path / f"{name}-001-001.fits") + for name in ("segmentation", "sexcat")} + fits.PrimaryHDU(seg).writeto(paths["segmentation"]) + objects = fits.BinTableHDU.from_columns([ + fits.Column(name="NUMBER", format="J", array=number), + fits.Column(name="X_IMAGE", format="E", array=x.astype(np.float32)), + fits.Column(name="Y_IMAGE", format="E", array=y.astype(np.float32)), + fits.Column(name="VIGNET", format=f"{STAMP * STAMP}E", + array=vignets.reshape(len(number), -1), + dim=f"({STAMP},{STAMP})"), + fits.Column(name="THETA_J2000", format="E", + array=np.zeros(len(number))), + fits.Column(name="X_IMAGE_DBL", format="D", array=x), + fits.Column(name="Y_IMAGE_DBL", format="D", array=y), + ], name="LDAC_OBJECTS") + imhead = fits.BinTableHDU.from_columns( + [fits.Column(name="Field Header Card", format="80A", + array=np.array(["HISTORY x"]))], name="LDAC_IMHEAD") + fits.HDUList([fits.PrimaryHDU(), imhead, objects]).writeto( + paths["sexcat"]) + return paths, seg_true + + +def _add(paths): + ss.add_seg_vignet(paths["sexcat"], paths["segmentation"]) + with fits.open(paths["sexcat"]) as hdul: + return [h.name for h in hdul], hdul["LDAC_OBJECTS"].data.copy() + + +def test_neither_rint_nor_float32_positions_give_the_centres(): + """The premise: np.rint of the double positions misses objects 3 and 4, + and SExtractor's rule on the float32 positions misses object 6.""" + x = np.array([o[1] for o in OBJECTS]) + y = np.array([o[2] for o in OBJECTS]) + col = np.array([o[3] for o in OBJECTS]) + row = np.array([o[4] for o in OBJECTS]) + rint_miss = (np.rint(x - 1) != col) | (np.rint(y - 1) != row) + assert np.flatnonzero(rint_miss).tolist() == [2, 3] + x32 = x.astype(np.float32).astype(np.float64) + y32 = y.astype(np.float32).astype(np.float64) + f32_miss = ((ss.vignet_centre(x32) != col) + | (ss.vignet_centre(y32) != row)) + assert np.flatnonzero(f32_miss).tolist() == [5] + + +@pytest.mark.parametrize("flat", [False, True], ids=["noisy", "flat"]) +def test_seg_vignet_is_registered_with_vignet(tmp_path, flat): + """SEG_VIGNET is the check image on VIGNET's grid for every object, the + half-pixel ties included, even where the image cannot tell the two + candidate centres apart: int32, VIGNET's shape, 0 off the image.""" + paths, seg_true = _scene(tmp_path, flat=flat) + names, data = _add(paths) + assert names == ["PRIMARY", "LDAC_IMHEAD", "LDAC_OBJECTS"] + seg_vignets = data["SEG_VIGNET"] + assert seg_vignets.dtype.kind == "i" and seg_vignets.dtype.itemsize == 4 + assert seg_vignets.shape == data["VIGNET"].shape + npt.assert_array_equal(seg_vignets, seg_true) + centre = STAMP // 2 + npt.assert_array_equal(seg_vignets[:, centre, centre], data["NUMBER"]) + + +def test_seg_vignet_replaces_the_double_positions(tmp_path): + """The written catalogue has the columns SExtractor writes without the + double positions, plus SEG_VIGNET, and every other HDU and column as + they were.""" + paths, _ = _scene(tmp_path) + with fits.open(paths["sexcat"]) as hdul: + before = hdul["LDAC_OBJECTS"].data.copy() + imhead = hdul["LDAC_IMHEAD"].data.copy() + _, after = _add(paths) + kept = [name for name in before.names if name not in DBL] + assert after.names == kept + ["SEG_VIGNET"] + for name in kept: + npt.assert_array_equal(after[name], before[name]) + with fits.open(paths["sexcat"]) as hdul: + npt.assert_array_equal(hdul["LDAC_IMHEAD"].data, imhead) + + +def test_seg_vignet_needs_the_double_positions(tmp_path): + paths, _ = _scene(tmp_path) + with fits.open(paths["sexcat"]) as hdul: + objects = hdul["LDAC_OBJECTS"] + cols = [c for c in objects.columns if c.name not in DBL] + hdus = [hdul[0].copy(), hdul[1].copy(), + fits.BinTableHDU.from_columns(cols, name="LDAC_OBJECTS")] + fits.HDUList(hdus).writeto(paths["sexcat"], overwrite=True) + with pytest.raises(ValueError, match="X_IMAGE_DBL"): + _add(paths) + + +def test_param_file_adds_the_double_positions(tmp_path): + """The runner's parameter file is the configured one plus the two + double-precision positions, appended after its last parameter.""" + dot_param = tmp_path / "default.param" + dot_param.write_text("NUMBER # running number\nX_IMAGE\nVIGNET(51,51)") + out = ss.seg_vignet_param_file(str(dot_param), str(tmp_path / "out.param")) + lines = [line.split("#")[0].strip() + for line in open(out).read().splitlines()] + assert [line for line in lines if line] == [ + "NUMBER", "X_IMAGE", "VIGNET(51,51)"] + DBL + assert dot_param.read_text().count("DBL") == 0 + + +def test_sextractor_caller_names_its_check_images(tmp_path): + """The runner finds the SEGMENTATION check image by type.""" + caller = ss.SExtractorCaller( + [str(tmp_path / "image-001-001.fits"), + str(tmp_path / "weight-001-001.fits")], + str(tmp_path), "-001-001", "d.sex", "d.param", "d.conv", + True, False, False, False, False, False, False, + check_image=["BACKGROUND", "SEGMENTATION"], + ) + assert caller.check_paths == { + "BACKGROUND": f"{tmp_path}/background-001-001.fits", + "SEGMENTATION": f"{tmp_path}/segmentation-001-001.fits", + } + + +# --- SEG_VIGNET through the join to the UNIONS catalogue ------------------- +# +# UberSeg (uberseg_mask, seg_has_neighbour, Ngmix._check_central_seg_label) +# compares each stamp's labels with the row's NUMBER. The check image is +# labelled with SExtractor's NUMBERs; the join replaces NUMBER with the +# catalogue's, and relabels every stamp through the whole SExtractor -> UNIONS +# map, the footprints of rows it drops included (UNMATCHED_LABEL). The scene +# makes old and new numbers collide: each paired row's new NUMBER is another +# object's SExtractor NUMBER, one of them the dropped detection's. + +CROWD_STAMP = 9 +# (SExtractor NUMBER, x, y, UNIONS NUMBER or None for no partner) +CROWD = [ + (1, 10.0, 10.0, 2), + (2, 13.0, 10.0, None), + (3, 10.0, 13.0, 1), + (4, 22.0, 22.0, 3), +] +EXT_ONLY = (7, 5.0, 25.0) +EXT_HEADER = ( + "# 1 NUMBER Running object number\n" + "# 2 X_IMAGE Object position along x [pixel]\n" + "# 3 Y_IMAGE Object position along y [pixel]\n" +) + + +def _crowd_seg(): + seg = np.zeros((30, 30), np.int32) + for number, x, y, _ in CROWD: + col, row = int(x) - 1, int(y) - 1 + seg[row - 1:row + 2, col - 1:col + 2] = number + return seg + + +def _write_crowd(sexcat_path, seg_path): + """SExtractor's outputs for CROWD: the sexcat (with the double positions + seg_vignet_param_file asks for) and the SEGMENTATION check image.""" + number = np.array([o[0] for o in CROWD], np.int32) + x = np.array([o[1] for o in CROWD]) + y = np.array([o[2] for o in CROWD]) + n = len(number) + size = CROWD_STAMP * CROWD_STAMP + objects = fits.BinTableHDU.from_columns([ + fits.Column(name="NUMBER", format="J", array=number), + fits.Column(name="X_IMAGE", format="E", array=x.astype(np.float32)), + fits.Column(name="Y_IMAGE", format="E", array=y.astype(np.float32)), + fits.Column(name="VIGNET", format=f"{size}E", + array=np.ones((n, size), np.float32), + dim=f"({CROWD_STAMP},{CROWD_STAMP})"), + fits.Column(name="X_IMAGE_DBL", format="D", array=x), + fits.Column(name="Y_IMAGE_DBL", format="D", array=y), + ], name="LDAC_OBJECTS") + imhead = fits.BinTableHDU.from_columns( + [fits.Column(name="Field Header Card", format="80A", + array=np.array(["HISTORY x"]))], name="LDAC_IMHEAD") + fits.HDUList([fits.PrimaryHDU(), imhead, objects]).writeto(sexcat_path) + fits.PrimaryHDU(_crowd_seg()).writeto(seg_path) + + +def _write_crowd_external(path): + rows = [(new, x, y) for _, x, y, new in CROWD if new is not None] + rows.append(EXT_ONLY) + path.write_text(EXT_HEADER + "".join( + f"{n:10d} {x:11.4f} {y:11.4f}\n" for n, x, y in rows)) + return str(path) + + +def _old_stamps(): + """The check image's stamps in SExtractor's numbering, row by row.""" + col = np.array([int(o[1]) - 1 for o in CROWD]) + row = np.array([int(o[2]) - 1 for o in CROWD]) + return ss.cut_stamps(_crowd_seg(), col, row, CROWD_STAMP, 0) + + +def _run_runner(tmp_path, monkeypatch, match): + """sextractor_runner as the workflow runs it under blend_handling: + uberseg, with SExtractor replaced by the CROWD outputs.""" + import logging + + from shapepipe.modules import sextractor_runner as runner_module + from shapepipe.pipeline.config import CustomParser + + out = tmp_path / "output" + tmp = tmp_path / "tmp" + out.mkdir() + tmp.mkdir() + num = "-001-001" + + def fake_execute(command_line): + _write_crowd(out / f"sexcat{num}.fits", out / f"segmentation{num}.fits") + return "", "All done" + + monkeypatch.setattr(runner_module, "execute", fake_execute) + dot_param = tmp_path / "default.param" + dot_param.write_text("NUMBER\nX_IMAGE\nY_IMAGE\nVIGNET(9,9)\n") + ext = _write_crowd_external(tmp_path / "CFIS_cat-001-001.cat") + config = CustomParser() + config["SEXTRACTOR_RUNNER"] = { + "EXEC_PATH": "source-extractor", + "DOT_SEX_FILE": "d.sex", "DOT_PARAM_FILE": str(dot_param), + "DOT_CONV_FILE": "d.conv", "WEIGHT_IMAGE": "True", + "FLAG_IMAGE": "False", "PSF_FILE": "False", + "DETECTION_IMAGE": "False", "DETECTION_WEIGHT": "False", + "ZP_FROM_HEADER": "False", "BKG_FROM_HEADER": "False", + "CHECKIMAGE": "BACKGROUND, SEGMENTATION", + "SEG_VIGNET": "${SP_SEG_VIGNET:-False}", + "MATCH_CATALOGUE": "${SP_MATCH_CATALOGUE:-}", + "MATCH_RADIUS": "1.0", "MATCH_MIN_FRACTION": "0.98", + "MATCH_TOLERATED_UNPAIRED": "20", "MAKE_POST_PROCESS": "False", + } + monkeypatch.setenv("SP_SEG_VIGNET", "True") + monkeypatch.setenv("SP_MATCH_CATALOGUE", ext if match else "") + runner_module.sextractor_runner( + [str(tmp_path / f"image{num}.fits"), + str(tmp_path / f"weight{num}.fits")], + {"output": str(out), "tmp": str(tmp)}, num, config, + "SEXTRACTOR_RUNNER", logging.getLogger("test"), + ) + with fits.open(out / f"sexcat{num}.fits") as hdul: + return hdul["LDAC_OBJECTS"].data.copy() + + +def test_runner_seg_vignet_carries_the_final_number(tmp_path, monkeypatch): + """Through the runner, each row's own footprint carries its final + (UNIONS) NUMBER and nothing else does, though every new NUMBER is some + other object's SExtractor NUMBER; the dropped detection's footprint is + UNMATCHED_LABEL. Fails if the join runs before add_seg_vignet, or relabels + only the paired labels.""" + from shapepipe.modules.sextractor_package import match_catalogue as mc + + data = _run_runner(tmp_path, monkeypatch, match=True) + paired = [i for i, o in enumerate(CROWD) if o[3] is not None] + old = _old_stamps()[paired] + old_number = np.array([CROWD[i][0] for i in paired]) + new_number = np.array([CROWD[i][3] for i in paired]) + npt.assert_array_equal(data["NUMBER"], new_number) + seg = data["SEG_VIGNET"] + assert "X_IMAGE_DBL" not in data.names + centre = CROWD_STAMP // 2 + npt.assert_array_equal(seg[:, centre, centre], data["NUMBER"]) + dropped = CROWD[1][0] + for i in range(len(paired)): + npt.assert_array_equal(seg[i] == new_number[i], + old[i] == old_number[i]) + npt.assert_array_equal(seg[i] == mc.UNMATCHED_LABEL, + old[i] == dropped) + npt.assert_array_equal(seg[i] == 0, old[i] == 0) + # Rows 0 and 1 overlap the dropped detection, whose SExtractor label (2) + # is row 0's new NUMBER. + assert (seg[0] == mc.UNMATCHED_LABEL).any() + assert (old[0] == new_number[0]).any() + + +def test_runner_seg_vignet_without_a_join_keeps_sextractor_numbers( + tmp_path, monkeypatch, +): + """Image simulations (empty MATCH_CATALOGUE): every row stays, and the + stamps are the check image's, labelled with SExtractor's NUMBER.""" + data = _run_runner(tmp_path, monkeypatch, match=False) + npt.assert_array_equal(data["NUMBER"], [o[0] for o in CROWD]) + npt.assert_array_equal(data["SEG_VIGNET"], _old_stamps()) + + +def test_uberseg_sees_the_same_neighbours_after_the_join(tmp_path, + monkeypatch): + """UberSeg only asks own-versus-other, so the relabelled stamps give the + neighbour mask and flag the SExtractor-labelled ones give.""" + from shapepipe.modules.ngmix_package.ngmix import ( + seg_has_neighbour, + uberseg_mask, + ) + + data = _run_runner(tmp_path, monkeypatch, match=True) + paired = [i for i, o in enumerate(CROWD) if o[3] is not None] + old = _old_stamps()[paired] + for i, j in enumerate(paired): + new_seg, number = data["SEG_VIGNET"][i], data["NUMBER"][i] + npt.assert_array_equal( + uberseg_mask(new_seg, number), + uberseg_mask(old[i], CROWD[j][0])) + assert (seg_has_neighbour(new_seg, number) + == seg_has_neighbour(old[i], CROWD[j][0])) + assert seg_has_neighbour(data["SEG_VIGNET"][0], data["NUMBER"][0]) + assert not seg_has_neighbour(data["SEG_VIGNET"][2], data["NUMBER"][2]) + + +def test_a_partial_relabel_would_collide(): + """The negative control: relabelling only the paired labels leaves the + dropped detection's SExtractor label 2 in row 0's stamp, equal to row 0's + new NUMBER, so UberSeg would take the neighbour for the object.""" + old = _old_stamps()[0] + partial = old.copy() + for number, _, _, new in CROWD: + if new is not None: + partial[old == number] = new + assert ((partial == 2) & (old != 1)).any() + + +def _join_with_numbers(case, numbers): + """Join CROWD to a catalogue whose three objects carry ``numbers`` + (written as given), and return the joined NUMBER column.""" + from shapepipe.modules.sextractor_package import match_catalogue as mc + + case.mkdir() + _write_crowd(case / "sexcat.fits", case / "seg.fits") + rows = zip(numbers, [10.0, 10.0, 22.0], [10.0, 13.0, 22.0]) + ext = case / "ext.cat" + ext.write_text(EXT_HEADER + "".join( + f"{n:>10} {x:11.4f} {y:11.4f}\n" for n, x, y in rows)) + mc.match_catalogue(str(case / "sexcat.fits"), str(ext)) + with fits.open(case / "sexcat.fits") as hdul: + return hdul["LDAC_OBJECTS"].data["NUMBER"].tolist() + + +@pytest.mark.parametrize("numbers", [ + [5, 5, 6], # repeated + [0, 5, 6], # not positive + [-3, 5, 6], + ["5.5", 7, 6], # not an integer + ["5.0", 7, 6], # read as a float + [2**31, 5, 6], # past int32 + [2**32 - 1, 5, 6], # would narrow to -1, UNMATCHED_LABEL +]) +def test_join_rejects_numbers_that_could_collide(tmp_path, numbers): + """Unique catalogue NUMBERs that the int32 columns hold exactly are what + keep a relabelled stamp's own footprint apart from every other one.""" + with pytest.raises(ValueError, match="NUMBER"): + _join_with_numbers(tmp_path / "case", numbers) + + +def test_join_accepts_the_largest_int32_number(tmp_path): + from shapepipe.modules.sextractor_package import match_catalogue as mc + + assert _join_with_numbers(tmp_path / "case", [1, mc.MAX_NUMBER, 6]) == [ + 1, mc.MAX_NUMBER, 6] + + +def test_seg_vignet_stays_out_of_the_final_catalogue(tmp_path): + """make_cat drops SEG_VIGNET with VIGNET: the stamps are ngmix inputs, + not catalogue columns.""" + from shapepipe.modules.make_cat_package import make_cat + + _write_crowd(tmp_path / "sexcat-001-001.fits", tmp_path / "seg.fits") + ss.add_seg_vignet(str(tmp_path / "sexcat-001-001.fits"), + str(tmp_path / "seg.fits")) + final = make_cat.prepare_final_cat_file(str(tmp_path), "-001-001") + make_cat.save_sextractor_data(final, str(tmp_path / "sexcat-001-001.fits")) + with fits.open(tmp_path / "final_cat-001-001.fits") as hdul: + names = hdul["RESULTS"].data.names + assert "VIGNET" not in names and "SEG_VIGNET" not in names + assert "NUMBER" in names diff --git a/tests/science/test_defect_recovery.py b/tests/science/test_defect_recovery.py new file mode 100644 index 000000000..eb582d26d --- /dev/null +++ b/tests/science/test_defect_recovery.py @@ -0,0 +1,134 @@ +"""Shear recovery for the defects the central veto keeps. + +Physics invariant: every defect the central veto keeps leaves both additive terms +|c1|, |c2| < 5e-4 and both diagonal multiplicative terms |m11|, |m22| < 1%, +from the full 2x2 response matrix. A filled defect biases m anisotropically +(m11 and m22 can differ tenfold), so a scalar m would hide it. +Interpolated defects (columns, full and finite 3-px bleeds, single pixels) +are checked at ``EPOCH_INTERPOLATED_DEFECT_RADIUS`` and two pixels beyond +it, on galaxies with half-light radius 0.3" and 0.5" through a 0.7" PSF, +round and with ellipticity (0.05, 0.02), and on a 0.7" galaxy through a +0.9" PSF. These pass only because the interpolated pixels' quarter turns +also lose their weight: an unsymmetrized column at 8 px gives +c1 = -1.3e-3. Defects too wide to interpolate (a 4-column cluster, a 5-px +bleed, edge bands) are noise-filled, and are checked at +``EPOCH_CENTRAL_DEFECT_RADIUS`` on the same three galaxies; the 0.7" +galaxy through the 0.9" PSF sets that radius (a 4-column cluster at 10 px +gives m11 = -6.6%). The cases sit at the radii themselves, so lowering +either below its calibrated value turns this red. + +The bound is checked against the veto alone, not the masked-fraction cut: +the edge bands at the noise-fill radius cover about 25% of the stamp, which +``EPOCH_MASKED_FRACTION_CUT`` (10%) drops in production. That they stay +within the bound is the evidence that the fraction cut is not a bias +control, so it can be chosen for DES comparability and robustness alone. + +Positive control: a 3-px bleed three pixels inside the interpolated-defect +radius breaks the bound. +""" + +import json + +import numpy as np +import pytest + +from shapepipe.modules.ngmix_package.defect_interpolation import ( + interpolable_defects, +) +from shapepipe.modules.ngmix_package.ngmix import ( + EPOCH_CENTRAL_DEFECT_RADIUS, + EPOCH_INTERPOLATED_DEFECT_RADIUS, + central_defect_vetoes, + defect_mask, +) +from tests.helpers.defect_response import defect_response + +N = 51 +CENTRE = N // 2 +RI = int(np.ceil(EPOCH_INTERPOLATED_DEFECT_RADIUS)) +RN = int(np.ceil(EPOCH_CENTRAL_DEFECT_RADIUS)) +ROUND = (0.0, 0.0) +ELLIPTICAL = (0.05, 0.02) +SEEDS = range(6) + + +def geometry(kind, distance): + """A detector defect whose nearest pixel is ``distance`` px from the + stamp centre.""" + bad = np.zeros((N, N), dtype=bool) + near = CENTRE + distance + if kind == "pixel": + bad[CENTRE, near] = True + elif kind == "column": + bad[:, near] = True + elif kind == "bleed": + bad[:, near:near + 3] = True + elif kind == "finite_bleed": + bad[CENTRE - 5:CENTRE + 6, near:near + 3] = True + elif kind == "cluster4": + bad[:, near:near + 4] = True + elif kind == "wide_bleed": + bad[:, near:near + 5] = True + elif kind == "edge": + bad[:, near:] = True + return bad + + +INTERPOLATED = ("column", "bleed", "finite_bleed", "pixel") +CASES = ( + [(k, d, h, 0.7, ROUND) for k in INTERPOLATED + for d in (RI, RI + 2) for h in (0.3, 0.5)] + + [(k, RI, 0.5, 0.7, ELLIPTICAL) for k in INTERPOLATED] + + [(k, RI, 0.7, 0.9, psf) for k in ("bleed", "finite_bleed") + for psf in (ROUND, ELLIPTICAL)] + + [(k, RN, h, 0.7, ROUND) for k in ("wide_bleed", "edge") + for h in (0.3, 0.5)] + + [("edge", RN, 0.5, 0.7, ELLIPTICAL)] + + [(k, RN, 0.7, 0.9, psf) for k in ("cluster4", "edge") + for psf in (ROUND, ELLIPTICAL)] + + [("edge", N - CENTRE - 5, 0.5, 0.7, psf) for psf in (ROUND, ELLIPTICAL)] +) + + +def case_id(case): + kind, distance, hlr, fwhm, psf_shear = case + psf = "elliptical" if any(psf_shear) else "round" + return f"{kind}-{distance}px-hlr{hlr}-psf{fwhm}-{psf}" + + +def recover(bad, hlr, fwhm, psf_shear, tmp_path): + result = defect_response(bad, hlr=hlr, psf=fwhm, seeds=SEEDS, + psf_shear=psf_shear) + (tmp_path / "recovery.json").write_text(json.dumps(result, indent=2)) + return np.abs(result["m"]).max(), np.abs(result["c"]).max(), result + + +@pytest.mark.parametrize("kind,distance,hlr,fwhm,psf_shear", CASES, + ids=[case_id(c) for c in CASES]) +def test_kept_defects_recover_shear_on_both_axes(kind, distance, hlr, fwhm, + psf_shear, tmp_path): + """Failure modes: a veto radius is below its calibrated value; the + interpolated pixels' quarter-turn orbit keeps its weight (a one-sided + hole in the likelihood); the fill mask is symmetrized; a wide hole is + interpolated; raw defect values leak into metacal; the check reads m11 + alone and misses m22, or c1 alone.""" + bad = geometry(kind, distance) + masked = defect_mask(np.ones((N, N)), bad.astype(np.int32)) + np.testing.assert_array_equal(masked, bad) + interpolated = interpolable_defects(masked) + assert interpolated.any() == (kind in INTERPOLATED) + assert not central_defect_vetoes(masked, interpolated) + m, c, result = recover(bad, hlr, fwhm, psf_shear, tmp_path) + assert m < 0.01, result + assert c < 5e-4, result + + +def test_vetoed_bleed_breaks_the_bound(tmp_path): + """Positive control: a 3-px bleed three pixels inside the + interpolated-defect radius, on the 0.3" galaxy, gives |c| > 1e-3 and + |m| > 1.5%. The veto drops it.""" + bad = geometry("bleed", RI - 3) + assert central_defect_vetoes(bad, interpolable_defects(bad)) + m, c, result = recover(bad, 0.3, 0.7, ROUND, tmp_path) + assert c > 1e-3, result + assert m > 0.015, result diff --git a/tests/unit/test_workflow_defect_weighting.py b/tests/unit/test_workflow_defect_weighting.py new file mode 100644 index 000000000..c7312f4de --- /dev/null +++ b/tests/unit/test_workflow_defect_weighting.py @@ -0,0 +1,71 @@ +"""The run config's ``defect_weighting`` reaches ngmix's DEFECT_WEIGHTING. + +The workflow reads ``defect_weighting`` (Snakefile), tile_ngmix exports it as +``SP_DEFECT_WEIGHTING``, and config_tile_Ng_template.ini expands it into +``[NGMIX_RUNNER] DEFECT_WEIGHTING``; empty takes the module default. These +files never see each other at run time, so their agreement is asserted here, +statically, plus the expansion through ShapePipe's own config parser. +""" + +import ast +import re +from pathlib import Path + +import pytest + +from shapepipe.modules.ngmix_package.ngmix import ( + DEFECT_WEIGHTING, + DEFECT_WEIGHTINGS, +) +from shapepipe.pipeline.config import CustomParser + +REPO_ROOT = Path(__file__).resolve().parents[2] +WORKFLOW = REPO_ROOT / "workflow" +TEMPLATE = WORKFLOW / "config" / "cfis" / "config_tile_Ng_template.ini" + + +def test_the_default_is_an_option(): + assert DEFECT_WEIGHTING in DEFECT_WEIGHTINGS + assert set(DEFECT_WEIGHTINGS) == {"des_y6", "fourfold_zero", "hole", + "full"} + + +def test_the_snakefile_reads_the_options_from_the_module_it_runs(): + """The Snakefile validates ``defect_weighting`` against the tuple it + parses out of ngmix.py; parse it the same way here.""" + snakefile = (WORKFLOW / "Snakefile").read_text() + assert 'config.get("defect_weighting")' in snakefile + assert "DEFECT_WEIGHTINGS" in snakefile + tree = ast.parse( + (REPO_ROOT / "src" / "shapepipe" / "modules" / "ngmix_package" + / "ngmix.py").read_text() + ) + parsed = next( + ast.literal_eval(node.value) for node in tree.body + if isinstance(node, ast.Assign) + and [getattr(t, "id", None) for t in node.targets] + == ["DEFECT_WEIGHTINGS"] + ) + assert parsed == DEFECT_WEIGHTINGS + + +def test_tile_ngmix_exports_it_and_tile_vignets_carries_it(): + """tile_ngmix exports SP_DEFECT_WEIGHTING; tile_vignets carries the + value as a param, so a change reruns the whole group, never the chunks + alone.""" + rules = (WORKFLOW / "rules" / "tile.smk").read_text() + ngmix = rules[rules.index("rule tile_ngmix:"):] + vignets = rules[rules.index("rule tile_vignets:"): + rules.index("rule tile_ngmix:")] + assert re.search(r'"SP_DEFECT_WEIGHTING":\s*DEFECT_WEIGHTING', ngmix) + assert re.search(r"defect_weighting\s*=\s*DEFECT_WEIGHTING", vignets) + + +@pytest.mark.parametrize("exported", ["", "des_y6"]) +def test_the_template_expands_the_export(monkeypatch, exported): + parser = CustomParser() + parser.read(TEMPLATE) + monkeypatch.setenv("SP_DEFECT_WEIGHTING", exported) + assert parser.getexpanded("NGMIX_RUNNER", "DEFECT_WEIGHTING") == exported + monkeypatch.delenv("SP_DEFECT_WEIGHTING") + assert parser.getexpanded("NGMIX_RUNNER", "DEFECT_WEIGHTING") == "" diff --git a/tests/unit/test_workflow_tile_detection.py b/tests/unit/test_workflow_tile_detection.py index 498a015de..ae75aac5c 100644 --- a/tests/unit/test_workflow_tile_detection.py +++ b/tests/unit/test_workflow_tile_detection.py @@ -153,3 +153,28 @@ def test_catalogue_retrieve_follows_the_prefix(source, retrieve): config = {"tile_detection": "unions_catalogue", "inputs": {"catalogues": source}} assert run_config.catalogue_source(config) == (source, retrieve) + + +# --- blend handling -------------------------------------------------------- + + +def test_blend_handlings_mirror_the_ngmix_module(): + """run_config.BLEND_HANDLINGS is a copy; the copy must stay true. + + The Snakefile validates `blend_handling:` against it in the launcher venv, + outside the container where shapepipe is importable, so the tuple is + mirrored rather than imported, and read out of the source text here for + the same reason: this file is container-free. + """ + import re + + src = (REPO_ROOT / "src" / "shapepipe" / "modules" / "ngmix_package" + / "ngmix.py").read_text() + match = re.search(r"^BLEND_HANDLINGS = \(([^)]*)\)", src, re.M) + assert match, "ngmix.py no longer defines BLEND_HANDLINGS" + assert _load("run_config").BLEND_HANDLINGS == tuple( + part.strip().strip('"\'') for part in match.group(1).split(",") + if part.strip()) + committed = _load("run_config").load( + str(REPO_ROOT / "workflow" / "config.yaml")) + assert committed["blend_handling"] == "uberseg" diff --git a/tests/workflow/README.md b/tests/workflow/README.md index b26309227..3ff6139fe 100644 --- a/tests/workflow/README.md +++ b/tests/workflow/README.md @@ -44,7 +44,7 @@ The API context remains open while tests inspect jobs and closes before the fixt ## Campaign-boundary pin -`params_pin.json` pins SHA-256 digests of: +`params_pin.json` pins the default campaign (data, psfex, `blend_handling: uberseg`), and `params_pin_noisefill.json` the same campaign with `blend_handling: noisefill`, so a campaign that ran as noisefill stays resumable. Each pins SHA-256 digests of: - `unit_pre()` rendered for every stage; - every rule's shell template; @@ -77,7 +77,10 @@ Apply mutations only to a disposable checkout, run the named test without `--upd | `test_final_cat_merge_reads_every_ready_tile` | Drop one ready tile; append an out-of-scope tile. | | `test_products_use_products_dir_and_run_name` | Rename either merged catalogue or the persist manifest; route products to scratch; derive `CAMPAIGN` from the products directory's basename. | | `test_tile_store_is_unique_per_campaign` | Make `LOCAL_TAG` conditional on `image_sims` (or constant); give one tile_shape member a different store path. | +| `test_blend_handling_reaches_ngmix_only_under_uberseg` | Make the Snakefile or config.yaml default `noisefill`; export `SP_BLEND_HANDLING` under noisefill, or `SP_SEG_VIGNET` only under uberseg; drop `blend_env(...)` from either tile_detect or tile_ngmix; export the wrong value; set `NGMIX_SEG_MEM_MB` or `DETECT_SEG_MEM_MB` to 0; give a committed ini's `SEG_VIGNET` / `BLEND_HANDLING` a literal or the wrong default. | +| `test_unknown_blend_handling_fails_during_parse` | Remove the Snakefile's `blend_handling` check. | | `test_missing_run_fails_during_parse` | Remove `run` from `run_config.REQUIRED`; literal paths must still receive the required-key diagnostic, not a later `KeyError`. | | `test_unit_pre_changes_at_campaign_boundary` | Append a line to `unit_pre`; change one rule's `params.pre`; change one shell; change a rendered thread count; make `LOCAL_TAG` empty for data. | +| `test_noisefill_plan_changes_at_campaign_boundary` | Export anything under noisefill; change one rule's `params.pre` or shell. | | `test_params_pin_ignores_fixture_root` | Remove fixture-root or run-dir-hash normalization. | | `test_params_is_a_rerun_trigger_under_both_profiles` | Remove `params` from candide or nibi's `rerun-triggers`. | diff --git a/tests/workflow/conftest.py b/tests/workflow/conftest.py index 134a9c62a..f08beaaa8 100644 --- a/tests/workflow/conftest.py +++ b/tests/workflow/conftest.py @@ -22,7 +22,7 @@ def campaign(request, tmp_path): @pytest.fixture def resolve_dag(monkeypatch): """Expose the resolver so tests can also assert parse-time failures.""" - return lambda campaign: resolve(campaign, monkeypatch) + return lambda campaign, **kwargs: resolve(campaign, monkeypatch, **kwargs) @pytest.fixture diff --git a/tests/workflow/harness.py b/tests/workflow/harness.py index 1c8420f26..abe0783ad 100644 --- a/tests/workflow/harness.py +++ b/tests/workflow/harness.py @@ -206,8 +206,12 @@ def load_profile(name): @contextmanager -def resolve(campaign, monkeypatch): - """Resolve ``all`` in an isolated state directory, without running jobs.""" +def resolve(campaign, monkeypatch, launch_env=None): + """Resolve ``all`` in an isolated state directory, without running jobs. + + ``launch_env`` adds variables to the scrubbed environment, as if the + launching shell had exported them. + """ from snakemake import workflow as sm_workflow scripts = REPO / "workflow" / "scripts" @@ -234,6 +238,8 @@ def resolve(campaign, monkeypatch): "XDG_CACHE_HOME": campaign.root / "cache", }.items(): patch.setenv(name, str(value)) + for name, value in (launch_env or {}).items(): + patch.setenv(name, str(value)) profile = load_profile("candide") try: with SnakemakeApi() as api: diff --git a/tests/workflow/params_pin.json b/tests/workflow/params_pin.json index 7b839ba7e..c739bb61f 100644 --- a/tests/workflow/params_pin.json +++ b/tests/workflow/params_pin.json @@ -11,7 +11,7 @@ "final_cat_merge": "e7f46859c4503a2220713d7bb2507555515d0a9632d780b20f14c59e32210023", "prepare_all_tiles": "b8f872a22adf014e25a7fa5198f49b71a6fe9e56042ed82b682bc8763970a844", "star_cat_merge": "6277450958474af5270982fa35360f2f237a29f7533c526ee9265dfd5acc07a0", - "tile_detect": "8336b148769e9d43b64f0d945c05e7d163c64dba4e6287e8c1964d1502e3ffe1", + "tile_detect": "24d015063fb07ba4a48b961284fa1345675f13e167acc62a2c7b397795113b82", "tile_exp_forest": "7447ab4a1049de5f0b5c81e5f9ed2a8644c7bdab85cb0a060bfde81261a89b28", "tile_find_exposures": "8704317871744996c44351c2836fcb222d7986a602e9e046a90d684ba7b3c184", "tile_get_catalogue": "7e3f889a955a14b0b917a015c2a8bc90e433a89b1cc5e00b8ca94695c4c93d3c", @@ -19,12 +19,12 @@ "tile_make_cat": "ea546ec59bcd13c0f8c9ee2c7fa5dde4773975277eed8f46ab63e6d124db0535", "tile_merge_cats": "ff21216ea804dccc2d2c290d2b2499d5d05f0c34c0a56993c233f43fe3c06bdb", "tile_merge_headers": "7a344849d62936e2f5598dc8731a2c4947eff2c4c7218b1a731dd2a9577e7111", - "tile_ngmix": "5192e65a72b3b6186b29d7bbecfe48751c7bc9ec68a61083ef4092db9f86f02f", + "tile_ngmix": "8ad2a80b8398d10e91daf939ebc3a5d24916a5360125d3031161b8d3cd3ab186", "tile_uncompress": "1e2b01acbf9708e0371070fb01c5b9568d7efb5f89d1835fc6bf91e2c8b60cb3", "tile_vignets": "ab52bf2c6ede04915c77f30a44be0cf707c4609ddf8ee768bfaf598cfff31481" }, "schema": 1, - "sha256": "1c792868397e4a2f1144d68d153c4a0a23e9ffd07ecdb95b98896d1b990c8ce1", + "sha256": "fcdbfc84d1178cfb5a13be005ce7de577e8a4b8774bb5980176bd194a68d2e42", "unit_pre": { "exp_get_images": "8dec850af212879f225fcf27a5f1281e1a075264c7b97d38c2214395d360168c", "exp_psf": "f2358ddf7385918dc5033d10b37f6dc97a15d02b071a3ea0a4619a5f7e6f5bec", diff --git a/tests/workflow/params_pin_noisefill.json b/tests/workflow/params_pin_noisefill.json new file mode 100644 index 000000000..5b755e8a8 --- /dev/null +++ b/tests/workflow/params_pin_noisefill.json @@ -0,0 +1,43 @@ +{ + "algorithm": "sha256", + "rules": { + "all": "572122d8d1901e12ff591b30adf405f8920be8183129befececbb819f3392ed8", + "clean_exposure": "22cb76b13a5205d20a02a9bd3b8c8bea5ea2801555e24dd2b84b11145f7e79d9", + "clean_tile": "a5c07b0461526ed407df36a291deb866d4181524b4c8e0fbc3dd047fd9d28479", + "exp_get_images": "71e76ff7f96af1c5d2b85697cc5819e2271911f177a2075253f3b8cfa268c1a9", + "exp_persist": "302e2837542bc1102430c27c81c600b7cda32e8bddcb5fd60d33950987609fff", + "exp_psf": "2c4f6d00f1939ccbf05b4982a0202a0ff92a4373a727f4e00f3aaab4eba03352", + "exp_split": "6e954f8f3d06f44d3f164675912ce27d6216648855d04968f9168bd7d0f2c4fa", + "final_cat_merge": "e7f46859c4503a2220713d7bb2507555515d0a9632d780b20f14c59e32210023", + "prepare_all_tiles": "b8f872a22adf014e25a7fa5198f49b71a6fe9e56042ed82b682bc8763970a844", + "star_cat_merge": "6277450958474af5270982fa35360f2f237a29f7533c526ee9265dfd5acc07a0", + "tile_detect": "24d015063fb07ba4a48b961284fa1345675f13e167acc62a2c7b397795113b82", + "tile_exp_forest": "7447ab4a1049de5f0b5c81e5f9ed2a8644c7bdab85cb0a060bfde81261a89b28", + "tile_find_exposures": "8704317871744996c44351c2836fcb222d7986a602e9e046a90d684ba7b3c184", + "tile_get_catalogue": "7e3f889a955a14b0b917a015c2a8bc90e433a89b1cc5e00b8ca94695c4c93d3c", + "tile_get_images": "331a67e747f211ebf4c14b946a7af7f9fc9f55243d69d2791d74aecc3ca228c3", + "tile_make_cat": "ea546ec59bcd13c0f8c9ee2c7fa5dde4773975277eed8f46ab63e6d124db0535", + "tile_merge_cats": "ff21216ea804dccc2d2c290d2b2499d5d05f0c34c0a56993c233f43fe3c06bdb", + "tile_merge_headers": "7a344849d62936e2f5598dc8731a2c4947eff2c4c7218b1a731dd2a9577e7111", + "tile_ngmix": "5bfc037fb09657cb83d98142c05bf8bd54a99f0bfa878bf6a632f8267defd1fe", + "tile_uncompress": "1e2b01acbf9708e0371070fb01c5b9568d7efb5f89d1835fc6bf91e2c8b60cb3", + "tile_vignets": "ab52bf2c6ede04915c77f30a44be0cf707c4609ddf8ee768bfaf598cfff31481" + }, + "schema": 1, + "sha256": "63d8f40e72c9294f610c4403a4e618ee6965c522801e17e8f135ed06176d9351", + "unit_pre": { + "exp_get_images": "8dec850af212879f225fcf27a5f1281e1a075264c7b97d38c2214395d360168c", + "exp_psf": "f2358ddf7385918dc5033d10b37f6dc97a15d02b071a3ea0a4619a5f7e6f5bec", + "exp_split": "358fa8bbe59680d4f9839e007b343dd25fed733cf3156dccd30d305d33ac9480", + "tile_detect": "adad5d671fa65dd04433e1c82a725b845635e9d2e368a1e70833ed990df2d55a", + "tile_find_exposures": "c5922fb507f6fd040a179b53fba0818697661016c9688984b6ce249186dc6986", + "tile_get_catalogue": "b124d12252617abb8df8a98d6234ddfd6cc503e19fa72c9a0c48c21559d6f48c", + "tile_get_images": "45b47c44bfeb34ac973b89d4e752c8c82e78028f9f0da379c44627005b45e279", + "tile_make_cat": "7579a52e32e76523c0b77d75c468a47d7f2e2ca0c55865f40e9628a7481cd79d", + "tile_merge_cats": "69cfe94ba941d2ba7b1ce24883961b47b191b80aff381da1865c6bc38de0c354", + "tile_merge_headers": "8590cfa3281c88d43eb8c4fe5760176a13e928a419cbea9bea3a8f88b0634b42", + "tile_ngmix": "b6b75b553a62df54aea6c337dcde1e0cdcd0fe30000e86880c1a4689e42c4a72", + "tile_uncompress": "91a15538491e53ee2b0d52b0472909374270918e4b4c80880c4f5a8329e40161", + "tile_vignets": "a6111cef708aa3c9fb0144b49de9f781eef84d2096ba4f8a3de9bb45b1a3774d" + } +} diff --git a/tests/workflow/test_dag.py b/tests/workflow/test_dag.py index b23493ecc..abfed9750 100644 --- a/tests/workflow/test_dag.py +++ b/tests/workflow/test_dag.py @@ -1,5 +1,7 @@ """Resolved-job checks for campaign scope, product paths, and PSF custody.""" +import hashlib +import os import re from collections import Counter from pathlib import Path @@ -162,3 +164,162 @@ def test_mccd_is_refused_during_parse(tmp_path, resolve_dag): with pytest.raises(WorkflowError, match=r"psf_model=mccd: PSF persistence"): with resolve_dag(campaign): pytest.fail("psf_model=mccd must be refused at parse time") + + +# --- blend_handling --------------------------------------------------------- + +# First-attempt memory of the stages that hold SEG_VIGNET, under every +# blend_handling: the base plus the seg stamps' share. +SEG_MEM_MB = {"tile_detect": 4000 + 3000, "tile_ngmix": 5000 + 500} +# The exports every blend_handling sets, and those uberseg adds. +SEG_EXPORTS = {"tile_detect": {"SP_SEG_VIGNET": "True"}} +BLEND_EXPORTS = {"tile_ngmix": {"SP_BLEND_HANDLING": "uberseg"}} +# The option each stage's committed ini reads the export through, and the +# value it must resolve to with and without it. +BLEND_OPTIONS = {"tile_detect": ("SEG_VIGNET", "False", "True"), + "tile_ngmix": ("BLEND_HANDLING", "noisefill", "uberseg")} + + +def _jobs(dag, campaign): + """Every job's prologue, shell and memory, keyed by rule and wildcards, + with the campaign root and the run-dir hash in the node-local store name + made location-free.""" + run_hash = hashlib.sha1(str(campaign.run_dir).encode()).hexdigest()[:8] + + def normalize(value): + if not isinstance(value, str): + return value + return (value.replace(run_hash, "") + .replace(str(campaign.root), "")) + + return { + (job.rule.name, tuple(sorted(job.wildcards_dict.items()))): tuple( + normalize(v) for v in (getattr(job.params, "pre", None), + job.shellcmd, job.resources.get("mem_mb"))) + for job in dag.jobs + } + + +def _surface(campaign, resolve_dag): + """The active rules and :func:`_jobs`.""" + with resolve_dag(campaign) as dag: + return dag.rule_names, _jobs(dag, campaign) + + +def _committed_value(shell, config_dir, option, env, monkeypatch): + """``option`` of the module section of the ini ``shell`` runs, expanded + under ``env`` as ShapePipe expands it.""" + from shapepipe.pipeline.config import CustomParser + + name = shell.split('shapepipe_run -c "$SP_CONFIG/')[1].split('"')[0] + parser = CustomParser() + parser.optionxform = str + assert parser.read(config_dir / name) + section = next(s for s in parser.sections() if s.endswith("_RUNNER")) + for key in ("SP_SEG_VIGNET", "SP_BLEND_HANDLING"): + monkeypatch.delenv(key, raising=False) + for key, value in env.items(): + monkeypatch.setenv(key, value) + return parser.getexpanded(section, option) + + +@pytest.mark.parametrize("detection", ["sextractor", "unions_catalogue"]) +def test_blend_handling_reaches_ngmix_only_under_uberseg( + tmp_path, resolve_dag, monkeypatch, detection): + """A campaign without the knob plans exactly the uberseg campaign. Both + handlings export SP_SEG_VIGNET to tile_detect, which turns its ini's + SEG_VIGNET on, and carry the seg stamps' memory in tile_detect and + tile_ngmix. Against explicit noisefill, uberseg only adds its export to + tile_ngmix's prologue, which turns BLEND_HANDLING to uberseg.""" + surfaces = {} + for blend in (None, "noisefill", "uberseg"): + campaign = Campaign(tmp_path / str(blend), "data", "psfex") + campaign.config["tile_detection"] = detection + if blend is not None: + campaign.config["blend_handling"] = blend + campaign.write_config() + surfaces[blend] = _surface(campaign, resolve_dag) + + assert surfaces["uberseg"] == surfaces[None] + rules, noisefill = surfaces["noisefill"] + uberseg_rules, uberseg = surfaces["uberseg"] + assert uberseg_rules == rules + assert uberseg.keys() == noisefill.keys() + config_dir = Path(__file__).parents[2] / "workflow" / "config" / "cfis" + for key, (pre, shell, mem) in uberseg.items(): + rule = key[0] + nf_pre, nf_shell, nf_mem = noisefill[key] + exports = BLEND_EXPORTS.get(rule, {}) + lines = {f"export {name}='{value}'" for name, value in exports.items()} + + def without_exports(text): + if text is None: + return None + return "\n".join(line for line in text.split("\n") + if line not in lines) + + # The rendered shell carries the prologue, so it moves with it and + # nowhere else. + assert without_exports(shell) == nf_shell, key + if pre is None: + assert nf_pre is None and not exports, key + continue + assert lines <= set(pre.split("\n")), key + assert without_exports(pre) == nf_pre, key + if exports: + option, default, value = BLEND_OPTIONS[rule] + assert _committed_value(shell, config_dir, option, {}, + monkeypatch) == default + assert _committed_value(shell, config_dir, option, exports, + monkeypatch) == value + assert mem == nf_mem, key + if rule in SEG_MEM_MB: + assert mem == SEG_MEM_MB[rule], key + seg_exports = SEG_EXPORTS.get(rule, {}) + if seg_exports: + assert {f"export {name}='{value}'" + for name, value in seg_exports.items()} <= set( + nf_pre.split("\n")), key + option, default, value = BLEND_OPTIONS[rule] + assert _committed_value(shell, config_dir, option, seg_exports, + monkeypatch) == value + + +def test_unknown_blend_handling_fails_during_parse(tmp_path, resolve_dag): + campaign = Campaign(tmp_path / "campaign", "data", "psfex") + campaign.config["blend_handling"] = "mof" + campaign.write_config() + with pytest.raises(WorkflowError, match=r"Invalid blend_handling='mof'"): + with resolve_dag(campaign): + pytest.fail("an unknown blend_handling must fail at parse time") + + +INHERITED_BLEND_ENV = { + f"{prefix}{name}": value + for prefix in ("", "APPTAINERENV_", "SINGULARITYENV_") + for name, value in (("SP_SEG_VIGNET", "True"), + ("SP_BLEND_HANDLING", "uberseg")) +} + + +def test_noisefill_ignores_blend_variables_in_the_launch_shell( + tmp_path, resolve_dag): + """A noisefill campaign launched from a shell that still exports the + uberseg variables plans exactly the campaign launched from a clean one, + and the parse leaves none of them in the environment jobs inherit (the + slurm executor submits with --export=ALL from this process).""" + campaigns = {} + for name in ("clean", "dirty"): + campaigns[name] = Campaign(tmp_path / name, "data", "psfex") + campaigns[name].config["blend_handling"] = "noisefill" + campaigns[name].write_config() + clean, dirty = campaigns["clean"], campaigns["dirty"] + _, clean_jobs = _surface(clean, resolve_dag) + + with resolve_dag(dirty, launch_env=INHERITED_BLEND_ENV) as dag: + leaked = sorted(set(INHERITED_BLEND_ENV) & set(os.environ)) + dirty_jobs = _jobs(dag, dirty) + assert leaked == [] + assert dirty_jobs == clean_jobs + for _, shell, _ in dirty_jobs.values(): + assert "SP_BLEND_HANDLING" not in (shell or "") diff --git a/tests/workflow/test_params_pin.py b/tests/workflow/test_params_pin.py index faeb43158..c97a8177b 100644 --- a/tests/workflow/test_params_pin.py +++ b/tests/workflow/test_params_pin.py @@ -7,6 +7,9 @@ from tests.workflow.params import params_pin PIN = Path(__file__).with_name("params_pin.json") +# blend_handling: noisefill's plan, pinned on its own so a campaign that ran as +# noisefill stays resumable with the knob set explicitly. +NOISEFILL_PIN = Path(__file__).with_name("params_pin_noisefill.json") BOUNDARY_MESSAGE = ( "params.pre is a rerun trigger under both profiles; this change reruns " "every finished unit of a resumed campaign — land it at a campaign " @@ -23,6 +26,21 @@ def test_unit_pre_changes_at_campaign_boundary(psfex_dag, pytestconfig): assert actual == json.loads(PIN.read_text()), BOUNDARY_MESSAGE +def test_noisefill_plan_changes_at_campaign_boundary(tmp_path, resolve_dag, + pytestconfig): + """The same pin for a campaign that sets blend_handling: noisefill.""" + campaign = Campaign(tmp_path / "campaign", "data", "psfex") + campaign.config["blend_handling"] = "noisefill" + campaign.write_config() + with resolve_dag(campaign) as dag: + actual = params_pin(dag) + if pytestconfig.getoption("--update-params-pin"): + NOISEFILL_PIN.write_text( + json.dumps(actual, indent=2, sort_keys=True) + "\n") + assert NOISEFILL_PIN.is_file(), BOUNDARY_MESSAGE + assert actual == json.loads(NOISEFILL_PIN.read_text()), BOUNDARY_MESSAGE + + def test_params_pin_ignores_fixture_root(tmp_path, resolve_dag): """A different temporary campaign directory cannot require a new pin.""" pins = [] diff --git a/universes/committed.yaml b/universes/committed.yaml index 2a517229c..30402df47 100644 --- a/universes/committed.yaml +++ b/universes/committed.yaml @@ -52,10 +52,11 @@ analyses: galaxy_pixel_weights: rms_vignet_weights psf_likelihood_noise: psf_noise_1em5 megacam_ccd_flip: megapipe_flip - defect_fill: noise - blend_handling: none - epoch_masked_fraction_cut: one_third - central_defect_veto: disabled + defect_fill: interpolate + blend_handling: uberseg + defect_weighting: fourfold_zero + epoch_masked_fraction_cut: ten_percent + central_defect_veto: fixed_radii catalogue_assembly: decisions: star_galaxy_classification: deferred_downstream diff --git a/workflow/README.md b/workflow/README.md index 4f0b135c3..80f2565c5 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -35,6 +35,13 @@ uv pip install 'snakemake>=9,<10' 'snakemake-executor-plugin-slurm>=2.7,<3' # (DR6), or the join fails. `sextractor` (the default for image sims) keeps # SExtractor's own NUMBER. Either way make_cat writes # TILE_UNIQUE_ID = tile_id * 10**6 + NUMBER. +# `blend_handling` is ngmix's neighbour treatment: `uberseg` (the default) or +# `noisefill`. Under both, tile_detect adds the coadd segmentation stamps to +# the sexcat as SEG_VIGNET and ngmix's neighbours are the other objects' +# footprints in them. It is chosen per campaign: flipping it on an existing +# run dir reruns tile_ngmix and the rest of the shape chain for every finished +# tile, and reclaimed exposures make that destructive, so a comparison needs +# its own `run:`. # The committed launcher loads apptainer/1.4.5 + the /project venv, so a # fresh shell always has the right state. diff --git a/workflow/Snakefile b/workflow/Snakefile index ac9d0760c..d6cce08db 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -31,6 +31,7 @@ the manifest says "this stage succeeded", the log says "here is what happened" (the contract is argued in completeness.py's docstring). """ +import ast import fnmatch import functools import hashlib @@ -188,6 +189,25 @@ try: except ValueError as err: raise WorkflowError(str(err)) from None +# How ngmix treats a stamp's neighbours; tile.smk's blend handling section +# carries it to tile_detect and tile_ngmix. +BLEND_HANDLING = config.get("blend_handling", "uberseg") +if BLEND_HANDLING not in run_config.BLEND_HANDLINGS: + raise WorkflowError( + f"Invalid blend_handling={BLEND_HANDLING!r}; expected one of " + f"{sorted(run_config.BLEND_HANDLINGS)}.") + +# The committed inis read SP_SEG_VIGNET and SP_BLEND_HANDLING with the modules' +# own defaults, and only tile.smk's blend_env exports them (SP_BLEND_HANDLING +# under uberseg, the workflow's default). Drop any copy the +# launching shell carries, bare or as the APPTAINERENV_/SINGULARITYENV_ forms +# that survive --cleanenv, so a job inherits none: the slurm executor submits +# with --export=ALL from this process's environment, and each job re-parses +# this file before its shell runs. +for _prefix in ("", "APPTAINERENV_", "SINGULARITYENV_"): + for _name in ("SP_SEG_VIGNET", "SP_BLEND_HANDLING"): + os.environ.pop(_prefix + _name, None) + # SP_PHASE is set by bin/sp and NOWHERE else: `prepare`/`compute` on the two # invocations of `sp run`, `passthrough` on direct commands (`sp --unlock`, `sp # --dag`, `sp exp_psf ...`). It gates the two parse-time side effects — the index @@ -231,6 +251,24 @@ scattergather: NGMIX_CHUNKS = workflow._scatter["ngmix"] +# How ngmix weights interpolated defects (DEFECT_WEIGHTING in +# config_tile_Ng_template.ini, exported by tile_ngmix as SP_DEFECT_WEIGHTING). +# Unset: empty, and ngmix takes its own default. The allowed values are read +# from the ngmix module this campaign runs, so a typo fails here, at parse time, +# rather than in every tile's ngmix job. +DEFECT_WEIGHTING = config.get("defect_weighting") or "" +_NGMIX_PY = (Path(workflow.basedir).parent / "src" / "shapepipe" / "modules" + / "ngmix_package" / "ngmix.py") +_ngmix_tree = ast.parse(_NGMIX_PY.read_text()) +DEFECT_WEIGHTINGS = next( + ast.literal_eval(node.value) for node in _ngmix_tree.body + if isinstance(node, ast.Assign) + and [getattr(t, "id", None) for t in node.targets] == ["DEFECT_WEIGHTINGS"]) +if DEFECT_WEIGHTING and DEFECT_WEIGHTING not in DEFECT_WEIGHTINGS: + raise WorkflowError( + f"Invalid defect_weighting={DEFECT_WEIGHTING!r}; expected one of " + f"{list(DEFECT_WEIGHTINGS)}.") + # --- parse-time index build + load (D1) ------------------------------------ # The COMPUTE invocation's parse IS the index build, and it runs UNCONDITIONALLY # there — no "some Fe output exists" guard. That guard used to make a diff --git a/workflow/config.yaml b/workflow/config.yaml index f144a8a0e..831835c2b 100644 --- a/workflow/config.yaml +++ b/workflow/config.yaml @@ -20,6 +20,16 @@ input_type: data # campaign's merged catalogues. Required, and set by the run config. # run: smk-g6 +# How ngmix treats a stamp's neighbours (its BLEND_HANDLING). tile_detect +# writes the coadd segmentation stamps into the sexcat as SEG_VIGNET under +# both, and the neighbours are the other objects' footprints in them: +# `uberseg` zeroes their weight; `noisefill`, ngmix's own default, zero-weights +# and noise-fills the -1e30 VIGNET markers on them. Chosen per campaign: +# flipping it on an existing run dir reruns tile_ngmix and everything after it +# for every finished tile, and reclaimed exposures make that destructive -- a +# comparison needs its own `run:`. +blend_handling: uberseg + # Default settings per input_type, can be overridden by a user-defined run config file. # - tile_detection: both run SExtractor on the tile. `unions_catalogue` (data) # also fetches the UNIONS per-tile catalogue and joins the detections to it, @@ -211,6 +221,10 @@ max_mem_mb: 750000 # ngmix chunks per tile. ngmix_chunks: 8 +# How ngmix weights interpolated defects: des_y6, fourfold_zero, hole or full +# (astra decision shape_measurement.defect_weighting). Unset: fourfold_zero. +# defect_weighting: des_y6 + # Node-local tile store (vignets, WCS). Unset: the profile's /local/scratch # bind (nibi NVMe, candide node /tmp). Dense image-sim campaigns on candide # want a shared directory instead. diff --git a/workflow/config/cfis/config_tile_Ng_template.ini b/workflow/config/cfis/config_tile_Ng_template.ini index 5d4c510e5..829042475 100644 --- a/workflow/config/cfis/config_tile_Ng_template.ini +++ b/workflow/config/cfis/config_tile_Ng_template.ini @@ -113,6 +113,14 @@ BKG_RMS_VIGNET_PATH = $NGMIX_VIGNET_DIR/vignetmaker_runner_run_2/output/backgrou # width, this bounds per-worker memory). SAVE_BATCH = 250 +# DEFECT_WEIGHTING: how interpolated defects are weighted (des_y6, +# fourfold_zero, hole or full; see ngmix.defect_weighting_masks). The workflow +# exports SP_DEFECT_WEIGHTING from the run config's `defect_weighting`; empty +# takes the default, fourfold_zero (DEFECT_WEIGHTING in +# shapepipe/modules/ngmix_package/ngmix.py). +# @sc [decision:shape_measurement.defect_weighting] +DEFECT_WEIGHTING = ${SP_DEFECT_WEIGHTING:-} + # Magnitude zero-point # @sc [decision:photometric_zeropoint] MAG_ZP = 30.0 @@ -122,3 +130,9 @@ MAG_ZP = 30.0 # via ShapePipe's getexpanded (env-expanded, not plain getint). ID_OBJ_MIN = $NGMIX_ROW_MIN ID_OBJ_MAX = $NGMIX_ROW_MAX + +# Neighbour treatment: noisefill, or uberseg, which reads the tile catalogue's +# SEG_VIGNET column. The workflow sets SP_BLEND_HANDLING to uberseg under its +# default blend_handling: uberseg, and leaves it unset under noisefill +# (workflow/config.yaml). +BLEND_HANDLING = ${SP_BLEND_HANDLING:-noisefill} diff --git a/workflow/config/cfis/config_tile_Sx.ini b/workflow/config/cfis/config_tile_Sx.ini index 43a59c493..6ca400b03 100644 --- a/workflow/config/cfis/config_tile_Sx.ini +++ b/workflow/config/cfis/config_tile_Sx.ini @@ -112,6 +112,14 @@ BKG_FROM_HEADER = False # OBJECTS, -OBJECTS, SEGMENTATION, APERTURES CHECKIMAGE = BACKGROUND, SEGMENTATION +# Write the SEGMENTATION check image's stamps, cut on each VIGNET's grid, as +# the int32 SEG_VIGNET column ngmix's blend handling reads under both +# noisefill and uberseg (needs the SEGMENTATION check image; SExtractor also +# computes X/Y_IMAGE_DBL, which centre the stamps and are left out of the +# sexcat). The workflow sets SP_SEG_VIGNET under every blend_handling +# (workflow/rules/tile.smk). +SEG_VIGNET = ${SP_SEG_VIGNET:-False} + # File name suffix for the output sextractor files (optional) SUFFIX = sexcat diff --git a/workflow/config/cfis/final_cat.param b/workflow/config/cfis/final_cat.param index fb8b3f3f7..aeab8b412 100644 --- a/workflow/config/cfis/final_cat.param +++ b/workflow/config/cfis/final_cat.param @@ -48,6 +48,8 @@ NGMIX_G2_PSF_ORIG_NOSHEAR # Number of epochs (exposures) N_EPOCH NGMIX_N_EPOCH +# epochs dropped because building their ngmix observation raised +NGMIX_N_EPOCH_FAILED # Per-epoch identity and PSF shape, slot n = 1..12 (config_tile_Mc.ini's # N_EPOCH_SLOTS): EXP_ID_n/CCD_n name the exposure and CCD of epoch n, aligned @@ -165,6 +167,13 @@ HSM_RHO4_PSF_12 # Blend flag: coadd seg stamp held a non-central footprint (shapepipe#776) NGMIX_NEIGHBOUR_FLAG +# Defect diagnostics over the kept epochs: epochs with interpolated pixels, +# min distance (px) from the centre to an interpolated / noise-filled defect +# pixel, -1 when none (ngmix Postage_stamp.record_defects) +NGMIX_N_EPOCH_INTERP +NGMIX_MIN_DIST_INTERP +NGMIX_MIN_DIST_NOISEFILL + ## Shape measurement outputs ## Ngmix: model fitting diff --git a/workflow/config/cfis_image_sims/final_cat.param b/workflow/config/cfis_image_sims/final_cat.param index 9db095b66..f90495bfe 100644 --- a/workflow/config/cfis_image_sims/final_cat.param +++ b/workflow/config/cfis_image_sims/final_cat.param @@ -31,6 +31,13 @@ NGMIX_MCAL_FLAGS NGMIX_MCAL_TYPES_FAIL NGMIX_NEIGHBOUR_FLAG +# Defect diagnostics over the kept epochs: epochs with interpolated pixels, +# min distance (px) from the centre to an interpolated / noise-filled defect +# pixel, -1 when none (ngmix Postage_stamp.record_defects) +NGMIX_N_EPOCH_INTERP +NGMIX_MIN_DIST_INTERP +NGMIX_MIN_DIST_NOISEFILL + # PSF ellipticity (original image PSF) NGMIX_G1_PSF_ORIG_NOSHEAR NGMIX_G2_PSF_ORIG_NOSHEAR @@ -38,6 +45,8 @@ NGMIX_G2_PSF_ORIG_NOSHEAR # Number of epochs (exposures) N_EPOCH NGMIX_N_EPOCH +# epochs dropped because building their ngmix observation raised +NGMIX_N_EPOCH_FAILED ## Shape measurement outputs ## Ngmix: model fitting diff --git a/workflow/rules/tile.smk b/workflow/rules/tile.smk index f607e3617..1f3ded7a1 100644 --- a/workflow/rules/tile.smk +++ b/workflow/rules/tile.smk @@ -54,6 +54,16 @@ catalogues; unpaired rows leave the catalogue. The catalogue is SExtractor run by MegaPipe on the same DR6 image with the same configuration, so the join is exact; on other pixels (a DR5 image) it fails loudly. +``blend_handling: uberseg``, the default, gives ngmix UberSeg's hard mask +instead of its own noise-fill default. Both read one coadd segmentation stamp +per object from the sexcat's SEG_VIGNET column: uberseg masks by it, and +noisefill takes as neighbours only the -1e30 VIGNET markers on another +object's footprint in it. tile_detect cuts it from SExtractor's SEGMENTATION +check image on the grid of each object's VIGNET, before the join, which +relabels every stamp to the catalogue's NUMBER (-1 on the footprints of rows +that leave, 0 on sky). The switch is one prologue variable and nothing else +(blend_env, below). + There is no `tile_mask` rule, and there will not be one (PR #847). ShapePipe generates no masks: tiles have no instrument flag image of their own, so tile_detect runs SExtractor with FLAG_IMAGE = False against @@ -472,6 +482,25 @@ rule tile_merge_headers: shell: sp_shell("tile_merge_headers", "config_tile_Mh_exp.ini") +# --- blend handling ------------------------------------------------------- + +# The blend-handling exports into a stage's prologue. tile_detect writes +# SEG_VIGNET (config_tile_Sx.ini, SEG_VIGNET = ${SP_SEG_VIGNET:-False}) under +# both handlings: noisefill keeps only the -1e30 markers on another object's +# footprint in it (split_tile_markers), uberseg masks by it. uberseg (the +# default) also sets BLEND_HANDLING (config_tile_Ng_template.ini, +# ${SP_BLEND_HANDLING:-noisefill}), which reruns only tile_ngmix on a flip. +BLEND_ENV = { + "tile_detect": {"SP_SEG_VIGNET": "True"}, + **({"tile_ngmix": {"SP_BLEND_HANDLING": "uberseg"}} + if BLEND_HANDLING == "uberseg" else {}), +} + + +def blend_env(stage): + return BLEND_ENV.get(stage, {}) + + # The UNIONS per-tile catalogue (CFIS..r.cat), from a local mirror or # from vos, the way tile_get_images fetches the image; tile_detect joins its # SExtractor detections to it. Reads only tile_numbers.txt, which unit_pre @@ -509,15 +538,28 @@ if TILE_DETECTION == "unions_catalogue": def detect_env(tile): - """tile_detect's prologue exports: the catalogue to join, if any. + """tile_detect's prologue exports: the catalogue to join, if any, and + SP_SEG_VIGNET. - Empty under tile_detection: sextractor (image simulations), so that no - value left exported in the submitting shell reaches the job. + SP_MATCH_CATALOGUE is empty under tile_detection: sextractor (image + simulations), so that no value left exported in the submitting shell + reaches the job. """ - if TILE_DETECTION != "unions_catalogue": - return {"SP_MATCH_CATALOGUE": ""} - gic = f"{tile_dir(tile)}/output/run_sp_tile_Gic/get_images_runner/output" - return {"SP_MATCH_CATALOGUE": f"{gic}/CFIS_cat{unit_num(tile)}.cat"} + match = "" + if TILE_DETECTION == "unions_catalogue": + gic = f"{tile_dir(tile)}/output/run_sp_tile_Gic/get_images_runner/output" + match = f"{gic}/CFIS_cat{unit_num(tile)}.cat" + return {"SP_MATCH_CATALOGUE": match, **blend_env("tile_detect")} + + +# tile_detect's extra memory for SEG_VIGNET: add_seg_vignet reads the 400 MB +# SEGMENTATION check image and adds an int32 SEG_VIGNET column the size of +# VIGNET, which the join then holds twice. MEASURED on DR6 tile 202.301 +# (36,022 detections; sextractor_runner as tile_detect runs it, without the +# post-processing): peak RSS 1.53 GiB without SEG_VIGNET, 2.93 GiB with it. +# Scaled to the 1.84 GiB worst tile above, it needs ~3.5 GiB; 7000 MB is 2x +# that. +DETECT_SEG_MEM_MB = 3000 # SExtractor object detection on the tile; under unions_catalogue joined to @@ -549,11 +591,11 @@ rule tile_detect: # 4000 MB is 2.1x the worst tile, and an OOM retries at 8000. The runtime # is ~7x the slowest tile, for /scratch I/O on nibi (a 400 MB image and # weight in, a ~430 MB catalogue out). nibi bills max(cores, mem_GB/4), so - # the job bills 1 core-equivalent. + # the job bills 1 core-equivalent; plus DETECT_SEG_MEM_MB above. threads: 1 retries: 1 resources: - mem_mb = lambda wc, attempt: 4000 * attempt, + mem_mb = lambda wc, attempt: (4000 + DETECT_SEG_MEM_MB) * attempt, runtime = lambda wc, attempt: 20 * attempt shell: sp_shell("tile_detect", "config_tile_Sx.ini") @@ -591,7 +633,12 @@ rule tile_vignets: # it carries the splitter's fingerprint too. Same hash, same constant — # the Snakefile's NGMIX_RANGE_HASH argues what it guards, and # tile_ngmix's copy below carries the mid-campaign-edit warning. - range_hash = NGMIX_RANGE_HASH + range_hash = NGMIX_RANGE_HASH, + # tile_ngmix exports it, so it is a params change there; carried here + # too so that changing it reruns the whole group from tile_vignets, + # never the chunks alone (see the mid-campaign deletion note on + # tile_ngmix's range_hash). + defect_weighting = DEFECT_WEIGHTING # 8, not 16, for the same reason tile_ngmix is 1: `-b {threads}` is SMP # batch size over input FILE SETS, and a tile is one set -- this run's own # log says "Batch size: 16 / Total number of processes: 1". 16 was the @@ -614,6 +661,13 @@ rule tile_vignets: sp_shell("tile_vignets", f"config_tile_PiViVi_{PSF_MODEL}.ini", check_args=' --run-dir "$SP_LOCAL" --unit {wildcards.tile}') +# Each ngmix chunk's extra memory for SEG_VIGNET: Tile_cat holding the tile's +# SEG_VIGNET column, a view into the loaded table like VIGNET. Measured on DR6 +# 202.301 (36065 rows, 51x51): Tile_cat peak and resident RSS 955 MiB with the +# column, 595 without (+360 MiB, the column's size). 500 covers 186.307's +# ~37.9k rows (~375 MiB) with a third to spare. +NGMIX_SEG_MEM_MB = 500 + # ngmix shape measurement — N chunks per tile (D4). Each chunk LOOKS UP its own # CLOSED catalogue-row range in the file tile_vignets materialised at the top of this # group job (TILE_NGMIX_RANGES); the ranges are knowable only at EXECUTION time, @@ -644,7 +698,9 @@ rule tile_ngmix: f"{TILE_DIR}/logs/tile_ngmix_{{chunk}}.json" params: pre = lambda wc: unit_pre("tile_ngmix", wc.tile, - env={"SP_NGMIX_CHUNK": wc.chunk, "NGMIX_N_CHUNKS": NGMIX_CHUNKS}, + env={"SP_NGMIX_CHUNK": wc.chunk, "NGMIX_N_CHUNKS": NGMIX_CHUNKS, + "SP_DEFECT_WEIGHTING": DEFECT_WEIGHTING, + **blend_env("tile_ngmix")}, # Two steps, not `eval "$(...)"`: a command substitution inside eval # discards the script's exit status, so a missing sexcat would fall # through to shapepipe_run with an unset range and fail as something @@ -803,7 +859,10 @@ rule tile_ngmix: # margin over the worst tile measured, and the first thing to give would # be the page cache holding the store. Take it only with a measurement # of cache behaviour under pressure, not on the arithmetic alone. - mem_mb = lambda wc, attempt: 5000 * attempt, + # + # Each chunk also holds the tile's SEG_VIGNET column, + # int32 and the size of VIGNET: NGMIX_SEG_MEM_MB below. + mem_mb = lambda wc, attempt: (5000 + NGMIX_SEG_MEM_MB) * attempt, # 120 on the FIRST attempt, and ATTEMPT-SCALED after it. MEASURED # two ways, and the second is why the margin is thinner than it looks: # * alone (job 20795277, tile 198.305 chunk 1): ~76 min for 3547 diff --git a/workflow/scripts/run_config.py b/workflow/scripts/run_config.py index 669a3ec60..7cf9c18c3 100644 --- a/workflow/scripts/run_config.py +++ b/workflow/scripts/run_config.py @@ -28,6 +28,10 @@ # never mention `$run` must still set it. REQUIRED = ("run", "tile_list", "inputs.tiles", "inputs.exposures", "outputs.run_dir", "outputs.index_db") +# ngmix's BLEND_HANDLING values, which `blend_handling` takes. A copy, because +# the Snakefile validates it outside the container; tests/unit holds it equal +# to ngmix_package.ngmix.BLEND_HANDLINGS. +BLEND_HANDLINGS = ("noisefill", "uberseg") def merge(base, over):