Fix nfev_fit int64 dtype lock crashing batch-saved ngmix output (#795)#802
Open
cailmdaley wants to merge 1 commit into
Open
Fix nfev_fit int64 dtype lock crashing batch-saved ngmix output (#795)#802cailmdaley wants to merge 1 commit into
cailmdaley wants to merge 1 commit into
Conversation
compile_results filled nfev_fit with np.nan on failed fits, the only int-like column with a float fallback. With SAVE_BATCH enabled, save_results locks column dtypes from batch 1; if that batch has no failures the column locks to int64, and the first later batch with a failure feeds NaN into it, raising "cannot convert float NaN to integer" on append. Use ngmix's own -1 failed/absent-nfev sentinel instead of NaN, keeping the column int64 across every batch. Fixes #795
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
compile_resultsfilled thenfev_fitcolumn withnp.nanon failed fits — the only int-like output column with a float fallback.SAVE_BATCHenabled,save_resultslocks column dtypes from batch 1 (freshsave_as_fitswrite) and appends every later batch into a structured array typed to those locked dtypes. If batch 1 happens to have zero failed fits,nfev_fitlocks to int64 (FITS'K'); the first later batch containing a failed fit then feeds a NaN into that int64 column, raisingValueError: cannot convert float NaN to integerat append time — crashing tiles mid-run, exactly at a batch boundary as reported.-1failed/absent-nfevsentinel (already documented in the adjacent comment) instead ofnp.nan.nfev_fitis now int64 with-1meaning failed/absent, dtype-stable across every batch.Test plan
test_save_results_batches_survive_a_later_failed_fitdrives the realsave_resultsover a two-batch sequence (batch 1 all-success, batch 2 with a failed fit type) — fails on develop with the exact reported crash atngmix.py:612, passes with the fix.tests/module/test_ngmix.py— 18 passed (in-container).pytest -m "not slow" tests/science— 11 passed, no regressions.nfevreference in the codebase — no consumer reads or NaN-masks the output column.Fixes #795
— Claude Fable on behalf of Cail
🤖 Generated with Claude Code
https://claude.ai/code/session_01J9qEVRCWEekT5ACNB3rjkq