Skip to content

Fix first-run checkpoint download race#844

Merged
m-reuter merged 3 commits into
Deep-MI:devfrom
arvraz:fix/checkpoint-race-and-numpy2-cross
Jul 22, 2026
Merged

Fix first-run checkpoint download race#844
m-reuter merged 3 commits into
Deep-MI:devfrom
arvraz:fix/checkpoint-race-and-numpy2-cross

Conversation

@arvraz

@arvraz arvraz commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Two independent bugs surfaced while running the full pipeline (run_fastsurfer.sh, seg + surface) on a fresh install. Both are fixed here; they are unrelated and can be split into separate PRs if preferred.

1. Checkpoint download race → PytorchStreamReader failed reading file data/1

On a fresh run (empty checkpoints/), the CorpusCallosum module crashes:

RuntimeError: PytorchStreamReader failed reading file data/1: file read failed

which then cascades (callosum.CC.orig.mgz missing → paint_cc_into_pred.py → no aseg.auto.mgz → segstats fails).

Cause: download_checkpoint() in FastSurferCNN/utils/checkpoint.py writes the response directly to the final path, while check_and_download_ckpts() guards downloads with if not checkpoint_path.exists() — i.e. it checks existence, not completeness. CorpusCallosum/fastsurfer_cc.py loads the localization and segmentation models in two parallel threads, and both call download_checkpoints(cc=True). One thread creates the (70 MB) file and begins writing; the other sees it "exists", skips the download, and torch.loads the partially-written file.

Fix: download to a temporary file in the same directory and os.replace() it into place, so the checkpoint only ever appears at its final path once complete.

2. Corpus callosum measurements fail under NumPy ≥ 2

Every CC slice fails with:

ValueError: Both input arrays must be (arrays of) 3-dimensional vectors,
but they are 2 and 2 dimensional instead.

so all CC morphometry is silently dropped (cc_num_failed_slices = number of slices).

Cause: NumPy 2.0 removed the 2-element form of np.cross. CorpusCallosum/shape/subsegment_contour.py and CorpusCallosum/utils/mapping_helpers.py use it only to get the sign (z-component) of the angle between two in-slice 2D vectors. This affects the pinned requirements.txt (numpy 2.4.4) as well as newer releases.

Fix: compute the 2D cross-product scalar directly, a[0]*b[1] - a[1]*b[0] (equivalent, and NumPy-version-independent).

Verification

  • Deleted the CC checkpoints and re-ran the CC module: fresh concurrent download succeeds, no PytorchStreamReader, both checkpoints torch.load valid.
  • CC measurements now compute: callosum.CC.all_slices.json written with cc_num_failed_slices: 0 (previously all slices failed).
  • Full run_fastsurfer.sh (seg + surface) completes end-to-end with valid surfaces and thickness.
  • ruff check clean; test/image (387 tests) still pass.

arvraz added 2 commits July 14, 2026 17:10
download_checkpoint() wrote the HTTP response straight to the final path, so a
concurrent caller could observe the file through the exists() guard in
check_and_download_ckpts() while it was still being written and torch.load a
truncated checkpoint. This reliably crashes the CorpusCallosum module on a
fresh run, where the localization and segmentation models are loaded in
parallel threads that both call download_checkpoints():

    RuntimeError: PytorchStreamReader failed reading file data/1: file read failed

Download to a temporary file in the same directory and os.replace() it into
place, so the checkpoint only ever becomes visible at its final path once it
is complete.
NumPy 2.0 removed the 2-element form of np.cross, which now raises
"Both input arrays must be (arrays of) 3-dimensional vectors". The corpus
callosum shape code used np.cross only to obtain the sign (z-component) of the
angle between two in-slice 2D vectors, so every slice's measurements failed:

    ValueError: Both input arrays must be (arrays of) 3-dimensional vectors,
    but they are 2 and 2 dimensional instead.

Compute the 2D cross-product scalar directly (a[0]*b[1] - a[1]*b[0]). This is
equivalent and works on the pinned numpy (2.4.4) as well as newer releases.
@m-reuter

Copy link
Copy Markdown
Member

Hi thanks for the contribution.

The numpy 2.5 fix is a duplicate from here: #842

We will check the race condition. I think the same checkpoint files should not be downloaded twice, the localiser should download only its own weights and the CC segmentation its own. Alternatively, all weights should be downloaded before the parallel block starts. We should avoid duplicate parallel downloads.

@m-reuter m-reuter changed the title Fix first-run checkpoint download race and CC measurements under NumPy 2 Fix first-run checkpoint download race Jul 21, 2026
@m-reuter
m-reuter requested a review from ClePol July 21, 2026 15:55
@ClePol

ClePol commented Jul 22, 2026

Copy link
Copy Markdown
Member

Hi arvaz, thanks for contributing and making us aware of the download concurrency issues!

I have pushed a modified version to the branch. This version

  1. Adresses the concurrency for a single case by downloading only the localisation and segmentation models respectively in it's thread. This allows for the intended parallel download while avoiding the collision.
  2. To make the downloads safer in case of running multiple cases it follows your approach of creating a temporary file that is only named checkpoint after the download has completed. However the previous implementation did not give read permissions to the group, which would have broken our docker install setup. So I replaced it with a different tempfile setup.
  3. I reverted the CC changes as this was already merged in Fix CorpusCallosum NumPy 2.5 compatibility #842

From my side this is now ready for merging

@m-reuter
m-reuter merged commit 2a94ee9 into Deep-MI:dev Jul 22, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants