Fix five silent-failure bugs in the streaming pipeline#5
Open
marinerhemant wants to merge 5 commits into
Open
Conversation
laue_orchestrator waited on lsu.wait_for_port(), which only watches the socket
and knows nothing about the daemon process. That failed in both directions:
- A daemon that is merely slow to start was killed. Startup reads a multi-GB
orientation database and then initialises a CUDA context; on a cold page
cache or a contended GPU that can exceed the fixed 180 s budget, and the run
aborted with "Daemon did not open port in time" despite nothing being wrong.
Observed on a 7.2 GB / 100 M-orientation database: both phases reached
"Initializing CUDA..." and were SIGKILLed at the 3-minute mark, losing the
launch. The identical scan started cleanly on retry.
- A daemon that died instantly (bad params, missing file, GPU error) still
held the orchestrator for the full 180 s before reporting anything, and the
reported message blamed a timeout rather than the actual exit.
Replace it with _wait_for_daemon_port(), which polls the port and the process
together: fail immediately with the daemon's exit code when it is genuinely
dead, keep waiting while it is alive, and log progress every 30 s so a slow
start is visible rather than silent. The timeout is retained as a backstop for
an alive-but-wedged daemon and raised 180 s -> 900 s, with the error pointing at
--port-timeout.
Verified: dead daemon returns in 2.0 s with its exit code (was: 900 s);
slow-but-alive daemon is waited for and succeeds; alive-but-silent daemon still
honours the cap.
The CONFIG block says 'set WATCH="" to batch an existing folder', but the
default was applied with ${WATCH:-"--watch"}, and the ":-" form substitutes the
default when the variable is unset *or empty*. Setting WATCH="" therefore still
produced --watch, so a run over an already-complete folder silently stayed in
watch mode: it indexed every frame, then parked forever waiting for more, and
never ran post-processing or wrote any output.h5 until someone noticed and
touched STOP_LAUE by hand.
Use ${WATCH-"--watch"} so an explicitly empty WATCH means batch mode while an
unset WATCH still defaults to watching. Verified: WATCH="" -> "", unset ->
"--watch".
This matters for unattended batch processing, where nothing is watching to
notice that a finished scan is sitting idle.
The flush wait broke as soon as solutions.txt existed and was non-empty, slept 2 s, and SIGTERMed the daemon. But solutions.txt is appended to as each frame is fitted, so its existence means the FIRST frame landed, not the last. The daemon is routinely behind — it logs "receive queue was full N times (backpressure from processing)" — so terminating it two seconds after the first result silently discards everything still queued. Observed on a 6561-frame batch run: the last 31 frames were missing, contiguously, with no error in any log. The scan reported success. Watch-mode runs had been masking this, because a finished scan sits idle until someone touches STOP_LAUE, which happens to give the daemon minutes-to-hours to drain; in batch mode (WATCH="") there is no such pause and the tail is lost every time. Wait for solutions.txt to stop growing instead: that is the daemon actually draining. Quiescence window is max(flush_time, 10 s), with an hour-long backstop and a warning that explicitly says results may be truncated if it trips. Verified against a synthetic writer: the loop does not break while the file is still growing, and captures every byte written.
…scans
frame_mapping is written by BOTH the sender thread and the consumer thread, while
the consumer also serialises it to JSON every save_interval frames. Nothing
guarded it, so json.dump() iterated the dict while the other thread inserted into
it and raised:
RuntimeError: dictionary changed size during iteration
laue_image_server.py _consumer_thread -> save_frame_mapping
laue_stream_utils.py json.dump(mapping, f, indent=1)
That exception killed the consumer thread. The server then shut down through its
normal path and reported
Done: 15955 sent, 0 skipped, 9599.3s total
on a 40,401-frame scan -- a silent 61% truncation presented as success, with
post-processing then run over the partial result. Nothing in the orchestrator or
the logs flagged it as a failure.
The window scales with frame count and with backpressure, which is why it went
unnoticed on 6.5k-frame scans and appeared on a 40k-frame one that logged 15,933
queue-full events.
Fixes:
- add mapping_lock; take it around every frame_mapping mutation in both threads
- periodic save takes a snapshot under the lock and writes the file outside it,
and can no longer kill the consumer: a failed periodic save is now a warning,
since the final save after the producer finishes is the authoritative one
- final save also snapshots under the lock
Verified with a threaded reproducer: unguarded, dumps raise RuntimeError while a
writer mutates concurrently; with the lock, 400/400 dumps succeed.
… calls N The launcher set CUDA_VISIBLE_DEVICES=$gpu without CUDA_DEVICE_ORDER. CUDA then enumerates by its default FASTEST_FIRST ordering, which need not match nvidia-smi indices on a host with mixed GPUs. Observed on a shared machine with two H200s (nvidia-smi 0,1) and two RTX PRO 6000s (2,3): a run configured for GPU 0/1 placed its daemon on physical GPU 2 -- a card already running another user's multi-day job -- while the intended H200s sat completely idle. Nothing in the logs indicated the substitution. Pin CUDA_DEVICE_ORDER=PCI_BUS_ID at launch so the configured index always selects the intended card. This matters most on shared systems, where the failure mode is quietly competing for someone else's GPU.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five defects found while running ~150,000 Laue frames through the streaming pipeline
across four machines during the July 2026 34-ID-E beam time. Four share a signature:
the failure reported success, so partial or misplaced work looked like a clean run.
Each commit is standalone and independently verified.
1. Port wait killed healthy daemons (
38f69c6)wait_for_port()is process-blind and enforced a flat 180 s budget. Daemon startup readsa multi-GB orientation database and then initialises a CUDA context; on a cold cache or a
contended GPU that legitimately exceeds it. Observed: 276 s on one host, so every
launch there would have failed with a message blaming a timeout. It also held the
orchestrator for the full 180 s when the daemon had already crashed.
Now polls the port and the process: fails immediately with the exit code when the daemon
is dead, waits while it is alive, logs progress every 30 s, cap raised to 900 s.
2.
WATCH=""never worked (ec64ad4)The CONFIG block documents
WATCH=""to batch an existing folder, but${WATCH:-...}substitutes the default for an empty value too. Batch runs silently stayed in watch mode:
they indexed every frame, then parked forever, writing no output until someone noticed.
3. Daemon terminated before draining (
d6c8107)The flush wait broke as soon as
solutions.txtexisted, then killed the daemon 2 s later.That file is appended per frame, so its existence means the first frame landed, not the
last. Observed: the final 31 frames of a 6,561-frame scan lost, contiguously, reported as
success. Watch-mode runs masked this by idling before shutdown; batch mode truncates every
scan. Now waits for the file to stop growing.
4.
frame_mappingJSON race (edca650)frame_mappingis written by both the sender and consumer threads while the consumerserialises it to JSON. Unguarded,
json.dumpiterated it mid-insert and raisedRuntimeError: dictionary changed size during iteration, killing the consumer thread. Theserver then shut down through its normal path reporting
Done: 15955 sent, 0 skipped— a 61% truncation on a 40,401-frame scan, presented as aclean run, with post-processing then run over partial data.
Adds a lock around every mutation; the periodic save snapshots under it and can no longer
kill the thread. Verified with a threaded reproducer.
5.
CUDA_DEVICE_ORDERunset (abdcbc3)CUDA_VISIBLE_DEVICES=$gpuwithoutCUDA_DEVICE_ORDERuses CUDA's FASTEST_FIRST ordering,which need not match nvidia-smi indices. Observed on a shared host: a run configured for
GPU 0/1 placed its daemon on physical GPU 2 — another user's multi-day job — while the
intended cards sat idle. Nothing in the logs indicated the substitution.
Verification
daemon is waited for; alive-but-silent still honours the cap
WATCH:""→ batch, unset →--watchSuggested follow-up
Regression tests for all five —
tests/already has a characterization suite, and each ofthese is cheaply testable without a GPU. Without them, all five can silently return.