Repeatedly triggering `FlushNow` occasionally seems to cause the NDFileHDF5 plugin to freeze

[jwlodek]

In our automated data collection code, we call flush now whenever we post updates about how many frames were collected for each detector onto our message bus so that clients can leverage the SWMR support for visualization or analysis, guaranteeing that the frame is available on-disk.

Recently, we started seeing the call to FlushNow occasionally lock up the plugin:

XF:27ID1-BI{Kinetix-Det:1}HDF1:FlushNow 2026-07-08 12:04:12.928794 Done  
XF:27ID1-BI{Kinetix-Det:1}HDF1:FlushNow 2026-07-08 12:04:15.931710 Flush  
XF:27ID1-BI{Kinetix-Det:1}HDF1:FlushNow 2026-07-08 12:04:15.952663 Done  
XF:27ID1-BI{Kinetix-Det:1}HDF1:FlushNow 2026-07-08 12:04:18.957255 Flush  
XF:27ID1-BI{Kinetix-Det:1}HDF1:FlushNow 2026-07-08 12:04:18.983072 Done  
XF:27ID1-BI{Kinetix-Det:1}HDF1:FlushNow 2026-07-08 12:04:21.997243 Flush  <== Here, the previous flushes returned in less than a second, this one blocked for over 10 seconds.
XF:27ID1-BI{Kinetix-Det:1}HDF1:FlushNow 2026-07-08 12:04:34.097083 Done  

The above is a camonitor of the PV showing how the acquisition code will write 1 to FlushNow every 3 seconds, until the lockup.

There are no error messages that I can see in the IOC shell. The IOC remains otherwise completely responsive, but the HDF plugin has it's array rate drop to zero.

I can observe this with any value for the flush on nth frame PV (1 or 10000). One note is that the underlying network storage was recently upgraded from a lustre filesystem to a NetApp volume (nfs4). We didn't see this prior to the switch as far as I know. The ADCore version and driver (as well as the acquisition code) were all unchanged. I upgraded the ADCore version from 3-10 to the most recent master branch, but the behavior is the same.

[MarkRivers]

This is what AI says about using SWMR with NetApp.

HDF5 SWMR (Single-Writer/Multiple-Reader) is not officially supported and generally does not work on NetApp storage.

Why SWMR and NetApp Don't Mix

Network File Systems:
SWMR relies entirely on a file system that supports strict POSIX write semantics to maintain correct data ordering. NetApp filers export storage over NFS or SMB, which are network file systems known to cause write-ordering issues.

File Locking: HDF5 uses strict file locking schemes which often fail, conflict, or cause performance degradation when operating over network-mounted volumes.

Data Corruption Risks: Using SWMR on a NetApp drive frequently results in file corruption, crashes, or inaccessible data because the reader may attempt to access chunks before the writer's blocks are fully synced.

[MarkRivers]

This is what AI says about SWMR and NFS4:

In short: No, standard HDF5 SWMR does not reliably work on NFSv4. The HDF5 SWMR User Guide and The HDF Group explicitly state that SWMR is not supported on network file systems (NFS, SMB, etc.).

Why It Fails
Standard SWMR relies on strict POSIX write() semantics and cache flush ordering. Network file systems like NFSv4 do not guarantee this strict order across different nodes or clients, meaning readers may pull incomplete metadata or corrupt files.

The Alternative: VFD SWMR
If your data must reside on NFSv4, The HDF Group developed VFD SWMR (Virtual File Driver SWMR). It addresses traditional SWMR limitations by using a specialized metadata file and snapshotting.

However, even with VFD SWMR:

• It is not fully recommended or natively expected to work flawlessly with NFSv4 out-of-the-box in all configurations.
• It generally requires high-performance parallel file systems like GPFS or Lustre for proper synchronization guarantees.

[jwlodek]

It appears this may not be related to FlushNow or SWMR. I've been testing without explicitly calling flush and with SWMR off, and was still able to reproduce. The plugin is active, has frames in it's queue, but the array rate is zero, and the plugin appears unresponsive. The rest of the IOC responds OK.

[jwlodek]

I cannot reproduce this when writing to a local disk. I've reached out to the data engineering group who maintains our storage to see if they might have some insight.