`h5py` Error: "Unable to synchronously open file"

[deangeckt]

Description

I am encountering a crash when attempting to load the network, despite having successfully run the download-pretrained command (all models and h5 files appear to be downloaded correctly).

The error is FileNotFoundError: [Errno 2] Unable to synchronously open file.

This is not a missing file issue (the files exist at the path); it appears to be an issue with how datamate/flyvis attempts to access or open the HDF5 files on certain systems. The internal h5py call fails to acquire the file handle, possibly due to specific flags (like SWMR) or read-before-write logic used in the library.

I reproduced this on:

1. Local Machine: Windows
2. Cloud: Google Colab (Linux)

Traceback (from colab)

/usr/local/lib/python3.12/dist-packages/h5py/_hl/files.py in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
    239         if swmr and swmr_support:
    240             flags |= h5f.ACC_SWMR_READ
--> 241         fid = h5f.open(name, flags, fapl=fapl)
    242     elif mode == 'r+':
    243         fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()
h5py/h5f.pyx in h5py.h5f.open()

FileNotFoundError: [Errno 2] Unable to synchronously open file (unable to open file: name = 'data/flyvis_data/connectome/ConnectomeFromAvgFilters_0006/edges/n_syn.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

Investigation & Fix

The issue seems to be in datamate.io._write_h5. The original implementation likely attempts to open the file in a mode (e.g., r+ or with specific checking logic) that conflicts with the file state or OS file handling on these platforms.

I fixed this by monkey-patching _write_h5 to bypass the complex read/check logic and force a standard write operation (mode="w"). This confirms the issue is with the opening method, not the file existence.

Working Patch (on local windows machine)

Here is the override I am using to get it to work:

import h5py
from pathlib import Path
import datamate.io
import datamate.directory

def fixed_write_h5(path, val):
    """
    A Windows-safe replacement that skips the 'read-before-write' check.
    """
    path = Path(path)
    path.parent.mkdir(parents=True, exist_ok=True)

    # Simple, safe write. No checking 'f["data"]', no 'unlink'.
    # This completely bypasses the logic causing your KeyError.
    with h5py.File(path, mode="w", libver="latest") as f:
        f.create_dataset("data", data=val)

# Patch 1: The definition
datamate.io._write_h5 = fixed_write_h5

# Patch 2: The copy inside directory.py (THIS IS THE KEY FIX)
if hasattr(datamate.directory, "_write_h5"):
    datamate.directory._write_h5 = fixed_write_h5
    print(" -> Patched datamate.directory._write_h5")
else:
    print(" -> Warning: Could not find _write_h5 in directory module")


print("Importing flyvis...")
from flyvis import NetworkView

This patch allows the library to load and function correctly.

---

[lappalainenj]

Hey, thanks for raising this issue.

It seems to be more related to datamate than to flyvis -- would you mind raising it directly on the datamate repository?

If your solution works across all unit tests, I'm also happy to review a pull request to adopt it, in particular as it seems like it simplifies the code.