Same context as the other two issues, 1 and 2 but now I have realized (I should have thought about this before) that dandi-cli has no support for NWB stored as Zarr: a .nwb.zarr. Currently, this is handled as a generic Zarr asset, so none of the NWB machinery ever sees it.
In my specific case described on the issues above two things happend:
- The validation does not stop the upload in its tracks if the referenced videos are missing.
dandi organize skips the .nwb.zarr file silently, no entry, no warning, no error.
I am not sure adding support is mechanically hard but there is a dependency problem. Reading of the file is supported by the backend independent pynwb.read_nwb, which is able to handle the two backends as long as hdmf-zarr is installed. That covers _get_pynwb_metadata, and validate already goes through pynwb.validate(paths=...) on its main path. Then there are the following sites that read quickly and where doing the full nwb read is probably too heavy, they will require a backend specific branch:
get_nwb_version, which opens the file only to read one root attribute.get_object_id, same thing for object_id.get_neurodata_types and its _scan_neurodata_types helper, which walk the raw tree looking for neurodata_type.nwb_has_external_links, the gate that refuses files whose content lives in other files. hdmf-zarr has links with a source too, so this ports the same way.
I can take the above if that makes sense for you but the thorny issue is dependencies. nwb.zarr will require Python 3.12 and zarr 3.2 or above, see the version discussion in hdmf-zarr. Currently you have zarr pinned to zarr>=2.18.0,<=3.1.5. Is there a reason for this? How could we move forward?
Same context as the other two issues, 1 and 2 but now I have realized (I should have thought about this before) that dandi-cli has no support for NWB stored as Zarr: a
.nwb.zarr. Currently, this is handled as a generic Zarr asset, so none of the NWB machinery ever sees it.In my specific case described on the issues above two things happend:
dandi organizeskips the.nwb.zarrfile silently, no entry, no warning, no error.I am not sure adding support is mechanically hard but there is a dependency problem. Reading of the file is supported by the backend independent
pynwb.read_nwb, which is able to handle the two backends as long as hdmf-zarr is installed. That covers_get_pynwb_metadata, andvalidatealready goes throughpynwb.validate(paths=...)on its main path. Then there are the following sites that read quickly and where doing the full nwb read is probably too heavy, they will require a backend specific branch:get_nwb_version, which opens the file only to read one root attribute.get_object_id, same thing forobject_id.get_neurodata_typesand its_scan_neurodata_typeshelper, which walk the raw tree looking forneurodata_type.nwb_has_external_links, the gate that refuses files whose content lives in other files. hdmf-zarr has links with asourcetoo, so this ports the same way.I can take the above if that makes sense for you but the thorny issue is dependencies. nwb.zarr will require Python 3.12 and zarr 3.2 or above, see the version discussion in hdmf-zarr. Currently you have zarr pinned to
zarr>=2.18.0,<=3.1.5. Is there a reason for this? How could we move forward?