Add support for handling zarr urls

[stephenworsley]

Closes #6967

[codecov]

Codecov Report

❌ Patch coverage is 94.04762% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.20%. Comparing base (c2ffc9b) to head (1aa431e).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/iris/fileformats/netcdf/saver.py	83.33%	3 Missing ⚠️
lib/iris/loading.py	86.66%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7113      +/-   ##
==========================================
+ Coverage   90.18%   90.20%   +0.02%     
==========================================
  Files          91       91              
  Lines       25102    25160      +58     
  Branches     4706     4718      +12     
==========================================
+ Hits        22638    22696      +58     
  Misses       1685     1685              
  Partials      779      779              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[pp-mo]

Some Notes

Since @pp-mo @stephenworsley discussed this yesterday, here's my take on some things we found...

• I pointed out that I'm not too sure about the approach here , because I think it undermines the intent of the "schemes"

  • if done more rigorously, I think it should be possible to supply a url to a remote zarr dataset, similar to the OpenDAP protocol supported by the netcdf file loader
    • .. but I don't know if there's a server solution that actually supports such url-based remote access ?
  • .. and admittedly the whole framework of "schemes" is a bit clunky :
    Notably, in the case of netcdf files, we can access from a filepath, a url, or an open dataset ...
    • first, we have separate "schemes" for handling filepaths, urls, and dataset(objects), which are managed here
    • we then supply separate "format agents" for the three different routes here , here and here
    • but then all of those call the same iris.fileformats.netcdf.loader.load_cubes handler function
      • .. and other code in the CFReader then needs to separately detect+handle the open-dataset case anyway.

• we also found that saving to zarr by this means currently fails for lazy data streaming (though OK for all-real data)

  • since we added "deferred writes" for netcdf (Lazy netcdf saves #5191), our standard approach is to create a variable and not write the data. We then close the file, and the proxies re-open it in modify ("r+") mode, to fill in a chunk of data.
  • .. it looks like this simply doesn't work with netCDF4-python any more if it is writing Zarr :
    when such a file is re-opened, no such variable exists (the metadata is not saved ?)
  • .. it's not clear though whether this is an essential limitation of Zarrr itself (controlling metadata cannot exist when there are no recorded chunks for a given array ??). Or possibly, this is just a problem with the netCDF4-python implementation.
  • we could however fix this by giving the 'lazy stream' operation an extra mode control, to be specifically turned on for Zarr ..
    This would use a "da.store" directly when creating the original file, i.e. a "third way" for this code to function, where the 'store' operation is just da.store -- which is exactly what the code used to do before Lazy netcdf saves #5191.
    N.B. I already found in Lazy netcdf saves #5191 that you can't create a write proxy and just 'da.store' it immediately (? at least for process-based/distributed scheduler? ), since it needs to re-open the file with write privilege, which will then fail (since the file is already, or "still" open for write). So, to store immediately, you must call da.store directly.

• we need to discuss whether this makes this whole approach too awkward for supporting Zarr via this route?
  @ukmo-ccbunney to reconsider versus alternative mechanisms in Zarr I/O #6961 ??

[scitools-ci]

Templating

This PR includes changes that may be worth sharing via templating. For each file listed below, please either:

• Action the suggestion via a pull request editing/adding the relevant file in the SciTools/.github templates/ directory. ¹
• Raise an issue against the SciTools/.github repo for the above action if you really don't have 10mins spare right now. Include an assignee, to avoid it being forgotten.
• Dismiss the suggestion if the changes are not suitable for templating.

You will need to dismiss this review before this PR can be merged. Recommend the reviewer does this as their final action before merging, as this text will continually update as commits come in.

Template candidates

The following changed files are not currently templated, but their parent directories suggest they may be good candidates for a new template to be created:

• docs/src/common_links.inc

Footnotes

1. Include this text in the PR body to avoid any notifications about applying the template changes back to the source repo!
   @scitools-templating: please no update notification on: iris ↩

[ukmo-ccbunney]

Looking good so far. Just a few suggestions and comments.

[ukmo-ccbunney]

Just a thought I meant to add to the original review...

[ukmo-ccbunney]

OK. Everything mostly looks good. I've made a few suggested changes to fix the failing test.

I've also pushed up some whatsnew fragments that Pak Lam and myself put together, see 90236d2

[ukmo-ccbunney]

Spotted a slight oddity...

[ukmo-ccbunney]

I've noticed that this zarr file isn't loading into a single cube.

If you load this via iris.load_cube() it fails to merge into a single cube.
Testing on my VDI shows that it actually loads into two cubes - the second being the surface_altitude cube.
The equality test still passes (as it takes the first cube via slice [0]), so surface_altitude is getting added as an AuxCoord on the first cube ok, so it is a bit odd.

Furthermore, when accessing the data on my VDI, I get the following errors thrown from the HDF5 layer, although they don't actually stop me from loading the data (so more warnings). I don't see the same warnings in the CI tests though (I'm running the same version of netcdf)

HDF5-DIAG: Error detected in HDF5 (1.14.6) thread 1:
  #000: H5F.c line 496 in H5Fis_accessible(): unable to determine if file is accessible as HDF5
    major: File accessibility
    minor: Not an HDF5 file
  #001: H5VLcallback.c line 3913 in H5VL_file_specific(): file specific failed
    major: Virtual Object Layer
    minor: Can't operate on object
  #002: H5VLcallback.c line 3848 in H5VL__file_specific(): file specific failed
    major: Virtual Object Layer
    minor: Can't operate on object
  #003: H5VLnative_file.c line 344 in H5VL__native_file_specific(): error in HDF5 file check
    major: File accessibility
    minor: Can't get value
  #004: H5Fint.c line 1055 in H5F__is_hdf5(): unable to open file
    major: File accessibility
    minor: Unable to initialize object
  #005: H5FD.c line 787 in H5FD_open(): can't open file
    major: Virtual File Layer
    minor: Unable to open file
  #006: H5FDsec2.c line 323 in H5FD__sec2_open(): unable to open file: name = 'file:///tmp/4dwe.zarr#mode=nczarr', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0
    major: File accessibility
    minor: Unable to open file

[ukmo-ccbunney]

So, after some investigation, it has become apparent that the realistic_4d_w_everything cube doesn't round trip if you save and load it as a netcdf file either - the same merge errors apply.

The same applied with iris.COMBINE_POLICY.set("comprehensive")

So this is not a NcZarr loading issue.

I can't immediately see why the file doesn't round trip, but it is clearly no the fault of the nczarr loading, so we can ignore that in the context of this PR.

The issue of the HDF warnings still applies, but this is just noise, rather than an error (the data is still read correctly) and it is not showing in the CI tests - only locally for me. I would be interested if @trexfeathers sees similar noise when running locally?

Other than the HDF5 noise, I am happy with the the implementation of the NcZarr functionality.

[ukmo-ccbunney]

After a bit of poking, I've narrowed down the HDF noise to the netCDF4.Dataset call in _ThreadSafeWrapper.__getitem__ where the dataset is repeatedly opened/closed.

Furthermore, it is only a problem when dask is used - if i force the data to be loaded as concrete data (by upping the minimum data size threshold) then there are no HDF warnings on the call the netCDF4.Dataset. Assumedly this has something to do with dask making lots of concurrent calls to op the dataset due to multiple chunks trying to read from it?

If i set dask.config.set('scheduler', 'single-threaded') the error also goes away, which gives further evidence that it is a dask concurrency issue.

At the moment, I don't know WHY it is a problem for NcZarr and OK for NetCDF...

[ukmo-ccbunney]

Right - I've got Copilot on board now to help me with the dask interaction.

It has suggested the following test that nicely proves that the HDF5 noise issue is actually not related to dask or Iris, but is generated when any worker-thread makes a call to netcdf4.Dataset with a NCZarr URI:

>>> from concurrent.futures import ThreadPoolExecutor
>>> import netCDF4

>>> uri = "file:///tmp/4dwe.zarr#mode=nczarr,file"

>>> def f():
>>>     ds = netCDF4.Dataset(uri)
>>>     ds.close()

>>>  # run on main thread - produces no noise:
>>> f()

>>> # Run on worker thread - generates HDF errors:
>>> with ThreadPoolExecutor(max_workers=1) as ex:
>>>     ex.submit(f).result()  # worker thread

HDF5-DIAG: Error detected in HDF5 (1.14.6) thread 3:
  #000: H5F.c line 496 in H5Fis_accessible(): unable to determine if file is accessible as HDF5
    major: File accessibility
    minor: Not an HDF5 file

As the tests pass, I think we can consider this just slightly annoying "noise" as a result of Dask accessing NCZarr datasets via it's worker thread(s).

If necessary, to quieten the warnings we could recommend either:

• filtering the HDF5 warnings
• Using the single-threaded dask scheduler (although potential performance impact?)

[ukmo-ccbunney]

I did a bit of hunting around and there are several other issues on GitHub that sounds similar (although relating to writing files, I think they root cause is the same).

E.g.:

• libnetcdf 4.9.3 fails to write from multi-threaded application conda-forge/libnetcdf-feedstock#215
• Thread safety and the HDF5 error stack in 4.9.3 Unidata/netcdf-c#3193
• H5Fis_accessible(): unable to determine if file is accessible as HDF5 HDFGroup/hdf5#6498 (this gut says the error doesn't occur when he switches to a faster local filesystem).

Copilot suggests that the HDF errors are expected (because a Zarr file is not HDF5) and are also emitted on the main thread , but are suppressed. However, the error suppression is only initialised on the main thread and not inherited by any spawned threads, hence why it shows when running on on anything other than "main thread".
There is a comment in the second link listed about that back this up:

I discovered that when creating a new NetCDF file from a second thread, when the first thread has already initialized the NetCDF C library (and therefore the HDF5 library's error handling), that the HDF5's error stack messages are printed/leaked. These error messages are for expected failures/error cases of the HDF5 library.

This relates to writing, but I think the underlying issue is the same.

---

So the bottom line: I believe that the code is safe as is and the HDF errors are just leaked noise that would normally have been suppressed on the main thread. I am happy that the NCZarr implementation in Iris functions as it should and the noise is not related to anything in Iris.

@trexfeathers - I feel you should make a final call on this one. I am happy to approve this PR, but are you happy with the potential for noisy HDF errors? Do you want to add a line in the docs to suggest setting the "single-threaded" scheduler in Dask if users see these errors?