Konstntokas/support nested zarr

.github/workflows/tests.yml

@@ -16,7 +16,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10", "3.11", "3.12"]
+        python-version: ["3.12", "3.13", "3.14"]
 
     steps:
     - uses: actions/checkout@v3

.gitignore

@@ -1,5 +1,6 @@
 testdata/
 site/
+examples/outputs/
 
 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -162,5 +163,8 @@ cython_debug/
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 .idea/
 
+# Visual Studio Code
+.vscode/
+
 # Downloaded assets
 docs/assets/downloaded/

CHANGES.md

@@ -1,5 +1,10 @@
 ## Version 0.8.1 (in development)
 
+* Migrated to the Zarr v3 Python API. Supports writing Zarr v2 and v3 stores, 
+  with Zarr v2 as the default. (#111)
+* Added support for nested Zarr structures represented by `xarray.DataTree`,
+  including atomic rollback of complete DataTree slices and optional per-group
+  variable encoding configuration. (#113)
 * Replaced dev tools black and flake8 by isort and ruff. 
 * Reformatted code and fixed linter issues.
 * Added `environment.yml` to setup a conda/mamba development environment. 

README.md

@@ -28,7 +28,8 @@ awesome Python packages [xarray](https://docs.xarray.dev/) and [zarr](https://za
 The objective of `zappend` is enabling geodata scientists and developers to 
 robustly create large data cubes. The tool performs transaction-based dataset 
 appends to existing data cubes in the 
-[Zarr format](https://zarr.readthedocs.io/en/stable/spec/v2.html). If an error 
+[Zarr format](https://zarr.readthedocs.io/en/stable/). Targets use Zarr v2 by
+default and may opt into Zarr v3. If an error
 occurs during an append step — typically due to I/O problems or out-of-memory 
 conditions — `zappend` will automatically roll back the operation, ensuring that 
 the existing data cube maintains its structural integrity. The design drivers 
@@ -48,7 +49,8 @@ The `zappend` tool provides the following features:
   created, effectively preventing concurrent dataset modifications.
 * **Transaction-based dataset appends**: On failure during an append step, 
   the transaction is rolled back, so that the target dataset remains valid and 
-  preserves its integrity.
+  preserves its integrity. A nested `xarray.DataTree` slice is rolled back as
+  a whole if appending any of its groups fails.
 * **Filesystem transparency**: The target dataset may be generated and updated 
   in any writable filesystems supported by the 
   [fsspec](https://filesystem-spec.readthedocs.io/) package. 
@@ -61,7 +63,8 @@ The `zappend` tool provides the following features:
   [`zappend`](cli.md) command or from Python. When used from Python using the 
   [`zappend()`](api.md) function, slice datasets can be passed as local file 
   paths, URIs, as datasets of type 
-  [xarray.Dataset](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html), or as custom
+  [xarray.Dataset](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html),
+  [xarray.DataTree](https://docs.xarray.dev/en/stable/generated/xarray.DataTree.html), or as custom
   [slice sources](https://bcdev.github.io/zappend/guide/#slice-sources).
 
 

docs/cli.md

@@ -10,6 +10,8 @@ Usage: zappend [OPTIONS] [SLICES]...
 
   The zappend command concatenates the dataset SLICES along a given append
   dimension, e.g., "time" (the default) for geospatial satellite observations.
+  A nested Zarr slice is opened as an xarray.DataTree and its groups are
+  appended as one transaction.
   Each append step is atomic, that is, the append operation is a transaction
   that can be rolled back, in case the append operation fails. This ensures
   integrity of the target data cube given by TARGET or in CONFIG.

docs/config.md

@@ -76,6 +76,10 @@ Variable metadata.
                 Its value is `null`.
           - Disable chunking in all dimensions.
             Its value is `null`.
+      - `shards`:
+        Zarr v3 shard sizes for each dimension of the variable. Each shard size
+        must be a multiple of the corresponding chunk size. The items of the
+        list are positive integers.
       - `fill_value`:
         Must be one of the following:
           - Type _number_.
@@ -108,6 +112,30 @@ Variable metadata.
   - `attrs`:
     Arbitrary variable metadata attributes.
 
+### `group_variables`
+
+Type _object_.
+Optional variable-configuration overrides for individual `xarray.DataTree`
+groups. Keys are absolute Zarr group paths such as `"/group_0"` or `"/group_1"`; each
+value uses the same syntax as [`variables`](#variables).
+
+For a group, its configuration is merged over the global `variables`
+configuration. If a group has no entry here, normal zappend behavior applies:
+the global `variables` setting is used if present, otherwise encoding such as
+chunk sizes is taken from that group's first slice.
+
+```yaml
+group_variables:
+  "/group_0":
+    var_0:
+      encoding:
+        chunks: [31, 450, 450]
+  "/group_1":
+    var_0:
+      encoding:
+        chunks: [31, 450, 450]
+```
+
 ### `attrs`
 
 Type _object_.
@@ -134,8 +162,7 @@ Defaults to `"keep"`.
 
 ### `zarr_version`
 
-The Zarr version to be used.
-Its value is `2`.
+The Zarr version to be used. Allowed values are `2` and `3`.
 Defaults to `2`.
 
 ## Data I/O - Target
@@ -192,7 +219,7 @@ Must be one of the following:
 ### `slice_source`
 
 Type _string_.
-The fully qualified name of a class or function that receives a slice item as argument(s) and provides the slice dataset. If a class is given, it must be derived from `zappend.api.SliceSource`. If the function is a context manager, it must yield an `xarray.Dataset`. If a plain function is given, it must return any valid slice item type. Refer to the user guide for more information.
+The fully qualified name of a class or function that receives a slice item as argument(s) and provides a slice object. If a class is given, it must be derived from `zappend.api.SliceSource`. If the function is a context manager, it must yield an `xarray.Dataset` or `xarray.DataTree`. If a plain function is given, it must return any valid slice item type. Refer to the user guide for more information.
 
 ### `slice_source_kwargs`
 

docs/guide.md

@@ -31,10 +31,11 @@ one, if their paths are given as URIs prefixed with a filesystem protocol such a
 dedicated configuration settings. More on this is given
 in section [_Data I/O_](#data-io) below.
 
-!!! note "Zarr Format v2"
+!!! note "Zarr formats"
     By default, `zappend` uses the [Zarr storage specification 2](https://zarr.readthedocs.io/en/stable/spec/v2.html)
-    and has only been tested with this version. The `zarr_version` setting can be used 
-    to change it, e,g, to `3`, but any other value than `2` is currently unsupported.
+    for compatibility with existing clients. Set `zarr_version` to `3` to write
+    Zarr v3. The target's format is fixed when it is created; use the same
+    setting for all later appends.
 
 The tool takes care of generating the target dataset from slice datasets, but doesn't 
 care how the slice datasets are created. Hence, when using the Python `zappend()` 
@@ -336,6 +337,26 @@ using the `chunks` setting in the encoding of each variable.
 The value of `chunks` can also be `null`, which means no chunking is 
 desired and the variable's data array will be persisted as one block. 
 
+For Zarr v3, `shards` may additionally be configured for a variable. A shard
+contains an integer number of regular chunks, reducing object counts while
+retaining smaller read/write chunks. Each shard size must be at least its
+corresponding chunk size and an exact multiple of it. Shards are not available
+for Zarr v2.
+
+```json
+{
+    "zarr_version": 3,
+    "variables": {
+        "chl": {
+            "encoding": {
+                "chunks": [1, 512, 512],
+                "shards": [8, 2048, 2048]
+            }
+        }
+    }
+}
+```
+
 By default, the chunking of the coordinate variable corresponding to the append 
 dimension will be its dimension size in the first slice dataset. Often, the size 
 will be `1` or another small number. Since `xarray` loads coordinates eagerly 
@@ -532,6 +553,10 @@ immediately after a target dataset modification failed. The rollback include res
 all changed files and removing added files. After the rollback you can analyse what
 went wrong and try to continue appending slices at the point it failed.
 
+For an `xarray.DataTree` slice, this transaction spans all data-bearing Zarr
+groups in the tree. If appending one group fails, changes already made to the
+other groups for that same slice are rolled back too.
+
 To allow for rollbacks, a slice append operation is treated as a transaction,
 hence temporary files must be written, e.g., to record required rollback actions and to
 save backup files with the original data. The location of the temporary transaction 
@@ -610,8 +635,26 @@ to `zappend`. Slice datasets can be provided in various ways.
   of slice items or provide a Python generator that provides the slice items.
 
 Each slice item can be a local file path or URI of type `str` or `FileObj`, 
-a dataset of type `xarray.Dataset`, or a `SliceSource` object explained in more detail 
-below.
+a dataset of type `xarray.Dataset`, a nested dataset of type `xarray.DataTree`,
+or a `SliceSource` object explained in more detail below.
+
+#### Nested Zarr Groups and DataTree
+
+Nested Zarr inputs are represented by
+[`xarray.DataTree`](https://docs.xarray.dev/en/stable/generated/xarray.DataTree.html).
+Pass a DataTree directly, return one from a `SliceSource`, or pass a path to a
+nested Zarr store: zappend opens the latter as a DataTree automatically.
+
+All DataTree slices in one run must have the same group paths. DataTree and
+Dataset slices cannot be mixed in one run. Each data-bearing group is appended
+along the configured append dimension, and the complete DataTree slice is
+rolled back if any group append fails.
+
+Use `group_variables` to override variable encoding for individual absolute
+group paths. Group settings override global `variables`; groups without an
+override use global `variables`, or the encoding of their first slice when no
+global setting exists. See the [configuration reference](config.md#group_variables)
+and `examples/datatree.ipynb` for a complete example.
 
 #### Paths and URIs
 
@@ -661,14 +704,18 @@ An alternative to providing the slice dataset as path or URI is using the
 `zappend.api.FileObj` class, which combines a URI with dedicated filesystem 
 storage options.
 
-#### Dataset Objects
+#### Dataset and DataTree Objects
 
 You can also use dataset objects of type 
 [`xarray.Dataset`](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html) 
 as slice item. Such objects may originate from opening datasets from some storage, e.g., 
 `xarray.open_dataset(slice_store, ...)` or by composing, aggregating, resampling 
 slice datasets from other datasets and dataset variables.
 
+Nested inputs can instead be passed as
+[`xarray.DataTree`](https://docs.xarray.dev/en/stable/generated/xarray.DataTree.html)
+objects. The same resource-management and persistence considerations apply.
+
 !!! note "Datasets are not closed automatically"
     If you pass [`xarray.Dataset`](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html) objects to `zappend` they will not be
     automatically closed. This may become be issue, if you have many datasets 
@@ -701,8 +748,8 @@ at the cost of additional i/o. It therefore defaults to `false`.
 
 A slice source gives you full control about how a slice dataset is created, loaded, 
 or generated and how its bound resources, if any, are released. In its simplest form, 
-a slice source is a plain Python function that can take any arguments and returns 
-an `xarray.Dataset`:
+a slice source is a plain Python function that can take any arguments and returns
+an `xarray.Dataset` or `xarray.DataTree`:
 
 ```python
 import xarray as xr
@@ -740,8 +787,8 @@ def get_dataset(path: str) -> xr.Dataset:
 You can also implement your slice source as a class derived from the abstract 
 `zappend.api.SliceSource` class. Its interface methods are:
 
-* `get_dataset()`: a zero-argument method that returns the slice dataset of type 
-  `xarray.Dataset`. You must implement this abstract method. 
+* `get_dataset()`: a zero-argument method that returns the slice object of type
+  `xarray.Dataset` or `xarray.DataTree`. You must implement this abstract method.
 * `close()`: Optional method. Put your cleanup code here. 
   (in zappend < v0.7, the `close` method was called `dispose`).
 

docs/index.md

@@ -11,7 +11,8 @@ awesome Python packages [xarray](https://docs.xarray.dev/) and [zarr](https://za
 The objective of `zappend` is enabling geodata scientists and developers to 
 robustly create large data cubes. The tool performs transaction-based dataset 
 appends to existing data cubes in the 
-[Zarr format](https://zarr.readthedocs.io/en/stable/spec/v2.html). If an error 
+[Zarr format](https://zarr.readthedocs.io/en/stable/). Targets use Zarr v2 by
+default and may opt into Zarr v3. If an error
 occurs during an append step — typically due to I/O problems or out-of-memory 
 conditions — `zappend` will automatically roll back the operation, ensuring that 
 the existing data cube maintains its structural integrity. The design drivers 
@@ -39,7 +40,8 @@ The `zappend` tool provides the following features:
   [`zappend`](cli.md) command or from Python. When used from Python using the 
   [`zappend()`](api.md) function, slice datasets can be passed as local file 
   paths, URIs, as datasets of type 
-  [xarray.Dataset](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html), or as custom
+  [xarray.Dataset](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html),
+  [xarray.DataTree](https://docs.xarray.dev/en/stable/generated/xarray.DataTree.html), or as custom
   [slice sources](guide.md#slice-sources).
 
 ## How It Works
@@ -48,6 +50,9 @@ At its core, `zappend` calls the [to_zarr()](https://docs.xarray.dev/en/stable/g
 for each dataset slice it receives and either creates the target dataset if it does 
 not exist yet or updates it with the current slice, if it already exists.
 
+For nested Zarr stores, it applies the same operation to each data-bearing
+`xarray.DataTree` group as one atomic slice transaction.
+
 If there is no target dataset yet, `zappend` does the following:
 
 * get target dataset outline and encoding from configuration and first slice dataset;

docs/requirements.md

@@ -7,11 +7,15 @@ of the `zappend` tool.
 
 * Create a target Zarr dataset by appending Zarr dataset slices along a 
   given *append dimension*, usually `time`.   
+* Create a target nested Zarr store by appending compatible
+  `xarray.DataTree` slices along the append dimension of every data-bearing
+  group.
 * The tool takes care of modifying the target dataset using the slices,
   but doesn't care how the slice datasets are created.
 * Slice datasets may be given as URIs with storage options or as 
   in-memory datasets of type 
   [xarray.Dataset](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html)
+  or [xarray.DataTree](https://docs.xarray.dev/en/stable/generated/xarray.DataTree.html)
   or 
   [xcube.core.mldataset.MultiLevelDataset](https://xcube.readthedocs.io/en/latest/mldatasets.html).
 * Target and slices are allowed to live in different filesystems.

environment.yml

@@ -11,8 +11,8 @@ dependencies:
   - numcodecs
   - numpy
   - pyyaml
-  - xarray <2026.4 # constrain to <2026.4 to retain compatibility with zarr 2.
-  - zarr >=2.18,<3
+  - xarray <2026.4
+  - zarr >=3,<4
   # Dev Dependencies
   - isort
   - pytest
@@ -32,3 +32,6 @@ dependencies:
   - h5netcdf
   - s3fs
   - scipy
+  # Development Dependencies - Demos
+  - jupyterlab
+  - matplotlib