Use xbeam to append?

[langmore]

I have some xbeam pipelines that write Zarr2. I'd like to append. It's easy enough to do new_data.to_zarr(path, mode='-a', consolidated=True) to append a small amount of data...but I wonder if there's a way to use xbeam to append a much larger set.

[shoyer]

I think this would be relatively straightforward to add, and potentially quite useful.

With the current version of Xarray-Beam, this could be done as follows:

1. Expand the Zarr file on disk, by calling some variation of new_data.to_zarr(path, mode='a', append_dim='time', compute=False)
2. Get the new template template, chunks = xbeam.open_zarr(path)
3. Use xbeam.ChunkToZarr(path, template, zarr_chunks=chunks, needs_setup=False) to create a pipeline for write new chunks to disk

There are probably a few subtleties here, so it would indeed be nice to have mode and append_dim arguments on ChunksToZarr.

[langmore]

Closed by the PR demonstrating append.

[shoyer]

Let's leave this open for adding an append feature into Xarray beam itself.