diff --git a/changes/4289.doc.md b/changes/4289.doc.md new file mode 100644 index 0000000000..a5630d5c5d --- /dev/null +++ b/changes/4289.doc.md @@ -0,0 +1,3 @@ +Documentation examples use fresh in-memory stores unless they demonstrate +persistent storage. Array, group and quickstart ZIP examples use their own +temporary directories without deleting existing reader data. diff --git a/docs/quick-start.md b/docs/quick-start.md index 48d92caa4e..bd2bcc315a 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -29,7 +29,7 @@ import numpy as np # Create a 2D Zarr array z = zarr.create_array( - store="data/example-1.zarr", + store={}, shape=(100, 100), chunks=(10, 10), dtype="f4", @@ -43,7 +43,7 @@ print(z.info) Here, we created a 2D array of shape `(100, 100)`, chunked into blocks of `(10, 10)`, and filled it with random floating-point data. This array was -written to a `LocalStore` in the `data/example-1.zarr` directory. +written to a fresh in-memory store. ### Compression and Filters @@ -53,7 +53,7 @@ Zarr supports data compression and filters. For example, to use Blosc compressio # Create a 2D Zarr array with Blosc compression z = zarr.create_array( - store="data/example-2.zarr", + store={}, shape=(100, 100), chunks=(10, 10), dtype="f4", @@ -79,7 +79,7 @@ Zarr allows you to create hierarchical groups, similar to directories: ```python exec="true" session="quickstart" source="above" result="ansi" # Create nested groups and add arrays -root = zarr.group("data/example-3.zarr", overwrite=True) +root = zarr.group(store={}) foo = root.create_group(name="foo") bar = root.create_array( name="bar", shape=(100, 10), chunks=(10, 10), dtype="f4" @@ -94,7 +94,7 @@ spam[:] = np.arange(10) print(root.tree()) ``` -This creates a group hierarchy with a group (`foo`) and two arrays (`bar` and `spam`). +This creates an in-memory group hierarchy with a group (`foo`) and two arrays (`bar` and `spam`). ### Batch Hierarchy Creation @@ -104,7 +104,7 @@ Suppose we want to copy existing groups and arrays into a new storage backend: ```python exec="true" session="quickstart" source="above" result="code" # Create nested groups and add arrays -root = zarr.group("data/example-4.zarr", attributes={'name': 'root'}, overwrite=True) +root = zarr.group(store={}, attributes={'name': 'root'}) foo = root.create_group(name="foo") bar = root.create_array( name="bar", shape=(100, 10), chunks=(10, 10), dtype="f4" @@ -126,15 +126,20 @@ be done in a separate step. ## Persistent Storage -Zarr supports persistent storage to disk or cloud-compatible backends. While examples above -utilized a [`zarr.storage.LocalStore`][], a number of other storage options are available. +Zarr supports persistent storage to disk or cloud-compatible backends. While the examples +above all used in-memory stores, a number of persistent storage options are available. A single-file store can also be created using the [`zarr.storage.ZipStore`][]: ```python exec="true" session="quickstart" source="above" +from pathlib import Path +from tempfile import TemporaryDirectory + +zip_directory = TemporaryDirectory(prefix="zarr-quickstart-") +zip_path = Path(zip_directory.name) / "example.zip" # Store the array in a ZIP file -store = zarr.storage.ZipStore("data/example-5.zip", mode="w") +store = zarr.storage.ZipStore(zip_path, mode="w") z = zarr.create_array( store=store, @@ -155,12 +160,14 @@ To open an existing array from a ZIP file: ```python exec="true" session="quickstart" source="above" result="ansi" # Open the ZipStore in read-only mode -store = zarr.storage.ZipStore("data/example-5.zip", read_only=True) +store = zarr.storage.ZipStore(zip_path, read_only=True) z = zarr.open_array(store, mode='r') # read the data as a NumPy Array print(z[:]) +store.close() +zip_directory.cleanup() ``` Zarr also integrates seamlessly with cloud object storage such as Amazon S3 and Google diff --git a/docs/user-guide/arrays.md b/docs/user-guide/arrays.md index e8df79a102..26b1a161ac 100644 --- a/docs/user-guide/arrays.md +++ b/docs/user-guide/arrays.md @@ -10,16 +10,14 @@ import numpy as np ```python exec="true" session="arrays" source="above" result="ansi" import zarr -z = zarr.create_array(store="memory://arrays-demo", shape=(10000, 10000), chunks=(1000, 1000), dtype='int32') +z = zarr.create_array(store={}, shape=(10000, 10000), chunks=(1000, 1000), dtype='int32') print(z) ``` The code above creates a 2-dimensional array of 32-bit integers with 10000 rows and 10000 columns, divided into chunks where each chunk has 1000 rows and 1000 columns (and so there will be 100 chunks in total). The data is written to an -in-memory store: when `fsspec` is installed, a `memory://` URL resolves to a -[`zarr.storage.FsspecStore`][] backed by fsspec's in-memory filesystem; otherwise a -[`zarr.storage.ManagedMemoryStore`][] is used. See the [Storage guide](storage.md) +in-memory [`zarr.storage.MemoryStore`][]. See the [Storage guide](storage.md) for more details on stores, and [Persistent arrays](#persistent-arrays) for details on storing arrays in other stores. See [Data types](data_types.md) for an in-depth look at the data types supported @@ -82,12 +80,18 @@ persistence of data between sessions. To do this, we can change the store argument to point to a filesystem path: ```python exec="true" session="arrays" source="above" -z1 = zarr.create_array(store='data/example-1.zarr', shape=(10000, 10000), chunks=(1000, 1000), dtype='int32', overwrite=True) +from pathlib import Path +from tempfile import TemporaryDirectory + +array_directory = TemporaryDirectory(prefix='zarr-arrays-') +array_path = Path(array_directory.name) / 'example-1.zarr' +z1 = zarr.create_array(store=array_path, shape=(10000, 10000), chunks=(1000, 1000), dtype='int32') ``` The array above will store its configuration metadata and all compressed chunk -data in a directory called `'data/example-1.zarr'` relative to the current working -directory. The [`zarr.create_array`][] function provides a convenient way +data at `array_path` inside a temporary directory. This directory is automatically +removed when `array_directory` is cleaned up; choose your own path to retain data +between Python sessions. The [`zarr.create_array`][] function provides a convenient way to create a new persistent array or continue working with an existing array. Note, there is no need to close an array: data are automatically flushed to disk, and files are automatically closed whenever an array is modified. @@ -104,7 +108,7 @@ z1[:, 0] = np.arange(10000) Check that the data have been written and can be read again: ```python exec="true" session="arrays" source="above" result="ansi" -z2 = zarr.open_array('data/example-1.zarr', mode='r') +z2 = zarr.open_array(array_path, mode='r') print(np.all(z1[:] == z2[:])) ``` @@ -116,8 +120,10 @@ path; `zarr.save_array` accepts `mode="w"` to do so. E.g.: ```python exec="true" session="arrays" source="above" result="ansi" a = np.arange(10) -zarr.save_array('data/example-2.zarr', a, mode="w") -print(zarr.load('data/example-2.zarr')) +saved_array_path = Path(array_directory.name) / 'example-2.zarr' +zarr.save_array(saved_array_path, a, mode='w') +print(zarr.load(saved_array_path)) +array_directory.cleanup() ``` Please note that there are a number of other options for persistent array @@ -129,7 +135,7 @@ A Zarr array can be resized, which means that any of its dimensions can be increased or decreased in length. For example: ```python exec="true" session="arrays" source="above" result="ansi" -z = zarr.create_array(store='data/example-3.zarr', shape=(10000, 10000), dtype='int32', chunks=(1000, 1000), overwrite=True) +z = zarr.create_array(store={}, shape=(10000, 10000), dtype='int32', chunks=(1000, 1000)) z[:] = 42 print(f"Original shape: {z.shape}") z.resize((20000, 10000)) @@ -145,7 +151,7 @@ used to append data to any axis. E.g.: ```python exec="true" session="arrays" source="above" result="ansi" a = np.arange(10000000, dtype='int32').reshape(10000, 1000) -z = zarr.create_array(store='data/example-4.zarr', shape=a.shape, dtype=a.dtype, chunks=(1000, 100), overwrite=True) +z = zarr.create_array(store={}, shape=a.shape, dtype=a.dtype, chunks=(1000, 100)) z[:] = a print(f"Original shape: {z.shape}") z.append(a) @@ -206,7 +212,7 @@ argument accepted by all array creation functions. For example: ```python exec="true" session="arrays" source="above" result="ansi" compressors = zarr.codecs.BloscCodec(cname='zstd', clevel=3, shuffle='bitshuffle') data = np.arange(100000000, dtype='int32').reshape(10000, 10000) -z = zarr.create_array(store='data/example-5.zarr', shape=data.shape, dtype=data.dtype, chunks=(1000, 1000), compressors=compressors, overwrite=True) +z = zarr.create_array(store={}, shape=data.shape, dtype=data.dtype, chunks=(1000, 1000), compressors=compressors) z[:] = data print(z.compressors) ``` @@ -241,7 +247,7 @@ compressor. To create an array without any compression, set `compressors=None`: ```python exec="true" session="arrays" source="above" result="ansi" -z_no_compress = zarr.create_array(store='data/example-uncompressed.zarr', shape=(10000, 10000), chunks=(1000, 1000), dtype='int32', compressors=None, overwrite=True) +z_no_compress = zarr.create_array(store={}, shape=(10000, 10000), chunks=(1000, 1000), dtype='int32', compressors=None) print(f"Compressors: {z_no_compress.compressors}") ``` @@ -250,7 +256,7 @@ here is an array using Gzip compression, level 1: ```python exec="true" session="arrays" source="above" result="ansi" data = np.arange(100000000, dtype='int32').reshape(10000, 10000) -z = zarr.create_array(store='data/example-6.zarr', shape=data.shape, dtype=data.dtype, chunks=(1000, 1000), compressors=zarr.codecs.GzipCodec(level=1), overwrite=True) +z = zarr.create_array(store={}, shape=data.shape, dtype=data.dtype, chunks=(1000, 1000), compressors=zarr.codecs.GzipCodec(level=1)) z[:] = data print(f"Compressors: {z.compressors}") ``` @@ -265,7 +271,7 @@ from zarr.codecs.numcodecs import LZMA lzma_filters = [dict(id=lzma.FILTER_DELTA, dist=4), dict(id=lzma.FILTER_LZMA2, preset=1)] compressors = LZMA(filters=lzma_filters) data = np.arange(100000000, dtype='int32').reshape(10000, 10000) -z = zarr.create_array(store='data/example-7.zarr', shape=data.shape, dtype=data.dtype, chunks=(1000, 1000), compressors=compressors, overwrite=True) +z = zarr.create_array(store={}, shape=data.shape, dtype=data.dtype, chunks=(1000, 1000), compressors=compressors) print(f"Compressors: {z.compressors}") ``` @@ -290,7 +296,7 @@ from zarr.codecs.numcodecs import Delta filters = [Delta(dtype='int32')] compressors = zarr.codecs.BloscCodec(cname='zstd', clevel=1, shuffle='shuffle') data = np.arange(100000000, dtype='int32').reshape(10000, 10000) -z = zarr.create_array(store='data/example-9.zarr', shape=data.shape, dtype=data.dtype, chunks=(1000, 1000), filters=filters, compressors=compressors, overwrite=True) +z = zarr.create_array(store={}, shape=data.shape, dtype=data.dtype, chunks=(1000, 1000), filters=filters, compressors=compressors) print(z.info_complete()) ``` @@ -315,7 +321,7 @@ coordinates. E.g.: ```python exec="true" session="arrays" source="above" result="ansi" data = np.arange(10) ** 2 -z = zarr.create_array(store='data/example-10.zarr', shape=data.shape, dtype=data.dtype, overwrite=True) +z = zarr.create_array(store={}, shape=data.shape, dtype=data.dtype) z[:] = data print(z[:]) print(z.get_coordinate_selection([2, 5])) @@ -333,7 +339,7 @@ e.g.: ```python exec="true" session="arrays" source="above" result="ansi" data = np.arange(15).reshape(3, 5) -z = zarr.create_array(store='data/example-11.zarr', shape=data.shape, dtype=data.dtype, overwrite=True) +z = zarr.create_array(store={}, shape=data.shape, dtype=data.dtype) z[:] = data print(z[:]) ``` @@ -377,7 +383,7 @@ Items can also be extracted by providing a Boolean mask. E.g.: ```python exec="true" session="arrays" source="above" result="ansi" data = np.arange(10) ** 2 -z = zarr.create_array(store='data/example-12.zarr', shape=data.shape, dtype=data.dtype, overwrite=True) +z = zarr.create_array(store={}, shape=data.shape, dtype=data.dtype) z[:] = data print(z[:]) ``` @@ -398,7 +404,7 @@ Here's a multidimensional example: ```python exec="true" session="arrays" source="above" result="ansi" data = np.arange(15).reshape(3, 5) -z = zarr.create_array(store='data/example-13.zarr', shape=data.shape, dtype=data.dtype, overwrite=True) +z = zarr.create_array(store={}, shape=data.shape, dtype=data.dtype) z[:] = data print(z[:]) ``` @@ -441,7 +447,7 @@ example, this allows selecting a subset of rows and/or columns from a ```python exec="true" session="arrays" source="above" result="ansi" data = np.arange(15).reshape(3, 5) -z = zarr.create_array(store='data/example-14.zarr', shape=data.shape, dtype=data.dtype, overwrite=True) +z = zarr.create_array(store={}, shape=data.shape, dtype=data.dtype) z[:] = data print(z[:]) ``` @@ -469,7 +475,7 @@ For convenience, the orthogonal indexing functionality is also available via the ```python exec="true" session="arrays" source="above" result="ansi" data = np.arange(15).reshape(3, 5) -z = zarr.create_array(store='data/example-15.zarr', shape=data.shape, dtype=data.dtype, overwrite=True) +z = zarr.create_array(store={}, shape=data.shape, dtype=data.dtype) z[:] = data print(z.oindex[[0, 2], :]) # select first and third rows ``` @@ -495,7 +501,7 @@ orthogonal indexing is also available directly on the array: ```python exec="true" session="arrays" source="above" result="ansi" data = np.arange(15).reshape(3, 5) -z = zarr.create_array(store='data/example-16.zarr', shape=data.shape, dtype=data.dtype, overwrite=True) +z = zarr.create_array(store={}, shape=data.shape, dtype=data.dtype) z[:] = data print(np.all(z.oindex[[0, 2], :] == z[[0, 2], :])) ``` @@ -508,7 +514,7 @@ a subset of chunk aligned rows and/or columns from a 2-dimensional array. E.g.: ```python exec="true" session="arrays" source="above" data = np.arange(100).reshape(10, 10) -z = zarr.create_array(store='data/example-17.zarr', shape=data.shape, dtype=data.dtype, chunks=(3, 3), overwrite=True) +z = zarr.create_array(store={}, shape=data.shape, dtype=data.dtype, chunks=(3, 3)) z[:] = data ``` @@ -541,7 +547,7 @@ print(z.blocks[0, 1:3]) Data can also be modified. Let's start by a simple 2D array: ```python exec="true" session="arrays" source="above" -z = zarr.create_array(store='data/example-18.zarr', shape=(6, 6), dtype=int, chunks=(2, 2), overwrite=True) +z = zarr.create_array(store={}, shape=(6, 6), dtype=int, chunks=(2, 2)) ``` Set data for a selection of items: @@ -584,7 +590,7 @@ performance guide. Sharded arrays can be created by providing the `shards` parameter to [`zarr.create_array`][]. ```python exec="true" session="arrays" source="above" result="ansi" -a = zarr.create_array('data/example-20.zarr', shape=(10000, 10000), shards=(1000, 1000), chunks=(100, 100), dtype='uint8', overwrite=True) +a = zarr.create_array(store={}, shape=(10000, 10000), shards=(1000, 1000), chunks=(100, 100), dtype='uint8') a[:] = (np.arange(10000 * 10000) % 256).astype('uint8').reshape(10000, 10000) print(a.info_complete()) ``` diff --git a/docs/user-guide/cli.md b/docs/user-guide/cli.md index 77f50f5eaf..9afdeb4f68 100644 --- a/docs/user-guide/cli.md +++ b/docs/user-guide/cli.md @@ -62,9 +62,10 @@ To open the array/group using the new metadata use: import zarr # create a small array to open (stands in for the migrated store) -zarr.create_array("data/cli-demo.zarr", shape=(4, 4), chunks=(2, 2), dtype="i4", overwrite=True) +store = {} +zarr.create_array(store=store, shape=(4, 4), chunks=(2, 2), dtype="i4") -zarr_with_v3_metadata = zarr.open("data/cli-demo.zarr", zarr_format=3) +zarr_with_v3_metadata = zarr.open(store=store, zarr_format=3) ``` Once you are happy with the conversion, you can run the following to remove the old v2 metadata: diff --git a/docs/user-guide/groups.md b/docs/user-guide/groups.md index ada5c5ec84..7cc712368f 100644 --- a/docs/user-guide/groups.md +++ b/docs/user-guide/groups.md @@ -8,7 +8,7 @@ To create a group, use the [`zarr.create_group`][] function: ```python exec="true" session="groups" source="above" result="ansi" import zarr -root = zarr.create_group(store="memory://groups-demo") +root = zarr.create_group(store={}) print(root) ``` @@ -83,10 +83,17 @@ print(root.tree()) The [`zarr.open_group`][] function provides a convenient way to create or re-open a group stored in a directory on the file-system, with sub-groups stored in -sub-directories, e.g.: +sub-directories. This example uses a temporary directory that is automatically +removed when `group_directory` is cleaned up; use a path of your choice to retain +data between Python sessions: ```python exec="true" session="groups" source="above" result="ansi" -root = zarr.open_group('data/group.zarr', mode='w') +from pathlib import Path +from tempfile import TemporaryDirectory + +group_directory = TemporaryDirectory(prefix='zarr-groups-') +group_path = Path(group_directory.name) / 'group.zarr' +root = zarr.open_group(group_path, mode='w') print(root) ``` @@ -147,8 +154,10 @@ from zarr.storage import LocalStore from pprint import pprint import io +# Keep this LocalStore example in the session's own temporary directory. +hierarchy_path = Path(group_directory.name) / 'hierarchy.zarr' node_spec = {'a/b/c': GroupMetadata()} -nodes_created = dict(create_hierarchy(store=LocalStore(root='data'), nodes=node_spec, overwrite=True)) +nodes_created = dict(create_hierarchy(store=LocalStore(root=hierarchy_path), nodes=node_spec, overwrite=True)) # Report nodes (pprint is used for cleaner rendering in the docs) output = io.StringIO() pprint(nodes_created, stream=output, width=60) @@ -171,7 +180,7 @@ Diagnostic information about arrays and groups is available via the `info` property. E.g.: ```python exec="true" session="groups" source="above" result="ansi" -root = zarr.group(store="memory://diagnostics-demo") +root = zarr.group(store={}) foo = root.create_group('foo') bar = foo.create_array(name='bar', shape=1000000, chunks=100000, dtype='int64') bar[:] = 42 @@ -196,4 +205,5 @@ Groups also have the [`zarr.Group.tree`][] method, e.g.: ```python exec="true" session="groups" source="above" result="ansi" print(root.tree()) +group_directory.cleanup() ``` diff --git a/docs/user-guide/performance.md b/docs/user-guide/performance.md index d1ff0e1b75..d884b6cfb9 100644 --- a/docs/user-guide/performance.md +++ b/docs/user-guide/performance.md @@ -159,7 +159,7 @@ def timed_write(write_empty_chunks): data = np.random.randint(0, 255, shape) dtype = 'uint8' arr = zarr.create_array( - f'data/example-{write_empty_chunks}.zarr', + store={}, shape=shape, chunks=chunks, dtype=dtype,