From 148ba7c8936d451f067fd5000b8b3c6435a9020c Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Sat, 10 Oct 2026 12:11:45 +0200 Subject: [PATCH 1/3] docs: list non-conformant and unregistered metadata in a checked ledger Add `zarr.core.metadata.ledger.LEDGER`, one entry per kind of stored metadata that zarr reads or writes although it does not conform to a Zarr specification or uses a name the zarr-extensions registry does not define: which software wrote it (including which zarr releases), whether zarr still writes it, and how zarr reads it now. A new Metadata compatibility docs page is generated from it by the mkdocs hook. Every site that reads or writes such metadata carries a `# NON-CONFORMANT: ` or `# UNREGISTERED: ` comment. tests/test_ledger.py fails if a repair in `ARRAY_REPAIRS` is in no entry, if a marker names no entry or has the wrong tag, if an entry has no marker, or if a `v3_unstable_dtype_warning` call is unmarked. The repair module no longer claims to be the only lenient reader of stored metadata. Assisted-by: ClaudeCode:claude-opus-5-5 --- docs/contributing.md | 2 + docs/metadata-compatibility.md | 21 ++ mkdocs.yml | 1 + mkdocs_hooks.py | 33 ++++ src/zarr/api/asynchronous.py | 1 + src/zarr/codecs/bytes.py | 1 + src/zarr/codecs/numcodecs/_codecs.py | 2 + src/zarr/core/_json.py | 2 + src/zarr/core/dtype/npy/bytes.py | 4 + src/zarr/core/dtype/npy/float.py | 2 + src/zarr/core/dtype/npy/int.py | 1 + src/zarr/core/dtype/npy/string.py | 1 + src/zarr/core/dtype/npy/structured.py | 3 + src/zarr/core/group.py | 5 + src/zarr/core/metadata/ledger.py | 269 ++++++++++++++++++++++++++ src/zarr/core/metadata/repair.py | 20 +- src/zarr/core/metadata/v2.py | 1 + src/zarr/metadata/migrate_v3.py | 1 + tests/test_ledger.py | 89 +++++++++ 19 files changed, 454 insertions(+), 5 deletions(-) create mode 100644 docs/metadata-compatibility.md create mode 100644 src/zarr/core/metadata/ledger.py create mode 100644 tests/test_ledger.py diff --git a/docs/contributing.md b/docs/contributing.md index 6054e86e97..b5f04a5d2f 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -455,6 +455,8 @@ The Zarr library is an implementation of a file format standard defined external If an existing Zarr format version changes, or a new version of the Zarr format is released, then the Zarr library will generally require changes. It is very likely that a new Zarr format will require extensive breaking changes to the Zarr library, and so support for a new Zarr format in the Zarr library will almost certainly come in a new `major` release. When the Zarr library adds support for a new Zarr format, there may be a period of accelerated changes as developers refine newly added APIs and deprecate old APIs. In such a transitional phase breaking changes may be more frequent than usual. +The Zarr library sometimes reads or writes stored metadata that does not conform to a Zarr specification, or that uses a name the [zarr-extensions registry](https://github.com/zarr-developers/zarr-extensions) does not define. It reads such metadata when software wrote it and users hold data stored with it, and it may write such metadata when no conformant form exists yet. Each kind of such metadata is an entry in `zarr.core.metadata.ledger.LEDGER`, which the [Metadata compatibility](metadata-compatibility.md) page lists. The entry records which software wrote the metadata, including which releases of this library, and how the library reads it now. The code that reads or writes it carries a `# NON-CONFORMANT: ` or `# UNREGISTERED: ` comment naming the entry, and `tests/test_ledger.py` fails if the code and the ledger disagree. A warning about such metadata describes the metadata and what to do about it; it does not name the release that wrote it. + ### Deprecation policy Our versioning policy (above) commits us to minimizing the effort users spend upgrading. A deprecation cycle is the main tool we use to honor that commitment. Rather than removing or changing public behavior abruptly, we first ship a release that keeps the old behavior working while warning that it is going away, and only remove it in a later release. This section defines when a deprecation cycle is required, how to decide whether a removal is worth it, and the concrete steps a deprecation must follow. diff --git a/docs/metadata-compatibility.md b/docs/metadata-compatibility.md new file mode 100644 index 0000000000..4b8056e9be --- /dev/null +++ b/docs/metadata-compatibility.md @@ -0,0 +1,21 @@ +# Metadata compatibility + +This page lists every kind of stored metadata that zarr-python reads or writes +although it does not conform to the +[Zarr specifications](https://zarr-specs.readthedocs.io/) (non-conformant), or +uses a name that the +[zarr-extensions registry](https://github.com/zarr-developers/zarr-extensions) +does not define (unregistered). Other Zarr implementations may refuse to read +such metadata, or read it differently. + +For each kind of metadata, the list says which software wrote it and how the +current release of `zarr` reads it. Some are read by a repair: zarr reads the +metadata as the valid metadata it was meant to be, and stores that valid +metadata the next time it writes the array's metadata. Where a repair needs you +to act, opening the array warns and says what to do. + +The list is generated from `zarr.core.metadata.ledger` when the documentation +is built. A test checks that ledger against the code that reads and writes +these kinds of metadata, so the list stays complete. + + diff --git a/mkdocs.yml b/mkdocs.yml index 4d06701a87..cc12bfb9a8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -102,6 +102,7 @@ nav: - 'zarr-metadata ↪': https://zarr.readthedocs.io/projects/zarr-metadata/ - 'zarr-indexing ↪': https://zarr.readthedocs.io/projects/zarr-indexing/ - release-notes.md + - metadata-compatibility.md - roadmap.md - contributing.md - Blog: diff --git a/mkdocs_hooks.py b/mkdocs_hooks.py index 88368926b9..e13c17b1cd 100644 --- a/mkdocs_hooks.py +++ b/mkdocs_hooks.py @@ -19,6 +19,8 @@ if TYPE_CHECKING: from markdown import Markdown from mkdocs.config.defaults import MkDocsConfig + from mkdocs.structure.files import Files + from mkdocs.structure.pages import Page # Mirrors markdown_exec's _to_bool: everything but these means "true". _FALSY = {"", "no", "off", "false", "0"} @@ -80,3 +82,34 @@ def on_config(config: MkDocsConfig) -> MkDocsConfig: } ) return config + + +# The metadata compatibility page holds this marker where the list of +# non-conformant and unregistered metadata goes. It is generated at build time from +# ``zarr.core.metadata.ledger``, which ``tests/test_ledger.py`` checks against the +# source, so the page lists what the code reads and writes. +_LEDGER_MARKER = "" + + +def on_page_markdown(markdown: str, page: Page, config: MkDocsConfig, files: Files) -> str: + if _LEDGER_MARKER not in markdown: + return markdown + return markdown.replace(_LEDGER_MARKER, ledger_markdown()) + + +def ledger_markdown() -> str: + """Render ``zarr.core.metadata.ledger.LEDGER`` as Markdown: the metadata the current + zarr still writes, then the metadata it only reads.""" + from zarr.core.metadata.ledger import LEDGER + + sections = [] + for heading, still_written in (("Still written", True), ("No longer written", False)): + items = [ + f"- **{entry.title}** ({entry.conformance})\n" + f" - Written by: {entry.written_by}\n" + f" - Current behavior: {entry.reading}" + for entry in LEDGER.values() + if entry.still_written is still_written + ] + sections.append(f"## {heading}\n\n" + "\n".join(items)) + return "\n\n".join(sections) diff --git a/src/zarr/api/asynchronous.py b/src/zarr/api/asynchronous.py index 5725b3d13a..4e55201fea 100644 --- a/src/zarr/api/asynchronous.py +++ b/src/zarr/api/asynchronous.py @@ -256,6 +256,7 @@ async def consolidate_metadata( v = dataclasses.replace(v, consolidated_metadata=ConsolidatedMetadata(metadata={})) members_metadata[k] = v + # UNREGISTERED: v3-consolidated-metadata if any(m.zarr_format == 3 for m in members_metadata.values()): warnings.warn( "Consolidated metadata is currently not part in the Zarr format 3 specification. It " diff --git a/src/zarr/codecs/bytes.py b/src/zarr/codecs/bytes.py index 622161c458..aee32d8d32 100644 --- a/src/zarr/codecs/bytes.py +++ b/src/zarr/codecs/bytes.py @@ -73,6 +73,7 @@ def to_dict(self) -> dict[str, JSON]: def evolve_from_array_spec(self, array_spec: ArraySpec) -> Self: if isinstance(array_spec.dtype, Struct): + # NON-CONFORMANT: struct-bytes-codec-without-endian if array_spec.dtype.has_multi_byte_fields(): if self.endian is None: warnings.warn( diff --git a/src/zarr/codecs/numcodecs/_codecs.py b/src/zarr/codecs/numcodecs/_codecs.py index c980743166..085d791e51 100644 --- a/src/zarr/codecs/numcodecs/_codecs.py +++ b/src/zarr/codecs/numcodecs/_codecs.py @@ -60,6 +60,7 @@ def _expect_name_prefix(codec_name: str) -> str: return codec_name.removeprefix(CODEC_PREFIX) +# UNREGISTERED: numcodecs-codecs def _parse_codec_configuration(data: dict[str, JSON]) -> dict[str, JSON]: parsed_name, parsed_configuration = parse_named_configuration(data) if not parsed_name.startswith(CODEC_PREFIX): @@ -112,6 +113,7 @@ def from_dict(cls, data: dict[str, JSON]) -> Self: codec_config = _parse_codec_configuration(data) return cls(**codec_config) + # UNREGISTERED: numcodecs-codecs def to_dict(self) -> dict[str, JSON]: codec_config = self.codec_config.copy() codec_config.pop("id", None) diff --git a/src/zarr/core/_json.py b/src/zarr/core/_json.py index 66f6236f44..f2740b4c00 100644 --- a/src/zarr/core/_json.py +++ b/src/zarr/core/_json.py @@ -38,6 +38,7 @@ def buffer_to_json(buffer: Buffer) -> JSON: """Parse the contents of a `Buffer` as a JSON value.""" # json.loads is typed as returning Any; the result is by definition JSON. + # NON-CONFORMANT: json-nan-tokens return cast("JSON", json.loads(buffer.to_bytes())) @@ -96,6 +97,7 @@ def json_to_buffer( """ if prototype is None: prototype = default_buffer_prototype() + # NON-CONFORMANT: json-nan-tokens return prototype.buffer.from_bytes(json.dumps(obj, indent=indent, allow_nan=allow_nan).encode()) diff --git a/src/zarr/core/dtype/npy/bytes.py b/src/zarr/core/dtype/npy/bytes.py index daeb105819..53bbfca748 100644 --- a/src/zarr/core/dtype/npy/bytes.py +++ b/src/zarr/core/dtype/npy/bytes.py @@ -383,6 +383,7 @@ def to_json( if zarr_format == 2: return {"name": self.to_native_dtype().str, "object_codec_id": None} elif zarr_format == 3: + # UNREGISTERED: null-terminated-bytes v3_unstable_dtype_warning(self) return { "name": self._zarr_v3_name, @@ -781,6 +782,7 @@ def to_json(self, zarr_format: ZarrFormat) -> RawBytesJSON_V2 | RawBytesJSON_V3: if zarr_format == 2: return {"name": self.to_native_dtype().str, "object_codec_id": None} elif zarr_format == 3: + # UNREGISTERED: raw-bytes v3_unstable_dtype_warning(self) return {"name": self._zarr_v3_name, "configuration": {"length_bytes": self.length}} raise ValueError(f"zarr_format must be 2 or 3, got {zarr_format}") # pragma: no cover @@ -1045,6 +1047,7 @@ def _check_json_v3(cls, data: DTypeJSON) -> TypeGuard[Literal["variable_length_b True if the input is a valid representation of this class in Zarr V3, False otherwise. """ + # UNREGISTERED: variable-length-bytes return data in (cls._zarr_v3_name, "bytes") @classmethod @@ -1139,6 +1142,7 @@ def to_json( if zarr_format == 2: return {"name": "|O", "object_codec_id": self.object_codec_id} elif zarr_format == 3: + # UNREGISTERED: variable-length-bytes v3_unstable_dtype_warning(self) return self._zarr_v3_name raise ValueError(f"zarr_format must be 2 or 3, got {zarr_format}") # pragma: no cover diff --git a/src/zarr/core/dtype/npy/float.py b/src/zarr/core/dtype/npy/float.py index 0fbd0c3e45..996ded6211 100644 --- a/src/zarr/core/dtype/npy/float.py +++ b/src/zarr/core/dtype/npy/float.py @@ -278,6 +278,7 @@ def from_json_scalar(self, data: JSON, *, zarr_format: ZarrFormat) -> Scalar: if zarr_format == 2: if check_json_float_v2(data): return self._cast_scalar_unchecked(float_from_json_v2(data)) + # NON-CONFORMANT: numeric-fill-value-spellings elif check_json_floatish_str(data): return self._cast_scalar_unchecked(float(data)) else: @@ -287,6 +288,7 @@ def from_json_scalar(self, data: JSON, *, zarr_format: ZarrFormat) -> Scalar: elif zarr_format == 3: if check_json_float_v3(data): return self._cast_scalar_unchecked(float_from_json_v3(data)) + # NON-CONFORMANT: numeric-fill-value-spellings elif check_json_floatish_str(data): return self._cast_scalar_unchecked(float(data)) else: diff --git a/src/zarr/core/dtype/npy/int.py b/src/zarr/core/dtype/npy/int.py index b7198fcb0e..39ba3cf091 100644 --- a/src/zarr/core/dtype/npy/int.py +++ b/src/zarr/core/dtype/npy/int.py @@ -205,6 +205,7 @@ def from_json_scalar(self, data: JSON, *, zarr_format: ZarrFormat) -> Scalar: """ if check_json_int(data): return self._cast_scalar_unchecked(data) + # NON-CONFORMANT: numeric-fill-value-spellings if check_json_intish_float(data): return self._cast_scalar_unchecked(int(data)) diff --git a/src/zarr/core/dtype/npy/string.py b/src/zarr/core/dtype/npy/string.py index 18a3860132..c145762ca9 100644 --- a/src/zarr/core/dtype/npy/string.py +++ b/src/zarr/core/dtype/npy/string.py @@ -405,6 +405,7 @@ def item_size(self) -> int: return self.length * self.code_point_bytes +# NON-CONFORMANT: string-fill-value-number def check_vlen_string_json_scalar(data: object) -> TypeGuard[int | str | float]: """ Check if the input is a valid JSON scalar for a variable-length string. diff --git a/src/zarr/core/dtype/npy/structured.py b/src/zarr/core/dtype/npy/structured.py index f421835309..1038805acc 100644 --- a/src/zarr/core/dtype/npy/structured.py +++ b/src/zarr/core/dtype/npy/structured.py @@ -465,6 +465,7 @@ def to_json(self, zarr_format: ZarrFormat) -> StructuredJSON_V2 | StructuredJSON ] return {"name": fields, "object_codec_id": None} elif zarr_format == 3: + # NON-CONFORMANT: structured-data-type v3_unstable_dtype_warning(self) fields = [ [f_name, f_dtype.to_json(zarr_format=zarr_format)] # type: ignore[list-item] @@ -706,6 +707,7 @@ def _check_json_v3(cls, data: DTypeJSON) -> TypeGuard[StructJSON_V3]: # type: i return ( isinstance(data, dict) and set(data.keys()) == {"name", "configuration"} + # NON-CONFORMANT: structured-data-type and data["name"] in ("struct", "structured") and isinstance(data["configuration"], dict) and set(data["configuration"].keys()) == {"fields"} @@ -791,6 +793,7 @@ def from_json_scalar(self, data: JSON, *, zarr_format: ZarrFormat) -> np.void: else: field_values.append(field_dtype.default_scalar()) return self._cast_scalar_unchecked(tuple(field_values)) + # NON-CONFORMANT: structured-data-type elif check_json_str(data): as_bytes = bytes_from_json(data, zarr_format=zarr_format) return self._scalar_bytes_to_void(as_bytes, zarr_format) diff --git a/src/zarr/core/group.py b/src/zarr/core/group.py index 7792fa58de..6abe954327 100644 --- a/src/zarr/core/group.py +++ b/src/zarr/core/group.py @@ -423,6 +423,7 @@ def to_buffer_dict(self, prototype: BufferPrototype) -> dict[str, Buffer]: # it's an array d[f"{k}/{ZARRAY_JSON}"] = v else: + # NON-CONFORMANT: v2-consolidated-group-entries d[f"{k}/{ZGROUP_JSON}"] = { "zarr_format": self.zarr_format, "consolidated_metadata": { @@ -471,6 +472,9 @@ def from_dict(cls, data: dict[str, Any], *, path: str | None = None) -> GroupMet raise NodeTypeValidationError( f"Invalid value for 'node_type'. Expected 'group' or None. Got {node_type!r}." ) + # NON-CONFORMANT: group-consolidated-metadata-null + # UNREGISTERED: v3-consolidated-metadata + # NON-CONFORMANT: v2-consolidated-group-entries consolidated_metadata = data.pop("consolidated_metadata", None) if consolidated_metadata: data["consolidated_metadata"] = ConsolidatedMetadata.from_dict( @@ -495,6 +499,7 @@ def to_dict(self) -> dict[str, Any]: result = asdict(replace(self, consolidated_metadata=None)) result.update(result.pop("extra_fields")) if self.consolidated_metadata is not None: + # UNREGISTERED: v3-consolidated-metadata result["consolidated_metadata"] = self.consolidated_metadata.to_dict() else: # Leave consolidated metadata unset if it's None diff --git a/src/zarr/core/metadata/ledger.py b/src/zarr/core/metadata/ledger.py new file mode 100644 index 0000000000..44da2d9402 --- /dev/null +++ b/src/zarr/core/metadata/ledger.py @@ -0,0 +1,269 @@ +"""The ledger of stored metadata that does not conform to the Zarr specifications. + +Each entry is one kind of stored metadata that is not conformant (the specification +forbids it) or not registered (it uses a name the zarr-extensions registry does not +define), which zarr-python writes or reads. An entry records which software wrote it, +whether zarr still writes it, and how zarr reads it now. The Metadata compatibility page +of the documentation lists this ledger. + +Every place in `src/zarr` that reads or writes such metadata carries a comment naming +its entry, `# NON-CONFORMANT: ` or `# UNREGISTERED: `, so a reader of the code +finds the entry and the entry can be checked against the code: `tests/test_ledger.py` +fails if a marker names no entry, if an entry has no marker, or if a repair in +`zarr.core.metadata.repair.ARRAY_REPAIRS` is in no entry. + +The ledger records zarr releases; the warnings zarr gives describe the metadata only. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import TYPE_CHECKING, Final, Literal + +from zarr.core.metadata import repair + +if TYPE_CHECKING: + from collections.abc import Mapping + + from zarr.core.metadata.repair import Repair + +type Conformance = Literal["non-conformant", "unregistered"] + + +@dataclass(frozen=True, kw_only=True) +class Entry: + """One kind of non-conformant or unregistered stored metadata. Text is Markdown.""" + + title: str + """What the stored metadata holds.""" + conformance: Conformance + """`non-conformant` if a specification forbids it, `unregistered` if it uses a name the + zarr-extensions registry does not define.""" + written_by: str + """Which software wrote it, and which releases.""" + still_written: bool + """Whether the current zarr writes it.""" + reading: str + """How the current zarr reads it, and whether it warns.""" + repairs: tuple[Repair, ...] = () + """The repairs in `zarr.core.metadata.repair` that read it.""" + + +MARKERS: Final[Mapping[Conformance, str]] = { + "non-conformant": "NON-CONFORMANT", + "unregistered": "UNREGISTERED", +} +"""The comment tag that marks the code of an entry of each conformance.""" + + +LEDGER: Final[Mapping[str, Entry]] = { + "chunk-size-zero": Entry( + title=( + "A regular chunk size of `0` or `false` (Zarr format 2 `chunks`, Zarr format 3 " + "`regular` `chunk_shape`)" + ), + conformance="non-conformant", + written_by=( + "zarr 3.0 to 3.3 for a Zarr format 2 array created with a zero-length axis, and " + "zarr 3.0 to 3.1 for a Zarr format 3 one; zarr 2.18.7 to 3.2.1 for an explicit chunk " + "size of `0` or `False` on an axis of any length" + ), + still_written=False, + reading=( + "Repaired: read as chunk size 1 (the inner chunk size of a sharded array). " + "Silent on a zero-length axis; on an axis of positive length it warns that the " + "array holds only its fill value. Writing chunks first stores the repaired " + "metadata." + ), + repairs=(repair._invalid_chunk_sizes_v2, repair._invalid_chunk_sizes_v3), + ), + "chunk-size-true": Entry( + title=( + "A chunk size of JSON `true`, in a regular chunk shape, in the inner chunk shape " + "of a sharding codec, or as a rectilinear chunk edge length" + ), + conformance="non-conformant", + written_by="zarr 3.0 and 3.2 for a chunk size of `True`", + still_written=False, + reading="Repaired silently: read as 1.", + repairs=( + repair._invalid_chunk_sizes_v2, + repair._invalid_chunk_sizes_v3, + repair._invalid_inner_chunk_sizes_v3, + repair._invalid_edge_lengths_v3, + ), + ), + "regular-grid-edge-lists": Entry( + title=( + "A `regular` chunk grid whose `chunk_shape` mixes chunk sizes with lists of chunk " + "edge lengths, such as `[2, [5, 10, 5]]`" + ), + conformance="non-conformant", + written_by="zarr 3.2.0 and 3.2.1 for `chunks=(2, (5, 10, 5))`", + still_written=False, + reading=( + "Repaired with a warning: read as the rectilinear chunk grid it describes, " + "without the `array.rectilinear_chunks` flag. Re-saving the metadata, or " + "writing chunks, stores that rectilinear grid, which requires the flag." + ), + repairs=(repair._invalid_chunk_sizes_v3,), + ), + "rectilinear-float-edges": Entry( + title="A rectilinear chunk edge length that is an integral JSON float, such as `4.0`", + conformance="non-conformant", + written_by="zarr 3.2 for float chunk edges", + still_written=False, + reading=( + "Repaired silently: read as the integer it equals. A float anywhere else in a " + "chunk grid is rejected." + ), + repairs=(repair._invalid_edge_lengths_v3,), + ), + "v2-empty-filters": Entry( + title="Zarr format 2 `filters: []`", + conformance="non-conformant", + written_by="zarr 3.0.0 to 3.0.3 for `filters=[]`", + still_written=False, + reading="Read as `filters: null`, with a warning.", + ), + "group-consolidated-metadata-null": Entry( + title='A Zarr format 3 group with `"consolidated_metadata": null`', + conformance="non-conformant", + written_by="zarr 3.0.0 to 3.1.3, for every Zarr format 3 group", + still_written=False, + reading="Read silently as a group with no consolidated metadata.", + ), + "struct-bytes-codec-without-endian": Entry( + title=( + 'A `bytes` codec with no `endian` (`{"name": "bytes"}`) on an array whose ' + "structured data type has multi-byte fields" + ), + conformance="non-conformant", + written_by="zarr 3.1.x", + still_written=False, + reading="Read as little-endian, with a warning.", + ), + "structured-data-type": Entry( + title=( + 'The `structured` data type: `"name": "structured"`, fields as `[name, data_type]` ' + "pairs, and a base64 fill value" + ), + conformance="non-conformant", + written_by=( + "zarr 3.1.x for every structured data type; the current zarr writes it only for " + "a `zarr.dtype.Structured` constructed explicitly" + ), + still_written=True, + reading=( + "Read silently as the `struct` data type, which is how the zarr-extensions " + "registry defines it: a legacy alias that implementations may read but must not " + "write. Written with an `UnstableSpecificationWarning`." + ), + ), + "variable-length-bytes": Entry( + title='The `"variable_length_bytes"` data type', + conformance="unregistered", + written_by=( + "zarr 3.1.0 and later, for variable-length bytes, where zarr 3.0 wrote the " + "registered name `bytes`" + ), + still_written=True, + reading=( + "Read, and so is the registered name `bytes` with a base64 fill value; the " + "fill value `[]` that zarr 3.0 wrote is rejected. Written with an " + "`UnstableSpecificationWarning`; re-saving metadata that names `bytes` stores " + "`variable_length_bytes`." + ), + ), + "null-terminated-bytes": Entry( + title='The `"null_terminated_bytes"` data type, for NumPy `S` data types', + conformance="unregistered", + written_by="zarr 3.1.0 and later", + still_written=True, + reading="Read. Written with an `UnstableSpecificationWarning`.", + ), + "raw-bytes": Entry( + title='The `"raw_bytes"` data type, for NumPy `V` data types', + conformance="unregistered", + written_by="zarr 3.1.0 and later", + still_written=True, + reading="Read. Written with an `UnstableSpecificationWarning`.", + ), + "numcodecs-codecs": Entry( + title=( + "Codecs named `numcodecs.` (`numcodecs.delta`, `numcodecs.bz2`, ...), with " + "the numcodecs configuration of the codec" + ), + conformance="unregistered", + written_by=( + "zarr 3.1.3 and later, through `zarr.codecs.numcodecs`, which the Zarr format 3 " + "conversion of `zarr migrate` also uses; before that, numcodecs' `numcodecs.zarr3` " + "module wrote the same names" + ), + still_written=True, + reading=( + "Read when numcodecs has a codec of that id; the `configuration` member is " + "required, and is passed to numcodecs as it is. Written without a warning." + ), + ), + "v3-consolidated-metadata": Entry( + title=( + "Consolidated metadata in a Zarr format 3 group, as a `consolidated_metadata` " + 'member with `"must_understand": false`' + ), + conformance="unregistered", + written_by="zarr 3.0.0 and later, by `zarr.consolidate_metadata`", + still_written=True, + reading=( + "Read. Readers that do not know it ignore it, because of `must_understand`. " + "`zarr.consolidate_metadata` warns when it writes it." + ), + ), + "v2-consolidated-group-entries": Entry( + title=( + "Zarr format 2 consolidated metadata (`.zmetadata`) whose group entries carry a " + "`consolidated_metadata` member" + ), + conformance="non-conformant", + written_by="zarr 3.0.0 and later", + still_written=True, + reading=( + "Read: the member marks a group with no members, as the Zarr format 3 " + "consolidated metadata does." + ), + ), + "json-nan-tokens": Entry( + title=( + "The tokens `NaN`, `Infinity` and `-Infinity` in a metadata document, which are not " + "JSON, for a non-finite float in user attributes" + ), + conformance="non-conformant", + written_by=( + "zarr 2, zarr 3.0.0 to 3.0.6, and zarr 3.1.1 and later (zarr 3.0.7 to 3.0.10 wrote the " + 'strings `"NaN"`, `"Infinity"` and `"-Infinity"`; zarr 3.1.0 refused to write them)' + ), + still_written=True, + reading="Read silently in any metadata document. Written without a warning.", + ), + "numeric-fill-value-spellings": Entry( + title=( + "An integer fill value written as an integral float (`0.0`) or as a string " + '(`"42"`), or a float fill value written as a decimal string (`"3.14"`)' + ), + conformance="non-conformant", + written_by="Other software; found in public data. No zarr release wrote it.", + still_written=False, + reading="Read silently as the number it spells; re-saving stores that number.", + ), + "string-fill-value-number": Entry( + title="A number as the fill value of a variable-length string array", + conformance="non-conformant", + written_by="zarr 2, which gave such arrays a fill value of `0` by default", + still_written=False, + reading=( + 'Read silently as the string the number prints as (`0` as `"0"`), in both Zarr ' + "formats; re-saving stores that string." + ), + ), +} +"""Every kind of non-conformant or unregistered stored metadata zarr reads or writes.""" diff --git a/src/zarr/core/metadata/repair.py b/src/zarr/core/metadata/repair.py index d8050b9f3d..01f3c306db 100644 --- a/src/zarr/core/metadata/repair.py +++ b/src/zarr/core/metadata/repair.py @@ -1,9 +1,9 @@ """Repairs that read invalid stored array metadata documents written by older software. -This is the only place invalid metadata is read leniently. A repair maps a stored -array metadata document (parsed JSON) to a valid one. `ArrayV2Metadata.from_dict` and -`ArrayV3Metadata.from_dict` apply the repairs for their Zarr format, so every path -that parses a stored document, including consolidated metadata, goes through them. +A repair maps a stored array metadata document (parsed JSON) to a valid one. +`ArrayV2Metadata.from_dict` and `ArrayV3Metadata.from_dict` apply the repairs for their +Zarr format, so every path that parses a stored document, including consolidated +metadata, goes through them. A reading warns only where the user must act on it; a reading that gives what zarr read from the same document before these repairs existed is silent, so a document @@ -15,7 +15,9 @@ writes chunks under it. A repair that only respells a value zarr already read the same way (`true` as 1, `4.0` as 4) moves no chunks, so it is not marked. -To read another kind of invalid document, add a repair to `ARRAY_REPAIRS`. +To read another kind of invalid document, add a repair to `ARRAY_REPAIRS` and the kind +of document it reads to `zarr.core.metadata.ledger.LEDGER`, which lists every kind of +non-conformant or unregistered metadata zarr reads or writes, repaired or not. """ from __future__ import annotations @@ -191,6 +193,8 @@ def _read_chunk_shape( return edges, Reading(moves_chunks, warning) +# NON-CONFORMANT: chunk-size-zero +# NON-CONFORMANT: chunk-size-true def _invalid_chunk_sizes_v2(doc: ArrayDocument) -> tuple[ArrayDocument, Reading] | None: shape = doc.get("shape") if not _is_int_list(shape): @@ -241,6 +245,7 @@ def _read_codecs(codecs: list[JSON]) -> list[JSON] | None: return [stored if new is None else new for stored, new in zip(codecs, read, strict=True)] +# NON-CONFORMANT: chunk-size-true def _invalid_inner_chunk_sizes_v3(doc: ArrayDocument) -> tuple[ArrayDocument, Reading] | None: stored = doc.get("codecs") if not isinstance(stored, list) or (codecs := _read_codecs(stored)) is None: @@ -261,6 +266,9 @@ def _inner_chunk_shape(doc: ArrayDocument) -> list[int] | None: return [] +# NON-CONFORMANT: chunk-size-zero +# NON-CONFORMANT: chunk-size-true +# NON-CONFORMANT: regular-grid-edge-lists def _invalid_chunk_sizes_v3(doc: ArrayDocument) -> tuple[ArrayDocument, Reading] | None: grid = doc.get("chunk_grid") shape = doc.get("shape") @@ -341,6 +349,8 @@ def _respelled(read: JSON, stored: JSON) -> bool: return type(read) is not type(stored) +# NON-CONFORMANT: chunk-size-true +# NON-CONFORMANT: rectilinear-float-edges def _invalid_edge_lengths_v3(doc: ArrayDocument) -> tuple[ArrayDocument, Reading] | None: grid = doc.get("chunk_grid") if not (isinstance(grid, Mapping) and grid.get("name") == "rectilinear"): diff --git a/src/zarr/core/metadata/v2.py b/src/zarr/core/metadata/v2.py index 136267fc58..47e428dc54 100644 --- a/src/zarr/core/metadata/v2.py +++ b/src/zarr/core/metadata/v2.py @@ -192,6 +192,7 @@ def from_dict(cls, data: dict[str, Any], *, path: str | None = None) -> ArrayV2M expected = {x.name for x in fields(cls)} expected |= {"dtype", "chunks"} + # NON-CONFORMANT: v2-empty-filters # check if `filters` is an empty sequence; if so use None instead and raise a warning filters = _data.get("filters") if ( diff --git a/src/zarr/metadata/migrate_v3.py b/src/zarr/metadata/migrate_v3.py index ad177e19fb..5019436737 100644 --- a/src/zarr/metadata/migrate_v3.py +++ b/src/zarr/metadata/migrate_v3.py @@ -263,6 +263,7 @@ def _convert_compressor( return compressor_codec +# UNREGISTERED: numcodecs-codecs def _find_numcodecs_zarr3(numcodecs_codec: numcodecs.abc.Codec) -> Codec: """Find matching zarr.codecs.numcodecs codec (if it exists)""" diff --git a/tests/test_ledger.py b/tests/test_ledger.py new file mode 100644 index 0000000000..c257774c11 --- /dev/null +++ b/tests/test_ledger.py @@ -0,0 +1,89 @@ +"""The ledger of non-conformant and unregistered metadata matches the source. + +`zarr.core.metadata.ledger.LEDGER` lists every kind of stored metadata zarr reads or +writes that does not conform to the Zarr specifications or is not registered, and the +Metadata compatibility page is generated from it. The code that reads or writes each +kind carries a `# NON-CONFORMANT: ` or `# UNREGISTERED: ` comment. These tests +fail if the code and the ledger disagree, so neither can change without the other. +""" + +from __future__ import annotations + +import ast +import importlib.util +import io +import re +import tokenize +from pathlib import Path + +import zarr +from zarr.core.metadata.ledger import LEDGER, MARKERS +from zarr.core.metadata.repair import ARRAY_REPAIRS + +SRC = Path(zarr.__file__).parent +ROOT = SRC.parent.parent +MARKER = re.compile(r"#\s*(NON-CONFORMANT|UNREGISTERED):\s*(\S*)") + + +def _markers() -> list[tuple[str, int, str, str]]: + """Every marker comment in `src/zarr`: (file, line, tag, key).""" + return [ + (str(path.relative_to(SRC)), token.start[0], match[1], match[2]) + for path in sorted(SRC.rglob("*.py")) + for token in tokenize.generate_tokens(io.StringIO(path.read_text()).readline) + if token.type == tokenize.COMMENT and (match := MARKER.match(token.string)) + ] + + +def test_every_repair_is_in_the_ledger() -> None: + """Every repair applied to stored documents is listed, and every listed repair is + applied.""" + applied = {repair for repairs in ARRAY_REPAIRS.values() for repair in repairs} + listed = {repair for entry in LEDGER.values() for repair in entry.repairs} + assert {r.__name__ for r in applied - listed} == set(), "repairs missing from the ledger" + assert {r.__name__ for r in listed - applied} == set(), "ledger names unapplied repairs" + + +def test_every_marker_names_an_entry_of_its_conformance() -> None: + wrong = [ + f"{file}:{line}: {tag}: {key!r}" + for file, line, tag, key in _markers() + if key not in LEDGER or MARKERS[LEDGER[key].conformance] != tag + ] + assert wrong == [], "markers that name no ledger entry, or the wrong tag:\n" + "\n".join(wrong) + + +def test_every_entry_has_a_marker() -> None: + marked = {key for *_, key in _markers()} + assert set(LEDGER) - marked == set(), "ledger entries no code is marked with" + + +def test_unstable_data_type_warnings_are_marked() -> None: + """A data type that warns that it has no Zarr format 3 specification writes + unregistered or non-conformant metadata, so its warning carries a marker on the line + before it.""" + marked = {(file, line) for file, line, *_ in _markers()} + unmarked = [ + f"{path.relative_to(SRC)}:{node.lineno}" + for path in sorted(SRC.rglob("*.py")) + for node in ast.walk(ast.parse(path.read_text())) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "v3_unstable_dtype_warning" + and (str(path.relative_to(SRC)), node.lineno - 1) not in marked + ] + assert unmarked == [] + + +def test_metadata_compatibility_page_lists_the_ledger() -> None: + """The docs page holds the marker the docs hook replaces, and the hook renders every + entry.""" + page = (ROOT / "docs" / "metadata-compatibility.md").read_text() + spec = importlib.util.spec_from_file_location("mkdocs_hooks", ROOT / "mkdocs_hooks.py") + assert spec is not None + assert spec.loader is not None + hooks = importlib.util.module_from_spec(spec) + spec.loader.exec_module(hooks) + assert hooks._LEDGER_MARKER in page + rendered = hooks.ledger_markdown() + assert [entry.title for entry in LEDGER.values() if entry.title not in rendered] == [] From 2401ca77470d973a4a3ffdb6281872cd9d836a2f Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Sat, 10 Oct 2026 12:43:11 +0200 Subject: [PATCH 2/3] docs: record whether zarrs, tensorstore and zarr 2 read each ledger entry Add `Entry.other_readers` and `READERS_TESTED` to the metadata ledger, and render both on the Metadata compatibility page. The results come from reading a sample store of every entry with zarrs 0.23.14, tensorstore 0.1.85 and zarr 2.18.7 (Zarr format 2 samples only), with the zarr 3.1.6 and zarr 2.18.7 samples written by those releases. Assisted-by: ClaudeCode:claude-opus-5-5 --- docs/metadata-compatibility.md | 5 +-- mkdocs_hooks.py | 9 ++++-- src/zarr/core/metadata/ledger.py | 55 ++++++++++++++++++++++++++++++++ 3 files changed, 64 insertions(+), 5 deletions(-) diff --git a/docs/metadata-compatibility.md b/docs/metadata-compatibility.md index 4b8056e9be..97d16bd154 100644 --- a/docs/metadata-compatibility.md +++ b/docs/metadata-compatibility.md @@ -8,8 +8,9 @@ uses a name that the does not define (unregistered). Other Zarr implementations may refuse to read such metadata, or read it differently. -For each kind of metadata, the list says which software wrote it and how the -current release of `zarr` reads it. Some are read by a repair: zarr reads the +For each kind of metadata, the list says which software wrote it, how the +current release of `zarr` reads it, and whether other implementations read it +(the end of the page says which versions were tested). Some are read by a repair: zarr reads the metadata as the valid metadata it was meant to be, and stores that valid metadata the next time it writes the array's metadata. Where a repair needs you to act, opening the array warns and says what to do. diff --git a/mkdocs_hooks.py b/mkdocs_hooks.py index e13c17b1cd..3ac20025a7 100644 --- a/mkdocs_hooks.py +++ b/mkdocs_hooks.py @@ -99,17 +99,20 @@ def on_page_markdown(markdown: str, page: Page, config: MkDocsConfig, files: Fil def ledger_markdown() -> str: """Render ``zarr.core.metadata.ledger.LEDGER`` as Markdown: the metadata the current - zarr still writes, then the metadata it only reads.""" - from zarr.core.metadata.ledger import LEDGER + zarr still writes, then the metadata it only reads, then which other implementations + were tested.""" + from zarr.core.metadata.ledger import LEDGER, READERS_TESTED sections = [] for heading, still_written in (("Still written", True), ("No longer written", False)): items = [ f"- **{entry.title}** ({entry.conformance})\n" f" - Written by: {entry.written_by}\n" - f" - Current behavior: {entry.reading}" + f" - Current behavior: {entry.reading}\n" + f" - Other implementations: {entry.other_readers}" for entry in LEDGER.values() if entry.still_written is still_written ] sections.append(f"## {heading}\n\n" + "\n".join(items)) + sections.append(f"## Other implementations tested\n\n{READERS_TESTED}.") return "\n\n".join(sections) diff --git a/src/zarr/core/metadata/ledger.py b/src/zarr/core/metadata/ledger.py index 44da2d9402..e62a2e1c44 100644 --- a/src/zarr/core/metadata/ledger.py +++ b/src/zarr/core/metadata/ledger.py @@ -45,6 +45,8 @@ class Entry: """Whether the current zarr writes it.""" reading: str """How the current zarr reads it, and whether it warns.""" + other_readers: str + """Whether the implementations in `READERS_TESTED` read it.""" repairs: tuple[Repair, ...] = () """The repairs in `zarr.core.metadata.repair` that read it.""" @@ -55,6 +57,13 @@ class Entry: } """The comment tag that marks the code of an entry of each conformance.""" +READERS_TESTED: Final = ( + "zarrs 0.23.14 (Rust crate, `Array::open` then reading every element, else `Group::open`), " + "tensorstore 0.1.85 (`zarr` and `zarr3` drivers, which open arrays only), and zarr 2.18.7 " + "(Zarr format 2 only), on 2026-10-10, each against a sample store of every entry" +) +"""Which other implementations `Entry.other_readers` was tested with, and how.""" + LEDGER: Final[Mapping[str, Entry]] = { "chunk-size-zero": Entry( @@ -75,6 +84,9 @@ class Entry: "array holds only its fill value. Writing chunks first stores the repaired " "metadata." ), + other_readers=( + "zarrs and tensorstore reject it. zarr 2.18.7 fails to open it on a zero-length axis." + ), repairs=(repair._invalid_chunk_sizes_v2, repair._invalid_chunk_sizes_v3), ), "chunk-size-true": Entry( @@ -86,6 +98,11 @@ class Entry: written_by="zarr 3.0 and 3.2 for a chunk size of `True`", still_written=False, reading="Repaired silently: read as 1.", + other_readers=( + "zarrs and tensorstore reject it, in every place it was tested (regular chunk shape, " + "sharding inner chunk shape, rectilinear edges; tensorstore reads no rectilinear " + "grid). zarr 2.18.7 reads the Zarr format 2 form." + ), repairs=( repair._invalid_chunk_sizes_v2, repair._invalid_chunk_sizes_v3, @@ -106,6 +123,7 @@ class Entry: "without the `array.rectilinear_chunks` flag. Re-saving the metadata, or " "writing chunks, stores that rectilinear grid, which requires the flag." ), + other_readers="zarrs and tensorstore reject it.", repairs=(repair._invalid_chunk_sizes_v3,), ), "rectilinear-float-edges": Entry( @@ -117,6 +135,10 @@ class Entry: "Repaired silently: read as the integer it equals. A float anywhere else in a " "chunk grid is rejected." ), + other_readers=( + "zarrs rejects it (it reads a rectilinear grid with integer edges). tensorstore reads " + "no rectilinear grid." + ), repairs=(repair._invalid_edge_lengths_v3,), ), "v2-empty-filters": Entry( @@ -125,6 +147,7 @@ class Entry: written_by="zarr 3.0.0 to 3.0.3 for `filters=[]`", still_written=False, reading="Read as `filters: null`, with a warning.", + other_readers="zarrs, tensorstore and zarr 2.18.7 read it.", ), "group-consolidated-metadata-null": Entry( title='A Zarr format 3 group with `"consolidated_metadata": null`', @@ -132,6 +155,7 @@ class Entry: written_by="zarr 3.0.0 to 3.1.3, for every Zarr format 3 group", still_written=False, reading="Read silently as a group with no consolidated metadata.", + other_readers="zarrs opens the group.", ), "struct-bytes-codec-without-endian": Entry( title=( @@ -142,6 +166,10 @@ class Entry: written_by="zarr 3.1.x", still_written=False, reading="Read as little-endian, with a warning.", + other_readers=( + "tensorstore reads it, field by field, as little-endian. zarrs reads no `structured` " + "data type." + ), ), "structured-data-type": Entry( title=( @@ -159,6 +187,7 @@ class Entry: "registry defines it: a legacy alias that implementations may read but must not " "write. Written with an `UnstableSpecificationWarning`." ), + other_readers="tensorstore reads it, field by field. zarrs rejects the `structured` name.", ), "variable-length-bytes": Entry( title='The `"variable_length_bytes"` data type', @@ -174,6 +203,7 @@ class Entry: "`UnstableSpecificationWarning`; re-saving metadata that names `bytes` stores " "`variable_length_bytes`." ), + other_readers="zarrs reads it. tensorstore rejects the name.", ), "null-terminated-bytes": Entry( title='The `"null_terminated_bytes"` data type, for NumPy `S` data types', @@ -181,6 +211,7 @@ class Entry: written_by="zarr 3.1.0 and later", still_written=True, reading="Read. Written with an `UnstableSpecificationWarning`.", + other_readers="zarrs and tensorstore reject the name.", ), "raw-bytes": Entry( title='The `"raw_bytes"` data type, for NumPy `V` data types', @@ -188,6 +219,10 @@ class Entry: written_by="zarr 3.1.0 and later", still_written=True, reading="Read. Written with an `UnstableSpecificationWarning`.", + other_readers=( + "tensorstore opens it as an array of bytes with an extra trailing dimension. zarrs " + "rejects the name." + ), ), "numcodecs-codecs": Entry( title=( @@ -205,6 +240,11 @@ class Entry: "Read when numcodecs has a codec of that id; the `configuration` member is " "required, and is passed to numcodecs as it is. Written without a warning." ), + other_readers=( + "zarrs reads some of these codecs (`numcodecs.zlib`, `numcodecs.bz2`, " + "`numcodecs.fixedscaleoffset`) and rejects others (`numcodecs.delta`, `numcodecs.lz4`, " + "`numcodecs.crc32`). tensorstore rejects every one tested." + ), ), "v3-consolidated-metadata": Entry( title=( @@ -218,6 +258,7 @@ class Entry: "Read. Readers that do not know it ignore it, because of `must_understand`. " "`zarr.consolidate_metadata` warns when it writes it." ), + other_readers="zarrs opens the group.", ), "v2-consolidated-group-entries": Entry( title=( @@ -231,6 +272,7 @@ class Entry: "Read: the member marks a group with no members, as the Zarr format 3 " "consolidated metadata does." ), + other_readers="zarrs opens the group; zarr 2.18.7 opens the consolidated group.", ), "json-nan-tokens": Entry( title=( @@ -244,6 +286,11 @@ class Entry: ), still_written=True, reading="Read silently in any metadata document. Written without a warning.", + other_readers=( + "zarrs rejects the document (`zarr.json`, or `.zattrs` with Zarr format 2), and " + "tensorstore rejects such a `zarr.json`. tensorstore reads a Zarr format 2 array, " + "whose `.zattrs` it does not read; zarr 2.18.7 reads it." + ), ), "numeric-fill-value-spellings": Entry( title=( @@ -254,6 +301,10 @@ class Entry: written_by="Other software; found in public data. No zarr release wrote it.", still_written=False, reading="Read silently as the number it spells; re-saving stores that number.", + other_readers=( + "zarrs rejects every form. tensorstore reads an integer fill value of `0.0` and " + "rejects the string forms. zarr 2.18.7 reads the Zarr format 2 string form." + ), ), "string-fill-value-number": Entry( title="A number as the fill value of a variable-length string array", @@ -264,6 +315,10 @@ class Entry: 'Read silently as the string the number prints as (`0` as `"0"`), in both Zarr ' "formats; re-saving stores that string." ), + other_readers=( + "zarrs and zarr 2.18.7 read the Zarr format 2 form; zarrs rejects the Zarr format 3 " + "form. tensorstore reads no variable-length string array." + ), ), } """Every kind of non-conformant or unregistered stored metadata zarr reads or writes.""" From e27c509200500ccf93d913f8dc531ee5a5c72ff1 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Sat, 10 Oct 2026 19:05:12 +0200 Subject: [PATCH 3/3] docs: add the changelog fragment for the metadata ledger Assisted-by: ClaudeCode:claude-opus-5-5 --- changes/4507.doc.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 changes/4507.doc.md diff --git a/changes/4507.doc.md b/changes/4507.doc.md new file mode 100644 index 0000000000..fc53d1a08b --- /dev/null +++ b/changes/4507.doc.md @@ -0,0 +1 @@ +A new Metadata compatibility page lists every kind of stored metadata that zarr reads or writes although it does not conform to a Zarr specification, or uses a name the zarr-extensions registry does not define. For each kind, the page says which software wrote it, how zarr reads it now, and whether zarrs, tensorstore and zarr 2 read it. The page is generated from `zarr.core.metadata.ledger`, which a test checks against the code that reads and writes such metadata.