docs: list non-conformant and unregistered metadata in a checked ledger

changes/4507.doc.md

@@ -0,0 +1 @@
+A new Metadata compatibility page lists every kind of stored metadata that zarr reads or writes although it does not conform to a Zarr specification, or uses a name the zarr-extensions registry does not define. For each kind, the page says which software wrote it, how zarr reads it now, and whether zarrs, tensorstore and zarr 2 read it. The page is generated from `zarr.core.metadata.ledger`, which a test checks against the code that reads and writes such metadata.

docs/contributing.md

@@ -455,6 +455,8 @@ The Zarr library is an implementation of a file format standard defined external
 
 If an existing Zarr format version changes, or a new version of the Zarr format is released, then the Zarr library will generally require changes. It is very likely that a new Zarr format will require extensive breaking changes to the Zarr library, and so support for a new Zarr format in the Zarr library will almost certainly come in a new `major` release. When the Zarr library adds support for a new Zarr format, there may be a period of accelerated changes as developers refine newly added APIs and deprecate old APIs. In such a transitional phase breaking changes may be more frequent than usual.
 
+The Zarr library sometimes reads or writes stored metadata that does not conform to a Zarr specification, or that uses a name the [zarr-extensions registry](https://github.com/zarr-developers/zarr-extensions) does not define. It reads such metadata when software wrote it and users hold data stored with it, and it may write such metadata when no conformant form exists yet. Each kind of such metadata is an entry in `zarr.core.metadata.ledger.LEDGER`, which the [Metadata compatibility](metadata-compatibility.md) page lists. The entry records which software wrote the metadata, including which releases of this library, and how the library reads it now. The code that reads or writes it carries a `# NON-CONFORMANT: <key>` or `# UNREGISTERED: <key>` comment naming the entry, and `tests/test_ledger.py` fails if the code and the ledger disagree. A warning about such metadata describes the metadata and what to do about it; it does not name the release that wrote it.
+
 ### Deprecation policy
 
 Our versioning policy (above) commits us to minimizing the effort users spend upgrading. A deprecation cycle is the main tool we use to honor that commitment. Rather than removing or changing public behavior abruptly, we first ship a release that keeps the old behavior working while warning that it is going away, and only remove it in a later release. This section defines when a deprecation cycle is required, how to decide whether a removal is worth it, and the concrete steps a deprecation must follow.

docs/metadata-compatibility.md

@@ -0,0 +1,22 @@
+# Metadata compatibility
+
+This page lists every kind of stored metadata that zarr-python reads or writes
+although it does not conform to the
+[Zarr specifications](https://zarr-specs.readthedocs.io/) (non-conformant), or
+uses a name that the
+[zarr-extensions registry](https://github.com/zarr-developers/zarr-extensions)
+does not define (unregistered). Other Zarr implementations may refuse to read
+such metadata, or read it differently.
+
+For each kind of metadata, the list says which software wrote it, how the
+current release of `zarr` reads it, and whether other implementations read it
+(the end of the page says which versions were tested). Some are read by a repair: zarr reads the
+metadata as the valid metadata it was meant to be, and stores that valid
+metadata the next time it writes the array's metadata. Where a repair needs you
+to act, opening the array warns and says what to do.
+
+The list is generated from `zarr.core.metadata.ledger` when the documentation
+is built. A test checks that ledger against the code that reads and writes
+these kinds of metadata, so the list stays complete.
+
+<!-- metadata-ledger -->

mkdocs.yml

@@ -102,6 +102,7 @@ nav:
       - 'zarr-metadata ↪': https://zarr.readthedocs.io/projects/zarr-metadata/
       - 'zarr-indexing ↪': https://zarr.readthedocs.io/projects/zarr-indexing/
   - release-notes.md
+  - metadata-compatibility.md
   - roadmap.md
   - contributing.md
   - Blog:

mkdocs_hooks.py

@@ -19,6 +19,8 @@
 if TYPE_CHECKING:
     from markdown import Markdown
     from mkdocs.config.defaults import MkDocsConfig
+    from mkdocs.structure.files import Files
+    from mkdocs.structure.pages import Page
 
 # Mirrors markdown_exec's _to_bool: everything but these means "true".
 _FALSY = {"", "no", "off", "false", "0"}
@@ -80,3 +82,37 @@ def on_config(config: MkDocsConfig) -> MkDocsConfig:
         }
     )
     return config
+
+
+# The metadata compatibility page holds this marker where the list of
+# non-conformant and unregistered metadata goes. It is generated at build time from
+# ``zarr.core.metadata.ledger``, which ``tests/test_ledger.py`` checks against the
+# source, so the page lists what the code reads and writes.
+_LEDGER_MARKER = "<!-- metadata-ledger -->"
+
+
+def on_page_markdown(markdown: str, page: Page, config: MkDocsConfig, files: Files) -> str:
+    if _LEDGER_MARKER not in markdown:
+        return markdown
+    return markdown.replace(_LEDGER_MARKER, ledger_markdown())
+
+
+def ledger_markdown() -> str:
+    """Render ``zarr.core.metadata.ledger.LEDGER`` as Markdown: the metadata the current
+    zarr still writes, then the metadata it only reads, then which other implementations
+    were tested."""
+    from zarr.core.metadata.ledger import LEDGER, READERS_TESTED
+
+    sections = []
+    for heading, still_written in (("Still written", True), ("No longer written", False)):
+        items = [
+            f"- **{entry.title}** ({entry.conformance})\n"
+            f"    - Written by: {entry.written_by}\n"
+            f"    - Current behavior: {entry.reading}\n"
+            f"    - Other implementations: {entry.other_readers}"
+            for entry in LEDGER.values()
+            if entry.still_written is still_written
+        ]
+        sections.append(f"## {heading}\n\n" + "\n".join(items))
+    sections.append(f"## Other implementations tested\n\n{READERS_TESTED}.")
+    return "\n\n".join(sections)

src/zarr/api/asynchronous.py

@@ -256,6 +256,7 @@ async def consolidate_metadata(
             v = dataclasses.replace(v, consolidated_metadata=ConsolidatedMetadata(metadata={}))
             members_metadata[k] = v
 
+    # UNREGISTERED: v3-consolidated-metadata
     if any(m.zarr_format == 3 for m in members_metadata.values()):
         warnings.warn(
             "Consolidated metadata is currently not part in the Zarr format 3 specification. It "

src/zarr/codecs/bytes.py

@@ -73,6 +73,7 @@ def to_dict(self) -> dict[str, JSON]:
 
     def evolve_from_array_spec(self, array_spec: ArraySpec) -> Self:
         if isinstance(array_spec.dtype, Struct):
+            # NON-CONFORMANT: struct-bytes-codec-without-endian
             if array_spec.dtype.has_multi_byte_fields():
                 if self.endian is None:
                     warnings.warn(

src/zarr/codecs/numcodecs/_codecs.py

@@ -60,6 +60,7 @@ def _expect_name_prefix(codec_name: str) -> str:
     return codec_name.removeprefix(CODEC_PREFIX)
 
 
+# UNREGISTERED: numcodecs-codecs
 def _parse_codec_configuration(data: dict[str, JSON]) -> dict[str, JSON]:
     parsed_name, parsed_configuration = parse_named_configuration(data)
     if not parsed_name.startswith(CODEC_PREFIX):
@@ -112,6 +113,7 @@ def from_dict(cls, data: dict[str, JSON]) -> Self:
         codec_config = _parse_codec_configuration(data)
         return cls(**codec_config)
 
+    # UNREGISTERED: numcodecs-codecs
     def to_dict(self) -> dict[str, JSON]:
         codec_config = self.codec_config.copy()
         codec_config.pop("id", None)

src/zarr/core/_json.py

@@ -38,6 +38,7 @@
 def buffer_to_json(buffer: Buffer) -> JSON:
     """Parse the contents of a `Buffer` as a JSON value."""
     # json.loads is typed as returning Any; the result is by definition JSON.
+    # NON-CONFORMANT: json-nan-tokens
     return cast("JSON", json.loads(buffer.to_bytes()))
 
 
@@ -96,6 +97,7 @@ def json_to_buffer(
     """
     if prototype is None:
         prototype = default_buffer_prototype()
+    # NON-CONFORMANT: json-nan-tokens
     return prototype.buffer.from_bytes(json.dumps(obj, indent=indent, allow_nan=allow_nan).encode())
 
 

src/zarr/core/dtype/npy/bytes.py

@@ -383,6 +383,7 @@ def to_json(
         if zarr_format == 2:
             return {"name": self.to_native_dtype().str, "object_codec_id": None}
         elif zarr_format == 3:
+            # UNREGISTERED: null-terminated-bytes
             v3_unstable_dtype_warning(self)
             return {
                 "name": self._zarr_v3_name,
@@ -781,6 +782,7 @@ def to_json(self, zarr_format: ZarrFormat) -> RawBytesJSON_V2 | RawBytesJSON_V3:
         if zarr_format == 2:
             return {"name": self.to_native_dtype().str, "object_codec_id": None}
         elif zarr_format == 3:
+            # UNREGISTERED: raw-bytes
             v3_unstable_dtype_warning(self)
             return {"name": self._zarr_v3_name, "configuration": {"length_bytes": self.length}}
         raise ValueError(f"zarr_format must be 2 or 3, got {zarr_format}")  # pragma: no cover
@@ -1045,6 +1047,7 @@ def _check_json_v3(cls, data: DTypeJSON) -> TypeGuard[Literal["variable_length_b
             True if the input is a valid representation of this class in Zarr V3, False otherwise.
         """
 
+        # UNREGISTERED: variable-length-bytes
         return data in (cls._zarr_v3_name, "bytes")
 
     @classmethod
@@ -1139,6 +1142,7 @@ def to_json(
         if zarr_format == 2:
             return {"name": "|O", "object_codec_id": self.object_codec_id}
         elif zarr_format == 3:
+            # UNREGISTERED: variable-length-bytes
             v3_unstable_dtype_warning(self)
             return self._zarr_v3_name
         raise ValueError(f"zarr_format must be 2 or 3, got {zarr_format}")  # pragma: no cover

src/zarr/core/dtype/npy/float.py

@@ -278,6 +278,7 @@ def from_json_scalar(self, data: JSON, *, zarr_format: ZarrFormat) -> Scalar:
         if zarr_format == 2:
             if check_json_float_v2(data):
                 return self._cast_scalar_unchecked(float_from_json_v2(data))
+            # NON-CONFORMANT: numeric-fill-value-spellings
             elif check_json_floatish_str(data):
                 return self._cast_scalar_unchecked(float(data))
             else:
@@ -287,6 +288,7 @@ def from_json_scalar(self, data: JSON, *, zarr_format: ZarrFormat) -> Scalar:
         elif zarr_format == 3:
             if check_json_float_v3(data):
                 return self._cast_scalar_unchecked(float_from_json_v3(data))
+            # NON-CONFORMANT: numeric-fill-value-spellings
             elif check_json_floatish_str(data):
                 return self._cast_scalar_unchecked(float(data))
             else:

src/zarr/core/dtype/npy/int.py

@@ -205,6 +205,7 @@ def from_json_scalar(self, data: JSON, *, zarr_format: ZarrFormat) -> Scalar:
         """
         if check_json_int(data):
             return self._cast_scalar_unchecked(data)
+        # NON-CONFORMANT: numeric-fill-value-spellings
         if check_json_intish_float(data):
             return self._cast_scalar_unchecked(int(data))
 

src/zarr/core/dtype/npy/string.py

@@ -405,6 +405,7 @@ def item_size(self) -> int:
         return self.length * self.code_point_bytes
 
 
+# NON-CONFORMANT: string-fill-value-number
 def check_vlen_string_json_scalar(data: object) -> TypeGuard[int | str | float]:
     """
     Check if the input is a valid JSON scalar for a variable-length string.

src/zarr/core/dtype/npy/structured.py

@@ -465,6 +465,7 @@ def to_json(self, zarr_format: ZarrFormat) -> StructuredJSON_V2 | StructuredJSON
             ]
             return {"name": fields, "object_codec_id": None}
         elif zarr_format == 3:
+            # NON-CONFORMANT: structured-data-type
             v3_unstable_dtype_warning(self)
             fields = [
                 [f_name, f_dtype.to_json(zarr_format=zarr_format)]  # type: ignore[list-item]
@@ -706,6 +707,7 @@ def _check_json_v3(cls, data: DTypeJSON) -> TypeGuard[StructJSON_V3]:  # type: i
         return (
             isinstance(data, dict)
             and set(data.keys()) == {"name", "configuration"}
+            # NON-CONFORMANT: structured-data-type
             and data["name"] in ("struct", "structured")
             and isinstance(data["configuration"], dict)
             and set(data["configuration"].keys()) == {"fields"}
@@ -791,6 +793,7 @@ def from_json_scalar(self, data: JSON, *, zarr_format: ZarrFormat) -> np.void:
                 else:
                     field_values.append(field_dtype.default_scalar())
             return self._cast_scalar_unchecked(tuple(field_values))
+        # NON-CONFORMANT: structured-data-type
         elif check_json_str(data):
             as_bytes = bytes_from_json(data, zarr_format=zarr_format)
             return self._scalar_bytes_to_void(as_bytes, zarr_format)

src/zarr/core/group.py

@@ -423,6 +423,7 @@ def to_buffer_dict(self, prototype: BufferPrototype) -> dict[str, Buffer]:
                         # it's an array
                         d[f"{k}/{ZARRAY_JSON}"] = v
                     else:
+                        # NON-CONFORMANT: v2-consolidated-group-entries
                         d[f"{k}/{ZGROUP_JSON}"] = {
                             "zarr_format": self.zarr_format,
                             "consolidated_metadata": {
@@ -471,6 +472,9 @@ def from_dict(cls, data: dict[str, Any], *, path: str | None = None) -> GroupMet
             raise NodeTypeValidationError(
                 f"Invalid value for 'node_type'. Expected 'group' or None. Got {node_type!r}."
             )
+        # NON-CONFORMANT: group-consolidated-metadata-null
+        # UNREGISTERED: v3-consolidated-metadata
+        # NON-CONFORMANT: v2-consolidated-group-entries
         consolidated_metadata = data.pop("consolidated_metadata", None)
         if consolidated_metadata:
             data["consolidated_metadata"] = ConsolidatedMetadata.from_dict(
@@ -495,6 +499,7 @@ def to_dict(self) -> dict[str, Any]:
         result = asdict(replace(self, consolidated_metadata=None))
         result.update(result.pop("extra_fields"))
         if self.consolidated_metadata is not None:
+            # UNREGISTERED: v3-consolidated-metadata
             result["consolidated_metadata"] = self.consolidated_metadata.to_dict()
         else:
             # Leave consolidated metadata unset if it's None