Repository navigation
Conversation
…ery reader takes
`to_json`, `to_key_value` and `canonicalize` wrote a field with nothing
to configure by its bare name: `"crc32c"`, `"default"`. The spec allows
that since v3.1, but a Zarr v3.0 reader takes no short-hand name in
`codecs` (core L585-L592), and no zarr-python release reads one there or
in `chunk_key_encoding`, so documents the package wrote could not be
opened by them. Every extension point but a data type is now written as
an object, `{"name": ...}`; a data type with nothing to configure keeps
its bare name, as core data types have been written since v3.0.
Re-writing zarr-python's own documents through the model, zarr-python
now opens all of them, where 14 to 22 of 80 failed.
Assisted-by: ClaudeCode:claude-opus-5-5
…ry says what the package decided Four users of the new APIs found the same rough edges. A problem's message showed the Python value -- `got None`, `got (1,)` -- and named shapes in Python's words, "a sequence", "a mapping", a one-member Literal as a one-element tuple, a union of two objects as "an object or an object". Values now show as the JSON the document holds, shapes in JSON's words, and a value of another JSON type than a closed set's is `invalid_type`, so the kind tells a wrong type from a wrong value. `help` of a validator printed its default scope in full, some 14,000 characters; a scope's repr now says how many definitions it holds. The README's validation boundary says what the package decides where the specs leave it open: non-finite numbers in attributes, written back as bare tokens; `must_understand: false` refused at every extension point; a chunk length of 0 along a dimension of length 0. The models' I/O methods have docstrings, and no public docstring names a private function. Assisted-by: ClaudeCode:claude-opus-5-5
…de the problems The v3 array validator read every extension point, the chunks the codecs are handed and each codec's stage, then kept only the problems, so a caller wanting any of it -- a core-only policy, each codec's stage, what was left unjudged -- resolved every field again. The reading is now returned as a ZarrV3ArrayMetadataReading, and validate_array_metadata_v3 is its problems. fields() gives each field with where it sits in the document, and after it the fields it holds, as the new fields_of gives them for any field resolve read. Resolved gains the kind a field was read as, which a field nothing in scope claims keeps too: before, a nested field nothing claimed told nothing of its kind, and a pipelines hook naming data types nothing claims read them as codecs. Built by hand, a Resolved is checked as resolve builds one: its kind one of the five, and its definition of that kind. A field its rules refuse keeps the fields it read inside it, whose problems it already reported, so fields() leaves none of them out. Assisted-by: ClaudeCode:claude-opus-5-5
…, and one read builds a document's reading and model - `resolve` gives one of three frozen dataclasses, built with keywords: `Read`, by the definition in scope that claims the field's name, with its definition and configuration; `Unclaimed`, when nothing in scope claims it; or `Refused`, with the definition that refused it, if any. `Resolved` is their union, for `match`. Each answers `name`, `read_as`, `definition` and `nested`. They compare by what they read, not how they were spelled: a configuration holds each field in it as `to_json` writes it. - Models hold those fields. `ZarrV3NamedConfig`, `ZarrV3MetadataField` (the model's and the pydantic one), `ZarrV3ArrayMetadataPartial` and `ZarrV3GroupMetadataPartial` are gone. - `read_array_metadata_v3` returns one reading, holding every problem and, when there is none, the model built by the same read; `read_group_metadata_v3` does the same for a group, reading each document its consolidated metadata holds once and holding the model of each that has no problem. `from_json` is the model, or the problems raised; `validate_*` are the problems, and build no models. - A model holds no scope, as a pydantic model holds no validation context. `update(context=..., **members)` reads only the members it is given, in that scope; the fields the model holds pass through as they were read; `UNSET` leaves a member out; a group keeps the documents its consolidated metadata holds unless it is given others. The pydantic field types read in the scope their validation context holds. - `to_key_value` reads the document it writes by the model's own fields, and refuses one with a problem, so a model changed by hand is never written invalid. - A scope pickles as its definitions and copies as itself; the core definitions compare equal once pickled; `configuration_of` takes an equal definition. A read copies no document first, reads each member once, and reads a field a configuration holds a frame deeper than it. - The changelog describes what ships: #370's and #372's fragments fold into #373's, and the statements of 4434, 4436 and 4443 that this replaces are dropped from theirs. Assisted-by: ClaudeCode:claude-opus-5-5
…ys, as a discriminated union reads its tag - `read_node_metadata_v3(value, *, context)` reads a v3 `zarr.json` of either kind: as `read_array_metadata_v3` or `read_group_metadata_v3` reads it, as its `node_type` says -- the tag of a union, as pydantic's discriminator and zod's discriminated union read one -- or, when it says neither or is not an object, as `ZarrV3UnknownNodeReading`, with the problem, reading nothing else of it. `ZarrV3NodeMetadataReading` is the three; `validate_node_metadata_v3` gives the problems. - `node_metadata_from_json_v3` and `node_metadata_from_key_value_v3` build the model of either kind, `ZarrV3NodeMetadata`, as the models' own `from_json` and `from_key_value` build one, and as pydantic's `TypeAdapter` validates a discriminated union from Python and JSON. - A group's consolidated metadata reads each document it holds as a node, so one of no node type is in `reading.consolidated`, as unknown, rather than missing. Its problem is located as before, and reads as a missing key when the entry has no `node_type`, and at the entry when it is not an object. Assisted-by: ClaudeCode:claude-opus-5-5
…ries what was found and what was expected Two borrowings from pydantic and zod. Bounds on the type. A number's type carries its bounds, in annotated-types' vocabulary as pydantic reads it: a gzip `level` is `Annotated[int, Interval(ge=0, le=9)]`. The checker holds a value to `Gt`, `Ge`, `Lt`, `Le` and `Interval`, one bound from each side, at any depth: one problem per value, whose message says what the type admits and whose `ctx` holds the bounds. A bound is held as the number it equals, so Python's `Annotated` cache, which takes `Ge(True)` for `Ge(1)`, cannot make a verdict or a message depend on import order. `Annotated` may also carry a note, a string or a `Doc`. Any other metadata is a `TypeError` when the type is compiled, so the checker and pydantic never read one type two ways. `typeddict_keys` keeps a member's metadata. The bounds that were rules move onto the types: gzip, zstd, blosc's `clevel` and `blocksize`, the regular and rectilinear grids, the shard's inner chunk shape, the eight integer fill values, byte values, and the numpy time types' `scale_factor` and ticks. `_InRange`, `byte_value_problems` and the numpy time rules are gone. Problems carry data. `ValidationProblem` gains `input`, the JSON the value handed in holds at `loc` (`UNSET` where nothing is, or where what is there is not JSON, so an error always pickles), and `ctx`, what was expected: a type's bounds by pydantic's names, or `expected` for a closed set. Neither takes part in equality or the repr, and `ctx` is read-only. Every reader fills `input` from the value its caller handed it, so a rule says only where a problem is. `UNSET` moves below the problem record. BREAKING CHANGE: `typeddict_keys(...).members` keeps `Annotated` metadata, and `check` and `Definition.check` hold values to the bounds a type carries. A definition's rules are asked only of a configuration within its bounds, as pydantic's after-validators are, so a value out of bounds hides a rule's problem until it is fixed: blosc's `typesize` behind a bad `clevel`, an `r16` fill value's count behind a bad byte. A definition that reuses a configuration TypedDict takes its bounds with it. A v2 `order` or `dimension_separator`, and a consolidated envelope's `must_understand`, are reported as values outside a closed set, `expected one of ["C", "F"], got "Q"`, and a `must_understand` that is not a boolean as `invalid_type`. A numpy time fill value out of range is reported as out of its integer range, without naming "NaT". Assisted-by: ClaudeCode:claude-opus-5-5
… and canonical_of spells a field read already `with_problems(fields, problems)` gives each field of one read -- a reading's `fields()` and `problems`, or `fields_of` a field and the problems `resolve` gave with it -- with the problems located in it, in the fields it holds too: those it was read with, and those the document found with it where it stands, its place in the pipeline, the chunk it is handed, the array's shape. A field with none is valid there. A function of the problems, as zod's `flattenError` is of the issues, grouping them at every depth; a problem in no field is in none's. `canonical_of(resolved, problems)` spells a field a scope has read already, given its problems, in its simplest equivalent spelling, without reading it again: None for a field with a problem, as `canonicalize` gives, so a key its TypedDict does not declare is never erased. `canonicalize` is `canonical_of(*resolve(...))`. Assisted-by: ClaudeCode:claude-opus-5-5
…s its specification says "Chunk sizes must be greater than zero" (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/chunk-grids/regular-grid/index.rst#L40), along a dimension of length 0 too. `RegularChunkGridConfiguration` bounded its lengths with `Ge(0)`, and its shape rules took a 0 on an empty dimension, reading the core specification's "The chunk shape elements are non-zero when the corresponding dimensions of the arrays have non-zero length" as allowing it. That sentence says less than the grid's own. The bound is now `Ge(1)`, so a 0 is an `invalid_value` at its place wherever it is, and the shape rules check only that the grid has the array's dimensionality. `create_default` already wrote 1 there. BREAKING CHANGE: a regular grid with a chunk length of 0 on a dimension of length 0, as zarr-python 3.0 and 3.1 wrote one, is now refused. Assisted-by: ClaudeCode:claude-opus-5-5
… and a zarr.json As pydantic's `TypeAdapter(...).json_schema()` and zod's `toJSONSchema` write theirs, in draft 2020-12: - `json_schema(shape)`, in `zarr_metadata.typed_json`, writes a TypedDict as `check` reads it. A bound is JSON Schema's keyword for it, the stricter where a type and its `NewType` both say one, and a `Doc` the `description`. Each TypedDict and type alias is written once, in `$defs`, under its name. - `field_json_schema(kind, context)`, in `zarr_metadata.v3.definition`, writes a field of one kind as a scope reads it: each definition's field, and a name none of them is written with, with any configuration. Raw bits' name is matched to its end, so a validator that matches patterns as Python does takes no final newline for it. - `node_metadata_json_schema_v3(context=...)`, in `zarr_metadata.model`, writes a `zarr.json`: its fill value held to the data type it names, and a group's consolidated metadata holding documents. `Schemas` writes each shape, asking a caller's `SchemaLeaf` first, as the checker asks a `Leaf`, and hands back a schema sharing nothing. A schema says what the types say and not what the rules say, so every JSON document the package finds nothing wrong with, the schema accepts. Property tests hold the typed_json schema to the checker's reference, bounds in layers to what `check` holds a value to, and fields and documents changed in one or two places to soundness. JSON Schema takes `1.0` for an integer. Only a `Doc` is a description, as in zod: the package's docstrings are written for Python's readers. The six field aliases move to `v3._common`, so `ZarrV3ArrayMetadataJSON` says what each extension point is: `data_type: DataTypeField`, `codecs: tuple[CodecField, ...]`. To a type checker and to `check` they are the JSON they were, and the model's tables of extension points are read off the TypedDict. `shape` holds integers of at least 0, which `check` now holds it to. A data type whose `fill_value` holds a metadata field is refused when it is built, since the checker reads a fill value as a value and a schema would write it as a field. Assisted-by: ClaudeCode:claude-opus-5-5
…key_value writes it as it is As pydantic validates in `__init__`: a v3 or v2 model's constructor reads its document by its own fields, the read the writer made before, raises `MetadataValidationError` with every problem, an extra field named as a member among them, and holds its members as that read refines them, in containers of its own, as pydantic holds what its `__init__` coerced. So a model built by hand, or changed by `dataclasses.replace` or a v2 `update`, is refused at the change rather than at the write; none is built invalid, none shares a container with what it was built of, and each writes as JSON. Each field, a `Read` or an `Unclaimed`, is held as the scope read it. The reads build their models with a private `construct`, as pydantic's `model_construct` builds one, so a model a read builds is not read twice, and `to_key_value` serializes without reading again. A group checks its own members, since each document its consolidated metadata holds is a model that checked itself; a consolidated path that is not a string is a `TypeError`. `ZarrV2ArrayMetadata.create_default(chunks=...)` without a shape they fit is refused, as the v3 model refuses a grid its shape does not take. The clauses about the checking writer that this makes untrue are dropped from 373's and 4420's fragments. BREAKING CHANGE: building or replacing a model into one whose document has a problem raises, and a model holds the containers a read would, not the ones it was given; a container changed in place is not checked again. Assisted-by: ClaudeCode:claude-opus-5-5
…nknown key wherever it sits `ProblemKind` defines `unknown_key` as a key an object's type does not declare where the type is closed, and a v3 configuration reported one so. A v2 `.zgroup`, a `.zmetadata` envelope, a group's `consolidated_metadata` and a metadata field's envelope reported the same case as `invalid_value`, so a reader that tolerates what another writer added -- netCDF-C's `_nczarr_*` keys (zarr-python#2296) -- could not filter it by kind. Each is now `unknown_key`, with the checker's message, "unexpected key 'x'". A definition's `check` and `judge` give a configuration back when a field it holds has a stray member, reported and left out, as the checker leaves out a key a closed TypedDict does not declare; a `must_understand` of `false` still refuses it. Assisted-by: ClaudeCode:claude-opus-5-5
…s in `read_node_metadata_v3` reads a document by the node type it names, and one that names none was reported only as missing its `node_type`. A crawler meeting the root `zarr.json` of zarr-python 2's draft of v3 (zarr-python#2982), whose `zarr_format` is a URL, learned "no node type", not "not v3". A document the dispatch cannot follow is now judged by its `zarr_format` as well: missing, or other than 3, is a problem beside the node type's. Assisted-by: ClaudeCode:claude-opus-5-5
…below its group
A group's consolidated metadata holds the hierarchy below the group:
zarr-python keeps the document of the node at /a/b of that hierarchy,
the group its root, at the key a/b. Nothing judged the keys or the tree
they make. A key that is no node's path ("", "__a", "a/../b", "/a"), a
document below an array's, and a document whose group is missing were
read without a problem; zarr-python raises a bare KeyError on three of
them and keeps the rest. Each is now a problem at its entry, a missing
group a missing_key, and ZarrV3ConsolidatedMetadata refuses them when it
is built. A listed group's own listing lists what the group lists, at
the joined key and of the same node type: a node it lists alone is
dropped by zarr-python's reader, which keeps the flat listing.
The rules are a hierarchy's, not consolidated metadata's: a private
hierarchy_problems judges the node type of each node, by its node path,
as the spec's tree, so a model of a whole hierarchy can use it too. Every
problem is one per value, saying its reasons at once -- a path of a
thousand bad names is one problem, a chain of missing groups one
missing_key at the nearest -- so what is reported weighs what was read.
NodeName and NodePath, in zarr_metadata.v3, are the spec's node names
and paths, modelled on zarrs' types, and zarr_metadata.model judges a
string by them with validate/is/parse_node_name_v3 and their node_path
twins.
Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-fable-5-1
…ment A v3 array model compared its members with ==, so "NaN" and "0x7fc00000" were two float32 fill values, 0.0 and -0.0 one, a blosc with and without the typesize that noshuffle ignores two codecs though canonical_of spelled them alike, and a model holding a NaN attribute never equalled its own copy. Two models are now equal when they mean the same document: what the package interprets compares by its canonical spelling, and what it does not compares as JSON text, which tells true from 1 and -0.0 from 0.0 and takes NaN for itself. A data type's definition says which spelling of a fill value is its value's own, DataTypeDefinition.fill_value_canonical, and canonical_fill_value spells one, or gives UNSET for a fill value with a problem: the float types read a number as a float64, as JSON parsers and numpy do, and spell a value by its shortest number, a named value, or a NaN's bits; complex by part; the numpy time types' -2**63 as "NaT"; bytes as base64; a struct field by field. A field, Read, Unclaimed or Refused, compares by field_key: its definition and its canonical configuration as text, with the fields it holds by their own keys; default, v2, sharding_indexed and zstd gain a canonical folding their spec defaults. Every model and field hashes as it compares. Assisted-by: ClaudeCode:claude-opus-5-5 Assisted-by: ClaudeCode:claude-fable-5-1
…of the readers This commit fixes what six review passes found, beyond the fixes already in #381. Nesting depth. A reader now walks at most 256 levels (JSON_DEPTH) and reports a deeper level as an invalid_value problem. Before, a small document nested a thousand levels deep made every validator, reader, parser and from_key_value raise RecursionError, and json.loads on store bytes made of 100,000 "[" did the same (now an invalid_json problem). A chain of groups, each holding the next in its consolidated metadata, was also unbounded. Each document is now read from its position in the outer document, so depth is counted from the outer root. is_json and the is_* guards stop at the same depth. A v2 dtype of field records is checked as JSON before its shape is read. Readers, writers and comparisons use one stack frame per level; the v2 models no longer use copy.deepcopy, which used two. validate_json takes the location of its value, so the v2 validator counts depth from the document root as the v3 readers do. arrays_to_tuples uses one frame per level. A problem copied by with_input or prefixed is not re-checked. A struct fill value is judged in linear time. An integer too long to print is shown by its bit length, a value too deeply nested is described instead of printed, and a non-string key is shown with Python's repr. Hand-built field objects. A Read or Unclaimed object placed inside a document passed to a reader is not JSON and is now refused. Before, it passed validation as if it had been read, so a model could write a document that the same validator then refused. Only a model's own fields are treated as already read. Absent values. A reading's data_type, chunk_grid and chunk_key_encoding are UNSET when the document has no such key, and Refused.json is UNSET when the field was not JSON. Both used to be None, which is also what a JSON null becomes. A field with no name is a missing_key problem. A consolidated_metadata of null is an invalid_type problem; it is no longer read as absent, and neither JSON Schema admits it. ZarrV3ConsolidatedMetadata declares must_understand as False. Extension names. An extension name must match ^[a-z][a-z0-9-_.]+$ or be a URI in RFC 3986 characters, so Python's re and ECMA-262 read the schema pattern the same way. Any other name is refused before a definition is looked up; before, any string was read as an unknown extension. well_named checks this, validate_metadata_field_v3 enforces it, the JSON Schema's unclaimed branch carries the pattern, and a definition with an invalid name raises TypeError when built. A raw-bits name with more than 100 digits is treated as an extension name, not a size. Definition functions always receive a read-only view of the configuration, and an error raised by a canonical function names its definition. Pydantic. The pydantic types raise ValidationError with one line error per problem, carrying type, loc, input and ctx. For a missing key the input is the object that lacks it. A message containing a ctx placeholder like {expected} is passed as the ctx's "message" entry, because pydantic renders messages as templates with no escaping. Tests. New tests pin each behavior the mutation review found unpinned: the stricter of two bounds per keyword, the unclaimed envelope in the schema, no empty attributes written for a group, value_at at index 0, every integer type's range, union naming, layered Annotated metadata, and construct's default factory. The float-bits property test draws sign, exponent and mantissa separately so it reaches normal values. A shard nested to the depth cap is read, written, hashed, pickled, deep-copied and walked; one level deeper reports the depth problem. The same holds for a v2 document and a v3 fill value at the cap. Docs: two spec anchors fixed, the struct data type rendered, a note on the JSONValue export, a typed TypeAdapter example, with_problems compared to zod's treeifyError, and the float fill value rounding cited to zarrs and tensorstore. Assisted-by: ClaudeCode:claude-fable-5-1
…t calling repr `shown` relied on `repr` raising RecursionError for a deeply nested value. Python 3.14 limits recursion by the real C stack, so on Linux a list nested 100,000 levels deep printed fine and the problem message was the whole value. Now any value with a problem past `JSON_DEPTH` is reported as "a value nested too deep to show", on every interpreter and platform. The test nests one level past the limit, where repr always works. Assisted-by: ClaudeCode:claude-opus-5-5
…SON values `shown_by_python` caught RecursionError so that a non-JSON value nested deeper than repr could handle was described instead of raising. No parsed document can contain such a value: sets and frozensets are not JSON. Its test only passed where the platform's stack happened to overflow at the chosen depth. Values a document can hold that nest past `JSON_DEPTH` are already reported as too deep by `shown`, without repr. Assisted-by: ClaudeCode:claude-opus-5-5
- `shown` reports "too deep" only for the depth problem itself. A non-JSON value sitting exactly on the last allowed level is now printed instead of being mislabeled. - Complex fill-value rules keep each component problem's `input` and `ctx`; only the `loc` is moved under the component index. - `Schemas.defined` restores every `$defs` entry, name and use count a failed write touched, not only its own name. - The node JSON Schema docstring no longer says `consolidated_metadata` may be `null`. Both the schema and the validator refuse that. - The 382 bugfix changelog fragment is rewritten for users without naming internals. - "modelled" is spelled "modeled" in new text. Assisted-by: ClaudeCode:claude-opus-5-5
…lish Assisted-by: ClaudeCode:claude-fable-5-1
The strict readers refuse documents that some writers produced. Repair is now a separate step that a caller asks for by name: `read_repaired_node_metadata_v3` applies `repair_node_metadata_v3` to undo each known writer bug, then reads the result with `read_node_metadata_v3`, and returns the reading together with the list of repairs made. The set of repairable documents is closed. Each repair applies only when the document's members match that repair's TypedDict: `ZarrV3ZeroChunkArrayMetadataJSON` for the chunk length of 0 along an empty dimension that zarr-python 3.0 and 3.1 wrote, and `ZarrV3NullConsolidatedGroupMetadataJSON` for the `consolidated_metadata: null` that zarr-python 3.0.x wrote. Documents inside consolidated metadata are repaired too. Anything else is left for the strict read to report. Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
…lish Assisted-by: ClaudeCode:claude-fable-5-1
Two scopes are equal when they hold the same definitions under the same kinds and names, regardless of construction order. Equal scopes hash alike. Before this, `Context` was not hashable at all because its tables are mapping proxies. Assisted-by: ClaudeCode:claude-fable-5-1
…Error `Claims` maps `(kind, filed name)` to the definition that read it, or None when nothing claimed it. `Conflict` records one disagreement: the key, what was claimed, what was found, and where. `ScopeConflictError` carries every conflict, as `MetadataValidationError` carries every problem. Assisted-by: ClaudeCode:claude-fable-5-1
…d for each name `claims_of` takes the fields of one reading, nested fields included, and returns the definition that read each name, keyed as the scope files it (raw bits under `r*`). A name claimed two different ways raises `ScopeConflictError`. Assisted-by: ClaudeCode:claude-fable-5-1
…s of a field `refines(field, other)` is True when `field` holds everything `other` holds: the same definition and canonical configuration where both were read, and a read field over an unclaimed one with the same name and JSON (a gain). The reverse direction is a loss, two different definitions are a conflict, and both return False. Two fields that refine each other are equal. Assisted-by: ClaudeCode:claude-fable-5-1
`Context.disagreements(claims)` reports, for each claim, whether this scope reads it the same way, would gain a definition, or conflicts with it (a different definition, or none where one was claimed). `Context.joined` returns the union of several scopes and raises `ScopeConflictError` when two of them file different definitions under one name. Assisted-by: ClaudeCode:claude-fable-5-1
Assisted-by: ClaudeCode:claude-fable-5-1
`refines` compared a read field's written JSON to the unclaimed field's
with Python `==`. That treated `"crc32c"` and `{"name": "crc32c"}` as
different fields and `true` as equal to `1`, so the order was not
transitive across spellings. A gain is now compared the way `Unclaimed`
equality works: by name and by the configuration as JSON text. A refused
field refines only itself, so a read field that holds one still refines
itself. `Claims` is a real type at run time, not a string, so it works in
signatures. The property test now checks reflexivity, transitivity,
mutual refinement equals equality, and substitutivity, over readings of
several documents in several scopes.
Assisted-by: ClaudeCode:claude-fable-5-1
Assisted-by: ClaudeCode:claude-fable-5-1
…eld, UnclaimedField, RefusedField Read, Unclaimed, Refused and their union Resolved become AcceptedField, UnclaimedField, RefusedField and ResolvedField: the noun says these are metadata fields as a scope read them, and Accepted is the opposite of Refused where Read read like a verb. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…n; its check step is private A definition reads a configuration as the model readers read a document: the pair of what was read and every problem, never raising. The type-check step that only read_configuration called is _check_configuration; the public shape check stays zarr_metadata.typed_json.check. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…developers#4495) * fix: zero default scalar for bytes fields in structured dtypes * docs: name changelog entry after zarr-developers#4495
…st (zarr-developers#4496) zarr-developers#4495 also changed the default of `U` fields in structured dtypes from "0" to "". Add `U4` to the regression test and name string fields in the changelog entry. Assisted-by: ClaudeCode:claude-opus-5-5
…developers#4488) Bumps the python-dependencies group with 8 updates: | Package | From | To | | --- | --- | --- | | [google-crc32c](https://github.com/googleapis/python-crc32c) | `1.8.0` | `1.9.0` | | [msgspec](https://github.com/msgspec/msgspec) | `0.21.1` | `0.22.0` | | [coverage](https://github.com/coveragepy/coveragepy) | `7.16.1` | `7.16.2` | | [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.168.1` | `6.168.3` | | [uv](https://github.com/astral-sh/uv) | `0.12.18` | `0.12.21` | | [botocore](https://github.com/boto/botocore) | `1.43.101` | `1.43.106` | | [ruff](https://github.com/astral-sh/ruff) | `0.16.8` | `0.16.9` | | [astroid](https://github.com/pylint-dev/astroid) | `4.3.1` | `4.3.3` | Updates `google-crc32c` from 1.8.0 to 1.9.0 - [Release notes](https://github.com/googleapis/python-crc32c/releases) - [Changelog](https://github.com/googleapis/python-crc32c/blob/main/CHANGELOG.md) - [Commits](https://github.com/googleapis/python-crc32c/commits) Updates `msgspec` from 0.21.1 to 0.22.0 - [Release notes](https://github.com/msgspec/msgspec/releases) - [Changelog](https://github.com/msgspec/msgspec/blob/main/docs/changelog.md) - [Commits](msgspec/msgspec@0.21.1...0.22.0) Updates `coverage` from 7.16.1 to 7.16.2 - [Release notes](https://github.com/coveragepy/coveragepy/releases) - [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst) - [Commits](coveragepy/coveragepy@7.16.1...7.16.2) Updates `hypothesis` from 6.168.1 to 6.168.3 - [Release notes](https://github.com/HypothesisWorks/hypothesis/releases) - [Commits](HypothesisWorks/hypothesis@v6.168.1...v6.168.3) Updates `uv` from 0.12.18 to 0.12.21 - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](astral-sh/uv@0.12.18...0.12.21) Updates `botocore` from 1.43.101 to 1.43.106 - [Commits](boto/botocore@1.43.101...1.43.106) Updates `ruff` from 0.16.8 to 0.16.9 - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.16.8...0.16.9) Updates `astroid` from 4.3.1 to 4.3.3 - [Release notes](https://github.com/pylint-dev/astroid/releases) - [Changelog](https://github.com/pylint-dev/astroid/blob/main/ChangeLog) - [Commits](pylint-dev/astroid@v4.3.1...v4.3.3) --- updated-dependencies: - dependency-name: google-crc32c dependency-version: 1.9.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: msgspec dependency-version: 0.22.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: coverage dependency-version: 7.16.2 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: hypothesis dependency-version: 6.168.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: uv dependency-version: 0.12.21 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: botocore dependency-version: 1.43.106 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: ruff dependency-version: 0.16.9 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: astroid dependency-version: 4.3.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
…pers#4051) * docs: add a deprecation policy to the contributing guide Closes zarr-developers#2924. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: neutral tone * Add missing component from NEP23 * docs: align the deprecation policy with the versioning section and the code Review follow-ups on the deprecation policy: - Explain who each warning class reaches: DeprecationWarning is hidden outside __main__ and test runners, FutureWarning is shown to everyone, which a behavior change needs. - Drop the "release first appeared in" requirement from the message; the release type is decided at release time under EffVer. - Widen the recently-introduced-API exception to match the versioning section, and make the experimental promotion re-export explicitly exempt. - Extend the minimum period to behavior changes, name raised exceptions as observable behavior, and note that a removal in a minor release is not SemVer (review threads). - Rename the changelog fragment to the PR number. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * use more sentences --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Davis Vann Bennett <davis.v.bennett@gmail.com>
Bumps [cairosvg](https://github.com/Kozea/CairoSVG) from 2.9.0 to 2.9.1. - [Release notes](https://github.com/Kozea/CairoSVG/releases) - [Changelog](https://github.com/Kozea/CairoSVG/blob/main/NEWS.rst) - [Commits](Kozea/CairoSVG@2.9.0...2.9.1) --- updated-dependencies: - dependency-name: cairosvg dependency-version: 2.9.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
updates: - [github.com/astral-sh/ruff-pre-commit: v0.16.6 → v0.16.10](astral-sh/ruff-pre-commit@v0.16.6...v0.16.10) - [github.com/DavidAnson/markdownlint-cli2: v0.23.2 → v0.23.3](DavidAnson/markdownlint-cli2@v0.23.2...v0.23.3) - [github.com/numpy/numpydoc: v1.11.0rc0 → v1.11.0](numpy/numpydoc@v1.11.0rc0...v1.11.0) - [github.com/zizmorcore/zizmor-pre-commit: v1.30.0 → v1.30.1](zizmorcore/zizmor-pre-commit@v1.30.0...v1.30.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
* fix: bound memory used by resize chunk cleanup Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix: delete resize chunks with a worker pool instead of lockstep batches Batches sized to async.concurrency were awaited one after another, so each group drained to zero in-flight deletions while waiting on its slowest request. A fixed pool of workers pulling from the shared lazy coordinate iterator keeps `concurrency` deletions in flight with the same memory bound. Also rename the changelog fragment to the PR number. Assisted-by: ClaudeCode:claude-opus-5-5 * fix: cancel remaining resize deletions after the first failure With gather, a failed deletion raised out of resize while the other workers kept deleting in the background, racing any retry. A TaskGroup cancels them; the first exception is unwrapped from the ExceptionGroup so resize keeps raising the store's own error. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(array): move the resize deletion pool into concurrent_foreach and size it to the job Review follow-ups on the bounded resize cleanup: - Add `zarr.core.common.concurrent_foreach`, a lazy bounded-concurrency sibling of `concurrent_map`: `limit` workers share one iterator, the first failure stops further items and cancels in-flight calls, and the failure is re-raised as itself. Simultaneous failures are kept as the `__cause__` group instead of being dropped, and `BaseExceptionGroup` is handled so non-Exception failures are unwrapped too. - A call that fails before its first await no longer fans out to every worker, so a read-only store raises once. - `_resize` computes the number of chunks to delete in O(ndim), skips the pool when it is zero (which subsumes the only-growing guard), and sizes the pool to `min(limit, n_delete)`. - The changelog fragment states the new cancel-on-first-failure semantics. - The pool-saturation test asserts the bound instead of the exact asyncio interleaving; helper semantics are tested in tests/test_common.py. Assisted-by: ClaudeCode:claude-fable-5-1 --------- Co-authored-by: 2160039878-cyber <285580214+2160039878-cyber@users.noreply.github.com> Co-authored-by: OpenAI Codex <noreply@openai.com> Co-authored-by: Davis Vann Bennett <davis.v.bennett@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Bumps [multidict](https://github.com/aio-libs/multidict) from 6.7.1 to 6.9.1. - [Release notes](https://github.com/aio-libs/multidict/releases) - [Changelog](https://github.com/aio-libs/multidict/blob/master/CHANGES.rst) - [Commits](aio-libs/multidict@v6.7.1...v6.9.1) --- updated-dependencies: - dependency-name: multidict dependency-version: 6.9.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Bumps the uv group with 1 update in the /packages/zarr-http-server directory: [multidict](https://github.com/aio-libs/multidict). Updates `multidict` from 6.7.1 to 6.9.1 - [Release notes](https://github.com/aio-libs/multidict/releases) - [Changelog](https://github.com/aio-libs/multidict/blob/master/CHANGES.rst) - [Commits](aio-libs/multidict@v6.7.1...v6.9.1) --- updated-dependencies: - dependency-name: multidict dependency-version: 6.9.1 dependency-type: indirect dependency-group: uv ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.1.8 to 3.1.9. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@3.1.8...3.1.9) --- updated-dependencies: - dependency-name: werkzeug dependency-version: 3.1.9 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
…opers#4487) Bumps the actions group with 1 update in the / directory: [github-community-projects/issue-metrics](https://github.com/github-community-projects/issue-metrics). Updates `github-community-projects/issue-metrics` from 5.0.2 to 5.0.3 - [Release notes](https://github.com/github-community-projects/issue-metrics/releases) - [Commits](github-community-projects/issue-metrics@a7dc2fb...6715286) --- updated-dependencies: - dependency-name: github-community-projects/issue-metrics dependency-version: 5.0.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
…#4483) * fix: reset the codec-pipeline thread pool after fork A forked child inherits codec_pipeline._pool pointing at threads that do not exist in the child, plus the parent's idle-semaphore permits. Submissions up to the permit count are consumed without spawning a thread, so small FusedCodecPipeline reads hang forever. Add an after_in_child hook that drops the inherited pool, size, and lock. The lock is recreated because it may have been held by a thread that no longer exists in the child. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: add changelog entry for zarr-developers#4483 Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: barlowa124 <barlowa124@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
* DOC: remove result="ansi" causing empty code blocks in user guide Closes zarr-developers#3506. Made-with: Cursor * DOC: fix store migration guide: remove broken row, add LRUStoreCache note Removes truncated table row and adds migration note for LRUStoreCache. Closes zarr-developers#2699. Closes zarr-developers#2857. Made-with: Cursor * docs: point LRUStoreCache migration note at CacheStore Move `LRUStoreCache` into the renamed-stores table, since it was never deprecated in 2.x, and point it at the experimental `zarr.experimental.cache_store.CacheStore`. Replace the nonexistent `fsspec.implementations.caching` module with `fsspec.implementations.cached`, and add the changelog fragment. Assisted-by: ClaudeCode:claude-opus-5-5 --------- Co-authored-by: Leo Ji <nuglifeleoji@gmail.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>
…uses a scope of the other A kind declares its format, class CodecDefinition(Definition[C], kind=True, format=3); Context.of refuses definitions of two formats and Context.format says which one a scope reads; ZarrV2Context and ZarrV3Context type the scopes, and every v2 and v3 entry point, resolve, and the pydantic bridge's bare-Context rule raise TypeError for a scope of the other format. A scope that files nothing is of no format and reads in either. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e's inner pipelines are read-only A field handed out by a model or a reading could be edited in place, which left its cached key, == and hash unchanged while codecs == and refines drifted. Fields and stages now hold read-only views at every level, as the models' attributes already do, and pickle as what they were built from. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… refines is transitive through == An accepted field refined an unclaimed one only when their JSON was written alike, while == compares canonical configurations, so a ⊑ b ⊑ u could hold with a ⋢ u. A gain now re-reads what the unclaimed field wrote in the scope the accepted field's own claims make, and holds when that read is the field. Reading one field asks a Scope for its format and claimant, which a Context is and a field's claims are. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…the group lists Two documents for one node path, one in the group's flat listing and another in a listed group's own listing, were accepted; the nested entry is now a problem when the two read otherwise. Arrays compare by array_key_of, what their models compare, and groups by their own members. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…fill value past float64, are read, not raised on An integer of more digits than the interpreter converts to text crashed the readers and the model's key with ValueError while validate_* found no problem; it is an invalid_value problem where it sits, the dimension lengths included. A v2 float or complex fill value written as an integer past the largest float64 raised OverflowError; it reads as the infinity of its sign, as the v3 float types read one. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…nflicts An exception's default reduce calls the constructor with its message, so a pickled or copied ScopeConflictError held each character of the message as a conflict. It reduces to its conflicts. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…dth; ScopeConflictError keeps its notes A number past the largest float32 or float16 reads as the infinity of its sign, as v3 reads one and NumPy stores it, not only past float64; a complex component is held at half the itemsize. A pickled ScopeConflictError keeps its notes and attributes, as a ValueError does. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…l consolidated metadata too; depth notes Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… paths and hierarchy, sized fill values, enum values, scope invariants A conflict names the field as the document writes it and tells two definitions of one name apart. is_group_metadata_v3 asks tuples of the documents a group lists. A .zmetadata key with a '.' or '..' segment is a problem, and the nodes of a .zmetadata make a hierarchy, as a v3 group's consolidated metadata does. A void or struct fill value decodes to exactly the item's size and a byte string's to at most it. A str, int or float subclass, a StrEnum or IntEnum member, is refined to the value JSON writes for it. Context(tables) keeps the invariants Context.of keeps. The raw-bits pattern's docstring says what the package does: a hundred digits is more than any size, so more is a name. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… reading and the filing hook as the package names things DataTypeField and the other v3 field aliases are ZarrV3DataTypeField and so on, as ZarrV2DataTypeField and ZarrV3MetadataFieldJSON are named; ZarrV3UnknownNodeReading is ZarrV3UnknownNodeMetadataReading, as its siblings are; a kind's spelled hook is filed_under, which says what it answers. ZarrV3NodeMetadataInput says what an Input is beside JSON and Update. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…wall-clock bound a slow runner meets pymdown-extensions 12.2 changed the Highlight extension's constructor, which mkdocstrings 1.0.6 called the old way, so the Read the Docs build of the package failed; 1.0.7 takes the change, pinned in the package and the root together, as the docs groups say. The hierarchy timing test allows 60 seconds for work a slow runner took 10 for; the quadratic walk it guards against takes minutes. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…document-context-stack
Owner
Author
|
🤖 AI text below 🤖 The agent closed this fork PR: its branch is now reviewed upstream as zarr-developers#4490, which carries the whole stack with the changelog consolidated under that number. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
This is the zarr-metadata work from fork PRs #370 through #397 as one pull request, with upstream
mainmerged in. It is the whole (document, context) design: every metadata field is read against a definition in a scope, and every model is its document and the scope it was read in.Each field of a document — a v3 data type, chunk grid, chunk key encoding, codec or storage transformer; a v2 dtype or numcodecs codec — is read as
Readby the definition in scope that claims its name,Unclaimedwhen nothing does, orRefusedwith every problem.read_array_metadata_v3,read_group_metadata_v3andread_array_metadata_v2read a document once and return everything the read found, the model among it. A model is built only from a document in a scope (CORE_AND_EXTENSIONSorCORE_V2by default) and is never invalid; two models are equal when their documents mean the same in their scopes;updatereads new members in the model's own scope,with_contextandrefined_inread the document in another,refinesorders models by information. Scopes compare, join and report disagreements. JSON Schemas are exported for a TypedDict, a field in a scope and azarr.json. Known writer bugs are undone below the strict model byread_repaired_node_metadata_v3andread_repaired_consolidated_metadata_v2. The changelog fragments underpackages/zarr-metadata/changes/describe every change for users.Breaking: models are no longer dataclasses built from typed members;
construct, the...PartialTypedDicts,ZarrV3NamedConfig/ZarrV3MetadataFieldand the requiredcontextonupdateare gone; a structured v2 dtype withfill_value: 0is refused.The stacked PRs this replaces (bottom-up)
#370, #371, #372, #373, #374, #376, #377, #378, #379, #381, #382, #391, #392, #393, #395, #396, #397. Each carries its own description and review history; their fragments are in this branch under their numbers.
🤖 Generated with Claude Code