Skip to content

Add the offset chunk key encoding and Array.prepend - #4516

Open
rhoadesScholar wants to merge 6 commits into
zarr-developers:mainfrom
rhoadesScholar:offset-chunk-key-encoding
Open

rhoadesScholar wants to merge 6 commits into
zarr-developers:mainfrom
rhoadesScholar:offset-chunk-key-encoding

Conversation

@rhoadesScholar

@rhoadesScholar rhoadesScholar commented Oct 15, 2026 •

Copy link
Copy Markdown

Adds the offset chunk key encoding (zarr-developers/zarr-extensions#80): the default key
of chunk_coords + offset, one integer per dimension, any sign. Under it an array grows in
the negative direction of an axis by whole chunks as a metadata-only change, and
Array.prepend(data, axis), the mirror of Array.append, does exactly that: shape grows,
offset drops, one zarr.json write, every existing chunk keeps its key, then data is
written in front.

z = zarr.create_array(store, shape=(6, 4), chunks=(3, 4), dtype="int32",
                      chunk_key_encoding={"name": "offset", "offset": (0, 0)})
z[:] = 1
z.prepend(np.full((6, 4), 2, dtype="int32"))   # shape (12, 4); keys c/-2/0, c/-1/0, c/0/0, c/1/0

Before / after. Today the only ways to add data before index 0 touch every chunk: rename
each chunk key (filesystem only), get/set/delete each chunk through the store API, or copy the
array. Measured on one machine (NVMe, 64³ uint8 chunks, two chunk rows added in front):

chunks rename each file store get/set/delete copy to new array metadata edit (prepend minus the write)
512 0.03 s 0.76 s 0.32 s 0.004 s
4 096 0.13 s 5.9 s 3.1 s 0.006 s
32 768 (8 GiB) 1.0 s 44 s 26 s 0.016 s

Script: https://github.com/rhoadesScholar/zarr-chunk-offset/tree/main/bench. The test
test_prepend_writes_in_front_and_leaves_every_existing_chunk_untouched asserts the property
directly: every pre-existing key holds the same bytes after the growth, and only the new keys
are added.

Design

  • OffsetChunkKeyEncoding(offset, separator="/") in zarr.core.chunk_key_encodings,
    registered as "offset". The offset length is checked against the shape when
    ArrayV3Metadata is built, so a wrong-rank offset fails at creation, not at the first key.
  • The ChunkKeyEncodingParams shortcut ({"name": ..., "separator": ...}) now carries
    offset into the configuration; before, that branch rebuilt the configuration with only
    separator.
  • prepend mirrors append (same signature, same shape check) with one constraint the
    format imposes: the data must span a whole number of chunks (shards, when sharded) along the
    axis, since a sub-chunk growth cannot leave the existing chunks in place. On a default or
    v2 array it raises a TypeError naming the encoding rather than renaming chunks behind
    the caller's back; switching an existing array is a one-line metadata rewrite that the
    plugin's adopt performs and that could become a core helper later.
  • Readers without the encoding refuse the array (unknown name; must_understand cannot be
    false for a chunk key encoding), which is the reason this is a new name and not a field on
    the regular grid.

Tests: 20 new (encoding round trips, negative components, rank checks, the shortcut, prepend
on plain, sharded, dot-separated and negative-axis arrays, refusals); tests/test_array.py, tests/test_metadata, tests/test_api and
tests/test_chunk_key_encodings.py pass; mypy and ruff clean. Docs: a "Prepending"
paragraph in the arrays guide and a changelog fragment.

Issues

Refs zarr-developers/zarr-specs#9 and zarr-developers/zarr-extensions#30; the registration PR
is zarr-developers/zarr-extensions#80.

rhoadesScholar and others added 3 commits October 15, 2026 21:34
The `offset` encoding stores each chunk under the `default` key of its
grid coordinate plus a fixed per-dimension integer offset, so that an
array grows in the negative direction of an axis by whole chunks as a
metadata-only change: `shape` grows, `offset` drops, and every existing
chunk keeps its key. `Array.prepend_chunks` performs that edit and
`Array.prepend` grows and writes in one call; both refuse an array on
another encoding rather than rename its chunks.

The offset must have one entry per dimension, checked when the metadata
is built; the `ChunkKeyEncodingParams` shortcut carries `offset` through
to the configuration instead of dropping it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CSaEKF9nB55gU7PQgQ22X5
The store key addresses a shard, so the unit of growth is the chunk
grid's shape, not the inner chunk; and a negative axis has to be
normalised before the preserved-shape check and the write selection use
it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CSaEKF9nB55gU7PQgQ22X5
Drops the public prepend_chunks; Array.prepend grows the array by the
data's extent (whole chunks, or shards when sharded) and writes it.
Docstrings and the docs paragraph are cut to what the API needs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CSaEKF9nB55gU7PQgQ22X5
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CSaEKF9nB55gU7PQgQ22X5
@codecov

codecov Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.74%. Comparing base (bd0dc84) to head (7501ff3).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4516      +/-   ##
==========================================
+ Coverage   94.71%   94.74%   +0.02%     
==========================================
  Files          94       94              
  Lines       13663    13741      +78     
==========================================
+ Hits        12941    13019      +78     
  Misses        722      722              
Files with missing lines Coverage Δ
src/zarr/core/array.py 98.48% <100.00%> (+0.04%) ⬆️
src/zarr/core/chunk_key_encodings.py 99.08% <100.00%> (+0.51%) ⬆️
src/zarr/core/metadata/v3.py 96.92% <100.00%> (+0.01%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rhoadesScholar
rhoadesScholar marked this pull request as ready for review October 16, 2026 00:13
@rhoadesScholar

Copy link
Copy Markdown
Author

Ready for review; the registry entry is zarr-developers/zarr-extensions#80. cc @d-v-b

rhoadesScholar and others added 2 commits October 16, 2026 00:45
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CSaEKF9nB55gU7PQgQ22X5
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CSaEKF9nB55gU7PQgQ22X5
@d-v-b

d-v-b commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

thanks for this PR! I really want to see this feature added to zarr! But I think we want to be sure that we support negative chunk indices consistently across the ecosystem, so I'll hold off on reviewing until we get alignment from the other implementers on the zarr-extensions PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants