feat: url-pipeline core — parser, adapter ABC, registry, store hooks

changes/4192.feature.md

@@ -0,0 +1,25 @@
+Added core support for URL pipelines (https://github.com/jbms/url-pipeline):
+`|`-chained URLs that address zarr data through nested storage layers, e.g.
+`s3://bucket/data.zip|zip:|zarr3:`. This PR adds the parser, the single-method
+`zarr.abc.url_pipeline.URLPipelineAdapter` interface, and the
+`zarr.url_adapters` entry-point group through which third-party packages
+(e.g. Icechunk) register adapters for their own schemes. Adapters for a scheme
+are loaded lazily and individually. Builtin adapters (`zip:`,
+`zarr2:`/`zarr3:`) follow in separate pull requests.
+
+Behavior notes:
+
+- The `|` character is now reserved as the pipeline delimiter in every string
+  store specification, and no percent-escape is decoded; pass a `pathlib.Path`
+  to address a local file whose name contains `|`. URLs without a `|` (and
+  without a registered root adapter scheme) are handled exactly as before —
+  registered adapters cannot intercept zarr's native `file:`/`memory:`
+  routing, and fsspec chained URLs (`zip::s3://...`) keep flowing to fsspec.
+- Inside a pipeline, `memory:` and `file:` roots follow the URL pipeline
+  spec's semantics (spelling equivalences; `file:` must be absolute, with at
+  most a `localhost` authority).
+- Mode `"a"` (open-or-create, the `zarr.open` default) on a *read-only* store
+  now serves the "open" half instead of raising upfront, for all stores;
+  unambiguous write modes (`"w"`, `"w-"`, `"r+"`) still raise.
+- For root-adapter URLs (e.g. `gh://org/repo`), `storage_options` are handed
+  to the adapter and are not validated as used by `make_store`.

docs/api/zarr/abc/index.md

@@ -11,3 +11,4 @@ Abstract base classes for extending Zarr-Python.
 - **[zarr.abc.metadata](./metadata.md)** - Creating metadata classes compatible with the Zarr API
 - **[zarr.abc.numcodec](./numcodec.md)** - Protocols and classes for modeling codec interface used by numcodecs
 - **[zarr.abc.store](./store.md)** - ABC for implementing Zarr stores and managing getting and setting bytes in a store
+- **[zarr.abc.url_pipeline](./url_pipeline.md)** - ABC for implementing [URL pipeline](https://github.com/jbms/url-pipeline) adapters

docs/api/zarr/abc/url_pipeline.md

@@ -0,0 +1,5 @@
+---
+title: url_pipeline
+---
+
+::: zarr.abc.url_pipeline

docs/user-guide/storage.md

@@ -103,6 +103,31 @@ print(group)
 - a [`Store`][zarr.abc.store.Store] or [`StorePath`][zarr.storage.StorePath] -
   see explicit store creation below.
 
+## URL Pipelines {#user-guide-url-pipelines}
+
+Zarr supports [URL pipelines](https://github.com/jbms/url-pipeline): `|`-chained URLs
+that address zarr data through nested storage layers, read left to right. The first
+sub-URL locates a resource with a conventional URL; each subsequent sub-URL names an
+*adapter* that reinterprets everything to its left (e.g.
+`s3://bucket/data.zip|zip:|zarr3:`). Adapters are provided by packages through the
+`zarr.url_adapters` entry-point group — see
+[`zarr.abc.url_pipeline`][zarr.abc.url_pipeline] for the adapter interface. Builtin
+adapters (`zip:`, `zarr2:`/`zarr3:`) are under development and will expand this
+section. URLs without a `|` (and without a registered root scheme) are handled
+exactly as before.
+
+`storage_options` passed to `zarr.open` apply to the *root* sub-URL (e.g. fsspec
+options for `s3://...`); adapters may consume adapter-specific, namespaced keys.
+Non-dict forms of `storage_options` are reserved for future per-segment
+configuration.
+
+The `|` character is reserved as the pipeline delimiter in every string store
+specification, and no percent-escape is decoded: to address a local file whose
+*name* contains `|` (or `#`), pass a `pathlib.Path` instead of a string.
+Registered adapters cannot intercept zarr's native `file:` and `memory:` root
+schemes, and fsspec's chained-URL syntax (`zip::s3://...`) keeps flowing to
+fsspec.
+
 ## Explicit Store Creation
 
 In some cases, it may be helpful to create a store instance directly. Zarr-Python offers

mkdocs.yml

@@ -43,6 +43,7 @@ nav:
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.metadata</code>': api/zarr/abc/metadata.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.numcodec</code>': api/zarr/abc/numcodec.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.store</code>': api/zarr/abc/store.md
+        - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.url_pipeline</code>': api/zarr/abc/url_pipeline.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.api</code>':
         - api/zarr/api/index.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.api.asynchronous</code>': api/zarr/api/asynchronous.md

src/zarr/abc/url_pipeline.py

@@ -0,0 +1,256 @@
+"""
+Abstract base class and data model for URL pipeline adapters.
+
+A URL pipeline is a `|`-separated chain of sub-URLs, read outer-to-inner,
+as specified by https://github.com/jbms/url-pipeline. The first sub-URL (the
+*root*) locates a resource using a conventional URL, and each subsequent
+sub-URL names an *adapter* that reinterprets everything to its left:
+
+    s3://bucket/data.zip|zip:path/inside|zarr3:
+
+Third-party packages provide adapters by subclassing
+[`URLPipelineAdapter`][zarr.abc.url_pipeline.URLPipelineAdapter] and
+registering the class under the `zarr.url_adapters` entry-point group,
+using the URL scheme as the entry-point name.
+"""
+
+from __future__ import annotations
+
+import enum
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from typing import TYPE_CHECKING, Any
+
+from zarr.errors import URLPipelineError
+
+if TYPE_CHECKING:
+    from zarr.abc.store import Store
+    from zarr.core.common import AccessModeLiteral, ZarrFormat
+
+__all__ = [
+    "AdapterResolution",
+    "PipelineContext",
+    "PipelineSegment",
+    "URLPipelineAdapter",
+]
+
+
+class _Unset(enum.Enum):
+    token = 0
+
+
+_UNSET = _Unset.token
+
+
+@dataclass(frozen=True)
+class PipelineSegment:
+    """
+    One `|`-delimited sub-URL of a URL pipeline.
+
+    Attributes
+    ----------
+    scheme : str
+        The lowercased URL scheme. Empty string only for a schemeless root
+        (a bare local path), which is treated as opaque text.
+    body : str
+        The text after `scheme:` and before any `?`. Interpretation is
+        scheme-defined; it is **not** URL-normalized, so case-significant
+        content (e.g. icechunk snapshot IDs) is preserved.
+    query : str | None
+        The raw query string after `?`, or None. Interpretation is
+        scheme-defined.
+    raw : str
+        The exact original sub-URL text, preserved for lossless
+        reconstruction of the pipeline.
+    """
+
+    scheme: str
+    body: str
+    query: str | None
+    raw: str
+
+    def __str__(self) -> str:
+        return self.raw
+
+
+@dataclass(frozen=True)
+class AdapterResolution:
+    """
+    The result of resolving a URL pipeline (or a prefix of one).
+
+    Attributes
+    ----------
+    store : Store
+        The resolved store.
+    path : str
+        Residual path *within* the store that the pipeline addresses
+        (e.g. `"path/to/node"` for `...|icechunk://tag.v1/path/to/node`).
+        Empty string when the pipeline addresses the store root.
+    zarr_format : ZarrFormat | None
+        Zarr format selected by a format segment (`zarr2:`/`zarr3:`),
+        or None if unspecified. A wrapper adapter that re-wraps a preceding
+        resolution must carry every field it does not change forward —
+        prefer `dataclasses.replace(preceding, store=..., path=...)` over
+        reconstructing, so fields added later are never silently dropped.
+    """
+
+    store: Store
+    path: str = ""
+    zarr_format: ZarrFormat | None = None
+
+
+@dataclass(frozen=True)
+class PipelineContext:
+    """
+    Context handed to a [`URLPipelineAdapter`][zarr.abc.url_pipeline.URLPipelineAdapter]
+    describing the pipeline to the left of its segment.
+
+    Attributes
+    ----------
+    preceding : tuple[PipelineSegment, ...]
+        The parsed sub-URLs to the left of the adapter's segment, outer to
+        inner. Empty when the adapter's segment is the pipeline root.
+    mode : AccessModeLiteral | None
+        The access mode requested by the caller (e.g. `zarr.open(mode=...)`),
+        or None when unspecified. Adapters for read-only resources should
+        raise for unambiguous write modes (`"w"`, `"w-"`, `"r+"`) and
+        open read-only otherwise. `"a"` (the `zarr.open` default) means
+        open-or-create: read-only adapters serve the "open" half, and any
+        subsequent write fails at the store level.
+    storage_options : dict[str, Any] | None
+        Options passed by the caller. By convention these configure the
+        *root* sub-URL (e.g. fsspec options); adapters may consume
+        adapter-specific keys, and should namespace them (e.g.
+        `myscheme_credentials`) to avoid collisions with other segments'
+        backends. An adapter that consumes keys should strip them before
+        resolving the rest of the pipeline, by passing the reduced mapping
+        to [`resolve_preceding`][zarr.abc.url_pipeline.PipelineContext.resolve_preceding].
+        Non-dict forms of the caller-facing `storage_options` argument are
+        reserved for future per-segment configuration (one mapping per
+        pipeline segment); this attribute will remain a single mapping —
+        the one addressed to this adapter's segment.
+    """
+
+    preceding: tuple[PipelineSegment, ...]
+    mode: AccessModeLiteral | None
+    storage_options: dict[str, Any] | None
+
+    @property
+    def read_only(self) -> bool:
+        """
+        True when the caller requires a read-only store (`mode == "r"`).
+
+        Adapters must construct their store read-only when this is set
+        (the resolver enforces it afterwards); when it is False, they may
+        construct a writable store if the underlying resource supports
+        writing.
+        """
+        return self.mode == "r"
+
+    @property
+    def preceding_url(self) -> str:
+        """
+        The pipeline to the left of this segment, reconstructed exactly.
+
+        An adapter that consumes this string instead of calling
+        [`resolve_preceding`][zarr.abc.url_pipeline.PipelineContext.resolve_preceding]
+        takes ownership of the *entire* preceding pipeline: it must
+        validate every preceding segment itself and raise
+        [`URLPipelineError`][zarr.errors.URLPipelineError] for segments it
+        does not understand, so that no segment is ever silently ignored.
+        """
+        return "|".join(segment.raw for segment in self.preceding)
+
+    async def resolve_preceding(
+        self,
+        *,
+        mode: AccessModeLiteral | _Unset | None = _UNSET,
+        storage_options: dict[str, Any] | _Unset | None = _UNSET,
+    ) -> AdapterResolution:
+        """
+        Resolve the preceding pipeline into a store.
+
+        This is the entry point for *wrapper* adapters (e.g. `zip:`) that
+        operate on the resource produced by the segments to their left. It
+        composes with any preceding adapters, because each segment is
+        resolved by its own adapter. Adapters backed by their own I/O
+        machinery (e.g. `icechunk:`) may instead consume
+        [`preceding_url`][zarr.abc.url_pipeline.PipelineContext.preceding_url]
+        and never materialize the intermediate store — subject to the
+        ownership contract documented there.
+
+        Parameters
+        ----------
+        mode : AccessModeLiteral | None, optional
+            Override the mode used to resolve the preceding pipeline.
+            Wrapper adapters that only read the preceding resource should
+            pass `mode="r"` so the root is opened read-only and without
+            create-on-open side effects, regardless of the caller's mode.
+            When omitted, the caller's mode is used.
+        storage_options : dict | None, optional
+            Override the options forwarded to the preceding pipeline. An
+            adapter that consumed adapter-specific keys should pass the
+            remaining mapping here (or None when nothing remains), so the
+            root store never sees keys that were not addressed to it.
+            When omitted, the caller's options are forwarded unchanged.
+        """
+        from zarr.storage._url_pipeline import _resolve
+
+        if not self.preceding:
+            raise URLPipelineError(
+                "this adapter segment is at the pipeline root; "
+                "there is no preceding sub-URL to resolve"
+            )
+        return await _resolve(
+            self.preceding,
+            mode=self.mode if isinstance(mode, _Unset) else mode,
+            storage_options=(
+                self.storage_options if isinstance(storage_options, _Unset) else storage_options
+            ),
+        )
+
+
+class URLPipelineAdapter(ABC):
+    """
+    Handler for one URL pipeline scheme.
+
+    Subclasses implement a single classmethod,
+    [`open_pipeline_segment`][zarr.abc.url_pipeline.URLPipelineAdapter.open_pipeline_segment],
+    and are registered under the `zarr.url_adapters` entry-point group with
+    the URL scheme as the entry-point name:
+
+        [project.entry-points."zarr.url_adapters"]
+        mypackage.myscheme = "mypackage.zarr_adapter:MyAdapter"
+
+    Nonstandard schemes should be vendor-prefixed (`vendor.scheme`) per the
+    URL pipeline specification.
+
+    An adapter is used in two positions:
+
+    - as an *adapter segment*: `s3://bucket/repo|icechunk://tag.v1` — the
+      context carries the preceding sub-URLs;
+    - as a *root scheme*: `gh://org/repo` — `context.preceding` is empty.
+    """
+
+    @classmethod
+    @abstractmethod
+    async def open_pipeline_segment(
+        cls, segment: PipelineSegment, context: PipelineContext
+    ) -> AdapterResolution:
+        """
+        Resolve `segment` (in the context of the pipeline to its left)
+        into a store and an optional residual path within that store.
+
+        The returned store must already be open and must honor
+        `context.read_only` (the resolver additionally enforces it by
+        downgrading — or rejecting — a writable store when the caller
+        required read-only).
+
+        This coroutine runs on zarr's internal I/O event loop. It must not
+        block (do I/O through async APIs or a thread executor) and must not
+        call zarr's synchronous API (`zarr.open`, `Group.open`, or anything
+        else that uses `zarr.core.sync.sync`) — doing so raises
+        `SyncError`. To open the preceding pipeline, use
+        [`resolve_preceding`][zarr.abc.url_pipeline.PipelineContext.resolve_preceding].
+        """
+        ...