feat(crawl): add ZenRowsCrawlClient for the Crawl API

DEVELOPMENT.md

@@ -19,14 +19,19 @@ Makefile targets below.
 
 ```
 src/zenrows/
-├── __init__.py               # re-exports both clients
+├── __init__.py               # re-exports the clients
 ├── client.py                 # ZenRowsClient (legacy sync scraper)
-└── batch/
-    ├── __init__.py           # ZenRowsBatchClient + key models
+├── batch/
+│   ├── __init__.py           # ZenRowsBatchClient + key models
+│   ├── client.py             # hand-written typed facade
+│   ├── _transport.py         # httpx wrapper, RFC 7807 → exceptions
+│   ├── errors.py             # BatchAPIError, ProblemDetail
+│   └── models.py             # GENERATED — pydantic v2 (do not edit)
+└── crawl/
+    ├── __init__.py           # ZenRowsCrawlClient + models
     ├── client.py             # hand-written typed facade
-    ├── _transport.py         # httpx wrapper, RFC 7807 → exceptions
-    ├── errors.py             # BatchAPIError, ProblemDetail
-    └── models.py             # GENERATED — pydantic v2 (do not edit)
+    ├── errors.py             # CrawlAPIError
+    └── models.py             # hand-written pydantic v2 models
 ```
 
 The Batch SDK is split deliberately:
@@ -42,12 +47,21 @@ This way the wire types stay in lockstep with the OpenAPI document
 while the ergonomic surface (method names, helpers, retries, URL
 override) stays in our control.
 
+The Crawl client uses the shared HTTP transport (`_transport.py`: key
+header, retries, problem+json) with `CrawlAPIError.from_response` as
+its error mapping, and the shared `poll_until` loop. Its models are
+hand-written: the surface is a few small schemas, so there is no Crawl
+codegen step.
+Keep them tolerant: unknown fields are ignored, and every response enum
+gets `_missing_ = classmethod(open_enum_missing)`.
+
 ## Common tasks
 
 | Make target     | What it does |
 |-----------------|--------------|
 | `make sync`     | `uv sync --all-extras` |
-| `make test`     | `uv run pytest` |
+| `make test`     | `uv run pytest` (offline; e2e tests deselected) |
+| `make test-e2e` | End-to-end tests against a live API (see below) |
 | `make check`    | `ty check` + `ruff check` + `ruff format --check` (CI mode) |
 | `make typecheck`| `ty check src` (static types; `models.py` excluded) |
 | `make lint`     | `ruff check --fix` |
@@ -56,6 +70,38 @@ override) stays in our control.
 | `make build`    | Build wheel + sdist via hatchling |
 | `make clean`    | Drop caches + build outputs |
 
+## Running the Crawl e2e test
+
+`tests/e2e/test_crawl_e2e.py` drives a real crawl through the client:
+create (with `include_patterns`, `output_format="html"`), wait, read the
+results, one page and the NDJSON download, list, stop, and a 404. It
+costs a few credits. It is marked `e2e`, which `make test` deselects,
+and it skips unless the three required variables below are set.
+
+| Variable | |
+|---|---|
+| `ZENROWS_API_KEY` | required: a key with Crawl access |
+| `ZENROWS_CRAWL_BASE_URL` | required: the API base, e.g. `https://api.zenrows.com/v1`; the test passes it as `base_url=` |
+| `ZENROWS_E2E_CRAWL_URL` | required: the page the crawl starts from |
+| `ZENROWS_E2E_CRAWL_INCLUDE` | optional: an include pattern; every result must contain it |
+
+```bash
+export ZENROWS_API_KEY=zr_...                       # a key with Crawl access
+export ZENROWS_CRAWL_BASE_URL=https://api.zenrows.com/v1
+export ZENROWS_E2E_CRAWL_URL=https://example.com/products/
+export ZENROWS_E2E_CRAWL_INCLUDE=/product/          # optional
+make test-e2e
+```
+
+Pick a start page whose links include a few pages matching the pattern:
+the test crawls at depth 1 with `max_items=3` and `output_format="html"`.
+
+To test against a local or staging deployment, point
+`ZENROWS_CRAWL_BASE_URL` at its `/v1` base instead. When the account
+has reached its limit of active jobs (429 `too_many_crawls`: 3 by default,
+shared with its Batch jobs), the test waits and retries (up to 5 minutes)
+before creating its crawl.
+
 ## Refreshing the OpenAPI spec
 
 `docs/openapi.yaml` is the SDK-local copy of the spec. `make generate`

Makefile

@@ -1,13 +1,18 @@
-.PHONY: install sync test lint format typecheck check generate docs clean build bump release
+.PHONY: install sync test test-e2e lint format typecheck check generate docs clean build bump release
 
 # Bootstrap: install + dev deps, build the local venv.
 install sync:
 	uv sync --all-extras
 
-# Run the suite.
+# Run the suite (offline; e2e tests are deselected).
 test:
 	uv run pytest
 
+# Run the end-to-end tests against a live API. Needs ZENROWS_API_KEY,
+# ZENROWS_CRAWL_BASE_URL and ZENROWS_E2E_CRAWL_URL; see DEVELOPMENT.md.
+test-e2e:
+	uv run pytest -m e2e -s tests/e2e
+
 # Static type check (ty — Astral). Shipped surface only; tests are
 # covered by the suite. Generated models.py is excluded in pyproject.
 typecheck:

README.md

@@ -10,14 +10,17 @@
 SDK to access [Zenrows](https://www.zenrows.com/) APIs directly from Python.
 Zenrows handles proxies rotation, headless browsers, and CAPTCHAs for you.
 
-This package ships two clients:
+This package ships three clients:
 
   - **`ZenRowsClient`** — the original synchronous scraping client.
     One URL in, one HTML/JSON response out. Best for ad-hoc scraping.
   - **`ZenRowsBatchClient`** — the async-job / Batch API client. Submit
     thousands of URLs as one job, poll for results, optionally upload a
     CSV of URLs in one call. Best for offline / bulk pipelines.
     _(Private beta — [contact support](mailto:support@zenrows.com) for access.)_
+  - **`ZenRowsCrawlClient`** — the Crawl API client. Give it one start
+    URL; it follows links and returns the URLs (and, optionally, the
+    pages) it finds behind it. _(Beta.)_
 
 ## Table of Contents
 
@@ -37,6 +40,10 @@ This package ships two clients:
   - [Act on an id without a GET](#act-on-an-id-without-a-get)
   - [Scheduled jobs & webhooks](#scheduled-jobs--webhooks)
   - [Error handling](#error-handling)
+- [Quickstart — Crawl API (beta) (`ZenRowsCrawlClient`)](#quickstart--crawl-api-beta-zenrowscrawlclient)
+  - [Read the pages](#read-the-pages)
+  - [List and stop crawls](#list-and-stop-crawls)
+  - [Crawl errors](#crawl-errors)
 - [Contributing](#contributing)
 - [License](#license)
 
@@ -485,6 +492,115 @@ The full Batch surface (jobs, runs, tasks, results, content, history,
 file_inputs, HMAC keys) is reachable via methods on `ZenRowsBatchClient`.
 See `src/zenrows/batch/client.py` or `help(ZenRowsBatchClient)`.
 
+## Quickstart — Crawl API (beta) (`ZenRowsCrawlClient`)
+
+Crawl is in Beta: its API may still change.
+
+A crawl starts from one URL, follows the links on each page up to
+`depth` hops, and keeps the URLs that match your patterns, up to
+`max_items`. It stays on the start URL's registrable domain;
+subdomains count. It runs in the background: `create` returns at once with
+`status="running"`, and `wait` blocks until it ends. Without
+`output_format` a crawl returns URLs only; `output_format="html"` also
+returns each kept URL's page.
+
+```python
+import os
+from zenrows import ZenRowsCrawlClient
+
+client = ZenRowsCrawlClient(api_key=os.environ["ZENROWS_API_KEY"])
+
+crawl = client.create(
+    "https://example.com/products/",
+    depth=1,                         # link hops from the start URL
+    max_items=50,                    # stop after keeping 50 URLs (default 10)
+    max_pages=100,                   # stop after fetching 100 pages (default 10)
+    include_patterns=["/product/"],  # keep only URLs containing this
+)
+crawl = client.wait(crawl.crawl_id)                 # Crawl, terminal (timeout 600 s)
+print(crawl.status.value, crawl.stop_reason, crawl.coverage)
+
+for result in client.iter_results(crawl.crawl_id):  # follows next_cursor
+    print(result.url)
+```
+
+`wait` returns a `failed` crawl rather than raising (read `crawl.error`),
+and raises `WaiterTimeout` on timeout without stopping the crawl.
+`iter_results` does not poll: it returns at the first empty page, so on a
+running crawl it yields only the URLs kept so far. Call `wait` first to
+read every result. `get(crawl_id, cursor=..., limit=...)` returns one raw
+page.
+
+### Read the pages
+
+Pass `output_format="html"` to also fetch the page of every kept URL:
+
+```python
+crawl = client.create(start_url, depth=1, output_format="html")
+client.wait(crawl.crawl_id)
+
+for result in client.iter_results(crawl.crawl_id):
+    if result.content_id:                           # content_status == fetched
+        html = client.get_content(crawl.crawl_id, result)
+
+# Or everything in one NDJSON file: {"url", "content_status", "content"} per line.
+saved = client.download(crawl.crawl_id, "out/crawl.jsonl")
+print(saved.path, saved.status)                    # status from X-Crawl-Status
+
+with client.iter_download(crawl.crawl_id) as download:  # streamed, parsed
+    print(download.status)
+    for line in download:
+        print(line.url, len(line.content or ""))
+```
+
+Both downloads carry the crawl's `status` when the file was read. On a
+`running` crawl the file holds only the results kept so far.
+
+### List and stop crawls
+
+```python
+for c in client.iter_crawls():          # newest first, without results
+    print(c.crawl_id, c.status.value)
+
+client.stop(crawl_id)                   # idempotent; read final counts with get()
+```
+
+### Crawl errors
+
+Every non-2xx raises `CrawlAPIError`. Branch on `status_code` and `code`:
+
+```python
+from zenrows.crawl import CrawlAPIError
+
+try:
+    client.create(start_url, depth=1)
+except CrawlAPIError as exc:
+    if exc.not_enabled:                 # 403 REQS008
+        print("Crawl is not enabled for this account")
+    elif exc.code == "too_many_crawls": # 429: active jobs limit reached
+        print("retry in", exc.retry_after, "s")
+    elif exc.status_code in (400, 422):
+        print(exc.detail)
+    else:
+        raise
+```
+
+| Status | `code` | What to do |
+|---|---|---|
+| 400 | `invalid_request`, `unknown_parameter`, `invalid_cursor` | Fix the request. Do not retry it as is. |
+| 403 | `REQS008` | Crawl is not enabled for the account (`exc.not_enabled`). |
+| 404 | `crawl_not_found`, `content_not_found` | Check the id. |
+| 409 | `idempotency_request_in_flight` | Retry after the first request with this key finishes. |
+| 422 | `invalid_parameter`, `invalid_start_url`, `domain_not_allowed` | Fix the request. Do not retry it as is. |
+| 422 | `idempotency_key_reused` | Use a new key, or no key. Do not retry it as is. |
+| 429 | `too_many_crawls` | The account has reached its limit of active jobs (3 by default), shared with its Batch jobs. Retry after `exc.retry_after` seconds. `create` does not retry it. |
+
+`code` is None when the response has no problem body with a code (for
+example a 502 from a proxy). A crawl that ended `failed` is not an error
+response: read `crawl.error.code` and `crawl.error.detail` on it. Response enums (`CrawlStatus`, `StopReason`,
+`ContentStatus`, ...) are open: a value added later parses as an
+`UNKNOWN` member that keeps the raw value.
+
 ## Contributing
 Pull requests are welcome. For significant changes, please open an issue first to discuss what you would like to change.
 

examples/11_crawl.py

@@ -0,0 +1,73 @@
+"""11: Crawl a site → wait → read URLs and pages. Crawl is in Beta.
+
+Demonstrates the Crawl client (`ZenRowsCrawlClient`):
+  - `create(url, depth=..., include_patterns=..., output_format="html")`
+    starts a crawl and returns at once, `running`.
+  - `wait(crawl_id)` blocks until it ends and returns the final `Crawl`
+    (a `failed` crawl is returned, with `error` set).
+  - `iter_results(crawl_id)` pages through the kept URLs.
+  - `get_content(crawl_id, result)` reads one kept URL's HTML.
+  - `download(crawl_id, path)` saves every result as NDJSON and
+    returns the path and the crawl's status.
+
+Run with:
+    export ZENROWS_API_KEY=zr_...
+    python examples/11_crawl.py --url https://example.com/products/
+"""
+
+import argparse
+import os
+
+from zenrows import ZenRowsCrawlClient
+from zenrows.crawl import CrawlAPIError, CrawlStatus
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument("--url", required=True, help="the page to start from")
+    parser.add_argument("--depth", type=int, default=1)
+    parser.add_argument("--max-items", type=int, default=5)
+    parser.add_argument(
+        "--include", action="append", help="keep URLs containing this (default /product/)"
+    )
+    parser.add_argument("--out", default="crawl.jsonl")
+    args = parser.parse_args()
+
+    client = ZenRowsCrawlClient(api_key=os.environ["ZENROWS_API_KEY"])
+
+    try:
+        crawl = client.create(
+            args.url,
+            depth=args.depth,
+            max_items=args.max_items,
+            include_patterns=args.include or ["/product/"],
+            output_format="html",
+        )
+    except CrawlAPIError as exc:
+        if exc.not_enabled:
+            raise SystemExit("Crawl is not enabled for this account.") from exc
+        raise
+    print(f"started {crawl.crawl_id}")
+
+    crawl = client.wait(crawl.crawl_id)
+    cov = crawl.coverage
+    print(
+        f"{crawl.crawl_id} {crawl.status.value}: {cov.items_found} URLs kept, "
+        f"{cov.pages_fetched} pages fetched ({cov.pages_failed} failed)"
+    )
+    if crawl.status is CrawlStatus.FAILED and crawl.error:
+        raise SystemExit(f"crawl failed: {crawl.error.code.value}: {crawl.error.detail}")
+
+    for result in client.iter_results(crawl.crawl_id):
+        status = result.content_status.value if result.content_status else "-"
+        print(f"  [{status}] {result.url}")
+        if result.content_id:
+            html = client.get_content(crawl.crawl_id, result)
+            print(f"           {len(html)} chars of HTML")
+
+    saved = client.download(crawl.crawl_id, args.out)
+    print(f"wrote {saved.path}")
+
+
+if __name__ == "__main__":
+    main()

examples/README.md

@@ -1,6 +1,7 @@
 # Examples
 
-Runnable samples for the Batch API client (`ZenRowsBatchClient`).
+Runnable samples for the Batch API client (`ZenRowsBatchClient`) and
+the Crawl API client (`ZenRowsCrawlClient`, `11_crawl.py`).
 Each script reads the API key from the environment:
 
 ```bash
@@ -27,6 +28,7 @@ uv run python examples/02_download_to_dir.py --job-id 01J...
 | `08_download_all_results.py`   | `download_all_results()` — server-side export zip of a whole run |
 | `09_scheduled_jobs.py`         | scheduled cadences (`Rate`/`Calendar`/`At`), `pause`/`resume`/`update_schedule`, per-fire runs, `retry_failed` |
 | `10_lightweight_handles.py`    | `client.job(id)` / `client.run(id, rid)` — act on a known id without a GET (`.data` lazily fetches) |
+| `11_crawl.py`                  | Crawl: `create` → `wait` → `iter_results` + `get_content` → `download` (NDJSON) |
 
 The samples are deliberately small — each demonstrates one feature
 end-to-end so they double as living documentation.

pyproject.toml

@@ -112,5 +112,9 @@ exclude = ["src/zenrows/batch/models.py"]
 # --- pytest ---
 [tool.pytest.ini_options]
 testpaths = ["tests"]
-addopts = "-ra --strict-markers"
+# e2e tests hit a live API; `make test-e2e` selects them with `-m e2e`.
+addopts = "-ra --strict-markers -m 'not e2e'"
 asyncio_mode = "auto"
+markers = [
+    "e2e: end-to-end tests against a live ZenRows API (opt-in, see DEVELOPMENT.md)",
+]

src/zenrows/__init__.py

@@ -1,17 +1,21 @@
 """Zenrows Python SDK.
 
-Two clients live here:
+Three clients live here:
 
   - `ZenRowsClient` — the original synchronous scraping API client.
     Backward-compatible with pre-1.4 usage; freshened-up internals.
 
   - `ZenRowsBatchClient` — the new async-job / batch API client.
     Built on a generated OpenAPI core with a hand-written ergonomic
     facade on top.
+
+  - `ZenRowsCrawlClient` — the Crawl API client: start a crawl from
+    one URL and read back the URLs (and pages) it finds.
 """
 
 from zenrows.__version__ import __version__
 from zenrows.batch import ZenRowsBatchClient
 from zenrows.client import ZenRowsClient
+from zenrows.crawl import ZenRowsCrawlClient
 
-__all__ = ["ZenRowsBatchClient", "ZenRowsClient", "__version__"]
+__all__ = ["ZenRowsBatchClient", "ZenRowsClient", "ZenRowsCrawlClient", "__version__"]