feat(crawl): add ZenRowsCrawlClient for the Crawl API

DEVELOPMENT.md

@@ -19,14 +19,19 @@ Makefile targets below.
 
 ```
 src/zenrows/
-├── __init__.py               # re-exports both clients
+├── __init__.py               # re-exports the clients
 ├── client.py                 # ZenRowsClient (legacy sync scraper)
-└── batch/
-    ├── __init__.py           # ZenRowsBatchClient + key models
+├── batch/
+│   ├── __init__.py           # ZenRowsBatchClient + key models
+│   ├── client.py             # hand-written typed facade
+│   ├── _transport.py         # httpx wrapper, RFC 7807 → exceptions
+│   ├── errors.py             # BatchAPIError, ProblemDetail
+│   └── models.py             # GENERATED — pydantic v2 (do not edit)
+└── crawl/
+    ├── __init__.py           # ZenRowsCrawlClient + models
     ├── client.py             # hand-written typed facade
-    ├── _transport.py         # httpx wrapper, RFC 7807 → exceptions
-    ├── errors.py             # BatchAPIError, ProblemDetail
-    └── models.py             # GENERATED — pydantic v2 (do not edit)
+    ├── errors.py             # CrawlAPIError
+    └── models.py             # hand-written pydantic v2 models
 ```
 
 The Batch SDK is split deliberately:
@@ -42,12 +47,21 @@ This way the wire types stay in lockstep with the OpenAPI document
 while the ergonomic surface (method names, helpers, retries, URL
 override) stays in our control.
 
+The Crawl client uses the shared HTTP transport (`_transport.py`: key
+header, retries, problem+json) with `CrawlAPIError.from_response` as
+its error mapping. Its models are
+hand-written: the surface is a few small schemas, so there is no Crawl
+codegen step.
+Keep them tolerant: unknown fields are ignored, and every response enum
+gets `_missing_ = classmethod(open_enum_missing)`.
+
 ## Common tasks
 
 | Make target     | What it does |
 |-----------------|--------------|
 | `make sync`     | `uv sync --all-extras` |
-| `make test`     | `uv run pytest` |
+| `make test`     | `uv run pytest` (offline; e2e tests deselected) |
+| `make test-e2e` | End-to-end tests against a live API (see below) |
 | `make check`    | `ty check` + `ruff check` + `ruff format --check` (CI mode) |
 | `make typecheck`| `ty check src` (static types; `models.py` excluded) |
 | `make lint`     | `ruff check --fix` |
@@ -56,6 +70,38 @@ override) stays in our control.
 | `make build`    | Build wheel + sdist via hatchling |
 | `make clean`    | Drop caches + build outputs |
 
+## Running the Crawl e2e test
+
+`tests/e2e/test_crawl_e2e.py` drives a real crawl through the client:
+create (with `include_patterns`, `output_format="html"`), wait, read the
+results, one page and the NDJSON download, list, stop, and a 404. It
+costs a few credits. It is marked `e2e`, which `make test` deselects,
+and it skips unless the three required variables below are set.
+
+| Variable | |
+|---|---|
+| `ZENROWS_API_KEY` | required: a key with Crawl access |
+| `ZENROWS_CRAWL_BASE_URL` | required: the API base, e.g. `https://api.zenrows.com/v1`; the test passes it as `base_url=` |
+| `ZENROWS_E2E_CRAWL_URL` | required: the page the crawl starts from |
+| `ZENROWS_E2E_CRAWL_INCLUDE` | optional: an include pattern; every result must contain it |
+
+```bash
+export ZENROWS_API_KEY=zr_...                       # a key with Crawl access
+export ZENROWS_CRAWL_BASE_URL=https://api.zenrows.com/v1
+export ZENROWS_E2E_CRAWL_URL=https://example.com/products/
+export ZENROWS_E2E_CRAWL_INCLUDE=/product/          # optional
+make test-e2e
+```
+
+Pick a start page whose links include a few pages matching the pattern:
+the test crawls at depth 1 with `max_items=3` and `output_format="html"`.
+
+To test against a local or staging deployment, point
+`ZENROWS_CRAWL_BASE_URL` at its `/v1` base instead. When the account
+has reached its limit of active jobs (429 `too_many_crawls`: 3 by default,
+shared with its Batch jobs), the test waits and retries (up to 5 minutes)
+before creating its crawl.
+
 ## Refreshing the OpenAPI spec
 
 `docs/openapi.yaml` is the SDK-local copy of the spec. `make generate`

Makefile

@@ -1,28 +1,33 @@
-.PHONY: install sync test lint format typecheck check generate docs clean build bump release
+.PHONY: install sync test test-e2e lint format typecheck check generate docs clean build bump release
 
 # Bootstrap: install + dev deps, build the local venv.
 install sync:
 	uv sync --all-extras
 
-# Run the suite.
+# Run the suite (offline; e2e tests are deselected).
 test:
 	uv run pytest
 
+# Run the end-to-end tests against a live API. Needs ZENROWS_API_KEY,
+# ZENROWS_CRAWL_BASE_URL and ZENROWS_E2E_CRAWL_URL; see DEVELOPMENT.md.
+test-e2e:
+	uv run pytest -m e2e -s tests/e2e
+
 # Static type check (ty — Astral). Shipped surface only; tests are
 # covered by the suite. Generated models.py is excluded in pyproject.
 typecheck:
 	uv run ty check src
 
 # Lint + format + type check (CI mode — no fixes).
 check: typecheck
-	uv run ruff check src/ tests/
-	uv run ruff format --check src/ tests/
+	uv run ruff check src/ tests/ acceptance/
+	uv run ruff format --check src/ tests/ acceptance/
 
 # Lint + format (writes fixes).
 lint:
-	uv run ruff check --fix src/ tests/
+	uv run ruff check --fix src/ tests/ acceptance/
 format:
-	uv run ruff format src/ tests/
+	uv run ruff format src/ tests/ acceptance/
 
 # Regenerate the pydantic v2 models from the backend's canonical spec.
 # docs/openapi.yaml is the SDK-local copy of the spec; refresh it from the

README.md

@@ -10,14 +10,17 @@
 SDK to access [Zenrows](https://www.zenrows.com/) APIs directly from Python.
 Zenrows handles proxies rotation, headless browsers, and CAPTCHAs for you.
 
-This package ships two clients:
+This package ships three clients:
 
   - **`ZenRowsClient`** — the original synchronous scraping client.
     One URL in, one HTML/JSON response out. Best for ad-hoc scraping.
   - **`ZenRowsBatchClient`** — the async-job / Batch API client. Submit
     thousands of URLs as one job, poll for results, optionally upload a
     CSV of URLs in one call. Best for offline / bulk pipelines.
     _(Private beta — [contact support](mailto:support@zenrows.com) for access.)_
+  - **`ZenRowsCrawlClient`** — the Crawl API client. Give it one start
+    URL; it follows links and returns the URLs (and, optionally, the
+    pages) it finds behind it. _(New.)_
 
 ## Table of Contents
 
@@ -37,6 +40,10 @@ This package ships two clients:
   - [Act on an id without a GET](#act-on-an-id-without-a-get)
   - [Scheduled jobs & webhooks](#scheduled-jobs--webhooks)
   - [Error handling](#error-handling)
+- [Quickstart — Crawl API (new) (`ZenRowsCrawlClient`)](#quickstart--crawl-api-new-zenrowscrawlclient)
+  - [Read the pages](#read-the-pages)
+  - [List and stop crawls](#list-and-stop-crawls)
+  - [Crawl errors](#crawl-errors)
 - [Contributing](#contributing)
 - [License](#license)
 
@@ -485,6 +492,127 @@ The full Batch surface (jobs, runs, tasks, results, content, history,
 file_inputs, HMAC keys) is reachable via methods on `ZenRowsBatchClient`.
 See `src/zenrows/batch/client.py` or `help(ZenRowsBatchClient)`.
 
+## Quickstart — Crawl API (new) (`ZenRowsCrawlClient`)
+
+Crawl is still evolving: new features are coming, limits may be tuned, and the changelog announces each change.
+
+A crawl starts from one URL, follows the links on each page up to
+`depth` hops, and keeps the URLs that match your patterns, up to
+`max_items`. It stays on the start URL's registrable domain;
+subdomains count. It runs in the background: `create` returns at once with
+`status="running"`, and `wait` blocks until it ends. Without
+`output_format` a crawl returns URLs only; `output_format="html"` also
+returns each kept URL's page.
+
+```python
+import os
+from zenrows import ZenRowsCrawlClient
+
+client = ZenRowsCrawlClient(api_key=os.environ["ZENROWS_API_KEY"])
+
+crawl = client.create(
+    "https://www.scrapingcourse.com/ecommerce/",
+    depth=1,                         # link hops from the start URL
+    max_items=50,                    # stop after keeping 50 URLs (default 10)
+    max_pages=100,                   # stop after fetching 100 pages (default 10)
+    include_patterns=["/product/"],  # keep only URLs containing this
+)
+crawl = client.wait(crawl.crawl_id)            # polls up to timeout=600 s
+print(crawl.status.value, crawl.stop_reason, crawl.coverage)
+
+for result in client.results(crawl.crawl_id):  # follows next_cursor
+    print(result.url)
+```
+
+`wait` returns the crawl when it ends, or when `timeout` (seconds,
+default 600) runs out. It does not raise: on timeout the crawl is still
+`running` (call `wait` again, or `stop`), and a `failed` crawl comes
+back with `crawl.error` set. `results` does not poll: it returns at the first empty page, so on a
+running crawl it yields only the URLs kept so far. Call `wait` first to
+read every result. `results(crawl_id, limit=...)` sets the page size of
+each request. `get(crawl_id, cursor=..., limit=...)` returns one raw page.
+
+The client takes these options: `base_url` (default
+`https://api.zenrows.com/v1`), `retries` (default 3; set 0 to disable)
+and `timeout` (seconds per request, default 30):
+
+```python
+client = ZenRowsCrawlClient(api_key, base_url="https://api.zenrows.com/v1", retries=3, timeout=30)
+```
+
+### Read the pages
+
+Pass `output_format="html"` to also fetch the page of every kept URL:
+
+```python
+crawl = client.create(start_url, depth=1, output_format="html")
+client.wait(crawl.crawl_id)
+
+for result in client.results(crawl.crawl_id):
+    if result.content_url:                       # content_status == fetched
+        html = client.content(crawl.crawl_id, result)
+
+# Or everything in one NDJSON file: {"url", "content_status", "content"} per line.
+with client.download(crawl.crawl_id) as download:  # streamed, parsed
+    print(download.status)                         # from X-Crawl-Status
+    for line in download.lines:
+        print(line.url, line.content_status)
+```
+
+`content` takes a result, its `content_url`, or its content id.
+`download` returns a `CrawlDownload`: its `status` is the crawl's status
+when the file was read, and its `lines` yields parsed lines. On a `running` crawl the
+file holds only the results kept so far.
+
+### List and stop crawls
+
+```python
+page = client.list(limit=20)            # newest first, without results
+for c in page.crawls:
+    print(c.crawl_id, c.status.value)
+# Pass page.next_cursor as cursor= for the next page; it is None on the last.
+
+client.stop(crawl_id)                   # idempotent; read final counts with get()
+```
+
+### Crawl errors
+
+Every non-2xx raises `CrawlAPIError`. Branch on `status_code` and `code`:
+
+```python
+from zenrows import CrawlAPIError
+from zenrows.crawl import CRAWL_NOT_ENABLED_CODE
+
+try:
+    client.create(start_url, depth=1)
+except CrawlAPIError as exc:
+    if exc.code == CRAWL_NOT_ENABLED_CODE:  # 403 REQS008
+        print("Crawl is not enabled for this account")
+    elif exc.code == "too_many_crawls": # 429: active jobs limit reached
+        print("retry in", exc.retry_after, "s")
+    elif exc.status_code in (400, 422):
+        print(exc.detail)
+    else:
+        raise
+```
+
+| Status | `code` | What to do |
+|---|---|---|
+| 400 | `invalid_request`, `unknown_parameter`, `invalid_cursor` | Fix the request. Do not retry it as is. |
+| 403 | `REQS008` | Crawl is not enabled for the account (`CRAWL_NOT_ENABLED_CODE`). |
+| 404 | `crawl_not_found`, `content_not_found` | Check the id. |
+| 409 | `idempotency_request_in_flight` | Retry after the first request with this key finishes. |
+| 422 | `invalid_parameter`, `invalid_start_url`, `domain_not_allowed` | Fix the request. Do not retry it as is. |
+| 422 | `idempotency_key_reused` | Use a new key, or no key. Do not retry it as is. |
+| 429 | `too_many_crawls` | The account has reached its limit of active jobs (3 by default), shared with its Batch jobs. Retry after `exc.retry_after` seconds. `create` does not retry it. |
+| 503 | `crawl_busy` | `stop` could not be saved yet; the crawl is still running. `stop` retries it up to `retries` times, waiting `exc.retry_after` seconds, then raises it. Repeating the stop is safe. |
+
+`code` is None when the response has no problem body with a code (for
+example a 502 from a proxy). A crawl that ended `failed` is not an error
+response: read `crawl.error.code` and `crawl.error.detail` on it. Response enums (`CrawlStatus`, `StopReason`,
+`ContentStatus`, ...) are open: a value added later parses as an
+`UNKNOWN` member that keeps the raw value.
+
 ## Contributing
 Pull requests are welcome. For significant changes, please open an issue first to discuss what you would like to change.
 

acceptance/crawl/.gitignore

@@ -0,0 +1 @@
+out/

acceptance/crawl/README.md

@@ -0,0 +1,111 @@
+# Crawl first-install acceptance test: Python SDK
+
+This folder tests the Python SDK (`zenrows` on PyPI) the way a new customer meets it. It builds the release
+artifact, installs it in a clean container with nothing else, follows the README's Crawl
+quickstart, and checks every Crawl feature against the live API. The result is one report.
+
+It tests this repo only, so it runs on its own.
+
+## What it needs
+
+- bash, git and Docker. Nothing else on the host.
+- Network access to GitHub, the package registries and the Crawl API.
+- A Zenrows API key with Crawl access, in `ZENROWS_API_KEY`.
+
+## How to run it
+
+From the repo root:
+
+```bash
+export ZENROWS_API_KEY="$(cat path/to/key-file)"   # keep it out of your shell history
+bash acceptance/crawl/run.sh              # build, run every check, write the report
+bash acceptance/crawl/run.sh --no-key     # only the checks that need no key; the live checks are SKIP
+bash acceptance/crawl/run.sh --report     # rebuild the report from out/
+REF=v1.2.3 bash acceptance/crawl/run.sh   # test a tag, branch or SHA from GitHub instead of this checkout
+```
+
+Without `ZENROWS_API_KEY`, `run.sh` stops and asks for it. The report is
+`acceptance/crawl/out/crawl-first-install-report.md`; `out/` is not committed.
+
+Every input is an environment variable with a default in `versions.env`: `CRAWL_URL`
+(default `https://www.scrapingcourse.com/ecommerce/`), `CRAWL_INCLUDE` (default `/product/`),
+`REF`, `CRAWL_API_BASE`, the images (`node:20`, `python:3.12-slim`, `golang:1.25`, pinned by
+digest) and the tool versions. `OUT_DIR` moves the output.
+
+## What a run does
+
+1. **Source.** By default, this checkout, including changes not yet committed. With `REF`, the
+   repo at that ref from GitHub.
+2. **Build.** In its own container, the release artifact: `pip install` of the wheel `python -m build` makes.
+3. **Clean test container.** It gets only the artifact (read-only), the checks in `surface/`
+   (read-only), the inputs, and the key by name (`-e ZENROWS_API_KEY`). It has no source, no
+   caches from the build, and no tokens.
+4. **README first.** It runs the README's Crawl quickstart as written. The only changes are the
+   ones the cost rules need: the key, `CRAWL_URL`, `CRAWL_INCLUDE`, `max_pages` 10 or less, and
+   no html. The report lists each change.
+5. **Checks 1 to 21 and the extras.** Each writes one JSON result:
+   `{surface, check, title, status, expected, actual, evidence}`, with `status` PASS, FAIL or
+   SKIP. The evidence is the command and its trimmed output, with the key redacted.
+6. **Clean-up.** It stops every crawl the run started and reads each one's final status and
+   `coverage.pages_fetched`. It runs even when a check fails, or on Ctrl-C.
+7. **Report.** Verdict, environment, install steps, feature matrix, findings, cost and
+   clean-up. Last, the run checks that the key appears in no file under `out/`.
+
+Some checks don't use the live API, so they behave the same on every run. They point the
+client at a small local stand-in server that records what it sends. These are: "only the
+fields you set are sent", "the error code is empty when the body has none", and the timing
+and signal checks.
+
+## The checks
+
+| # | Feature | Expected |
+| --- | --- | --- |
+| 1 | Create | returns at once: `running`, a `crawl_id`, the echoed parameters; only the fields set are sent |
+| 2 | Create with html | results get `content_status` and, once fetched, `content_url` |
+| 3 | Idempotency | same key and body → same `crawl_id`; different body → 422 `idempotency_key_reused` (MCP, CLI: `CRAWL_INVALID_REQUEST` with the advice to send a new key) |
+| 4 | Wait | returns the ended crawl; `failed` is returned, not raised (CLI: exit 1) |
+| 5 | Wait runs out | a 1 s timeout on a running crawl returns it `running`, no error (CLI: exit 0; MCP, CLI: a `note`) |
+| 6 | Get | `limit` 2, then the cursor: one page each; `next_cursor` moves on and is null on the last page |
+| 7 | List | `limit` 2, then the next page: newest first; `next_cursor` absent or null on the last page |
+| 8 | Results | every kept URL once, each matching `CRAWL_INCLUDE`; on a running crawl, what is kept so far (MCP, CLI: `partial: true`) |
+| 9 | Content | by content id and by `content_url`: the same HTML |
+| 10 | Download | a status, and one line per result with `url`, `content_status`, `content` |
+| 11 | Stop | running → `stopped`, `stop_reason: user`; ended → answers as it ended |
+| 12 | Not found | 404 `crawl_not_found` (MCP, CLI: `CRAWL_NOT_FOUND` with `crawl_id` and `server_code`) |
+| 13 | Invalid input | `depth` 0 → 422 `invalid_parameter` (MCP, CLI: `CRAWL_INVALID_REQUEST`) |
+| 14 | Gated option | `output_format` json → 422 `invalid_parameter`; if it is accepted, the key is in a beta segment, not a bug |
+| 15 | Bad key | the key `invalid` → 401 (MCP, CLI: `AUTH_INVALID`) |
+| 16 | Errors carry the server code | SDKs: `code` holds it, empty when the body has none; MCP, CLI: `server_code` |
+| 17 | Naming | the API's operation and parameter names, in this surface's casing |
+| 18 | Copy | Crawl is "new", never "beta"; one "still evolving" line; the 429 limit reads as the account's active jobs, shared with Batch |
+| 19 | Failed crawl | an include pattern that matches nothing ends `failed` with `no_items_found`; SDKs return it, not raise it; MCP and CLI say `ok: false` (CLI: exit 1, `CRAWL_FAILED`) |
+| 20 | Results after stop | the html crawl is stopped while its pages are still being fetched: reading its results ends at once (MCP and CLI: `partial: false`, no note to call again), and the same URLs come back once the pages in flight finish |
+| 21 | Documented defaults | a create without `max_pages` and `max_items` gets the default the docs state (10) |
+
+Extra checks for this surface: `mypy --strict` sees the package as typed (`py.typed`) and reads the real signatures.
+
+## What it costs
+
+Every crawl uses `depth` 1 and `max_pages` 10 or less. A run starts five crawls: the README run,
+one URLs-only crawl, one idempotency crawl that is stopped at once, one html crawl that is
+stopped with its pages in flight (check 20), and one that keeps nothing (check 19). On the default `CRAWL_URL`, a URLs-only crawl fetches 1 page and the
+html crawl 6, so a run costs about **10 pages**, and at most 50 if every crawl reaches its
+`max_pages`. Check 9 starts a sixth crawl, one html item, when the stop in check 20 lands
+before any page is fetched. It adds at most 10 pages. The report lists each crawl id and its `pages_fetched`.
+
+An account can run 3 active jobs at once, shared with Batch jobs. A run has at most 2 crawls
+running at a time. If a create still gets a 429, the run waits `Retry-After` and tries again.
+It never sends requests to trigger the limit. Don't run two copies against one account at the
+same time.
+
+## How to read the report
+
+- **Verdict:** ready, unless there is a blocker or major finding.
+- **Feature matrix:** PASS, FAIL or SKIP, with a short reason. SKIP means the check does not
+  apply to this surface, the client rejects the input before the API, a crawl ended before
+  the check could act on it, or there was no key.
+- **Findings:** one entry per FAIL, most severe first, with the command and output. An install
+  or README failure is a blocker; a wrong answer from a feature is major; naming, copy, the
+  defaults and the extras are minor.
+- `out/results/checks/*.json` holds every result; `out/results/container.log` is the full
+  output.