feat(crawl): add ZenRowsCrawlClient for the Crawl API

[StefanoGuerrini]

This PR replaces #30. The code is the same. The history drops the AI co-author trailer. The review of the first version is here.

What this adds

A zenrows.crawl package with ZenRowsCrawlClient, also exported from the top level (from zenrows import ZenRowsCrawlClient). Crawl is New: the README and the client docstrings say "(new)", and the README and the package docstring add one line that Crawl is still evolving. There is no runtime warning.

• ZenRowsCrawlClient(api_key, *, base_url=None, timeout=30.0, retries=3, verify_ssl=True, user_agent=..., httpx_args=None). The default base URL is https://api.zenrows.com/v1. Override it with base_url= only; the client reads no environment variable.
• create(url, *, depth, max_items=None, max_pages=None, include_patterns=None, exclude_patterns=None, output_format: Literal["html"] | None = None, idempotency_key=None) -> Crawl sends only the fields you set. output_format is checked by the type hint only; the API decides. Without output_format, a crawl returns URLs only. A crawl stays on the start URL's registrable domain; subdomains count.
• get(crawl_id, *, cursor=None, limit=None) -> CrawlWithResults
• list(*, cursor=None, limit=None) -> CrawlList returns one page of crawls, newest first. next_cursor is None on the last page.
• results(crawl_id, *, limit=None) follows next_cursor, limit URLs per request, and returns at the first empty page or a null cursor. It does not poll: on a running crawl it yields only what the crawl has kept so far. Call wait first to read every result.
• stop(crawl_id) -> CrawlStop is idempotent.
• content(crawl_id, content) -> str returns one kept URL's HTML. content is a CrawlResult, its content_url, or a content id.
• download(crawl_id) -> CrawlDownload opens the NDJSON file. Iterate its lines property for parsed DownloadLines, and read .status (from X-Crawl-Status). It is also a context manager. On a running crawl, status is running and the file holds only what the crawl has kept so far.
• DownloadLine.content is str | dict | None, so a line with object content parses.
• wait(crawl_id, *, timeout=600.0) -> Crawl polls with backoff (2 s, x1.5, up to 15 s) until the crawl ends or timeout seconds pass. It never raises for the crawl's state. On timeout it returns the crawl, still running; call wait again or stop. A failed crawl is returned; read crawl.error.code and crawl.error.detail.
• Responses parse into pydantic v2 models. Unknown fields are ignored, and every response enum is open: a value added later parses as UNKNOWN and keeps its raw value.
• Errors raise CrawlAPIError, with status_code, code, detail, problem and retry_after. code is None when the body has no code. The README lists every code:
  • 400 invalid_request / unknown_parameter / invalid_cursor.
  • 403 REQS008 reads "Crawl is not enabled for this account". Compare exc.code with the exported CRAWL_NOT_ENABLED_CODE.
  • 404 crawl_not_found / content_not_found.
  • 409 idempotency_request_in_flight: retry after the first request finishes.
  • 422 invalid_parameter / invalid_start_url / domain_not_allowed, and idempotency_key_reused (use a new key or no key).
  • 429 too_many_crawls, with retry_after: the account has reached its limit of active jobs (3 by default), shared with its Batch jobs. create never retries it, with or without an idempotency key.
• The package now ships a py.typed marker (in the wheel and the sdist), so type checkers use its types.

The SDK's shared HTTP transport now takes the error mapping and logger as constructor arguments, gains request() and stream() helpers, and takes the retryable statuses per request. ProblemDetail.code is None when the body has no code; BatchAPIError keeps its "internal" fallback. Existing clients behave as before.

Tests

• make check passes (ty, ruff check, ruff format --check).
• make test: 205 passed, 1 deselected (the e2e test).
• Unit tests cover wait returning a running crawl on timeout and a failed crawl, list paging with a cursor, and results sending limit.
• New e2e test, tests/e2e/test_crawl_e2e.py:
  • It is marked e2e and deselected from make test.
  • It skips unless ZENROWS_API_KEY, ZENROWS_CRAWL_BASE_URL and ZENROWS_E2E_CRAWL_URL are set. It passes ZENROWS_CRAWL_BASE_URL as base_url=. ZENROWS_E2E_CRAWL_INCLUDE is optional.
  • Run it with make test-e2e. DEVELOPMENT.md has a "Running the Crawl e2e test" section.
• I ran make test-e2e against a live deployment before the method renames: 1 passed. I did not run it again after them.

Docs: a Crawl API (new) quickstart section in the README (with the client options base_url, retries and timeout), examples/11_crawl.py, and DEVELOPMENT.md notes.

Release: this PR does not change the version. Do a minor bump (1.6.0) in the release commit.