fix(crawl): retry stop on 503 crawl_busy, honoring a capped Retry-After

[gonzalo-zenrows]

Stacked on #31 (feat/crawl-client-v2): it merges after #31, and the diff shows only this change.

Summary

ZenRowsCrawlClient.stop now retries a 503 with problem code crawl_busy, waiting Retry-After seconds between tries. The client caps that wait at 30 s, so a large server value cannot hang a stop. Before this change, the client raised the 503 while the crawl kept running.

Problem

POST /v1/crawls/{crawl_id}/stop can collide with the crawl while it saves its own progress. The API retries that collision on the server for under 2 s. If the collision stays, the API answers 503 with problem code: crawl_busy and Retry-After: 1. The API saves nothing, the crawl keeps running, and a second stop is safe.

stop raised that 503 at once. Stop is a POST without an Idempotency-Key, so the shared transport never retries it. The caller saw an error while the crawl kept running.

Change

Case	stop behavior
503 with code crawl_busy	Retries up to retries more times, then raises the CrawlAPIError. retries is the existing client option (default 3); retries=0 turns the retry off.
Wait between tries	The Retry-After value, at most 30 s. A missing, negative, NaN or infinite value gives 1 s.
503 with another code, any other error	Raises at once, as before.
Every other method	No change.

The docstrings of retries and stop mention crawl_busy. The crawl errors table in README.md adds a 503 crawl_busy row.

Tests

New tests in tests/test_crawl_client.py use respx and record each time.sleep call:

• 503 crawl_busy, then 200: stop returns the stopped crawl after one 1.0 s wait.
• 503 crawl_busy on every call, with retries=2: stop raises it after 3 calls, with waits [1.0, 1.0].
• A 503 with another code: stop does not retry.
• A parametrized test checks the wait for each Retry-After value: 3600 waits 30.0 s, -1 and nan wait 1.0 s, 2 waits 2.0 s.

How to try it

make check   # ty, ruff check, ruff format --check
make test

Squash-merge with the title as the message: the branch commits carry trailer lines that should not land.

[StefanoGuerrini]

Merged into the feature branch with a merge commit. It ships to main with #31.