fix(batch): read the stopped run, retry without duplicate jobs, and recover every unfinished task

README.md

@@ -132,12 +132,35 @@ validation + credit estimation always work (no key needed).
 
 ```bash
 zenrows batch estimate jobs.jsonl                    # validate the spec + estimate credits (local)
-zenrows batch create jobs.jsonl --wait               # submit a job and wait for it to finish
+zenrows batch create jobs.jsonl --follow             # submit a job and wait for it to finish
 zenrows batch status <job-id>                        # run status + progress stats
+zenrows batch wait <job-id> --timeout 600000         # wait up to 600000 ms (or --timeout 600s)
 zenrows batch results <job-id> --out results.jsonl   # collect results (paginated)
 zenrows batch cancel <job-id>                        # stop an in-flight run
+zenrows batch retry-failed <job-id> --include-pending  # rerun failed and unfinished tasks
 ```
 
+`--wait` is an alias of `--follow`. `create` and `retry-failed` send an
+`Idempotency-Key` and reuse it for their own automatic retries, so a dropped
+connection never starts a second job or run. Running a command again sends a
+new key unless you pass one: a repeated `create` creates another job, and a
+repeated `retry-failed`, once the previous run has ended, may start another
+run of the tasks that still match, re-scraping and billing them again. With
+the same `--idempotency-key <key>`, the same file and flags, within 24 hours
+of the first request, a repeat cannot create a duplicate (after 24 hours the
+key is forgotten). Its answer is not guaranteed to be the original: the API
+may replay it, answer 409 (no recorded answer yet, or a rerun whose run is
+still in progress), or refuse it before looking at the key, for example with
+a 402 when the account is out of credits. Unless the repeat returns the job,
+check your jobs before doing anything else. When a submit or rerun ends without a usable answer, the
+error carries the key and prints the exact command to repeat with it.
+
+Before submitting, `create` checks the run against the local policy:
+`max_pages_per_run` (default 1000 URLs) and `max_credits_per_run` (default
+10000 estimated credits, where each Adaptive Stealth Mode task counts as 25).
+Raise them for large jobs with `zenrows policy set max_pages_per_run <n>` and
+`zenrows policy set max_credits_per_run <n>`.
+
 `jobs.jsonl` is one JSON object per line, each with a `url` (plus optional
 per-line overrides like `js_render`, `premium_proxy`, `proxy_country`,
 `extract`). Validate and estimate locally first, then submit with `create`.
@@ -150,8 +173,8 @@ Extract — structured data instead of raw HTML:
 {"url": "https://example.com/reviews", "extract": "auto", "external_id": "r1"}
 ```
 
-An Extract task costs the same as a regular one (1 credit at base tier), so
-`batch estimate` prices it correctly. Results for an Extract task carry two
+Extract adds no charge of its own: an Extract task costs what the same task
+costs without it (1 credit at base tier), so `batch estimate` prices it correctly. Results for an Extract task carry two
 keys: `html` (the raw page) and `parsed` (the structured data) — validate a
 sample with `zenrows extract <url>` before running the full batch.
 

agent-plugin/skills/batch-jobs/SKILL.md

@@ -21,19 +21,23 @@ anti-bot advantage becomes obvious — thousands to millions of URLs.
 | `batch_create` | Submit a job (list of URL tasks + optional scrape/extract params) |
 | `batch_status` | Poll run status + progress stats |
 | `batch_results` | Page results (cursor pagination; filter successful/failed/all) |
+| `batch_wait` | Block until the run reaches a terminal status |
 | `batch_cancel` | Stop an in-flight run |
 
-Optional if exposed by the server: `batch_wait` to block until terminal status.
-
 ## Workflow
 
 1. Validate the workflow on **one** URL with `scrape` or `extract`.
-2. `batch_create` with the full task list (each task: `url` + optional
-   overrides like `js_render`, `premium_proxy`, `proxy_country`, `mode`,
-   `autoparse`, `extract`, `external_id`, `metadata`). `extract: "auto"`
-   returns structured data instead of raw HTML, at the same credit cost as a
-   regular task; the result carries `html` and `parsed` keys.
-3. Poll `batch_status` until terminal (`completed` / `stopped` / …).
+2. `batch_create` with the full task list. Each task is `url` plus optional
+   `external_id` and `metadata`, and per-task scrape overrides like
+   `js_render`, `premium_proxy`, `proxy_country`, `mode`, `autoparse` or
+   `extract`. Put those overrides under the task's `zenrows_params`
+   (`{"url": "...", "zenrows_params": {"extract": "auto"}}`): the MCP server
+   reads them there, and some versions silently ignore the same keys set flat
+   on the task. `extract: "auto"` returns structured data instead of
+   raw HTML, at the same credit cost as a regular task; the result carries
+   `html` and `parsed` keys.
+3. Call `batch_wait`, or poll `batch_status`, until terminal (`completed`,
+   `failed`, `stopped` or `deleted`). A `failed` run carries `failure_reason`.
 4. Collect with `batch_results`.
 5. `batch_cancel` if the user aborts or the run should stop early.
 

agent-plugin/skills/cost-control/SKILL.md

@@ -18,9 +18,10 @@ These multipliers apply when you enable the flags (on `scrape` or `extract`):
 - Premium proxies (`premium_proxy`): **10×**
 - Both: **25×**
 
-`extract` `mode=auto` is currently free; billing may apply
-later. Do not promise permanent free extract pricing. Autoparse / CSS still
-follow the multipliers above when stealth flags are set.
+Extract adds no charge of its own: an `extract` request costs what the
+underlying fetch costs, multipliers included. Do not promise that Extract
+pricing will never change. Autoparse / CSS follow the same multipliers when
+stealth flags are set.
 
 ## Browser sessions (`browser_*`)
 

agent-plugin/skills/extract/SKILL.md

@@ -21,7 +21,8 @@ extract({ url, proxy_country: "US" })               # geo-targeted
 ```
 
 `mode=auto` (`extract=auto`) works on any domain Zenrows has prepared, with any
-key, and is currently free; billing may apply later. If the
+key. Extract adds no charge of its own: a request costs what the underlying
+fetch costs, multipliers included. If the
 domain is not enabled (AUTH010) or not prepared yet (REQS007), the tool retries
 once with Autoparse by default (`fallback_autoparse`).
 

agent-plugin/skills/zenrows/SKILL.md

@@ -66,8 +66,8 @@ If the user needs login, clicks, forms, sessions, or persistent state:
 | Primitive | MCP tools | Notes |
 | --- | --- | --- |
 | Protected Fetch | `scrape` | Default markdown; HTML / plaintext / PDF / screenshot options |
-| Extract | `extract` | Structured JSON (`extract=auto`, autoparse, css_extractor). `extract=auto` is currently free; billing may apply later |
-| Batch | `batch_create`, `batch_status`, `batch_results`, `batch_cancel` | Same credits as Fetch; a 403 maps to `BATCH_ACCESS_DENIED` |
+| Extract | `extract` | Structured JSON (`extract=auto`, autoparse, css_extractor). Extract adds no charge of its own: a request costs what the underlying fetch costs |
+| Batch | `batch_create`, `batch_status`, `batch_results`, `batch_wait`, `batch_cancel` | Same credits as Fetch; a 403 maps to `BATCH_ACCESS_DENIED` |
 | Browser | `browser_*` (start with `browser_navigate`) | Escalation-only; same backend as Browser Sessions |
 
 ## Cost notes

recipes/batch-estimate/RECIPE.md

@@ -28,6 +28,6 @@ URL; cost multipliers are basic 1x / JS 5x / proxy 10x / both 25x.
 
 Once the spec is clean, submit it:
 ```
-zenrows batch create jobs.jsonl --wait
+zenrows batch create jobs.jsonl --follow
 ```
 See [[batch-jobs]] for the full cloud flow.

skills/batch-jobs/SKILL.md

@@ -34,22 +34,52 @@ structured data instead of raw HTML.
 {"url": "https://example.com/products", "extract": "auto", "external_id": "p1"}
 ```
 
-An Extract task costs the same as a regular one, so `batch estimate` prices it
-correctly. The result carries two keys: `html` (the
+Extract adds no charge of its own: an Extract task costs what the same task
+costs without it, so `batch estimate` prices it correctly. The result carries two keys: `html` (the
 raw page) and `parsed` (the structured data). Validate on a single URL with
 `zenrows extract <url>` before running the full batch.
 
 ## Cloud (needs a key)
 ```
-zenrows batch create jobs.jsonl [--proxy-country cc] [--wait]
+zenrows batch create jobs.jsonl [--proxy-country cc] [--follow [--timeout ms]]
 zenrows batch create jobs.jsonl --manual [--js-render] [--premium-proxy]
 zenrows batch status <id>             # run status + stats
-zenrows batch results <id> [--status successful|failed|all] [--out results.jsonl]
-zenrows batch wait <id> [--timeout ms]
+zenrows batch results <id> [--status successful|failed|processing|all] [--out results.jsonl]
+zenrows batch wait <id> [--timeout ms] # default 600000 ms (600s also works); fails if paused
 zenrows batch cancel <id>             # stop an in-flight run
-zenrows batch retry-failed <id>       # rerun only the failed tasks (new run)
+zenrows batch retry-failed <id> [--include-pending]   # rerun failed (and unfinished) tasks
 ```
 
+`create` and `retry-failed` send an `Idempotency-Key` and reuse it for their
+own automatic retries, so a dropped connection never starts a second job or
+run. Running a command again sends a new key unless you pass one: a repeated
+`create` creates another job, and a repeated `retry-failed`, once the
+previous run has ended, may start another run of the tasks that still match.
+Either re-scrapes and bills those tasks again. With the same
+`--idempotency-key <key>`, the same file and flags, within 24 hours of the
+first request, a repeat cannot create a duplicate (after 24 hours the key is
+forgotten). Its answer is not guaranteed to be the original: the API may
+replay it, answer 409 (no recorded answer yet, or a rerun whose run is still
+in progress), or refuse it before looking at the key (for example 402, out of
+credits). When a submit or rerun ends without a
+usable answer (no response, a 5xx, a refusal after a lost attempt, or a
+success answer that can't be read), the error carries `idempotency_key` and
+prints the exact command to repeat; run that, not a fresh command. If it
+does not return the job, or reports that an earlier attempt may have gone
+through, check `zenrows batch status <id>` (or the dashboard) before running
+anything else. With a workspace, `create` and `retry-failed` also record the
+key in their run artifact. If `create --follow` fails after the job was
+created, the error names the job: follow it with `zenrows batch wait <id>`,
+do not create it again. Status reads retry transient errors on their own.
+`--timeout` takes milliseconds, as before; `600s` and `600000ms` also work.
+`--wait` still works as an alias of `--follow`.
+
+`--include-pending` reruns every task that did not finish: `failed`, `pending`
+(never started) and `processing` (cut off mid-flight). `wait` fails right
+away when a run is paused (`BATCH_FAILED`, `reason: run_paused`, exit 1, the
+same code and exit as the timeout it would otherwise reach): a paused run
+picks up no new task, though tasks already started may still finish.
+
 Every task runs in Adaptive Stealth Mode (`mode=auto`) by default, like `fetch`
 and `extract`: Zenrows enables JS rendering and premium proxies only on the pages
 that need them, and bills only the configuration that succeeds. A task that sets
@@ -68,7 +98,29 @@ A run ends in one of `completed`, `failed`, `stopped`, or `deleted`. `status`,
 `failure_reason` is `api_key_cap_reached`, the key hit its own credit cap
 (`KEY_CREDIT_CAP_REACHED`): the account still has credits, other keys keep
 working, and the tasks not yet run stay pending. Wait for the cap to reset
-(`zenrows usage`) or raise the cap; do not resubmit the job with the same key.
+(`zenrows usage`) or raise the cap, then run what is left with
+`zenrows batch retry-failed <id> --include-pending`; do not resubmit the job.
+A run that failed with `insufficient_credits` keeps `BATCH_FAILED` and works
+the same way once credits are added. Errors carry the Batch API's own code as
+`server_code`, and a finer `reason` (`insufficient_credits`,
+`api_key_cap_reached`, `too_many_active_jobs`, `too_many_scheduled_jobs`,
+`run_paused`, `wait_timeout`) next to the unchanged `code`. A `429`
+(`BATCH_QUOTA_EXCEEDED`) means an account limit: 3 active jobs and 5
+scheduled jobs by default. A closed job counts as active while its run is in
+progress (wait for it, or stop it), an open job until it is deleted, or
+closed and its run has ended, and a scheduled job counts toward its limit
+until it is deleted.
+
+## Local policy caps
+
+`create` checks the run against `.zenrows/policy.json` before anything is sent:
+`max_pages_per_run` (default 1000 URLs) and `max_credits_per_run` (default
+10000 estimated credits). The estimate is an upper bound and counts each
+Adaptive Stealth Mode task as 25, so with the defaults a run of more than 400
+auto-mode URLs stops with `POLICY_LIMIT_EXCEEDED`. Raise the caps for large
+jobs, e.g. `zenrows policy set max_pages_per_run 50000` and
+`zenrows policy set max_credits_per_run 1250000`; the error names the exact
+command for the run you tried.
 
 Deferred (documented, not yet in the CLI): CSV upload, open/queue jobs,
 scheduled jobs, webhooks/HMAC, and ZIP export.

skills/trace-debug/SKILL.md

@@ -34,7 +34,7 @@ zenrows trace export <run-id>     # JSON for sharing
   connectivity and `zenrows config show`.
 - `AUTH_INVALID` → re-check the key, `zenrows login --api-key …`.
 - `KEY_CREDIT_CAP_REACHED` → this API key hit one of its own credit caps (HTTP
-  402, gateway code `AUTH014`, Batch `api_key_cap_reached`). The account still
+  402, code `AUTH014`). The account still
   has credits and its other keys keep working. Do not retry and do not escalate:
   `zenrows usage` shows the cap and when it resets. Wait for the reset, or have
   the account owner raise or remove the cap at
@@ -43,8 +43,23 @@ zenrows trace export <run-id>     # JSON for sharing
   local policy credit limit was hit). Do not retry-loop; `zenrows usage` shows
   when credits renew. Top up or upgrade, or claim the account if it is an
   auto-created Free plan.
-- `BATCH_FAILED` → the batch run ended `failed`; `failure_reason` and
-  `failure_detail` in `zenrows batch status <id> --json` say why.
+- Batch's 402 keeps the codes above: `POLICY_MAX_CREDITS_EXCEEDED` (out of
+  credits, `reason: insufficient_credits`) or `KEY_CREDIT_CAP_REACHED`
+  (`reason: api_key_cap_reached`). A run that stopped for lack of credits ends
+  `failed` with `BATCH_FAILED` and `reason: insufficient_credits`. Afterwards
+  run the tasks that did not finish with
+  `zenrows batch retry-failed <id> --include-pending`.
+- `BATCH_QUOTA_EXCEEDED` → Batch's 429 (`reason: too_many_active_jobs` or
+  `too_many_scheduled_jobs`): the account reached an active-job (3)
+  or scheduled-job (5) limit, shared with Crawl. A closed job frees its slot
+  when its run ends (wait, or stop it), an open job when it is deleted, or
+  closed and its run has ended, and a scheduled job only when it is deleted.
+- `BATCH_FAILED` → any other Batch error (400, 409, 5xx), a run that ended
+  `failed`, or a `wait` that timed out (`reason: wait_timeout`) or found the
+  run paused (`reason: run_paused`); `server_code` holds the API's own code, and `failure_reason` /
+  `failure_detail` in `zenrows batch status <id> --json` say why a run failed.
+  A Batch 401, 403 and 404 keep their own codes: `AUTH_INVALID`,
+  `BATCH_ACCESS_DENIED` and `BATCH_NOT_FOUND`.
 - `CRAWL_NOT_ENABLED` → the API answered 403 `REQS008`: Crawl is not
   enabled for this account. Do not retry; ask Zenrows support for access.
 - `CRAWL_INSUFFICIENT_CREDITS` / `CRAWL_KEY_CAP_REACHED` → Crawl's 402: the

skills/zenrows/SKILL.md

@@ -41,8 +41,9 @@ If the user has a start page (listing, index, section) but not the item URLs:
 If the user has many URLs (bulk):
   → Use Batch.                      (validate one page with fetch/extract first,
                                      then `zenrows batch create <jobs.jsonl>`)
-                                     Batch is the cheap path at scale — do NOT
-                                     fan out fetch/extract per URL for bulk work.
+                                     Batch is the reliable path at scale (same
+                                     per-request credits as fetch); do NOT fan
+                                     out fetch/extract per URL for bulk work.
 
 If the user needs login, clicks, forms, sessions, or persistent state:
   → Use Interact / Browser Sessions.(zenrows browser)    [escalation-only]
@@ -79,7 +80,7 @@ Run `zenrows status` for the live capability matrix. As of this toolkit:
 | MCP | `zenrows mcp` | **available**: `https://mcp.zenrows.com/mcp` |
 
 - Extract: `extract=auto` falls back to Autoparse on a domain not prepared yet.
-- Batch: the cheapest path at scale (create, status, results). Estimate specs locally with no key.
+- Batch: the reliable path at scale, at the same per-request credits as Fetch (create, status, results, wait, retry-failed). Estimate specs locally with no key.
 - Crawl: collects the URLs behind one start page, optionally with each page's HTML.
 - Browser: escalation-only, for pages Fetch can't handle; same backend as the MCP `browser_*` tools.