feat(batch): agents could not recover, inspect or manage batch jobs from the CLI

[emirlan-zen]

Why

zenrows batch disagreed with the Batch API in ways that sent agents down the wrong path:

• batch cancel read the stopped run as a job, so it always printed status: unknown.
• A run that ran out of credits or lost its subscription got BATCH_FAILED and "fix the tasks, then retry-failed". There was nothing to fix in the tasks, and retry-failed only reran failed tasks. The tasks the failure left pending could not be reached at all.
• The default local policy stopped any Adaptive Stealth Mode batch over 400 URLs.
• The estimate ignored job-level flags (--premium-proxy priced at 1 instead of 10) and read "false" as on.
• POST tasks were sent with method and body inside zenrows_params, so the API rejected the whole job.
• Every 409 came back as BATCH_FAILED "Fix the reported problem", with no server_code.
• create sent no Idempotency-Key. A timed-out submit that the API had accepted could be resubmitted as a second, billed job.
• wait hung on a paused run, ignored --timeout 10s, and exited 1 with no state when the time ran out.

What changes

Fixes

• cancel, pause and resume read the run the API answers with. cancel on an ended run prints "Nothing to stop" and the job as it ended, the way crawl stop does.

• A failed run gets the code and advice of the matching refused request:

  • insufficient_credits maps to POLICY_MAX_CREDITS_EXCEEDED.
  • subscription_inactive maps to AUTH_INVALID with Billing advice.
  • api_key_cap_reached maps to KEY_CREDIT_CAP_REACHED.
  • Any other reason stays BATCH_FAILED.

  Each carries server_code (the failure_reason), job_id, and the command that reruns the unfinished tasks. The exit code is 1.

• Each lifecycle 409 gets its own code and next action: BATCH_RUN_IN_PROGRESS, BATCH_RUN_ENDED, BATCH_NO_MATCHING_TASKS, BATCH_RERUN_WINDOW_EXPIRED, BATCH_REQUEST_IN_FLIGHT, and BATCH_CONFLICT for the rest. The 410 is BATCH_RESULT_EXPIRED, a content 422 is BATCH_TASK_FAILED, and a content 409 is BATCH_TASK_NOT_READY.

• Other error fixes:

  • Every Batch error carries server_code and job_id.
  • A 402 "Subscription has expired." now uses subscription wording instead of "out of credits".
  • The 429 advice names the slot rules and zenrows batch list / zenrows crawl list instead of a <id> placeholder.
  • The doubled period is gone.

• create and rerun send an Idempotency-Key, generated or from --idempotency-key. They retry network errors and 502/503/504 up to 3 times under the same key.

  • Reads retry 429/502/503/504.
  • cancel, pause and resume retry 502/503/504.
  • If the outcome is still unknown after the retries, the error carries idempotency_key and the exact, shell-quoted command that repeats the request safely. This also holds when a later attempt is refused (for example a 402, which the API checks before the key) after an earlier attempt got no answer.

• POST tasks send method and body at task level. They get no Adaptive Stealth Mode, and they are refused locally with js_render, js_instructions or json_response.

• estimate and create check each line against the API's limits before anything is sent:

  • url: http or https, 2048 bytes at most.
  • external_id: 128 characters at most, from [A-Za-z0-9._-].
  • metadata: 20 string entries, keys up to 64 characters, values up to 1024 bytes, 4 KiB in total.
  • body: 16 KiB at most.
  • Boolean params: true or false only.

• The estimate prices tasks on what is submitted: job-level flags, nested zenrows_params, and "false". Adaptive Stealth Mode is priced as a range (1 to 25). estimate takes create's flags. The JSON adds estimated_credits and estimated_credits_min.

• Default policy caps: 10,000 URLs and 250,000 estimated credits per run, up from 1,000 and 10,000. With these defaults, a run of up to 10,000 URLs passes in any mode, since an Adaptive Stealth Mode task counts at its 25-credit upper bound. POLICY_LIMIT_EXCEEDED now prints the exact zenrows policy set ... command. Workspaces created earlier keep the caps stored in their policy.json.

• --timeout changes:

  • It takes 500ms, 30s, 5m or 1h. A bare number keeps its old meaning, milliseconds.
  • An invalid value is rejected instead of silently falling back to the default.
  • create --follow and rerun --follow take it.

• Wait changes:

  • A wait that runs out exits 0 and prints the run's state with a note, the way crawl wait does.
  • A paused run returns at once, exit 0, with paused: true and a resume note.
  • Transient errors while polling are ridden out, within the timeout: each request is cut off at the deadline and no retry sleeps past it.
  • Ctrl-C stops the wait, not the job, and exits 130.

• results:

  • Accepts --status processing and --run <run_id>.
  • Reports partial: true while the run is in progress or still receiving its tasks, or when the run's state cannot be read.
  • With --download, skips expired rows with the reason and counts them.

• JSON output adds job_id, run_id, paused, ingest_status, accepted_tasks, rerun_of, retried_tasks, inherited_tasks and the full job. Existing keys (jobId, estimatedCredits, ...) stay.

New commands and flags

• batch list [--status open|closed] [--type regular|scheduled] [--limit n] [--cursor c]
• batch content <id> <task_id> [--run <run_id>] [--out <file>]
• batch pause <id>, batch resume <id>
• batch rerun <id> reruns the failed and pending tasks, so successful tasks are never billed again by surprise. This matches the MCP batch_rerun default. --status pending,processing,failed,successful picks the statuses, and --all reruns every task. --all and --status cannot be combined.
• batch retry-failed <id> --include-pending, the same as rerun --status failed,pending,processing.
• batch delete <id> --yes
• create --name, --external-id and --metadata key=value (repeatable).

Schedules, webhooks, ZIP exports, CSV inputs, open jobs, the runs list and task history stay API/SDK-only. The help text, README and skills say so.

Docs: help text, the README Batch section, skills/batch-jobs (rewritten in the crawl skill's layout, with Limits and Errors sections), skills/trace-debug, skills/zenrows, the MCP copy in agent-plugin/skills/batch-jobs, the template and the recipe (--follow instead of --wait).

Behaviour changes to call out in the release notes

• batch wait and create --follow exit 0, not 1, when the wait runs out. The output carries the state and a note.
• Invalid --timeout values are rejected.
• 409, 410 and content 422 errors have new codes instead of BATCH_FAILED.
• A credit-failed run is POLICY_MAX_CREDITS_EXCEEDED, and a subscription-failed run is AUTH_INVALID, instead of BATCH_FAILED.
• New default policy caps. crawl --max-pages is capped by the same max_pages_per_run, so its local default cap also rises to 10000.

Overlap with open PRs

• fix(crawl): say a crawl with nothing new to follow completes, so users stop chasing their patterns #50: the crawl "default 1000" lines in README.md, skills/crawl/SKILL.md and the crawl help text sit inside fix(crawl): say a crawl with nothing new to follow completes, so users stop chasing their patterns #50's hunks. These are one-line conflicts.
• feat(extract): remove the autoparse fallback and add zenrows extract prepare #47: skills/zenrows/SKILL.md line 82 ("cheapest path at scale") is next to feat(extract): remove the autoparse fallback and add zenrows extract prepare #47's hunk.
• src/core/errors.ts only gets added lines (new codes, job_id, idempotency_key). quotaExhausted is not touched. fix(crawl): exit 1 when a crawl stops because credits ran out #49's status: null change is not needed: the run-failure error sets its own likely_cause.

Testing

• npm run typecheck
• npm test: 411 pass. The new tests/batch-consistency.test.ts has 40 tests using the API's wire shapes. The /stop fixture now returns a run.
• npm run build, then every subcommand from dist/ against a local mock of the Batch API: estimate, create (201, 202, 503 then success with the same key, 503 exhausted, 402, 402 expired, 429), list, status, wait (timeout, paused, Ctrl-C exits 130), results (partial, --run, --download with expired rows), content (200, 409, 410, 422), cancel (running, ended), pause (running, ended), resume, rerun and retry-failed (default, --all, --status, --all with --status, --include-pending, each 409), and delete (with and without --yes).

There is no lint script in this repo.