fix(batch): read the stopped run, retry without duplicate jobs, and recover every unfinished task

[emirlan-zen]

This PR fixes several ways zenrows batch misreports a run, prices it, retries it, or gets stuck. It touches cancel, retry-failed, wait, results and create.

Nothing that works today changes. The existing error codes, exit codes, JSON keys, flags and --timeout units all stay as they are. New detail comes in new fields (server_code, reason, idempotency_key, paused) and new flags. The only differences in existing behaviour are a few values the Batch API already rejects, which are now refused locally before sending. The machine diff against main under Compatibility shows this.

batch cancel reads the run that stop returns

POST /jobs/{id}/stop returns the refreshed run, not the job. The client read the response as a job and looked for latest_run, so the run's status and stats never showed.

before: ✓ Stopped job j1 · status: unknown              --json: {"ok":true,"jobId":"j1"}
after:  ! Stopped job j1 · status: stopped
          3/5 completed · 3 successful · 0 failed        --json: {"ok":true,"jobId":"j1","status":"stopped","stats":{...}}

A successful stop call still exits 0 with ok: true, whatever the run's status, as before. The status, stats and failure fields are additions. The test fixtures mocked /stop with a job-shaped body, which hid the bug; they now use the run shape.

Recovering a run that did not finish

• New flag --include-pending on retry-failed. It reruns ?status=failed,pending,processing, so it covers every task that did not finish.
  • A stopped or credit-failed run leaves unstarted tasks pending and can leave tasks cut off mid-flight as processing.
  • A partial rerun only re-enqueues the statuses it names, so a filter without processing would carry those rows over unfinished, and the new run could never complete.
  • Plain retry-failed (failed only) is unchanged.
  • When a stopped run has tasks that did not complete, status points at this flag.
• 409 answers keep BATCH_FAILED, with specific advice. Instead of "fix the problem and retry":
  • no_matching_tasks, partial_rerun_window_expired (10 days) and run_not_stoppable each get their own next step.
  • run_not_terminal separates a run that is still in progress from a run whose task ingestion never completed.
  • idempotency_key_conflict is covered below.

Automatic retries never start a second job

• Status reads retry. GETs retry 429/502/503/504 up to 3 times, with backoff and Retry-After, the same rule crawl uses. A transient 502 no longer aborts wait or loses the pages a paginated results had already fetched.
• Writes retry under one key. create and retry-failed send an Idempotency-Key. That's a fresh one by default, or your own with the new --idempotency-key. They reuse it for their own automatic retries on 502/503/504 and on connection errors. A 429 on a POST is an account limit and is not retried.
• What a retry under that key can get back. Within the key's 24 hours, it cannot start a second job or run. Its answer is not guaranteed to be the original, though. The Batch API may:
  • replay the original;
  • answer 409, because the key has no recorded answer yet, or because a rerun's lost first attempt already started its run;
  • refuse before it looks at the key, for example with a 402 when the account is out of credits.
• Key validation. --idempotency-key must be 1 to 128 characters, with no surrounding whitespace and no control characters. HTTP strips a header's surrounding whitespace, so a blank key would reach the API as no key at all. This is a new flag, so nothing that worked before is refused.

When a write ends with an unknown result

Only create (POST /jobs) and retry-failed (POST /jobs/{id}/rerun) create billable work. Their result is unknown, and the error keeps its usual code while adding idempotency_key, in each of these cases:

#	How the call ends	Test (tests/batch-uncertain.test.ts)
A	No answer on every attempt (connection error)	A: create and rerun with no answer on every attempt keep the key
B	No answer on every attempt (client-side timeout)	B: a write that times out on every attempt keeps the key
C	A 5xx on the last attempt	C: create and rerun whose last answer is a 5xx keep the key
D	A refusal after an attempt that got no usable answer: a 402 (out of credits, or a key cap), a 409, a 429 or a 400. The earlier attempt may have been accepted.	five D: tests
E	A 2xx whose answer is empty, not JSON, or has no job_id. The request was accepted, so the job or run probably exists.	E: a 2xx with an empty, truncated or job_id-less answer keeps the key (create and rerun)
F	create --follow: the job was created, then following it failed. The error names the job and says not to create it again.	F: create --follow that fails after the job exists names the job and says not to create again

In cases A to E the error says not to resubmit with a new key, and prints the exact command that repeats the request with its key:

• --idempotency-key <the key that was sent> comes right after the subcommand, so it stays an option even after a --;
• the caller's own key option is dropped;
• every argument is shell-quoted.

Tests run the printed command through sh and through the CLI, and check that the repeat sends the same key. A repeat within 24 hours of the first attempt cannot create a duplicate. After that, the API forgets the key.

Definite outcomes carry no key: a refusal on the first attempt, and every GET. cancel sends no key and is not retried, because the API treats a repeated stop as a no-op. With a workspace, create and retry-failed record the key in their run artifact.

Running a command again without the key is not deduped. A repeated create creates another job. A repeated retry-failed, once the previous run has ended, may start another run of the tasks that still match, or get no_matching_tasks. The README, skill and help text say so.

Errors: same codes, more detail

Every code stays as it was for the situation that produced it. The new fields come on top:

Situation	code (unchanged)	New fields
402, out of credits	POLICY_MAX_CREDITS_EXCEEDED	server_code, reason: insufficient_credits
402 api_key_cap_reached	KEY_CREDIT_CAP_REACHED	server_code, reason: api_key_cap_reached
429, 3 active jobs	BATCH_QUOTA_EXCEEDED	server_code, reason: too_many_active_jobs
429, 5 scheduled jobs	BATCH_QUOTA_EXCEEDED	server_code, reason: too_many_scheduled_jobs
run failed, api_key_cap_reached	KEY_CREDIT_CAP_REACHED	server_code, reason
run failed, insufficient_credits	BATCH_FAILED	server_code, reason: insufficient_credits, credit advice
wait timed out	BATCH_FAILED	reason: wait_timeout
401 / 403 / 404	AUTH_INVALID / BATCH_ACCESS_DENIED / BATCH_NOT_FOUND	server_code
anything else (400, 409, 5xx)	BATCH_FAILED	server_code

The 429 advice now says how each kind of slot is freed:

• a closed job frees its slot when its run ends (wait, or stop it);
• an open job, when it is deleted, or once it is closed and its run has ended;
• a scheduled job, only when it is deleted.

Credit estimate and submitted params

• The estimate now prices what is submitted. estimateCredits used to price only flat per-line flags. It ignored job-level flags and nested zenrows_params, and treated the string "false" as on. Two URLs with --premium-proxy printed ~2 credits while sending premium_proxy: true (real cost 20). Each task is now priced on the params it is submitted with: job params, then the task's zenrows_params, then its flat keys. A task key set to null is not sent, so the job's value applies, both for the estimate and for the proxy_country check.
• The policy cap still blocks exactly what it blocked before. The max_credits_per_run check keeps using the old estimate, so a run that passed the cap before still passes. When the corrected estimate is over the cap, create prints a warning with the exact zenrows policy set command, and submits as before.
• Boolean params. The API accepts JSON true/false and the strings "true"/"false" in any case. It rejects anything else, 1 and "yes" included, with a 400 that fails the whole job. The CLI now sends "true"/"false", and refuses other values before sending (see below).

Smaller fixes

• metadata is checked as string-to-string before submitting. The API accepts nothing else.
• results --status processing is accepted. The API supports it.
• wait --timeout still takes milliseconds, and a bare number is read exactly as before.
  • 600s (seconds) and 600000ms are also accepted, and must be positive.
  • A value that isn't a number was silently ignored before, and the 600000 ms default applied; it still is, now with a warning.
  • create --follow takes --timeout too.
• wait on a paused run fails right away with BATCH_FAILED (reason: run_paused) and exit 1. That's the same code and exit as the timeout it used to wait for. A paused run picks up no new task. status shows a paused run with "paused": true. A stopped run that kept pause_state is not reported as paused.
• Policy cap errors name the exact zenrows policy set command and state the default.
• Docs and skills:
  • The policy defaults are documented: 1000 URLs and 10000 estimated credits, where an Adaptive Stealth Mode task counts as 25.
  • "Batch is the cheapest path" now reads "the reliable path at scale, at Fetch's credits".
  • The Extract cost wording is the same everywhere: Extract adds no charge of its own.
  • batch_wait is listed as an MCP tool.
  • Examples use --follow, and --wait stays as an alias.

Compatibility

A compatibility script, run locally and not part of this PR, builds origin/main (c8eefa6) and this branch, then runs 44 scenarios against both through a local mock of the Batch API. The scenarios cover every subcommand, every error status, failed, paused and stopped runs, timeouts, policy caps and malformed answers. For each scenario it compares the exit code, ok, the error code, and the full JSON key shape. It also diffs --help, each subcommand's flags, and the ErrorCode union.

Check	Result
JSON keys removed, or changed type	none in 44 scenarios
Exit codes changed	none
ok changed	none
Flags removed	none. Added: create --idempotency-key, create --timeout, retry-failed --idempotency-key, retry-failed --include-pending
ErrorCode values removed or added	none
Error code changed	2 scenarios, both values the API already rejects with a 400, now refused locally: a numeric metadata value and js_render: 1 (BATCH_FAILED becomes INVALID_USAGE)
JSON keys added	error.server_code, error.reason, error.idempotency_key, error.suggested_commands[] (now filled where it was empty), and on cancel status, stats, failure_reason; paused on status
--help	2 lines reworded (results --status adds processing; wait --timeout <ms> adds the default and the s unit), 8 lines added for the new flags

Testing

• npm test: 430 passed, 0 failed, run under nice -n 19. origin/main has 364, so this branch adds 66:

  • 1 batch cancel test in tests/batch-command.test.ts;
  • 42 in tests/batch-hardening.test.ts;
  • 23 in tests/batch-uncertain.test.ts.

• main's tests still pass. The error-code assertions in its own tests/batch-api.test.ts and tests/batch-command.test.ts are back to main's codes, with the new server_code/reason asserted next to them. The only fixture changed is the /stop body, which now has the run shape the API actually returns.

• npm run typecheck: clean.

• Coverage of the changed lines. I ran node --test --experimental-test-coverage on a sourcemapped build and intersected the lcov report with the lines this branch adds: 717 of 717 changed executable lines are covered (100%).

  File	Changed lines covered	Whole file
  src/adapters/batch.ts	117/117	100%
  src/cli/commands/batch.ts	252/252	86.6%
  src/core/batch-api.ts	338/338	99.7%
  src/core/errors.ts	6/6	100%
  src/core/policy.ts	4/4	97.9%

• Built CLI against a local mock server:

  • A rerun answered 502, then 402: the error kept POLICY_MAX_CREDITS_EXCEEDED and the credit advice, added that the earlier attempt may have been accepted, and printed the repeat command with the key.
  • A create answered 201 with truncated JSON: the CLI reported that the job probably exists and printed the repeat command.
  • A caller key my "key" it's: the command came out quoted, and sh read the key back intact.
  • create ... -- flaky.jsonl with all 4 answers lost: the printed command put --idempotency-key before the --. Running it sent the same key and got the original job back.
  • create --follow whose status reads kept failing: the error named the job and pointed to zenrows batch wait <id>.

Needs a human decision

• Enforcing the corrected estimate. The policy credit cap keeps checking the old estimate, so nothing that passed before is blocked; it now only warns. Enforcing the corrected estimate would block some runs that pass today, so it belongs in a major release, if anyone wants it.
• Server-side replay order for reruns. The Batch API checks the previous run's status, and the account's credits, before the Idempotency-Key. So a retried rerun whose first attempt went through gets a 409 (or a 402) instead of the replayed answer. The CLI handles that safely and says so; an exact replay needs the API to check the key first.
• Billing for target 404/410 responses. This PR only changes Extract wording. The broader rule (successful requests and target 404/410 responses are billed) should be confirmed with billing before any customer-facing copy relies on it.
• Policy defaults stay at 1000 pages and 10000 credits. This PR documents them, but doesn't raise them.

Open points from the last fact-check

A final independent fact-check of this branch against the Batch API source left these points open:

• Execution claims. The fact-check ran read-only, so it could not re-run the suite, the coverage run, the compatibility diff or the mock sessions above. They come from local runs on this branch.
• Released MCP behaviour. Anything said about the published MCP package is based on its current source, not on a check of the published package.