fix(crawl): exit 1 when a crawl stops because credits ran out

README.md

@@ -203,7 +203,12 @@ run out (default 600). When the time runs out, the command exits 0 and prints
 the crawl with status `running` and the hint `zenrows crawl wait <crawl-id>`
 (under `--json`, in `note`); the crawl keeps running. Ctrl-C stops the wait, not the crawl, and exits 130;
 resume with `zenrows crawl wait <crawl-id>`. A crawl that ends `failed` exits 1
-with `CRAWL_FAILED`.
+with `CRAWL_FAILED`. A crawl that ends `stopped` because the account ran out of
+credits or the API key hit a credit cap also exits 1, with the code a refused
+create gets: `CRAWL_INSUFFICIENT_CREDITS` or `CRAWL_KEY_CAP_REACHED`, and the
+API's `insufficient_credits` or `api_key_cap_reached` as `server_code`. Older
+deployments end such a crawl `failed`; it gets the same codes. Its fetched pages
+stay readable.
 
 Errors carry the API's code as `server_code`, `crawl_id` on any call about one
 crawl, and `retry_after` on a 429. When the account has reached its limit of

skills/crawl/SKILL.md

@@ -77,8 +77,15 @@ zenrows crawl stop <id>                      # stop it; kept URLs stay readable
 ```
 
 A crawl ends `completed` (nothing left, or a cap hit: `stop_reason`
-`max_items` / `max_pages`), `stopped` (you stopped it) or `failed` (`error.code`
-and `error.detail` say why). Only `failed` exits non-zero (`CRAWL_FAILED`).
+`max_items` / `max_pages`), `stopped` or `failed` (`error.code` and
+`error.detail` say why). `stopped` with `stop_reason` `user` means you stopped
+it. `stopped` with no `stop_reason` and `error.code` `insufficient_credits` or
+`api_key_cap_reached` means the account ran out of credits or the API key hit a
+credit cap; the pages fetched before that stay readable. `failed`, and a
+`stopped` crawl with an `error`, exit non-zero: `CRAWL_INSUFFICIENT_CREDITS` or
+`CRAWL_KEY_CAP_REACHED` when credits ended it (the same codes a refused
+`create` gets, also when an older deployment reports it `failed`), else
+`CRAWL_FAILED`. `server_code` carries the API's `error.code`.
 `stop` on a crawl that already ended prints the status it ended with.
 
 With `--json`, `create`, `get`, `wait` and `stop` print
@@ -112,15 +119,15 @@ The "Retry?" column is for errors that remain after these retries.
 | --- | --- | --- |
 | `CRAWL_NOT_ENABLED` | 403 `REQS008`: Crawl is not enabled for this account | no |
 | `AUTH_INVALID` | 401: the API key was rejected | no, log in again |
-| `CRAWL_INSUFFICIENT_CREDITS` | 402: the account is out of credits (claim the account or add credits) | after credits renew or are added |
-| `CRAWL_KEY_CAP_REACHED` | 402 `AUTH014` or `api_key_cap_reached`: this API key reached its credit cap | after the cap resets or is raised |
+| `CRAWL_INSUFFICIENT_CREDITS` | 402 on `create`, or a crawl that ended on `insufficient_credits`: the account is out of credits (claim the account or add credits) | after credits renew or are added |
+| `CRAWL_KEY_CAP_REACHED` | 402 `AUTH014` or `api_key_cap_reached` on `create`, or a crawl that ended on `api_key_cap_reached`: this API key reached its credit cap | after the cap resets or is raised |
 | `CRAWL_INVALID_REQUEST` | 400 or 422: fix the request. For `idempotency_key_reused`, send a new key or no key | no, not as is |
 | `CRAWL_REQUEST_IN_FLIGHT` | 409: the same Idempotency-Key is still in flight | yes, once the first request ends |
 | `CRAWL_NOT_FOUND` / `CRAWL_CONTENT_NOT_FOUND` | 404: wrong id, or the page has no content | no |
 | `CRAWL_TOO_MANY_CRAWLS` | 429 `too_many_crawls` on `create`: the account's active-job limit | yes, after `retry_after` |
 | `CRAWL_RATE_LIMITED` | any other 429: requests are being throttled | yes, after `retry_after` |
 | `CRAWL_BUSY` | 503 `crawl_busy` on `stop`: the stop was not saved yet and the crawl still runs | yes, run the stop again after `retry_after` |
-| `CRAWL_FAILED` | the crawl ended `failed`, or another error | 5xx: yes, with backoff |
+| `CRAWL_FAILED` | the crawl ended `failed`, or `stopped` with an error other than credits, or another error | 5xx: yes, with backoff |
 | `BACKEND_UNAVAILABLE` | the API could not be reached | yes |
 
 Rules: start small (`--depth 1`, low caps) and check the results before a

skills/trace-debug/SKILL.md

@@ -49,12 +49,16 @@ zenrows trace export <run-id>     # JSON for sharing
   enabled for this account. Do not retry; ask Zenrows support for access.
 - `CRAWL_INSUFFICIENT_CREDITS` / `CRAWL_KEY_CAP_REACHED` → Crawl's 402: the
   account is out of credits, or this key reached its credit cap (`AUTH014`).
+  The same codes come back for a crawl that credits ended (`stopped`, or
+  `failed` on older deployments; `server_code` `insufficient_credits` /
+  `api_key_cap_reached`); its fetched pages stay readable.
   Same advice as `POLICY_MAX_CREDITS_EXCEEDED` / `KEY_CREDIT_CAP_REACHED`.
 - `CRAWL_TOO_MANY_CRAWLS` → 429 `too_many_crawls`: the account has reached
   its limit of active jobs (3 by default), shared with its Batch jobs.
   Nothing was created; retry after `retry_after` seconds (see
   `zenrows crawl list`).
-- `CRAWL_FAILED` → the crawl ended `failed`; `error.code` in
+- `CRAWL_FAILED` → the crawl ended `failed`, or `stopped` with an error
+  other than credits; `error.code` in
   `zenrows crawl get <id> --json` says why, with `error.detail`. See
   [[crawl]].
 - `PARAM_CONFLICT_AUTO_MANUAL` → drop the managed flags or add `--manual`.

src/cli/commands/crawl.ts

@@ -30,7 +30,8 @@ import {
   type CreateCrawlParams,
 } from "../../core/crawl-api.ts";
 import { asNumber, asString, parse, type Command, type RunContext } from "../command.ts";
-import { ToolkitError } from "../../core/errors.ts";
+import { ToolkitError, isKeyCapReached, keyCapReached, quotaExhausted } from "../../core/errors.ts";
+import { readAccount } from "../../core/agent-account.ts";
 import { printError, writeOut, writeOutStream } from "../output.ts";
 
 /** The API's default for `max_pages` (and `max_items`) when the flag is unset. */
@@ -439,7 +440,9 @@ async function stopCmd(rest: string[], ctx: RunContext): Promise<number> {
   assertUsable("crawl");
   const apiKey = requireApiKey();
   const s = await stopCrawl(id, { apiKey });
-  const headline = s.status === "stopped" ? `Stopped crawl ${id}` : `Nothing to stop: crawl ${id} had already ended`;
+  // A crawl stopped on credits carries `error`: it had already ended, this call did not stop it.
+  const headline =
+    s.status === "stopped" && !s.error ? `Stopped crawl ${id}` : `Nothing to stop: crawl ${id} had already ended`;
   return printCrawl(s, ctx.json || values.json === true, headline);
 }
 
@@ -462,34 +465,51 @@ async function waitUntilEnd(id: string, opts: { apiKey: string; timeoutMs?: numb
 }
 
 /**
- * The error for a crawl that ended `failed`, or null for any other state. The
- * crawl's own `error.code` picks the next action.
+ * The error for a crawl that ended `failed`, or `stopped` with an `error` (the
+ * account ran out of credits or the API key hit a credit cap), or null for any
+ * other state. A crawl the credits ended gets the code and advice a create
+ * refused with 402 gets (`CRAWL_INSUFFICIENT_CREDITS`, `CRAWL_KEY_CAP_REACHED`),
+ * whether the API reports it `stopped` or, as older deployments do, `failed`.
+ * Any other `error.code` picks the next action of a `CRAWL_FAILED`.
  */
 export function crawlFailure(c: Crawl | CrawlStop): ToolkitError | null {
-  if (c.status !== "failed") return null;
+  const creditStop = c.status === "stopped" && !!c.error;
+  if (c.status !== "failed" && !creditStop) return null;
   const code = c.error?.code;
+  const ids = { server_code: code, crawl_id: c.crawl_id };
+  const where = `crawl ${c.crawl_id}`;
+  const detail = c.error?.detail || undefined;
+  const getCmd = `zenrows crawl get ${c.crawl_id}`;
+  if (code && isKeyCapReached(code)) {
+    const cap = keyCapReached(where, { status: null, detail }).toJSON();
+    return new ToolkitError({ ...cap, ...ids, code: "CRAWL_KEY_CAP_REACHED", suggested_commands: [...(cap.suggested_commands ?? []), getCmd] });
+  }
+  if (code === "insufficient_credits") {
+    const acct = readAccount();
+    const quota = quotaExhausted(where, acct?.unclaimed ? acct.claimUrl : undefined, { status: null, detail }).toJSON();
+    return new ToolkitError({ ...quota, ...ids, code: "CRAWL_INSUFFICIENT_CREDITS", suggested_commands: [...(quota.suggested_commands ?? []), getCmd] });
+  }
   const why = [code, c.error?.detail].filter(Boolean).join(": ");
   const next: Record<string, string> = {
-    insufficient_credits: "The account ran out of credits. Check `zenrows usage`, add credits, then start a new crawl.",
     seed_unreachable: "The start URL could not be fetched (the site may be blocking it). Check it with `zenrows fetch <url>`.",
     no_items_found:
       "Nothing matched. Loosen --include-patterns / --exclude-patterns, raise or drop --depth, or check the start page with `zenrows fetch <url>`.",
   };
   return new ToolkitError({
     code: "CRAWL_FAILED",
-    message: `Crawl ${c.crawl_id} failed.`,
+    message: creditStop ? `Crawl ${c.crawl_id} stopped before it finished.` : `Crawl ${c.crawl_id} failed.`,
     likely_cause: why || "The crawl ended in status failed without a reason.",
     next_action: (code && next[code]) || "Read the error, fix the cause, then start a new crawl.",
-    suggested_commands: [`zenrows crawl get ${c.crawl_id}`],
-    server_code: code,
-    crawl_id: c.crawl_id,
+    suggested_commands: [getCmd],
+    ...ids,
   });
 }
 
 /**
  * Print a crawl (or a stop answer), structured under --json, and return the exit
- * code: 1 when it ended `failed`, else 0. `stopped` is a deliberate outcome, so
- * it exits 0 but prints as a warning. `extra` carries the results a `get` read,
+ * code: 1 when it ended `failed`, or `stopped` because credits ran out or the
+ * API key hit a credit cap, else 0. A `stopped` crawl the caller stopped is a
+ * deliberate outcome, so it exits 0 but prints as a warning. `extra` carries the results a `get` read,
  * or the note of a wait that ran out.
  */
 function printCrawl(

src/core/crawl-api.ts

@@ -42,7 +42,7 @@ export interface CrawlCoverage {
   items_found: number;
 }
 
-/** Why a crawl ended `failed` (part of the crawl, not an error response). */
+/** Why a crawl ended `failed`, or `stopped` because credits ran out or the API key hit a credit cap (part of the crawl, not an error response). */
 export interface CrawlRunError {
   code: string;
   detail: string;

src/core/errors.ts

@@ -109,12 +109,14 @@ export class ToolkitError extends Error {
  * (so `claimUrl` is surfaced); for a real logged-in account there is nothing to
  * claim — the step is to add credits / upgrade in the dashboard. `status`/`detail`
  * let the caller record the actual upstream response (402 usage-limit, 429
- * quota, …) instead of a hardcoded status.
+ * quota, …) instead of a hardcoded status. `status: null` is for a crawl the
+ * credits stopped: its status call succeeded, so there is no HTTP error to cite,
+ * only `url` (the crawl).
  */
 export function quotaExhausted(
   url: string,
   claimUrl?: string,
-  opts: { status?: number; detail?: string } = {},
+  opts: { status?: number | null; detail?: string } = {},
 ): ToolkitError {
   // Say that the allowance comes back. Without it this reads as a permanent paywall,
   // which is how the API's own AUTH004 text reads ("Purchase a new subscription to
@@ -126,10 +128,11 @@ export function quotaExhausted(
     ? `You are on the Zenrows Free plan. Claim your account to keep your usage and add credits: ${claimUrl}. ${renewLine}`
     : `You are out of Zenrows credits. ${renewLine} To carry on now, add a credit pack (${BILLING_TOPUP_URL} opens the purchase directly) or upgrade your plan: ${PLANS_URL}`;
   const detail = opts.detail ? `${opts.detail.replace(/\.\s*$/, "")}. ` : "";
+  const where = opts.status === null ? `Stopped ${url}` : `HTTP ${opts.status ?? 429} for ${url}`;
   return new ToolkitError({
     code: "POLICY_MAX_CREDITS_EXCEEDED",
     message: "Zenrows request quota exhausted.",
-    likely_cause: `${detail}HTTP ${opts.status ?? 429} for ${url}`,
+    likely_cause: `${detail}${where}`,
     next_action: claimLine,
     suggested_commands: ["zenrows usage"],
   });

tests/crawl-command.test.ts

@@ -335,6 +335,61 @@ test("crawl get exits 0 for completed and stopped crawls", async () => {
   }
 });
 
+// What the server says when credits end a crawl, as error.detail.
+const creditStopDetail: Record<string, string> = {
+  insufficient_credits: "The crawl stopped because your account has no credits left.",
+  api_key_cap_reached: "The crawl stopped because your API key reached one of its credit caps.",
+};
+
+test("a crawl the credits ended exits 1 with the code a 402 create gets, stopped or failed", async () => {
+  const cases = [
+    { status: "stopped", server: "insufficient_credits", code: "CRAWL_INSUFFICIENT_CREDITS", advice: /zenrows usage/ },
+    { status: "stopped", server: "api_key_cap_reached", code: "CRAWL_KEY_CAP_REACHED", advice: /cap to reset/ },
+    // Older deployments end such a crawl `failed`: callers see the same code.
+    { status: "failed", server: "insufficient_credits", code: "CRAWL_INSUFFICIENT_CREDITS", advice: /zenrows usage/ },
+    { status: "failed", server: "api_key_cap_reached", code: "CRAWL_KEY_CAP_REACHED", advice: /cap to reset/ },
+  ];
+  for (const k of cases) {
+    const label = `${k.status} ${k.server}`;
+    await withCrawlWorkspace(
+      () =>
+        json({
+          ...crawlBody,
+          status: k.status,
+          error: { code: k.server, detail: creditStopDetail[k.server] },
+          results: [{ url: "https://example.com/product/a" }],
+          next_cursor: null,
+        }),
+      async () => {
+        const { code: exit, out } = await run(["get", "c_1"]);
+        assert.equal(exit, 1, label);
+        assert.equal(out.ok, false, label);
+        assert.equal(out.status, k.status, label);
+        assert.equal(out.error.code, k.code, label);
+        assert.equal(out.error.server_code, k.server, label);
+        assert.equal(out.error.crawl_id, "c_1", label);
+        assert.ok(out.error.likely_cause.includes(`${creditStopDetail[k.server]} Stopped crawl c_1`), label);
+        assert.match(out.error.next_action, k.advice, label);
+        assert.ok(out.error.suggested_commands.includes("zenrows crawl get c_1"), label);
+        assert.equal(out.results.length, 1, label);
+      },
+    );
+  }
+});
+
+test("a crawl stopped with any other error code exits 1 with CRAWL_FAILED", async () => {
+  await withCrawlWorkspace(
+    () => json({ ...crawlBody, status: "stopped", error: { code: "internal", detail: "Something broke." }, results: [], next_cursor: null }),
+    async () => {
+      const { code, out } = await run(["get", "c_1"]);
+      assert.equal(code, 1);
+      assert.equal(out.error.code, "CRAWL_FAILED");
+      assert.equal(out.error.server_code, "internal");
+      assert.match(out.error.message, /stopped before it finished/);
+    },
+  );
+});
+
 test("crawl get on an unknown id exits 1 with CRAWL_NOT_FOUND", async () => {
   await withCrawlWorkspace(
     () => json({ code: "crawl_not_found", title: "Crawl not found", detail: "No crawl c_x.", status: 404 }, 404),