fix(extract): describe Extract as generally available, on any prepared domain

README.md

@@ -37,7 +37,7 @@ that endpoint with extraction parameters, not a separate product.
 | Command | What it does | Status (this build) |
 | --- | --- | --- |
 | `zenrows fetch` | Fetch — retrieve a protected page | **available** — `GET https://api.zenrows.com/v1/` |
-| `zenrows extract` | Extract — `extract=auto` (domain-gated open beta; falls back to Autoparse) / CSS / Markdown | **beta** — same `/v1/` |
+| `zenrows extract` | Extract: `extract=auto` (any prepared domain; falls back to Autoparse on one not prepared yet), CSS, or Markdown | available, same `/v1/` |
 | `zenrows batch` | Batch: fan out over many URLs | **available**: `https://async.api.zenrows.com/v1`; local validate/estimate need no key |
 | `zenrows crawl` | Crawl (new) — collect the URLs behind one start page, optionally with each page's HTML | **available** — `https://api.zenrows.com/v1/crawls` |
 | `zenrows browser` | Browser Sessions REST API (same backend as MCP `browser_*`) | **available** — escalation-only; bills by bandwidth + time |

agent-plugin/skills/cost-control/SKILL.md

@@ -18,7 +18,7 @@ These multipliers apply when you enable the flags (on `scrape` or `extract`):
 - Premium proxies (`premium_proxy`): **10×**
 - Both: **25×**
 
-`extract` `mode=auto` is **open beta** and currently free; billing may apply
+`extract` `mode=auto` is currently free; billing may apply
 later. Do not promise permanent free extract pricing. Autoparse / CSS still
 follow the multipliers above when stealth flags are set.
 

agent-plugin/skills/extract/SKILL.md

@@ -20,10 +20,10 @@ extract({ url, mode: "css", css_extractor: '{"title":"h1","price":".price"}' })
 extract({ url, proxy_country: "US" })               # geo-targeted
 ```
 
-`mode=auto` (`extract=auto`) is **open beta**: richest on enabled domains,
-currently free; billing may apply later. If the domain is not enabled (AUTH010)
-or not prepared yet (REQS007), the tool retries once with Autoparse by default
-(`fallback_autoparse`).
+`mode=auto` (`extract=auto`) works on any domain Zenrows has prepared, with any
+key, and is currently free; billing may apply later. If the
+domain is not enabled (AUTH010) or not prepared yet (REQS007), the tool retries
+once with Autoparse by default (`fallback_autoparse`).
 
 Every mode uses Adaptive Stealth Mode by default: pass just the URL, even for
 protected or dynamic sites. Set `js_render` or `premium_proxy` only to force a
@@ -45,7 +45,7 @@ scrape({ url, outputs: "emails,links" })            # or "*" for all built-ins
 
 ## When to use which
 
-- **`mode=auto`**: site-tailored fields on beta-enabled domains.
+- **`mode=auto`**: site-tailored fields on any prepared domain.
 - **`mode=autoparse`**: any website; also the automatic AUTH010 / REQS007 fallback.
 - **`mode=css`**: known fields/selectors; deterministic.
 - **Markdown via `scrape`**: feed clean content to an LLM yourself.

agent-plugin/skills/zenrows/SKILL.md

@@ -66,7 +66,7 @@ If the user needs login, clicks, forms, sessions, or persistent state:
 | Primitive | MCP tools | Notes |
 | --- | --- | --- |
 | Protected Fetch | `scrape` | Default markdown; HTML / plaintext / PDF / screenshot options |
-| Extract | `extract` | Structured JSON (`extract=auto`, autoparse, css_extractor). `extract=auto` open beta (currently free; billing may apply later) |
+| Extract | `extract` | Structured JSON (`extract=auto`, autoparse, css_extractor). `extract=auto` is currently free; billing may apply later |
 | Batch | `batch_create`, `batch_status`, `batch_results`, `batch_cancel` | Same credits as Fetch; a 403 maps to `BATCH_ACCESS_DENIED` |
 | Browser | `browser_*` (start with `browser_navigate`) | Escalation-only; same backend as Browser Sessions |
 

docs/capabilities.md

@@ -8,8 +8,6 @@ the single source of truth that prevents hallucinated execution.
 - **available** — a documented endpoint exists and the command does real work.
 - **available-but-needs-confirmation** — likely available; verify per account.
 - **experimental** — exists but not yet promoted to a stable status.
-- **beta** — open beta; usable by any key. Product-specific limits (e.g. Extract
-  domain coverage) are handled by adapters / API errors, not by blocking the CLI.
 - **planned** — no documented endpoint yet; local spec / validation only.
 - **not-implemented** / **deprecated** — not usable.
 
@@ -20,7 +18,7 @@ Classification is based on the public Zenrows documentation:
 | Capability | Backend evidence | Status |
 | --- | --- | --- |
 | `protected_fetch` | Fetch `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available |
-| `extract` | Extract `GET https://api.zenrows.com/v1/` via `extract=auto` (domain-gated open beta; CLI falls back to `autoparse`), plus `autoparse`, `css_extractor`, `outputs`, `response_type=markdown\|plaintext` | beta |
+| `extract` | Extract `GET https://api.zenrows.com/v1/` via `extract=auto` (any prepared domain; CLI falls back to `autoparse` on one not prepared yet), plus `autoparse`, `css_extractor`, `outputs`, `response_type=markdown\|plaintext` | available |
 | `batch` | Batch `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header). Local JSONL spec validation + credit estimation work with no key. | available |
 | `crawl` | Crawl `https://api.zenrows.com/v1/crawls` (`X-API-Key` header; same host as Fetch/Extract). Follows the links on each page from one start URL, up to `depth` when set (optional; no depth limit when omitted), on the start URL's registrable domain (subdomains count); returns URLs, and with `output_format: "html"` each page's HTML. Subcommands: `create`, `get`, `list`, `results`, `content`, `download`, `stop`, `wait`. Without Crawl access the API returns 403 `REQS008` → `CRAWL_NOT_ENABLED`. | available |
 | `browser` | Browser Sessions REST API `https://mcp.zenrows.com/browser/sessions/*` (Bearer; same backend as `@zenrows/mcp` `browser_*`); CDP via `zenrows browser connect`. Escalation-only (prefer fetch/extract); on by default, opt out via `policy.allow_browser`; bills by bandwidth + session time (15-min max) | available |

registry/capabilities.json

@@ -13,11 +13,11 @@
     "extract": {
       "key": "extract",
       "label": "Extract (extract=auto / Autoparse / CSS / Markdown)",
-      "status": "beta",
+      "status": "available",
       "command": "zenrows extract",
       "backend": "Extract — GET https://api.zenrows.com/v1/ (extract=auto, autoparse, css_extractor, outputs, response_type)",
       "requiresAuth": true,
-      "notes": "Open beta. Default is extract=auto (domain-gated); CLI falls back to autoparse on AUTH010. Autoparse / CSS / outputs / markdown work on any domain."
+      "notes": "Default is extract=auto, which works on any prepared domain with any key; CLI falls back to autoparse on REQS007 (domain not prepared yet) or AUTH010. Autoparse / CSS / outputs / markdown work on any domain."
     },
     "batch": {
       "key": "batch",

registry/skills.json

@@ -26,7 +26,7 @@
       "name": "extract",
       "type": "skill",
       "description": "Turn protected pages into structured data with Autoparse / CSS / Markdown.",
-      "status": "beta",
+      "status": "available",
       "requires_backend_capabilities": ["extract"],
       "requires_auth": true,
       "version": "0.1.0",

skills/extract/SKILL.md

@@ -12,7 +12,7 @@ access + extraction**, not generic LLM parsing.
 
 ## Methods
 ```
-zenrows extract <url>                              # extract=auto (default); falls back to Autoparse if domain not enabled
+zenrows extract <url>                              # extract=auto (default); falls back to Autoparse if the domain is not prepared yet
 zenrows extract <url> --autoparse                  # general-purpose Autoparse (any domain)
 zenrows extract <url> --css '{"title":"h1","price":".price"}'   # selector map
 zenrows extract <url> --outputs emails,links       # built-in output filters → JSON
@@ -21,8 +21,8 @@ zenrows extract <url> --validate                   # fail if not valid JSON
 ```
 
 ## When to use which
-- **Extract** (`extract=auto`, default): richest fields on domains enabled for the open beta.
-- **Autoparse**: any website; also the automatic fallback when Extract is not enabled for the domain.
+- **Extract** (`extract=auto`, default): richest fields; works on any domain Zenrows has prepared.
+- **Autoparse**: any website; also the automatic fallback when the domain is not prepared for Extract yet.
 - **CSS**: you know the exact fields/selectors and want determinism.
 - **Markdown/plaintext**: feed clean content to an LLM yourself.
 

skills/zenrows/SKILL.md

@@ -72,7 +72,7 @@ Run `zenrows status` for the live capability matrix. As of this toolkit:
 | Primitive | Command | Status |
 | --- | --- | --- |
 | Protected Fetch | `zenrows fetch` | available (`GET /v1/`) |
-| Extract (extract=auto / Autoparse/CSS/Markdown) | `zenrows extract` | beta (same `/v1/`; extract=auto falls back to autoparse) |
+| Extract (extract=auto / Autoparse/CSS/Markdown) | `zenrows extract` | available (same `/v1/`; extract=auto falls back to autoparse) |
 | Batch | `zenrows batch` | available: runs cloud jobs (create/status/results); cheapest path at scale. Estimate specs locally with no key |
 | Crawl | `zenrows crawl` | new — collect the URLs behind one start page, optionally with each page's HTML |
 | Browser | `zenrows browser` | available (Browser Sessions REST API / MCP) |

src/adapters/extract.ts

@@ -2,7 +2,7 @@
  * Extract adapter.
  *
  * Structured extraction on the same `/v1/` Fetch and Extract API via:
- *   - extract=auto         → site-tailored Extract (open beta, domain-gated)
+ *   - extract=auto         → site-tailored Extract (any prepared domain)
  *   - autoparse=true       → general-purpose Autoparse (any domain)
  *   - css_extractor=<json> → selector-based field extraction
  *   - outputs=<filters>    → built-in output filters → JSON
@@ -38,7 +38,7 @@ export interface ExtractOutcome extends FetchOutcome {
   method: ExtractMethod;
   /** Parsed JSON when the method yields structured data; otherwise undefined. */
   data?: unknown;
-  /** Raw HTML from `extract=auto` beta responses (validation aid). */
+  /** Raw HTML from `extract=auto` responses (validation aid). */
   html?: string;
   /** True when Extract fell back to Autoparse because the domain is not enabled. */
   fellBackToAutoparse?: boolean;
@@ -47,6 +47,11 @@ export interface ExtractOutcome extends FetchOutcome {
    * array / non-JSON). Lets callers warn instead of reporting a silent success.
    */
   empty?: boolean;
+  /**
+   * True when Extract answered `no_data_extracted` (HTTP 422): it knows this kind
+   * of page but no field came back filled. Always reported with `empty: true`.
+   */
+  noDataExtracted?: boolean;
 }
 
 /** True when a parsed extraction result carries no usable structured data. */
@@ -117,6 +122,20 @@ async function runExtractOnce(
   };
 
   const outcome = await runFetch(fetchOpts, config, policy, apiKey);
+  if (outcome.result.noDataExtracted) {
+    if (opts.validate) {
+      throw new ToolkitError({
+        code: "EXTRACT_VALIDATION_FAILED",
+        message: "Extract found no data on this page.",
+        likely_cause:
+          "Extract knows this kind of page, but none of the fields came back filled (HTTP 422 no_data_extracted). The request was not charged.",
+        next_action:
+          "Check that the page rendered fully: try --manual --js-render with --wait-for <selector>. Protected sites may also need --premium-proxy.",
+        suggested_commands: [`zenrows extract ${opts.url} --manual --js-render --premium-proxy`],
+      });
+    }
+    return { ...outcome, method, empty: true, noDataExtracted: true };
+  }
   const { data, html, empty } = parseExtractBody(method, outcome.result.body, opts);
 
   return { ...outcome, method, data, html, empty };
@@ -150,7 +169,7 @@ function parseExtractBody(
 
   let data: unknown = parsed;
   let html: string | undefined;
-  // extract=auto beta shape: { parsed, html }. Prefer `parsed` for callers.
+  // extract=auto shape: { parsed, html }. Prefer `parsed` for callers.
   if (method === "extract" && parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
     const envelope = parsed as { parsed?: unknown; html?: unknown };
     if ("parsed" in envelope) {

src/adapters/protected-fetch.ts

@@ -34,7 +34,7 @@ export interface FetchOptions {
   cssExtractor?: string;
   autoparse?: boolean;
   /**
-   * Site-tailored Extract (`extract=auto`). Open beta, domain-gated.
+   * Site-tailored Extract (`extract=auto`). Works on any prepared domain.
    * Takes precedence over autoparse / css_extractor / outputs / response_type.
    */
   extract?: boolean;

src/cli/commands/extract.ts

@@ -1,7 +1,7 @@
 /**
  * `zenrows extract <url> [flags]` → Extract (structured extraction on `/v1/`).
  *
- * Default: `extract=auto` (site-tailored Extract, open beta). If the domain is
+ * Default: `extract=auto` (site-tailored Extract). If the domain is
  * not enabled or prepared (AUTH010 / REQS007), falls back once to Autoparse. Opt into Autoparse
  * directly with `--autoparse`. Other methods: `--css`, `--outputs`, `--output`.
  */
@@ -24,7 +24,7 @@ export const extract: Command = {
   usage: "zenrows extract <url> [--autoparse | --css <json> | --outputs <filters> | --output md|text] [flags]",
   help: [
     "Methods:",
-    "  (default)              extract=auto — site-tailored Extract (open beta); falls back to Autoparse if the domain is not enabled or prepared",
+    "  (default)              extract=auto: site-tailored Extract; falls back to Autoparse if the domain is not enabled or prepared",
     "  --autoparse            general-purpose Autoparse (any domain, no Extract fallback)",
     "  --css <json>           CSS selector map, e.g. '{\"title\":\"h1\",\"price\":\".price\"}'",
     "  --outputs <list>       built-in output filters → JSON: emails, phone_numbers, headings,",
@@ -165,7 +165,11 @@ export const extract: Command = {
         `Extracted via ${via} · ${outcome.result.body.length} bytes · ${formatRequestCost(outcome.result.costUsd, outcome.result.costCredits)} · run ${runId}`,
       );
       if (runDir) log.dim(`  artifact: ${runDir}`);
-      if (outcome.empty) {
+      if (outcome.noDataExtracted) {
+        log.warn(
+          "Extract found no data on this page (HTTP 422 no_data_extracted, not charged). Check that the page rendered fully: try --manual --js-render with --wait-for <selector>. Protected sites may also need --premium-proxy.",
+        );
+      } else if (outcome.empty) {
         log.warn(
           "No structured data extracted — the page may render via JavaScript, or this layout isn't auto-extractable. Try --manual --js-render, or --css with explicit selectors.",
         );
@@ -180,6 +184,7 @@ export const extract: Command = {
               method: outcome.method,
               fellBackToAutoparse: outcome.fellBackToAutoparse ?? false,
               empty: outcome.empty ?? false,
+              noDataExtracted: outcome.noDataExtracted ?? false,
               httpStatus: outcome.result.status,
               data: outcome.data ?? null,
               bytes: outcome.result.body.length,

src/core/capabilities.ts

@@ -28,8 +28,8 @@ export function getCapability(key: string): Capability | undefined {
 const USABLE: ReadonlySet<CapabilityStatus> = new Set<CapabilityStatus>([
   "available",
   "available-but-needs-confirmation",
-  // Open beta: any key can call; product-specific limits (e.g. Extract domain
-  // coverage, Batch 403) are handled by the adapters / API errors, not here.
+  // Beta: any key can call; product-specific limits are handled by the
+  // adapters / API errors, not here.
   "beta",
 ]);
 

src/core/http.ts

@@ -37,6 +37,12 @@ export interface ScraperResult {
   concurrencyRemaining: number | null;
   /** The request URL with the apikey redacted — safe to log/persist. */
   redactedUrl: string;
+  /**
+   * True when an Extract request answered 422 with code `no_data_extracted`:
+   * Extract knows this kind of page but none of the fields came back filled.
+   * The request is not charged, and the same request returns the same answer.
+   */
+  noDataExtracted?: boolean;
 }
 
 export interface ScraperParams {
@@ -147,10 +153,31 @@ export async function scrape(
       code: "EXTRACT_DOMAIN_NOT_ENABLED",
       message: "Extract has not been prepared for this domain yet.",
       likely_cause: zrErrorDetail(body) ?? "This domain is not prepared for Extract.",
-      next_action: "Retry with --autoparse for general-purpose extraction on any site.",
+      next_action:
+        "Retry with --autoparse for general-purpose extraction on any site, or get the domain prepared for Extract (POST /v1/extract/prepared-domains).",
       suggested_commands: [`zenrows extract ${params.url} --autoparse`],
     });
   }
+  // 422 no_data_extracted on an Extract request: the page was processed and had
+  // nothing to extract. It is an answer, not a failure to reach the target, so
+  // return it for the extract adapter to report as an empty result instead of
+  // advising escalation (which would repeat a slow, deterministic request).
+  if (res.status === 422 && params.extract !== undefined && zrErrorCode(body) === "NO_DATA_EXTRACTED") {
+    return { ...result, noDataExtracted: true };
+  }
+  // 422 blocked_document on an Extract request: the fetched page was a bot wall,
+  // an error page or an empty shell, so Extract had nothing real to read. Not
+  // charged. The fix is a stronger fetch of the same URL, not another extractor.
+  if (res.status === 422 && params.extract !== undefined && zrErrorCode(body) === "BLOCKED_DOCUMENT") {
+    throw new ToolkitError({
+      code: "EXTRACT_FAILED",
+      message: "Extract could not read this page: the site answered with a block or error page (HTTP 422 blocked_document, not charged).",
+      likely_cause: zrErrorDetail(body) ?? "The fetched page was a bot challenge, an error page or a page that had not finished loading.",
+      next_action:
+        "Retry with a stronger fetch: --manual --js-render --premium-proxy. Check the URL if the site may be returning an error page.",
+      suggested_commands: [`zenrows extract ${params.url} --manual --js-render --premium-proxy`],
+    });
+  }
   if (res.status === 401 || res.status === 403) {
     // Distinguish Zenrows auth errors from target-site 403s. Zenrows auth
     // failures surface AUTH00x codes in the body.
@@ -187,16 +214,16 @@ export async function scrape(
     }
   }
   if (res.status === 402 && isZenrowsErrorEnvelope(body)) {
-    // AUTH010 on extract=auto means the target domain is not in the Extract
-    // beta — distinct from credit exhaustion (AUTH004). Let the extract
+    // AUTH010 on extract=auto means the target domain is not enabled for
+    // Extract, distinct from credit exhaustion (AUTH004). Let the extract
     // adapter fall back to autoparse.
     if (params.extract !== undefined && zrErrorCode(body) === "AUTH010") {
       throw new ToolkitError({
         code: "EXTRACT_DOMAIN_NOT_ENABLED",
         message: "Extract is not enabled for this domain yet.",
-        likely_cause: zrErrorDetail(body) ?? "This domain is not part of the Extract open beta.",
+        likely_cause: zrErrorDetail(body) ?? "This domain is not enabled for Extract.",
         next_action:
-          "Retry with --autoparse for general-purpose extraction on any site, or contact Zenrows support to enable this domain for Extract.",
+          "Retry with --autoparse for general-purpose extraction on any site, or get the domain prepared for Extract (POST /v1/extract/prepared-domains).",
         suggested_commands: [`zenrows extract ${params.url} --autoparse`],
       });
     }

templates/extract-pricing-pages/README.md

@@ -13,5 +13,5 @@ zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse
 zenrows extract <product-url> --css '{"title":"h1","price":".price"}' --validate
 ```
 
-Requires `extract` (open beta). Start on one page, confirm the fields, then
+Requires `extract`. Start on one page, confirm the fields, then
 fan out `zenrows extract` across the rest of your URLs.