feat(extract): replace the autoparse fallback with zenrows extract prepare

README.md

@@ -37,7 +37,7 @@ that endpoint with extraction parameters, not a separate product.
 | Command | What it does | Status (this build) |
 | --- | --- | --- |
 | `zenrows fetch` | Fetch: retrieve a protected page | **available**: `GET https://api.zenrows.com/v1/` |
-| `zenrows extract` | Extract: turn a page into structured data with `extract=auto` on any prepared domain (falls back to Autoparse on one not prepared yet), CSS selectors, or Markdown | **available**: `GET https://api.zenrows.com/v1/?extract=auto` |
+| `zenrows extract` | Extract: turn a page into structured data with `extract=auto` on any prepared domain (prepare one yourself with `zenrows extract prepare`), CSS selectors, or Markdown | **available**: `GET https://api.zenrows.com/v1/?extract=auto` |
 | `zenrows batch` | Batch: run Fetch and Extract over many URLs in one job; specs validate and estimate locally with no key | **available**: `POST https://async.api.zenrows.com/v1/jobs` |
 | `zenrows crawl` | Crawl (new): collect the URLs behind one start page, optionally with each page's HTML | **available**: `POST https://api.zenrows.com/v1/crawls` |
 | `zenrows browser` | Browser Sessions: a real cloud browser over REST (same backend as MCP `browser_*`); escalation-only, billed by bandwidth + time | **available**: `POST https://mcp.zenrows.com/browser/sessions` |
@@ -56,7 +56,7 @@ npm install -g @zenrows/cli # puts `zenrows` on PATH (466 ms, zero deps)
 zenrows init
 zenrows fetch https://httpbin.io/html # auto-provisions a Free plan account on first use
 zenrows extract https://www.owler.com/company/meltwater # extract=auto on a prepared domain
-zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse # Autoparse (any domain)
+zenrows extract https://www.scrapingcourse.com/ecommerce/ --css '{"title":"h1"}' # CSS selectors (any domain)
 ```
 
 > Prefer no global install? Prefix each command with `npx -y @zenrows/cli`
@@ -117,12 +117,22 @@ zenrows fetch <url> --proxy-country us --wait-for ".price"
 
 ```bash
 zenrows extract https://www.owler.com/company/meltwater                # extract=auto (prepared domain)
-zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse   # Autoparse (any domain)
+zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse   # deprecated Autoparse (any domain)
 zenrows extract <url> --css '{"title":"h1","price":".price"}' --validate
 zenrows extract <url> --outputs emails,links                            # built-in output filters → JSON
 zenrows extract <url> --output markdown
 ```
 
+On a domain Extract hasn't prepared yet, `zenrows extract` fails with REQS007
+and prints the command to prepare it. Preparing takes a few minutes and costs
+no credits; each new domain uses one slot of the monthly allowance.
+
+```bash
+zenrows extract prepare example.com --url https://example.com/p/1 --url https://example.com/list --wait
+zenrows extract prepare status example.com   # progress + monthly allowance
+zenrows extract https://example.com/p/1      # then extract again
+```
+
 ## 9. Batch
 
 Zenrows **Batch** (`https://async.api.zenrows.com/v1`) fans a

agent-plugin/README.md

@@ -62,7 +62,7 @@ Use MCP tool names — not CLI `zenrows fetch`:
 | --- | --- |
 | `zenrows` | Decision tree across all primitives |
 | `protected-fetch` | `scrape` |
-| `extract` | `extract` (fallback: `scrape` + autoparse / css_extractor) |
+| `extract` | `extract` (fallback: `scrape` + css_extractor); on REQS007 `prepare_domain` then `prepared_domain_status` |
 | `batch-jobs` | `batch_create`, `batch_status`, `batch_results`, `batch_cancel` |
 | `interact-browser` | `browser_*` (escalation only) |
 | `cost-control` | Cost multipliers + escalate-with-evidence |

agent-plugin/skills/cost-control/SKILL.md

@@ -19,8 +19,8 @@ These multipliers apply when you enable the flags (on `scrape` or `extract`):
 - Both: **25×**
 
 `extract` `mode=auto` is currently free; billing may apply
-later. Do not promise permanent free extract pricing. Autoparse / CSS still
-follow the multipliers above when stealth flags are set.
+later. Do not promise permanent free extract pricing. CSS (and the deprecated
+Autoparse) still follow the multipliers above when stealth flags are set.
 
 ## Browser sessions (`browser_*`)
 

agent-plugin/skills/extract/SKILL.md

@@ -14,28 +14,53 @@ Convert protected pages into structured data. Value is **protected page access
 Use **`extract`** when you need structured JSON fields rather than a full page:
 
 ```
-extract({ url })                                    # mode=auto (default)
-extract({ url, mode: "autoparse" })                 # general Autoparse (any domain)
+extract({ url })                                    # mode=auto (default), returns `parsed` only
+extract({ url, include_html: true })                # also return the full page HTML
+extract({ url, mode: "autoparse" })                 # deprecated Autoparse (any domain)
 extract({ url, mode: "css", css_extractor: '{"title":"h1","price":".price"}' })
 extract({ url, proxy_country: "US" })               # geo-targeted
 ```
 
-`mode=auto` (`extract=auto`) works on any domain Zenrows has prepared, with any
-key, and is currently free; billing may apply later. If the
-domain is not enabled (AUTH010) or not prepared yet (REQS007), the tool retries
-once with Autoparse by default (`fallback_autoparse`).
+`mode=auto` (`extract=auto`) works on any prepared domain, with any key, and
+is currently free; billing may apply later. The result carries `parsed` only;
+the page HTML is most of the response size, so ask for it with
+`include_html: true` only when you need it.
 
 Every mode uses Adaptive Stealth Mode by default: pass just the URL, even for
 protected or dynamic sites. Set `js_render` or `premium_proxy` only to force a
 fixed configuration; that turns Adaptive Stealth Mode off. Check `empty` in the
 result: true means no field came back with a value.
 
+## Domain not prepared yet (REQS007): prepare it, then extract again
+
+`mode=auto` works on any prepared domain, and you can prepare one yourself. On
+a domain Extract hasn't prepared yet, `extract` returns a tool error (`REQS007`;
+legacy `AUTH010`) whose `next_tool` holds the `prepare_domain` arguments. It does
+not retry with Autoparse unless you set `fallback_autoparse: true`. The flow:
+
+```
+extract({ url })                                          # 1. error REQS007
+prepare_domain({ domain: "example.com",
+  example_urls: ["https://example.com/p/1", "https://example.com/list"] })
+                                                          # 2. returns at once, status "preparing"
+prepared_domain_status({ domain: "example.com" })         # 3. every ~30 s until status "ready"
+extract({ url })                                          # 4. extract again
+```
+
+- `example_urls`: 1 to 5 real pages you want to extract, one per kind of page.
+- Preparing takes a few minutes and costs no credits. Each new domain uses one
+  slot of the account's monthly allowance; `prepared_domain_status` returns it
+  (`used` / `limit` / `remaining`) and a `next` field with what to do now.
+- `out_of_allowance`: no slots left this month; use `mode=css`.
+  `overloaded` / `disabled`: retry in a few minutes, do not loop.
+  A `failed` status carries `failure.code`; follow `next`.
+
 ## Fallback: `scrape` extraction params
 
 If `extract` is unavailable, similar outcomes are available on **`scrape`**:
 
 ```
-scrape({ url, autoparse: true })
+scrape({ url, autoparse: true })                    # deprecated
 scrape({ url, css_extractor: '{"title":"h1","price":".price"}' })
 scrape({ url, outputs: "emails,links" })            # or "*" for all built-ins
 ```
@@ -46,7 +71,8 @@ scrape({ url, outputs: "emails,links" })            # or "*" for all built-ins
 ## When to use which
 
 - **`mode=auto`**: site-tailored fields on any prepared domain.
-- **`mode=autoparse`**: any website; also the automatic AUTH010 / REQS007 fallback.
+- **`mode=autoparse`** (deprecated): any website, only when asked for. Do not
+  build new work on it.
 - **`mode=css`**: known fields/selectors; deterministic.
 - **Markdown via `scrape`**: feed clean content to an LLM yourself.
 
@@ -57,7 +83,7 @@ Stealth flags (`js_render`, `premium_proxy`) still apply the usual multipliers
 ## Rules
 
 - Validate on a single page before `batch_create` ([[batch-jobs]]).
-- If Autoparse misses fields, switch to `css_extractor`, or add `js_render`
+- If a field is missing, switch to `css_extractor`, or add `js_render`
   for JS-heavy pages.
 - Prefer MCP `extract` / `scrape` over CLI `zenrows extract` when the MCP
   server is connected.

agent-plugin/skills/protected-fetch/SKILL.md

@@ -34,7 +34,7 @@ scrape({ url, js_render: true, wait_for: ".price" })
 - Prefer `response_type: "markdown"` for LLM consumption; use `html` only when
   you need raw markup.
 - For **structured** fields, prefer the `extract` tool (or `scrape` with
-  `autoparse` / `css_extractor` / `outputs`) — see [[extract]].
+  `css_extractor` / `outputs`) — see [[extract]].
 - On failure, escalate with evidence before `browser_*` ([[interact-browser]]).
 - Cost model: [[cost-control]].
 

agent-plugin/skills/zenrows/SKILL.md

@@ -36,7 +36,7 @@ If the user has a known URL and wants page content:
 
 If the user has a known URL and wants structured data:
   → Use Extract.                    (MCP tool: extract)
-                                     Fallback: scrape with autoparse / css_extractor / outputs
+                                     Fallback: scrape with css_extractor / outputs
 
 If the user has many URLs (bulk):
   → Use Batch.                      (validate one page with scrape/extract first,
@@ -66,7 +66,8 @@ If the user needs login, clicks, forms, sessions, or persistent state:
 | Primitive | MCP tools | Notes |
 | --- | --- | --- |
 | Protected Fetch | `scrape` | Default markdown; HTML / plaintext / PDF / screenshot options |
-| Extract | `extract` | Structured JSON (`extract=auto`, autoparse, css_extractor). `extract=auto` is currently free; billing may apply later |
+| Extract | `extract` | Structured JSON (`extract=auto` on any prepared domain, css_extractor; autoparse deprecated). `extract=auto` is currently free; billing may apply later |
+| Prepare a domain | `prepare_domain`, `prepared_domain_status` | After an `extract` REQS007: prepare the domain with 1 to 5 example URLs, poll until ready, extract again. No credits; uses the monthly allowance |
 | Batch | `batch_create`, `batch_status`, `batch_results`, `batch_cancel` | Same credits as Fetch; a 403 maps to `BATCH_ACCESS_DENIED` |
 | Browser | `browser_*` (start with `browser_navigate`) | Escalation-only; same backend as Browser Sessions |
 

docs/capabilities.md

@@ -18,7 +18,7 @@ Classification is based on the public Zenrows documentation:
 | Capability | Backend evidence | Status |
 | --- | --- | --- |
 | `protected_fetch` | Fetch `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available |
-| `extract` | Extract `GET https://api.zenrows.com/v1/` via `extract=auto` (any prepared domain; CLI falls back to `autoparse` on one not prepared yet), plus `autoparse`, `css_extractor`, `outputs`, `response_type=markdown\|plaintext` | available |
+| `extract` | Extract `GET https://api.zenrows.com/v1/` via `extract=auto` (any prepared domain; on one not prepared yet the CLI reports REQS007 with the command to prepare it: `zenrows extract prepare <domain> --url <page>`, which calls `POST /v1/extract/prepared-domains`), plus deprecated `autoparse`, `css_extractor`, `outputs`, `response_type=markdown\|plaintext` | available |
 | `batch` | Batch `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header). Local JSONL spec validation + credit estimation work with no key. | available |
 | `crawl` | Crawl `https://api.zenrows.com/v1/crawls` (`X-API-Key` header; same host as Fetch/Extract). Follows the links on each page from one start URL, up to `depth` when set (optional; no depth limit when omitted), on the start URL's registrable domain (subdomains count); returns URLs, and with `output_format: "html"` each page's HTML. Subcommands: `create`, `get`, `list`, `results`, `content`, `download`, `stop`, `wait`. Without Crawl access the API returns 403 `REQS008` → `CRAWL_NOT_ENABLED`. | available |
 | `browser` | Browser Sessions REST API `https://mcp.zenrows.com/browser/sessions/*` (Bearer; same backend as `@zenrows/mcp` `browser_*`); CDP via `zenrows browser connect`. Escalation-only (prefer fetch/extract); on by default, opt out via `policy.allow_browser`; bills by bandwidth + session time (15-min max) | available |

registry/capabilities.json

@@ -12,12 +12,12 @@
     },
     "extract": {
       "key": "extract",
-      "label": "Extract (extract=auto / Autoparse / CSS / Markdown)",
+      "label": "Extract (extract=auto / CSS / Markdown)",
       "status": "available",
       "command": "zenrows extract",
-      "backend": "Extract — GET https://api.zenrows.com/v1/ (extract=auto, autoparse, css_extractor, outputs, response_type)",
+      "backend": "Extract — GET https://api.zenrows.com/v1/ (extract=auto, css_extractor, outputs, response_type; autoparse deprecated)",
       "requiresAuth": true,
-      "notes": "Default is extract=auto, which works on any prepared domain with any key; CLI falls back to autoparse on REQS007 (domain not prepared yet) or AUTH010. Autoparse / CSS / outputs / markdown work on any domain."
+      "notes": "Default is extract=auto, which works on any prepared domain with any key. On a domain not prepared yet (REQS007, legacy AUTH010) the CLI reports the error and does not retry; prepare the domain with `zenrows extract prepare <domain> --url <page> [--wait]` (POST /v1/extract/prepared-domains: 1 to 5 example URLs, a few minutes, no credits) and check it with `zenrows extract prepare status <domain>`. CSS / outputs / markdown work on any domain; autoparse is deprecated and runs only with --autoparse."
     },
     "batch": {
       "key": "batch",

registry/skills.json

@@ -25,13 +25,13 @@
     {
       "name": "extract",
       "type": "skill",
-      "description": "Turn protected pages into structured data with Autoparse / CSS / Markdown.",
+      "description": "Turn protected pages into structured data with Extract / CSS / Markdown (Autoparse is deprecated).",
       "status": "available",
       "requires_backend_capabilities": ["extract"],
       "requires_auth": true,
       "version": "0.1.0",
       "path": "skills/extract",
-      "tags": ["extract", "autoparse", "structured-data"]
+      "tags": ["extract", "structured-data"]
     },
     {
       "name": "batch-jobs",

registry/templates.json

@@ -14,7 +14,7 @@
     {
       "name": "extract-pricing-pages",
       "type": "template",
-      "description": "Extract structured pricing fields from product pages (Autoparse / CSS).",
+      "description": "Extract structured pricing fields from product pages (extract=auto / CSS).",
       "status": "available",
       "requires_backend_capabilities": ["extract"],
       "requires_auth": true,

skills/extract/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: extract
-description: Turn protected pages into structured data with Extract / Autoparse / CSS / Markdown.
+description: Turn protected pages into structured data with Extract / CSS / Markdown (Autoparse is deprecated).
 version: 0.1.0
 requires_backend_capabilities: [extract]
 ---
@@ -12,23 +12,51 @@ access + extraction**, not generic LLM parsing.
 
 ## Methods
 ```
-zenrows extract <url>                              # extract=auto (default); falls back to Autoparse if the domain is not prepared yet
-zenrows extract <url> --autoparse                  # general-purpose Autoparse (any domain)
+zenrows extract <url>                              # extract=auto (default), any prepared domain
+zenrows extract <url> --autoparse                  # deprecated general-purpose Autoparse (any domain)
 zenrows extract <url> --css '{"title":"h1","price":".price"}'   # selector map
 zenrows extract <url> --outputs emails,links       # built-in output filters → JSON
 zenrows extract <url> --output markdown            # Markdown conversion
 zenrows extract <url> --validate                   # fail if not valid JSON
 ```
 
 ## When to use which
-- **Extract** (`extract=auto`, default): richest fields; works on any domain Zenrows has prepared.
-- **Autoparse**: any website; also the automatic fallback when the domain is not prepared for Extract yet.
+- **Extract** (`extract=auto`, default): richest fields; works on any prepared
+  domain (you can prepare one yourself, see below).
+- **Autoparse** (deprecated): only with `--autoparse`. Never automatic; do not
+  build new work on it.
 - **CSS**: you know the exact fields/selectors and want determinism.
 - **Markdown/plaintext**: feed clean content to an LLM yourself.
 
+## Domain not prepared yet (REQS007): prepare it, then extract again
+
+`extract=auto` works on any prepared domain, and you can prepare one yourself.
+On a domain Extract hasn't prepared yet, `zenrows extract` fails with
+`EXTRACT_DOMAIN_NOT_ENABLED` (API code `403 REQS007`; legacy `402 AUTH010`) and
+prints the command to run. It does not retry with Autoparse. The flow:
+
+```
+zenrows extract https://example.com/p/1             # 1. fails: Extract hasn't prepared example.com yet
+zenrows extract prepare example.com \
+  --url https://example.com/p/1 --url https://example.com/list --wait
+                                                    # 2. prepare it and wait until ready (a few minutes)
+zenrows extract https://example.com/p/1             # 3. extract again
+```
+
+- `--url`: 1 to 5 real pages you want to extract, one per kind of page.
+- Without `--wait` the command returns at once; then poll with
+  `zenrows extract prepare status example.com` until it says `ready`.
+- Preparing costs no credits. Each new domain uses one slot of the account's
+  monthly allowance; `prepare status` shows `used` / `limit` / `remaining`.
+- `EXTRACT_PREPARE_OUT_OF_ALLOWANCE`: no slots left this month; use `--css`.
+  `EXTRACT_PREPARE_UNAVAILABLE`: retry in a few minutes, do not loop.
+  A `failed` status prints the next step for its failure code.
+
 ## Rules
 - Validate on a single page before scaling across many URLs.
-- If Autoparse misses fields, switch to `--css` with explicit selectors. Auto
+- Check `empty` in `--json` output: true means every field came back null or
+  empty, so the result is not data.
+- If a field is missing, switch to `--css` with explicit selectors. Auto
   mode already renders JS when the target needs it, so reach for
   `--manual --js-render` only after auto mode has returned an incomplete page:
   it costs several times a basic call ([[cost-control]]) and it makes the