Repository navigation
feat(extract): replace the autoparse fallback with zenrows extract prepare - #47
Draft
zenrows-dev-pool[bot] wants to merge 3 commits into
Draft
zenrows-dev-pool[bot] wants to merge 3 commits into
zenrows-dev-pool[bot] wants to merge 3 commits into
Conversation
…g back to autoparse On 403 REQS007 (domain not prepared yet) or the legacy 402 AUTH010, `zenrows extract` used to retry with the deprecated autoparse. That hid the prepare step, charged for an Autoparse call and often returned unusable data. The error is now reported, not retried, and names the next step: Extract hasn't prepared <domain> yet. Prepare it with POST /v1/extract/prepared-domains (a few example URLs, a few minutes, no credits), then retry; or run with --autoparse. `--autoparse` stays as the explicit opt-in and now warns that Autoparse is deprecated. The adapter's `fallbackAutoparse` option defaults to false. The empty-result check now treats a payload whose fields are all null or empty (recursively) as empty, so page scaffolding no longer reads as data. Skills, docs and the capability matrix describe the prepare step, mark autoparse deprecated and drop the fallback wording.
Removing the autoparse fallback leaves REQS007 as the answer on a domain Extract has not prepared yet, so CLI users need a way to prepare one. - `zenrows extract prepare <domain> --url <u> [--url <u>...]` submits the domain with 1 to 5 example URLs to POST /v1/extract/prepared-domains and returns at once. `--wait` (off by default) polls until ready or failed, up to `--timeout`. On 409 in_progress it follows the running preparation. - `zenrows extract prepare status <domain>` shows progress, the failure code and detail, the account's monthly allowance and the next step. - Errors name the next step: EXTRACT_PREPARE_INVALID (400), EXTRACT_PREPARE_NOT_FOUND (404), EXTRACT_PREPARE_IN_PROGRESS (409), EXTRACT_PREPARE_OUT_OF_ALLOWANCE (429), EXTRACT_PREPARE_UNAVAILABLE (503 overloaded / disabled), EXTRACT_PREPARE_FAILED (failed status, with the step for its failure code), EXTRACT_PREPARE_TIMEOUT. - The REQS007 error now prints the exact command: zenrows extract prepare <domain> --url <url>. Skills, README and the capability matrix teach the flow: extract, REQS007, prepare, poll, extract again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
zenrows extractno longer retries with the deprecated Autoparse when Extract has not prepared a domain. It reports the error with the exact command to prepare the domain, and this PR adds that command, so users can prepare a domain themselves and extract again.Flow:
zenrows extract <url>→ REQS007 →zenrows extract prepare <domain> --url <url> --wait→zenrows extract <url>.Behaviour
403 REQS007/ legacy402 AUTH010onextract=autoautoparse=true;fellBackToAutoparse: true; Autoparse chargedEXTRACT_DOMAIN_NOT_ENABLEDwith the message below; nothing retriedzenrows extract prepare <domain> --url <u> [--url <u>...] [--wait [--timeout <s>]]andzenrows extract prepare status <domain>--autoparsefallbackAutoparsetruefalse(explicit opt-in kept)null/""/[](for example page scaffolding)empty: falseempty: true, so the usual warning (or--validatefailure) appliesThe REQS007 message, short so agents can copy it:
suggested_commandsare the same command with--wait,zenrows extract prepare status example.com, and the original extract. The error code staysEXTRACT_DOMAIN_NOT_ENABLED, so callers that match on it keep working.zenrows extract preparePOST /v1/extract/prepared-domains(X-API-Key) with{"domain", "example_urls"}.--urltakes 1 to 5 full http(s) URLs; anything else is rejected before any request. A URL orwww.host is normalized to the domain.--waitpollsGET /v1/extract/prepared-domains/{domain}every 15 seconds untilreadyorfailed(default timeout 900 seconds).prepare statusshows status, state, per-page-kind roles, the failure, the monthly allowance (GET .../allowance: used / limit / remaining) and the next step. If the allowance call fails, the status is still shown.prepare status ..., or the extract once ready)in_progressEXTRACT_PREPARE_IN_PROGRESSonly when this account cannot read itinvalid_argument/ 409idempotency_conflictEXTRACT_PREPARE_INVALID(withreason)not_found(status)EXTRACT_PREPARE_NOT_FOUND, with the prepare commandout_of_allowanceEXTRACT_PREPARE_OUT_OF_ALLOWANCE(with used / limit); do not retry this month, use--cssoverloaded/disabledEXTRACT_PREPARE_UNAVAILABLE; retry in a few minutes, do not loopfailed(with--wait)EXTRACT_PREPARE_FAILED, exit 1, with the next step for each documented failure code--waitruns outEXTRACT_PREPARE_TIMEOUT, with the status commandDocs and skills
skills/extract,skills/zenrows,agent-plugin/skills/*,README.md,docs/capabilities.md, the competitor-intelligence workflow, the pricing template andregistry/*.jsonnow teach the extract → REQS007 → prepare → poll → extract flow. They mark Autoparse deprecated, drop "falls back to autoparse", and say "any prepared domain (you can prepare one yourself)". Theagent-pluginextract skill describes the MCP server'sprepare_domain/prepared_domain_statustools and new defaults, which ship in the matching zenrows-mcp PR.Release
No version change here. Per
docs/contributing.md, release-please sets the version from thefeat(extract)title after merge.Tests
npm run typecheck: clean. The repo has no linter beyondtsc.npm test: 384 passed, 0 failed. New:tests/extract-prepare.test.tscovers submit,--waitto ready,--waitto failed,--waittimeout, 409 followed and 409 unreadable, 400, 429, 503 overloaded and disabled,--urlvalidation, status with and without the allowance, status 404, every failure code, and routing fromzenrows extract. Updated: REQS007 and AUTH010 print the prepare command and are not retried; the fallback runs only withfallbackAutoparse: true; all-null nested payloads are empty; the capability notes name the prepare commands.prepare statuson a ready domain,prepare quotes.toscrape.com --url ... --wait(preparing → ready, then printed the extract command), andprepare statuson a domain never submitted (404 mapped).