Repository navigation
fix(crawl): say a crawl with nothing new to follow completes, so users stop chasing their patterns - #50
Merged
Conversation
`crawl create --output-format markdown` stores each kept URL's page converted to Markdown, read back with `crawl content` and `crawl download` as `--output-format html` already was. A value that is neither is still refused locally, before any network call. A crawl also reports `discovery`, the strategies it found URLs with, so `crawl get --json` carries it instead of dropping it.
The lock names the API description this surface was synced to, by its blob id, so a later drift check compares against it instead of reporting the surface as never synced. Two operations in that description are not served to callers and no surface exposes them; the lock lists them as pending, so a change to either does not read as drift here.
The Crawl API now ends a crawl that runs out of credits, or whose API key hits a credit cap, as `stopped` with `error.code` `insufficient_credits` or `api_key_cap_reached` and no `stop_reason`, instead of `failed`. The CLI treated every `stopped` crawl as a deliberate stop, so such a crawl would have exited 0 with `ok: true`. It now exits 1 with `CRAWL_FAILED` and the API's code as `server_code`, as it did when the crawl ended `failed`, with next-step advice for both codes. `crawl stop` on such a crawl says it had already ended. The crawl skill, the trace-debug skill and the README say so.
A crawl that ends `stopped` because the account ran out of credits or the API key hit a credit cap now exits 1 with `CRAWL_INSUFFICIENT_CREDITS` or `CRAWL_KEY_CAP_REACHED`, the codes a create refused with 402 already gets, instead of `CRAWL_FAILED`. It carries the same next-step advice as those codes, the API's `error.code` as `server_code`, its `error.detail` in `likely_cause`, and `crawl_id`. A `failed` crawl with either of those codes, as older deployments report it, maps the same way, so callers see one code whatever the server version. Any other error on a stopped or failed crawl stays `CRAWL_FAILED`. `quotaExhausted` takes `status: null`, as `keyCapReached` does, for an error read from a crawl rather than an HTTP 402. The crawl skill, the trace-debug skill and the README say so.
The fixtures now carry the error.detail the server returns when credits end a crawl, one per error code.
The Crawl API now applies `include_patterns` and `exclude_patterns` only to the URLs a crawl returns. The crawl still opens on-site pages that fail them, to reach pages that match, and each of those fetches counts toward `max_pages` and is billed. A crawl whose patterns match no URL it read ends `completed` with no results instead of `failed` with `no_items_found`, and `no_items_found` now only means the pages read held nothing to follow. `crawl get`, `crawl wait` and `create --follow` print a note when a completed crawl with patterns kept nothing, telling the caller to loosen the patterns or raise `--depth` or `--max-pages`. The exit code stays 0. The `no_items_found` next action points at the start page instead of the patterns. The `--max-pages` and pattern help, the README, the crawl skill and acceptance check 19 say so.
The lock lives beside the contract, not in a surface. main already dropped it, and this branch had brought it back.
…to fix/crawl-patterns-output-filter
…imit the walk too Patterns limit both the pages a crawl opens and the URLs it returns, so the help, README and skill no longer say the crawl opens or bills pages that fail them. A completed crawl with 0 results still exits 0, and with patterns set its note says to check them against the URLs the start page links to. no_items_found now means Extract found no data on the start page, and seed_unreachable also covers an empty start page.
zenrows-dev-pool
Bot
changed the title
…tput-filter # Conflicts: # skills/crawl/SKILL.md # src/cli/commands/crawl.ts # tests/crawl-command.test.ts
…y matches the API The next_action and the skill now say the crawl read the start page but found no data it could use, and suggest starting from a listing or product page.
StefanoGuerrini
marked this pull request as ready for review
October 16, 2026 13:56
StefanoGuerrini
approved these changes
Oct 16, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before: The Crawl API ended a crawl that kept nothing
failedwithno_items_found, sozenrows crawl waitexited 1 withCRAWL_FAILEDand told the user to loosen their patterns. An earlier version of this PR also said patterns filter only the URLs a crawl returns, and that pages opened to walk past them count toward--max-pagesand are billed. That is not how Crawl works.After:
completedwith 0 results. The CLI printsok: trueand exits 0.--include-patternsor--exclude-patterns, the CLI adds a note (under--json, innote) to check them against the URLs the start page links to.no_items_foundstill exits 1 withCRAWL_FAILED. Itsnext_actionnow says the crawl read the start page but found no data it could use, and suggests starting from a listing or product page.seed_unreachableexits 1 withCRAWL_FAILED. Itsnext_actionnow also covers a start URL that returned an empty page.How: Merged the current base into this branch with merge commits, twice. The second merge, of
main, reported conflicts inskills/crawl/SKILL.md,src/cli/commands/crawl.tsandtests/crawl-command.test.ts. In each,mainheld the credit-stop change, now squash-merged there, and this branch held the same text plus the 0-result wording. The resolution kept both.crawlFailurecarries the newnext_actiontext for both codes.noMatchNotesets the patterns note only for acompletedcrawl with 0 items and patterns set. README,skills/crawl/SKILL.mdand the help text drop the walk and billing wording. Acceptance row 19 and its surface check assertcompleted, 0 results, noerror, exit 0 and a note.Tests:
npm run typecheck: clean.npm run build: clean.npm test: 376 tests, 376 pass, 0 fail.no_items_foundcrawl, and an empty start page failingseed_unreachable.