Repository navigation
crawler: 10 s fetch, 30 s render, slow or broken targets parked 30 days (0.49.0) - #194
Merged
Merged
Conversation
…ys (0.49.0) Fleet crawler rule (2026-11-08): "crawlers should timeout after 10 seconds for fetch, 30 for browser. Flag to recheck later in a month." - politeGet: one 10 s budget (CRAWL_FETCH_TIMEOUT_MS) for the name lookup, every redirect hop and the body; was 20 s per hop with no body cutoff. - Obscura render: waitUntil load (not networkidle0), a 1.5 s bounded settle, 30 s total (CRAWL_RENDER_TIMEOUT_MS); out of budget rejects RenderTimeout. Obscura's own trouble is ObscuraError and never parks a target. - src/failures.ts: timeout/DNS/TLS/connection/5xx/429 classification. robots.txt failing that way parks the host 30 days (it used to read a timeout as "no rules" and time out every page 3x). Page failures are not thrown to BullMQ: DNS/TLS/connection/429 park the host, a timeout or 5xx parks the page, 5 within an hour park the host. 429 honours Retry-After, capped. - sites gains status, last_error, failed_at, retry_after (added once, in the site store's schema list); takeDue skips retry_after in the future, so a flagged host gets exactly one recheck after it; success clears the flag. - Redis pitcrawl:down:<site>, pitcrawl:pdown:<urlid>, pitcrawl:fails:<site>, read in the same pipeline as the cached robots.txt (no extra round trip). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ralyodio
changed the title
sh1pt-actions-fleet
Bot
force-pushed
the
crawl-10s-30s-30d
branch
from
November 8, 2026 17:04
aaae4d5 to
eadba90
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fleet crawler rule (2026-11-08): 10 s per plain fetch, 30 s per browser render, and a slow or broken target is flagged and left alone 30 days, then rechecked once.
politeGet: one 10 s budget (CRAWL_FETCH_TIMEOUT_MS) for DNS, every redirect and the body (was 20 s per hop).waitUntil: load+ 1.5 s bounded settle, 30 s total (CRAWL_RENDER_TIMEOUT_MS); timeout rejects (RenderTimeout). Obscura's own outages (ObscuraError) never park a target.src/failures.ts: timeout/DNS/TLS/connection/5xx/429. robots.txt failure parks the host 30 d (a timeout used to mean "no rules"). Page failures of those kinds are no longer thrown to BullMQ's 3 attempts: host-level kinds park the host, timeout/5xx park the page, 5 in an hour park the host. 429 honours Retry-After, capped at 30 d.sites:status, last_error, failed_at, retry_after(added once via the site store's schema list);takeDueskips futureretry_after; success clears the flag.test/failures.test.ts+ Redis/Postgres cases inredis.test.ts(run locally against throwaway redis:8 / postgres:17: 139 pass).🤖 Generated with Claude Code