Repository navigation
crawler: tunable site-start concurrency; frontier pool 3 per process (0.49.2) - #196
Merged
Merged
Conversation
…3 (0.49.2) Two ceilings hit scaling toward 1M pages/h (2026-11-08): - Site starts ran max(2, concurrency/4) = 8 at once on dev2. A start is mostly politeness waits (robots, sitemaps, home page, probes, 1.5 s apart), so too few sites were active: 144k page jobs delayed, 0 ready. CRAWL_SITE_CONCURRENCY. - Each process opened up to 8 frontier connections; 11 processes held 70 idle, the pitcrawl role's cap, and the 12th could not connect, so its site starts hung. CRAWL_DATABASE_POOL, default 3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two ceilings hit while scaling toward 1M pages/h:
max(2, concurrency/4)= 8 at once. A start is mostly politeness waits (robots, sitemaps, home page and probes, 1.5 s apart), so too few sites were active: 144k page jobs were delayed and 0 were ready.CRAWL_SITE_CONCURRENCYnow sets it, and dev2 will run 64.pitcrawlrole's cap, so the 12th could not connect and its site starts hung.CRAWL_DATABASE_POOLnow sets it, defaulting to 3.Crawler: 131 tests pass.
🤖 Generated with Claude Code