Repository navigation
Conversation
github-actions
Bot
added
the
t-tooling
Mantisus
marked this pull request as ready for review
October 15, 2026 21:59
vdusek
reviewed
Oct 16, 2026
Mantisus
force-pushed
the
check-readiness
branch
2 times, most recently
from
October 18, 2026 17:01
3629140 to
a30518d
Compare
vdusek
reviewed
Oct 21, 2026
|
|
||
| request_manager = await self.get_request_manager() | ||
| return not await request_manager.is_empty() | ||
| return (await request_manager.check_readiness()).status == 'ready' |
Collaborator
There was a problem hiding this comment.
Note: The pool calls __is_finished_function and then __is_task_ready_function on every pass, and both now run check_readiness(). On a queue that is empty but not finished, each call makes two backend calls (is_empty() + is_finished()), so a pass costs 4 calls where it used to cost 2. That covers the end of a crawl, waiting on in-progress requests, and throttler cooldowns. On the platform these are API calls, and ThrottlingRequestManager repeats them for every domain queue. Could the status from __is_finished_function be reused here, or is that too tied to the call order in the pool?
✍️ Drafted by Claude Code
vdusek
reviewed
Oct 21, 2026
Mantisus
force-pushed
the
check-readiness
branch
2 times, most recently
from
October 21, 2026 18:44
bbe9b52 to
09c2557
Compare
Mantisus
force-pushed
the
check-readiness
branch
from
October 23, 2026 20:07
09c2557 to
57c0d67
Compare
Pijukatel
reviewed
Oct 26, 2026
Mantisus
force-pushed
the
check-readiness
branch
from
October 27, 2026 15:07
57c0d67 to
1560e63
Compare
vdusek
approved these changes
Oct 28, 2026
vdusek
left a comment
vdusek
left a comment
Collaborator
There was a problem hiding this comment.
Two more minor comments, otherwise LGTM
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Mantisus
force-pushed
the
check-readiness
branch
from
October 28, 2026 12:29
e6c0520 to
c787aaa
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
RequestLoaderandRequestManagerreplaceis_empty()andis_finished()with a singlecheck_readiness(). It returns one of four statuses, exported fromcrawlee.request_loaders:ReadyStatus,WaitingStatus,StalledStatusorFinishedStatus.WaitingStatus.ready_atsays when a source expects to have work again, andThrottlingRequestManagersets it from its cooldowns.StalledStatus.reasonsays why a source can't make progress. Storage clients keepis_empty()andis_finished(). The upgrade guide covers the migration.ThrottlingRequestManager.get_stall_reason()is removed. The crawler reads a stall from the status, so it now also detects a stalled domain when theThrottlingRequestManageris wrapped in aRequestManagerTandem.RequestManagerTandemmoves a request from its loader only when the loader is ready, so aSitemapRequestLoaderthat's still loading sitemaps no longer holds up the requests already in the manager.RequestLoader.fetch_next_request()no longer has to wait for a request. ANonemeans nothing is pending right now, not that the loader is finished.SitemapRequestLoader.fetch_next_request()returnsNoneonce its sitemaps are loaded, instead of waiting until every fetched request is handled.Issues
is_empty/is_finishedwithcheck_readiness()and add pacing signals toRequestManager#2296✍️ Drafted by Claude Code