Skip to content

fix: isolate default request queues per crawler - #2210

Open
snowyukitty wants to merge 2 commits into
apify:masterfrom
snowyukitty:fix/per-crawler-default-queue
Open

fix: isolate default request queues per crawler#2210
snowyukitty wants to merge 2 commits into
apify:masterfrom
snowyukitty:fix/per-crawler-default-queue

Conversation

@snowyukitty

@snowyukitty snowyukitty commented Sep 6, 2026

Copy link
Copy Markdown

Use a separate construction index to retain the first crawler's global default request queue while routing each later implicit manager through the crawler ID-derived stable alias. Reset the counter during unit isolation and cover ordinary, explicit-ID, injected-manager, repeated-ID, and adaptive construction paths.

Verification

Recorded against 2b4ac0104a82723e0b39febc26325dd2b66425d0. From the repository root, with the project and test dependencies plus uvloop installed, the Python test invocation was:

python3 -u -c "import asyncio, uvloop, pytest, sys; asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()); sys.exit(pytest.main(['-p', 'no:rerunfailures', '--timeout=60', '-q', '-o', 'addopts=', 'tests/unit/crawlers/_basic', '-k', 'explicit_queue or request_source_tandem or crawler_get_storages or share_state or own_state or share_stats or consecutive_runs or default_storages or other_storages or storage_client_overwrite or state_persistence or default_request_queue']))"

Result: 21 passed, 104 deselected, 81 warnings. This selected six new default-queue tests and fifteen existing queue/state/storage tests. The warnings were one unknown flaky marker and 80 multiprocessing fork deprecations, also observed on the clean base. The test environment selected this checkout's src on PYTHONPATH.

This was a focused run using uvloop, not the full unit suite. The adaptive-crawler test checks construction and queue ownership without launching a browser.

@janbuchar

Copy link
Copy Markdown
Collaborator

To whoever will review this: please focus on parity with current crawlee-js master, this is a fairly recent addition. Also it's probably v2 material?

@snowyukitty

Copy link
Copy Markdown
Author

Thanks. I checked the queue-selection rule against current crawlee-js master: the first constructed crawler keeps the global default queue, and later implicit managers use __default_<crawler id>__. Injected managers are retained. This PR preserves Python's existing automatic ID allocation; JS derives automatic IDs from its construction index, so the ID allocation itself is not identical.

This changes behavior for applications that relied on multiple crawlers implicitly sharing a queue, so the release placement matters. The PR currently targets master; which branch should it target for v2?

The test label in the description is too opaque. The recorded run covered six new default-queue tests and fifteen existing queue/state/storage tests: 21 passed, 104 deselected. It used uvloop and did not run the full suite or launch a browser.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants