fix: isolate default request queues per crawler - #2210
Conversation
|
To whoever will review this: please focus on parity with current crawlee-js master, this is a fairly recent addition. Also it's probably v2 material? |
|
Thanks. I checked the queue-selection rule against current crawlee-js master: the first constructed crawler keeps the global default queue, and later implicit managers use This changes behavior for applications that relied on multiple crawlers implicitly sharing a queue, so the release placement matters. The PR currently targets The test label in the description is too opaque. The recorded run covered six new default-queue tests and fifteen existing queue/state/storage tests: 21 passed, 104 deselected. It used uvloop and did not run the full suite or launch a browser. |
Use a separate construction index to retain the first crawler's global default request queue while routing each later implicit manager through the crawler ID-derived stable alias. Reset the counter during unit isolation and cover ordinary, explicit-ID, injected-manager, repeated-ID, and adaptive construction paths.
Verification
Recorded against
2b4ac0104a82723e0b39febc26325dd2b66425d0. From the repository root, with the project and test dependencies plus uvloop installed, the Python test invocation was:python3 -u -c "import asyncio, uvloop, pytest, sys; asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()); sys.exit(pytest.main(['-p', 'no:rerunfailures', '--timeout=60', '-q', '-o', 'addopts=', 'tests/unit/crawlers/_basic', '-k', 'explicit_queue or request_source_tandem or crawler_get_storages or share_state or own_state or share_stats or consecutive_runs or default_storages or other_storages or storage_client_overwrite or state_persistence or default_request_queue']))"Result: 21 passed, 104 deselected, 81 warnings. This selected six new default-queue tests and fifteen existing queue/state/storage tests. The warnings were one unknown
flakymarker and 80 multiprocessing fork deprecations, also observed on the clean base. The test environment selected this checkout'ssrconPYTHONPATH.This was a focused run using uvloop, not the full unit suite. The adaptive-crawler test checks construction and queue ownership without launching a browser.