fix(docker): serve a hooked request from its own browser, not the shared pool

[mvletter]

A declarative hook changes the browser context it runs in, and the change outlives the request. add_cookies installs cookies into the context; the context belongs to a crawler handed out by crawler_pool.get_crawler(), which is shared — the permanent one for the default browser config, in practice almost every request. release_crawler() only decrements active_requests, so the next request inherits whatever the last one left behind.

Reproduction

Unmodified unclecode/crawl4ai:0.9.3, CRAWL4AI_HOOKS_ENABLED=true, four sequential POSTs to /crawl against a URL that echoes the cookies it received (https://httpbin.org/cookies):

#	request	result
1	no hooks	no cookie — correct
2	add_cookies hook	cookie present — correct
3	no hooks	cookie from #2
4	no hooks	cookie from #2

Sequential, so it does not need concurrent requests to happen.

The same is visible without hooks at all: crawl https://httpbin.org/cookies/set?probe=x, then crawl https://httpbin.org/cookies as a separate request, and the cookie the site set during the first crawl is still there. Cookies stay scoped to their own domain, so this does not cross sites — but it does cross requests, and on a shared deployment that means it crosses callers.

Why scoping the hook does not fix it

Worth stating, because it is the obvious first attempt and it looks like it works. I built it: one dispatcher per hook point on the strategy, resolving the actual hook from a ContextVar set only on the requesting task. Instrumented, the scoping was correct — request 2's hook set was verifiably empty:

KLAIPROBE dispatcher point=on_page_context_created ... hooks=['on_page_context_created']
KLAIPROBE dispatcher point=on_page_context_created ... hooks=[]

And request 2 still came back carrying request 1's cookie. The cookie is in the context, not in the hook.

The change

A request that brings hooks gets a browser of its own, disposed when it finishes. That is the shape the PDF branch already used, so this mostly makes an existing idea explicit:

• crawler_pool.get_unpooled_crawler() starts a crawler outside the pool, behind the same memory guard get_crawler() applies, and marks it pooled = False.
• _dispose_crawler() reads that marker instead of inferring "not pooled" from the strategy type, which only ever recognised PDFCrawlerStrategy.
• The non-streaming handler's inline close/release branch is replaced by that same helper, so both handlers dispose crawlers through one path.

Requests without hooks are untouched: same pool, same counters, same lifetimes.

Cost is one browser start per hooked request. Hooks are opt-in and off by default, and callers that use them tend to batch many URLs into one request, so this is per batch rather than per page.

Tests

tests/docker/test_pool_release.py gains two cases. They import the real crawler_pool rather than a copy of its logic, so they fail if the module changes underneath them — the existing cases in that file re-implement release_crawler locally ("mirrors the logic that will be added to ..."), which cannot catch a change in the code it is meant to cover.

12 passed

The four-request sequence above was re-run against a build of this branch: #3 and #4 come back clean, #2 still gets its cookie.