refactor(http-clients)!: Handle redirects and cookies in the HttpClient base class and drop crawl()

docs/examples/code_examples/file_download_stream.py

@@ -14,9 +14,9 @@ async def main() -> None:
     # been read yet.
     crawler = FileDownloadCrawler(
         stream=True,
-        # Bounds establishing the connection and receiving the response headers.
+        # Bounds the whole download: the connection, the response headers and the body.
         navigation_timeout=timedelta(minutes=5),
-        # The body is downloaded inside the handler, so this bounds the transfer itself.
+        # The body is downloaded inside the handler, so this bounds the transfer as well.
         request_handler_timeout=timedelta(minutes=5),
         # Limit the crawl to max requests. Remove or increase it for crawling all links.
         max_requests_per_crawl=10,

docs/guides/code_examples/http_clients/parsel_httpx_example.py

@@ -8,7 +8,6 @@ async def main() -> None:
     http_client = HttpxHttpClient(
         # Optional additional keyword arguments for `httpx2.AsyncClient`.
         timeout=10,
-        follow_redirects=True,
     )
 
     crawler = ParselCrawler(

docs/guides/extending_crawlee.mdx

@@ -132,11 +132,11 @@ An HTTP client performs network calls for crawlers. Swapping it changes the tran
 
 The contract is <ApiLink to="class/HttpClient">`HttpClient`</ApiLink>:
 
-- `crawl` performs a request inside the crawler's pipeline and returns the result the crawler consumes.
-- `send_request` covers standalone calls made from a handler.
-- `stream` yields a response you read incrementally.
+- `_fetch` sends a single request and returns the response with its body still unread. It doesn't follow redirects or keep cookies.
 - `cleanup` releases whatever the client holds open.
 
+The base class builds `send_request` on top of `_fetch`. The crawler's pipeline and the handlers both call it. It follows redirects and sends and stores the session cookies, so every client handles them the same way.
+
 Crawlee ships <ApiLink to="class/ImpitHttpClient">`ImpitHttpClient`</ApiLink>, <ApiLink to="class/HttpxHttpClient">`HttpxHttpClient`</ApiLink>, and <ApiLink to="class/CurlImpersonateHttpClient">`CurlImpersonateHttpClient`</ApiLink>.
 
 See the [HTTP clients guide](./http-clients) for the libraries these clients wrap and their installation requirements.

docs/guides/http_clients.mdx

@@ -101,15 +101,14 @@ python -m pip install 'crawlee[all]'
 
 Crawlee provides an abstract base class, <ApiLink to="class/HttpClient">`HttpClient`</ApiLink>, which defines the interface that all HTTP clients must implement. This allows you to create custom HTTP clients tailored to your specific requirements.
 
-HTTP clients are responsible for several key operations:
+To create a custom HTTP client, inherit from <ApiLink to="class/HttpClient">`HttpClient`</ApiLink> and implement two methods:
 
-- sending HTTP requests and receiving responses,
-- managing cookies and sessions,
-- handling headers and authentication,
-- managing proxy configurations,
-- connection pooling with timeout management.
+- `_fetch` sends a single request, through the proxy from `proxy_info` if there is one, and returns the response without reading its body. It doesn't follow redirects or keep cookies of its own. Its `timeout` bounds both receiving the response and reading the body, and without one the default timeout of the underlying library applies.
+- `cleanup` closes the connections and other resources the client holds.
 
-To create a custom HTTP client, you need to inherit from the <ApiLink to="class/HttpClient">`HttpClient`</ApiLink> base class and implement all required abstract methods. Your implementation must be async-compatible and include proper cleanup and resource management to work seamlessly with Crawlee's concurrent processing model.
+The response is a subclass of the abstract <ApiLink to="class/HttpResponse">`HttpResponse`</ApiLink> class. Its body is read on `read` or `read_stream`, and `aclose` releases the connection when the body isn't needed. The base class provides the async context manager, which closes the response on exit.
+
+The base class provides `send_request` on top of `_fetch`. It sends the cookies of the session with every request, stores the cookies the server sets, and follows up to 10 redirects unless the client is created with `follow_redirects=False`. A custom client therefore handles redirects and cookies the same way as the built-in ones.
 
 ## Conclusion
 

docs/guides/secure_scraping.mdx

@@ -63,7 +63,7 @@ If only your crawler's storage sees the response, the same request is a *blind*
 A hostile site can try to use up your CPU, memory, or queue rather than read anything:
 
 - A *crawler trap* is a maze of randomly generated URLs linking to more generated pages. It's built to keep a crawler running forever and grow its queue without bound. Bound how far link-following goes with <ApiLink to="class/BasicCrawlerOptions#max_crawl_depth">`max_crawl_depth`</ApiLink>, and cap a whole run with <ApiLink to="class/BasicCrawlerOptions#max_requests_per_crawl">`max_requests_per_crawl`</ApiLink>. Both are unlimited by default, so set them yourself.
-- A single *oversized or slow response* can tie up a worker or blow up memory. Both the fetch and the handler are capped at 60 seconds by default. <ApiLink to="class/HttpCrawlerOptions#navigation_timeout">`navigation_timeout`</ApiLink> covers the fetch and is passed straight to the HTTP client. <ApiLink to="class/BasicCrawlerOptions#request_handler_timeout">`request_handler_timeout`</ApiLink> covers the handler. Raise them if a legitimate target needs longer, but keep the cap as tight as your workload allows.
+- A single *oversized or slow response* can tie up a worker or blow up memory. Both the fetch and the handler are capped at 60 seconds by default. <ApiLink to="class/HttpCrawlerOptions#navigation_timeout">`navigation_timeout`</ApiLink> covers the fetch, including reading the body, and is passed straight to the HTTP client. <ApiLink to="class/BasicCrawlerOptions#request_handler_timeout">`request_handler_timeout`</ApiLink> covers the handler. Raise them if a legitimate target needs longer, but keep the cap as tight as your workload allows.
 - A *decompression bomb* is an archive that expands to gigabytes when unpacked. If your crawler downloads and extracts archives, cap the output stream during extraction. The declared size is part of the attack and easy to forge.
 
 ## Untrusted content

docs/upgrading/upgrading_to_v2.md

@@ -201,6 +201,79 @@ for urls in batches:
 
 In v1, `RequestQueue.purge()` worked around a storage client that didn't support purging. It dropped an unnamed queue and opened the default queue in its place, and it skipped a named queue with a warning. In v2, it raises the client's `NotImplementedError`, the same as Crawlee JS. The Apify platform's storage client doesn't support purging, so on the platform, call `drop()` or open a new queue instead.
 
+## HTTP clients
+
+`HttpClient` follows the design of Crawlee JS v4. A client sends a single request, and the base class handles redirects and cookies the same way for every client.
+
+### Responses are read lazily
+
+`HttpClient.crawl` and `HttpClient.stream` are removed. `send_request` returns the response as soon as its headers arrive and reads the body only on `read` or `read_stream`. A response whose body you don't read keeps its connection until you call `aclose`. Use the response as an async context manager to close it:
+
+```python
+# v1
+async with http_client.stream(url) as response:
+    async for chunk in response.read_stream():
+        ...
+
+# v2
+async with await http_client.send_request(url) as response:
+    async for chunk in response.read_stream():
+        ...
+```
+
+Responses returned by `context.send_request` are closed after the request is processed, so a request handler doesn't need to close them.
+
+### `timeout` covers the redirects and the body
+
+The `timeout` of `send_request` is one deadline for the whole redirect chain and for reading the body, the same as Crawlee JS. A body read past it raises `TimeoutError`. Without a `timeout`, the default of the underlying library applies. It's 30 seconds for `ImpitHttpClient`, including reading the body, and for `CurlImpersonateHttpClient`, and 5 seconds per operation for `HttpxHttpClient`. In v1, `ImpitHttpClient` and `CurlImpersonateHttpClient` disabled the timeout altogether in that case. To change the default, pass `timeout` when you create the client.
+
+The crawlers pass the remaining `navigation_timeout` to the HTTP client, so it bounds reading the body too. With `FileDownloadCrawler(stream=True)`, the deadline also covers the body streamed in the request handler, with every HTTP client. In v1, only `ImpitHttpClient` bounded the streamed body, while `HttpxHttpClient` and `CurlImpersonateHttpClient` bounded only the connection. For long downloads, raise `navigation_timeout` together with `request_handler_timeout`.
+
+### `send_request` follows at most 10 redirects
+
+`send_request` follows up to 10 redirects and raises `RuntimeError` when a server redirects more often. The limit isn't configurable. In v1, each client raised the `TooManyRedirects` error of its library instead, after 20 redirects with `ImpitHttpClient` and `HttpxHttpClient` and after 30 with `CurlImpersonateHttpClient`. Every client now takes `follow_redirects`. With `follow_redirects=False`, `send_request` returns the first response as it is. The redirect options of the underlying libraries are ignored with a warning. For the list, see [Some client arguments are ignored](#some-client-arguments-are-ignored).
+
+### Client headers follow the redirect rules
+
+The `headers` argument of a client still applies to every request. Its `Authorization`, `Proxy-Authorization` and `Cookie` headers are dropped when a redirect leaves the origin, the same as the headers passed with a request. With `HttpxHttpClient`, the headers of the client now win over the generated browser headers. In v1, the generated browser headers replaced them.
+
+### Some client arguments are ignored
+
+The following keyword arguments of the underlying libraries are now ignored with a warning:
+
+- `cookies` of every client and `cookie_jar` of `ImpitHttpClient`, since cookies are kept in sessions.
+- `auth` and `params` of `HttpxHttpClient` and `CurlImpersonateHttpClient`, which would reach every request of a redirect chain. Pass the credentials in the `Authorization` header and the query in the URL instead.
+- `max_redirects` of every client and `allow_redirects` of `CurlImpersonateHttpClient`, since `send_request` follows the redirects.
+
+### Custom clients implement `_fetch`
+
+A custom client implements `_fetch` and `cleanup` instead of `crawl`, `send_request` and `stream`. `_fetch` sends one request and returns the response without following redirects or keeping cookies. The client inherits `send_request` from `HttpClient`.
+
+For example:
+
+```python
+# v1
+class MyHttpClient(HttpClient):
+    async def crawl(self, request, **kwargs): ...
+
+    async def send_request(self, url, **kwargs): ...
+
+    def stream(self, url, **kwargs): ...
+
+    async def cleanup(self): ...
+
+
+# v2
+class MyHttpClient(HttpClient):
+    async def _fetch(self, url, *, method, headers, payload, proxy_info, timeout): ...
+
+    async def cleanup(self): ...
+```
+
+`HttpClient.__init__` is no longer abstract, so a custom client doesn't have to define it. A custom `__init__` must call `super().__init__()`, because the base `__init__` stores the `follow_redirects` and `headers` options that `send_request` reads.
+
+`HttpResponse` is now an abstract base class instead of a protocol, so a custom response subclasses it. It has three new abstract members, `url`, `headers_set_cookie` and `aclose`, and works as an async context manager that closes the response.
+
 ## Logs changed
 
 If you configure loggers by name or match log messages, update them for the following changes.

src/crawlee/_types.py

@@ -566,7 +566,9 @@ class SendRequestFunction(Protocol):
     """A function for sending HTTP requests.
 
     It simplifies the process of sending HTTP requests. It is implemented by the crawling context and is used
-    within request handlers to send additional HTTP requests to target URLs.
+    within request handlers to send additional HTTP requests to target URLs. The body of the response is not read yet.
+    The response is closed once the request is processed, and until then it stays in memory with the body read by
+    `read`, so a handler that sends many requests keeps all of their responses.
     """
 
     def __call__(
@@ -585,7 +587,8 @@ def __call__(
             method: The HTTP method to use.
             headers: The headers to include in the request.
             payload: The payload to include in the request.
-            timeout: Maximum time allowed to process the request.
+            timeout: Maximum time allowed to receive the response, including all redirects, and to read its body.
+                Without it, the default timeout of the HTTP client applies.
 
         Returns:
             The HTTP response received from the server.

src/crawlee/_utils/http.py

@@ -2,13 +2,25 @@
 
 from __future__ import annotations
 
+import asyncio
 import re
+from contextlib import suppress
 from datetime import UTC, datetime, timedelta
 from email.utils import parsedate_to_datetime
 from logging import getLogger
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from crawlee.http_clients import HttpResponse
 
 logger = getLogger(__name__)
 
+_MAX_DRAINED_BODY_SIZE = 64 * 1024
+"""The largest unread body that `drain_small_body` reads."""
+
+_MAX_DRAIN_SECONDS = 1
+"""How long `drain_small_body` waits for the body."""
+
 _CHARSET_PATTERN = re.compile(r'(?:^|;)\s*charset\s*=\s*(?:"([^"]*)"|([^;\s]*))', re.IGNORECASE)
 
 
@@ -66,3 +78,16 @@ def parse_retry_after_header(value: str | None) -> timedelta | None:
         pass
 
     return None
+
+
+async def drain_small_body(response: HttpResponse) -> None:
+    """Read the body of a response known to be small, so that closing the response keeps its HTTP/1.1 connection.
+
+    `httpx2` and `impit` drop the connection of a response closed before its body arrives.
+    """
+    content_length = response.headers.get('content-length', '')
+    if content_length.isdigit() and int(content_length) <= _MAX_DRAINED_BODY_SIZE:
+        # If reading fails or takes too long, closing the response drops the connection anyway.
+        with suppress(Exception):
+            async with asyncio.timeout(_MAX_DRAIN_SECONDS):
+                await response.read()

src/crawlee/_utils/robots.py

@@ -57,15 +57,18 @@ async def load(cls, url: str, http_client: HttpClient, proxy_info: ProxyInfo | N
         try:
             response = await http_client.send_request(url, proxy_info=proxy_info)
 
-            status_code = response.status_code
-            if is_status_code_client_error(status_code):
-                # A 4xx response means the file is unavailable, so crawling is unrestricted (RFC 9309, section 2.3.1.3).
-                body = b'User-agent: *\nAllow: /'
-            elif is_status_code_server_error(status_code):
-                # A 5xx response means the file is unreachable, so assume complete disallow (RFC 9309, section 2.3.1.4).
-                body = b'User-agent: *\nDisallow: /'
-            else:
-                body = await response.read()
+            async with response:
+                status_code = response.status_code
+                if is_status_code_client_error(status_code):
+                    # A 4xx response means the file is unavailable, so crawling is unrestricted (RFC 9309, section
+                    # 2.3.1.3).
+                    body = b'User-agent: *\nAllow: /'
+                elif is_status_code_server_error(status_code):
+                    # A 5xx response means the file is unreachable, so assume complete disallow (RFC 9309, section
+                    # 2.3.1.4).
+                    body = b'User-agent: *\nDisallow: /'
+                else:
+                    body = await response.read()
             robots = Protego.parse(body.decode('utf-8'))
 
         except Exception as e:

src/crawlee/_utils/sitemap.py

@@ -400,9 +400,10 @@ async def _fetch_and_process_sitemap(
     while retries_left > 0:
         retries_left -= 1
         try:
-            async with http_client.stream(
+            response = await http_client.send_request(
                 sitemap_url, method='GET', headers=SITEMAP_HEADERS, proxy_info=proxy_info, timeout=timeout
-            ) as response:
+            )
+            async with response:
                 status_code = response.status_code
                 if not _is_successful_sitemap_status(status_code):
                     # Retryable statuses raise and route to the retry handler below; the rest skip this source.
@@ -740,6 +741,8 @@ def _check_and_add(url: str) -> bool:
                 response = await http_client.send_request(
                     candidate, method=method_for_checking, proxy_info=proxy_info, timeout=request_timeout
                 )
+                # Only the status code is checked, the body is never read.
+                await response.aclose()
                 if is_status_code_successful(response.status_code) and _check_and_add(candidate):
                     yield candidate
             except ProxyError:

src/crawlee/crawlers/_abstract_http/_abstract_http_crawler.py

@@ -123,6 +123,7 @@ def _create_static_content_crawler_pipeline(self) -> ContextPipeline[ParsedHttpC
             .compose(self._make_http_request)
             .compose(self._execute_post_navigation_hooks)
             .compose(self._handle_status_code_response)
+            .compose(self._read_http_response)
             .compose(self._parse_http_response)
             .compose(self._handle_blocked_request_by_content)
         )
@@ -280,16 +281,30 @@ async def _make_http_request(self, context: BasicCrawlingContext) -> AsyncGenera
             The original crawling context enhanced by HTTP response.
         """
         async with self._shared_navigation_timeouts[id(context.request)] as remaining_timeout:
-            result = await self._http_client.crawl(
-                request=context.request,
+            response = await self._http_client.send_request(
+                context.request.url,
+                method=context.request.method,
+                headers=context.request.headers,
+                payload=context.request.payload,
                 session=context.session,
                 proxy_info=context.proxy_info,
-                statistics=self._statistics,
                 timeout=remaining_timeout,
             )
 
-        context.request.state = RequestState.AFTER_NAV
-        yield HttpCrawlingContext.from_basic_crawling_context(context=context, http_response=result.http_response)
+        try:
+            self._statistics.register_status_code(response.status_code)
+            context.request.loaded_url = response.url
+            context.request.state = RequestState.AFTER_NAV
+            yield HttpCrawlingContext.from_basic_crawling_context(context=context, http_response=response)
+        finally:
+            await response.aclose()
+
+    async def _read_http_response(self, context: HttpCrawlingContext) -> AsyncGenerator[HttpCrawlingContext, None]:
+        """Read the body of the HTTP response within the navigation timeout."""
+        async with self._shared_navigation_timeouts[id(context.request)]:
+            await context.http_response.read()
+
+        yield context
 
     async def _handle_status_code_response(
         self, context: HttpCrawlingContext