[Mantisus]

Description

RequestManager has a new abstract method, record_pacing_signal(). The crawler reports HTTP 429 responses and robots.txt Crawl-delay directives through it as a RateLimitedSignal or a MinIntervalSignal, both exported from crawlee.request_loaders. RequestQueue refuses signals, RequestManagerTandem forwards them to its manager, and ThrottlingRequestManager paces the domain. The crawler no longer checks the request manager's type, so a ThrottlingRequestManager inside a RequestManagerTandem now receives 429s and crawl-delays too. The upgrade guide covers the migration.

• ThrottlingRequestManager.record_domain_delay() and set_crawl_delay() are replaced by record_pacing_signal(), and is_throttled() and record_success() are removed. base_delay and max_delay must be positive.
• A RequestThrottledError raised in the request handler reports the rate limit itself, so it starts the domain's backoff without a prior call. The request is deferred only if the request manager paces its domain, and is handled as an ordinary error otherwise.
• The warnings about a 429 or a Crawl-delay that the request manager doesn't pace are logged once per domain, and the warning at the start of run() about respect_robots_txt_file is removed.

Issues

• Part of: Replace is_empty/is_finished with check_readiness() and add pacing signals to RequestManager #2296
• Follow-up: refactor(request-loaders)!: Replace is_empty and is_finished with check_readiness #2334
• Closed: ThrottlingRequestManager has no effect behind RequestManagerTandem #2238

✍️ Drafted by Claude Code