Add option to respect robots.txt disallows

[tw4l]

Fixes #631

This PR introduces a --robots CLI flag to the crawler. When enabled, the crawler will attempt to fetch robots.txt for each host encountered during crawling, and if found will check if pages are disallowed before adding them to the crawl queue.

Fetched robots.txt bodies are cached by their URL in Redis using an LRU cache mechanism which retains the 100 most recently accessed ones in the cache to prevent memory usage from getting out of control.

Robots.txt bodies are parsed and checked for page allow/disallow status using the https://github.com/samclarke/robots-parser library, which is the most active and well-maintained implementation I could find with TypeScript types.

I have added basic tests that check the log lines to ensure robots.txt is being fetched and cached, but I haven't been able to find a robots.txt on a Webrecorder-managed domain with disallows to test that disallowed URLs are not actually queued. Maybe we could set up a single disallow on our website's robots.txt for this purpose?

Manual testing

1. In util/constants.ts, set ROBOTS_CACHE_LIMIT to 2
2. Re-build the crawler image and run a crawl across multiple hosts, with --robots and --logging debug enabled and a mixture of URLs that do and do not have a robots.txt specified, and for those that do, a mixture of URLs that will be allowed and disallowed, e.g.:

docker compose build
docker compose run crawler crawl --url https://forums.gentoo.org --url https://forums.gentoo.org/search.php --url https://forums.gentoo.org/statistics.php --url https://forums.gentoo.org/viewforum-f-33.html\?sid\=ed69503a6d2b866c8bcc28307f8e7d59 --url https://webrecorder.net/browsertrix --url https://webrecorder.net --url https://bl.uk --url https://bl.uk/collection --url https://bl.uk/visit --url https://www.kb.dk --url https://www.kb.dk/README.txt --url https://www.kb.dk/search/ --url https://www.kb.dk/en/find-materials --url https://bitarchivist.net --limit 20 --scopeType page --collection robots-cache --robots --generateWACZ --logging debug,errors

3. Inspect the crawl log for relevant messages with robots and links context. What should be present:

Log lines for fetching and caching robots.txt

{"timestamp":"2025-09-30T14:45:03.888Z","logLevel":"debug","context":"robots","message":"Fetching robots.txt","details":{"url":"https://forums.gentoo.org/robots.txt"}}
{"timestamp":"2025-09-30T14:45:04.226Z","logLevel":"debug","context":"robots","message":"Caching robots.txt body","details":{"url":"https://forums.gentoo.org/robots.txt"}}

Log lines for using cached robots.txt

{"timestamp":"2025-09-30T14:45:04.234Z","logLevel":"debug","context":"robots","message":"Using cached robots.txt body","details":{"url":"https://forums.gentoo.org/robots.txt"}}

Log lines for not queueing page URLs disallowed by robots

{"timestamp":"2025-09-30T14:45:04.235Z","logLevel":"debug","context":"links","message":"Page URL not queued, disallowed by robots.txt","details":{"url":"https://forums.gentoo.org/search.php"}}

Log lines for deleting least-recently accessed cached robots.txt when over cache limit

{"timestamp":"2025-09-30T14:45:04.843Z","logLevel":"debug","context":"robots","message":"Deleting cached robots.txt, over cache limit","details":{"url":"https://forums.gentoo.org/robots.txt"}}

Log lines for hosts where robots.txt cannot be found or fetched

{"timestamp":"2025-09-30T14:45:05.619Z","logLevel":"debug","context":"robots","message":"Robots.txt not fetched","details":{"url":"https://bitarchivist.net/robots.txt","status":404}}

[ikreymer]

This handles the robots.txt matches for pages, but not for subpage resources. Perhaps we should have a:
--robots all|pages option, with all also intercepting all the requests (like blockrules) which can be handled with interceptRequest() similar to blockrules.

Perhaps we have a RobotsHandler that handles both interception + page queue filtering?

[ikreymer]

This handles the robots.txt matches for pages, but not for subpage resources. Perhaps we should have a:
--robots all|pages option, with all also intercepting all the requests (like blockrules) which can be handled with interceptRequest() similar to blockrules.

Perhaps we have a RobotsHandler that handles both interception + page queue filtering?

Based on user feedback, decided to keep just keep this focused on pages only, can add other options later if there is a need.

---

Added some minor refactoring in: e9e3738:

• move robots logic to ./src/utils/robots.ts
• add --robotsAgent arg, defaulting to Browsertrix/1.x
• remove debug logging for 'using cached robots' as its too verbose (for every link), harder to see other message
• cache empty "" robots responses and treat as allow all
• treat non-200 and non-429 and non-503 responses as empty ""

Added the cacheing of empty "", and error responses as "", otherwise, these were getting fetched every time and not cached.

[ikreymer]

Hm, while the retry mechanism is good and something we should perhaps add for required crawl resources (see: #921), here it could potentially lead to indefinite waiting just to fetch robots.txt, for every link. If a site is rate limiting access to robots.txt, that could potentially slow down the whole crawl.

I think we should reevaluate this in a later pass, and remove for now, also adding a timedRun() around the robots fetch, perhaps limiting to 10 seconds at most

[ikreymer]

Probably the way to handle this would be to fetch robots async, outside of the main crawler loop, and also avoid starting a fetch multiple times for the same domain.

[ikreymer]

Also added quick batching of multiple requests, to avoid sending multiple requests for robots.txt in one crawler.
Could further improve for multi-instance deployments, but should be good first pass.