[security] RobotsParser.can_fetch() disables TLS verification (ssl=False) when fetching robots.txt

[liuchunyi-buaa]

Description

In crawl4ai/utils.py (line 344), RobotsParser.can_fetch() issues the robots.txt request with ssl=False:

session.get(robots_url, timeout=2, ssl=False)

Certificate validation is disabled, so an on-path attacker (MITM, rogue DNS, malicious proxy) can impersonate the target host and feed arbitrary robots.txt content, silently controlling whether the crawler honors Disallow rules for the target site.

Impact is limited (robots.txt is an advisory protocol and the request carries no credentials), so I'm filing this as a hardening issue rather than a private security report.

Suggested fix

Keep TLS verification enabled for https:// URLs; allow opt-in disabling only for explicitly configured hosts:

ssl = parsed.scheme.lower() != "https"  # or an explicit per-request override

Reproduction

1. Route the target host through a proxy that presents a self-signed certificate.
2. Observe the robots.txt request succeeding despite the invalid certificate, and the tampered Disallow rules taking effect.

[zhongxiao-chang]

Claiming this one. I can send a PR against develop that keeps certificate verification on for https:// robots.txt fetches.

Plan:

• crawl4ai/utils.py:344 — drop the unconditional ssl=False so aiohttp uses its default (verified) SSL context, while keeping an explicit opt-out for hosts that legitimately need verification disabled.
• Regression test asserting the robots.txt fetch no longer runs with an unverified context (fake session, no network).

Confirmed on develop: RobotsParser.can_fetch() still issues session.get(robots_url, timeout=2, ssl=False) at line 344.

If you would rather handle this as a private advisory than a public fix, say so and I'll stand down.

[liuchunyi-buaa]

Thanks for picking this up — and yes, a public fix via PR against develop is exactly the path we were hoping for. That's precisely why this was filed as a regular issue rather than a private advisory: please go ahead with the public PR, no need to stand down or treat it privately.

Your plan matches what we had in mind:

1. drop the unconditional ssl=False so aiohttp uses its verified default SSL context,
2. keep an explicit opt-in for hosts that legitimately need verification disabled,
3. add the regression test (fake session, no network).

Line 344 on develop matches what we observed on our side. Happy to review the PR once it's up — thanks for the quick turnaround!

[ntohidi]

@liuchunyi-buaa

Thanks. Two things:

Our SECURITY.md asks that [security] reports go to unclecode@crawl4ai.com and nasrin@crawl4ai.com, not a public issue. Low impact here, so no harm, but please use that next time.

It's already fixed on our side and will ship in the next release. Not on develop yet.

Closing as fixed.