[Bug]: when backend deployed on aws restricting to crawl content due to bot restriction

[nikhilbobe-07]

crawl4ai version

0.8.6

Expected Behavior

should extract content on aws also

Current Behavior

extracting content on local

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

Code snippets

OS

macos,windows

Python version

3.14

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response