Your blog article advertises deceptive practices

[flying-sheep]

Your scraper does the right thing, congrats!

scrapingant-client-python/scrapingant_client/client.py

Line 29 in a0cff7e

self.user_agent = f'ScrapingAnt Client/{version} ({sys.platform}; Python/{platform.python_version()});'

But your blog entry here: https://scrapingant.com/blog/change-user-agent-httpx
advertises to pretend to be a browser. Please change that blog entry.

[kami4ka]

@flying-sheep thanks!

It is a really debatable area. The web is moving towards less strict bot policies due to all AI agents and more search engine competition, but it is still insanely botophobic.

[flying-sheep]

The web is moving towards less strict bot policies due to all AI agents […]

Huh! Everything I see is the exact opposite!

Namely that because all the AI scrapers behave so much worse than what came before, a lot of web pages that let everything go through until now started to deploy anti-AI-crawler tools like CloudFlare’s turnstile, nepenthes, anubis, e.g. ReadTheDocs, and a lot of really important places like https://gitlab.gnome.org/.