Repository navigation
Send a standard browser Accept header for ad network homepage checks - #126
Merged
Merged
Conversation
Some site firewalls answer 403 to the bare "text/html,application/xhtml+xml" list, which browsers never send. From OVH, with the same stack127bot user agent, 350cc.com and 1teachmore.lk return 403 to it and the full page to the standard browser value. About 40% of live correction checks were inconclusive, and 14 of 15 sampled were HTTP status failures. The user agent and the HTML content-type check are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RaviTejaKNTS
temporarily deployed
to
production
November 13, 2026 16:41 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The ad network checker requests homepages with
Accept: text/html,application/xhtml+xml. Browsers never send that exact value, and some site firewalls answer it with 403. From OVH, with the samestack127bot/1.0user agent:350cc.comand1teachmore.lk: 403 with the old header, 200 with the full page with the standard browser value (and with curl's*/*).13milers.com: 403 either way.In the production homepage-miss correction running now, about 40% of live checks are inconclusive. 14 of 15 sampled were
http_statusfailures.Change
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.Accept: text/plain.Checks
npm run quality: 113 core tests passed.🤖 Generated with Claude Code