Repository navigation
Stop blocked homepage checks from ending live ad network relationships - #124
Merged
Merged
Conversation
…ures A failed or challenged homepage fetch counted as "network not found" for networks that need page evidence, and unchanged ads.txt replies added further misses. Three misses ended live relationships: on 2026-11-13, 1,786 left events landed in one day, and 76% of a live sample still ran AdSense. Blocked, failed, empty and challenge pages are now inconclusive and retried with backoff. Unchanged ads.txt reuses its saved parse but no longer stands in for page evidence. Page evidence older than 30 days makes a relationship stale (not ended), and public checked dates use successful checks only. The per-domain deadline keeps the hourly job inside its budget. correct-homepage-misses.mjs restores relationships only with fresh matching evidence, removes their false left events, and repairs daily counts in bounded, checkpointed batches. The runbook covers backups, a frontend maintenance window, and 25-domain then 100-domain batches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RaviTejaKNTS
temporarily deployed
to
production
November 13, 2026 13:06 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The ad network checker ended live relationships. A failed or challenged homepage fetch (OVH gets bot challenges and 403s from some sites) counted as "network not found" for networks that need page evidence: AdSense, MGID, Taboola, Teads, and others. Unchanged ads.txt replies then added misses without a new page fetch. On 2026-11-13, 1,786
leftevents landed in one day. A live re-check of 120 sites marked as having left AdSense found 91 (76%) still run it.Change
scripts/ad-networks/correct-homepage-misses.mjsrestores wrongly ended or stale relationships only with fresh matching evidence, removes their falseleftevents, and repairs daily counts in bounded, checkpointed batches. New migration: a checkpoint table, backend access only.content_docs/ad-networks/production-runbook.md: backups, a frontend maintenance window, a 25-domain batch, then 100-domain batches.Review
Two reviews by a separate reviewer. Round 1: 2 blockers and 5 should-fix findings, all fixed. Round 2: no blockers; the 2 should-fix and 2 minor findings were fixed by the lead (expiry uses the saved evidence time; a valid absence can also expire an old confirmation; checker wording; removed the manual migration fallback).
Checks
npm run quality: TypeScript and 113 core tests passed.After merge
Run the correction on production following the runbook, starting with 25 domains.
🤖 Generated with Claude Code