Keep the OVH job checkout current and stop discovery failing on one source

content_docs/ad-networks/maintenance.md

@@ -2,7 +2,7 @@
 
 ## Recurring production jobs (since 2026-10-19)
 
-The jobs run as plain cron on the OVH server and write straight to the production database. Niche sorting runs on homelab with its logged-in Claude CLI and the reviewed Haiku alias; no model API key is needed. All ad-network pages stay unpublished and return 404. The [production runbook](production-runbook.md) covers paths, checks, updates, pausing and recovery.
+The jobs run as plain cron on the OVH server and write straight to the production database. Niche sorting runs on homelab with its logged-in Claude CLI and the reviewed Haiku alias; no model API key is needed. All ad-network pages stay unpublished and return 404. The [production runbook](production-runbook.md) covers paths, checks, updates, pausing and recovery. The [hourly checkout updater](../../scripts/stack127/checkout/README.md) keeps OVH job code current. Manual updates and the first install run start only at UTC minutes 15–28, protecting scheduled jobs from losing their turn to updater locks. Set its hold before a maintenance window, wait for an active update, and remove your hold only after the window is complete. Keep the runbook's exact cron prefixes and cache timer steps.
 
 | Job | Schedule, UTC | Work |
 | --- | --- | --- |
@@ -19,7 +19,7 @@ The jobs run as plain cron on the OVH server and write straight to the productio
 
 The crontab and its wrapper are [`scripts/ad-networks/ovh/`](../../scripts/ad-networks/ovh/). One session-level PostgreSQL advisory lock lets only one heavy job run at a time. A scheduled job that finds it taken retries every minute for up to 6 hours, and verification steps aside while one waits. `flock` stops the same job from starting twice. Site-based enrichment stops admitting work after 30 minutes or 5,000 sites. Source-feed jobs need a complete feed and have an outer timeout. Existing host limits remain authoritative.
 
-`node --experimental-strip-types scripts/ad-networks/refresh.mjs --job <job> --environment development` prints the full collector, sync, and revalidation plan without changing anything. `--apply` runs it. Production additionally requires `--environment production --confirm stack127-production`. Every scheduled run ends with a consistent reviewed sync and an authenticated on-demand origin revalidation. Completed checkpoints are still synced when a collector fails, and the run still counts as failed. A failed sync or revalidation never reports success. Snapshots and backups also use this final pipeline. Normal operation has no separate human sync step.
+`node --experimental-strip-types scripts/ad-networks/refresh.mjs --job <job> --environment development` prints the full collector, sync, and revalidation plan without changing anything. `--apply` runs it. Production additionally requires `--environment production --confirm stack127-production`. Every scheduled run ends with a consistent reviewed sync and an authenticated on-demand origin revalidation. Completed checkpoints are still synced when a collector fails. Discovery logs one isolated source failure as a warning and succeeds if another source completed; all sources failing or the same source failing on consecutive runs fails the job. Source status persists in SQLite `source_state` (or the existing JSON state file for PostgreSQL discovery). A successful HTTP 304 clears failure status while preserving feed metadata. Database write failures always fail. Other collector failures still fail the run. A failed sync or revalidation never reports success. Snapshots and backups also use this final pipeline. Normal operation has no separate human sync step.
 
 Sync's database triggers already queue `ad_networks:all` and affected page changes. The runner's extra revalidation immediately expires the origin data/page caches. It deliberately does not delete queued events. The existing Supabase cache worker verifies/purges the legacy host, or dispatches the Cloudflare static frontend refresh, before acknowledging events. In Cloudflare mode the public files update after that workflow completes. No collector publishes routes.
 
@@ -47,13 +47,22 @@ Job failures reuse the existing `GITHUB_ALERT_TOKEN` and the repository's GitHub
 
 Keep the one-time homelab RDAP pass and gather supervisor running. When both finish, follow "Merge the homelab RDAP pass" in the runbook: `merge-rdap.mjs`, run through `refresh --job rdap-import`, merges only newer RDAP checkpoints and registration dates by domain into OVH's store, then syncs and revalidates. It never replaces the working SQLite file, niches, site checks, sources, or relationships.
 
-## GitHub preview workflows
+### Outbrain seller-feed access
 
-The repository discovery and verification workflows are manual-only previews. Neither has a schedule or passes `--apply`. Discovery needs no database credential in preview mode. Verification reads candidates from PostgreSQL and writes only its summary artifact, without saving database results. Its default preview batch is 100 domains.
+On November 14, 2026, plain GETs to `https://www.outbrain.com/sellers.json` returned HTTP 406 with an empty body from both OVH and the Mac. From OVH, `application/json`, wildcard, broader JSON/plain-text and full HTML Accept values all failed with the unchanged `stack127bot/1.0 (+https://stack127.com/ad-networks)` identity. Compression and language negotiation also failed. The response included `X-Served-By: cache-lcy-…`, `Cache-Control: private, max-age=600`, and `Content-Length: 0`.
 
-The verifier rejects `--store postgres --apply`. Apply verification to the SQLite working store, then copy reviewed results through `sync`; this keeps one verification write path. PostgreSQL previews still load protected review signals.
+This is an unresolved upstream refusal; the response does not identify its exact rule. It is not proven to be an Accept-only problem, unlike the homepage cases in PR #126. Discovery (including curl downloads) and publisher-name requests now consistently accept JSON, plain text and wildcard content while keeping the bot identity. This does not establish that Outbrain is fixed. The source `sellers-json:teads-outbrain` is paused in `sources.mjs` (since November 14, 2026), so the weekly job stops failing on it. Its last good run on October 27 read 974 domains and added no new ones. To try it again, run `npm run ad-networks:discover -- --kind sellers-json` with the source re-enabled in a reviewed change; do not impersonate a browser. Existing SQLite `last_status='failed'` also counts as the preceding failed run under the new policy.
 
-These workflows do not change the independent homelab workers described below. Keep production pages unpublished and draft tables private. Before enabling a production sync or schedule, review the queue import, permissions, identity rules, sample results, and limits. Publication is a separate registry decision.
+## Manual previews on OVH
+
+The manual GitHub discovery and verification workflows have been removed. Use the OVH checkout for previews, without `--apply`. First set the [checkout hold](../../scripts/stack127/checkout/README.md#maintenance-hold-and-manual-commands) and wait for an active updater. Remove your hold after the preview. Discovery needs no database credential. Verification reads the protected candidates and writes a local summary, without saving database results.
+
+```sh
+ssh ovh 'cd /srv/stack127 && npm run ad-networks:discover -- --store postgres --kind sellers-json'
+ssh ovh 'cd /srv/stack127 && set -a && . /srv/stack127-data/ad-networks.env && set +a && npm run ad-networks:verify -- --store postgres --limit 100 --concurrency 4 --time-budget-minutes 10'
+```
+
+The verifier rejects `--store postgres --apply`. Apply verification to the SQLite working store, then copy reviewed results through `sync`; this keeps one verification write path. PostgreSQL previews still load protected review signals. Publication is a separate registry decision.
 
 ## Phase 2 one-time import
 
@@ -143,7 +152,7 @@ The default is a dry run. `--apply` is required for database writes. `--source`
 
 HTTP ETag, Last-Modified, candidate counts, and source failures are stored in SQLite. A 304 response skips an unchanged source. Source count drops greater than 50% cause a failure for review; disappearance from a source never deletes a candidate. The old PostgreSQL path remains available with `--store postgres` for a later controlled sync.
 
-Check the workflow log for failed sources, unusually low domain counts, and changes in redirects or robots rules. When a source URL changes, update the manifest and this guide, run its dry run, then apply that source in development. Keep `source_key` stable across URL changes so existing provenance remains idempotent. Before a new source is scheduled, record its exact URL, extraction method, permitted use, robots result, and review date.
+Check the OVH discovery log for failed sources, unusually low domain counts, and changes in redirects or robots rules. When a source URL changes, update the manifest and this guide, run its dry run, then apply that source in development. Keep `source_key` stable across URL changes so existing provenance remains idempotent. Before a new source is scheduled, record its exact URL, extraction method, permitted use, robots result, and review date.
 
 ## Verify publisher relationships
 

content_docs/ad-networks/plan.md

@@ -143,7 +143,7 @@ This phase establishes the page and database contract. It does not assign a netw
 
 Database history is reconciled through the cleanup release. Keep the canonical versions in both managed projects. The old development ledger gaps and SQL experiments are historical context, not pending migrations.
 
-The GitHub discovery and verification workflows remain manual-only, without `--apply`. They preview candidates and checks without saving database changes. Keep their production schedules disabled until a separate activation review. The independent homelab SQLite workers remain separate.
+The manual GitHub discovery and verification workflows have been removed. Run previews without `--apply` in the OVH checkout, using the hold and commands in [the maintenance guide](maintenance.md#manual-previews-on-ovh). Recurring production jobs run on OVH.
 
 ### 2. Gather the initial domain list
 

content_docs/ad-networks/production-runbook.md

@@ -18,6 +18,8 @@ On OVH (`ssh ovh`, user `ubuntu`, UTC):
 
 The env file also sets `GITHUB_ALERT_TOKEN`, `AD_NETWORK_DB_PATH`, and `AD_NETWORKS_STATE_DIRECTORY`. Never print it. Homelab owns env values; Mac copies must not be edited.
 
+The shared checkout updates at minute 25 each hour through the [installed updater](../../scripts/stack127/checkout/README.md). Cron and manual updates start only at UTC minutes 15–28, so the ten-minute timeout plus 30-second kill period releases job locks before :40. It skips maintenance holds and active jobs, uses fast-forward only, and installs dependencies only when needed. Never pull or install packages underneath a running job.
+
 The wrapper is [`run-job.sh`](../../scripts/ad-networks/ovh/run-job.sh). The full schedule is [`crontab.txt`](../../scripts/ad-networks/ovh/crontab.txt). Homelab's `stack127-ad-network-niche.timer` runs daily at 06:00 UTC (11:30 IST), with up to 3,000 sites per run.
 
 ## Counts in titles and headings
@@ -67,13 +69,13 @@ Ordinary vacuum makes dead space reusable; it does not generally shrink files on
 The coordinator performs these steps after review and release authorization. Keep cron removed and pause homelab's niche timer while doing the first sync.
 
 1. Confirm the latest database backup and SQLite backup are usable. Wait for any old ad-network processes to stop. Keep the existing OVH working store.
-2. Update OVH to the reviewed release:
+2. Let the [hourly checkout updater](../../scripts/stack127/checkout/README.md) reach the reviewed release (install it with that guide first if needed). Run it by hand only at UTC minutes 15–28, before opening a maintenance window. The first install run has the same time limit:
 
    ```bash
-   ssh ovh 'cd /srv/stack127 && git pull --ff-only && npm ci --omit=dev --ignore-scripts && git rev-parse HEAD'
+   ssh ovh '/bin/bash /srv/stack127-data/checkout/run-job.sh stack127-checkout-update && git -C /srv/stack127 rev-parse HEAD'
    ```
 
-   Check that SHA against the released revision. Check the env target privately: `AD_NETWORKS_RUNTIME_ENVIRONMENT=production`, loopback port 5432, and login `stack127_app`. Source it with `set -a; . /srv/stack127-data/ad-networks.env; set +a`. Do not use a parser that leaves surrounding quotes in values.
+   Exit 76 means outside UTC minutes 15–28, held or busy; wait for the next safe window and retry without bypassing locks. Check that SHA against the released revision. Set the checkout hold and wait for the updater using the linked guide before the manual steps below; remove your hold only after all release/maintenance work is complete. Check the env target privately: `AD_NETWORKS_RUNTIME_ENVIRONMENT=production`, loopback port 5432, and login `stack127_app`. Source it with `set -a; . /srv/stack127-data/ad-networks.env; set +a`. Do not use a parser that leaves surrounding quotes in values.
 3. If the separate evidence cleanup is approved, run its SQL as the database administrator. Each pass selects at most 5,000 keys per table before projecting JSON. Temporary cursors advance in one database session, including through unchanged batches. Stop when both scanned counts are zero, even if earlier trimmed counts were zero. For example, on OVH from `/srv/stack127`:
 
    ```bash
@@ -191,6 +193,11 @@ From the operator's local checkout, pause frontend dispatch and save the existin
 ```bash
 # Local shell. The implementation agent must not run these production mutations.
 export ADN_WINDOW=$(date -u +%Y%m%dT%H%M%SZ)
+# Hold code first; refuse an existing hold. Wait for any update already in flight.
+ssh ovh 'set -eu
+install -d -m 755 /srv/stack127-data/state
+(set -C; : > /srv/stack127-data/state/checkout-hold)
+flock -w 660 /srv/stack127-data/checkout/updater.lock true'
 ssh ovh python3 - "$ADN_WINDOW" <<'PYTHON'
 import pathlib
 import subprocess
@@ -214,7 +221,7 @@ ssh ovh 'sudo -n systemctl stop stack127-cache-worker.timer'
 ssh ovh 'while systemctl is-active --quiet stack127-cache-worker.service; do sleep 2; done'
 ```
 
-Coordinate other writers and releases for the same window. The authorized operator pauses homelab niche imports with `ssh homelab 'systemctl --user stop stack127-ad-network-niche.timer'` and waits until `ssh homelab 'systemctl --user is-active stack127-ad-network-niche.service'` reports inactive. Do not kill an active importer. These are release instructions, not permission for the implementation agent to operate on homelab. Wait for existing OVH collectors, syncs and retry wrappers to finish; inspect with:
+Keep this hold through the complete window. Update code before setting it, using the updater only at UTC minutes 15–28; never bypass the start window while collectors are paused. It survives restarts, but does not pause data jobs or dispatch. If waiting for the updater fails, leave the hold and investigate before proceeding. Coordinate other writers and releases for the same window. The authorized operator pauses homelab niche imports with `ssh homelab 'systemctl --user stop stack127-ad-network-niche.timer'` and waits until `ssh homelab 'systemctl --user is-active stack127-ad-network-niche.service'` reports inactive. Do not kill an active importer. These are release instructions, not permission for the implementation agent to operate on homelab. Wait for existing OVH collectors, syncs and retry wrappers to finish; inspect with:
 
 ```bash
 ssh ovh 'ps -eo pid,etime,args | grep -E "[s]cripts/ad-networks/(ovh/run-job.sh|refresh.mjs|verify.mjs|sync.mjs|niche-import.mjs)"'
@@ -389,6 +396,8 @@ PYTHON
 ssh ovh 'crontab -l'
 ```
 
+After the complete window and public acceptance pass, remove only the checkout hold created in step 1 with `ssh ovh 'rm /srv/stack127-data/state/checkout-hold'`. If no new window was opened, do not remove an existing hold belonging to another operator. The next minute-25 update catches up.
+
 After the next hourly `:40` run finishes, inspect `ssh ovh 'tail -n 100 /srv/stack127-data/logs/verification.log'`. Require a `start verification` timestamp after `$ADN_RESUMED_AT` and its following `end verification exit 0`. An old exit line or an active crontab alone does not pass. If that next run fails, keep the release incomplete and investigate before claiming acceptance.
 
 Check the durable `ad_networks/all` queue event drains after a successful refresh and the next hourly verification exits successfully. Do not remove the backups until acceptance. The whole false-loss spike **will not disappear**: blocked checks, changed signals and historical misses without proof remain unresolved. Missing old snapshots mean unknown history, not zero losses.