fix(source-control): re-read the pages near expiry when the verdict listing over-counts

.changes/pr-pipeline/fix-verify-listing-early-deletion-9c4e2a17.md

@@ -0,0 +1,8 @@
+---
+bump: patch
+---
+
+### Fixed
+
+- The verdict reader's carried listing first re-reads the pages around entries near expiry when it holds more than GitHub's live count. That re-read has a page cap and an id-read cap, and drops an omitted entry only on a 404. The over-count then takes at most one whole listing a job.
+- The merge queue's cold-read headroom check (`cold_cost` in pr-require-checks.yml) now counts those reads, up to 58 more requests a cold read: 1,681 at 7,522 artifacts and 264 FAILs.

.changes/source-control/fix-verify-listing-early-deletion-5b1e9c3a.md

@@ -0,0 +1,28 @@
+---
+bump: patch
+---
+
+### Fixed
+
+- The verdict gate reader (`babysit_verify_evidence.py`) settles an over-count from early deletion
+  before it falls back to a whole listing. GitHub can delete artifacts before their `expires_at`.
+  When the incremental listing or a whole listing's top-up holds more than GitHub's live count, the
+  reader first re-reads only the pages around the carried entries that expire within
+  `EARLY_DELETION_LEAD`. Those pages add every id they hold. An id inside their span that they omit
+  is dropped only when its own read answers 404 or shows it expired, because pages read one at a
+  time can skip a live entry at a boundary when an artifact above it is deleted. The re-read takes
+  at most `NEAR_EXPIRY_PAGE_CAP` (24) pages and `NEAR_EXPIRY_ID_READ_CAP` (34) id reads per listing.
+  A count it leaves unsettled, or an id read with any other answer, goes to the single whole listing
+  added in #11926, which refuses `verify:unreadable` when its top-up misses the count. An id read
+  that answers a 5xx is read again after each of `LISTING_WAITS`, as a page is; one still failing
+  then fails the read as transient, so a top-up refuses without noting a failed listing for the job.
+- An over-count now takes that whole listing at most once a job, even when its top-up comes up
+  short. Once that listing starts paging, it is noted beside the saved listing
+  (`.overcount-fallback`) with its outcome. A later reader in the job whose pages over-count then
+  refuses without another whole listing, keeping the caller-limit prefix when that listing ran out
+  of a caller's time. A refusal before paging leaves the listing to the next reader.
+- On the carried listing behind the `verify:unreadable` refusal of #11903, the re-read finished
+  incrementally in 21 page reads and 15.9 s, measured before the per-id reads were added. With
+  #11926 alone, that over-count goes to a whole listing of about 133 pages, past verify-cap's 120 s
+  limit, so it refuses (derived from the measured page count, not re-run). Each
+  `babysit-listing-miss` line also reports `region_pages`.

.github/workflows/pr-require-checks.yml

@@ -6390,13 +6390,16 @@ jobs:
             # most 13 pages each, LISTING_CARRY_MAX_AGE), then one count read
             # and LISTING_READS (3) whole listings, each up to
             # 1 + len(LISTING_WAITS) (3) attempts that page from the first,
-            # and a top-up after each (LISTING_TOP_UP_PAGES, 7). A whole
+            # and a top-up after each (LISTING_TOP_UP_PAGES, 7), plus at most
+            # NEAR_EXPIRY_PAGE_CAP (24) pages near expiry read again when a
+            # count is over, and NEAR_EXPIRY_ID_READ_CAP (34) artifacts read
+            # by id that those pages omit. A whole
             # listing pages ceil(total_count / 100), plus one for uploads while
             # it pages, so a cold read costs at most
-            # 3 x FAILs + 3 x 13 + 1 + 3 x (3 x pages + 7), plus the pages of
-            # this step's own listing, which every entry the queue builds at
-            # once also reads, cold_cost below: 1,623 at 7,522 artifacts and
-            # 264 FAILs, 2,021 at 10,450 and 300.
+            # 3 x FAILs + 3 x 13 + 1 + 3 x (3 x pages + 7) + 24 + 34, plus the
+            # pages of this step's own listing, which every entry the queue
+            # builds at once also reads, cold_cost below: 1,681 at 7,522
+            # artifacts and 264 FAILs, 2,079 at 10,450 and 300.
             # test_babysit_verify_evidence runs these lines against those caps
             # at several sizes. It covers this path only: an
             # entry with a carried run cache but no carried listing pays the
@@ -6438,7 +6441,7 @@ jobs:
                 exit 1
               fi
               listing_pages=$(((total + 99) / 100 + 1))
-              cold_cost=$((listing_pages + 3 * fails + 3 * 13 + 1 + 3 * (3 * listing_pages + 7)))
+              cold_cost=$((listing_pages + 3 * fails + 3 * 13 + 1 + 3 * (3 * listing_pages + 7) + 24 + 34))
               rate=$(read_api "the GITHUB_TOKEN rate limit" rate_limit) || rate=
               if ! remaining=$(jq -er '.resources.core.remaining | numbers | floor' <<<"$rate" 2>/dev/null) ||
                 ! limit=$(jq -er '.resources.core.limit | numbers | floor' <<<"$rate" 2>/dev/null) ||

docs/conventions/pr-pipeline/README.md

@@ -2007,7 +2007,8 @@ at once goes cold together, so the step first lists every artifact page once and
 starts only when the token's core pool then has at least one cold read's worst case under the
 listing's caps (the step's own listing, 3 reads per listed FAIL, 3 incremental reads of at most 13
 pages each, one count read, and 3 whole listings of up to 3 attempts of `ceil(total_count / 100) + 1`
-pages, each with a top-up of at most 7: 1,623 at 7,522 artifacts and 264 FAILs) times
+pages, each with a top-up of at most 7, at most 24 pages near expiry read again and 34 artifacts read by id:
+1,681 at 7,522 artifacts and 264 FAILs) times
 `max_entries_to_build`, plus the
 read floor (`limits.api` `read-floor` of the planned limit, inlined in the job env), left.
 An entry with a carried run cache but no carried listing pays the listing's part
@@ -2648,16 +2649,25 @@ hand-started run of main's pr-merge.yml uploaded, which `.github/scripts/pr-merg
 fetches as the head-ref map is fetched): it reads pages newest id first down past that listing's
 highest id until an artifact created two hours before that listing was read, keeps that listing's
 entries below the lowest id those pages hold, less those expired by the first page's `Date`, and is
-kept only when it holds exactly that page's `total_count`. Any other count, a missing, older or
+kept only when it holds exactly that page's `total_count`. A count above it first reads again the pages
+around kept entries near expiry, up to a page cap, since GitHub can delete an artifact up to an hour
+before its `expires_at` (`EARLY_DELETION_LEAD`, `NEAR_EXPIRY_PAGE_CAP`). Those pages can skip an
+entry at a boundary, so an entry they omit is dropped only when its own read answers 404, up to an
+id-read cap (`NEAR_EXPIRY_ID_READ_CAP`). Any count still other than it, a missing, older or
 malformed base, or any failed read lists every page, so a carried listing that leaves out a FAIL
 lists every page too (the rule and its basis: `babysit_verify_evidence.py`, "Carried reads").
 A whole listing is held to the same count: deduplicated by id, added to the whole listings read
 before it and checked by a top-up from the first page, since one can skip an artifact at a page
 boundary while an earlier one expires (#10486); after three whole listings that never hold the
 live count the candidate refuses `verify:unreadable`, and after the first when its top-up holds more
-than the count or does not read, which another whole listing cannot cure. A listing that fails once
+than the count or does not read, or the pages read before it held more, which another whole listing
+cannot cure. A listing that fails once
 it began a whole listing is noted beside the saved listing, and every later reader in the job
-refuses on that note without paging.
+refuses on that note without paging. An over-count is the exception to that note's limits: once the
+one whole listing an over-count takes starts paging, a later reader in the job whose pages
+over-count refuses without another, even when that listing failed transiently or on a caller's
+limit (then with the caller-limit prefix). A refusal before paging leaves that listing to the next
+reader.
 The lane token holds no Actions permission, so pr-merge.yml grants its job `actions: read` and passes the job's `GITHUB_TOKEN` as
 `BABYSIT_VERIFY_GH_TOKEN`; the reader uses it as `GH_TOKEN` for the artifact listing, the run
 records, the download of a verdict.json (by the listed artifact's id, one request where `gh run