docs(skill): backfill scheduler, per-spec liveness driver, outage gotchas

.claude/skills/babysit-pipeline/SKILL.md

@@ -144,17 +144,56 @@ apart, then polls until each metadata file lands, and exits with
 `run_in_background: true`; it skips libraries already on main, so
 re-running it after a partial result retries exactly the gaps.
 
-**Two specs in parallel is fine — but only in this mode** (~4 specs/h
-vs ~2); ten concurrent `impl-generate` runs showed no rate-limit
-effects. It works because `run_spec.sh` decides per library from
-`origin/main` metadata. `poll_spec.sh` and `monitor_spec.sh` must
+**Several specs in parallel is fine — but only in this mode.** It
+works because `run_spec.sh` decides per library from `origin/main`
+metadata and reads liveness per spec (its own generate runs, its own
+open `implementation/<spec>/*` PRs), so drivers never mistake each
+other's runs for progress. `poll_spec.sh` and `monitor_spec.sh` must
 still run one spec at a time: their stall logic reads *any* active
 `impl-*` run as belonging to the spec they are watching (§3), so a
 second spec in flight makes them call a stalled spec healthy. Never
-mix the two modes on the same queue. Keep a ledger —
-`done.log` / `deferred.log` next to the queue file in `agentic/runs/`
-— and append the result line before dispatching the next spec, so a
-compaction or a crashed session can resume without recounting.
+mix the two modes on the same queue. The slot count is the user's
+call: **2 is the default**, 4 with their OK. A 5-slot trial on
+2026-09-01 was clean on the GitHub side (API quota untouched, impl-*
+runner wait ≤4 min; only CodeQL piled up) — the binding limit is the
+Claude usage window, which is why the user pulled it back the same
+evening. Reducing means: stop launching and let in-flight PRs drain;
+never cancel runs that already spent Claude time.
+
+**For an unattended queue use the scheduler**,
+`.claude/skills/babysit-pipeline/run_queue.sh <queue-dir> [slots]`,
+started detached so it survives the session, with both streams
+captured:
+
+```bash
+setsid nohup .claude/skills/babysit-pipeline/run_queue.sh agentic/runs/<run> 2 \
+  > agentic/runs/<run>/queue.out 2>&1 &      # the 2 is the slot count
+```
+
+It keeps `[slots]` drivers in flight over
+`<queue-dir>/full_queue.txt`, skips libraries already on main and
+pairs recorded as `CONFIRMED GAP` in `deferred.log`, harvests every
+driver's `RESULT=` line into `done.log` / `deferred.log`, and holds
+new launches for 15 min while any throttle sign is present: GitHub
+core quota below 800, **three or more distinct generate pairs failing
+within 25 min** (§4's cluster rule), or a rate-limit signature in a
+failed impl-* log. `DEADLINE=<epoch>` stops new launches at a fixed
+time. A `rescue_specs.txt` (`<spec> <lib...>` lines) is re-checked
+ahead of the queue and launched only once nothing for that spec is in
+flight — the way to re-queue a spec that an outage cut short without
+auto-closing PRs that are still under review or repair. Stop it with
+`pkill -f run_queue.sh`; drivers keep watching (all their dispatches
+went out at start), so "finish the current specs, start nothing new"
+is exactly that. Keep the ledger next to the queue file in
+`agentic/runs/` — a compaction or a crashed session resumes from it
+without recounting.
+
+**Report every ~10 minutes** from one cycle script: GitHub quota,
+runner queue depth and impl-* queue age, impl-* failures of the last
+30 min with the first `##[error]` line of each new one, the scheduler
+log tail, per-spec progress from `origin/main` metadata, open impl
+PRs with labels. A run of ~1600 pipeline runs on 2026-09-02 needed no
+other signal.
 
 **The one-retry rule.** A library that comes back missing gets
 exactly one fresh targeted dispatch before it is deferred. This is
@@ -197,8 +236,38 @@ nothing failed.
   Nothing re-dispatches it: watchdog case 3 fires once, then only
   logs `already retried by watchdog — needs manual attention`. Audit
   it against metadata before trusting it — of 87 such labels on
-  2026-08-24, **42 sat on implementations that had since landed**.
-  Take the label as a hint to check, never as the coverage answer.
+  2026-08-24, **42 sat on implementations that had since landed**
+  (impl-merge clears the label on merge only since #11197). Take the
+  label as a hint to check, never as the coverage answer.
+- **A provider outage looks like a capability cliff.** On 2026-09-02
+  03:15–03:45 UTC every Claude step ended `is_error:true` with
+  `Internal error: directory mismatch ... tsconfig.json`; 27 pairs
+  across three specs burned all three attempts in twenty minutes and
+  two whole specs came back 0/6. That is a throttle signal, not six
+  gaps: the scheduler's cluster rule held launches, and every pair
+  regenerated fine once the incident was over. Since #11199 such
+  failures no longer spend the pair's budget; before it, expect a
+  re-dispatch inside the 12-hour window to run without auto-retry.
+- **Rescuing an outage-hit spec has an order.** First let the
+  pipeline finish what it can: PRs in `ai-review-failed` and stalled
+  repairs are the watchdog's job (`gh workflow run
+  watchdog-stuck-jobs.yml -f stale_hours=2` when you cannot wait for
+  the cron; since #11198 it also covers `ai-rejected` +
+  `ai-attempt-N` with a crashed repair — before that, dispatch
+  `impl-repair.yml` yourself with the watchdog's parameters
+  `pr_number`, `specification_id`, `library`, `attempt`). Only then
+  put the spec on `rescue_specs.txt`: the driver's dispatch
+  auto-closes every open PR of the pair, so a regeneration fired while
+  a repair is mid-flight throws that work away.
+- **`Merge: PR #N` failing five times with "Head branch is out of
+  date" while `mergeable` stays `UNKNOWN` is a stuck PR object, not a
+  branch problem.** Seen on #10850 (2026-09-01): `update-branch` had
+  pushed a merge commit, the branch ref moved, the pull request never
+  registered it. Re-running impl-merge cannot help. With the user's OK
+  push an empty commit to the PR branch (`git commit-tree` on the
+  remote head, plain push) so GitHub re-syncs and recomputes
+  mergeability, then re-dispatch `impl-merge.yml -f pr_number=N`;
+  the merge itself stays with the workflow.
 - **"Agent reports success, writes no file"** is a live intermittent
   failure (8 of 85 generate runs on 2026-08-24, ~9%): the Claude step
   ends `"subtype":"success","is_error":false` and the next step fails

.claude/skills/babysit-pipeline/run_queue.sh

@@ -0,0 +1,248 @@
+#!/usr/bin/env bash
+# Queue scheduler for the gap backfill (SKILL.md §5): keep up to $SLOTS
+# run_spec.sh drivers in flight over a queue file, skipping libraries that are
+# already on origin/main and pairs recorded as CONFIRMED GAP in deferred.log.
+# Refills a slot as soon as a driver exits and harvests each driver's RESULT line
+# into done.log / deferred.log. Before every launch it checks for throttle signs
+# (GitHub API quota, a cluster of generate failures, rate-limit signatures in
+# failed impl-* logs) and waits instead of launching while any is present.
+#
+# Usage: run_queue.sh <queue-dir> [slots]
+#   <queue-dir>  holds full_queue.txt (`<n-missing> <spec> <lib...>`, one spec
+#                per line) and receives the ledger: done.log, deferred.log,
+#                queue.log, results/, and an optional rescue_specs.txt
+#   [slots]      drivers in flight at once (default 2; 4 only with the user's OK)
+# Env: MODEL (default sonnet), DEADLINE (epoch seconds; no launches after it),
+#      STAGGER (s between launches, 90), THROTTLE_WAIT (s, 900), ANYPLOT_REPO.
+# Run it detached so it survives the session, with both streams captured
+# (the second positional argument is the slot count, not a redirect):
+#   setsid nohup run_queue.sh agentic/runs/<run> 2 > agentic/runs/<run>/queue.out 2>&1 &
+# Stop it with `pkill -f run_queue.sh`; drivers keep running (after their
+# dispatches they only watch).
+#
+# `RUN_QUEUE_LIB=1 source run_queue.sh <queue-dir>` loads the functions for
+# testing without starting the scheduler loop — strict mode stays off then so
+# the caller's shell keeps its own options.
+if [ "${RUN_QUEUE_LIB:-}" != 1 ]; then
+  set -uo pipefail
+fi
+
+usage() {
+  echo "usage: $(basename "${BASH_SOURCE[0]}") <queue-dir> [slots]" >&2
+  exit 2
+}
+[ "$#" -ge 1 ] || usage
+Q="$(cd "$1" && pwd)" || usage
+[ -f "$Q/full_queue.txt" ] || { echo "error: $Q/full_queue.txt not found" >&2; exit 2; }
+SLOTS="${2:-2}"
+case "$SLOTS" in
+  ''|*[!0-9]*|0) echo "error: slots must be a positive integer, got '$SLOTS'" >&2; usage ;;
+esac
+
+# run_spec.sh lives next to this script; the repo is resolved the same way the
+# driver does it (ANYPLOT_REPO wins, then git from the script's location).
+R="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO="${ANYPLOT_REPO:-$(git -C "$R" rev-parse --show-toplevel 2>/dev/null || true)}"
+if [ -z "$REPO" ] || [ ! -d "$REPO/plots" ]; then
+  echo "error: could not resolve the anyplot repo root (tried \$ANYPLOT_REPO, then git from $R)." >&2
+  exit 2
+fi
+MODEL="${MODEL:-sonnet}"
+DEADLINE="${DEADLINE:-}"        # epoch seconds; no new launches at or after this
+STAGGER="${STAGGER:-90}"        # seconds between two launches
+THROTTLE_WAIT="${THROTTLE_WAIT:-900}"
+OUT="$Q/results"; mkdir -p "$OUT"
+LOG="$Q/queue.log"
+SEEN="$Q/queue_seen_runs.txt"; touch "$SEEN"
+HARVESTED="$Q/harvested.txt"; touch "$HARVESTED"
+# Specs to (re)generate ahead of the queue — lines of `<spec> <lib...>`. Used for
+# specs whose first pass was cut short (an outage, a lost dispatch). A rescue
+# launches only once nothing for that spec is in flight any more, so PRs that
+# are still being reviewed or repaired are never auto-closed by a re-dispatch.
+RESCUE="$Q/rescue_specs.txt"
+
+say() { echo "[$(date -u +%H:%M:%S)] $*" | tee -a "$LOG"; }
+
+lang_of() {
+  case "$1" in
+    ggplot2) echo r ;;
+    makie) echo julia ;;
+    chartjs|d3|echarts|highcharts|muix) echo javascript ;;
+    *) echo python ;;
+  esac
+}
+
+fetch_main() {
+  local i
+  for i in 1 2 3; do
+    git -C "$REPO" fetch origin main --quiet 2>/dev/null && return 0
+    sleep $(( i * 3 ))
+  done
+  say "WARN: git fetch origin main failed 3x; using possibly stale origin/main"
+  return 1
+}
+
+# Libraries of a spec that are neither on main nor a confirmed gap.
+missing_libs() {
+  local spec=$1; shift
+  local l m=""
+  for l in "$@"; do
+    git -C "$REPO" ls-tree --name-only origin/main -- \
+      "plots/$spec/metadata/$(lang_of "$l")/$l.yaml" 2>/dev/null | grep -q . && continue
+    grep -qE "^$spec $l CONFIRMED GAP" "$Q/deferred.log" 2>/dev/null && continue
+    m="$m $l"
+  done
+  echo "${m# }"
+}
+
+# Distinct specs with a driver process (the `bash -c "sleep N; ... run_spec.sh"`
+# wrapper and the script itself both match, so count spec names, not PIDs).
+# The scheduler's own command line never contains `run_spec.sh `, so no
+# self-exclusion is needed — and a PID-substring filter would drop unrelated
+# lines.
+running() {
+  pgrep -af 'run_spec\.sh ' | grep -oE 'run_spec\.sh [a-z0-9-]+' | sort -u | wc -l
+}
+driver_for() { pgrep -f "run_spec\.sh $1 " >/dev/null; }
+
+# A spec whose generate runs or implementation PRs are still in flight must not
+# be re-dispatched: a new impl-generate run auto-closes the open PR for the pair.
+spec_in_flight() {
+  local spec=$1 st out
+  for st in in_progress queued waiting; do
+    out=$(gh run list --workflow=impl-generate.yml --status "$st" --limit 100 --json displayTitle \
+      --jq ".[] | select(.displayTitle | test(\" for ${spec}\$\")) | .displayTitle" 2>/dev/null) || return 0
+    [ -n "$out" ] && return 0
+  done
+  out=$(gh pr list --state open --limit 100 --json headRefName \
+    --jq ".[] | select(.headRefName | startswith(\"implementation/${spec}/\")) | .headRefName" 2>/dev/null) || return 0
+  [ -n "$out" ] && return 0
+  return 1
+}
+
+# Prints a reason to hold new launches, or nothing when the coast is clear.
+throttle_reason() {
+  local rem fails since id wf sig
+  rem=$(gh api rate_limit --jq .resources.core.remaining 2>/dev/null) || { echo "gh api rate_limit failed"; return; }
+  if [ "${rem:-0}" -lt 800 ]; then echo "GitHub core quota low ($rem remaining)"; return; fi
+  fails=$(gh run list --workflow=impl-generate.yml --limit 100 --json conclusion,updatedAt,displayTitle \
+    --jq '[.[] | select(.conclusion=="failure" and (.updatedAt > (now - 1500 | todate))) | .displayTitle] | unique | length' 2>/dev/null) \
+    || { echo "gh run list failed"; return; }
+  if [ "${fails:-0}" -ge 3 ]; then echo "$fails distinct generate pairs failed in the last 25 min"; return; fi
+  since=$(date -u -d '30 min ago' +%Y-%m-%dT%H:%M:%SZ)
+  for wf in impl-generate impl-review impl-repair impl-merge; do
+    while read -r id; do
+      [ -z "$id" ] && continue
+      grep -qx "$id" "$SEEN" && continue
+      echo "$id" >> "$SEEN"
+      # POSIX ERE has no `\b`; the digit-boundary groups keep 429/529 from
+      # matching inside run ids or timestamps.
+      sig=$(gh run view "$id" --log-failed 2>/dev/null \
+        | grep -iEo 'rate.?limit[^"]{0,60}|(^|[^0-9])(429|529)([^0-9]|$)|overloaded[^"]{0,40}|too many requests|usage limit[^"]{0,40}' \
+        | head -1)
+      if [ -n "$sig" ]; then echo "rate-limit signature in $wf run $id: $sig"; return; fi
+    done < <(gh run list --workflow=$wf.yml --limit 100 --json databaseId,conclusion,updatedAt \
+      --jq ".[] | select(.conclusion==\"failure\" and .updatedAt > \"$since\") | .databaseId" 2>/dev/null)
+  done
+}
+
+# Copy each finished driver's RESULT line into the ledger, once. Rescue
+# launches write `<spec>.rescue.out`, so a spec harvested as PARTIAL on its
+# first pass is harvested again after the rescue (the key is the file name).
+harvest() {
+  local f key spec line
+  for f in "$OUT"/*.out; do
+    [ -e "$f" ] || continue
+    key=$(basename "$f" .out); spec=${key%.rescue}
+    grep -qx "$key" "$HARVESTED" && continue
+    line=$(grep -m1 '^RESULT=' "$f") || continue
+    echo "$key" >> "$HARVESTED"
+    case "$line" in
+      RESULT=COMPLETE*)
+        # Two shapes: `... libs=<list> after ~N min` after a real run, or
+        # `... (all libs already present)` when nothing was missing.
+        case "$line" in
+          *" libs="*) detail=$(sed -E 's/^RESULT=COMPLETE spec=[^ ]+ libs=//' <<<"$line") ;;
+          *)          detail="all libs already present" ;;
+        esac
+        echo "$spec COMPLETE ($detail, queue-runner) $(date -u +%F_%H:%M)" >> "$Q/done.log"
+        say "DONE  $spec" ;;
+      *)
+        echo "$spec $(echo "$line" | sed 's/^RESULT=//') (queue-runner; retry pending) $(date -u +%F_%H:%M)" >> "$Q/deferred.log"
+        say "DEFER $spec: ${line#RESULT=}" ;;
+    esac
+  done
+}
+
+wait_for_slot() {
+  while [ "$(running)" -ge "$SLOTS" ]; do sleep 60; harvest; done
+}
+hold_while_throttled() {
+  local reason
+  while reason=$(throttle_reason); [ -n "$reason" ]; do
+    say "THROTTLE: $reason; holding new launches for $((THROTTLE_WAIT/60)) min"
+    sleep "$THROTTLE_WAIT"; harvest
+  done
+}
+
+# One pass over the rescue list. Cheap checks first (git + pgrep); the API is
+# only consulted for a spec that actually has missing libs and no driver.
+rescue_pass() {
+  [ -s "$RESCUE" ] || return 0
+  local spec libs miss
+  while read -r spec libs; do
+    [ -z "${spec:-}" ] && continue
+    miss=$(missing_libs "$spec" $libs); [ -z "$miss" ] && continue
+    driver_for "$spec" && continue
+    spec_in_flight "$spec" && continue
+    wait_for_slot
+    hold_while_throttled
+    fetch_main
+    miss=$(missing_libs "$spec" $libs); [ -z "$miss" ] && continue
+    spec_in_flight "$spec" && continue
+    say "rescue $spec [$miss] (running=$(running))"
+    nohup env ANYPLOT_REPO="$REPO" "$R/run_spec.sh" "$spec" "$MODEL" $miss > "$OUT/$spec.rescue.out" 2>&1 &
+    sleep "$STAGGER"
+  done < "$RESCUE"
+}
+
+# `RUN_QUEUE_LIB=1 source run_queue.sh` loads the functions for testing without
+# starting the scheduler loop.
+if [ "${RUN_QUEUE_LIB:-}" = 1 ]; then return 0 2>/dev/null || exit 0; fi
+
+say "scheduler start: slots=$SLOTS model=$MODEL deadline=$([ -n "$DEADLINE" ] && date -u -d @"$DEADLINE" +%H:%M || echo none) UTC"
+while read -r n spec libs; do
+  [ -z "${spec:-}" ] && continue
+  if [ -n "$DEADLINE" ] && [ "$(date +%s)" -ge "$DEADLINE" ]; then
+    say "deadline reached; no new launches"; break
+  fi
+  harvest
+  rescue_pass
+  fetch_main
+  miss=$(missing_libs "$spec" $libs)
+  [ -z "$miss" ] && continue
+  if driver_for "$spec"; then say "skip  $spec: driver already running"; continue; fi
+  # Rescue candidates get first pick of a freed slot: re-check them every
+  # ~5 min while waiting.
+  i=0
+  while [ "$(running)" -ge "$SLOTS" ]; do
+    sleep 60; harvest
+    i=$((i+1)); [ $((i % 5)) -eq 0 ] && rescue_pass
+  done
+  if [ -n "$DEADLINE" ] && [ "$(date +%s)" -ge "$DEADLINE" ]; then
+    say "deadline reached while waiting for a slot; no new launches"; break
+  fi
+  hold_while_throttled
+  fetch_main
+  miss=$(missing_libs "$spec" $libs)
+  [ -z "$miss" ] && continue
+  if spec_in_flight "$spec"; then say "skip  $spec: generate run or PR still in flight (needs a look)"; continue; fi
+  say "launch $spec [$miss] (running=$(running))"
+  nohup env ANYPLOT_REPO="$REPO" "$R/run_spec.sh" "$spec" "$MODEL" $miss > "$OUT/$spec.out" 2>&1 &
+  sleep "$STAGGER"
+done < "$Q/full_queue.txt"
+
+say "queue pass finished; waiting for in-flight drivers"
+while [ "$(running)" -gt 0 ]; do sleep 60; harvest; done
+harvest
+say "scheduler exit"