diff --git a/.claude/skills/babysit-pipeline/SKILL.md b/.claude/skills/babysit-pipeline/SKILL.md index f3021e206d7..3a4b4306b5e 100644 --- a/.claude/skills/babysit-pipeline/SKILL.md +++ b/.claude/skills/babysit-pipeline/SKILL.md @@ -144,17 +144,56 @@ apart, then polls until each metadata file lands, and exits with `run_in_background: true`; it skips libraries already on main, so re-running it after a partial result retries exactly the gaps. -**Two specs in parallel is fine — but only in this mode** (~4 specs/h -vs ~2); ten concurrent `impl-generate` runs showed no rate-limit -effects. It works because `run_spec.sh` decides per library from -`origin/main` metadata. `poll_spec.sh` and `monitor_spec.sh` must +**Several specs in parallel is fine — but only in this mode.** It +works because `run_spec.sh` decides per library from `origin/main` +metadata and reads liveness per spec (its own generate runs, its own +open `implementation//*` PRs), so drivers never mistake each +other's runs for progress. `poll_spec.sh` and `monitor_spec.sh` must still run one spec at a time: their stall logic reads *any* active `impl-*` run as belonging to the spec they are watching (§3), so a second spec in flight makes them call a stalled spec healthy. Never -mix the two modes on the same queue. Keep a ledger — -`done.log` / `deferred.log` next to the queue file in `agentic/runs/` -— and append the result line before dispatching the next spec, so a -compaction or a crashed session can resume without recounting. +mix the two modes on the same queue. The slot count is the user's +call: **2 is the default**, 4 with their OK. A 5-slot trial on +2026-09-01 was clean on the GitHub side (API quota untouched, impl-* +runner wait ≤4 min; only CodeQL piled up) — the binding limit is the +Claude usage window, which is why the user pulled it back the same +evening. Reducing means: stop launching and let in-flight PRs drain; +never cancel runs that already spent Claude time. + +**For an unattended queue use the scheduler**, +`.claude/skills/babysit-pipeline/run_queue.sh [slots]`, +started detached so it survives the session, with both streams +captured: + +```bash +setsid nohup .claude/skills/babysit-pipeline/run_queue.sh agentic/runs/ 2 \ + > agentic/runs//queue.out 2>&1 & # the 2 is the slot count +``` + +It keeps `[slots]` drivers in flight over +`/full_queue.txt`, skips libraries already on main and +pairs recorded as `CONFIRMED GAP` in `deferred.log`, harvests every +driver's `RESULT=` line into `done.log` / `deferred.log`, and holds +new launches for 15 min while any throttle sign is present: GitHub +core quota below 800, **three or more distinct generate pairs failing +within 25 min** (§4's cluster rule), or a rate-limit signature in a +failed impl-* log. `DEADLINE=` stops new launches at a fixed +time. A `rescue_specs.txt` (` ` lines) is re-checked +ahead of the queue and launched only once nothing for that spec is in +flight — the way to re-queue a spec that an outage cut short without +auto-closing PRs that are still under review or repair. Stop it with +`pkill -f run_queue.sh`; drivers keep watching (all their dispatches +went out at start), so "finish the current specs, start nothing new" +is exactly that. Keep the ledger next to the queue file in +`agentic/runs/` — a compaction or a crashed session resumes from it +without recounting. + +**Report every ~10 minutes** from one cycle script: GitHub quota, +runner queue depth and impl-* queue age, impl-* failures of the last +30 min with the first `##[error]` line of each new one, the scheduler +log tail, per-spec progress from `origin/main` metadata, open impl +PRs with labels. A run of ~1600 pipeline runs on 2026-09-02 needed no +other signal. **The one-retry rule.** A library that comes back missing gets exactly one fresh targeted dispatch before it is deferred. This is @@ -197,8 +236,38 @@ nothing failed. Nothing re-dispatches it: watchdog case 3 fires once, then only logs `already retried by watchdog — needs manual attention`. Audit it against metadata before trusting it — of 87 such labels on - 2026-08-24, **42 sat on implementations that had since landed**. - Take the label as a hint to check, never as the coverage answer. + 2026-08-24, **42 sat on implementations that had since landed** + (impl-merge clears the label on merge only since #11197). Take the + label as a hint to check, never as the coverage answer. +- **A provider outage looks like a capability cliff.** On 2026-09-02 + 03:15–03:45 UTC every Claude step ended `is_error:true` with + `Internal error: directory mismatch ... tsconfig.json`; 27 pairs + across three specs burned all three attempts in twenty minutes and + two whole specs came back 0/6. That is a throttle signal, not six + gaps: the scheduler's cluster rule held launches, and every pair + regenerated fine once the incident was over. Since #11199 such + failures no longer spend the pair's budget; before it, expect a + re-dispatch inside the 12-hour window to run without auto-retry. +- **Rescuing an outage-hit spec has an order.** First let the + pipeline finish what it can: PRs in `ai-review-failed` and stalled + repairs are the watchdog's job (`gh workflow run + watchdog-stuck-jobs.yml -f stale_hours=2` when you cannot wait for + the cron; since #11198 it also covers `ai-rejected` + + `ai-attempt-N` with a crashed repair — before that, dispatch + `impl-repair.yml` yourself with the watchdog's parameters + `pr_number`, `specification_id`, `library`, `attempt`). Only then + put the spec on `rescue_specs.txt`: the driver's dispatch + auto-closes every open PR of the pair, so a regeneration fired while + a repair is mid-flight throws that work away. +- **`Merge: PR #N` failing five times with "Head branch is out of + date" while `mergeable` stays `UNKNOWN` is a stuck PR object, not a + branch problem.** Seen on #10850 (2026-09-01): `update-branch` had + pushed a merge commit, the branch ref moved, the pull request never + registered it. Re-running impl-merge cannot help. With the user's OK + push an empty commit to the PR branch (`git commit-tree` on the + remote head, plain push) so GitHub re-syncs and recomputes + mergeability, then re-dispatch `impl-merge.yml -f pr_number=N`; + the merge itself stays with the workflow. - **"Agent reports success, writes no file"** is a live intermittent failure (8 of 85 generate runs on 2026-08-24, ~9%): the Claude step ends `"subtype":"success","is_error":false` and the next step fails diff --git a/.claude/skills/babysit-pipeline/run_queue.sh b/.claude/skills/babysit-pipeline/run_queue.sh new file mode 100755 index 00000000000..c54be0a177a --- /dev/null +++ b/.claude/skills/babysit-pipeline/run_queue.sh @@ -0,0 +1,248 @@ +#!/usr/bin/env bash +# Queue scheduler for the gap backfill (SKILL.md §5): keep up to $SLOTS +# run_spec.sh drivers in flight over a queue file, skipping libraries that are +# already on origin/main and pairs recorded as CONFIRMED GAP in deferred.log. +# Refills a slot as soon as a driver exits and harvests each driver's RESULT line +# into done.log / deferred.log. Before every launch it checks for throttle signs +# (GitHub API quota, a cluster of generate failures, rate-limit signatures in +# failed impl-* logs) and waits instead of launching while any is present. +# +# Usage: run_queue.sh [slots] +# holds full_queue.txt (` `, one spec +# per line) and receives the ledger: done.log, deferred.log, +# queue.log, results/, and an optional rescue_specs.txt +# [slots] drivers in flight at once (default 2; 4 only with the user's OK) +# Env: MODEL (default sonnet), DEADLINE (epoch seconds; no launches after it), +# STAGGER (s between launches, 90), THROTTLE_WAIT (s, 900), ANYPLOT_REPO. +# Run it detached so it survives the session, with both streams captured +# (the second positional argument is the slot count, not a redirect): +# setsid nohup run_queue.sh agentic/runs/ 2 > agentic/runs//queue.out 2>&1 & +# Stop it with `pkill -f run_queue.sh`; drivers keep running (after their +# dispatches they only watch). +# +# `RUN_QUEUE_LIB=1 source run_queue.sh ` loads the functions for +# testing without starting the scheduler loop — strict mode stays off then so +# the caller's shell keeps its own options. +if [ "${RUN_QUEUE_LIB:-}" != 1 ]; then + set -uo pipefail +fi + +usage() { + echo "usage: $(basename "${BASH_SOURCE[0]}") [slots]" >&2 + exit 2 +} +[ "$#" -ge 1 ] || usage +Q="$(cd "$1" && pwd)" || usage +[ -f "$Q/full_queue.txt" ] || { echo "error: $Q/full_queue.txt not found" >&2; exit 2; } +SLOTS="${2:-2}" +case "$SLOTS" in + ''|*[!0-9]*|0) echo "error: slots must be a positive integer, got '$SLOTS'" >&2; usage ;; +esac + +# run_spec.sh lives next to this script; the repo is resolved the same way the +# driver does it (ANYPLOT_REPO wins, then git from the script's location). +R="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="${ANYPLOT_REPO:-$(git -C "$R" rev-parse --show-toplevel 2>/dev/null || true)}" +if [ -z "$REPO" ] || [ ! -d "$REPO/plots" ]; then + echo "error: could not resolve the anyplot repo root (tried \$ANYPLOT_REPO, then git from $R)." >&2 + exit 2 +fi +MODEL="${MODEL:-sonnet}" +DEADLINE="${DEADLINE:-}" # epoch seconds; no new launches at or after this +STAGGER="${STAGGER:-90}" # seconds between two launches +THROTTLE_WAIT="${THROTTLE_WAIT:-900}" +OUT="$Q/results"; mkdir -p "$OUT" +LOG="$Q/queue.log" +SEEN="$Q/queue_seen_runs.txt"; touch "$SEEN" +HARVESTED="$Q/harvested.txt"; touch "$HARVESTED" +# Specs to (re)generate ahead of the queue — lines of ` `. Used for +# specs whose first pass was cut short (an outage, a lost dispatch). A rescue +# launches only once nothing for that spec is in flight any more, so PRs that +# are still being reviewed or repaired are never auto-closed by a re-dispatch. +RESCUE="$Q/rescue_specs.txt" + +say() { echo "[$(date -u +%H:%M:%S)] $*" | tee -a "$LOG"; } + +lang_of() { + case "$1" in + ggplot2) echo r ;; + makie) echo julia ;; + chartjs|d3|echarts|highcharts|muix) echo javascript ;; + *) echo python ;; + esac +} + +fetch_main() { + local i + for i in 1 2 3; do + git -C "$REPO" fetch origin main --quiet 2>/dev/null && return 0 + sleep $(( i * 3 )) + done + say "WARN: git fetch origin main failed 3x; using possibly stale origin/main" + return 1 +} + +# Libraries of a spec that are neither on main nor a confirmed gap. +missing_libs() { + local spec=$1; shift + local l m="" + for l in "$@"; do + git -C "$REPO" ls-tree --name-only origin/main -- \ + "plots/$spec/metadata/$(lang_of "$l")/$l.yaml" 2>/dev/null | grep -q . && continue + grep -qE "^$spec $l CONFIRMED GAP" "$Q/deferred.log" 2>/dev/null && continue + m="$m $l" + done + echo "${m# }" +} + +# Distinct specs with a driver process (the `bash -c "sleep N; ... run_spec.sh"` +# wrapper and the script itself both match, so count spec names, not PIDs). +# The scheduler's own command line never contains `run_spec.sh `, so no +# self-exclusion is needed — and a PID-substring filter would drop unrelated +# lines. +running() { + pgrep -af 'run_spec\.sh ' | grep -oE 'run_spec\.sh [a-z0-9-]+' | sort -u | wc -l +} +driver_for() { pgrep -f "run_spec\.sh $1 " >/dev/null; } + +# A spec whose generate runs or implementation PRs are still in flight must not +# be re-dispatched: a new impl-generate run auto-closes the open PR for the pair. +spec_in_flight() { + local spec=$1 st out + for st in in_progress queued waiting; do + out=$(gh run list --workflow=impl-generate.yml --status "$st" --limit 100 --json displayTitle \ + --jq ".[] | select(.displayTitle | test(\" for ${spec}\$\")) | .displayTitle" 2>/dev/null) || return 0 + [ -n "$out" ] && return 0 + done + out=$(gh pr list --state open --limit 100 --json headRefName \ + --jq ".[] | select(.headRefName | startswith(\"implementation/${spec}/\")) | .headRefName" 2>/dev/null) || return 0 + [ -n "$out" ] && return 0 + return 1 +} + +# Prints a reason to hold new launches, or nothing when the coast is clear. +throttle_reason() { + local rem fails since id wf sig + rem=$(gh api rate_limit --jq .resources.core.remaining 2>/dev/null) || { echo "gh api rate_limit failed"; return; } + if [ "${rem:-0}" -lt 800 ]; then echo "GitHub core quota low ($rem remaining)"; return; fi + fails=$(gh run list --workflow=impl-generate.yml --limit 100 --json conclusion,updatedAt,displayTitle \ + --jq '[.[] | select(.conclusion=="failure" and (.updatedAt > (now - 1500 | todate))) | .displayTitle] | unique | length' 2>/dev/null) \ + || { echo "gh run list failed"; return; } + if [ "${fails:-0}" -ge 3 ]; then echo "$fails distinct generate pairs failed in the last 25 min"; return; fi + since=$(date -u -d '30 min ago' +%Y-%m-%dT%H:%M:%SZ) + for wf in impl-generate impl-review impl-repair impl-merge; do + while read -r id; do + [ -z "$id" ] && continue + grep -qx "$id" "$SEEN" && continue + echo "$id" >> "$SEEN" + # POSIX ERE has no `\b`; the digit-boundary groups keep 429/529 from + # matching inside run ids or timestamps. + sig=$(gh run view "$id" --log-failed 2>/dev/null \ + | grep -iEo 'rate.?limit[^"]{0,60}|(^|[^0-9])(429|529)([^0-9]|$)|overloaded[^"]{0,40}|too many requests|usage limit[^"]{0,40}' \ + | head -1) + if [ -n "$sig" ]; then echo "rate-limit signature in $wf run $id: $sig"; return; fi + done < <(gh run list --workflow=$wf.yml --limit 100 --json databaseId,conclusion,updatedAt \ + --jq ".[] | select(.conclusion==\"failure\" and .updatedAt > \"$since\") | .databaseId" 2>/dev/null) + done +} + +# Copy each finished driver's RESULT line into the ledger, once. Rescue +# launches write `.rescue.out`, so a spec harvested as PARTIAL on its +# first pass is harvested again after the rescue (the key is the file name). +harvest() { + local f key spec line + for f in "$OUT"/*.out; do + [ -e "$f" ] || continue + key=$(basename "$f" .out); spec=${key%.rescue} + grep -qx "$key" "$HARVESTED" && continue + line=$(grep -m1 '^RESULT=' "$f") || continue + echo "$key" >> "$HARVESTED" + case "$line" in + RESULT=COMPLETE*) + # Two shapes: `... libs= after ~N min` after a real run, or + # `... (all libs already present)` when nothing was missing. + case "$line" in + *" libs="*) detail=$(sed -E 's/^RESULT=COMPLETE spec=[^ ]+ libs=//' <<<"$line") ;; + *) detail="all libs already present" ;; + esac + echo "$spec COMPLETE ($detail, queue-runner) $(date -u +%F_%H:%M)" >> "$Q/done.log" + say "DONE $spec" ;; + *) + echo "$spec $(echo "$line" | sed 's/^RESULT=//') (queue-runner; retry pending) $(date -u +%F_%H:%M)" >> "$Q/deferred.log" + say "DEFER $spec: ${line#RESULT=}" ;; + esac + done +} + +wait_for_slot() { + while [ "$(running)" -ge "$SLOTS" ]; do sleep 60; harvest; done +} +hold_while_throttled() { + local reason + while reason=$(throttle_reason); [ -n "$reason" ]; do + say "THROTTLE: $reason; holding new launches for $((THROTTLE_WAIT/60)) min" + sleep "$THROTTLE_WAIT"; harvest + done +} + +# One pass over the rescue list. Cheap checks first (git + pgrep); the API is +# only consulted for a spec that actually has missing libs and no driver. +rescue_pass() { + [ -s "$RESCUE" ] || return 0 + local spec libs miss + while read -r spec libs; do + [ -z "${spec:-}" ] && continue + miss=$(missing_libs "$spec" $libs); [ -z "$miss" ] && continue + driver_for "$spec" && continue + spec_in_flight "$spec" && continue + wait_for_slot + hold_while_throttled + fetch_main + miss=$(missing_libs "$spec" $libs); [ -z "$miss" ] && continue + spec_in_flight "$spec" && continue + say "rescue $spec [$miss] (running=$(running))" + nohup env ANYPLOT_REPO="$REPO" "$R/run_spec.sh" "$spec" "$MODEL" $miss > "$OUT/$spec.rescue.out" 2>&1 & + sleep "$STAGGER" + done < "$RESCUE" +} + +# `RUN_QUEUE_LIB=1 source run_queue.sh` loads the functions for testing without +# starting the scheduler loop. +if [ "${RUN_QUEUE_LIB:-}" = 1 ]; then return 0 2>/dev/null || exit 0; fi + +say "scheduler start: slots=$SLOTS model=$MODEL deadline=$([ -n "$DEADLINE" ] && date -u -d @"$DEADLINE" +%H:%M || echo none) UTC" +while read -r n spec libs; do + [ -z "${spec:-}" ] && continue + if [ -n "$DEADLINE" ] && [ "$(date +%s)" -ge "$DEADLINE" ]; then + say "deadline reached; no new launches"; break + fi + harvest + rescue_pass + fetch_main + miss=$(missing_libs "$spec" $libs) + [ -z "$miss" ] && continue + if driver_for "$spec"; then say "skip $spec: driver already running"; continue; fi + # Rescue candidates get first pick of a freed slot: re-check them every + # ~5 min while waiting. + i=0 + while [ "$(running)" -ge "$SLOTS" ]; do + sleep 60; harvest + i=$((i+1)); [ $((i % 5)) -eq 0 ] && rescue_pass + done + if [ -n "$DEADLINE" ] && [ "$(date +%s)" -ge "$DEADLINE" ]; then + say "deadline reached while waiting for a slot; no new launches"; break + fi + hold_while_throttled + fetch_main + miss=$(missing_libs "$spec" $libs) + [ -z "$miss" ] && continue + if spec_in_flight "$spec"; then say "skip $spec: generate run or PR still in flight (needs a look)"; continue; fi + say "launch $spec [$miss] (running=$(running))" + nohup env ANYPLOT_REPO="$REPO" "$R/run_spec.sh" "$spec" "$MODEL" $miss > "$OUT/$spec.out" 2>&1 & + sleep "$STAGGER" +done < "$Q/full_queue.txt" + +say "queue pass finished; waiting for in-flight drivers" +while [ "$(running)" -gt 0 ]; do sleep 60; harvest; done +harvest +say "scheduler exit" diff --git a/.claude/skills/babysit-pipeline/run_spec.sh b/.claude/skills/babysit-pipeline/run_spec.sh index 3591e310cce..7090a693db6 100755 --- a/.claude/skills/babysit-pipeline/run_spec.sh +++ b/.claude/skills/babysit-pipeline/run_spec.sh @@ -51,16 +51,32 @@ meta_present() { # assumes a fresh `git fetch` already ran this iteration [ -n "$(git -C "$REPO" ls-tree --name-only origin/main -- "plots/${SPEC}/metadata/${lang}/$1.yaml" 2>/dev/null)" ] } -pipeline_active() { - for wf in impl-generate.yml impl-review.yml impl-repair.yml impl-merge.yml impl-review-retry.yml; do - local out - if ! out=$(gh run list --workflow="$wf" --limit 12 --json status \ - --jq '.[] | select(.status=="in_progress" or .status=="queued") | .status' 2>&1); then - printf 'WARN: gh run list %s failed; assuming active: %s\n' "$wf" "$out" >> "$LOG" +# Per-spec liveness. A missing lib is still in flight when impl-generate has a +# queued/in-progress run for THIS spec, or an open implementation// +# PR exists that was touched within the last hour (review, repair and merge all +# happen on that PR). The earlier repo-wide "any impl-* run active" signal broke +# both ways once several drivers ran side by side: it was almost always true +# (a dead spec hid behind its neighbours' runs) and, with a short list limit, +# it missed a long generate run whose entry had scrolled off (false PARTIAL +# while PRs were mid-repair). `--status` filters server-side, so the list +# length no longer matters. +spec_active() { + local st out + for st in in_progress queued waiting; do + if ! out=$(gh run list --workflow=impl-generate.yml --status "$st" --limit 100 \ + --json displayTitle \ + --jq ".[] | select(.displayTitle | test(\" for ${SPEC}\$\")) | .displayTitle" 2>&1); then + printf 'WARN: gh run list --status %s failed; assuming active: %s\n' "$st" "$out" >> "$LOG" return 0 fi grep -q . <<<"$out" && return 0 done + if ! out=$(gh pr list --state open --limit 100 --json headRefName,updatedAt \ + --jq ".[] | select((.headRefName | startswith(\"implementation/${SPEC}/\")) and (.updatedAt > (now - 3600 | todate))) | .headRefName" 2>&1); then + printf 'WARN: gh pr list failed; assuming active: %s\n' "$out" >> "$LOG" + return 0 + fi + grep -q . <<<"$out" && return 0 return 1 } @@ -71,7 +87,7 @@ pipeline_active() { # `?` on error, never 0: a masked API failure reading as "all healthy" is how a # quota outage gets mistaken for a slow queue. recent_generate_failures() { - gh run list --workflow=impl-generate.yml --limit 60 \ + gh run list --workflow=impl-generate.yml --limit 100 \ --json conclusion,updatedAt \ --jq "[.[] | select(.conclusion==\"failure\" and (.updatedAt > (now - 1500 | todate)))] | length" 2>/dev/null \ || echo "?" @@ -118,7 +134,9 @@ for i in "${!TODO[@]}"; do [ "$i" -lt $(( ${#TODO[@]} - 1 )) ] && sleep "$STAGGER" done -MAXMIN=$(( 30 + 15 * ${#TODO[@]} )); [ "$MAXMIN" -gt 150 ] && MAXMIN=150 +# Runner queueing grows with the number of drivers in flight, so the budget is +# a little more generous than the 2-slot version (6 libs: 150 min, cap 180). +MAXMIN=$(( 30 + 20 * ${#TODO[@]} )); [ "$MAXMIN" -gt 180 ] && MAXMIN=180 iters=$(( MAXMIN * 60 / INTERVAL )) idle=0; last_done=-1 for ((i=1; i<=iters; i++)); do @@ -133,7 +151,7 @@ for ((i=1; i<=iters; i++)); do exit 0 fi if [ "$done_n" -gt "$last_done" ]; then idle=0; last_done=$done_n - elif pipeline_active; then idle=0 + elif spec_active; then idle=0 else idle=$((idle+1)); fi printf '[%s iter %d/%d] %d/%d done, missing: %s (idle=%d)\n' \ "$(TZ=UTC date -u +%H:%M:%S)" "$i" "$iters" "$done_n" "${#TODO[@]}" "${missing[*]}" "$idle" >> "$LOG" diff --git a/CHANGELOG.md b/CHANGELOG.md index e9a1f6da2cb..eba9c022e7f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -81,6 +81,19 @@ aggregate instead: an italic *Catalog* line at the end of the version section an ### Changed +- **The `babysit-pipeline` skill gains the backfill scheduler and the driver's per-spec + liveness check** — `run_queue.sh [slots]` keeps N `run_spec.sh` drivers + in flight over a queue file, skips libraries already on main and confirmed gaps, + harvests every driver's result into the ledger, holds launches on throttle signs + (GitHub quota, ≥3 distinct generate pairs failing within 25 min, rate-limit + signatures in failed logs), and re-queues outage-hit specs from a rescue list only + once nothing for them is in flight. `run_spec.sh` now reads liveness per spec (its + own generate runs and open `implementation//*` PRs) instead of "any impl-* + run active", which had produced false PARTIAL verdicts with several drivers in + flight. `SKILL.md` §5 documents the slot-count rule, the reporting cycle, and new + gotchas from the 2026-09-02 run: a provider outage reads like a capability cliff, the + order in which to rescue an outage-hit spec, and the stuck-PR-object symptom behind + "Head branch is out of date". (#11201) - **CodeQL moves to an advanced-setup workflow that skips `plots/**`** — the default setup scanned five languages on every push to an `implementation/*` branch and every impl-* pull request; during the 4-slot backfill of 2026-09-02 up to 23 CodeQL runs