docs(skill): backfill scheduler, per-spec liveness driver, outage gotchas

[MarkusNeusinger]

Summary

• run_queue.sh <queue-dir> [slots] joins the babysit-pipeline skill: keeps N run_spec.sh drivers in flight over a queue file, skips libraries already on main and CONFIRMED GAP pairs, harvests every driver's RESULT= line into done.log/deferred.log, holds launches for 15 min on throttle signs (GitHub core quota < 800, ≥3 distinct generate pairs failing within 25 min, rate-limit signatures in failed impl-* logs), honours a DEADLINE, and re-queues outage-hit specs from rescue_specs.txt only once nothing for them is in flight. It ran the 2026-09-01/02 backfill unattended: 55 specs, 329 implementations, three automatic throttle holds during a Claude outage.
• run_spec.sh gets the per-spec liveness check that was only in the gitignored working copy so far: a missing library counts as in flight when this spec has a queued/in-progress generate run or an open implementation/<spec>/* PR touched within the hour. The repo-wide "any impl-* run active" signal produced false PARTIAL verdicts with several drivers in flight and missed long generate runs behind a short list limit. Timeout budget: 30 + 20 min per library, cap 180.
• SKILL.md §5 documents the slot-count rule (2 by default, 4 with the user's OK; the binding limit is the Claude usage window, not GitHub), the scheduler, the rescue list and the 10-minute reporting cycle; three new gotchas cover the outage signature (is_error:true + "Internal error"), the rescue order (watchdog/repair first, regeneration last, because a driver dispatch auto-closes open PRs), and the stuck-PR-object symptom behind five "Head branch is out of date" merge failures.

Plan

N/A

Test plan

• bash -n on both scripts; run_queue.sh without arguments prints usage; RUN_QUEUE_LIB=1 source run_queue.sh <dir> loads the functions and missing_libs returns empty for a fully covered spec.
• run_spec.sh is byte-identical to the copy that drove the 2026-09-02 run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SrKzcwZBnref1sWYtdXynu

[Copilot]

🟡 Changes recommended

The new scheduler has confirmed logic/documentation issues (rate-limit signature matching and slot counting) that can prevent throttling or exceed the configured concurrency.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR extends the .claude/skills/babysit-pipeline operational tooling by adding an unattended queue scheduler (run_queue.sh), improving the per-spec liveness signal in run_spec.sh, and documenting the new workflow and outage-handling gotchas in the babysitting skill docs, with a corresponding CHANGELOG.md entry.

Changes:

• Add run_queue.sh to keep multiple run_spec.sh drivers in flight, harvest results, and apply throttle holds.
• Replace the repo-wide “any impl-* active” heuristic in run_spec.sh with a per-spec liveness check.
• Update .claude/skills/babysit-pipeline/SKILL.md and CHANGELOG.md to document the scheduler, slot rules, and outage rescue guidance.

File summaries

File	Description
CHANGELOG.md	Adds an Unreleased entry describing the new scheduler + per-spec liveness improvements and new gotchas documentation.
.claude/skills/babysit-pipeline/SKILL.md	Documents the new scheduler workflow, slot-count guidance, reporting cadence, and additional operational gotchas.
.claude/skills/babysit-pipeline/run_spec.sh	Updates liveness detection to be per-spec (generate runs + recent PR activity) and adjusts timeout budgeting for multi-driver queueing.
.claude/skills/babysit-pipeline/run_queue.sh	Introduces the new queue scheduler that launches drivers, harvests results, and throttles based on quota/failure signatures.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

The new scheduler’s running() slot accounting can undercount active drivers due to an unsafe PID substring filter, which can cause the scheduler to exceed the requested concurrency.

Review details

Suppressed comments (1)

.claude/skills/babysit-pipeline/run_queue.sh:91

• running() filters the pgrep output with grep -v "$$", which can accidentally exclude other driver processes whose PID contains the current PID as a substring (because pgrep -af prints the PID as the first token). That can undercount active drivers and cause the scheduler to overfill slots.

running() {
  pgrep -af 'run_spec\.sh ' | grep -v "$$" | grep -oE 'run_spec\.sh [a-z0-9-]+' | sort -u | wc -l
}

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

run_queue.sh has concurrency-control correctness issues (slot argument validation and driver counting) that can break scheduling behavior in unattended runs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

.claude/skills/babysit-pipeline/run_queue.sh:91

• running() tries to exclude the current process via grep -v "$$", but pgrep -af 'run_spec\.sh ' should not match run_queue.sh in the first place. Filtering by PID substring can also drop unrelated driver lines whose command line happens to contain the same digits, undercounting running specs and potentially overfilling slots.

running() {
  pgrep -af 'run_spec\.sh ' | grep -v "$$" | grep -oE 'run_spec\.sh [a-z0-9-]+' | sort -u | wc -l
}

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

There are a couple of concrete correctness/robustness issues in the new scheduler/docs (ledger parsing edge case and “sourcing” side effects) that should be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

run_queue.sh can silently fail to record driver failures when a driver exits without emitting a RESULT= line, which can cause specs to be dropped from the ledger unnoticed in unattended runs.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.claude/skills/babysit-pipeline/run_queue.sh:160

• harvest() skips output files that don't contain a RESULT= line, which can silently drop a spec from the ledger (and from any retry workflow) if run_spec.sh exits early (e.g., repo resolution/auth/usage error) before printing RESULT=. For unattended runs this makes failures easy to miss because the queue advances but nothing is recorded in done.log/deferred.log.

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite