diff --git a/.abcd/development/intents/planned/itd-2609201925079472-an-autonomous-implementation-run-paces-itself-by-default-and.md b/.abcd/development/intents/planned/itd-2609201925079472-an-autonomous-implementation-run-paces-itself-by-default-and.md deleted file mode 100644 index 5cc4c5aa6..000000000 --- a/.abcd/development/intents/planned/itd-2609201925079472-an-autonomous-implementation-run-paces-itself-by-default-and.md +++ /dev/null @@ -1,87 +0,0 @@ ---- -id: itd-2609201925079472 -slug: an-autonomous-implementation-run-paces-itself-by-default-and -spec_id: spc-2609202134341288 -kind: standalone -suggested_kind: null -reclassification_history: [] -builds_on: [itd-2609201916151817] -related_intents: [itd-29] -severity: minor -impact: additive -origin: researcher-authored -production_mode: hand-written ---- - -# An autonomous run paces itself by default, and one flag sets the pace - -## Press Release - -> **A run started with `abcd build` (or `abcd drain`, or the `abcd implement` loop they hand over to) follows a pace it was never told: a working window, a pause after it, and a ceiling on lanes alive at once.** The three numbers come from one place, layered: a flag on this run wins over the repository's abcd configuration, which wins over the machine's, which wins over the bundled default; the run record names which layer applied. `--pace /` and `--sub-agents ` set them for one run. At the window's end the loop starts no new lane, lets a running lane finish its step and checkpoint to its branch, writes `next_eligible_at` into the state file and exits; an invocation before that time refuses and names it, so the pause survives a killed process and needs no sleeping model. Today every autonomous run carries its pace in its prompt as prose, and two runs on one machine cannot share it. -> -> "Two pilots, two prompts, two paces typed by hand, and no way to know the other one was keeping to its ceiling," said a technical facilitator who ran both. "Now the pace is a line in the config and a timestamp in the state file, and a run that starts early is refused." - -## Why This Matters - -Two pilots on one machine in the week of 2026-09-15 ran on two paces written into their prompts by hand, two hours of work then five off with two lanes here, two hours then four off with three lanes there, and neither could be sure the other honoured its ceiling. The pace exists because the model budget resets on a window; a run that ignores it runs into the reset mid-lane and loses the lane. - -## Mechanism - -We expect a pace read from layered configuration and enforced by a timestamp in the state file to be honoured by every run without being told, because the loop that reads the state is the only thing that starts a lane; shown wrong if a run starts a lane inside a pause or above the ceiling, which the run record would show, or if the budget window the pause exists for moves in a way minutes cannot express. - -## Scope Conditions - -- Holds for a run driven by the implement verb's loop; a session pacing itself by prompt is outside it. -- Holds per run: The ceiling counts this run's lanes and validators. -- The bundled default is a choice, not a measurement: The two pilots ran 120/300 with two lanes and 120/240 with three, and a repository overrides it in its configuration. - -## What's In Scope - -- The three numbers, their four layers, the flags, the window clock and `next_eligible_at` in the state file, the ceiling on this run's lanes and validators, the budget check before a run starts, and the checkpoint on a rate-limit response. - -## What's Out of Scope - -- A ceiling across runs on one machine: The register's (`itd-2609150819440345`), where a lane registry can live. -- Mid-run telemetry and an operator's hand verbs over the state file: Dropped with `itd-29` until wanted. - -## Decisions - -Settled on 2026-09-20 with one defensible answer each, on the product thinker's ruling that such a decision is recorded rather than asked (`iss-2609202055103741`): - -1. **Re-entrant pause.** The pause is `next_eligible_at` in the state file; an invocation before it refuses and names the time. No process sleeps. -2. **Units are minutes**, as the product thinker specified the flag; a quota-window signal, where a harness exposes one, is a later refinement and is recorded as such. -3. **Cross-run enforcement is the register's.** This intent bounds one run; the design review found a repository file cannot bound two runs across repositories. -4. **Layering is flag, then repository, then machine, then bundled**, the order the model-tier intent already uses. -5. **The bundled default is 120 minutes of work, 300 of pause, two lanes**, the product thinker's numbers for this repository's runs on 2026-09-20; a repository that measured otherwise writes its own. -6. **The budget check and the rate-limit checkpoint come here from `itd-29`**, superseded on 2026-09-20 by the implement verb: A run refuses to start when the estimated cost exceeds the remaining quota where the runner reports one, and a rate-limit response checkpoints the lane and ends the window early. -7. **A run works in parallel up to its ceiling** (ruling DR6, the product thinker, 2026-09-29, verbatim: "per-run agent limit: WORK IN PARALLEL — a run may build several pieces and run reviewers concurrently up to its limit (new build-loop work; then AC6 is testable)."). The validators of a round run side by side, the lanes of steps that do not need each other run side by side, and implementers and reviewers share the ceiling; criterion 6 is tested through the concurrent loop the spec's piece 6 designs. -8. **Steps run one after another unless their plan says otherwise, and a hand-back holds the siblings' landings** (rulings DR6b and DR6c, the product thinker, 2026-09-30). DR6b, verbatim: "(a) ONE AFTER ANOTHER BY DEFAULT: a step runs alongside earlier ones only if its plan says so; nothing already planned changes; reviews run side by side; the 2026-09-21 wording stands." DR6c, verbatim: "(c) FINISH, BUT HOLD THEM: when one piece is handed back, pieces in flight finish but nothing merges until the person re-plans; the person then decides whether the held pieces land as they are." A step's default `needs` is every earlier step, opted out of per step; after a hand-back the sibling lanes run to completion and each passing one is `held`, never armed, until the person releases or discards it with `implement step --release ` or `--discard ` (the spec's piece 6, criteria C6, C11 and C13). - -## Open Questions - -_None open._ - -## Acceptance Criteria - -- **Given** no flag and no configuration, **when** a run starts, **then** it runs on 120/300 with two lanes and the run record names the bundled layer. -- **Given** a repository configuration and a machine configuration that disagree, **when** a run starts without a flag, **then** the repository's values apply and the record says so. -- **Given** `--pace 90/240 --sub-agents 3`, **when** a run starts, **then** those values apply over every configured layer and the record names the flag. -- **Given** a window that has elapsed, **when** the loop is invoked, **then** it starts no lane, lets a running lane finish its current step and checkpoint to its branch, writes `next_eligible_at`, and exits 0 naming the time. -- **Given** `next_eligible_at` in the future, **when** the loop is invoked, **then** it refuses naming the time and changes no state. -- **Given** the ceiling reached, **when** the loop would start a lane or a validator, **then** it starts nothing, names the lanes alive, and exits; the next invocation fills the slot, and the record counts the minutes the slot was waited for. -- **Given** a runner that reports remaining quota and an estimate that exceeds it, **when** a run starts, **then** it refuses naming both numbers and writes no state; a runner that reports no quota is named and the check is skipped out loud. -- **Given** a rate-limit response from a runner mid-lane, **when** the loop reads it, **then** the lane is checkpointed to its branch, the window ends early with `next_eligible_at` set, and the record names the response. -- **Given** a malformed pace or ceiling, **when** a run starts, **then** it refuses naming the value and the accepted form, and writes no state. - -## Typed Links - -- **builds_on `itd-2609201916151817`** (the implement verb): The loop this pace bounds. -- **refines `itd-2609170822093401`** (the model tier): The same configuration layering. - -## Audit Notes - -_Empty. Populated by intent-auditor when intent moves to shipped/._ - -## Grounds - -- pursued: we expect a pace read from layered configuration and enforced by a timestamp in the state file to be honoured by every run without being told, because the loop that reads the state is the only thing that starts a lane; shown wrong if a run starts a lane inside a pause or above the ceiling, which the run record would show diff --git a/.abcd/development/intents/shipped/itd-2609201925079472-an-autonomous-implementation-run-paces-itself-by-default-and.md b/.abcd/development/intents/shipped/itd-2609201925079472-an-autonomous-implementation-run-paces-itself-by-default-and.md new file mode 100644 index 000000000..91c0455da --- /dev/null +++ b/.abcd/development/intents/shipped/itd-2609201925079472-an-autonomous-implementation-run-paces-itself-by-default-and.md @@ -0,0 +1,195 @@ +--- +id: itd-2609201925079472 +slug: an-autonomous-implementation-run-paces-itself-by-default-and +spec_id: spc-2609202134341288 +kind: standalone +suggested_kind: null +reclassification_history: [] +builds_on: [itd-2609201916151817] +related_intents: [itd-29] +severity: minor +impact: additive +origin: researcher-authored +production_mode: hand-written +--- + +# An autonomous run paces itself by default, and one flag sets the pace + +## Press Release + +> **A run started with `abcd build` (or `abcd drain`, or the `abcd implement` loop they hand over to) follows a pace it was never told: a working window, a pause after it, and a ceiling on lanes alive at once.** The three numbers come from one place, layered: a flag on this run wins over the repository's abcd configuration, which wins over the machine's, which wins over the bundled default; the run record names which layer applied. `--pace /` and `--sub-agents ` set them for one run. At the window's end the loop starts no new lane, lets a running lane finish its step and checkpoint to its branch, writes `next_eligible_at` into the state file and exits; an invocation before that time refuses and names it, so the pause survives a killed process and needs no sleeping model. Today every autonomous run carries its pace in its prompt as prose, and two runs on one machine cannot share it. +> +> "Two pilots, two prompts, two paces typed by hand, and no way to know the other one was keeping to its ceiling," said a technical facilitator who ran both. "Now the pace is a line in the config and a timestamp in the state file, and a run that starts early is refused." + +## Why This Matters + +Two pilots on one machine in the week of 2026-09-15 ran on two paces written into their prompts by hand, two hours of work then five off with two lanes here, two hours then four off with three lanes there, and neither could be sure the other honoured its ceiling. The pace exists because the model budget resets on a window; a run that ignores it runs into the reset mid-lane and loses the lane. + +## Mechanism + +We expect a pace read from layered configuration and enforced by a timestamp in the state file to be honoured by every run without being told, because the loop that reads the state is the only thing that starts a lane; shown wrong if a run starts a lane inside a pause or above the ceiling, which the run record would show, or if the budget window the pause exists for moves in a way minutes cannot express. + +## Scope Conditions + +- Holds for a run driven by the implement verb's loop; a session pacing itself by prompt is outside it. +- Holds per run: The ceiling counts this run's lanes and validators. +- The bundled default is a choice, not a measurement: The two pilots ran 120/300 with two lanes and 120/240 with three, and a repository overrides it in its configuration. + +## What's In Scope + +- The three numbers, their four layers, the flags, the window clock and `next_eligible_at` in the state file, the ceiling on this run's lanes and validators, the budget check before a run starts, and the checkpoint on a rate-limit response. + +## What's Out of Scope + +- A ceiling across runs on one machine: The register's (`itd-2609150819440345`), where a lane registry can live. +- Mid-run telemetry and an operator's hand verbs over the state file: Dropped with `itd-29` until wanted. + +## Decisions + +Settled on 2026-09-20 with one defensible answer each, on the product thinker's ruling that such a decision is recorded rather than asked (`iss-2609202055103741`): + +1. **Re-entrant pause.** The pause is `next_eligible_at` in the state file; an invocation before it refuses and names the time. No process sleeps. +2. **Units are minutes**, as the product thinker specified the flag; a quota-window signal, where a harness exposes one, is a later refinement and is recorded as such. +3. **Cross-run enforcement is the register's.** This intent bounds one run; the design review found a repository file cannot bound two runs across repositories. +4. **Layering is flag, then repository, then machine, then bundled**, the order the model-tier intent already uses. +5. **The bundled default is 120 minutes of work, 300 of pause, two lanes**, the product thinker's numbers for this repository's runs on 2026-09-20; a repository that measured otherwise writes its own. +6. **The budget check and the rate-limit checkpoint come here from `itd-29`**, superseded on 2026-09-20 by the implement verb: A run refuses to start when the estimated cost exceeds the remaining quota where the runner reports one, and a rate-limit response checkpoints the lane and ends the window early. +7. **A run works in parallel up to its ceiling** (ruling DR6, the product thinker, 2026-09-29, verbatim: "per-run agent limit: WORK IN PARALLEL — a run may build several pieces and run reviewers concurrently up to its limit (new build-loop work; then AC6 is testable)."). The validators of a round run side by side, the lanes of steps that do not need each other run side by side, and implementers and reviewers share the ceiling; criterion 6 is tested through the concurrent loop the spec's piece 6 designs. +8. **Steps run one after another unless their plan says otherwise, and a hand-back holds the siblings' landings** (rulings DR6b and DR6c, the product thinker, 2026-09-30). DR6b, verbatim: "(a) ONE AFTER ANOTHER BY DEFAULT: a step runs alongside earlier ones only if its plan says so; nothing already planned changes; reviews run side by side; the 2026-09-21 wording stands." DR6c, verbatim: "(c) FINISH, BUT HOLD THEM: when one piece is handed back, pieces in flight finish but nothing merges until the person re-plans; the person then decides whether the held pieces land as they are." A step's default `needs` is every earlier step, opted out of per step; after a hand-back the sibling lanes run to completion and each passing one is `held`, never armed, until the person releases or discards it with `implement step --release ` or `--discard ` (the spec's piece 6, criteria C6, C11 and C13). + +## Open Questions + +_None open._ + +## Acceptance Criteria + +- **Given** no flag and no configuration, **when** a run starts, **then** it runs on 120/300 with two lanes and the run record names the bundled layer. +- **Given** a repository configuration and a machine configuration that disagree, **when** a run starts without a flag, **then** the repository's values apply and the record says so. +- **Given** `--pace 90/240 --sub-agents 3`, **when** a run starts, **then** those values apply over every configured layer and the record names the flag. +- **Given** a window that has elapsed, **when** the loop is invoked, **then** it starts no lane, lets a running lane finish its current step and checkpoint to its branch, writes `next_eligible_at`, and exits 0 naming the time. +- **Given** `next_eligible_at` in the future, **when** the loop is invoked, **then** it refuses naming the time and changes no state. +- **Given** the ceiling reached, **when** the loop would start a lane or a validator, **then** it starts nothing, names the lanes alive, and exits; the next invocation fills the slot, and the record counts the minutes the slot was waited for. +- **Given** a runner that reports remaining quota and an estimate that exceeds it, **when** a run starts, **then** it refuses naming both numbers and writes no state; a runner that reports no quota is named and the check is skipped out loud. +- **Given** a rate-limit response from a runner mid-lane, **when** the loop reads it, **then** the lane is checkpointed to its branch, the window ends early with `next_eligible_at` set, and the record names the response. +- **Given** a malformed pace or ceiling, **when** a run starts, **then** it refuses naming the value and the accepted form, and writes no state. + +## Typed Links + +- **builds_on `itd-2609201916151817`** (the implement verb): The loop this pace bounds. +- **refines `itd-2609170822093401`** (the model tier): The same configuration layering. + +## Audit Notes + + +Fidelity review — receipt rcp-c411be255805 (verifier intent-auditor claude-fable-5-1). + +Provenance: intent-auditor@claude-fable-5-1 · rubric_hash sha256:effa65b3e9e88ff29433b443ec2be159522a8b0b71cf1434526514aa61edb13e · prompt_hash sha256:922986f5804db2851ba4918cd59959f541b9b4df9c2aa7a626ce91f58ab15c81 +Input attestations: diff:17490941a92ebc34fcddb14bf7a70c63b61b2b22..45b2057aa51375564f8a0a387b1d60cb84270904 on build/run-2610092052125685-lane-1@sha256:d925a34a08af3bd6325be86da1455a2e646dd87f6e4612769a7509b5a6d8f9ed; + +Acceptance rollup: MET 7 · MET_WITH_CONCERNS 2 · NOT_MET 0 · INCONCLUSIVE 0 + +Per-criterion verdicts: +- ac-1 — MET: the bundled constants are 120/300/2, a start with no flag and no configuration resolves to them with layer 'bundled', and the run record's pace line names the bundled layer; the test passed on the lane tip + evidence: internal/core/implement/loop/pace.go:28 — "BundledWorkMinutes = 120" + evidence: internal/core/implement/loop/pace.go:30 — "BundledSubAgents = 2" + evidence: internal/core/implement/loop/pace_test.go:69 — "func TestARunWithNoConfigurationRunsOnTheBundledPace" + evidence: internal/core/implement/loop/pace_test.go:82 — "the record names the pace and the bundled layer" +- ac-2 — MET: resolvePace reads flag, then repository, then machine, then bundled through layered.Load; with the two files disagreeing and no flag the run takes the repository's 100/200/4, the origin is .abcd/config.json and the record names it and not the machine's file + evidence: internal/core/implement/loop/pace.go:200 — "func resolvePace(roots layered.Roots, f paceFlags) (Pace, error)" + evidence: internal/core/implement/loop/pace_test.go:96 — "func TestTheRepositoryPaceWinsOverTheMachines" + evidence: internal/core/implement/loop/pace_test.go:106 — "wantPace(t, st.Pace, 100, 200, 4, "repo")" +- ac-3 — MET: the flags are set on the layered store above every file layer; with both files set, --pace 90/240 --sub-agents 3 resolves to 90/240/3 at layer 'flag' and the record carries the flags as typed + evidence: internal/core/implement/loop/pace.go:206 — "s.SetFlag(paceNamespace+"."+keyWorkMinutes, *f.work, f.paceOrigin)" + evidence: internal/core/implement/loop/pace_test.go:134 — "func TestThePaceFlagsWinOverEveryLayer" + evidence: internal/core/implement/loop/pace_test.go:144 — "st.Pace.WorkMinutes.Origin != "--pace 90/240" || st.Pace.SubAgents.Origin != "--sub-agents 3"" +- ac-4 — MET: advance closes an elapsed window before any move: closeWindow writes next_eligible_at and a 'pause' record line, the call returns nil error with the time in Next, the outstanding await's receipt is still taken inside the pause so the lane finishes its implement step on its branch, and the multi-lane test shows nothing starts on any lane + evidence: internal/core/implement/loop/loop.go:622 — "if until, ok := windowElapsed(*st, now); ok {" + evidence: internal/core/implement/loop/loop.go:666 — "st.NextEligibleAt = &until" + evidence: internal/core/implement/loop/pace_test.go:245 — "func TestAnElapsedWindowStartsNothingAndWritesNextEligibleAt" + evidence: internal/core/implement/loop/pace_test.go:289 — "the running lane checkpoints inside the pause" + evidence: internal/core/implement/loop/parallel_test.go:738 — "func TestAnElapsedWindowStartsNothingOnAnyLane" +- ac-5 — MET: the first thing advance does under the lock is refuse with a 'pause' contention naming next_eligible_at in RFC3339 when now is before it, returning false so nothing is written; the test compares the state bytes before and after and finds them equal + evidence: internal/core/implement/loop/loop.go:606 — "if st.NextEligibleAt != nil && now.Before(*st.NextEligibleAt) {" + evidence: internal/core/implement/loop/loop.go:607 — "the run is paused until "+st.NextEligibleAt.UTC().Format(time.RFC3339)" + evidence: internal/core/implement/loop/pace_test.go:297 — "a step inside the pause is refused naming the time" + evidence: internal/core/implement/loop/pace_test.go:300 — "a refused step inside the pause changes no state and performs nothing" +- ac-6 — MET: slotsInUse counts every outstanding await (implementers and validators alike) against the pace's sub_agents; at the ceiling moveAgent hands out nothing, sets CeilingReached, names every lane alive with its awaits, and holds the work in waiting with the time first held; the next receipt frees the slot and tookSlot writes a 'slot' record line counting the minutes waited (14 in the test) + evidence: internal/core/implement/loop/schedule.go:526 — "if st.slotsInUse() >= st.ceiling() {" + evidence: internal/core/implement/loop/schedule.go:667 — "took a freed slot after waiting %d minute(s) at the run's ceiling of %d" + evidence: internal/core/implement/loop/parallel_test.go:300 — "func TestTheCeilingCountsValidatorsAndHoldsTheWaitingWork" + evidence: internal/core/implement/loop/parallel_test.go:353 — "the intent-auditor of lane-1 took a freed slot after waiting 14 minute(s)" +- ac-7 — MET_WITH_CONCERNS: checkBudget runs in start after the checks and before any directory is made; a route whose quota is below its estimate refuses at stage 'budget' naming the route and both numbers, and the run tier is proven absent; a route reporting none gets a 'reports no quota, so its check is skipped' row in the result, the text and the record. Concern: no shipped runner implements QuotaReporter (quota.go says so and TestOnlyARunnerThatCanReportsAQuota proves host, claude and opencode all report none), so the refusal path is reachable only through the Options.Quota test seam; in production every route is skipped out loud today + evidence: internal/core/implement/loop/loop.go:328 — "budget, err := checkBudget(len(chk.steps), chk.Intent != "", o)" + evidence: internal/core/implement/loop/budget.go:131 — "reports %d tokens of quota left, and the run's estimate is %s" + evidence: internal/core/implement/loop/budget.go:124 — "reports no quota, so its check is skipped (an estimate of %s)" + evidence: internal/core/implement/loop/budget_test.go:37 — "func TestAnEstimateOverTheQuotaIsRefusedAndWritesNothing" + evidence: internal/core/implement/loop/budget_test.go:103 — "func TestARouteThatReportsNoQuotaIsSkippedOutLoud" + evidence: internal/core/runner/quota.go:8 — "No shipped runner does: the adapters read only" + evidence: internal/surface/cli/build_budget_surface_test.go:65 — "budget: the host reports no quota, so its check is skipped" +- ac-8 — MET_WITH_CONCERNS: both adapters map a rate-limit response to ReasonRateLimited, the loop's dispatcher pauses on it instead of falling back, and rateLimited writes under the lock a 'rate-limit' line naming lane, role, runner and abcd's detail, a 'checkpoint' line per lane in flight naming its branch and head commit, and next_eligible_at once (now plus the run's pause), exiting 0 naming the time; the two-lane test on real worktrees passes. Concerns: (1) 'checkpointed to its branch' is realised as a record line naming the branch and the commit it already holds (checkpointNote); the loop commits nothing, so uncommitted agent work is not checkpointed, which the lane report records as a settled choice; (2) only a routed runner's response is read, so with the default all-host routing the path can never fire; (3) the claude rate_limit_event/rejected and opencode statusCode 429 shapes are stated assumptions not checked against a live harness + evidence: internal/core/implement/loop/drive.go:152 — "if out.RateLimited != nil {" + evidence: internal/core/implement/loop/ratelimit.go:60 — "func rateLimited(repoRoot, runID, laneID string, aw Await, fl *runner.Failure, o Options) (StepResult, error)" + evidence: internal/core/implement/loop/ratelimit.go:89 — "if st.NextEligibleAt == nil || !now.Before(*st.NextEligibleAt) {" + evidence: internal/core/implement/loop/ratelimit.go:126 — "%s checkpointed to %s at %s, at its %s stage" + evidence: internal/core/runner/dispatch.go:57 — "PauseOnRateLimit bool" + evidence: internal/core/runner/claude.go:90 — "the live shape is owed to a person's check" + evidence: internal/core/implement/loop/ratelimit_test.go:43 — "func TestARateLimitResponseCheckpointsEveryLaneAndEndsTheWindow" + evidence: internal/core/runner/ratelimit_test.go:19 — "func TestARateLimitResponseIsItsOwnFailure" +- ac-9 — MET: parsePaceFlags refuses a flag that is not two digit runs round one slash, and resolvePace refuses a file value outside its range or a misspelt key, every refusal at stage 'pace' with paceForm as the remedy naming the accepted form; sixteen malformed cases (flags, repository and machine files) each name the value and leave the run tier absent + evidence: internal/core/implement/loop/pace.go:62 — "--pace takes < work-minutes>/< pause-minutes> in whole minutes (work 1 to 10080, pause 0 to 10080, e.g. 120/300)" + evidence: internal/core/implement/loop/pace.go:158 — "return f, paceRefusal(fmt.Sprintf("--pace %q is not a pace", layered.BoundKey(*pace)))" + evidence: internal/core/implement/loop/pace_test.go:155 — "func TestAMalformedPaceIsRefusedAndWritesNoState" + evidence: internal/core/implement/loop/pace_test.go:203 — "runTierAbsent(t, repo.Root())" + +Gap audit: +- honoured: + - the three numbers come from one layered place (flag, repository, machine, bundled) and the run record names which layer applied + evidence: internal/core/implement/loop/pace.go:200 — "func resolvePace(roots layered.Roots, f paceFlags) (Pace, error)" + evidence: internal/core/implement/loop/loop.go:389 — "Note: "pace " + pace.String() + "; the first window opens now"" + - the pause is re-entrant: next_eligible_at in the state file, an early invocation refused naming the time, no process sleeps + evidence: internal/core/implement/loop/loop.go:606 — "if st.NextEligibleAt != nil && now.Before(*st.NextEligibleAt) {" + evidence: internal/core/implement/loop/loop.go:666 — "st.NextEligibleAt = &until" + - the ceiling counts this run's implementers and validators together and a run works in parallel up to it + evidence: internal/core/implement/loop/schedule.go:68 — "func (s State) slotsInUse() int {" + evidence: internal/core/implement/loop/parallel_test.go:387 — "func TestAnImplementerAndReviewersShareTheSlots" + - the budget check runs before anything is written and a refused run leaves no state + evidence: internal/core/implement/loop/loop.go:326 — "The budget check (criterion 7) comes after the checks and before" + evidence: internal/core/implement/loop/budget_test.go:47 — "runTierAbsent(t, repo.Root())" + - a rate-limit response ends the window early for the whole run, next_eligible_at written once, with no fallback to the host and no fallback receipt + evidence: internal/core/implement/loop/ratelimit.go:89 — "if st.NextEligibleAt == nil || !now.Before(*st.NextEligibleAt) {" + evidence: internal/core/runner/ratelimit_test.go:60 — "a rate limit is no fallback: outcome %+v, receipts %v" +- diverged: + - a lane 'checkpointed to its branch' on a rate limit: delivered as a run-record line naming the branch and the commit it already holds; the loop makes no commit, so uncommitted in-flight work stays only in the lane's worktree + evidence: internal/core/implement/loop/ratelimit.go:15 — "The commits an agent made are on the branch already; the loop makes none of" + evidence: internal/core/implement/loop/ratelimit.go:126 — "%s checkpointed to %s at %s, at its %s stage" + - the rate-limit checkpoint applies to any agent mid-lane: delivered only for a role routed to a runner; a host-run agent (the default route) has no way to report a rate limit to the loop + evidence: internal/core/implement/loop/drive.go:152 — "if out.RateLimited != nil {" + evidence: internal/core/runner/dispatch.go:53 — "PauseOnRateLimit is set by a caller that pauses its run on a rate-limit" + - docs/reference/terminology.md now states the run 'checkpoints every lane in flight to its own branch' and resumes from the lanes' branches; the delivered checkpoint is a record line over commits already on the branch, so the sentence overstates what is persisted + evidence: docs/reference/terminology.md:53 — "it checkpoints every lane in flight to its own branch and writes the time the next window may start" + evidence: internal/core/implement/loop/ratelimit.go:118 — "func checkpointNote(repoRoot string, l Lane) string {" +- missing: + - a production quota source: no shipped runner implements QuotaReporter, so the estimate-versus-quota refusal (decision 6) never fires outside the test seam + evidence: internal/core/runner/quota.go:8 — "No shipped runner does: the adapters read only" + evidence: internal/core/runner/ratelimit_test.go:97 — "%s reports no quota: ok %v, err %v" + - a live check of the harness rate-limit event shapes the adapters parse (claude rate_limit_event status 'rejected', opencode error statusCode 429) + evidence: internal/core/runner/claude.go:90 — "the live shape is owed to a person's check, as the opencode" + evidence: internal/core/runner/opencode.go:114 — "live shape is owed to a person's check, as its other events' are" + +Scope-condition dispositions: +- cond-2609202134346930 — survived: every enforcement point (the pause refusal, the window close, the ceiling, the budget check and the rate-limit checkpoint) lives inside the implement verb's loop and its process driver; nothing reaches a session that paces itself by prompt + evidence: internal/core/implement/loop/loop.go:606 — "if st.NextEligibleAt != nil && now.Before(*st.NextEligibleAt) {" + evidence: internal/core/implement/loop/loop.go:328 — "budget, err := checkBudget(len(chk.steps), chk.Intent != "", o)" + evidence: internal/core/implement/loop/drive.go:145 — "PauseOnRateLimit: true," +- cond-2609202134341920 — survived: the ceiling is the run's own pace.sub_agents and slotsInUse counts only the awaits in this run's state file, implementers and validators alike; nothing counts another run's agents, and the rate-limit checkpoint frees the slot inside the same run's state + evidence: internal/core/implement/loop/schedule.go:60 — "func (s State) ceiling() int {" + evidence: internal/core/implement/loop/schedule.go:68 — "func (s State) slotsInUse() int {" + evidence: internal/core/implement/loop/parallel_test.go:300 — "func TestTheCeilingCountsValidatorsAndHoldsTheWaitingWork" +- cond-2609202134341625 — survived: the bundled 120/300/2 is three named constants recorded as the product thinker's choice, applied only when no layer sets a key, and a repository's .abcd/config.json pace block overrides each key on its own, as the repository-wins test shows + evidence: internal/core/implement/loop/pace.go:23 — "The bundled pace: 120 minutes of work, 300 of pause, two lanes (decision 5," + evidence: internal/core/implement/loop/pace_test.go:96 — "func TestTheRepositoryPaceWinsOverTheMachines" + + +## Grounds + +- pursued: we expect a pace read from layered configuration and enforced by a timestamp in the state file to be honoured by every run without being told, because the loop that reads the state is the only thing that starts a lane; shown wrong if a run starts a lane inside a pause or above the ceiling, which the run record would show diff --git a/.abcd/development/specs/open/spc-2609301921521360-the-budget-check-and-the-rate-limit-checkpoint.md b/.abcd/development/specs/closed/spc-2609301921521360-the-budget-check-and-the-rate-limit-checkpoint.md similarity index 100% rename from .abcd/development/specs/open/spc-2609301921521360-the-budget-check-and-the-rate-limit-checkpoint.md rename to .abcd/development/specs/closed/spc-2609301921521360-the-budget-check-and-the-rate-limit-checkpoint.md diff --git a/commands/build.md b/commands/build.md index bde4f050c..2a85a9b73 100644 --- a/commands/build.md +++ b/commands/build.md @@ -184,6 +184,24 @@ each other beside one another. A step runs beside earlier steps only when the spec's `- needs:` line under it says so (`- needs: none`, or `- needs: 1, 3` naming the steps it waits for); without the line it needs every step before it. +## The budget check + +Before a new run writes anything, it checks its budget. Each route the run's +roles take (the host, or a runner the machine routes a role to) is asked for the +quota it has left, and the run's estimate is compared with what each route +reports for its own roles: about 80000 tokens an agent, for the implementer and +the two reviewers on every step the run opens a lane for, and the +intent-auditor once (an issue run takes no audit). The estimate is the first +round, not the worst case: fix rounds are not counted. A route whose estimate +exceeds the quota it reports refuses the start at the `budget` stage with exit +2, naming the route, both numbers and the remedy, and nothing is written; tell +the user both numbers. A route that reports no quota is named and its check is +skipped out loud: the host never reports one (abcd cannot read a host session's +quota), and no shipped runner reports one yet. The payload's `budget` carries a +row per route (`route`, `roles`, `estimate_tokens`, `remaining_tokens`, null when +none was reported, `checked` and `detail`), and the run record's `budget` line +names the same. A resumed start checks no budget, and its `budget` is null. + ## Drive it The host session drives the loop, one move per call: @@ -250,6 +268,18 @@ started may still finish: hand its receipt back as usual. Before naming the time, and nothing changes; stop driving the run and invoke it again at or after that time, when a new window opens. The pause lives in the state file, so no process waits through it. + +A runner that answers a routed role with a rate-limit response ends the window +early for the whole run, since every lane spends the same budget; the role is +not handed to you as a fallback. The call frees that agent's slot, checkpoints +every lane with work in flight to its own branch (the run record's `checkpoint` +lines name each branch and the commit it holds), writes `next_eligible_at` once +(now plus the run's pause), records a `rate-limit` line naming the lane, the +role and the runner, and exits 0 with `rate_limit` (`lane`, `role`, `runner`, +`detail`) and `next_eligible_at` in the payload. An agent already out on +another lane may still hand its receipt back. Stop driving the run, tell the +user the lane and the time, and invoke it again at or after that time: the work +the response stopped is handed out afresh from its branch. `"${CLAUDE_PLUGIN_ROOT}/abcd" implement status --json` renders every run, its lanes and its record, and writes nothing. A lane's `worktree` is home-relative, or its directory name when it sits outside HOME; its `brief` and `receipt` keep diff --git a/commands/implement.md b/commands/implement.md index a622579fa..f29ebff29 100644 --- a/commands/implement.md +++ b/commands/implement.md @@ -285,7 +285,12 @@ pause, and exits 0 with `next_eligible_at` in the result and `next` naming the time; an agent already started may still hand its receipt back. Before the run's `next_eligible_at` a `step` is refused as a pause (exit 3) naming the time, and nothing changes; at or after it a new window opens and the stage -proceeds. +proceeds. A routed runner that answers with a rate-limit response ends the +window early instead of falling back to the host: the agent's slot is freed, +every lane with work in flight is checkpointed to its own branch, and the call +writes `next_eligible_at` once, records the lane, the role and the runner the +response came from, and exits 0 with `rate_limit` and `next_eligible_at` in the +result; at or after that time the stopped work is handed out afresh. Without `--run`, both act on the one run in progress in this checkout, and are refused naming the runs when there are several. A refusal exits 2 (3 on a pause diff --git a/docs/reference/cli/commands.md b/docs/reference/cli/commands.md index 0e0dd8a66..e4f36b9ab 100644 --- a/docs/reference/cli/commands.md +++ b/docs/reference/cli/commands.md @@ -278,6 +278,14 @@ ceiling is free. A lane whose validators still do not pass after its fix rounds handed back: it stops as unachievable with the last round's findings, the run starts nothing further for it, and `abcd implement step` refuses naming the hand-back. +A new run checks its budget before anything is written: each route the run's roles take +(the host, or a runner) is asked for the quota it has left, and the run's estimate, about +80000 tokens an agent for the implementer and the two reviewers on every step and the +intent-auditor once, is compared with what each route reports for its roles. A route whose +estimate exceeds its quota refuses the start naming both numbers, and writes nothing; a +route that reports none (the host, and every shipped runner) is named and its check is +skipped out loud, in the result and the run record. + The run then moves one step per `abcd implement step`, driven by the host session. The runner configuration is read before the run is created: roles..runner (host, @@ -2079,6 +2087,13 @@ nothing, writes next_eligible_at (now plus the run's pause) and exits 0 naming i agent already started may still hand back its receipt. Before next_eligible_at the call is refused as a pause and nothing changes; at or after it, a new window opens. +A runner that answers with a rate-limit response is not fallen back on, since every lane +spends the run's budget: its agent's slot is freed, every lane with work in flight is +checkpointed to its own branch (the run record names the branch and the commit it holds), +the window ends early with next_eligible_at written once (now plus the run's pause), the +record names the lane, the role and the runner the response came from, and the call exits +0 naming the time. At next_eligible_at the work is handed out afresh. + --run names the run; without it, the one run in progress in this checkout. Exit 2 on a refusal, exit 3 on a pause or a locked run state. diff --git a/docs/reference/terminology.md b/docs/reference/terminology.md index d383469bd..e9bc8fd9b 100644 --- a/docs/reference/terminology.md +++ b/docs/reference/terminology.md @@ -50,7 +50,7 @@ deep-linked. | **Authorization** | The process of verifying that a requested action or service is approved for a specific entity.[^authz] | **WATCHING** — deterministic gates decide what enters the record, but a permission model for agent actions is undesigned: the coordination draft rules an authorisation matrix out of its own scope and routes it to per-project permission templates (itd-33, itd-18 — both drafts; commit identity is itd-131's gate, shipped). | | **Autonomous agent** | A system situated in an environment, sensing and acting on it over time in pursuit of its own agenda (the classical taxonomy).[^auto] | **ADAPTS** — autonomy in abcd is bounded by construction: runs iterate under receipts and deterministic gates, and product decisions — adoption, sign-off, anything irreversible — stay human (*verifier selects, gates decide*: a verdict is a proposal; the human's adoption is the gate). | | **Context engineering** | Building dynamic systems that curate the optimal set of tokens — information and tools, in the right format — supplied to a model at inference time (popularised mid-2025 in vendor engineering documentation).[^ce] | **USES** — the modular rules loader (shipped; itd-3) is exactly this: prompt-triggered recall matching injects only the rule domains relevant to the turn, and a prompt matching nothing injects zero tokens. | -| **Durable execution** | Making code fault-tolerant by automatically persisting execution progress, so a failed process resumes from the last completed step instead of restarting.[^durable] | **WATCHING** — the record's run resilience is superseded (itd-29) and nothing resumable ships: the budget check before a run and the checkpoint on a rate-limit response are criteria of a planned pacing intent (itd-2609201925079472), which checkpoints a running lane to its branch and writes the time the next window may start into the run's state file — deliberately file-based and framework-free. | +| **Durable execution** | Making code fault-tolerant by automatically persisting execution progress, so a failed process resumes from the last completed step instead of restarting.[^durable] | **ADAPTS** — the record's run resilience (itd-29) was superseded by the implement loop's pacing (itd-2609201925079472): a run checks its budget before it starts wherever a route reports its quota, and on a runner's rate-limit response it checkpoints every lane in flight to its own branch and writes the time the next window may start into the run's state file, so the run resumes from its lanes' branches after the pause rather than replaying a step — deliberately file-based and framework-free. | | **Evaluations (evals)** | Systematic tests of model outputs against specified criteria; the peer-reviewed anchor is holistic, multi-metric evaluation.[^evals] | **USES** — a standing discipline: no judge ships unmeasured (itd-81) — calibration corpora with at least 40% known-good cases, scoring on recall *and* true-negative rate, and a false-positive ceiling that gates the prompt lock. The same discipline records a rejection of judge panels as redundant votes. | | **Grounding** | Supplying a model use-case-specific information at inference time so responses connect to accurate, verifiable sources.[^ground] | **ADAPTS** — abcd's version is *cite-or-be-dropped* (adr-35): a claim in a synthesised artefact carries its source citation or does not ship, and whatever could not be grounded is reported loudly as coverage blanks rather than papered over. | | **Guardrails** | Programmable controls on an application's inputs and outputs — validation, filtering, dialogue constraints — enforced outside the model.[^guard] | **ADAPTS** — abcd's equivalents are deterministic gates: lint families, receipt gates, and pre-commit guards all fail closed. The pre-tool-use shell-hazard guard (`abcd guard`) deliberately does not — it is *fail-open-loud*, a mistake filter rather than a security boundary (adr-42): a guard that cannot answer warns unmissably and lets the command run, because a broken guard must never brick a session. Two principles sharpen the industry sense: *enforcement claims are facts* (a claimed gate must exist) and *guards prove themselves* (a guard ships with the test that shows it fires). | diff --git a/internal/core/implement/loop/budget.go b/internal/core/implement/loop/budget.go new file mode 100644 index 000000000..0e06180fd --- /dev/null +++ b/internal/core/implement/loop/budget.go @@ -0,0 +1,148 @@ +package loop + +// budget.go is the budget check a run takes before it starts +// (itd-2609201925079472 criterion 7, decision 6, from itd-29; +// spc-2609301921521360). Each route the run's roles go through is asked for +// the quota it has left (runner.Config.QuotaFor): where one reports it, the +// share of the run's estimate its roles will spend is compared with it, and a +// run that exceeds it is refused naming both numbers before anything is +// written; a route that reports none (the host, whose quota abcd cannot read, +// and every shipped runner today) is named and the check is skipped out loud, +// in the result and in the run record's first lines. +// +// The estimate is itd-29's: about 80000 tokens an agent, an agent for each +// role a lane's first round takes (its implementer, the ruthless and the +// security reviewer), on every step the run opens a lane for, and the +// intent-auditor once on the lane that closes the spec (an issue run takes no +// audit). It is the run's first round, not its worst case: fix rounds are not +// counted, and a run that spends past its quota on them is what the rate-limit +// checkpoint (ratelimit.go) ends the window for. + +import ( + "context" + "fmt" + "strings" + + "github.com/intentdriven/abcd/internal/core/runner" +) + +// EstimateTokensPerAgent is the tokens the estimate counts for each agent a +// run starts (itd-29: "tasks × ~80k tokens"). +const EstimateTokensPerAgent = 80000 + +// StageBudget is the refusal stage, and the run record's, of the budget check. +const StageBudget = "budget" + +// QuotaReader reads the quota a route reports: ok false when it reports none. +type QuotaReader func(ctx context.Context, route string) (runner.Quota, bool, error) + +// BudgetRow is one route the run's roles go through, with its share of the +// estimate and what it reported. +type BudgetRow struct { + // Route is the host or a runner. + Route string `json:"route"` + // Roles are the roles the run hands this route. + Roles []string `json:"roles"` + // EstimateTokens is the estimate of what those roles spend over the run. + EstimateTokens int64 `json:"estimate_tokens"` + // RemainingTokens is the quota the route reported; null when it reports + // none and its check was skipped. + RemainingTokens *int64 `json:"remaining_tokens"` + // Checked is true when the route reported a quota and it covered the + // estimate. + Checked bool `json:"checked"` + // Detail is the row in words, as the run record carries it. + Detail string `json:"detail"` +} + +// Budget is the budget check a new run took. +type Budget struct { + // Steps are the spec steps the run opens lanes for. + Steps int `json:"steps"` + // TokensPerAgent is the estimate's figure for one agent. + TokensPerAgent int64 `json:"tokens_per_agent"` + Rows []BudgetRow `json:"rows"` +} + +// String is the run record's line for the check. +func (b Budget) String() string { + parts := make([]string, 0, len(b.Rows)) + for _, r := range b.Rows { + parts = append(parts, r.Detail) + } + return fmt.Sprintf("the budget check, at about %d tokens an agent over %d step(s): %s", + b.TokensPerAgent, b.Steps, strings.Join(parts, "; ")) +} + +// runRoles are the agents a run starts for each lane's first round, and the +// auditor it starts once for an intent's run. +var runRoles = []string{RoleImplementer, RoleRuthless, RoleSecurity} + +// checkBudget estimates what the run will spend on each route its roles go +// through and compares it with the quota each reports, refusing at the budget +// stage a route the estimate exceeds or whose report fails. It writes nothing. +func checkBudget(steps int, audits bool, o Options) (*Budget, error) { + roles := append([]string(nil), runRoles...) + if audits { + roles = append(roles, RoleAuditor) + } + var order []string + byRoute := map[string]*BudgetRow{} + for _, role := range roles { + route := runner.Host + if o.Runners != nil { + route = o.Runners.RouteFor(role).Runner + } + row, ok := byRoute[route] + if !ok { + row = &BudgetRow{Route: route} + byRoute[route] = row + order = append(order, route) + } + row.Roles = append(row.Roles, role) + agents := int64(steps) + if role == RoleAuditor { + agents = 1 + } + row.EstimateTokens += agents * EstimateTokensPerAgent + } + read := o.Quota + if read == nil { + read = o.Runners.QuotaFor + } + b := &Budget{Steps: steps, TokensPerAgent: EstimateTokensPerAgent} + for _, route := range order { + row := *byRoute[route] + name := routeName(route) + q, ok, err := read(context.Background(), route) + if err != nil { + return nil, refuse(StageBudget, "", "", fmt.Sprintf("%s did not report its quota: %v", name, err), + "correct what the reason names and start again; no state is written") + } + estimate := fmt.Sprintf("%d tokens for the %s", row.EstimateTokens, strings.Join(row.Roles, ", ")) + if !ok { + row.Detail = fmt.Sprintf("%s reports no quota, so its check is skipped (an estimate of %s)", name, estimate) + b.Rows = append(b.Rows, row) + continue + } + left := q.RemainingTokens + row.RemainingTokens = &left + if row.EstimateTokens > left { + return nil, refuse(StageBudget, "", "", fmt.Sprintf("%s reports %d tokens of quota left, and the run's estimate is %s "+ + "(%d step(s) at about %d tokens an agent)", name, left, estimate, steps, EstimateTokensPerAgent), + "start it once the quota has reset, route its roles elsewhere, or split the spec into fewer steps a run; no state is written") + } + row.Checked = true + row.Detail = fmt.Sprintf("%s reports %d tokens of quota left, which covers an estimate of %s", name, left, estimate) + b.Rows = append(b.Rows, row) + } + return b, nil +} + +// routeName names a route in a sentence. +func routeName(route string) string { + if route == runner.Host { + return "the host" + } + return "the " + route + " runner" +} diff --git a/internal/core/implement/loop/budget_test.go b/internal/core/implement/loop/budget_test.go new file mode 100644 index 000000000..7ac405ea4 --- /dev/null +++ b/internal/core/implement/loop/budget_test.go @@ -0,0 +1,153 @@ +package loop + +// budget_test.go is criterion 7 of itd-2609201925079472 (spc-2609301921521360, +// the budget check): where a route reports its remaining quota, a run whose +// estimate exceeds it is refused naming both numbers with no state written; a +// route that reports none is named and the check is skipped out loud. The +// fake runner is a quota reader on Options, standing in for a runner that +// reports one, since no shipped runner does. + +import ( + "context" + "errors" + "strconv" + "strings" + "testing" + + "github.com/intentdriven/abcd/internal/core/runner" +) + +// claudeImplementer routes the implementer to the claude runner; every other +// role stays on the host. +const claudeImplementer = `{"roles":{"implementer":{"runner":"claude"}},"runner":{"claude":{}}}` + +// quotas is a fake runner's quota report: the routes named report theirs, +// every other route reports none. +func quotas(left map[string]int64) QuotaReader { + return func(_ context.Context, route string) (runner.Quota, bool, error) { + n, ok := left[route] + return runner.Quota{RemainingTokens: n}, ok, nil + } +} + +// TestAnEstimateOverTheQuotaIsRefusedAndWritesNothing: two steps with the +// implementer on the claude runner estimate 2 × 80000 tokens there; a runner +// reporting 100000 left refuses the start naming both numbers, and no state is +// written. +func TestAnEstimateOverTheQuotaIsRefusedAndWritesNothing(t *testing.T) { + repo := loopRepo(t, readyIntent("", settledQuestions), specWithSteps("1. The parser\n2. The loop\n")) + o := Options{Runners: runnerConfig(t, claudeImplementer, ""), Quota: quotas(map[string]int64{runner.Claude: 100000})} + _, err := Start(repo.Root(), "itd-10", o) + r := mustRefusal(t, err) + want := strconv.Itoa(2 * EstimateTokensPerAgent) + if r.Stage != StageBudget || !strings.Contains(r.Reason, "100000") || !strings.Contains(r.Reason, want) || + !strings.Contains(r.Reason, runner.Claude) { + t.Fatalf("the refusal names the runner, its quota and the estimate %s: %+v", want, r) + } + runTierAbsent(t, repo.Root()) +} + +// TestAnEstimateWithinTheQuotaStartsAndTheRecordNamesTheCheck: a quota over +// the estimate starts the run; the result and the run record name the runner, +// both numbers, and the host's roles, whose quota is not reported, skipped out +// loud. +func TestAnEstimateWithinTheQuotaStartsAndTheRecordNamesTheCheck(t *testing.T) { + repo := loopRepo(t, readyIntent("", settledQuestions), specWithSteps("1. The parser\n2. The loop\n")) + o := Options{Runners: runnerConfig(t, claudeImplementer, ""), Quota: quotas(map[string]int64{runner.Claude: 1000000})} + res, err := Start(repo.Root(), "itd-10", o) + if err != nil { + t.Fatal(err) + } + b := res.Budget + if b == nil || len(b.Rows) != 2 { + t.Fatalf("the start reports a row per route: %+v", b) + } + byRoute := map[string]BudgetRow{} + for _, row := range b.Rows { + byRoute[row.Route] = row + } + c := byRoute[runner.Claude] + if !c.Checked || c.RemainingTokens == nil || *c.RemainingTokens != 1000000 || c.EstimateTokens != 2*EstimateTokensPerAgent || + strings.Join(c.Roles, ",") != RoleImplementer { + t.Fatalf("the claude row is checked with both numbers: %+v", c) + } + h := byRoute[runner.Host] + // Two steps of two reviewers each, and the auditor once on the closing lane. + if h.Checked || h.RemainingTokens != nil || h.EstimateTokens != 5*EstimateTokensPerAgent || !strings.Contains(h.Detail, "reports no quota") { + t.Fatalf("the host row is skipped out loud: %+v", h) + } + st, err := ReadState(repo.Root(), res.RunID) + if err != nil { + t.Fatal(err) + } + var note string + for _, e := range st.Record { + if e.Stage == StageBudget { + note = e.Note + } + } + for _, want := range []string{"claude", "1000000", strconv.Itoa(2 * EstimateTokensPerAgent), "host reports no quota", "skipped"} { + if !strings.Contains(note, want) { + t.Fatalf("the run record's budget line names %q: %q", want, note) + } + } + again, err := Start(repo.Root(), "itd-10", o) + if err != nil || !again.Resumed || again.Budget != nil { + t.Fatalf("a resumed start checks no budget again: %+v %v", again, err) + } +} + +// TestARouteThatReportsNoQuotaIsSkippedOutLoud: with no runner configured +// every role is the host's, which reports no quota; a runner that reports +// none is named the same way; the run starts either way. +func TestARouteThatReportsNoQuotaIsSkippedOutLoud(t *testing.T) { + for _, tc := range []struct { + name string + o func(t *testing.T) Options + want []string + }{ + {"host", func(*testing.T) Options { return Options{} }, []string{"host reports no quota"}}, + {"runner", func(t *testing.T) Options { + return Options{Runners: runnerConfig(t, claudeImplementer, ""), Quota: quotas(nil)} + }, []string{"host reports no quota", "claude runner reports no quota"}}, + {"shipped", func(t *testing.T) Options { return Options{Runners: runnerConfig(t, claudeImplementer, "")} }, + []string{"claude runner reports no quota"}}, + } { + t.Run(tc.name, func(t *testing.T) { + repo := loopRepo(t, readyIntent("", settledQuestions), specWithSteps("")) + res, err := Start(repo.Root(), "itd-10", tc.o(t)) + if err != nil { + t.Fatal(err) + } + if res.Budget == nil { + t.Fatal("the start reports the budget check") + } + for _, row := range res.Budget.Rows { + if row.Checked || row.RemainingTokens != nil { + t.Fatalf("a route reporting no quota is not checked: %+v", row) + } + } + st, _ := ReadState(repo.Root(), res.RunID) + text := recordText(st) + for _, want := range tc.want { + if !strings.Contains(text, want) { + t.Fatalf("the record names %q:\n%s", want, text) + } + } + }) + } +} + +// TestAQuotaThatCannotBeReadRefusesTheStart: a report that fails is not a +// report of none: the start is refused naming it and writes nothing. +func TestAQuotaThatCannotBeReadRefusesTheStart(t *testing.T) { + repo := loopRepo(t, readyIntent("", settledQuestions), specWithSteps("")) + o := Options{Runners: runnerConfig(t, claudeImplementer, ""), Quota: func(context.Context, string) (runner.Quota, bool, error) { + return runner.Quota{}, false, errors.New("the quota endpoint is down") + }} + _, err := Start(repo.Root(), "itd-10", o) + if r := mustRefusal(t, err); r.Stage != StageBudget || !strings.Contains(r.Reason, "endpoint is down") { + t.Fatalf("a failed report refuses the start: %+v", r) + } + runTierAbsent(t, repo.Root()) +} diff --git a/internal/core/implement/loop/drive.go b/internal/core/implement/loop/drive.go index 4099121f2..62c3fa285 100644 --- a/internal/core/implement/loop/drive.go +++ b/internal/core/implement/loop/drive.go @@ -21,7 +21,10 @@ package loop // receipt the verifier refuses leaves the lane awaiting, writes one // fallback receipt into the run's state (Fallbacks) and the record, and // hands the role to the host with the reason (criterion 3), which the run -// record counts per runner and per role (criterion 4). +// record counts per runner and per role (criterion 4); +// - a runner that answers with a rate-limit response is not fallen back on: +// the run is checkpointed and its window ends early (ratelimit.go, +// itd-2609201925079472 criterion 8). // // The runner is started outside the run's lock, which is held only for the // advance before it and the Receipt or the fallback write after it, so a @@ -137,14 +140,20 @@ func Drive(ctx context.Context, repoRoot, runID string, steps Stages, o Options, done = &r return nil }, - Record: func(fb runner.FallbackReceipt) error { return recordFallback(repoRoot, runID, res.Lane, fb, o) }, - Now: o.Now, + Record: func(fb runner.FallbackReceipt) error { return recordFallback(repoRoot, runID, res.Lane, fb, o) }, + Now: o.Now, + PauseOnRateLimit: true, } out, err := d.Dispatch(ctx, req) if err != nil { return res, refuse(StageRunner, "", res.Lane, err.Error(), "the lane still awaits its receipt: correct what the reason names and step again, or start the agent the step names by hand") } + if out.RateLimited != nil { + // Every lane spends the run's budget: the window ends early, and the + // role is not handed to the host (criterion 8, ratelimit.go). + return rateLimited(repoRoot, runID, res.Lane, aw, out.RateLimited, o) + } if out.Handoff { res.Fallback = out.Fallback if fb := out.Fallback; fb != nil { diff --git a/internal/core/implement/loop/drive_test.go b/internal/core/implement/loop/drive_test.go index 806c68c0f..40d25c9d3 100644 --- a/internal/core/implement/loop/drive_test.go +++ b/internal/core/implement/loop/drive_test.go @@ -64,6 +64,15 @@ func loopFakeHarness(mode string) int { argv, _ := json.Marshal(os.Args[1:]) _ = os.WriteFile(filepath.Join(dir, name+".argv.json"), argv, 0o600) } + if mode == "ratelimit" { + // The provider refuses the run at a usage limit: the rate-limit event + // as the claude runner assumes it, an error result, a non-zero exit, + // and no receipt written. + fmt.Println(`{"type":"system","subtype":"init","session_id":"fake-session-3","model":"fake-model"}`) + fmt.Println(`{"type":"rate_limit_event","rate_limit_info":{"status":"rejected","rateLimitType":"five_hour"},"session_id":"fake-session-3"}`) + fmt.Println(`{"type":"result","subtype":"error_during_execution","is_error":true,"result":"limit reached","session_id":"fake-session-3"}`) + return 1 + } prompt := os.Args[len(os.Args)-1] field := func(prefix string) string { sc := bufio.NewScanner(strings.NewReader(prompt)) diff --git a/internal/core/implement/loop/loop.go b/internal/core/implement/loop/loop.go index a3e109b89..4dbffb5ef 100644 --- a/internal/core/implement/loop/loop.go +++ b/internal/core/implement/loop/loop.go @@ -45,6 +45,14 @@ type Options struct { // Roots are where the pace's configuration layers are read; nil reads // them at layered.RootsFor(repoRoot). Roots *layered.Roots + // Runners is the runner configuration read before a new run starts + // (LoadRunners): the route each role takes, which the budget check asks + // for its quota. nil leaves every role on the host. + Runners *runner.Config + // Quota reads the quota a route reports for the budget check; nil asks + // Runners (runner.Config.QuotaFor). A test sets it to play a runner that + // reports one. + Quota QuotaReader } // StageClaim is the refusal stage of a start whose shared-run claim is refused @@ -189,6 +197,10 @@ type StartResult struct { // Pace is the run's pace, each number with the layer that supplied it. // Null for a run started before the loop paced a run. Pace *Pace `json:"pace"` + // Budget is the budget check the call took when it created the run, a + // row per route with its estimate and the quota it reported, or that it + // reported none; null on a resumed start, which checks none. + Budget *Budget `json:"budget"` // HandBack is set when the run's lane stands handed back to the person. HandBack *HandBack `json:"hand_back,omitempty"` Next string `json:"next"` @@ -231,7 +243,10 @@ type StepResult struct { // Fallback is the fallback receipt this call recorded when the runner a // role is routed to did not run it and the host is handed the role. Fallback *runner.FallbackReceipt `json:"fallback,omitempty"` - Next string `json:"next"` + // RateLimit is the rate-limit response a runner answered with in this + // call, which ended the run's window early (NextEligibleAt). + RateLimit *RateLimit `json:"rate_limit,omitempty"` + Next string `json:"next"` // handed is true when this call handed the lane to an agent, false when // it re-told an await an earlier call began: only the call that hands the // work out may start a runner for it. @@ -308,6 +323,12 @@ func start(repoRoot, key string, o Options, pick *RunPick) (StartResult, error) if !chk.OK { return StartResult{}, chk.refusal() } + // The budget check (criterion 7) comes after the checks and before + // anything is written: a run it refuses leaves no state. + budget, err := checkBudget(len(chk.steps), chk.Intent != "", o) + if err != nil { + return StartResult{}, err + } if err := fsutil.EnsureRealDirAll(repoRoot, RunRelDir, dirPerm); err != nil { return StartResult{}, fmt.Errorf("creating %s: %w", RunRelDir, err) } @@ -367,6 +388,7 @@ func start(repoRoot, key string, o Options, pick *RunPick) (StartResult, error) Note: "checks passed; " + st.Lanes[0].ID + " opened for " + laneWork(st, st.Lanes[0])}) st.Record = append(st.Record, Entry{At: now, Stage: StagePace, Note: "pace " + pace.String() + "; the first window opens now"}) + st.Record = append(st.Record, Entry{At: now, Stage: StageBudget, Note: budget.String()}) if pick != nil { rp := *pick rp.Lane = st.Lanes[0].ID @@ -386,6 +408,7 @@ func start(repoRoot, key string, o Options, pick *RunPick) (StartResult, error) } res = startResult(st, chk.Checks, false) res.Claim = claim + res.Budget = budget return nil }) return res, err diff --git a/internal/core/implement/loop/loop_test.go b/internal/core/implement/loop/loop_test.go index b289ab294..cd704b6f9 100644 --- a/internal/core/implement/loop/loop_test.go +++ b/internal/core/implement/loop/loop_test.go @@ -395,8 +395,8 @@ func TestStartCreatesOneLaneAndAStartAgainResumesIt(t *testing.T) { if len(st.Pending) != 1 || st.Pending[0].Number != 3 { t.Fatalf("the unlanded steps after the first wait as pending: %+v", st.Pending) } - if len(st.Record) != 2 || st.Record[0].Stage != "start" || st.Record[1].Stage != StagePace { - t.Fatalf("the record opens with the start, then names the pace: %+v", st.Record) + if len(st.Record) != 3 || st.Record[0].Stage != "start" || st.Record[1].Stage != StagePace || st.Record[2].Stage != StageBudget { + t.Fatalf("the record opens with the start, then names the pace and the budget check: %+v", st.Record) } fi, err := os.Stat(filepath.Join(repo.Root(), filepath.FromSlash(res.State))) if err != nil || fi.Mode().Perm() != filePerm { diff --git a/internal/core/implement/loop/ratelimit.go b/internal/core/implement/loop/ratelimit.go new file mode 100644 index 000000000..9aa722480 --- /dev/null +++ b/internal/core/implement/loop/ratelimit.go @@ -0,0 +1,135 @@ +package loop + +// ratelimit.go is the rate-limit checkpoint (itd-2609201925079472 criterion 8, +// decision 6, from itd-29; spc-2609301921521360). A runner that answers a role +// with a rate-limit response ran nothing, and the role is not handed to the +// host, because every lane of the run spends the same budget. Instead, under +// the run's lock: +// +// - the await the response answered is taken off its lane: its agent has +// stopped, so its slot is free, and the stage hands the work out afresh, +// to a fresh agent with the same brief, once the run may move again; +// - every lane with work in flight, the response's own included, is +// checkpointed to its own branch: the run record names the branch and the +// commit it holds, which is where the lane's next agent takes it up. The +// commits an agent made are on the branch already; the loop makes none of +// its own, so no commit carries text abcd did not compose or judge, and +// whatever an agent left uncommitted stays in the lane's worktree, which +// the loop never cleans under it. An agent still out on another lane may +// hand back its receipt, as in any pause; +// - the window ends early for the whole run: next_eligible_at is written +// once, the run's pause from now, as the window clock writes it; a +// response inside a pause already written leaves its time as it is; +// - the record names the lane, the role and the runner the response came +// from. +// +// The call exits as a closed window does, naming the time; before it every +// step is refused as a pause, and at it a window opens. The pause is the +// run's pace in minutes (decision 2): a reset time a harness reports beside +// its response is not read. + +import ( + "fmt" + "os" + "strings" + "time" + + "github.com/intentdriven/abcd/internal/core/runner" + "github.com/intentdriven/abcd/internal/gitutil" +) + +// The run record's stages of the checkpoint. +const ( + StageRateLimit = "rate-limit" + StageCheckpoint = "checkpoint" +) + +// RateLimit is the rate-limit response a call read: the lane and the role +// whose runner answered with it, and the runner. +type RateLimit struct { + Lane string `json:"lane"` + Role string `json:"role"` + Runner string `json:"runner"` + // Detail is abcd's own account of the response; it never carries the + // harness's text. + Detail string `json:"detail"` +} + +// rateLimited checkpoints the run on a rate-limit response from the runner +// the await on laneID was handed to, and ends its window. +func rateLimited(repoRoot, runID, laneID string, aw Await, fl *runner.Failure, o Options) (StepResult, error) { + var res StepResult + rl := RateLimit{Lane: laneID, Role: aw.Role, Runner: fl.Runner, Detail: fl.Detail} + err := mutate(repoRoot, runID, func(_ *os.Root, st *State) (bool, error) { + now := o.now() + for i := range st.Lanes { + if st.Lanes[i].ID != laneID { + continue + } + l := st.Lanes[i] + var kept []Await + for _, a := range l.Awaits { + if !samePath(repoRoot, a.Receipt, aw.Receipt) { + kept = append(kept, a) + } + } + l.Awaits = kept + st.Lanes[i] = l + } + st.Record = append(st.Record, Entry{At: now, Lane: laneID, Stage: StageRateLimit, + Note: fmt.Sprintf("the %s's runner %s answered with a rate-limit response (%s); its agent ran nothing to hand back, "+ + "so its slot is freed and the work is handed out afresh when the run moves again", aw.Role, fl.Runner, fl.Detail)}) + for _, i := range st.laneOrder() { + l := st.Lanes[i] + if l.ID != laneID && len(l.Awaits) == 0 { + continue + } + st.Record = append(st.Record, Entry{At: now, Lane: l.ID, Stage: StageCheckpoint, Note: checkpointNote(repoRoot, l)}) + } + if st.NextEligibleAt == nil || !now.Before(*st.NextEligibleAt) { + pause := BundledPauseMinutes + if st.Pace != nil { + pause = st.Pace.PauseMinutes.Value + } + until := now.Add(time.Duration(pause) * time.Minute) + opened := "the run's window" + if st.WindowStartedAt != nil { + opened = "the window opened at " + st.WindowStartedAt.UTC().Format(time.RFC3339) + } + st.NextEligibleAt = &until + st.Record = append(st.Record, Entry{At: now, Stage: "pause", + Note: fmt.Sprintf("%s ends early on %s's rate-limit response, since every lane spends the same budget; "+ + "no stage is taken before %s (a %d-minute pause)", opened, laneID, until.UTC().Format(time.RFC3339), pause)}) + } + st.UpdatedAt = now + res = idleResult(*st) + res.NextEligibleAt = st.NextEligibleAt + res.RateLimit = &rl + res.Next = fmt.Sprintf("the %s runner answered %s's %s with a rate-limit response; %s", fl.Runner, laneID, aw.Role, + pausedMove(*st, *st.NextEligibleAt)) + return true, nil + }) + return res, err +} + +// checkpointNote names where a lane stands on its branch at a rate-limit +// response: the branch and the commit it holds, and the agents it still has +// out. A branch that cannot be read is named as such. +func checkpointNote(repoRoot string, l Lane) string { + if l.Branch == "" { + return l.ID + " has no branch yet; it is taken up from its stage, " + string(l.Stage) + } + at := "a head that could not be read" + if tip, err := gitutil.Run(repoRoot, "rev-parse", "--verify", "--quiet", "refs/heads/"+l.Branch+"^{commit}", "--"); err == nil { + at = shortSHA(strings.TrimSpace(tip)) + } + note := fmt.Sprintf("%s checkpointed to %s at %s, at its %s stage", l.ID, l.Branch, at, l.Stage) + if len(l.Awaits) > 0 { + roles := make([]string, 0, len(l.Awaits)) + for _, a := range l.Awaits { + roles = append(roles, a.Role) + } + note += "; its " + strings.Join(roles, ", ") + " may still hand back a receipt" + } + return note +} diff --git a/internal/core/implement/loop/ratelimit_test.go b/internal/core/implement/loop/ratelimit_test.go new file mode 100644 index 000000000..76530cbbf --- /dev/null +++ b/internal/core/implement/loop/ratelimit_test.go @@ -0,0 +1,136 @@ +package loop + +// ratelimit_test.go is criterion 8 of itd-2609201925079472 (spc-2609301921521360, +// the rate-limit checkpoint): a runner's rate-limit response mid-lane, with two +// lanes in flight, ends the window early for the whole run; every lane with +// work in flight is checkpointed to its own branch, next_eligible_at is +// written once, and the record names the lane the response came from. The +// runner is the fake claude harness on PATH, playing a provider that refuses +// the run at a usage limit. + +import ( + "bytes" + "context" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/intentdriven/abcd/internal/core/runner" +) + +// rateLimitEnv puts the fake claude harness, in the given mode, first on PATH, +// ahead of git and the stub forge the fixture needs. +func rateLimitEnv(t *testing.T, mode string) { + t.Helper() + self, err := os.Executable() + if err != nil { + t.Fatal(err) + } + bin := t.TempDir() + if err := os.Symlink(self, filepath.Join(bin, "claude")); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) + t.Setenv(loopFakeEnv, mode) + t.Setenv(loopFakeLogEnv, t.TempDir()) +} + +// TestARateLimitResponseCheckpointsEveryLaneAndEndsTheWindow: lane 1 has its +// implementer out on the host and lane 2's implementer is routed to the claude +// runner, which answers with a rate-limit response. +func TestARateLimitResponseCheckpointsEveryLaneAndEndsTheWindow(t *testing.T) { + f := newParFixture(t, "1. One\n2. Two\n - needs: none\n", Options{SubAgents: strp("2"), Pace: strp("120/300")}) + f.stepUntil(t, "lane 1's implementer is out and lane 2 is ready for its own", func(st State) bool { + return len(st.Lanes) == 2 && len(st.Lanes[0].Awaits) == 1 && st.Lanes[1].Stage == StageImplement && len(st.Lanes[1].Awaits) == 0 + }) + oneHead := laneCommit(t, f.repo, f.lane(t, "lane-1"), "one.txt") + twoHead := strings.TrimSpace(f.repo.Git("rev-parse", "refs/heads/"+f.lane(t, "lane-2").Branch)) + + rateLimitEnv(t, "ratelimit") + store := &memTranscripts{} + f.now = f.now.Add(10 * time.Minute) + res, err := Drive(context.Background(), f.repo.Root(), f.runID, f.stages, f.opts(), + Runners{Config: runnerConfig(t, claudeImplementer, ""), Transcripts: store}) + if err != nil { + t.Fatalf("a rate-limit response is not a failure of the step: %v", err) + } + until := f.now.Add(300 * time.Minute) + if res.RateLimit == nil || res.RateLimit.Lane != "lane-2" || res.RateLimit.Role != RoleImplementer || res.RateLimit.Runner != runner.Claude { + t.Fatalf("the result names the lane, the role and the runner the response came from: %+v", res.RateLimit) + } + if res.NextEligibleAt == nil || !res.NextEligibleAt.Equal(until) || !strings.Contains(res.Next, until.Format(time.RFC3339)) { + t.Fatalf("the window ends early with next_eligible_at %s: %+v", until, res) + } + if len(store.stored) != 1 { + t.Fatalf("the rate-limited run's transcript is stored: %v", store.stored) + } + + st := f.state(t) + if st.NextEligibleAt == nil || !st.NextEligibleAt.Equal(until) { + t.Fatalf("next_eligible_at is written into the state: %v", st.NextEligibleAt) + } + if len(st.Fallbacks) != 0 { + t.Fatalf("a rate limit is not a fallback to the host: %+v", st.Fallbacks) + } + if len(st.Lanes[1].Awaits) != 0 || len(st.Lanes[0].Awaits) != 1 { + t.Fatalf("the rate-limited agent's slot is freed and lane 1's agent is still out: %+v", st.Lanes) + } + text := recordText(st) + for _, want := range []string{ + "lane-2 " + StageRateLimit + " the implementer's runner claude answered with a rate-limit response", + "lane-1 " + StageCheckpoint + " lane-1 checkpointed to " + st.Lanes[0].Branch + " at " + shortSHA(oneHead), + "lane-2 " + StageCheckpoint + " lane-2 checkpointed to " + st.Lanes[1].Branch + " at " + shortSHA(twoHead), + } { + if !strings.Contains(text, want) { + t.Fatalf("the record carries %q:\n%s", want, text) + } + } + if !strings.Contains(text, "ends early on lane-2's rate-limit response") { + t.Fatalf("the pause names the response's lane:\n%s", text) + } + + // A second response inside the pause, from the lane still in flight, + // frees its slot and is recorded, but next_eligible_at is written once. + second := f.now.Add(5 * time.Minute) + if _, err := rateLimited(f.repo.Root(), f.runID, "lane-1", f.await(t, "lane-1", RoleImplementer), + &runner.Failure{Runner: runner.Claude, Reason: runner.ReasonRateLimited, Detail: "again"}, Options{Now: func() time.Time { return second }}); err != nil { + t.Fatal(err) + } + st = f.state(t) + pauses, limits := 0, 0 + for _, e := range st.Record { + switch e.Stage { + case "pause": + pauses++ + case StageRateLimit: + limits++ + } + } + if !st.NextEligibleAt.Equal(until) || pauses != 1 || limits != 2 || st.SlotsInUse() != 0 { + t.Fatalf("next_eligible_at is written once for the run: %v, %d pause(s), %d response(s), %d slot(s):\n%s", + st.NextEligibleAt, pauses, limits, st.SlotsInUse(), recordText(st)) + } + + // Before next_eligible_at nothing moves and the state is unchanged. + before := stateBytes(t, f.repo.Root(), f.runID) + f.now = until.Add(-time.Minute) + _, err = advance(f.repo.Root(), f.runID, f.stages, f.opts()) + if r := mustRefusal(t, err); r.Stage != "pause" || !strings.Contains(r.Reason, until.Format(time.RFC3339)) { + t.Fatalf("a step inside the pause is refused naming the time: %+v", r) + } + if !bytes.Equal(before, stateBytes(t, f.repo.Root(), f.runID)) { + t.Fatal("a refused step changes nothing") + } + + // At next_eligible_at a window opens and each lane's work is handed out + // afresh from its branch, where it was checkpointed. + f.now = until + f.stepUntil(t, "both implementers are out again", func(st State) bool { + return len(st.Lanes[0].Awaits) == 1 && len(st.Lanes[1].Awaits) == 1 + }) + if st := f.state(t); st.NextEligibleAt != nil || st.Lanes[0].Branch == "" || st.Lanes[1].Branch == "" { + t.Fatalf("the new window runs on the lanes' own branches: %+v", st) + } +} diff --git a/internal/core/runner/claude.go b/internal/core/runner/claude.go index 9eb1e6352..87e645b0d 100644 --- a/internal/core/runner/claude.go +++ b/internal/core/runner/claude.go @@ -75,12 +75,54 @@ func (c *ClaudeCLI) Run(ctx context.Context, req Request) (Answer, []byte, error res, err := c.launch.run(ctx, Claude, bin, c.args(req), nil, req.Dir, req.timeout()) transcript := res.transcript() if err != nil { + // A harness refused at a usage limit exits non-zero: the response it + // printed first is the reason, not the exit. + if claudeRateLimited(res.stdout) { + return Answer{}, transcript, claudeLimit() + } return Answer{}, transcript, err } ans, err := parseClaude(res.stdout) return ans, transcript, err } +// claudeRateLimit is the event a rate limit is reported in, and the status +// that says the provider refused the run. The shape is the Agent SDK's +// rate-limit event as this build assumes it (a rate_limit_event whose +// rate_limit_info.status is "rejected"; "allowed" and "allowed_warning" let +// the run go on); the live shape is owed to a person's check, as the opencode +// events' are. +const ( + claudeRateLimitType = "rate_limit_event" + claudeRateLimitRejected = "rejected" +) + +// claudeLimit is the failure a rate-limit response is. Its detail is abcd's +// own: nothing of the harness's text reaches it. +func claudeLimit() *Failure { + return fail(Claude, ReasonRateLimited, "its provider refused the run at a usage limit (a rejected %s)", claudeRateLimitType) +} + +// claudeRateLimited reports whether out carries a rejected rate-limit event +// and no successful result after it. Lines that are not events are passed +// over: a stream cut short by the exit is read for what it holds. +func claudeRateLimited(out []byte) bool { + limited := false + for _, line := range bytes.Split(out, []byte("\n")) { + var ev claudeEvent + if json.Unmarshal(bytes.TrimSpace(line), &ev) != nil { + continue + } + switch { + case ev.Type == claudeRateLimitType && ev.RateLimit != nil && ev.RateLimit.Status == claudeRateLimitRejected: + limited = true + case ev.Type == "result" && !ev.IsError && ev.Subtype == "success": + limited = false + } + } + return limited +} + // claudeEvent is the part of a stream-json event the runner reads. type claudeEvent struct { Type string `json:"type"` @@ -89,6 +131,10 @@ type claudeEvent struct { Model string `json:"model"` IsError bool `json:"is_error"` Result string `json:"result"` + // RateLimit is a rate_limit_event's report; nil on every other event. + RateLimit *struct { + Status string `json:"status"` + } `json:"rate_limit_info"` } // parseClaude reads the event stream: one JSON object per line, the init @@ -123,6 +169,9 @@ func parseClaude(out []byte) (Answer, error) { result = &e } } + if (result == nil || result.IsError || result.Subtype != "success") && claudeRateLimited(out) { + return Answer{}, claudeLimit() + } if result == nil { return Answer{}, fail(Claude, ReasonUnparsable, "its output carries no result event") } diff --git a/internal/core/runner/dispatch.go b/internal/core/runner/dispatch.go index 143df4249..77dfc8d7d 100644 --- a/internal/core/runner/dispatch.go +++ b/internal/core/runner/dispatch.go @@ -52,6 +52,12 @@ type Dispatcher struct { Prepare func(runner string, req Request) error // Now is the clock the receipts are stamped with; time.Now when nil. Now func() time.Time + // PauseOnRateLimit is set by a caller that pauses its run on a rate-limit + // response (the loop's process driver, itd-2609201925079472 criterion 8): + // a runner that answers with one is not fallen back on, and the outcome + // carries the response in RateLimited instead, with no fallback receipt + // written. Unset, a rate-limit response is a failure like any other. + PauseOnRateLimit bool } // Outcome is one dispatch's result. @@ -66,6 +72,10 @@ type Outcome struct { Fallback *FallbackReceipt // Receipt is the role's receipt block. Receipt RoleReceipt + // RateLimited is the rate-limit response the runner answered with, when + // the dispatcher pauses on one: nothing ran the role, and no fallback was + // taken or recorded. + RateLimited *Failure } // Dispatch runs req by its role's route. @@ -96,6 +106,9 @@ func (d *Dispatcher) Dispatch(ctx context.Context, req Request) (Outcome, error) return out, nil } ans, err := d.runOn(ctx, landing, req) + if rl := d.rateLimit(err); rl != nil { + return out.limited(rl), nil + } if err != nil { return Outcome{}, fmt.Errorf("runner: the configured host %s did not run %s: %w", landing, req.Role, err) } @@ -106,6 +119,9 @@ func (d *Dispatcher) Dispatch(ctx context.Context, req Request) (Outcome, error) if err == nil { return out.ran(route.Runner, ans), nil } + if rl := d.rateLimit(err); rl != nil { + return out.limited(rl), nil + } var fl *Failure if !errors.As(err, &fl) { return Outcome{}, err @@ -132,6 +148,9 @@ func (d *Dispatcher) Dispatch(ctx context.Context, req Request) (Outcome, error) return out, nil } ans, err = d.runOn(ctx, landing, req) + if rl := d.rateLimit(err); rl != nil { + return out.limited(rl), nil + } if err != nil { return Outcome{}, fmt.Errorf("runner: %s %s for %s, and the configured host %s did not run it either: %w", route.Runner, fl.Reason, req.Role, landing, err) @@ -142,6 +161,24 @@ func (d *Dispatcher) Dispatch(ctx context.Context, req Request) (Outcome, error) // none is the route a receipt names when nothing could run the role. const none = "none" +// rateLimit is err's rate-limit response when the dispatcher pauses on one, +// else nil. +func (d *Dispatcher) rateLimit(err error) *Failure { + var fl *Failure + if d.PauseOnRateLimit && errors.As(err, &fl) && fl.Reason == ReasonRateLimited { + return fl + } + return nil +} + +// limited is the outcome of a run its runner's provider refused at a rate +// limit: nothing ran the role. +func (o Outcome) limited(fl *Failure) Outcome { + o.RateLimited = fl + o.Receipt.Route.Ran = none + return o +} + func (o Outcome) ran(route string, ans Answer) Outcome { o.Answer = &ans o.Receipt.Route.Ran = route diff --git a/internal/core/runner/main_test.go b/internal/core/runner/main_test.go index f2fde9b31..b23e3879c 100644 --- a/internal/core/runner/main_test.go +++ b/internal/core/runner/main_test.go @@ -92,6 +92,30 @@ func fakeHarness(mode string) int { fmt.Fprintln(os.Stdout, `{"type":"result","subtype":"error_during_execution","is_error":true,"result":"refused","session_id":"fake-session-1"}`) } return 0 + case "ratelimit", "ratelimit-passed": + // A harness whose provider refuses the run at a usage limit, exiting + // non-zero; and one whose limit warned and then let the run finish. + if name == "opencode" { + if mode == "ratelimit" { + fmt.Fprintln(os.Stdout, `{"type":"step_start","sessionID":"ses_fake1","part":{"type":"step-start"}}`) + fmt.Fprintln(os.Stdout, `{"type":"error","sessionID":"ses_fake1","error":{"name":"APIError","data":{"message":"rate limited","statusCode":429,"isRetryable":true}}}`) + return 1 + } + fmt.Fprintln(os.Stdout, `{"type":"text","sessionID":"ses_fake1","part":{"type":"text","text":"done"}}`) + return 0 + } + fmt.Fprintln(os.Stdout, `{"type":"system","subtype":"init","session_id":"fake-session-1","model":"fake-model"}`) + if mode == "ratelimit" { + fmt.Fprintln(os.Stdout, `{"type":"rate_limit_event","rate_limit_info":{"status":"rejected","rateLimitType":"five_hour","resetsAt":1790000000},"session_id":"fake-session-1"}`) + fmt.Fprintln(os.Stdout, `{"type":"result","subtype":"error_during_execution","is_error":true,"result":"limit reached","session_id":"fake-session-1"}`) + return 1 + } + if rec := promptField(prompt, "Receipt: "); rec != "" { + _ = os.WriteFile(rec, []byte(`{"ok":true}`), 0o600) + } + fmt.Fprintln(os.Stdout, `{"type":"rate_limit_event","rate_limit_info":{"status":"allowed_warning","rateLimitType":"five_hour"},"session_id":"fake-session-1"}`) + fmt.Fprintln(os.Stdout, `{"type":"result","subtype":"success","is_error":false,"result":"done","session_id":"fake-session-1"}`) + return 0 case "hugemodel", "ctrlmodel", "suffixmodel": // A harness whose init event reports a model: one past any bound, // one carrying control bytes, and a real id's bracketed suffix. diff --git a/internal/core/runner/opencode.go b/internal/core/runner/opencode.go index 223e1dcb9..5188a7373 100644 --- a/internal/core/runner/opencode.go +++ b/internal/core/runner/opencode.go @@ -76,6 +76,11 @@ func (o *OpenCodeCLI) Run(ctx context.Context, req Request) (Answer, []byte, err res, err := o.launch.run(ctx, OpenCode, bin, o.args(req), openCodeSeal, req.Dir, req.timeout()) transcript := res.transcript() if err != nil { + // A harness refused at a usage limit may exit non-zero: the error + // event it printed first is the reason, not the exit. + if openCodeRateLimited(res.stdout) { + return Answer{}, transcript, openCodeLimit() + } return Answer{}, transcript, err } ans, err := parseOpenCode(res.stdout) @@ -94,6 +99,41 @@ type openCodeEvent struct { Type string `json:"type"` Text string `json:"text"` } `json:"part"` + // Error is an error event's error; its status code is the provider's + // HTTP status when the error is an API error. + Error *struct { + Data struct { + StatusCode int `json:"statusCode"` + } `json:"data"` + } `json:"error"` +} + +// openCodeRateLimitStatus is the provider's status a rate limit is reported +// with: an error event whose error.data.statusCode is 429 (HTTP's Too Many +// Requests). The field is opencode's API error as this build assumes it; the +// live shape is owed to a person's check, as its other events' are. +const openCodeRateLimitStatus = 429 + +// openCodeLimit is the failure a rate-limit response is, in abcd's words. +func openCodeLimit() *Failure { + return fail(OpenCode, ReasonRateLimited, "its provider refused the run at a rate limit (an error event with status %d)", openCodeRateLimitStatus) +} + +// limited reports whether the event is a rate-limit response. +func (ev openCodeEvent) limited() bool { + return ev.Type == "error" && ev.Error != nil && ev.Error.Data.StatusCode == openCodeRateLimitStatus +} + +// openCodeRateLimited reports whether out carries a rate-limit response. +// Lines that are not events are passed over. +func openCodeRateLimited(out []byte) bool { + for _, line := range bytes.Split(out, []byte("\n")) { + var ev openCodeEvent + if json.Unmarshal(bytes.TrimSpace(line), &ev) == nil && ev.limited() { + return true + } + } + return false } // parseOpenCode reads the event stream: one JSON object per line. An error @@ -117,6 +157,9 @@ func parseOpenCode(out []byte) (Answer, error) { } switch ev.Type { case "error": + if ev.limited() { + return Answer{}, openCodeLimit() + } return Answer{}, fail(OpenCode, ReasonRefused, "it reported an error event") case "text": ans.Text = ev.Part.Text diff --git a/internal/core/runner/quota.go b/internal/core/runner/quota.go new file mode 100644 index 000000000..477229428 --- /dev/null +++ b/internal/core/runner/quota.go @@ -0,0 +1,62 @@ +package runner + +// quota.go is what a route reports of the budget left to it, the runner's +// half of the budget check a run takes before it starts (itd-2609201925079472 +// criterion 7, spc-2609301921521360). +// +// A route reports a quota only when its runner can say what is left without +// spending a turn to find out. No shipped runner does: the adapters read only +// the event stream a run prints, and no headless quota read is known to this +// build for either harness (whether one exists is owed to a person's check), +// and abcd cannot read a host session's. So each reports none, and the loop +// names it and skips the check out loud; a runner that gains the report +// implements QuotaReporter and is read from then on, with nothing else +// changing. + +import ( + "context" + "fmt" +) + +// Quota is what a route reports of the budget left to it. +type Quota struct { + // RemainingTokens is the tokens the route may still spend before its + // provider's limit refuses it. + RemainingTokens int64 `json:"remaining_tokens"` +} + +// QuotaReporter is a Runner that can report its remaining quota before a run +// starts. +type QuotaReporter interface { + Quota(ctx context.Context) (Quota, error) +} + +// QuotaFor reports the quota route reports, ok false when it reports none: +// the host, a runner this machine does not enable, or a runner that cannot +// report one. An error is a report that failed, which is not none. +func (c *Config) QuotaFor(ctx context.Context, route string) (Quota, bool, error) { + if c == nil || route == Host { + return Quota{}, false, nil + } + rc, ok := c.runners[route] + if !ok { + return Quota{}, false, nil + } + return quotaOf(ctx, c.adapter(rc)) +} + +// quotaOf reads r's quota when r can report one. +func quotaOf(ctx context.Context, r Runner) (Quota, bool, error) { + qr, ok := r.(QuotaReporter) + if !ok { + return Quota{}, false, nil + } + q, err := qr.Quota(ctx) + if err != nil { + return Quota{}, false, fmt.Errorf("runner: the %s runner's quota could not be read: %w", r.Name(), err) + } + if q.RemainingTokens < 0 { + return Quota{}, false, fmt.Errorf("runner: the %s runner reports a negative quota (%d tokens)", r.Name(), q.RemainingTokens) + } + return q, true, nil +} diff --git a/internal/core/runner/ratelimit_test.go b/internal/core/runner/ratelimit_test.go new file mode 100644 index 000000000..ebbccd977 --- /dev/null +++ b/internal/core/runner/ratelimit_test.go @@ -0,0 +1,115 @@ +package runner + +// ratelimit_test.go proves the runner's half of itd-2609201925079472's +// criteria 7 and 8 (spc-2609301921521360): a harness's rate-limit response is +// a failure of its own kind, which a dispatcher that pauses on it hands back +// rather than falling back on, and a route reports its remaining quota only +// when its runner can, which no shipped runner can. + +import ( + "context" + "errors" + "strings" + "testing" +) + +// TestARateLimitResponseIsItsOwnFailure: each harness's rate-limit response is +// a rate-limited failure, whether the harness exits non-zero after it or not; +// a limit that warned and let the run finish is an answer. +func TestARateLimitResponseIsItsOwnFailure(t *testing.T) { + for _, h := range []string{Claude, OpenCode} { + t.Run(h, func(t *testing.T) { + f := newFake(t, "ratelimit", h) + var r Runner = newClaude("") + if h == OpenCode { + r = newOpenCode("") + } + _, transcript, err := r.Run(context.Background(), f.request("scribe")) + var fl *Failure + if !errors.As(err, &fl) || fl.Reason != ReasonRateLimited { + t.Fatalf("err = %v, want a %s failure", err, ReasonRateLimited) + } + if len(transcript) == 0 { + t.Fatal("the transcript of a rate-limited run is still returned") + } + if strings.Contains(fl.Detail, "limit reached") || strings.Contains(fl.Detail, "rate limited") { + t.Fatalf("the detail is abcd's own, never the harness's text: %q", fl.Detail) + } + }) + } + f := newFake(t, "ratelimit-passed", Claude) + if _, _, err := newClaude("").Run(context.Background(), f.request("scribe")); err != nil { + t.Fatalf("a limit that only warned is not a rate-limit response: %v", err) + } +} + +// TestARateLimitedRunnerPausesRatherThanFallingBack is the dispatch half of +// criterion 8: a dispatcher that pauses on a rate limit hands the response +// back with no fallback receipt and no hand-off, since every route spends the +// run's budget; one that does not (the interviews) falls back as on any other +// failure. +func TestARateLimitedRunnerPausesRatherThanFallingBack(t *testing.T) { + f := newFake(t, "ratelimit", Claude, OpenCode) + h := newHarness(t, mustLoad(t, routed(routedMachine), ""), true) + h.d.PauseOnRateLimit = true + out, err := h.d.Dispatch(context.Background(), f.request("ruthless-reviewer")) + if err != nil { + t.Fatal(err) + } + if out.RateLimited == nil || out.RateLimited.Runner != OpenCode || out.RateLimited.Reason != ReasonRateLimited { + t.Fatalf("the rate-limit response is handed back: %+v", out) + } + if out.Handoff || out.Fallback != nil || out.Answer != nil || len(h.receipts) != 0 { + t.Fatalf("a rate limit is no fallback: outcome %+v, receipts %v", out, h.receipts) + } + if len(h.store.got) != 1 { + t.Fatalf("the rate-limited run's transcript is stored: %v", h.store.got) + } + + h = newHarness(t, mustLoad(t, routed(routedMachine), ""), true) + out, err = h.d.Dispatch(context.Background(), f.request("ruthless-reviewer")) + if err != nil { + t.Fatal(err) + } + if !out.Handoff || out.RateLimited != nil || len(h.receipts) != 1 || h.receipts[0].Reason != ReasonRateLimited { + t.Fatalf("without the pause a rate limit falls back as any failure does: %+v %v", out, h.receipts) + } +} + +// reporter is a runner that reports its quota. +type reporter struct { + q Quota + err error +} + +func (reporter) Name() string { return "reporter" } +func (reporter) Run(context.Context, Request) (Answer, []byte, error) { + return Answer{}, nil, errors.New("not run") +} +func (r reporter) Quota(context.Context) (Quota, error) { return r.q, r.err } + +// TestOnlyARunnerThatCanReportsAQuota is criterion 7's runner half: the host, +// a runner this machine does not enable, and each shipped runner report none; +// a runner that implements the report is read, and a negative one refused. +func TestOnlyARunnerThatCanReportsAQuota(t *testing.T) { + c := mustLoad(t, routedMachine, "") + for _, route := range []string{Host, Claude, OpenCode, "elsewhere"} { + if _, ok, err := c.QuotaFor(context.Background(), route); ok || err != nil { + t.Fatalf("%s reports no quota: ok %v, err %v", route, ok, err) + } + } + var nilConfig *Config + if _, ok, err := nilConfig.QuotaFor(context.Background(), Host); ok || err != nil { + t.Fatalf("no configuration reports no quota: %v %v", ok, err) + } + q, ok, err := quotaOf(context.Background(), reporter{q: Quota{RemainingTokens: 500000}}) + if err != nil || !ok || q.RemainingTokens != 500000 { + t.Fatalf("a reporter's quota is read: %+v %v %v", q, ok, err) + } + if _, _, err := quotaOf(context.Background(), reporter{q: Quota{RemainingTokens: -1}}); err == nil { + t.Fatal("a negative quota is refused") + } + if _, _, err := quotaOf(context.Background(), reporter{err: errors.New("down")}); err == nil { + t.Fatal("a report that fails is an error, not none") + } +} diff --git a/internal/core/runner/runner.go b/internal/core/runner/runner.go index 36f1f2d41..5f53ac33e 100644 --- a/internal/core/runner/runner.go +++ b/internal/core/runner/runner.go @@ -19,6 +19,13 @@ // the role, the runner asked for, the reason and the route that ran. Tally // counts those receipts per runner and per role for the run's summary. // +// A harness whose provider refused the run at a rate or usage limit fails +// with ReasonRateLimited; a dispatcher that pauses on it (the loop's process +// driver) hands the response back instead of falling back, so the loop can +// end the run's window early (itd-2609201925079472 criterion 8). A route +// reports its remaining quota through QuotaFor (quota.go) for the budget a run +// checks before it starts (criterion 7). +// // The package never prints. It starts processes only through Dispatch and the // adapters' Run, and never logs a harness in: a harness's credential is its // own, read from its own environment or configuration, and abcd never puts one @@ -117,6 +124,11 @@ const ( ReasonUnparsable Reason = "unparsable" // ReasonInvalid: the answer failed the contract's validator. ReasonInvalid Reason = "invalid" + // ReasonRateLimited: the harness reported that its provider refused the + // run at a rate or usage limit (itd-2609201925079472 criterion 8). A + // dispatcher that pauses on it (PauseOnRateLimit) hands it back rather + // than falling back, since every route of a run spends the same budget. + ReasonRateLimited Reason = "rate-limited" ) // Failure is a runner's failure. Detail is written by abcd, never copied from diff --git a/internal/surface/cli/build.go b/internal/surface/cli/build.go index 02726fac8..df58c31cc 100644 --- a/internal/surface/cli/build.go +++ b/internal/surface/cli/build.go @@ -130,6 +130,13 @@ func newBuildCommand(asJSON *bool) *cobra.Command { "ceiling is free. A lane whose validators still do not pass after its fix rounds is\n" + "handed back: it stops as unachievable with the last round's findings, the run starts nothing\n" + "further for it, and `abcd implement step` refuses naming the hand-back.\n\n" + + "A new run checks its budget before anything is written: each route the run's roles take\n" + + "(the host, or a runner) is asked for the quota it has left, and the run's estimate, about\n" + + "80000 tokens an agent for the implementer and the two reviewers on every step and the\n" + + "intent-auditor once, is compared with what each route reports for its roles. A route whose\n" + + "estimate exceeds its quota refuses the start naming both numbers, and writes nothing; a\n" + + "route that reports none (the host, and every shipped runner) is named and its check is\n" + + "skipped out loud, in the result and the run record.\n\n" + "The run then moves one step per `abcd implement step`, driven by the host session.\n\n" + "The runner configuration is read before the run is created: roles..runner (host,\n" + "the default, or a runner) and the runners this machine enables under runner. in\n" + @@ -163,10 +170,11 @@ func newBuildCommand(asJSON *bool) *cobra.Command { for _, n := range notes { fmt.Fprintln(cmd.ErrOrStderr(), termsafe.Sanitize(n)) } - if _, err := loadRunners(cmd, roots); err != nil { + cfg, err := loadRunners(cmd, roots) + if err != nil { return loopFail(cmd.OutOrStdout(), *asJSON, prefix, err) } - o := loop.Options{Session: session, Roots: &roots} + o := loop.Options{Session: session, Roots: &roots, Runners: cfg} if cmd.Flags().Changed("pace") { o.Pace = &pace } @@ -188,6 +196,7 @@ func newBuildCommand(asJSON *bool) *cobra.Command { fmt.Fprintf(w, "build %s: %s run %s\n", termsafe.Sanitize(args[0]), verb, res.RunID) fmt.Fprintf(w, " state: %s\n", res.State) renderPace(w, res.Pace) + renderBudget(w, res.Budget) renderLaneLine(w, res.Lane) renderPending(w, res.Pending) switch { @@ -252,10 +261,11 @@ func newBuildNextCommand(asJSON *bool) *cobra.Command { for _, n := range notes { fmt.Fprintln(cmd.ErrOrStderr(), termsafe.Sanitize(n)) } - if _, err := loadRunners(cmd, roots); err != nil { + cfg, err := loadRunners(cmd, roots) + if err != nil { return loopFail(cmd.OutOrStdout(), *asJSON, prefix, err) } - o := loop.Options{Session: session, Roots: &roots} + o := loop.Options{Session: session, Roots: &roots, Runners: cfg} if cmd.Flags().Changed("pace") { o.Pace = &pace } @@ -318,6 +328,7 @@ func renderNext(w io.Writer, res loop.NextResult) { fmt.Fprintf(w, " (the lane's worktree stage commits it as %s's first commit)\n", res.Start.Lane.ID) fmt.Fprintf(w, " state: %s\n", res.Start.State) renderPace(w, res.Start.Pace) + renderBudget(w, res.Start.Budget) renderLaneLine(w, res.Start.Lane) renderPending(w, res.Start.Pending) fmt.Fprintf(w, "next: %s\n", termsafe.Sanitize(fsutil.RedactHome(res.Start.Next))) @@ -333,6 +344,18 @@ func renderPace(w io.Writer, p *loop.Pace) { fmt.Fprintf(w, " pace: %s\n", termsafe.Sanitize(p.String())) } +// renderBudget renders the budget check a new run took, a line per route: the +// estimate and the quota it reported, or that it reported none and its check +// was skipped. A resumed start took none and renders nothing. +func renderBudget(w io.Writer, b *loop.Budget) { + if b == nil { + return + } + for _, r := range b.Rows { + fmt.Fprintf(w, " budget: %s\n", termsafe.Sanitize(r.Detail)) + } +} + // renderLaneLine renders one lane as a line, and its footprint once its stages // have made one. func renderLaneLine(w io.Writer, l loop.Lane) { @@ -529,6 +552,10 @@ func renderStepResult(w io.Writer, verb string, res loop.StepResult) { case res.HandBack != nil: fmt.Fprintf(w, "%s: HANDED BACK: %s's %s stopped as %s after %d fix round(s); the run starts nothing further for it\n", verb, res.RunID, res.Lane, res.HandBack.Verdict, res.HandBack.FixRounds) + case res.RateLimit != nil && res.NextEligibleAt != nil: + fmt.Fprintf(w, "%s: %s's window ended early on a rate-limit response; paused until %s\n", verb, res.RunID, res.NextEligibleAt.UTC().Format(time.RFC3339)) + fmt.Fprintf(w, " rate limit: the %s runner answered %s's %s (%s)\n", termsafe.Sanitize(res.RateLimit.Runner), res.RateLimit.Lane, + termsafe.Sanitize(res.RateLimit.Role), termsafe.Sanitize(res.RateLimit.Detail)) case res.NextEligibleAt != nil: fmt.Fprintf(w, "%s: %s's window has elapsed; paused until %s\n", verb, res.RunID, res.NextEligibleAt.UTC().Format(time.RFC3339)) case res.CeilingReached: @@ -658,6 +685,12 @@ func newImplementStepCommand(asJSON *bool) *cobra.Command { "nothing, writes next_eligible_at (now plus the run's pause) and exits 0 naming it; an\n" + "agent already started may still hand back its receipt. Before next_eligible_at the call\n" + "is refused as a pause and nothing changes; at or after it, a new window opens.\n\n" + + "A runner that answers with a rate-limit response is not fallen back on, since every lane\n" + + "spends the run's budget: its agent's slot is freed, every lane with work in flight is\n" + + "checkpointed to its own branch (the run record names the branch and the commit it holds),\n" + + "the window ends early with next_eligible_at written once (now plus the run's pause), the\n" + + "record names the lane, the role and the runner the response came from, and the call exits\n" + + "0 naming the time. At next_eligible_at the work is handed out afresh.\n\n" + "--run names the run; without it, the one run in progress in this checkout. Exit 2 on a\n" + "refusal, exit 3 on a pause or a locked run state.", Args: cobra.NoArgs, diff --git a/internal/surface/cli/build_budget_surface_test.go b/internal/surface/cli/build_budget_surface_test.go new file mode 100644 index 000000000..dd57bca73 --- /dev/null +++ b/internal/surface/cli/build_budget_surface_test.go @@ -0,0 +1,70 @@ +package cli + +import ( + "encoding/json" + "os" + "strings" + "testing" + + "github.com/intentdriven/abcd/internal/abcdhome" +) + +// The budget check at the surface (itd-2609201925079472 criterion 7, +// spc-2609301921521360): `abcd build` hands the runner configuration it read +// to the start, so a route the machine sends a role to is asked for its quota +// beside the host's; neither reports one, and each is named in the text, the +// JSON and the run record, skipped out loud. + +func TestBuildNamesEachRouteItsBudgetCheckSkipped(t *testing.T) { + buildRepo(t) + home := os.Getenv("HOME") + if err := os.MkdirAll(abcdhome.Path(home), 0o700); err != nil { + t.Fatal(err) + } + machine := `{"roles":{"implementer":{"runner":"claude"}},"runner":{"claude":{}}}` + if err := os.WriteFile(abcdhome.Path(home, "config.json"), []byte(machine), 0o600); err != nil { + t.Fatal(err) + } + out := mustImplement(t, "build", "itd-10", "--json") + var res struct { + Budget *struct { + Rows []struct { + Route string `json:"route"` + Roles []string `json:"roles"` + RemainingTokens *int64 `json:"remaining_tokens"` + Checked bool `json:"checked"` + Detail string `json:"detail"` + } `json:"rows"` + } `json:"budget"` + } + if err := json.Unmarshal([]byte(out), &res); err != nil { + t.Fatalf("%v: %s", err, out) + } + if res.Budget == nil || len(res.Budget.Rows) != 2 { + t.Fatalf("the payload carries a budget row per route: %s", out) + } + routes := map[string]bool{} + for _, r := range res.Budget.Rows { + routes[r.Route] = true + if r.Checked || r.RemainingTokens != nil || !strings.Contains(r.Detail, "reports no quota") { + t.Fatalf("a route reporting no quota is skipped out loud: %+v", r) + } + } + if !routes["claude"] || !routes["host"] { + t.Fatalf("the routed role's runner and the host are each asked: %s", out) + } + status := mustImplement(t, "implement", "status") + for _, want := range []string{"the claude runner reports no quota", "the host reports no quota"} { + if !strings.Contains(status, want) { + t.Fatalf("the run record names %q:\n%s", want, status) + } + } +} + +func TestBuildTextNamesTheBudgetCheck(t *testing.T) { + buildRepo(t) + code, out, errOut := implementCLI(t, "build", "itd-10") + if code != 0 || !strings.Contains(out, "budget: the host reports no quota, so its check is skipped") { + t.Fatalf("the text names the budget check: exit %d\n%s\n%s", code, out, errOut) + } +} diff --git a/internal/surface/cli/drain.go b/internal/surface/cli/drain.go index c3ef9c1a9..a87d69601 100644 --- a/internal/surface/cli/drain.go +++ b/internal/surface/cli/drain.go @@ -126,7 +126,13 @@ func runDrain(cmd *cobra.Command, asJSON bool, repoRoot string, maxLanes int, pa for _, n := range notes { fmt.Fprintln(cmd.ErrOrStderr(), termsafe.Sanitize(n)) } - o := loop.Options{Roots: &roots} + // The runner configuration is read as `abcd build` reads it: a run the + // drain starts checks its budget on the routes its roles take. + cfg, err := loadRunners(cmd, roots) + if err != nil { + return loopFail(cmd.OutOrStdout(), asJSON, prefix, err) + } + o := loop.Options{Roots: &roots, Runners: cfg} if cmd.Flags().Changed("pace") { o.Pace = &pace }