fix(cost): price only the calls no receipt covered - #1185
Conversation
`streamSandboxLeaf` read `usdEstimated` off the loop TOTAL whenever
`costUsdKnown` was false. That flag is an AND: `run-loop.ts` clears it on
any single `llm_call` carrying `usdKnown: false`, and again for an
iteration that saw no call at all. One unproven call therefore relabelled
every receipt-backed dollar beside it as a price, and `usd - usdEstimated`
— the quantity the estimate channel exists to make readable — reported
$0.00 billed against money a provider really had charged.
Measured on the real path with a scripted box: a leaf whose stream carried
`llm_call {costUsd: 0.5, usdKnown: false}` then a terminal
`done {totalCostUsd: 1.5}` (the ledger credits the $1.00 residual as a
provider receipt) settled `{usd: 1.5, usdKnown: false, usdEstimated: 1.5}`.
It now settles `usdEstimated: 0.5`, so the subtraction returns the $1.00
the receipt states.
The unproven part is summed where the per-call fact exists, in the loop
ledger that already accumulates `costUsd`, and carried on `Iteration` and
`LoopResult` beside it. This is the partition `spendFromUsageEvents` and
`bridge-executor` already apply per event; the leaf was the one settlement
path that folded first and classified afterwards.
Nothing is promoted: `usdKnown` is untouched, and budget admission reads
only `usd`/`usdKnown`, so a dollar-capped run admits and spends exactly
what it did before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`runDirectoryHolderIsLive` reads the lock file outside the mutation guard that `acquireRunDirectoryLock` evaluates the identical rule under, so its answer can be stale by the time a caller acts on it. The docstring named a supervisor picking up an abandoned directory as a caller without saying that such a caller must still acquire, which invites acting on a `false` that a concurrent acquire has already overtaken. Behaviour is unchanged; the function is for reporting and listing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`leakedReservations` was asserted only on the in-memory result, so nothing proved the finding outlives the process that made it. #1181's symptom was a persisted `failure.json` reporting `nodes: 0` with no settle record at all, which makes the durable artifact the one that mattered. The case runs the settled result through `settleRecordJson` — the same serializer `result.json` is written with, which refuses any value JSON would misstate — and asserts the leak and its assignment survive. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4a3cdcc to
91b1d82
Compare
|
Rebased onto main (was based on 0.211.0, four releases behind) and prepared as 0.216.0 with the CHANGELOG entry. Note on local test signal: this branch shows 22 failures when run on the dev host, but unmodified |
|
Verified locally, not by CI. Serialized (
The earlier 22-vs-18 full-suite numbers were parallelism under load average 327: Merging on that evidence. |
Follow-up to #1184, addressing the defects its reviews raised. Three commits; one is a behaviour fix, two are evidence and documentation.
What changed and why
fix(cost): price only the calls no receipt covered (#1175).streamSandboxLeafreadusdEstimatedoff the loop TOTAL whenevercostUsdKnownwas false. That flag is an AND —run-loop.ts:725clears it on any singlellm_callcarryingusdKnown: false,run-loop.ts:833clears it for an iteration that saw no call at all, andrun-loop.ts:1093ANDs it across iterations. So one unproven call relabelled every receipt-backed dollar beside it as a price, andusd - usdEstimated— the quantity the estimate channel exists to make readable — reported$0.00billed against money a provider really had charged. The unproven amount is now summed per call in the loop ledger that already accumulatescostUsd, and carried onIteration.unprovenCostUsdandLoopResult.unprovenCostUsdbeside it.docs(durable): state that the holder liveness answer is advisory.runDirectoryHolderIsLivereads the lock file outside the mutation guardacquireRunDirectoryLockevaluates the identical rule under. The docstring named a supervisor picking up an abandoned directory as a caller without saying such a caller must still acquire, which invites acting on afalsea concurrent acquire has already overtaken. No behaviour change.test(budget): pin the leaked reservation into the durable settle record.leakedReservationswas asserted only on the in-memory result. #1181's symptom was a persistedfailure.jsonreportingnodes: 0with no settle record at all, so the durable artifact is the one that mattered. The new case runs the settled result throughsettleRecordJson— the same serializerresult.jsonis written with — and asserts the leak and its assignment survive.Capability lifted / owner code extended
budget.tsmeterUsageEventfoldsusdEstimatedper cost EVENT (totals.usdEstimated += ev.usdEstimated, only underusdKnown: false), andbridge-executor.ts:986prices per turn rather than per run for the same reason.streamSandboxLeafwas the one settlement path that folded first and classified afterwards; it now reads the per-call sum the loop ledger already had the facts to produce.usdEstimated ⊆ usdclamp isassertValidSpend's own rule (budget.ts:usdEstimated must not exceed usd), not a new one.settleRecordJsonfromsrc/durable/settle-record.ts— the serializer that already refuses any value JSON would misstate — rather than asserting against a hand-built shape.@stableinterfaces.Measured evidence from this session
All of these were run in this worktree, through
createExecutor({ backend: 'sandbox', … })with a scripted box client — the real leaf path, not a stub.llm_call {costUsd: 0.5, usdKnown: false}then a terminaldone {totalCostUsd: 1.5}settled:{usd: 1.5, usdKnown: false, usdEstimated: 1.5}—usd - usdEstimated = $0.00billed. The ledger credits the$1.00residual of the cumulative terminal receipt asusdKnown: true(sandbox-events.tsbuildLlmCall: absentcostProvenanceon a canonical terminal means the provider receipt), so$1.00of stated billing read as zero.{usd: 1.5, usdKnown: false, usdEstimated: 0.5}—usd - usdEstimated = $1.00, matching the receipt. The emitted cost event carries the same partition.runtime.tshunk reverted to its previous form, the new mixed-settlement case fails (1 failed, 8 passed in that file). Before this change the whole suite passed with that hunk reverted — the cloud-child settlement had no coverage at all.spent.usdandspent.usdKnown(budget.tsreserve/reconcile), neverusdEstimated, so a dollar-capped run admits and spends exactly what it did before.usdKnownis untouched on every path.Test command and result
291 test files passed, 3 skipped (294 total); 3758 tests passed, 0 failed, 11 skipped (3769 total); 126.35s.
Also run, all clean:
pnpm run lint(biome, 699 files),pnpm run typecheck(tsc --noEmit+ examples),pnpm run build(tsdown),pnpm run check:api-surface(1978 exports across 13 entry points, record current),pnpm run check:version-bump(2 export change(s) needing a minor bump, paid for by 0.210.0 -> 0.211.0; 0.210.0 is the last published version, so the bump already on main covers these and no new release commit is needed),pnpm run docs:check,pnpm run verify:package,pnpm run check:publish-workflow,pnpm run check:upstream-evidence-workflow.Explicitly NOT in scope
sandbox-session.tsandenvironment-provider.tsare unchanged, deliberately. One review read them as the same defect class. They are not: both declare their whole dollar channel unproven at settlement (usdKnown: falsewhenever a turn ran, independent of what any call reported), and under that declarationusdEstimated == usdis the correct reading — nothing on the channel is claimed as billed.streamSandboxLeafis different because it honours per-call provenance (usdKnown = result.costUsdKnown !== false), so it is the only one of the three that can hold a mixture. Changing the other two would mean revisiting whether a sandbox-reported dollar is a receipt at all, which is the decision fix(accounting): cloud child spend is recorded as an unreported gap while the provider reports exact per-call cost #1175 was about and a separate change.budget.tsreconcileis unchanged. A review asked forassertValidSpend/validateResourceUnitsto be folded into the decided-then-thrown path so an unreadable spend settles its ticket. I implemented it and measured it, and it is the wrong trade: with the change, a run whose child reportedtokens.input: -1returnskind: 'winner'carryingspentTotal.tokens.input: -1withtokensKnown: trueand nospendGaps— an impossible count presented as a measured total on the one arm that promises a complete one. The stated motivation does not hold either: both behaviours charge the full ceiling rather than refunding it, sotokensLeftis identical (measured: 90,000 of 100,000 before and after, and a second spawn asking for the whole remainder is admitted either way). No capacity is stranded; the open ticket is what makes the winner arm refuse the corrupt total, so it is load-bearing. The change is reverted and not proposed.leakedReservations. A review asked for a newleaked-reservationjournal kind alongsideteardown-unconfirmed. The evidence is already durable:result.jsonserializes the wholeSupervisedResult, which is now pinned by test. A second copy inspawn-journal.jsonlwould widen the public journal-event union for evidence that already survives.strategy.ts→supervise-surface.tsstill dropsusdEstimatedwhen projecting a supervisedspentTotalintoAgenticRunResult, so a benchmark surface rebuilds aSpendwhose whole dollar reads as billed. Named in fix(observability): settle a leaked-reservation run, expose run-directory liveness, and name unproven dollars #1184 and still out of scope: it widens a public result type and is not the pursuit settlement fix(accounting): cloud child spend is recorded as an unreported gap while the provider reports exact per-call cost #1175 measured.maindeclares0.211.0; nothing is tagged or published. Unchanged by this PR.One correction to #1184's own description: it claimed
leakedReservationsis declared on both arms ofSupervisedResultand is dead on the winner arm. That is wrong — it is declared only on the two no-winner arms (types.ts:1701lifecycle,types.ts:1748driver-failed). There is no dead field.🤖 Generated with Claude Code