Skip to content

fix(supervise): budget refusal replaces the driver error of a failed attempt under a dollar cap #1176

Description

@drewstone

Summary

When a driver attempt in driveHarnessFromBackend fails and Runtime cannot prove the provider was not dispatched, Runtime meters the attempt as unknown spend.
Under a budget with maxUsd, the budget pool refuses that observation.
The catch around the failure metering then assigns the refusal to failure.
The run reports budget pool: cannot observe unknown dollar cost under a dollar-capped budget instead of the driver's own error.

Source (main at 7722cd4c)

  • supervise.ts#L1287-L1321: a failed attempt without providerDispatch: 'not_started' is metered as unmeteredSpend(0) (usdKnown: false) with telemetry: 'unknown-after-failure'.
  • scope.ts#L1649-L1689: meterInternal catches the observe refusal, appends the metered journal event, then rethrows.
  • budget.ts#L636-L649: observe throws ValidationError for usdKnown: false under a dollar cap before it changes any state. readout() (#L682, #L686) therefore still reports tokensKnown: true and usdKnown: true on a pool that nothing else has tainted.
  • supervise.ts#L1336-L1343: the catch sets failure = error when budget.tokensKnown !== false || (budget.usdCapped && budget.usdKnown !== false).
    The comment says the original failure is replaced only if the unknown-usage marker did not land.
    The marker does land in the journal, but the condition reads the pool readout, which the refusal leaves unchanged, so the condition is true and the driver's error is replaced.

The published 0.208.1 build (dist/supervise-B5kRyFJj.js) has the same logic.

Reproduction

Vitest file in tests/kernel/, run on main at 7722cd4c.
A fake bridge answers POST /v1/chat/completions with the HTTP 501 body cli-bridge returns for a pre-dispatch not_configured refusal.

Test file
import { createServer, type Server } from 'node:http'
import type { AddressInfo } from 'node:net'
import { afterEach, describe, expect, it } from 'vitest'
import { supervise } from '../helpers/runtime-with-test-brain'

const REFUSAL = 'backend opencode cannot replace its harness system prompt'

function fakeBridge(errorBody: Record<string, unknown>): Server {
  return createServer((req, res) => {
    if (req.method === 'GET' && req.url?.startsWith('/v1/capabilities')) {
      res.writeHead(200, { 'content-type': 'application/json' })
      res.end(JSON.stringify({ backend: 'fake' }))
      return
    }
    if (req.method === 'GET' && req.url === '/health') {
      res.writeHead(200, { 'content-type': 'application/json' })
      res.end(JSON.stringify({ status: 'ok' }))
      return
    }
    if (req.method === 'GET' && req.url === '/') {
      res.writeHead(200, { 'content-type': 'application/json' })
      res.end(JSON.stringify({ capabilities: {
        profileMaterialization: 'cli-bridge.profile-materialization.v2',
        usageCostProvenance: 'cli-bridge.usage-cost.v1',
        runtimeAttachments: { mcp: true },
      } }))
      return
    }
    req.resume()
    req.on('end', () => {
      res.writeHead(501, { 'content-type': 'application/json' })
      res.end(JSON.stringify({ error: errorBody }))
    })
  })
}

describe('driver failure under a dollar-capped budget', () => {
  let server: Server | undefined
  afterEach(async () => {
    if (server) await new Promise((resolve) => server?.close(resolve))
    server = undefined
  })

  for (const maxUsd of [undefined, 1]) {
    for (const dispatch of [undefined, 'not_started']) {
      it(`maxUsd=${maxUsd} provider_dispatch=${dispatch}`, async () => {
        server = fakeBridge({
          message: REFUSAL,
          type: 'not_configured',
          ...(dispatch === undefined ? {} : { provider_dispatch: dispatch }),
        })
        await new Promise<void>((resolve) => server?.listen(0, '127.0.0.1', resolve))
        const { port } = server.address() as AddressInfo
        const result = await supervise(
          { name: 'root', harness: 'codex', model: { provider: 'openai', default: 'test' },
            prompt: { systemPrompt: 'Lead.' } },
          'Choose.',
          {
            backend: { backend: 'bridge', bridgeUrl: `http://127.0.0.1:${port}`, bridgeBearer: 't' },
            budget: { maxIterations: 4, maxTokens: 10_000, ...(maxUsd === undefined ? {} : { maxUsd }) },
            driverRetry: { enabled: false },
          },
        )
        const message = result.kind === 'no-winner' ? String((result.error as Error)?.message) : ''
        expect(message).toContain(REFUSAL)
      })
    }
  }
})

Result: 3 passed, 1 failed.

maxUsd provider_dispatch in the 501 body reported cause spentTotal.usdKnown
unset absent BackendTransportError: bridgeExecutor: bridge 501: ... (stopped by retry-disabled) false
unset not_started BackendTransportError: bridgeExecutor: bridge 501: ... (stopped by retry-disabled) true
1 absent ValidationError: budget pool: cannot observe unknown dollar cost under a dollar-capped budget (stopped by terminal-error) false
1 not_started BackendTransportError: bridgeExecutor: bridge 501: ... (stopped by retry-disabled) true

Only the third row loses the driver's error.
The same shape occurred in a real supervised run on 0.208.1: zero model turns, zero tokens, and result.error.message named the budget refusal while the bridge log for that request held the not_configured refusal.

Expected

The driver's error remains the reported cause of driver-failed.
The accounting refusal is attached to it, for example as cause, instead of replacing it.
The unknown-spend metered event and usdKnown: false stay as they are.

The bridge side of this case, where the refusal should carry provider_dispatch: "not_started", is drewstone/cli-bridge#213.
That fix removes this specific trigger. A failure after a real dispatch, with no terminal accounting captured, still reaches the same substitution.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions