Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion api-surface.json
Original file line number Diff line number Diff line change
Expand Up @@ -1585,7 +1585,7 @@
"runBenchmark": "value a00b82e95062",
"runCancelRequestFile": "value 229aecaab891",
"runCancellationFile": "value 229aecaab891",
"runFinalizer": "value 34d3218fb8ef",
"runFinalizer": "value fc5202156e7a",
"runGraph": "value 7d72ff33c325",
"runInWorkspace": "value 624b01e4ba38",
"runIsolatedCheck": "value ebb5d56a353b",
Expand Down
10 changes: 9 additions & 1 deletion docs/api/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -34436,7 +34436,11 @@ readonly `T`[]
Run a finalizer over a settled-worker ledger under the delivered-only invariant: filter the
ledger to structurally delivered children, materialize their outputs, and hand the finalizer a
blob reader that throws on any ref outside that set. This is the one call site both driver arms
(the in-process tool-loop and the MCP-mounted harness) finalize through.
(the in-process tool-loop and the MCP-mounted harness) finalize through. When a parent declares
a deliverable, its candidate must also pass that check: children may have narrower assignments.
The default selects the highest-scoring child that passes the parent's check; custom finalizers
assemble their candidate before the check. A rejection leaves the parent incomplete; a thrown
oracle surfaces a validation error. Neither changes child validity.

#### Parameters

Expand All @@ -34462,6 +34466,10 @@ readonly [`FinalizerSettled`](#finalizersettled)[]

`Readonly`\<\{ `resources?`: `Readonly`\<`Record`\<`string`, \{ `unit`: `string`; `limit`: `number`; `remaining`: `number`; `reserved`: `number`; `committed`: `number`; `known`: `boolean`; \}\>\>; `tokensLeft`: `number`; `tokensKnown`: `boolean`; `cacheBreakdownKnown`: `boolean`; `usdLeft`: `number`; `usdCapped`: `boolean`; `usdKnown`: `boolean`; `iterationsLeft`: `number`; `deadlineMs`: `number`; `reservedTokens`: `number`; \}\>

###### deliverable?

[`DeliverableSpec`](#deliverablespec)\<`unknown`\>

#### Returns

`Promise`\<`unknown`\>
Expand Down
8 changes: 8 additions & 0 deletions docs/canonical-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,14 @@ A general "loop" primitive is the single most common modelling error in this rep

**The trap** is a single grammar (`defineLoop`, a `runXxxLoop`) spanning all of the above: there can't be one, because some are code and one is a model deciding. No new loop primitive lands without a tiny executable proof, **over real agents**, of the exact substrate join it claims to simplify.

A child can satisfy its own assignment while its parent's objective remains incomplete.
When the parent declares a `deliverable`, its finalizer's candidate must pass that check before the parent completes.
The default selects the highest-scoring child that also passes the parent's check.
Custom finalizers assemble their candidate before the check.
Child validity and delivery counts remain unchanged.
An external director can then use `repromptOnUnmet` to continue from a rejected candidate within its existing resource limits.
A thrown parent check reports a validation error through the existing driver failure record.

| I want to… | Use (import) | Do NOT build |
|---|---|---|
| Run one product chat turn with streamed events, ordered persistence hooks, and stable execution/turn identity | `handleChatTurn(...)` + `deriveExecutionId(...)`: `/durable`; pass the derived id as both `executionId` and `turnId` on initial dispatch | importing the broad package entry from an edge worker, treating `executionId` alone as dispatch idempotency, or rebuilding framing and persistence ordering in the product |
Expand Down
1 change: 1 addition & 0 deletions src/runtime/supervise/coordination-driver.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1289,6 +1289,7 @@ export function driverAgent(opts: DriverAgentOptions): Agent<unknown, unknown> {
blobs: opts.blobs,
tree: runTree(scope),
budget: scope.budget,
...(opts.deliverable ? { deliverable: opts.deliverable } : {}),
})
},
}
Expand Down
30 changes: 28 additions & 2 deletions src/runtime/supervise/finalizer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
*/

import { ValidationError } from '../../errors'
import type { DeliverableSpec } from './completion-gate'
import type { ResultBlobStore, Scope, TreeView } from './types'

/** One settled worker as the finalizer sees it — the ledger row (structural fields only). */
Expand Down Expand Up @@ -114,7 +115,11 @@ export const collectDelivered: SupervisorFinalizer = (ctx) => {
* Run a finalizer over a settled-worker ledger under the delivered-only invariant: filter the
* ledger to structurally delivered children, materialize their outputs, and hand the finalizer a
* blob reader that throws on any ref outside that set. This is the one call site both driver arms
* (the in-process tool-loop and the MCP-mounted harness) finalize through.
* (the in-process tool-loop and the MCP-mounted harness) finalize through. When a parent declares
* a deliverable, its candidate must also pass that check: children may have narrower assignments.
* The default selects the highest-scoring child that passes the parent's check; custom finalizers
* assemble their candidate before the check. A rejection leaves the parent incomplete; a thrown
* oracle surfaces a validation error. Neither changes child validity.
*/
export async function runFinalizer(
finalizer: SupervisorFinalizer,
Expand All @@ -123,6 +128,7 @@ export async function runFinalizer(
readonly blobs: ResultBlobStore
readonly tree: TreeView
readonly budget: Scope<unknown>['budget']
readonly deliverable?: DeliverableSpec
},
): Promise<unknown | undefined> {
const deliveredRows = args.settled.filter((w) => w.status === 'done' && w.valid === true)
Expand All @@ -146,11 +152,31 @@ export async function runFinalizer(
return args.blobs.get(outRef)
},
}
return finalizer({
const accepted = async (candidate: unknown): Promise<boolean> => {
if (candidate === undefined) return false
if (args.deliverable === undefined) return true
try {
return (await args.deliverable.check(candidate)) === true
} catch (error) {
throw new ValidationError(
`finalizer: parent completion check threw: ${error instanceof Error ? error.message : String(error)}`,
)
}
}
// A high-scoring partial assignment cannot displace a complete portfolio member. Aggregating
// finalizers still need every child-valid component, so their inputs must not be filtered here.
if (finalizer === bestDelivered && args.deliverable !== undefined) {
for (const output of [...delivered].sort((a, b) => (b.score ?? 0) - (a.score ?? 0))) {
if (await accepted(output.out)) return output.out
}
return undefined
}
const candidate = await finalizer({
delivered,
allSettled: args.settled,
tree: args.tree,
blobs: guardedBlobs,
budget: args.budget,
})
return (await accepted(candidate)) ? candidate : undefined
}
32 changes: 21 additions & 11 deletions src/runtime/supervise/supervisor-agent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -982,9 +982,18 @@ function buildSupervisorAgent(
const baseTokensLeft = scope.budget.tokensLeft
const contractDeclared = deps.deliverable !== undefined
const maxReprompts = deps.repromptOnUnmet ?? 0
let candidate: unknown
const finalize = () =>
runFinalizer(deps.finalizer ?? bestDelivered, {
settled: mcp.settled(),
blobs: deps.blobs,
tree: runTree(scope),
budget: scope.budget,
...(deps.deliverable ? { deliverable: deps.deliverable } : {}),
})
const readProgress = (): DriverProgressMark => {
const settled = mcp.settled()
// The same delivered-only rule the finalizer applies: settled `done` AND check-passed.
// Child delivery is progress, but its check may cover only the child's assignment.
const deliveredCount = settled.filter(
(w) => w.status === 'done' && w.valid === true,
).length
Expand All @@ -996,13 +1005,14 @@ function buildSupervisorAgent(
deliveredCount,
contract: !contractDeclared
? 'none'
: submitted || deliveredCount > 0
: submitted || candidate !== undefined
? 'met'
: 'unmet',
}
}
await runDriverWithRetry({
drive: async (attempt, reentry) => {
candidate = undefined
if (deps.controlDir !== undefined && deps.abortRun !== undefined) {
applyRunCancellation(deps.controlDir, deps.abortRun, () => new Date().toISOString())
}
Expand All @@ -1029,6 +1039,12 @@ function buildSupervisorAgent(
// an accepted submission the backend error propagates into the retry decision.
if (!mcp.submittedResult() && !mcp.isStopped()) throw error
}
// Decide this parent's completion before the retry loop reads progress. Cache the
// checked candidate so neither the finalizer nor its oracle runs twice on return.
if (contractDeclared && !mcp.submittedResult()) {
await mcp.drainResolved()
candidate = await finalize()
}
},
progress: readProgress,
budget: () => scope.budget,
Expand Down Expand Up @@ -1059,9 +1075,8 @@ function buildSupervisorAgent(
: {}),
...(deps.onDriverAttempt ? { onAttempt: deps.onDriverAttempt } : {}),
})
// Drain settled-but-unpulled children first — a gate-verified delivery the harness never
// awaited must still reach the finalize ledger.
await mcp.drainResolved()
// Without a parent oracle, preserve the single finalization after the driver finishes.
if (!contractDeclared) await mcp.drainResolved()
// Direct work is eligible only through `submit_result`, after the injected independent
// check passes. Raw harness prose remains ineligible.
const submitted = mcp.submittedResult()
Expand All @@ -1071,12 +1086,7 @@ function buildSupervisorAgent(
}
// The deliverable comes from the finalizer seam over DELIVERED children only — never the
// harness's own output (Foreman 0/18). Default keep-best.
return await runFinalizer(deps.finalizer ?? bestDelivered, {
settled: mcp.settled(),
blobs: deps.blobs,
tree: runTree(scope),
budget: scope.budget,
})
return contractDeclared ? candidate : await finalize()
} finally {
await mcp.close()
}
Expand Down
20 changes: 18 additions & 2 deletions tests/kernel/supervise-convenience.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,7 @@ describe('supervise — the one-call convenience (defaults blobs/perWorker/journ

it('uses a child-specific completion check when a managed child submits its own result', async () => {
let childCheckCalls = 0
const journal = new InMemorySpawnJournal()
const child = testAgentProfile('specialist', {
harness: 'opencode',
tools: runtimeToolDeclarations('submit_result'),
Expand Down Expand Up @@ -288,6 +289,8 @@ describe('supervise — the one-call convenience (defaults blobs/perWorker/journ
perWorker: { maxIterations: 4, maxTokens: 10_000 },
makeLeafAgent: () => deliveringLeaf('unused', {}),
driveHarness,
journal,
runId: 'parent-contract',
// The run-wide check deliberately rejects the child's output. A profile-managed child
// must instead receive the check selected for its exact authorized assignment.
deliverable: { check: () => false },
Expand All @@ -307,9 +310,22 @@ describe('supervise — the one-call convenience (defaults blobs/perWorker/journ
},
)

expect(result.kind).toBe('winner')
if (result.kind === 'winner') expect(result.out).toEqual({ answer: 42 })
// The child's own check is what judged the child: consulted exactly once, and it passed.
expect(childCheckCalls).toBe(1)
expect(await journal.loadTree('parent-contract')).toEqual(
expect.arrayContaining([
expect.objectContaining({
kind: 'settled',
id: 'parent-contract:s0',
status: 'done',
verdict: expect.objectContaining({ valid: true }),
}),
]),
)
// A child that satisfies its narrower assignment does not complete the parent. The run-wide
// check rejects the child's output, no continuation is configured, so the parent's contract
// stays unmet and the partial component is not promoted to the winner.
expect(result.kind).toBe('no-winner')
})

it('runDir makes the run durable and resumable; unset stays in-memory', async () => {
Expand Down
155 changes: 155 additions & 0 deletions tests/kernel/supervisor-agent.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1515,6 +1515,161 @@ describe('supervisorAgent — coordination bind + prompt hoisting on the harness
expect(tasks).toHaveLength(1)
})

it('EXTERNAL arm: a valid partial child leaves the parent contract unmet and re-enters its director', async () => {
const blobs = new InMemoryResultBlobStore()
const journal = new InMemorySpawnJournal()
const checked: unknown[] = []
const progress: unknown[] = []
let drives = 0
let finalizations = 0
const partial = { component: 'database', ready: true }
const complete = { answer: 42 }
const root = supervisorAgent(
testAgentProfile('sup', {
harness: 'pi',
tools: runtimeToolDeclarations('spawn_worker', 'await_event', 'submit_result'),
}),
{
blobs,
makeWorkerAgent: () => deliveringLeaf('component', partial),
perWorker,
driveHarness: async ({ coordinationMcpUrl }) => {
drives += 1
if (drives === 1) {
await jsonRpc(coordinationMcpUrl, 'tools/call', {
name: 'spawn_worker',
arguments: { profile: testAgentProfile('component'), task: 'build the database' },
})
await jsonRpc(coordinationMcpUrl, 'tools/call', {
name: 'await_event',
arguments: {},
})
return
}
await jsonRpc(coordinationMcpUrl, 'tools/call', {
name: 'submit_result',
arguments: { result: complete },
})
},
finalizer: ({ delivered }) => {
finalizations += 1
expect(delivered).toHaveLength(1)
return delivered[0]?.out
},
deliverable: {
check: (out) => {
checked.push(out)
return JSON.stringify(out) === JSON.stringify(complete)
},
},
repromptOnUnmet: 1,
onUnmetContract: (context) => {
progress.push(context.progress)
return { steer: 'Integrate the database into the complete product.' }
},
},
)

const result = await runSupervisor(root, blobs, journal)
expect(result.kind).toBe('winner')
if (result.kind === 'winner') expect(result.out).toEqual(complete)
expect(drives).toBe(2)
expect(finalizations).toBe(1)
expect(checked).toEqual([partial, complete])
expect(progress).toEqual([expect.objectContaining({ deliveredCount: 1, contract: 'unmet' })])
expect(await journal.loadTree('sup')).toEqual(
expect.arrayContaining([
expect.objectContaining({
kind: 'settled',
id: 'sup:s0',
status: 'done',
verdict: expect.objectContaining({ valid: true }),
}),
]),
)
})

it.each([
{ name: 'partial', out: { component: 'database' }, valid: false },
{ name: 'complete', out: { answer: 42 }, valid: true },
{ name: 'throwing check', out: { throw: true }, valid: false },
{ name: 'aggregate', out: { component: 'database' }, aggregate: true, valid: true },
])('both arms check the parent contract for a $name child candidate', async (scenario) => {
for (const harness of ['pi', 'cli-base'] as const) {
const blobs = new InMemoryResultBlobStore()
const journal = new InMemorySpawnJournal()
let checks = 0
let finalizations = 0
const root = supervisorAgent(
testAgentProfile('sup', {
harness,
tools: runtimeToolDeclarations('spawn_worker', 'await_event'),
}),
{
blobs,
makeWorkerAgent: () => deliveringLeaf('component', scenario.out),
perWorker,
...(harness === 'pi'
? {
driveHarness: async ({ coordinationMcpUrl }: Parameters<DriveHarness>[0]) => {
await jsonRpc(coordinationMcpUrl, 'tools/call', {
name: 'spawn_worker',
arguments: { profile: testAgentProfile('component'), task: 'build component' },
})
await jsonRpc(coordinationMcpUrl, 'tools/call', {
name: 'await_event',
arguments: {},
})
},
}
: {
brain: scriptedBrain([
{
toolCalls: [
{
name: 'spawn_worker',
arguments: {
profile: testAgentProfile('component'),
task: 'build component',
},
},
],
},
{ toolCalls: [{ name: 'await_event', arguments: {} }] },
{ content: 'done' },
]),
}),
finalizer: ({ delivered }) => {
finalizations += 1
return scenario.aggregate ? { answer: 42 } : delivered[0]?.out
},
deliverable: {
check: (out) => {
checks += 1
if (scenario.name === 'throwing check') throw new Error('oracle unavailable')
return JSON.stringify(out) === JSON.stringify({ answer: 42 })
},
},
},
)

const result = await runSupervisor(root, blobs, journal)
expect(result.kind, `${harness}: ${scenario.name}`).toBe(
scenario.valid ? 'winner' : 'no-winner',
)
if (scenario.name === 'throwing check' && result.kind === 'no-winner') {
expect(result.reason).toBe('driver-failed')
if (result.reason === 'driver-failed') {
expect(result.error.message).toContain(
'parent completion check threw: oracle unavailable',
)
}
}
expect(checks).toBe(1)
expect(finalizations).toBe(1)
}
})

it('EXTERNAL arm: a run the coordination server STOPPED is never re-prompted', async () => {
// The driver called `stop`. That was a decision, and Runtime refuses the re-prompt before the
// product hook is consulted, so no hook can talk the run past its own stop.
Expand Down
Loading
Loading