Skip to content

fix(supervise): separate completion from failure retries #1260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
drewstone merged 3 commits into from
Sep 16, 2026
Merged

fix(supervise): separate completion from failure retries #1260

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
60978f1
fix(supervise): separate completion from failure retries
drewstone Sep 16, 2026
5f47ff4
Merge remote-tracking branch 'origin/main' into fix/completion-driven…
drewstone Sep 16, 2026
78f2fdc
docs(supervise): describe completion-driven continuation
drewstone Sep 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,16 @@

## 0.232.0

External directors can use `repromptOnUnmet: 'until-complete'` with a completion check and finite positive budget deadline.
Successful continuations reuse the retained conversation without consuming `driverRetry.maxAttempts`.
That limit now counts failed invocations; numeric continuation caps retain their separate meaning.
Completion, explicit stop, cancellation, deadlines, resources, and failure limits still stop the driver.
Provider failures remain visible when recording unknown cost also fails, and that accounting refusal remains terminal.

Local regressions exercise 12 turns in one retained environment, interrupted continuation recovery, and refused further dispatch under unknown cost.
These checks do not establish live multi-hour fleet reliability.
Related system issue: SYS-011 — outcome: mitigates — proof: retained-owner recovery and driver-retry regression tests.

A child that DROPS now keeps its transcript, the receipt that says so has moved to where an
unavailable trace cannot hide it, and the thing itself is finally called what it is.

Expand Down
8 changes: 4 additions & 4 deletions api-surface.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -890,10 +890,10 @@
"DriverContractState": "type a287e992fd8a",
"DriverProgressMark": "type cb7863a2e835",
"DriverReentry": "type 856da6132418",
"DriverRepromptPolicy": "type 0761759617c8",
"DriverRepromptPolicy": "type 0f79f31dc99e",
"DriverRepromptRefusal": "type 2b7a1302459a",
"DriverRetryPolicy": "type b5607c21ee53",
"DriverUnmetContractContext": "type 9e6627e11f90",
"DriverUnmetContractContext": "type 2c4d72703ac4",
"DriverUnmetContractDecision": "type f62a992629d1",
"EVIDENCE_MAX_CHARS": "value 60366a39f55c",
"EdgeDeliveryOutcome": "type ede8c2ce5f1a",
Expand Down Expand Up @@ -1257,15 +1257,15 @@
"StructuralRolloutPolicy": "type d372912ef050",
"StructuralRolloutResult": "type 9248b72cae04",
"SuperviseDispatchOptions": "type 55e71c9c1e6b",
"SuperviseOptions": "type 7280b2a63be3",
"SuperviseOptions": "type 1047255658db",
"SuperviseOptionsForDispatch": "type 7d89526c1040",
"SuperviseRegistry": "type 4fd60c297f74",
"SuperviseRegistryTable": "type cc1468cd50c1",
"SuperviseSurfaceOptions": "type 8a7daaf98896",
"SuperviseSurfaceResult": "type 2c378dbc3193",
"SupervisedResult": "type 924803bb65bd",
"Supervisor": "type 7d9aff9cd744",
"SupervisorAgentDeps": "type b2683b00f9e6",
"SupervisorAgentDeps": "type 763549b3d5d6",
"SupervisorCleanupReceipt": "type d862eb60266d",
"SupervisorFinalizer": "type f8628e65536f",
"SupervisorNodeContext": "type b8e545bbb355",
Expand Down
5 changes: 4 additions & 1 deletion docs/agent-managed-compute/reliability.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,10 @@ Recovery validates the actual retained bytes before accepting another append.
The index is not a second durable record or a cross-process ownership fence.
Observer records detach their inputs before queued I/O, so later hook mutations cannot rewrite the evidence being saved.
Completed director invocations reset the consecutive transport-failure counter even when the pursuit remains incomplete.
The explicit total-attempt, deadline, cancellation, and resource bounds still apply.
The failure-attempt, deadline, cancellation, and resource bounds still apply.
Successful incomplete invocations do not consume `driverRetry.maxAttempts`; only failed invocations consume that allowance.
Use `repromptOnUnmet: 'until-complete'` with a completion check and finite positive budget deadline to omit the continuation count cap.
Numeric continuation caps retain their meaning, and zero still disables continuation.

The file run lock protects one local coordinator.
It does not fence provider mutations from a partitioned coordinator on another machine.
Expand Down
9 changes: 6 additions & 3 deletions docs/api/durable.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1749,7 +1749,7 @@ Per-re-entry record for every retried worker spawn — what makes a saturated ex

##### repromptOnUnmet?

> `readonly` `optional` **repromptOnUnmet?**: `number`
> `readonly` `optional` **repromptOnUnmet?**: `number` \| `"until-complete"`

How many times an EXTERNAL-harness driver that RETURNED with `deliverable` still unmet is
re-entered on the SAME live session with the unmet items.
Expand All @@ -1760,12 +1760,15 @@ runs): 376 of 376 winning runs ended on the driver's own completion, and the com
could only LABEL an undelivered result `valid:false`, never send the driver back for it.

A re-prompt is the retry path, not a second loop: same scope, same coordination server, same
live children, and the same budget, deadline, abort, and `driverRetry.maxAttempts` bounds. A
live children, and the same budget, deadline, and abort bounds. Successful continuations do
not consume `driverRetry.maxAttempts`, which counts failed invocations only. A
run the coordination server already stopped is never re-prompted — that stop was a decision.

Requires `deliverable`, and applies to every external manager with a completion check. A
recursive manager receives the check selected for its exact assignment. Refused for a
router-brained manager, which runs its turn loop in process. Omit/`0` = never.
Use `'until-complete'` with a finite positive budget deadline to remove the continuation cap.
Completion, explicit stop, cancellation, resource limits, and failure limits still stop work.

###### Inherited from

Expand All @@ -1776,7 +1779,7 @@ router-brained manager, which runs its turn loop in process. Omit/`0` = never.
> `readonly` `optional` **onUnmetContract?**: [`OnUnmetContract`](runtime.md#onunmetcontract)

Compose the re-entry instruction for an unmet contract, or return `'stop'` to end the run.
Requires `repromptOnUnmet >= 1`. Omit = Runtime's own instruction, which names what the run
Requires positive `repromptOnUnmet` or `'until-complete'`. Omit = Runtime's instruction, which names what the run
owes and reports how many workers passed the check.

###### Inherited from
Expand Down
43 changes: 25 additions & 18 deletions docs/api/runtime.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13906,10 +13906,9 @@ Consecutive failures that changed NOTHING (no metered spend, no settlement, no s

> `readonly` `optional` **maxAttempts?**: `number`

Absolute ceiling on attempts, regardless of progress. Default 8. The barren counter alone
cannot bound a driver that crashes every turn AFTER metering a little: each attempt looks like
progress, so without this backstop such a run would retry until it had eaten the entire
envelope. A caller who wants budget-only bounding sets this high deliberately.
Ceiling on failed invocations across this driver run, regardless of progress. Default 8,
minimum 1. Successful continuations do not consume this allowance or reset it.
This bounds repeated crashes that each make enough progress to reset the barren streak.

##### initialBackoffMs?

Expand Down Expand Up @@ -14083,7 +14082,7 @@ How many re-prompts this run has already issued.

##### maxReprompts

> `readonly` **maxReprompts**: `number`
> `readonly` **maxReprompts**: `number` \| `"until-complete"`

##### progress

Expand Down Expand Up @@ -14112,10 +14111,11 @@ How a completed-but-undelivered drive is re-entered. Absent = the historical beh

##### maxReprompts

> `readonly` **maxReprompts**: `number`
> `readonly` **maxReprompts**: `number` \| `"until-complete"`

How many times one run may re-enter its live session with the unmet items. `0` = never. Each
re-prompt also consumes an attempt, so `maxAttempts` bounds it too.
How many times one run may re-enter its live session with the unmet items. `0` = never.
`'until-complete'` removes the count cap and requires a finite positive scope deadline.
Budget, cancellation, explicit stop, and failure retry limits still apply.

##### onUnmetContract?

Expand Down Expand Up @@ -14955,7 +14955,7 @@ Per-re-entry record for every retried worker spawn — what makes a saturated ex

##### repromptOnUnmet?

> `readonly` `optional` **repromptOnUnmet?**: `number`
> `readonly` `optional` **repromptOnUnmet?**: `number` \| `"until-complete"`

How many times an EXTERNAL-harness driver that RETURNED with `deliverable` still unmet is
re-entered on the SAME live session with the unmet items.
Expand All @@ -14966,12 +14966,15 @@ runs): 376 of 376 winning runs ended on the driver's own completion, and the com
could only LABEL an undelivered result `valid:false`, never send the driver back for it.

A re-prompt is the retry path, not a second loop: same scope, same coordination server, same
live children, and the same budget, deadline, abort, and `driverRetry.maxAttempts` bounds. A
live children, and the same budget, deadline, and abort bounds. Successful continuations do
not consume `driverRetry.maxAttempts`, which counts failed invocations only. A
run the coordination server already stopped is never re-prompted — that stop was a decision.

Requires `deliverable`, and applies to every external manager with a completion check. A
recursive manager receives the check selected for its exact assignment. Refused for a
router-brained manager, which runs its turn loop in process. Omit/`0` = never.
Use `'until-complete'` with a finite positive budget deadline to remove the continuation cap.
Completion, explicit stop, cancellation, resource limits, and failure limits still stop work.

###### Inherited from

Expand All @@ -14982,7 +14985,7 @@ router-brained manager, which runs its turn loop in process. Omit/`0` = never.
> `readonly` `optional` **onUnmetContract?**: [`OnUnmetContract`](#onunmetcontract)

Compose the re-entry instruction for an unmet contract, or return `'stop'` to end the run.
Requires `repromptOnUnmet >= 1`. Omit = Runtime's own instruction, which names what the run
Requires positive `repromptOnUnmet` or `'until-complete'`. Omit = Runtime's instruction, which names what the run
owes and reports how many workers passed the check.

###### Inherited from
Expand Down Expand Up @@ -19118,7 +19121,7 @@ Per-re-entry record for every retried worker spawn — what makes a saturated ex

##### repromptOnUnmet?

> `readonly` `optional` **repromptOnUnmet?**: `number`
> `readonly` `optional` **repromptOnUnmet?**: `number` \| `"until-complete"`

How many times an EXTERNAL-harness driver that RETURNED with `deliverable` still unmet is
re-entered on the SAME live session with the unmet items.
Expand All @@ -19129,19 +19132,22 @@ runs): 376 of 376 winning runs ended on the driver's own completion, and the com
could only LABEL an undelivered result `valid:false`, never send the driver back for it.

A re-prompt is the retry path, not a second loop: same scope, same coordination server, same
live children, and the same budget, deadline, abort, and `driverRetry.maxAttempts` bounds. A
live children, and the same budget, deadline, and abort bounds. Successful continuations do
not consume `driverRetry.maxAttempts`, which counts failed invocations only. A
run the coordination server already stopped is never re-prompted — that stop was a decision.

Requires `deliverable`, and applies to every external manager with a completion check. A
recursive manager receives the check selected for its exact assignment. Refused for a
router-brained manager, which runs its turn loop in process. Omit/`0` = never.
Use `'until-complete'` with a finite positive budget deadline to remove the continuation cap.
Completion, explicit stop, cancellation, resource limits, and failure limits still stop work.

##### onUnmetContract?

> `readonly` `optional` **onUnmetContract?**: [`OnUnmetContract`](#onunmetcontract)

Compose the re-entry instruction for an unmet contract, or return `'stop'` to end the run.
Requires `repromptOnUnmet >= 1`. Omit = Runtime's own instruction, which names what the run
Requires positive `repromptOnUnmet` or `'until-complete'`. Omit = Runtime's instruction, which names what the run
owes and reports how many workers passed the check.

##### childSettleGraceMs?
Expand Down Expand Up @@ -20217,22 +20223,23 @@ Per-attempt record for the external driver — how an operator sees "failed afte

##### repromptOnUnmet?

> `readonly` `optional` **repromptOnUnmet?**: `number`
> `readonly` `optional` **repromptOnUnmet?**: `number` \| `"until-complete"`

How many times an EXTERNAL driver that RETURNED with `deliverable` still unmet is re-entered
on the SAME live session with the unmet items. The harness owns its own turn loop, so it can
end while the run has delivered nothing — 376 of 376 winning discovery-lab runs (2026-09-01)
ended on the driver's own completion, and the completion gate could only label that result,
never change it. A re-prompt reuses the retry path: same scope, same coordination server, same
live children, same budget/deadline/abort/attempt bounds. Requires `deliverable`; refused for
a router-brained supervisor, which runs its loop in process. Omit/`0` = never re-prompt.
live children, same budget/deadline/abort bounds. Successful turns do not consume failure
retries. Use `'until-complete'` with a finite positive scope deadline to omit the count cap.
Requires `deliverable`; refused for a router-brained supervisor. Omit/`0` = never re-prompt.

##### onUnmetContract?

> `readonly` `optional` **onUnmetContract?**: [`OnUnmetContract`](#onunmetcontract)

Compose the re-entry instruction for an unmet contract, or return `'stop'` to end the run.
Requires `repromptOnUnmet >= 1`. Omit = Runtime's own instruction.
Requires positive `repromptOnUnmet` or `'until-complete'`. Omit = Runtime's own instruction.

##### nodeContext?

Expand Down
27 changes: 17 additions & 10 deletions docs/api/testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -865,7 +865,7 @@ Per-re-entry record for every retried worker spawn — what makes a saturated ex

##### repromptOnUnmet?

> `readonly` `optional` **repromptOnUnmet?**: `number`
> `readonly` `optional` **repromptOnUnmet?**: `number` \| `"until-complete"`

How many times an EXTERNAL-harness driver that RETURNED with `deliverable` still unmet is
re-entered on the SAME live session with the unmet items.
Expand All @@ -876,12 +876,15 @@ runs): 376 of 376 winning runs ended on the driver's own completion, and the com
could only LABEL an undelivered result `valid:false`, never send the driver back for it.

A re-prompt is the retry path, not a second loop: same scope, same coordination server, same
live children, and the same budget, deadline, abort, and `driverRetry.maxAttempts` bounds. A
live children, and the same budget, deadline, and abort bounds. Successful continuations do
not consume `driverRetry.maxAttempts`, which counts failed invocations only. A
run the coordination server already stopped is never re-prompted — that stop was a decision.

Requires `deliverable`, and applies to every external manager with a completion check. A
recursive manager receives the check selected for its exact assignment. Refused for a
router-brained manager, which runs its turn loop in process. Omit/`0` = never.
Use `'until-complete'` with a finite positive budget deadline to remove the continuation cap.
Completion, explicit stop, cancellation, resource limits, and failure limits still stop work.

###### Inherited from

Expand All @@ -892,7 +895,7 @@ router-brained manager, which runs its turn loop in process. Omit/`0` = never.
> `readonly` `optional` **onUnmetContract?**: [`OnUnmetContract`](runtime.md#onunmetcontract)

Compose the re-entry instruction for an unmet contract, or return `'stop'` to end the run.
Requires `repromptOnUnmet >= 1`. Omit = Runtime's own instruction, which names what the run
Requires positive `repromptOnUnmet` or `'until-complete'`. Omit = Runtime's instruction, which names what the run
owes and reports how many workers passed the check.

###### Inherited from
Expand Down Expand Up @@ -1707,7 +1710,7 @@ Per-re-entry record for every retried worker spawn — what makes a saturated ex

##### repromptOnUnmet?

> `readonly` `optional` **repromptOnUnmet?**: `number`
> `readonly` `optional` **repromptOnUnmet?**: `number` \| `"until-complete"`

How many times an EXTERNAL-harness driver that RETURNED with `deliverable` still unmet is
re-entered on the SAME live session with the unmet items.
Expand All @@ -1718,12 +1721,15 @@ runs): 376 of 376 winning runs ended on the driver's own completion, and the com
could only LABEL an undelivered result `valid:false`, never send the driver back for it.

A re-prompt is the retry path, not a second loop: same scope, same coordination server, same
live children, and the same budget, deadline, abort, and `driverRetry.maxAttempts` bounds. A
live children, and the same budget, deadline, and abort bounds. Successful continuations do
not consume `driverRetry.maxAttempts`, which counts failed invocations only. A
run the coordination server already stopped is never re-prompted — that stop was a decision.

Requires `deliverable`, and applies to every external manager with a completion check. A
recursive manager receives the check selected for its exact assignment. Refused for a
router-brained manager, which runs its turn loop in process. Omit/`0` = never.
Use `'until-complete'` with a finite positive budget deadline to remove the continuation cap.
Completion, explicit stop, cancellation, resource limits, and failure limits still stop work.

###### Inherited from

Expand All @@ -1734,7 +1740,7 @@ router-brained manager, which runs its turn loop in process. Omit/`0` = never.
> `readonly` `optional` **onUnmetContract?**: [`OnUnmetContract`](runtime.md#onunmetcontract)

Compose the re-entry instruction for an unmet contract, or return `'stop'` to end the run.
Requires `repromptOnUnmet >= 1`. Omit = Runtime's own instruction, which names what the run
Requires positive `repromptOnUnmet` or `'until-complete'`. Omit = Runtime's instruction, which names what the run
owes and reports how many workers passed the check.

###### Inherited from
Expand Down Expand Up @@ -2382,15 +2388,16 @@ Per-attempt record for the external driver — how an operator sees "failed afte

##### repromptOnUnmet?

> `readonly` `optional` **repromptOnUnmet?**: `number`
> `readonly` `optional` **repromptOnUnmet?**: `number` \| `"until-complete"`

How many times an EXTERNAL driver that RETURNED with `deliverable` still unmet is re-entered
on the SAME live session with the unmet items. The harness owns its own turn loop, so it can
end while the run has delivered nothing — 376 of 376 winning discovery-lab runs (2026-09-01)
ended on the driver's own completion, and the completion gate could only label that result,
never change it. A re-prompt reuses the retry path: same scope, same coordination server, same
live children, same budget/deadline/abort/attempt bounds. Requires `deliverable`; refused for
a router-brained supervisor, which runs its loop in process. Omit/`0` = never re-prompt.
live children, same budget/deadline/abort bounds. Successful turns do not consume failure
retries. Use `'until-complete'` with a finite positive scope deadline to omit the count cap.
Requires `deliverable`; refused for a router-brained supervisor. Omit/`0` = never re-prompt.

###### Inherited from

Expand All @@ -2401,7 +2408,7 @@ How many times an EXTERNAL driver that RETURNED with `deliverable` still unmet i
> `readonly` `optional` **onUnmetContract?**: [`OnUnmetContract`](runtime.md#onunmetcontract)

Compose the re-entry instruction for an unmet contract, or return `'stop'` to end the run.
Requires `repromptOnUnmet >= 1`. Omit = Runtime's own instruction.
Requires positive `repromptOnUnmet` or `'until-complete'`. Omit = Runtime's own instruction.

###### Inherited from

Expand Down
Loading
Loading