Preflight error names a model the route ignores; a spawned node is absent from the settled record

[drewstone]

What happened

A supervised run with two children ran 3h17m (03:51:30Z to 07:08:37Z). Both children finished their work and wrote their results. The run then settled:

kind:   "no-winner"
reason: "driver-failed"
error:  DriverAttemptsExhaustedError: supervisor driver failed after 4 attempt(s)
        — stopped by no-progress; last cause: BackendTransportError: bridgeExecutor:
        bridge http://127.0.0.1:8921 did not answer for model
        "opencode/kimi-for-coding/k3": bridge GET
        /v1/capabilities?model=opencode%2Fkimi-for-coding%2Fk3 timed out
        (classified transient)

Why the cause as written is misleading

bridgeModelRouteRefusal is a preflight whose stated purpose is good: catch a routing error before a child is spawned and metered. It probes GET /v1/capabilities?model=<wireModel> with BRIDGE_RUN_STATE_TIMEOUT_MS.

Two facts make the reported cause wrong as an explanation:

1. BRIDGE_RUN_STATE_TIMEOUT_MS = 2_000. Two seconds.
2. /v1/capabilities is a static constant. In cli-bridge src/server.ts the handler returns a fixed object listing schema names, backend names and endpoint paths. It never consults the model, never touches a backend, never spawns anything. The ?model= query parameter does not change the response.

So a timeout on that route cannot mean the bridge lacked the model, could not route it, or was unhealthy. It can only mean the request did not complete a round trip within 2s. The error message names a model as though model availability were in question, which sends a reader to the wrong subsystem.

Independent check, after the run settled: the same bridge answered a capabilities probe in 4.5 ms (HTTP 401 without a bearer, so the listener is live and prompt). The bridge process never died and was never restarted.

The actual condition

Host load average was 66-76 for the hour around the failure, driven by the run's own children doing exact combinatorics plus unrelated local work. A 2s budget for an HTTP round trip on a host the runtime is itself deliberately saturating is not a safe gate.

The bridge's own log has no entry at 07:08Z at all (last write 04:42:41Z), so there is no harness-side record of the probe arriving. That absence is itself the evidence: the request did not reach a handler that logs.

The composition defect, which is the more serious half

The probe classifies its failure retryable: true and the runtime classifies it transient. It is then retried 4 times, makes no progress, and the correct transient classification is converted into a terminal no-winner for the whole run.

That is the wrong terminal state for this run, on two counts:

• Both children had already settled. The tree records s0: done and the second child had written its final artifacts, posted its verdict, written the joint decision record, and called coordination_stop.
• The run's actual product was complete and durable on disk before the probe ever ran.

A retryable transient that exhausts should not be able to relabel a finished run as having produced no winner. At minimum the settled record should distinguish "the work failed" from "the driver could not confirm it".

Second defect in the same record: a node that ran is absent from the tree

The observer journal records a third node:

04:15:04Z  agent.spawn   collab-decoder-20260911:s1:s0:spawn
04:30:58Z  agent.child   collab-decoder-20260911:s1:s0:settled

15m54s of life. The settled result.json reports tree.nodes with 2 entries. s1:s0 is in neither the tree nor spendGaps, and emitted no agent.turn meter.

Its spend is therefore not merely unreported, it is unnameable from the settled record: nothing in result.json says the node existed. The parent observed real usage for it from the trace (50,631 input / 16,059 output tokens, 44-span trace sha256:796e55b5…903e) while the Runtime recorded 0 iterations.

Arithmetic confirming the omission: the two director meters sum exactly to spentTotal on every channel (input 17,141,910 + 24,691,750 = 41,833,660; output 111,345 + 117,473 = 228,818; cacheRead 16,715,968 + 24,228,864 = 40,944,832; usd 29.3863104 + 15.1087325 = 44.4950429). The third node contributes exactly zero. spentTotal is a floor, not a total, and the record does not say so.

Third observation: the declared budgets bounded nothing

Both children carry {deadlineMs: 604800000, maxIterations: 64, maxTokens: 3000000}.

• Measured input was 17,141,910 and 24,691,750 tokens against maxTokens: 3000000. Whichever channel the cap is meant to count, it did not stop either child at 3M.
• iterations: 0 on every meter, so a 64-iteration cap had no counter to act on.
• tokensKnown: false and usdKnown: false throughout; usd: 44.4950429 is usdEstimated, and a cap enforced against an estimate did not bound the real channel.

Suggested direction

1. Give the preflight a timeout proportional to observed host conditions, or make it a non-fatal advisory. Its value is catching a routing error; a transport stall on a static route is not that.
2. Do not let an exhausted retryable transient produce no-winner when the tree shows settled children with durable artifacts. Report driver exhaustion as its own terminal state.
3. Rewrite the message so it names the endpoint and the timeout budget rather than the model, since the endpoint ignores the model.
4. Include every spawned node in tree.nodes and in spendGaps, including one that failed to meter, so the settled record can name what it could not measure.

Versions

@tangle-network/agent-runtime@0.210.0, @tangle-network/agent-eval@0.180.0, cli-bridge at /Users/drew/code/_wt/cli-bridge-8921, opencode backend, two providers (zai-coding-plan/glm-5.3, kimi-for-coding/k3).

Evidence retained at discovery-lab#682 (branch research/collab-decoder-20260911).

[drewstone]

Correction from the reporter — two claims above are wrong, retracting them

I re-read the record and the source after filing. Two of my claims are false and one is an overreach. Correcting them here rather than leaving them to be discovered.

Wrong: "after every child has succeeded"

The driver did not fail at the end of the run. It failed 51.5 minutes in, and the children then ran for 2 hours 25 minutes more and completed the entire research.

Corrected timeline, from observer.jsonl and the bridge log:

time	event
03:51:30.756Z	run starts
04:42:42.046Z	the root's own turn fails: BackendError: opencode: opencode error event without a message (code upstream). A genuine provider failure, not a timeout.
04:42:46.174Z	attempt 1 metered, telemetry: unknown-after-failure
04:42:50.920Z	attempt 2, known-zero-before-dispatch
04:42:55.489Z	attempt 3, known-zero-before-dispatch
04:43:01.792Z	attempt 4, known-zero-before-dispatch → maxConsecutiveFailures: 3 → no-progress
05:41:42Z	child s0 settles, done
07:08:37Z	child s1 settles; run finalizes and reports the 04:43 error as the settled cause

So the first failure was a real upstream provider error; only retries 2 through 4 hit the capability preflight. The title's framing is wrong and I have corrected it.

Wrong: "the declared budgets bounded nothing"

I compared maxTokens against the input channel. That is not the channel a budget counts. chargedTokens(usage) = usage.input - creditedCacheRead(usage) + usage.output, and creditedCacheRead credits the full cacheRead whenever the reported classes fit inside input, which they do here exactly.

Recomputed against maxTokens: 3000000:

• director-a: 17,141,910 − 16,715,968 + 111,345 = 537,287, 17.9% of cap
• director-b: 24,691,750 − 24,228,864 + 117,473 = 580,359, 19.3% of cap

The budget accounted correctly and neither child came close to its ceiling. My claim was an artifact of reading the wrong field. Retracted.

(iterations: 0 on every meter still looks odd next to a maxIterations cap, but I have not traced it and am not claiming it as a defect.)

Overreach: "a retryable transient should not produce a terminal no-winner"

madeProgress deliberately refuses to count spend and settlements while a declared check is outstanding, and says so:

Spend and settlements count only while no declared check is outstanding — with a check unmet they are the burn-rate reading this module's header measures and rejects.

That is documented intent, not an ordering bug, and I withdraw the suggestion that child settlements should have counted here.

The retry policy is also already configurable through deps.driverRetry (maxAttempts, maxConsecutiveFailures, backoff). I did not set it. The defaults that applied were maxAttempts: 8 and maxConsecutiveFailures: 3, and three barren attempts inside 20 seconds is simply too tight for a multi-hour run on a contended host. That is my misconfiguration, not a runtime defect — and in my case the operating surface I drive Runtime through did not expose driverRetry at all, which is a gap I am fixing on my side.

What I still believe is a real defect

Narrowed to what I verified in source:

1. BRIDGE_RUN_STATE_TIMEOUT_MS = 2_000 is a hardcoded module constant. No run option reaches it. When the retry policy is the intended mitigation for a fragile probe, the probe's own budget should be reachable too, or derived from the seam's configured timeout.

2. The message names a model for an endpoint that ignores models. GET /v1/capabilities in cli-bridge returns a static object; the ?model= parameter does not affect the response. bridge ... did not answer for model "opencode/kimi-for-coding/k3" therefore points a reader at provider routing when the real condition is a 2s socket budget. Naming the endpoint and the budget would have saved me an hour.

3. A node that ran is absent from the settled record. s1:s0 spawned 04:15:04Z, settled 04:30:58Z, and appears in neither tree.nodes (2 entries) nor spendGaps. The two director meters sum exactly to spentTotal on every channel, so the node contributed zero. spentTotal is a floor and the record cannot say so, because nothing in it names the third node.

Point 3 is the one I would fix first: an unmeasured node that the record cannot name is worse than one it reports as unmeasured.

[drewstone]

Second correction: the main defect was already fixed in 0.212.0. Retracting it.

A peer pointed this out and I verified it against the published tarballs rather than take it on trust.

The probe no longer borrows the run-state budget. In 0.210.0, which is what my run used, bridgeModelRouteRefusal called:

bridgeGet(seam, `/v1/capabilities?model=...`, BRIDGE_RUN_STATE_TIMEOUT_MS, signal)   // 2_000

In 0.212.0 and every version since, including 0.215.0:

BRIDGE_ROUTE_PROBE_TIMEOUT_MS = 30_000
bridgeGet(seam, `/v1/capabilities?model=...`, BRIDGE_ROUTE_PROBE_TIMEOUT_MS, signal)

Checked by unpacking @tangle-network/agent-runtime@0.212.0, @0.214.0 and @0.215.0 and grepping the constant and the call site. The probe has its own 30s budget, and per the peer the same change caches routed models per bridge. That is the whole of my point 1, fixed before I filed it.

And BRIDGE_RUN_STATE_TIMEOUT_MS staying at 2s is correct, not an oversight. A slow run-state read costs one skipped refresh; it does not gate a spawn. I was wrong to lump the two constants together — I grepped the constant name, saw 2s still present in 0.213.0, and concluded the defect survived without checking which call site consumed it. That is the same mistake as my first correction: reading one field and inferring the behaviour instead of following it to its use.

So the headline claim is gone, and with it the reason I filed. What remains is much smaller:

1. The message names a model for a route that ignores models. Still present in 0.215.0: did not answer for model ${JSON.stringify(wireModel)}. /v1/capabilities returns a static object in cli-bridge and the ?model= parameter does not affect the response, so a transport stall reads as a provider-routing problem. Now that the timeout is 30s this is rare, but when it fires it still points a reader at the wrong subsystem. Cosmetic, worth one line.

2. A spawned node absent from the settled record. s1:s0 spawned 04:15:04Z, settled 04:30:58Z, and appeared in neither tree.nodes (2 entries) nor spendGaps. The two director meters sum exactly to spentTotal on every channel, so that node contributed zero and nothing in the record can name it. I measured this on 0.210.0 and have not re-measured on 0.215.0, so treat it as unverified against current.

Retitling to match what is actually left. If a maintainer judges point 2 already covered by later work, close this — the evidence for it is a 0.210.0 run and I would rather it be closed than sit here asserting something stale.

Evidence for the original run remains at discovery-lab@6ddb2526.