Skip to content

fix(supervise): the pre-spawn bridge capacity check reads the admission lane, not the executor that actually gates a spawn #1186

Description

@drewstone

What happens

bridgeAdmissionRefusal decides whether a bridge has room before Runtime spawns. It reads only the admission lane:

const bulkActive = admission?.activeByClass?.bulk
const bulkMaxActive = admission?.bulkMaxActive
if ( typeof bulkMaxActive === "number" ) {
  if (bulkActive < bulkMaxActive) return void 0   // no refusal
  
}

(installed @tangle-network/agent-runtime@0.210.0, dist/redact-CE6Hfkrp.js)

The same /health body carries the counter that actually gates a spawn, and nothing reads it:

"admission": {"active": 2, "maxActive": 8, "bulkMaxActive": 6, "activeByClass": {"reserved": 0, "bulk": 2}},
"executor":  {"in_flight": 2, "max": 4, "queued": 0, "acquire_deadline_ms": 60000}

A bridge spawn waits on a host semaphore whose ceiling is BRIDGE_HOST_MAX_CONCURRENCY, default 4 (cli-bridge src/executors/host.ts), while the admission lane allows 6. So whenever between 4 and 6 bulk turns are live, the check says there is room, Runtime admits and spawns, and the turn then blocks on a semaphore that is already full.

What the operator sees

The turn waits the bridge's acquire_deadline_ms, default 60,000 ms, and is refused with capacity: true and provider_dispatch: not_started. Nothing in that sequence says "the host was full when we decided to spawn". It reads as a slow or failing provider, a minute after the decision that caused it.

The gap is not theoretical. A play registering maxLiveWorkers: 3 plus a root that turns through the same bridge has a peak demand of 4 concurrent turns, which equals the whole host. With three unrelated runs holding three slots, the pre-spawn check returned no refusal because bulk was 3 of 6.

Why a caller cannot compensate

There is no field for it. runner/pursuit.mjs sends backend, bridgeUrl, bridgeBearer and cwd; SUPERVISE_KEYS carries no acquire or concurrency key, so a registration cannot raise the ceiling or lengthen the wait. Raising BRIDGE_HOST_MAX_CONCURRENCY needs a bridge restart, which is currently fatal to every live run on that bridge (drewstone/cli-bridge#220). The only remaining option is an out-of-band launch precondition, which is what we shipped: a read-only gate that refuses to launch unless executor.in_flight is 0 of 4.

Expected

Read the executor counters when the bridge reports them, and refuse before spawning when executor.in_flight + intended concurrency > executor.max. The existing comment in that function already anticipates the shape: "Older bridges expose only the overall counters; those retain the prior fallback." Modern bridges expose executor, so prefer it and keep the admission lane as the fallback.

When a turn is refused after waiting on the semaphore, say that it was a capacity refusal rather than leaving it to look like a provider failure.

Related

drewstone/cli-bridge#220 (a mid-run restart is unrecoverable, which is why raising the ceiling is not an available workaround) and #221 (the turn deadline is wall-clock from process start).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions