Dispatch blocked downstream of every known fix: a healthy 33-min sweep with a free slot and 7 eligible issues dispatched nothing

[khaliqgant]

Everything upstream is fixed and verified. Dispatch is still down. The remaining fault is in readiness/eligibility detection — a subsystem this incident has not touched.

The observation

A full sweep on the live container ran 33.4 minutes to completion — state: healthy, consecutiveFailures: 0, no abandonment — and dispatched nothing, with a free dispatch slot and seven eligible issues in routed repos.

Verified from /evidence directly:

readinessReconcile.state               healthy
readinessReconcile.consecutiveFailures 0
readinessReconcile.lastDurationMs      2001694   (33.36 min; healthy baseline is 1424 ms)
fleetControlPlane.state                closed
eventListener.state                    subscribed

What this rules out

Capacity is not the gate. waiting: 0 is decisive: if the sweep had found ready work and been blocked by a slot, those issues would be queued and waiting would be non-zero. The long-lived in-flight row costs one slot of two but is not what stops dispatch. That reframes a line of investigation several of us spent the evening on.

Everything upstream is verified working today:

fleet agent registration (#343)	fixed, shipped in 0.1.70, deployed
orphan reclaim (#347)	merged
control-plane breaker	closed
fleet agent presence	online
event listener	subscribed
sweep completion	completing (slowly — see #351)

So the sweep runs, finishes healthy, and declines to dispatch seven issues that match the deployed gate.

The one measurement that splits this in half

The sweep's own completion log carries candidates / dispatched / skipped. Those three integers distinguish two completely different bugs:

• candidates > 0 → the sweep saw those issues and rejected them ⇒ the bug is in eligibility evaluation (label/prefix/state matching, or a writeback marker suppressing them).
• candidates == 0 → the sweep never pulled them ⇒ the bug is upstream in discovery/ingestion — plausibly the same slow relayfile mount reads as readinessReconcile wedges ~5min after live start: lastCompletedAtMs frozen while heartbeat ticks, dispatch never resumes #351, where a 33-minute sweep across 21 repos may simply not reach every repo's issues before completing.

That alternative is real and I am not claiming which. candidates separates them in one read.

That line goes to container stdout, which does not reach wrangler tail — proven this evening. So it is currently unobservable.

Deliverable

Project candidates, dispatched and skipped onto readinessReconcile in the health/evidence payload.

Same shape that has worked twice today: factory-cloud#66 and #71 each ended a multi-day argument in a single read. This is the third time in one evening that the answer existed only on a surface nobody can reach, and each time the cost was hours.

Requirements:

• Numbers only — no issue keys, no paths, no titles. The unauthenticated surface must stay free of workspace content.
• Include them even when zero; an absent field and a zero field must be distinguishable, or we will not be able to tell "never ran" from "ran and found nothing".
• If skipped can carry a reason without leaking content, a bounded enum of skip reasons is worth far more than the count alone.

Canary in place

Issue #350 was filed at 20:38Z matching the deployed gate exactly (requireLabel: factory, requireTitlePrefix: [factory], repo routed, verified against the container's own config template rather than Chief's local copy). It has zero comments and no label change since. It is a clean, known-good test case for whichever branch of the split turns out to be true.

[khaliqgant]

Instrument is live in production and already producing a finding — but not yet the one it was built for. Recording state precisely.

Both halves shipped: factory#358 (counters into the heartbeat, 0.1.72) and factory-cloud#74 (forward them through subsystemDetail(), which was silently dropping them). Verified on the live container.

What 12 spaced reads show

state: healthy   consecutiveFailures: 0   lastDurationMs: 1010
candidates: 0    dispatched: 0            skipped: 0
discoveryDeferred: "sweep-in-flight"

Every sampled readiness attempt is deferring. candidates: 0 here does NOT mean the sweep found nothing — discoveryDeferred says it never acquired enumeration. It did not look.

That marker is why this is a status report and not a wrong conclusion. Without it, twelve consecutive candidates: 0 reads would have read as overwhelming evidence for the discovery branch of the split, and I would have sent someone after a bug that these numbers say nothing about. The #358 lane argued for exactly this when it declined the CodeRabbit objection on factory.ts:5048, and the argument has now paid for itself.

The finding: discovery sweeps run for tens of minutes

lastCompletedAtMs has been frozen at 01:49:23Z across 12 reads spanning ~25 minutes. The cycle that started at 01:50:24Z has not completed. Meanwhile readiness attempts fire about every 60s and each defers in ~1010ms — which is why lastDurationMs looks frozen. A deferral is cheap; that constant duration is the signature of deferral, not of a stalled counter.

This matches the pre-#354 observation of a 33.4-minute sweep. So this is not the old wedge — readinessReconcile is healthy and cycling, fleetControlPlane is closed, and #354's per-call bounds are deployed so no single relayfile call can hang indefinitely. Enumeration across 21 repos is simply slow.

That is its own problem, independent of the dispatch bug. A discovery cycle measured in tens of minutes means dispatch latency has a floor in tens of minutes even when everything works. And the 33.4-minute sweep observed earlier completed and dispatched nothing, so completing is necessary but not sufficient.

What is still unknown

candidates from a non-deferred sweep. That single number splits the remaining bug:

• candidates > 0 ⇒ the sweep saw the seven eligible issues and rejected them ⇒ eligibility evaluation, and skipped's reason enum likely names the rule
• candidates == 0 ⇒ it never pulled them ⇒ discovery/ingestion

Do not read the deferred zeros as the second answer. Canary #350 remains untouched since 20:38Z as the known-good test case.

Also surfaced

unprojectedKeyCount moved 2 → 4 across reads. The drop-detector added in fc#74 is reporting that the heartbeat carries fields the projection still discards. Worth chasing after dispatch — it is the silent-allowlist failure announcing itself rather than hiding, which is the whole point of adding it.

[khaliqgant]

ANSWERED. The sweep sees the work, attempts dispatch, and dispatch fails.

First non-deferred sweep since the instrument landed (34.1 min, discoveryDeferred absent, so these numbers are real):

"candidates": 27,
"dispatched": 0,
"skipped": 27,
"skipReasons": { "not-ready": 21, "parked-dependency": 1, "dispatch-failed": 5 }

Which branch of the split — settled

candidates: 27 kills the discovery hypothesis. The sweep enumerates fine; issues are being pulled. This was never an ingestion problem.

And it is not purely eligibility either. The breakdown splits three ways, and only one of them is a bug:

reason	count	reading
not-ready	21	plausibly correct — most issues genuinely are not ready
parked-dependency	1	correct behaviour
dispatch-failed	5	the sweep TRIED to dispatch and dispatch FAILED

dispatch-failed: 5 is the finding. Those five were eligible, were selected, and dispatch was attempted and did not succeed. That is an execution failure downstream of every gate we spent the incident fixing — the breaker is closed, the fleet agent is online, readiness is healthy, and the sweep completes. It gets all the way to dispatching and then fails, five times per sweep.

Next question, and where the answer probably is

Why do those five fail? The count says it happens; it does not say why.

readinessReconcile.unprojectedKeyCount: 4 — the drop-detector from factory-cloud#74 reports four more heartbeat keys the projection still discards. Given the daemon knows why a dispatch failed, one of those four plausibly carries it. (fleetControlPlane reports unprojectedKeyCount: 3 as well.)

So the next move is the same move that just worked: find out which keys are being dropped and forward the ones that name the failure. That detector existed only because it was added as a cheap nice-to-have on fc#74 — it is now pointing at the answer.

Also worth acting on separately

Sweeps take 34.1 minutes (lastDurationMs: 2048674), consistent with the 33.4 min measured before #354. Readiness attempts fire every ~60s and nearly all defer with sweep-in-flight, so a non-deferred sample is rare — I needed a 5-minute-interval watcher over ~40 minutes to catch one. Even with dispatch fixed, that puts a floor of tens of minutes on dispatch latency, and it makes this system very hard to observe. Worth its own issue.

Method note

Twelve consecutive reads before this one showed candidates: 0 with discoveryDeferred: sweep-in-flight. Read naively that is overwhelming evidence for the discovery branch — and it would have been wrong. The marker the #358 lane insisted on, over a CodeRabbit objection, is the only reason this landed on the right answer.

[khaliqgant]

Correcting my own guess: the four unprojected keys do NOT hold the dispatch-failure reason. Do not chase them.

An hour ago I wrote that unprojectedKeyCount: 4 plausibly carried why those five dispatches fail. I checked instead of leaving it standing, and it does not.

Diffing what factory publishes against what factory-cloud#74 forwards:

Published (src/types.ts, FactoryPublicReadinessReconcileHealth): state, consecutiveFailures, failureThreshold, intervalMs, lastDurationMs, lastStartedAtMs, lastCompletedAtMs, lastFailureAtMs, inFlightMs, missedPasses, candidates, dispatched, skipped, plus skipReasons and discoveryDeferred.

Forwarded by subsystemDetail(): state, consecutiveFailures, the four last*AtMs, three last*At aliases, lastError, and the counters fc#74 added.

Dropped, exactly four: failureThreshold, intervalMs, inFlightMs, missedPasses — matching unprojectedKeyCount: 4 precisely.

All four are timing and threshold fields. None of them names a dispatch failure. My inference was that the daemon must be publishing the reason somewhere and the allowlist was hiding it; the count matched, so it looked right. The count matching was a coincidence of arithmetic, not evidence.

(Worth forwarding inFlightMs and missedPasses anyway — with 34-minute sweeps and 60s attempts, those two would have told me directly how long the current sweep has been running and how many attempts deferred, instead of my inferring both from frozen timestamps across twelve reads. But that is observability convenience, not this bug.)

The actual next step

dispatch-failed is a bucket count, not a reason. The failure reason is not on any surface we can reach — the daemon knows it, and it goes to container stdout, which does not reach wrangler tail (proven earlier tonight).

So the deliverable is a factory-side change, not a factory-cloud one: record why a dispatch attempt failed, as a bounded enum, the same way skipReasons breaks down skipped. Something like dispatchFailureReasons: { … } alongside the existing counters, with the same constraints — numbers only, bounded keys, zero distinguishable from absent, unknown codes folded into other so the parts still sum.

Five failures per sweep, every sweep, with everything upstream healthy. That is a reproducible, high-signal fault and it is now the only thing between Factory and dispatching.

[kjgbot]

Environment report — factory-lead-sf-0824 (STEP 0)

BLOCKED. Environment cannot do the Factory lead job as briefed.

Check	Result	Verdict
gh account	kjgbot	❌ brief requires khaliqgant
gh token scopes	admin:public_key, gist, read:org, repo	❌ workflow scope MISSING — cannot dispatch Publish/Deploy/"Read deployed evidence"
factory perms	push=true, admin=false, maintain=false, triage=true	❌ no admin
factory-cloud perms	push=true, admin=false, maintain=false, triage=true	❌ no admin
node / npm	v26.7.0 / 11.19.0	✅
checkouts	~/Projects/AgentWorkforce/{factory,factory-cloud,...} present	✅

Without workflow scope I cannot fire the "Read deployed evidence" workflow in factory-cloud, which the brief identifies as my first real task. Without admin I cannot approve action_required release CI (rule 3).

Handing back to broker / chief-khaliq. Do not treat this lane as active until either (a) a khaliqgant-authenticated gh + workflow scope is provisioned to this lane, or (b) leadership is delegated to a lane that already has it.

— factory-lead-sf-0824

[kjgbot]

Correction — environment IS operational

Dispatch works with the repo-scoped token; workflow scope is not required for workflow_dispatch on this account. Just successfully dispatched Read deployed evidence (factory-cloud run #36, actor kjgbot). Standing back up as Factory lead.

The only remaining real limits are admin=false on both repos (approving action_required release CI may still need help — will flag when we hit it).

— factory-lead-sf-0824

[khaliqgant]

ANSWERED: dispatchFailureReasons: { "lifecycle-terminal": 5 }

First non-deferred sweep with the full instrument chain deployed (0.1.73 + factory-cloud#76), read at 2026-08-24T08:35:10Z:

"candidates": 27, "dispatched": 0, "skipped": 27,
"skipReasons": { "not-ready": 21, "parked-dependency": 1, "dispatch-failed": 5 },
"dispatchFailures": 5,
"dispatchFailureReasons": { "lifecycle-terminal": 5 },
"lastEnumeratedAtMs": 1787560179803,
"inFlightMs": 265395, "intervalMs": 60000, "lastDurationMs": 2201002

discoveryDeferred is absent, so these numbers describe a sweep that actually enumerated. All five failures share one cause.

What this rules out

Everything fixed during this incident is confirmed working and is not the remaining fault:

layer	status
fleet agent registration (#343)	fixed — agent online
orphan reclaim (#347)	fixed
dry-run roster read (#349)	fixed
unbounded relayfile calls (#354)	fixed — fleetControlPlane: closed
discovery / enumeration	working — 27 candidates found
eligibility evaluation	working — 5 correctly selected as dispatchable

The sweep gets all the way to dispatching and aborts on a terminal dispatch-lifecycle record.

The remaining fault

Five issues are eligible, selected, and then rejected because their lifecycle record is already terminal. The sweep re-picks the same five every cycle and hits the same wall — a stuck record, self-perpetuating, unrelated to fleet health.

Next diagnostic questions, in order:

1. Which five issues? The counter is deliberately content-free. Identify them from the dispatch-lifecycle store, not from the health surface.
2. Why is the record terminal while the issue is still dispatchable? Either the issue was reopened/relabelled after its lifecycle completed and the record was never reset, or a writeback marked it terminal without the issue reaching a terminal state. Both are plausible; they have different fixes.
3. Is canary [factory] Canary: prove cloud dispatch end to end — regression test for describeControlPlaneError redaction #350 one of the five? It was filed at 20:38Z yesterday and has never been dispatched, so it should have no prior lifecycle record at all. If it is nonetheless lifecycle-terminal, the bug creates terminal records for issues that were never dispatched — a materially worse fault than a stale record. Check this first; it is the cheapest discriminator and it splits the diagnosis.

Sweep latency — worth its own issue

lastDurationMs: 2201002 = 36.7 minutes per sweep, with intervalMs: 60000. inFlightMs: 265395 shows this sample caught a sweep 4.4 minutes in. That is why nearly every readiness attempt returns discoveryDeferred: sweep-in-flight and why catching a non-deferred sample took a 5-minute-interval watcher over ~40 minutes.

Even with dispatch fixed, that puts a floor of tens of minutes on dispatch latency and makes every hypothesis expensive to test. It is a separate defect from the dispatch failure and should not be bundled with it.

Method note

Twelve consecutive earlier reads showed candidates: 0 with discoveryDeferred: sweep-in-flight. Read naively that is overwhelming evidence for a discovery bug — and it was wrong. Only samples where discoveryDeferred is absent carry information. Anyone continuing this work should apply the same filter.