Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 38 additions & 1 deletion docs/deployed-diagnostics.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,12 @@ logic of its own by design: the boundary lives in one place, in this repo, with
"inFlightSinceMs": 1787224595805, // when the oldest sweep still running began
"inFlightMs": 4560000, // this pass has run 76 minutes
"missedPasses": 76,
"lastErrorClass": "TimeoutError"
"lastErrorClass": "TimeoutError",
// The last COMPLETED sweep's arithmetic (#355). Absent until one completes.
"candidates": 7, // work units it pulled and evaluated
"dispatched": 0, // work units it dispatched
"skipped": 7, // work units it saw and declined
"skipReasons": { "dispatch-terminal": 7 }
},
"eventListener": { "state": "subscribed" },
"fleetControlPlane": { "state": "closed", "consecutiveFailures": 0, "failureThreshold": 3 }
Expand All @@ -86,6 +91,38 @@ logic of its own by design: the boundary lives in one place, in this repo, with
#296 — fall back to `lastStarted > lastCompleted`, which infers the same thing from timestamp order.
Prefer the published field: once a sweep has passed its deadline (below) the wait records a failure
while the sweep underneath it keeps running, and order alone then reports nothing in flight.
- **`candidates` / `dispatched` / `skipped`** — the *green-but-idle* case, and the fastest question to
ask when nothing is being dispatched and every state above reads healthy. On 2026-08-23 a sub-second
sweep with `state: healthy`, `consecutiveFailures: 0` and a free dispatch slot declined seven
eligible issues, and no surface anyone could reach said which half of the pipeline was at fault.

- `candidates > 0` — the sweep **saw** those issues and **rejected** them. The bug is in eligibility
evaluation, and `skipReasons` names which gate.
- `candidates == 0` — the sweep **never pulled** them. The bug is upstream, in discovery/ingestion.
- **the three fields absent entirely** — this daemon has not *completed* a sweep (or predates #355).
That is not a zero, and must not be read as one: it says nothing about either half. Check
`lastCompletedAtMs` and `inFlightMs`.

They describe the last sweep that settled **successfully**, the same tense as `lastDurationMs`;
`lastCompletedAtMs` dates them. A pass that failed leaves them untouched rather than zeroing them.

- **`discoveryDeferred: "sweep-in-flight"`** — the sweep returned immediately because another process
held the discovery lease, so it enumerated nothing. Without this, that pass is indistinguishable
from one that queried the provider and legitimately found no ready work: both publish
`candidates: 0`.

- **`skipReasons`** — `skipped` split by a closed vocabulary
(`FACTORY_SWEEP_SKIP_REASON_CODES`); zero-count codes are omitted, so an absent key is a zero, and
the counts always sum to `skipped`. `dispatch-terminal` and `dispatch-retry-limit` are the two that
never clear on their own — a work unit in either needs a human. `dispatch-backoff`,
`already-tracked` and `queued-or-escalated` resolve by themselves. `out-of-scope` and `not-ready`
mean the gate is working as configured and the issue does not match it — check the deployed
`safety` config against the issue rather than the daemon.

Counts only, by construction: issue keys, paths and titles carry customer project and repository
names and never cross onto this surface. The keys are rebuilt from the reader's own copy of the
vocabulary, so a record from another version cannot publish an arbitrary string as one.

- **`fleetControlPlane`** — an `open` circuit fails every spawn and resume fast, so it gates dispatch
as hard as a failing sweep. `closed` is the healthy value.
- **`state: "stalled"`** — derived, not written: an in-flight pass older than ten sweep intervals.
Expand Down
44 changes: 42 additions & 2 deletions src/cli/diagnose.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { telemetryErrorClass } from '../observability/error-class.js'
import { normalizePublicHealth } from '../orchestrator/public-health.js'
import type { FactoryPublicHealth } from '../types'
import type { FactoryPublicHealth, FactoryPublicReadinessReconcileHealth } from '../types'

/**
* `factory diagnose --deployed <url>` (#295).
Expand Down Expand Up @@ -315,15 +315,49 @@ function verdictFor(diagnosis: Omit<DeployedFactoryDiagnosis, 'verdict' | 'dispa
'not dispatching: the readiness loop is not running — this instance is not a live daemon.',
}
}
// #355. Everything above is a subsystem verdict, and every one of them read
// green through a total dispatch outage: a healthy sub-second sweep declined
// seven eligible issues and this line still said "dispatching". The last
// sweep's own arithmetic is the only thing on this surface that can
// contradict that, so it goes in the sentence rather than three lines below
// it.
return {
dispatching: true,
verdict:
'dispatching: readinessReconcile is healthy' +
(readiness?.intervalMs ? ` on a ${formatDuration(readiness.intervalMs)} cadence` : '') +
`, and the event listener is ${health.eventListener?.state ?? 'unknown'}.`,
`, and the event listener is ${health.eventListener?.state ?? 'unknown'}.` +
` Last completed sweep: ${formatSweepOutcome(readiness)}.`,
}
}

/**
* The last completed sweep, as a phrase that keeps zero and absent apart (#355).
*
* `candidates: 0` says discovery pulled nothing and the bug is upstream of
* eligibility; no `candidates` at all says this instance has not finished a
* sweep — or predates the counters — and says nothing about either. Rendering
* both as `0` would recreate the ambiguity the field exists to remove.
*/
export function formatSweepOutcome(
readiness: FactoryPublicReadinessReconcileHealth | undefined,
): string {
if (!readiness || readiness.candidates === undefined) {
return 'not reported (no sweep has completed, or this instance predates the counters)'
}
return `${readiness.candidates} candidate(s), ${readiness.dispatched ?? 0} dispatched, ` +
`${readiness.skipped ?? 0} skipped` +
(readiness.discoveryDeferred
? ' — it deferred to another process holding the discovery lease and enumerated nothing'
: '')
}

/** The skip breakdown, ordered as the record carries it. */
export function formatSkipReasons(counts: Record<string, number>): string {
const entries = Object.entries(counts).filter(([, count]) => count > 0)
return entries.length === 0 ? '—' : entries.map(([code, count]) => `${code}=${count}`).join(', ')
}

export function formatDuration(ms: number | undefined): string {
if (ms === undefined) return 'unknown'
if (ms < 1_000) return `${ms}ms`
Expand Down Expand Up @@ -466,6 +500,12 @@ export function renderDeployedDiagnosis(diagnosis: DeployedFactoryDiagnosis): st
lines.push(` lastStartedAt : ${formatInstant(readiness.lastStartedAtMs)}`)
lines.push(` lastCompletedAt : ${formatInstant(readiness.lastCompletedAtMs)}`)
lines.push(` lastFailureAt : ${formatInstant(readiness.lastFailureAtMs)}`)
// #355. `candidates === 0` and an absent `candidates` are opposite
// diagnoses, so the renderer must not collapse them into one dash.
lines.push(` last sweep : ${formatSweepOutcome(readiness)}`)
if (readiness.skipReasons) {
lines.push(` skip reasons : ${formatSkipReasons(readiness.skipReasons)}`)
}
}
const capacity = health.dispatchCapacity
if (capacity) {
Expand Down
6 changes: 6 additions & 0 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -490,6 +490,12 @@ export type {
TriageDecision,
TriageEngine,
} from './types'
export {
FACTORY_SWEEP_SKIP_REASON_CODES,
factorySweepSkipReasonCode,
factorySweepSkipReasonCounts,
} from './orchestrator/sweep-skip-reason'
export type { FactorySweepSkipReasonCode } from './orchestrator/sweep-skip-reason'
export type { FleetControlPlaneState, FleetControlPlaneStatus } from './fleet/control-plane-circuit'
export {
LOAD_EVIDENCE_CONTRACT,
Expand Down
26 changes: 21 additions & 5 deletions src/orchestrator/factory.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3209,6 +3209,7 @@ describe('FactoryLoop', () => {
expect(report.skipped).toContainEqual({
issue: { uuid: 'uuid-141', key: 'AR-141', path: dependentPath },
reason: 'parked on dependencies: AgentWorkforce/pear#140',
code: 'parked-dependency',
})
expect(factory.status().parked).toEqual([
expect.objectContaining({
Expand Down Expand Up @@ -3355,6 +3356,7 @@ describe('FactoryLoop', () => {
expect(report.skipped).toContainEqual({
issue: expect.objectContaining({ key: '36' }),
reason: 'parked on dependencies: AgentWorkforce/pear#35',
code: 'parked-dependency',
})
expect(factory.status().parked).toEqual([
expect.objectContaining({ issue: expect.objectContaining({ key: '36' }) }),
Expand Down Expand Up @@ -3512,6 +3514,7 @@ describe('FactoryLoop', () => {
expect(report.skipped).toEqual([{
issue: { uuid: 'AgentWorkforce/pear#46', key: '46', path: canonicalPath },
reason: 'live state is not ready-for-agent',
code: 'not-ready',
}])
expect(fleet.spawns).toEqual([])
})
Expand Down Expand Up @@ -3809,6 +3812,7 @@ describe('FactoryLoop', () => {
expect(report.skipped).toEqual([{
issue: { uuid: 'AgentWorkforce/pear#49', key: '49', path },
reason: 'live state is not ready-for-agent',
code: 'not-ready',
}])
expect(fleet.spawns).toEqual([])
})
Expand Down Expand Up @@ -5480,6 +5484,7 @@ describe('FactoryLoop', () => {
expect(report.skipped).toContainEqual({
issue: { uuid: 'AgentWorkforce/pear#59', key: '59', path: racedPath },
reason: 'live state changed during dispatch',
code: 'dispatch-failed',
})
expect(report.dispatched.map((result) => result.issue.key)).toEqual(['60'])
expect(fleet.spawns.map((spawn) => spawn.name)).toEqual([
Expand Down Expand Up @@ -5572,6 +5577,7 @@ describe('FactoryLoop', () => {
expect(report.skipped).toContainEqual({
issue: { uuid: 'AgentWorkforce/pear#59', key: '59', path: blockedPath },
reason: 'dispatch lifecycle already terminal',
code: 'dispatch-failed',
})
expect(report.dispatched.map((result) => result.issue.key)).toEqual(['60'])
expect(fleet.spawns.map((spawn) => spawn.name)).toEqual([
Expand Down Expand Up @@ -5605,6 +5611,7 @@ describe('FactoryLoop', () => {
// Sanitized: `run-once` prints the report as JSON, so the reason
// carries a classification rather than raw provider text.
reason: 'dispatch failed (TypeError)',
code: 'dispatch-failed',
})
expect(report.dispatched.map((result) => result.issue.key)).toEqual(['60'])
expect(factory.status().counters.dispatchItemFailuresSkipped).toBe(1)
Expand Down Expand Up @@ -5727,6 +5734,7 @@ describe('FactoryLoop', () => {
expect(report.skipped).toContainEqual({
issue: { uuid: 'AgentWorkforce/pear#59', key: '59', path: blockedPath },
reason: 'dispatch failed (Error)',
code: 'dispatch-failed',
})
expect(report.dispatched.map((result) => result.issue.key)).toEqual(['60'])
expect(fleet.spawns).toEqual([])
Expand Down Expand Up @@ -5918,6 +5926,10 @@ describe('FactoryLoop', () => {
expect(report.skipped).toEqual([{
issue: { uuid: 'AgentWorkforce/pear#53', key: '53', path },
reason: 'active dispatch claim or live agent still owns the issue',
// The batch already holds this issue, so it is the `already-tracked`
// gate that declines it — not the readiness gate that carries the
// same operator text one branch later.
code: 'already-tracked',
}])
expect(fleet.spawns.map((spawn) => spawn.name)).toEqual(['ar-53-babysit-pear'])
expect(fleet.spawns[0]).toMatchObject({
Expand Down Expand Up @@ -6234,6 +6246,7 @@ describe('FactoryLoop', () => {
issue: '55',
path,
reason: 'active dispatch claim or live agent still owns the issue',
code: 'not-ready',
},
])
} finally {
Expand Down Expand Up @@ -6705,8 +6718,8 @@ describe('FactoryLoop', () => {

expect(report.pulled.map((issue) => issue.key)).toEqual(['AR-1', 'AR-2', 'AR-3', 'AR-4'])
expect(report.dispatched.map((result) => result.issue.key)).toEqual(['AR-1', 'AR-2'])
expect(report.skipped).toContainEqual({ issue: { uuid: 'uuid-3', key: 'AR-3', path: issuePath(3) }, reason: 'queued or escalated' })
expect(report.skipped).toContainEqual({ issue: { uuid: 'uuid-4', key: 'AR-4', path: issuePath(4) }, reason: 'live state is not ready-for-agent' })
expect(report.skipped).toContainEqual({ issue: { uuid: 'uuid-3', key: 'AR-3', path: issuePath(3) }, reason: 'queued or escalated', code: 'queued-or-escalated' })
expect(report.skipped).toContainEqual({ issue: { uuid: 'uuid-4', key: 'AR-4', path: issuePath(4) }, reason: 'live state is not ready-for-agent', code: 'not-ready' })
expect(fleet.spawns).toHaveLength(4)
expect(factory.status().inFlight.map((issue) => issue.key)).toEqual(['AR-1', 'AR-2'])
expect(factory.status().queued.map((issue) => issue.key)).toEqual(['AR-3'])
Expand Down Expand Up @@ -7208,7 +7221,7 @@ describe('FactoryLoop', () => {
expect(report.pulled).toEqual([{ uuid: 'uuid-67-canonical', key: 'AR-67', path: canonicalPath }])
expect(report.dispatched).toEqual([])
expect(report.skipped).toEqual([
{ issue: { uuid: 'uuid-67-canonical', key: 'AR-67', path: canonicalPath }, reason: 'live state is not ready-for-agent' },
{ issue: { uuid: 'uuid-67-canonical', key: 'AR-67', path: canonicalPath }, reason: 'live state is not ready-for-agent', code: 'not-ready' },
])
expect(fleet.spawns).toEqual([])
})
Expand Down Expand Up @@ -7287,7 +7300,7 @@ describe('FactoryLoop', () => {

expect(report.dispatched).toEqual([])
expect(report.skipped).toEqual([
{ issue: { uuid: 'uuid-365', key: 'AR-365', path: issuePath(365) }, reason: 'dispatch already terminal' },
{ issue: { uuid: 'uuid-365', key: 'AR-365', path: issuePath(365) }, reason: 'dispatch already terminal', code: 'dispatch-terminal' },
])
expect(fleet.spawns.map((spawn) => spawn.name)).toEqual(['ar-365-impl-pear', 'ar-365-review'])
expect(factory.status().counters.dispatchTerminalReopened).toBeUndefined()
Expand Down Expand Up @@ -7758,6 +7771,7 @@ describe('FactoryLoop', () => {
expect(report.skipped).toContainEqual({
issue: { uuid: '40c7e780-59ad-47ee-8809-3a9b8434d8fb', key: 'AR-173', path: capturedStaleDoneCanonicalPath },
reason: 'live state is not ready-for-agent',
code: 'not-ready',
})
expect(mount.readPaths).not.toContain(byIdBareAliasPath)
expect(mount.readPaths).not.toContain(byIdCanonicalShapedAliasPath)
Expand Down Expand Up @@ -11010,6 +11024,7 @@ describe('FactoryLoop', () => {
expect(report.skipped).toContainEqual({
issue: { uuid: 'uuid-21', key: 'AR-21', path: unscopedPath },
reason: 'not factory-e2e scope',
code: 'out-of-scope',
})
expect(report.triaged).toEqual([])
expect(report.dispatched).toEqual([])
Expand Down Expand Up @@ -11095,6 +11110,7 @@ describe('FactoryLoop', () => {
expect(report.skipped).toContainEqual({
issue: { uuid: 'uuid-261', key: 'AR-261', path: draftPath },
reason: 'not reconciled real Linear issue',
code: 'not-dispatchable',
})
expect(report.triaged).toEqual([])
expect(report.dispatched).toEqual([])
Expand Down Expand Up @@ -19231,7 +19247,7 @@ describe('FactoryLoop', () => {
const report = await factory.runOnce()

expect(report.dispatched).toEqual([])
expect(report.skipped).toContainEqual({ issue: { uuid: 'uuid-20', key: 'AR-20', path: issuePath(20) }, reason: 'queued or escalated' })
expect(report.skipped).toContainEqual({ issue: { uuid: 'uuid-20', key: 'AR-20', path: issuePath(20) }, reason: 'queued or escalated', code: 'queued-or-escalated' })
expect(fleet.spawns).toEqual([])
const slackRoots = mount.writes.filter((write) => isSlackRootWritePath(write.path))
expect(slackRoots).toHaveLength(1)
Expand Down
Loading