Repository navigation
Description
Problem
The recurring terminal dialog signature (all_attempts_exhausted with an empty route object) continues on a deployed revision whose build and runtime revisions match the repository base. The newest capture shows twelve attempt_failed rows in four requeue rounds of exactly three attempts each (one external primary-route attempt plus one on each of the two self-hosted GPU lanes), each attempt failing immediately, in about 35 seconds total. Two live explanations fit the verified code: (1) an external farm-wide backend outage — an open issue documents GPU device faults on one self-hosted lane since the 2026-11-05 farm boot — exhausting every pass; and/or (2) an unhealed reverted dialog hop budget: the pinned tree confirms each routing pass stops after retry.max_hops started attempts, the dialog workflow is normalized to 5 hops only by a once-only guarded convergence pass, and the deployed partial-admin-update fix prevents future budget reverts but neither heals an already-reverted row nor clears the once-only marker. Per-attempt failure reasons are not retained, so the two cannot yet be separated; the deep stage must recover per-attempt reasons plus the live workflow row.
External cause: possible. Code defect: possible.
Observations
- Incident snapshot: terminal routing event all_attempts_exhausted for the dialog workflow with an empty route object; count 67 occurrences since first seen about two months ago, status diagnosing, last seen at capture time (the prior scoped analysis of the same signature counted 53, then 66, so occurrences continue under the current deployed code).
- Deployed runtime is healthy and current: runtime revision, build revision and repository base are identical, restart_count 0, no OOM kill, ample memory and disk, moderate load on 8 CPUs, database pool normal with zero waiting locks; local resource starvation, crash loops and lock exhaustion are ruled out.
- Scoped route-attempt capture for the newest occurrence: twelve attempt_failed rows numbered 1-12 forming four retry rounds of exactly three attempts each — one external primary-route attempt plus one attempt on each of the two self-hosted GPU lanes — rounds roughly five seconds apart, all twelve failing within about 35 seconds; per-attempt error strings and failure reasons are not retained.
- Attempt timing shows immediate failures (the two self-hosted lane attempts within each round share the same second; each round completes in about 0-5 seconds), i.e. fast rejects rather than full generation runs; the model field is populated only for one self-hosted lane and is absent on the external route attempts.
- The task record: dialog kind on the same queue, state failed, created to completed about 66 seconds including a queue-wait gap before the final run; the four observed rounds are consistent with the job-retry handler requeueing the whole turn up to its max attempts before emitting the terminal event with provider/model unset, matching the empty route object.
- Queue counters for the dialog lane: about 33k completed versus 124 failed (well under 0.5 percent), so dialog routing succeeds broadly and the trigger is conditional on provider or stored-config state, not constant.
- Verified in the pinned tree (crates/openplotva-app/src/routed_attempts.rs:369): a routing pass breaks once started_attempts reaches the workflow retry.max_hops, so a hop budget of 3 ends each pass after the three observed attempts and the pass never reaches deeper fallback assignments.
- Verified in the pinned tree (crates/openplotva-app/src/model_routing.rs:1970-1971, 2064, 2107): the convergence pass resets the dialog workflow to retry_max_hops 5 / retry_wall_ms 180000 and reorders the fallback tail only while the once-only app-setting marker llm.routing.dialog_fallback_cascade_v1 is absent and a readiness check passes; the marker is then written, so a live row whose budget was later reverted is never healed automatically.
- Verified in the pinned tree (crates/openplotva-storage/src/llm_routing.rs:430): the admin workflow update now COALESCEs omitted fields, keeping stored values; this prevents future silent budget reverts but restores nothing already reverted and leaves the once-only convergence marker in place.
- The prior scoped analysis of this same signature (different occurrence) retained a single lane attempt lasting about 98 seconds and suspected session-engine exits that hard-coded validator rejections as protocol faults; that funnel was reclassified by an already-deployed merge, and the new capture's fast multi-lane failure shape differs from that generation-then-reject timing.
Hypotheses
- Farm-wide backend outage (external): both self-hosted GPU lanes fault (an open issue documents GPU device faults on one lane since the 2026-11-05 farm boot) and the external primary route also fails fast in the same window; every pass exhausts its started attempts with immediate errors, four requeue rounds follow, and the turn terminates with the empty-route event.
- Unhealed reverted hop budget (our code/config interaction): the live dialog workflow row still carries a reverted 3-hop budget from an earlier partial admin update; the deployed COALESCE fix prevents future reverts but cannot heal it because the once-only convergence marker blocks re-application, so each pass stops after three started attempts and the designed fallback tail is never attempted — matching the exactly-three-attempts-per-round shape.
- Request-construction defect (our code): a malformed or oversized request produced for this specific prompt is rejected immediately by every lane including the external route and reproduces on every replay; without per-attempt error strings this cannot be separated from the outage story.
- Validator-rejection funnel (prior suspect, now less likely for this occurrence): the deployed reclassification routes validator rejections to same-model re-sampling, and immediate sub-5-second failures are inconsistent with generation-then-reject timing; this funnel may still contribute to other occurrences of the same terminal signature.
Supporting
- All three distinct lanes (one external, two self-hosted) fail immediately and repeatedly within the same occurrence — the shape an outage or a request-level defect produces, not a single-lane validation story.
- The open GPU-fault issue documents one self-hosted lane failing since the 2026-11-05 farm boot, giving a concrete external fault source active in the incident window.
- The exactly-three-attempts-per-round, four-round structure matches the code-verified hop cap (routed_attempts.rs:369) interacting with the job-retry requeue cap, and the fallback tail is never attempted in any round.
- The once-only convergence guard (model_routing.rs:1970, 2107) plus the COALESCE-only fix (llm_routing.rs:430) leaves a previously reverted dialog budget permanently unhealed, matching the open issue's account of a partial admin update reverting the 5-hop/180s budget.
Contradicting
- The outage story alone does not explain why no round ever starts a fourth or fifth attempt: even with all primary lanes faulting, a 5-hop pass should still start fallback-tail attempts and fail there, so the observed hard stop at three attempts per round is more directly explained by the hop-budget cap than by breaker state alone.
- The budget-revert story alone does not explain the failures themselves: a reduced hop count changes which attempts start, not whether they succeed, so an external or request-level fault must still be present for attempts to fail; the two hypotheses are complementary, not exclusive.
- Dialog throughput is broadly healthy (about 33k completed vs 124 failed), which argues against a constant request-construction defect affecting every turn and for a state-conditional trigger.
Related changes
- Issue Dialog turns fail when GPU0 NInfer crashes: GPU device faults since the 2026-11-05 farm boot #294 (open): documents GPU device faults on a self-hosted lane since the 2026-11-05 farm boot with dialog turns failing, and records the same twelve-attempt, four-round, three-lane shape with the fallback tail never attempted; primary external-cause candidate in the window.
- Issue Dialog turns terminate as provider_protocol_error after our own validators reject the answer: two session-engine exits bypass the model-output split and requeue to exhaustion #277 (closed): tracked the identical terminal signature attributed to session-engine exits typing validator rejections as protocol faults; fixed and deployed, so its recurrence under the deployed fix bounds but does not explain the new fast-fail shape.
- PR Route memory workflows through independent cascades #98 (closed): the cascade merge that introduced the guarded convergence pass, the once-only dialog-fallback marker and the 5-hop/180s dialog normalization (verified by pickaxe: commit 'Route memory workflows through independent cascades' introduced the marker); the guard's no-reapply behavior is central to the unhealed-budget hypothesis.
- Issue [configured resource]-draw attempts exhaust again: both models of the single draw provider report provider_unavailable in bursts; count 11 to 13, no code defect observed #279 (closed): an earlier attempts-exhausted signature on a different workflow attributed to provider-side unavailability on both models of a single provider; shows the terminal event is shared by several provider-outage funnels, so the empty-route event alone is not cause-specific.
- Issue Re-sample a rejected model before falling back to one without tools #151 (closed): re-sampling a rejected model before falling back underpins the deployed validator-rejection handling that makes the prior funnel an unlikely explanation for this occurrence's timing.
Missing
- Per-attempt failure reasons or error strings for the twelve attempts (the error field distinguishes provider-response classes from request-construction and validation classes); the scoped evidence tool retains no error strings.
- The live dialog workflow row's current retry_max_hops and retry_wall_ms values, and the circuit-breaker/busy/disabled state of the three lanes during the window — either would separate the hop-cap story from the outage story.
- Retry-stage job events (requeue round timestamps and last_retryable_reason) for this occurrence, which carry the terminal reason class that the snapshot omits.
- Whether the external-route attempts failed before model selection (they carry no model field), which the retained capture cannot answer.
Acceptance
- Deep stage: recover or re-capture one occurrence with per-attempt reasons plus the live workflow row; the causal story must reproduce the exact four-round, three-attempt, empty-route terminal shape from code plus observed state.
- If the reverted-budget defect is confirmed: after restoring a 5-hop dialog budget with the fallback tail enabled, a synthetic dialog task whose three primary lanes all fail fast must start fallback-tail attempts inside one pass instead of terminating after four rounds; the regression check would assert the pass continues past hop 3 while fallback assignments remain unattempted.
- If an outage-only story is confirmed: the same synthetic replay with healthy lanes succeeds and no code change is justified; the terminal event then attributes to provider-response classes with per-attempt reasons visible.
Impact
67 captured terminal events.
Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.
Activity