feat(buzz-agent): fail over to the next provider on quota/auth/5xx - #4551
feat(buzz-agent): fail over to the next provider on quota/auth/5xx#4551BarrettHolien wants to merge 3 commits into
Conversation
A single provider's quota wall took every managed agent offline: Config
held one provider, so a 429 (z.ai's weekly `1310`) flattened to
AgentError::Llm and killed the turn with nowhere else to go.
Adds an ordered failover chain. Config gains `fallback: Vec<Endpoint>`,
auto-derived from the provider slots the environment already defines
(Ollama Cloud, then OpenRouter), overridable with
BUZZ_AGENT_FALLBACK_PROVIDERS and disabled with `none`. Llm::complete
walks that chain, retrying the SAME turn on the next endpoint.
Failures are split into two classes. Quota (429/402), auth (401/403
after the token source spent its refresh), upstream 5xx/499, and
transport are cutover-class: another provider may not share them. A 400
or an unknown model is stop-class and returns immediately — it fails
identically everywhere, so walking the chain would only reach the same
answer more slowly. The class is carried by a new
AgentError::LlmUnavailable { kind, detail } at JSON-RPC -32003.
An in-process circuit breaker (health.rs, ported from cortex-discord's
lib/health.js) benches an endpoint after two consecutive cutover
failures, for 5min on quota/auth and 60s on 5xx/transport, doubling to a
15min cap. Without it every turn of a days-long quota outage re-pays for
discovering the wall. It fails open: if every endpoint is benched the
whole chain is tried anyway.
Two things that are easy to get wrong and are covered by tests:
- Each fallback answers with its OWN slot's model. BUZZ_AGENT_MODEL
applies to the primary only; sending `glm-5.2` to OpenRouter 404s.
- Each endpoint gets its own TokenSource. `Llm::auth` holds the
primary's credential, so a cutover reusing it would authenticate the
wrong provider and 401 every time.
A deployment with no fallback configured is unchanged, down to the
error text.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Barrett Holien <17663953+BarrettHolien@users.noreply.github.com>
…copy buzz-agent's failover chain returns AgentError::LlmUnavailable at -32003 when every configured provider failed. Without a case for it the desktop showed the raw message, which reads as an unexplained agent crash rather than "your providers are exhausted, add another to the chain". This surfaces ONLY when the whole chain is down. A turn a fallback rescued returns Ok and never reaches this path. Retargets the "unrecognized structured code" fixture from -32003 to -32004 — that test needs a code nothing claims, and -32003 is now taken. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Barrett Holien <17663953+BarrettHolien@users.noreply.github.com>
Summarization drives handoff and context compaction. Leaving it on the primary alone meant a quota wall still stranded a long conversation at the context ceiling — the same outage the completion path now survives, arriving by another door. Extracts the chain walk both paths now share: `attempt_chain` builds the ordered endpoints (primary on the caller's effective model, each fallback on its own slot's model), drops the ones the breaker has benched unless that would leave nothing, and hands back a Config aimed at each. `chain_exhausted` words the final error identically for both. The breaker is shared, so a primary benched by failed turns is skipped by summarization too rather than each path paying to rediscover the outage. Also pins BUZZ_AGENT_FALLBACK_PROVIDERS=none in the integration harness. `Command::env` adds to the inherited environment rather than replacing it, so on a box exporting a real provider key AND model these tests would auto-derive a chain and answer from a live provider the moment a canned response looked cutover-class — nondeterministic, and billable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Barrett Holien <17663953+BarrettHolien@users.noreply.github.com>
|
Pushed a third commit: failover now covers Summarization drives handoff and context compaction, so leaving it on the primary alone meant a quota wall still stranded a long conversation at the context ceiling — the same outage this PR exists to survive, arriving by another door. Both paths now share one walk. Also pins Note on a pre-existing flaky testWhile verifying, This is not from this PR. It reproduces on pristine |
Problem
A single provider's quota wall takes every managed agent offline.
Configholds one provider, so a 429 — z.ai's weekly1310, for instance — flattens toAgentError::Llm, becomes JSON-RPC-32000, and the turn dies with nowhere else to go. The desktop then shows a raw error that reads like an agent crash rather than an exhausted quota.What this does
Adds an ordered failover chain.
Configgainsfallback: Vec<Endpoint>, auto-derived from the provider slots the environment already defines (Ollama Cloud, then OpenRouter), overridable withBUZZ_AGENT_FALLBACK_PROVIDERSand disabled withnone.Llm::completewalks that chain, retrying the same turn on the next endpoint.Failures split into two classes:
A malformed request fails identically on every provider, so walking the chain would only reach the same answer more slowly while burning the fallbacks. The class rides on a new
AgentError::LlmUnavailable { kind, detail }at-32003, which the desktop maps to actionable copy.An in-process circuit breaker (
health.rs) benches an endpoint after two consecutive cutover failures — 5min for quota/auth, 60s for 5xx/transport, doubling to a 15min cap. Without it every turn of a multi-day quota outage re-pays for discovering the wall. It fails open: if every endpoint is benched, the whole chain is tried anyway, so stale breaker state can never be the reason a turn has nowhere to go.Two things that are easy to get wrong
Both are covered by tests, because both fail silently:
BUZZ_AGENT_MODELapplies to the primary only — sendingglm-5.2to OpenRouter is a guaranteed 404.TokenSource.Llm::authholds the primary's credential, so a cutover that reused it would authenticate the wrong provider and 401 every single time, which looks exactly like a bad fallback key.Compatibility
A deployment with no fallback configured is unchanged, down to the error text — there's a test pinning that. Existing error messages keep their wording; only the variant carrying them changed, so
-32003is purely additive.Testing
buzz-agent(31 new: 15 chain-derivation, 10 breaker state-machine, 6 end-to-end cutover against stub servers)cargo fmt --check,cargo clippy --all-targets,cargo check --workspace --all-targetsall cleanThe end-to-end tests assert the things worth asserting: that the fallback is asked for its own model, that a 400 never contacts the fallback, and that a benched primary stops being retried by the third turn.