Skip to content

feat(buzz-agent): fail over to the next provider on quota/auth/5xx - #4551

Open
BarrettHolien wants to merge 3 commits into
block:mainfrom
HolienTech:feat/agent-provider-failover
Open

feat(buzz-agent): fail over to the next provider on quota/auth/5xx#4551
BarrettHolien wants to merge 3 commits into
block:mainfrom
HolienTech:feat/agent-provider-failover

Conversation

@BarrettHolien

Copy link
Copy Markdown

Problem

A single provider's quota wall takes every managed agent offline. Config holds one provider, so a 429 — z.ai's weekly 1310, for instance — flattens to AgentError::Llm, becomes JSON-RPC -32000, and the turn dies with nowhere else to go. The desktop then shows a raw error that reads like an agent crash rather than an exhausted quota.

What this does

Adds an ordered failover chain. Config gains fallback: Vec<Endpoint>, auto-derived from the provider slots the environment already defines (Ollama Cloud, then OpenRouter), overridable with BUZZ_AGENT_FALLBACK_PROVIDERS and disabled with none. Llm::complete walks that chain, retrying the same turn on the next endpoint.

Failures split into two classes:

class statuses behavior
cutover 429/402 quota, 401/403 auth (after the token source spent its refresh), 5xx/499, transport try the next provider
stop 400, unknown model return immediately

A malformed request fails identically on every provider, so walking the chain would only reach the same answer more slowly while burning the fallbacks. The class rides on a new AgentError::LlmUnavailable { kind, detail } at -32003, which the desktop maps to actionable copy.

An in-process circuit breaker (health.rs) benches an endpoint after two consecutive cutover failures — 5min for quota/auth, 60s for 5xx/transport, doubling to a 15min cap. Without it every turn of a multi-day quota outage re-pays for discovering the wall. It fails open: if every endpoint is benched, the whole chain is tried anyway, so stale breaker state can never be the reason a turn has nowhere to go.

Two things that are easy to get wrong

Both are covered by tests, because both fail silently:

  • Each fallback answers with its own slot's model. BUZZ_AGENT_MODEL applies to the primary only — sending glm-5.2 to OpenRouter is a guaranteed 404.
  • Each endpoint gets its own TokenSource. Llm::auth holds the primary's credential, so a cutover that reused it would authenticate the wrong provider and 401 every single time, which looks exactly like a bad fallback key.

Compatibility

A deployment with no fallback configured is unchanged, down to the error text — there's a test pinning that. Existing error messages keep their wording; only the variant carrying them changed, so -32003 is purely additive.

Testing

  • 498 tests pass in buzz-agent (31 new: 15 chain-derivation, 10 breaker state-machine, 6 end-to-end cutover against stub servers)
  • 34 desktop tests pass for the error-copy mapping
  • cargo fmt --check, cargo clippy --all-targets, cargo check --workspace --all-targets all clean

The end-to-end tests assert the things worth asserting: that the fallback is asked for its own model, that a 400 never contacts the fallback, and that a benched primary stops being retried by the third turn.

BarrettHolien and others added 2 commits August 3, 2026 10:48
A single provider's quota wall took every managed agent offline: Config
held one provider, so a 429 (z.ai's weekly `1310`) flattened to
AgentError::Llm and killed the turn with nowhere else to go.

Adds an ordered failover chain. Config gains `fallback: Vec<Endpoint>`,
auto-derived from the provider slots the environment already defines
(Ollama Cloud, then OpenRouter), overridable with
BUZZ_AGENT_FALLBACK_PROVIDERS and disabled with `none`. Llm::complete
walks that chain, retrying the SAME turn on the next endpoint.

Failures are split into two classes. Quota (429/402), auth (401/403
after the token source spent its refresh), upstream 5xx/499, and
transport are cutover-class: another provider may not share them. A 400
or an unknown model is stop-class and returns immediately — it fails
identically everywhere, so walking the chain would only reach the same
answer more slowly. The class is carried by a new
AgentError::LlmUnavailable { kind, detail } at JSON-RPC -32003.

An in-process circuit breaker (health.rs, ported from cortex-discord's
lib/health.js) benches an endpoint after two consecutive cutover
failures, for 5min on quota/auth and 60s on 5xx/transport, doubling to a
15min cap. Without it every turn of a days-long quota outage re-pays for
discovering the wall. It fails open: if every endpoint is benched the
whole chain is tried anyway.

Two things that are easy to get wrong and are covered by tests:

- Each fallback answers with its OWN slot's model. BUZZ_AGENT_MODEL
  applies to the primary only; sending `glm-5.2` to OpenRouter 404s.
- Each endpoint gets its own TokenSource. `Llm::auth` holds the
  primary's credential, so a cutover reusing it would authenticate the
  wrong provider and 401 every time.

A deployment with no fallback configured is unchanged, down to the
error text.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Barrett Holien <17663953+BarrettHolien@users.noreply.github.com>
…copy

buzz-agent's failover chain returns AgentError::LlmUnavailable at -32003
when every configured provider failed. Without a case for it the desktop
showed the raw message, which reads as an unexplained agent crash rather
than "your providers are exhausted, add another to the chain".

This surfaces ONLY when the whole chain is down. A turn a fallback
rescued returns Ok and never reaches this path.

Retargets the "unrecognized structured code" fixture from -32003 to
-32004 — that test needs a code nothing claims, and -32003 is now taken.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Barrett Holien <17663953+BarrettHolien@users.noreply.github.com>
@BarrettHolien
BarrettHolien requested a review from a team as a code owner August 3, 2026 15:50
Summarization drives handoff and context compaction. Leaving it on the
primary alone meant a quota wall still stranded a long conversation at
the context ceiling — the same outage the completion path now survives,
arriving by another door.

Extracts the chain walk both paths now share: `attempt_chain` builds the
ordered endpoints (primary on the caller's effective model, each fallback
on its own slot's model), drops the ones the breaker has benched unless
that would leave nothing, and hands back a Config aimed at each.
`chain_exhausted` words the final error identically for both.

The breaker is shared, so a primary benched by failed turns is skipped by
summarization too rather than each path paying to rediscover the outage.

Also pins BUZZ_AGENT_FALLBACK_PROVIDERS=none in the integration harness.
`Command::env` adds to the inherited environment rather than replacing
it, so on a box exporting a real provider key AND model these tests would
auto-derive a chain and answer from a live provider the moment a canned
response looked cutover-class — nondeterministic, and billable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Barrett Holien <17663953+BarrettHolien@users.noreply.github.com>
@BarrettHolien

Copy link
Copy Markdown
Author

Pushed a third commit: failover now covers summarize, not just complete.

Summarization drives handoff and context compaction, so leaving it on the primary alone meant a quota wall still stranded a long conversation at the context ceiling — the same outage this PR exists to survive, arriving by another door.

Both paths now share one walk. attempt_chain builds the ordered endpoints (primary on the caller's effective model, each fallback on its own slot's model), drops the ones the breaker has benched unless that would leave nothing, and returns a Config aimed at each; chain_exhausted words the terminal error identically for both. Because the breaker is shared, a primary benched by failed turns is skipped by summarization too, rather than each path paying to rediscover the same outage.

Also pins BUZZ_AGENT_FALLBACK_PROVIDERS=none in the fake_llm integration harness. Command::env adds to the inherited environment rather than replacing it, so on a machine exporting a real provider key and model, those tests would auto-derive a chain and answer from a live provider the moment a canned response looked cutover-class — nondeterministic, and billable.

Note on a pre-existing flaky test

While verifying, fake_llm.rs intermittently failed under parallel load — cancelled_turn_with_usage_emits_notification_before_response and steer_folds_into_active_turn_without_cancelling, roughly 1 run in 3.

This is not from this PR. It reproduces on pristine 5e0efb0 with none of these changes, and still occurs here with failover pinned off; agent.rs and wire.rs are untouched by this branch. Flagging it in case it shows up in CI here — happy to open a separate issue if useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant