Repository navigation
Description
Problem
Python has no supported way to fail over from one model or provider to another when a call fails. Common triggers are a 429, a 5xx or "overloaded" response, a regional outage, or a capacity limit on an Azure OpenAI PTU deployment. .NET already has this: Microsoft.Extensions.AI 10.9+ ships FailoverChatClient and OrderedFailoverChatClient (dotnet/extensions#7662), and the .NET Agent Framework builds on them (#7641, sample Agent_Step24_MultiModelRouting). Python has nothing comparable.
Writing it yourself is not safe in Python. Every shipped client mixes FunctionInvocationLayer into the client class:
class OpenAIChatClient(FunctionInvocationLayer, ChatMiddlewareLayer, ChatTelemetryLayer, RawOpenAIChatClient): ...So the obvious wrapper ("try client A, if it raises try client B") fails over the whole tool loop, not the single model call that failed. If A has already executed tools and then fails on its next model call, B starts again from the original messages and executes the same tools a second time.
Minimal repro against main (scripted clients, no network):
SENT = []
@tool(approval_mode="never_require")
def send_email(to: str) -> str:
SENT.append(to); return f"sent to {to}"
class ScriptedClient(FunctionInvocationLayer, BaseChatClient):
# model call 1 -> asks for send_email; model call 2 -> answers
# primary is configured to raise ChatClientException("429") on model call 2
...
class NaiveFallbackClient:
async def get_response(self, messages, **kw):
for c in self.clients:
try: return await c.get_response(messages, **kw)
except ChatClientException: continue
await NaiveFallbackClient(primary, fallback).get_response(msgs, options={"tools": [send_email]})Output:
failover: primary: 429 Too Many Requests
final: done via fallback
send_email executions: 2 ['cfo@contoso.com', 'cfo@contoso.com']
The caller sees a successful response, but the side effect ran twice. This is the same class of problem as #9031 and #10172, but here the user's own resilience code causes it.
A second gap: there is no provider-neutral way to decide whether an error should trigger failover. Clients raise a plain ChatClientException with no status code and no retryable flag. Callers have to unwrap the provider SDK exception (openai.RateLimitError, Anthropic status_code, and so on) per provider. #9904 showed that even the shipped auto_retry.py sample got this wrong.
Use Case and Evidence
- Azure OpenAI capacity and quota: overflow from a PTU deployment to pay-as-you-go, or to a second region, when the first returns 429/503. This is standard production guidance for Azure OpenAI customers, and agents with tools are exactly the workloads that hit these limits.
- Provider outages: overflow from OpenAI to Anthropic or Bedrock (or the reverse) during an incident, without stopping a long-running harness or loop agent.
- Cost tiers: try a small or local model first and fall back to a larger hosted model on failure.
- Peer frameworks treat this as basic infrastructure: Pydantic AI
FallbackModel(withfallback_on), LangChainModelFallbackMiddleware(which runs per model call inside the agent loop), LiteLLM Router fallbacks, and MEAIFailoverChatClientin .NET. - Existing Python MAF content (for example, community posts on an "SLM-default, LLM-fallback" pattern) builds fallback by hand with workflow edges, at the run level, so the duplication described above still applies.
Desired Outcome
A Python developer can configure an ordered set of chat clients so that:
- A qualifying failure on one model call is retried on the next client for that same model call. The tool loop continues with the tool results already produced, and no tool runs twice.
- Streaming fails over only before the first update reaches the caller. After that, the error surfaces (the same rule MEAI uses).
- Cancellation never triggers failover.
- What counts as a qualifying failure (rate limit, server error or unavailable, timeout, optionally content filter) can be decided without importing provider SDK exception types.
- Telemetry records each attempt and which client finally served the call.
- Using clients that keep service-side conversation state (
conversation_id/previous_response_id) is either rejected or clearly documented, because that history cannot move between providers. .NET documents the same constraint for routing.
Alternatives Considered
auto_retry.py(tenacity + ChatMiddleware): retries the same model. It doesn't help with an outage or quota exhaustion, and it relies on provider-specific exception types (Python: [Bug]: auto_retry.py sample never retries a 429: the client raises ChatClientException, not RateLimitError #9904, Python: Fix the auto_retry sample so it retries 429 errors #10068).- Wrapping full clients: re-executes tools (see the repro in the Problem section).
- Composing
Raw*clients under a hand-builtFunctionInvocationLayer: possible, but undocumented. It also loses per-client middleware and telemetry, and every user has to rediscover the right layering. - Workflow conditional edges between agents: runs at the run level, so the same duplication applies, and it's heavy for a resilience concern.
Proposed Direction
Open to maintainer preference:
- Do failover at the per-model-call layer, below
FunctionInvocationLayer. One possible shape: aFailoverChatClient([client_a, client_b], should_failover=...)that owns a single function-invocation loop and calls only the inner, non-tool-looping path of each candidate. Another: a chat middleware like LangChain's, sinceChatMiddlewareLayeralready runs per model call. - Use semantics close to MEAI's
FailoverChatClient/OrderedFailoverChatClient(attempt reporting, failover only before output is committed), so .NET and Python behave the same. - A possibly separate, independently useful piece: give
ChatClientExceptionprovider-neutral classification, for examplestatus_code,is_transient/retry_after, or subclasses such as a rate-limit or service-unavailable exception, populated by the first-party clients. This would also simplify the retry guidance from Python: Add typical rate limiting handling for model clients #1362 without the framework having to own a retry policy. - Mark it experimental.
Scope and Non-Goals
- Not built-in retry or backoff for a single client. That stays user-owned, following the decision in Python: Add typical rate limiting handling for model clients #1362.
- Not semantic or content-based routing, and not manual mid-conversation route switching. Python parity with
RoutePersistingRoutingChatClient([Feature]: Allow switching from provider A to provider B without losing the chat history to the new provider #6865 / .NET: Add session-persisted chat client routing #7641) could be a follow-up. - No .NET changes. .NET is covered by MEAI.
Related: #1362, #6865, #7641, #9904, #10068, #9031, #10172, dotnet/extensions#7647, dotnet/extensions#7662
Language
Python
Contribution Intent
I am willing to implement this
Acknowledgements
- I searched existing issues and did not find a duplicate.
- I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.
Metadata
Metadata
Labels
Type
Projects
- StatusShow more project fieldsNo status
Activity
[-][Feature]: Model failover across chat clients that never re-runs tools (parity with .NET FailoverChatClient)[/-][+]Python: [Feature]: Model failover across chat clients that never re-runs tools (parity with .NET FailoverChatClient)[/+]eavanvalkenburg commentedon Nov 16, 2026
/plan The way we think about this is that is the type of scenario the
Raw...Clientis meant for, so we should start with exploring what that looks like, potentially as a sample that demonstrated how you can built the failover layer and then wrap it in the function-invocation layer and use itgithub-actions commentedon Nov 16, 2026
Summary
The issue asks for model failover in Python: if one model call fails (429, 5xx, outage), retry that same call on the next client. Wrapping whole shipped clients re-runs the tool loop, so tools can execute twice. The issue also asks for provider-neutral error classification.
Recommendation
Following your direction: start with a sample, not a new framework class. Code changes stay a possible follow-up once the sample shows what is missing.
Why this fits:
Raw...Clientclasses (for exampleRawOpenAIChatClient,RawAnthropicClient,RawGeminiChatClient) with the tool loop added on top byFunctionInvocationLayer. TheRawones make one model call and do not loop on tools.Rawclients, wrapped once inFunctionInvocationLayer, would fail over one model call at a time. Tool results already produced would stay. That is the behaviour the issue asks for.custom_chat_client.pyalready shows how to composeFunctionInvocationLayer,ChatMiddlewareLayerandChatTelemetryLayeraround a customBaseChatClient. It has no failover example. I found no existing failover sample.Trade-offs and open points the sample should settle:
Rawclients loses each client's own middleware and telemetry. The sample should show that the outer layers cover this, and say what is lost.ChatClientExceptioncarries no status code, so the sample will probably unwrap provider SDK errors itself. That would show whether a neutral error type is worth adding to the framework.conversation_id,previous_response_id) cannot move between providers. The sample should reject or document this.Proposed plan
This is a proposal; nothing has been changed.
FunctionInvocationLayercalls the inner client (_tools.py,_clients.py). Confirm that an inner client whose_inner_get_responseloops over severalRawclients is called once per model call.python/samples/02-agents/chat_client/. It has aFailoverRawClient(aBaseChatClientsubclass) that triesRawclients in order. Ashould_failovercallable decides which errors qualify. Cancellation never triggers failover. For streaming, it fails over only before the first update is yielded. Compose it asFunctionInvocationLayer -> ChatMiddlewareLayer -> ChatTelemetryLayer -> FailoverRawClient, and add it to the README.FailoverChatClientand a provider-neutral error classification onChatClientException. Those would need a separate ADR or maintainer agreement, as the issue's acknowledgements say.Done when the sample runs, the tool executes once despite a mid-loop failure, streaming and cancellation behave as described, and the README explains the conversation-state limit.
karthik-0306 commentedon Nov 16, 2026
Thanks Eduard van Valkenburg (@eavanvalkenburg), agreed: a sample over
Raw...Clients avoids the layering seam I used in #10364.I checked the approach with two
RawOpenAIChatClients insideFunctionInvocationLayer(SDK call mocked; the primary returns a tool call, then aRateLimitError). In both streaming and non-streaming mode the tool runs once and the fallback answers.One catch the sample must handle:
RawOpenAIChatClient(and soRawFoundryChatClient) hasSTORES_BY_DEFAULT = True. Without sendingstore=False, the fallback receives only the tool result plus the primary'sprevious_response_id, not the conversation. Withstore=False, it receives the full conversation, including earlier turns. A second limit to document: withstore=False, OpenAI rejects reasoning produced by another provider before sending the request, so the sample should fail over between deployments of the same provider.Shall I rework #10364 into that sample (offline scripted demo, tool runs once, streaming and cancellation rules, README)?