Skip to content

Python: [Feature]: Model failover across chat clients that never re-runs tools (parity with .NET FailoverChatClient) #10332

Description

Problem

Python has no supported way to fail over from one model or provider to another when a call fails. Common triggers are a 429, a 5xx or "overloaded" response, a regional outage, or a capacity limit on an Azure OpenAI PTU deployment. .NET already has this: Microsoft.Extensions.AI 10.9+ ships FailoverChatClient and OrderedFailoverChatClient (dotnet/extensions#7662), and the .NET Agent Framework builds on them (#7641, sample Agent_Step24_MultiModelRouting). Python has nothing comparable.

Writing it yourself is not safe in Python. Every shipped client mixes FunctionInvocationLayer into the client class:

class OpenAIChatClient(FunctionInvocationLayer, ChatMiddlewareLayer, ChatTelemetryLayer, RawOpenAIChatClient): ...

So the obvious wrapper ("try client A, if it raises try client B") fails over the whole tool loop, not the single model call that failed. If A has already executed tools and then fails on its next model call, B starts again from the original messages and executes the same tools a second time.

Minimal repro against main (scripted clients, no network):

SENT = []

@tool(approval_mode="never_require")
def send_email(to: str) -> str:
    SENT.append(to); return f"sent to {to}"

class ScriptedClient(FunctionInvocationLayer, BaseChatClient):
    # model call 1 -> asks for send_email; model call 2 -> answers
    # primary is configured to raise ChatClientException("429") on model call 2
    ...

class NaiveFallbackClient:
    async def get_response(self, messages, **kw):
        for c in self.clients:
            try: return await c.get_response(messages, **kw)
            except ChatClientException: continue

await NaiveFallbackClient(primary, fallback).get_response(msgs, options={"tools": [send_email]})

Output:

  failover: primary: 429 Too Many Requests
final: done via fallback
send_email executions: 2 ['cfo@contoso.com', 'cfo@contoso.com']

The caller sees a successful response, but the side effect ran twice. This is the same class of problem as #9031 and #10172, but here the user's own resilience code causes it.

A second gap: there is no provider-neutral way to decide whether an error should trigger failover. Clients raise a plain ChatClientException with no status code and no retryable flag. Callers have to unwrap the provider SDK exception (openai.RateLimitError, Anthropic status_code, and so on) per provider. #9904 showed that even the shipped auto_retry.py sample got this wrong.

Use Case and Evidence

  • Azure OpenAI capacity and quota: overflow from a PTU deployment to pay-as-you-go, or to a second region, when the first returns 429/503. This is standard production guidance for Azure OpenAI customers, and agents with tools are exactly the workloads that hit these limits.
  • Provider outages: overflow from OpenAI to Anthropic or Bedrock (or the reverse) during an incident, without stopping a long-running harness or loop agent.
  • Cost tiers: try a small or local model first and fall back to a larger hosted model on failure.
  • Peer frameworks treat this as basic infrastructure: Pydantic AI FallbackModel (with fallback_on), LangChain ModelFallbackMiddleware (which runs per model call inside the agent loop), LiteLLM Router fallbacks, and MEAI FailoverChatClient in .NET.
  • Existing Python MAF content (for example, community posts on an "SLM-default, LLM-fallback" pattern) builds fallback by hand with workflow edges, at the run level, so the duplication described above still applies.

Desired Outcome

A Python developer can configure an ordered set of chat clients so that:

  1. A qualifying failure on one model call is retried on the next client for that same model call. The tool loop continues with the tool results already produced, and no tool runs twice.
  2. Streaming fails over only before the first update reaches the caller. After that, the error surfaces (the same rule MEAI uses).
  3. Cancellation never triggers failover.
  4. What counts as a qualifying failure (rate limit, server error or unavailable, timeout, optionally content filter) can be decided without importing provider SDK exception types.
  5. Telemetry records each attempt and which client finally served the call.
  6. Using clients that keep service-side conversation state (conversation_id / previous_response_id) is either rejected or clearly documented, because that history cannot move between providers. .NET documents the same constraint for routing.

Alternatives Considered

Proposed Direction

Open to maintainer preference:

  • Do failover at the per-model-call layer, below FunctionInvocationLayer. One possible shape: a FailoverChatClient([client_a, client_b], should_failover=...) that owns a single function-invocation loop and calls only the inner, non-tool-looping path of each candidate. Another: a chat middleware like LangChain's, since ChatMiddlewareLayer already runs per model call.
  • Use semantics close to MEAI's FailoverChatClient / OrderedFailoverChatClient (attempt reporting, failover only before output is committed), so .NET and Python behave the same.
  • A possibly separate, independently useful piece: give ChatClientException provider-neutral classification, for example status_code, is_transient/retry_after, or subclasses such as a rate-limit or service-unavailable exception, populated by the first-party clients. This would also simplify the retry guidance from Python: Add typical rate limiting handling for model clients #1362 without the framework having to own a retry policy.
  • Mark it experimental.

Scope and Non-Goals

Related: #1362, #6865, #7641, #9904, #10068, #9031, #10172, dotnet/extensions#7647, dotnet/extensions#7662

Language

Python

Contribution Intent

I am willing to implement this

Acknowledgements

  • I searched existing issues and did not find a duplicate.
  • I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 13, 2026
  2. changed the title [-][Feature]: Model failover across chat clients that never re-runs tools (parity with .NET FailoverChatClient)[/-] [+]Python: [Feature]: Model failover across chat clients that never re-runs tools (parity with .NET FailoverChatClient)[/+] on Nov 13, 2026
  3. eavanvalkenburg commented on Nov 16, 2026

    @eavanvalkenburg
    Member

    /plan The way we think about this is that is the type of scenario the Raw...Client is meant for, so we should start with exploring what that looks like, potentially as a sample that demonstrated how you can built the failover layer and then wrap it in the function-invocation layer and use it

  4. github-actions commented on Nov 16, 2026

    @github-actions
    Contributor

    Summary

    The issue asks for model failover in Python: if one model call fails (429, 5xx, outage), retry that same call on the next client. Wrapping whole shipped clients re-runs the tool loop, so tools can execute twice. The issue also asks for provider-neutral error classification.

    Recommendation

    Following your direction: start with a sample, not a new framework class. Code changes stay a possible follow-up once the sample shows what is missing.

    Why this fits:

    • The shipped clients are Raw...Client classes (for example RawOpenAIChatClient, RawAnthropicClient, RawGeminiChatClient) with the tool loop added on top by FunctionInvocationLayer. The Raw ones make one model call and do not loop on tools.
    • So a failover client that holds several Raw clients, wrapped once in FunctionInvocationLayer, would fail over one model call at a time. Tool results already produced would stay. That is the behaviour the issue asks for.
    • custom_chat_client.py already shows how to compose FunctionInvocationLayer, ChatMiddlewareLayer and ChatTelemetryLayer around a custom BaseChatClient. It has no failover example. I found no existing failover sample.
    • I did not run anything. The claim that this avoids double tool execution is untested.

    Trade-offs and open points the sample should settle:

    • The issue notes that using Raw clients loses each client's own middleware and telemetry. The sample should show that the outer layers cover this, and say what is lost.
    • Error classification is the harder part. ChatClientException carries no status code, so the sample will probably unwrap provider SDK errors itself. That would show whether a neutral error type is worth adding to the framework.
    • Service-side conversation state (conversation_id, previous_response_id) cannot move between providers. The sample should reject or document this.

    Proposed plan

    This is a proposal; nothing has been changed.

    1. Read how FunctionInvocationLayer calls the inner client (_tools.py, _clients.py). Confirm that an inner client whose _inner_get_response loops over several Raw clients is called once per model call.
    2. Add a sample under python/samples/02-agents/chat_client/. It has a FailoverRawClient (a BaseChatClient subclass) that tries Raw clients in order. A should_failover callable decides which errors qualify. Cancellation never triggers failover. For streaming, it fails over only before the first update is yielded. Compose it as FunctionInvocationLayer -> ChatMiddlewareLayer -> ChatTelemetryLayer -> FailoverRawClient, and add it to the README.
    3. Use scripted fake clients in the sample's demo path so it runs without network. Make the first client fail on its second model call. Show that a tool with a side effect runs once. This is the check that resolves the issue's repro.
    4. Log each attempt and which client served the call.
    5. After the sample exists, decide on follow-ups. Examples are a supported FailoverChatClient and a provider-neutral error classification on ChatClientException. Those would need a separate ADR or maintainer agreement, as the issue's acknowledgements say.

    Done when the sample runs, the tool executes once despite a mid-loop failure, streaming and cancellation behave as described, and the README explains the conversation-state limit.

    Generated by Issue Planning for #10332 · copilot · auto · ~18 AIC · ⌖ ~1 AIC · ⊞ 4.8K · ◷

  5. karthik-0306 commented on Nov 16, 2026

    @karthik-0306
    ContributorAuthor

    Thanks Eduard van Valkenburg (@eavanvalkenburg), agreed: a sample over Raw...Clients avoids the layering seam I used in #10364.

    I checked the approach with two RawOpenAIChatClients inside FunctionInvocationLayer (SDK call mocked; the primary returns a tool call, then a RateLimitError). In both streaming and non-streaming mode the tool runs once and the fallback answers.

    One catch the sample must handle: RawOpenAIChatClient (and so RawFoundryChatClient) has STORES_BY_DEFAULT = True. Without sending store=False, the fallback receives only the tool result plus the primary's previous_response_id, not the conversation. With store=False, it receives the full conversation, including earlier turns. A second limit to document: with store=False, OpenAI rejects reasoning produced by another provider before sending the request, so the sample should fail over between deployments of the same provider.

    Shall I rework #10364 into that sample (offline scripted demo, tool runs once, streaming and cancellation rules, README)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

agentsUsage: [Issues, PRs], Target: Single agentpythonUsage: [Issues, PRs], Target: PythontriageUsage: [Issues], Target: All issues that still need to be triaged

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions