Repository navigation
Python: Add FailoverChatClient for per-model-call failover across chat clients - #10364
Karthik Thota (karthik-0306) wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Approval-resume requests can execute tools before forbidden storage or continuation options are rejected, while paths without a model call can bypass validation entirely.
5 open issues
What changed in this PR
Adds an experimental Python FailoverChatClient that retries individual model calls across ordered clients while keeping one tool-execution loop, addressing the duplicate-tool risk described in #10332.
Changes:
- Implements ordered failover, streaming output safeguards, and client-managed-history restrictions.
- Exports the client and registers its experimental feature identifier.
- Adds offline regression tests, contributor documentation, and a Foundry sample.
| File | Description |
|---|---|
python/samples/02-agents/chat_client/README.md |
Lists the failover sample and its prerequisites. |
python/samples/02-agents/chat_client/chat_client_failover.py |
Demonstrates failover between Foundry deployments. |
python/packages/core/tests/core/test_failover.py |
Tests tool execution, streaming, errors, history, middleware, and ownership. |
python/packages/core/AGENTS.md |
Documents failover behavior and constraints. |
python/packages/core/agent_framework/_feature_stage.py |
Registers CHAT_CLIENT_FAILOVER. |
python/packages/core/agent_framework/_failover.py |
Implements per-model-call failover. |
python/packages/core/agent_framework/__init__.pyi |
Adds the public typing export. |
python/packages/core/agent_framework/__init__.py |
Adds the lazy runtime export. |
Review effort: Balanced
Validate options before approval resume, propagate nested continuation names, omit other providers' reasoning per attempt, forward the caller session to candidate middleware, and close sample resources.
…orward sessions to nested failover clients
Nick (surlymochan)
left a comment
There was a problem hiding this comment.
Non-binding review (scoped: read _failover.py core, the docstring contract, and the test file list; the full 1593-line diff was not line-by-line reviewed. Not an approval).
What I verified:
- The constructor guards are in place: empty client list raises, the same client object twice raises, and entries without
get_response()raiseTypeError. - The failover contract is clearly stated: per model call (not per run), first client retried first on the next call, tool results are reused rather than re-executed, and
ResponseInvalidatedException/middleware control-flow exceptions always propagate.
Two questions:
_client_sourcehashes client type + model + service URL, but not credentials/tenant. Two clients pointed at the same model and URL with different API keys share one source id, so reasoning produced under one key is replayed to the other as if it were its own. Is that intended, or should the identity distinguish them?- The first client is retried first on every model call with no backoff or circuit-breaker. A flapping primary is therefore retried on every call of every run. Is that deliberate, or is a cooldown planned as a follow-up?
|
Nick (@surlymochan) Thanks for the careful read. Both questions are fair. 1. Source identity and credentials. Leaving credentials out is intentional. The identity is stored in conversation history, so it shouldn't contain anything derived from a secret. Many clients also authenticate with token providers rather than a static key. You're right that two accounts on the same model and URL (for example, two OpenAI organizations used for quota spillover) currently look like one source. The impact is limited:
If maintainers want this covered, I'd suggest letting callers name candidates explicitly, similar to named routes in .NET, rather than deriving identity from credentials. I'm happy to add that here or as a follow-up. 2. No cooldown. This is deliberate and matches MEAI's |


Motivation & Context
Production agents need to keep working when a model call fails with a 429, a 5xx, an overloaded response, or a regional outage. Python Agent Framework has no supported way to fail over to another deployment or provider. .NET gets this from
Microsoft.Extensions.AI(FailoverChatClient/OrderedFailoverChatClient, dotnet/extensions#7662).The obvious hand-written fallback is unsafe in Python. Every shipped client mixes
FunctionInvocationLayerinto the client class, so wrapping complete clients intry/exceptfails over the whole tool loop. If the primary runs a tool and then fails on its next model call, the fallback starts again from the original messages and runs the tool a second time. #10332 has a minimal repro againstmainin whichsend_emailruns twice while the caller sees a successful response.This PR adds failover at the per-model-call level, below the function-invocation loop, so tools that already ran are never re-executed.
Description & Review Guide
What are the major changes?
FailoverChatClient(agent_framework/_failover.py, feature IDCHAT_CLIENT_FAILOVER), exported fromagent_framework.FunctionInvocationLayer + ChatMiddlewareLayer + BaseChatClient. It owns the single tool loop, and its_inner_get_responsetries each candidate in order.FunctionInvocationLayer. That is the same entry the candidate's own loop uses for every model call. The candidate's chat middleware and telemetry run for each attempt (so each attempt gets its own span with the real model), and its tool loop does not.Exceptionfails over, matching MEAI's default, unlessshould_failover(exception)returnsFalse.ResponseInvalidatedExceptionandMiddlewareExceptionalways propagate, as spec 004 requires invalidation to propagate unchanged.store=True, or any service continuation option (the union of the candidates'SERVICE_CONTINUATION_OPTION_NAMES), raisesChatClientInvalidRequestExceptionbefore any client is called.STORES_BY_DEFAULT(for example, OpenAI Responses and Foundry) receivestore=Falsefor their attempt whenstoreis unset. Only those candidates get it, because providers such as Ollama and Mistral forward unknown option keys to their SDKs.conversation_idwould make the loop send only new tool results to a different provider.tests/core/test_failover.py, a sample atsamples/02-agents/chat_client/chat_client_failover.py, and a coreAGENTS.mdentry.What is the impact of these changes?
Additive and experimental. Existing clients and the function-calling loop are unchanged.
Before (hand-written fallback; re-runs tools on a later failure):
After:
Execution modes:
Ownership, defaults and placement:
function_invocation_configurationon the failover client or the agent. Function middleware and the tool loop of the candidate clients are bypassed.What do you want reviewers to focus on?
FunctionInvocationLayer(super(FunctionInvocationLayer, client).get_response) is an acceptable seam, or whether you'd prefer an explicit public "single model call" entry point, or a chat-middleware shape like LangChain'sModelFallbackMiddleware.store=Falseonly forSTORES_BY_DEFAULTcandidates.Exception, as MEAI does. Provider-neutral error classification (for example, a status code or transient flag onChatClientException) would makeshould_failovereasier to write. Python: [Feature]: Model failover across chat clients that never re-runs tools (parity with .NET FailoverChatClient) #10332 lists it as a possible separate follow-up, so it is not in this PR.Related Issue
Fixes #10332
Note: #10332 has not yet been marked
ready-for-implementation. I'm opening this as a draft reference implementation of the proposed direction and am happy to change its shape, split it, or close it in favour of a different approach, per maintainer preference. I found no other open PR for this issue.AI Assistance
AI assistance details: design investigation, implementation, tests, sample, documentation, and this PR description.
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.