Skip to content

Python: Add FailoverChatClient for per-model-call failover across chat clients - #10364

Draft
Karthik Thota (karthik-0306) wants to merge 3 commits into
microsoft:mainfrom
karthik-0306:feat/python-chat-client-failover
Draft

Karthik Thota (karthik-0306) wants to merge 3 commits into
microsoft:mainfrom
karthik-0306:feat/python-chat-client-failover

Conversation

@karthik-0306

@karthik-0306 Karthik Thota (karthik-0306) commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Motivation & Context

Production agents need to keep working when a model call fails with a 429, a 5xx, an overloaded response, or a regional outage. Python Agent Framework has no supported way to fail over to another deployment or provider. .NET gets this from Microsoft.Extensions.AI (FailoverChatClient / OrderedFailoverChatClient, dotnet/extensions#7662).

The obvious hand-written fallback is unsafe in Python. Every shipped client mixes FunctionInvocationLayer into the client class, so wrapping complete clients in try/except fails over the whole tool loop. If the primary runs a tool and then fails on its next model call, the fallback starts again from the original messages and runs the tool a second time. #10332 has a minimal repro against main in which send_email runs twice while the caller sees a successful response.

This PR adds failover at the per-model-call level, below the function-invocation loop, so tools that already ran are never re-executed.

Description & Review Guide

  • What are the major changes?

    • New experimental FailoverChatClient (agent_framework/_failover.py, feature ID CHAT_CLIENT_FAILOVER), exported from agent_framework.
      • It is FunctionInvocationLayer + ChatMiddlewareLayer + BaseChatClient. It owns the single tool loop, and its _inner_get_response tries each candidate in order.
      • Each candidate is called below its own FunctionInvocationLayer. That is the same entry the candidate's own loop uses for every model call. The candidate's chat middleware and telemetry run for each attempt (so each attempt gets its own span with the real model), and its tool loop does not.
      • It does not change the function-calling loop. The loop is reused as-is.
    • Failover rules:
      • Every model call starts with the first client again.
      • Any Exception fails over, matching MEAI's default, unless should_failover(exception) returns False.
      • Cancellation is never caught. ResponseInvalidatedException and MiddlewareException always propagate, as spec 004 requires invalidation to propagate unchanged.
      • When the last client fails, its exception propagates unchanged.
    • Streaming fails over only before an update with contents is released. Content-less updates (for example, response-created metadata) are held back and discarded if that attempt fails.
    • History must be client-managed:
      • store=True, or any service continuation option (the union of the candidates' SERVICE_CONTINUATION_OPTION_NAMES), raises ChatClientInvalidRequestException before any client is called.
      • Candidates with STORES_BY_DEFAULT (for example, OpenAI Responses and Foundry) receive store=False for their attempt when store is unset. Only those candidates get it, because providers such as Ollama and Mistral forward unknown option keys to their SDKs.
      • Without this, a provider-held conversation_id would make the loop send only new tool results to a different provider.
    • Candidates are caller-owned and are not closed.
    • Tests in tests/core/test_failover.py, a sample at samples/02-agents/chat_client/chat_client_failover.py, and a core AGENTS.md entry.
  • What is the impact of these changes?
    Additive and experimental. Existing clients and the function-calling loop are unchanged.

    Before (hand-written fallback; re-runs tools on a later failure):

    try:
        response = await primary.get_response(messages, options={"tools": [issue_refund]})
    except ChatClientException:
        response = await fallback.get_response(messages, options={"tools": [issue_refund]})

    After:

    from agent_framework import Agent, FailoverChatClient
    from agent_framework.foundry import FoundryChatClient
    
    client = FailoverChatClient([
        FoundryChatClient(project_endpoint=east_endpoint, model="gpt-4o", credential=credential),
        FoundryChatClient(project_endpoint=west_endpoint, model="gpt-4o", credential=credential),
    ])
    agent = Agent(client=client, tools=[issue_refund])

    Execution modes:

    • Non-streaming fails over on any qualifying error.
    • Streaming fails over only before content is released; after that, the error propagates. This is the same rule MEAI uses.

    Ownership, defaults and placement:

    • Configure tools, function middleware and function_invocation_configuration on the failover client or the agent. Function middleware and the tool loop of the candidate clients are bypassed.
    • Failover-level chat middleware runs once per model call, around all attempts. Candidate-level chat middleware runs once per attempt.
  • What do you want reviewers to focus on?

    • Whether calling a candidate below its FunctionInvocationLayer (super(FunctionInvocationLayer, client).get_response) is an acceptable seam, or whether you'd prefer an explicit public "single model call" entry point, or a chat-middleware shape like LangChain's ModelFallbackMiddleware.
    • The client-managed-history policy: rejecting service continuations, and adding store=False only for STORES_BY_DEFAULT candidates.
    • The default of failing over on every Exception, as MEAI does. Provider-neutral error classification (for example, a status code or transient flag on ChatClientException) would make should_failover easier to write. Python: [Feature]: Model failover across chat clients that never re-runs tools (parity with .NET FailoverChatClient) #10332 lists it as a possible separate follow-up, so it is not in this PR.
    • Validation is offline, with scripted clients that use the shipped client layering. I have not run the sample against a live Foundry project.

Related Issue

Fixes #10332

Note: #10332 has not yet been marked ready-for-implementation. I'm opening this as a draft reference implementation of the proposed direction and am happy to change its shape, split it, or close it in favour of a different approach, per maintainer preference. I found no other open PR for this issue.

AI Assistance

  • No material AI assistance was used.
  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: design investigation, implementation, tests, sample, documentation, and this PR description.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Approval-resume requests can execute tools before forbidden storage or continuation options are rejected, while paths without a model call can bypass validation entirely.

5 open issues
What changed in this PR

Adds an experimental Python FailoverChatClient that retries individual model calls across ordered clients while keeping one tool-execution loop, addressing the duplicate-tool risk described in #10332.

Changes:

  • Implements ordered failover, streaming output safeguards, and client-managed-history restrictions.
  • Exports the client and registers its experimental feature identifier.
  • Adds offline regression tests, contributor documentation, and a Foundry sample.
File Description
python/​samples/​02-agents/​chat_client/​README.md Lists the failover sample and its prerequisites.
python/​samples/​02-agents/​chat_client/​chat_client_failover.py Demonstrates failover between Foundry deployments.
python/​packages/​core/​tests/​core/​test_failover.py Tests tool execution, streaming, errors, history, middleware, and ownership.
python/​packages/​core/​AGENTS.md Documents failover behavior and constraints.
python/​packages/​core/​agent_framework/​_feature_stage.py Registers CHAT_CLIENT_FAILOVER.
python/​packages/​core/​agent_framework/​_failover.py Implements per-model-call failover.
python/​packages/​core/​agent_framework/​__init__.pyi Adds the public typing export.
python/​packages/​core/​agent_framework/​__init__.py Adds the lazy runtime export.

Review effort: Balanced

Validate options before approval resume, propagate nested continuation names, omit other providers' reasoning per attempt, forward the caller session to candidate middleware, and close sample resources.

@surlymochan Nick (surlymochan) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-binding review (scoped: read _failover.py core, the docstring contract, and the test file list; the full 1593-line diff was not line-by-line reviewed. Not an approval).

What I verified:

  • The constructor guards are in place: empty client list raises, the same client object twice raises, and entries without get_response() raise TypeError.
  • The failover contract is clearly stated: per model call (not per run), first client retried first on the next call, tool results are reused rather than re-executed, and ResponseInvalidatedException/middleware control-flow exceptions always propagate.

Two questions:

  1. _client_source hashes client type + model + service URL, but not credentials/tenant. Two clients pointed at the same model and URL with different API keys share one source id, so reasoning produced under one key is replayed to the other as if it were its own. Is that intended, or should the identity distinguish them?
  2. The first client is retried first on every model call with no backoff or circuit-breaker. A flapping primary is therefore retried on every call of every run. Is that deliberate, or is a cooldown planned as a follow-up?

@karthik-0306

Copy link
Copy Markdown
Contributor Author

Nick (@surlymochan) Thanks for the careful read. Both questions are fair.

1. Source identity and credentials. Leaving credentials out is intentional. The identity is stored in conversation history, so it shouldn't contain anything derived from a secret. Many clients also authenticate with token providers rather than a static key.

You're right that two accounts on the same model and URL (for example, two OpenAI organizations used for quota spillover) currently look like one source. The impact is limited:

  • Failover already sends the full transcript to every configured client, so this isn't a new data flow.
  • The likely outcome is that the second account rejects the first account's encrypted reasoning. That is just another failed attempt, so failover moves on, and only the last client's failure surfaces. I haven't verified the rejection against the live service.

If maintainers want this covered, I'd suggest letting callers name candidates explicitly, similar to named routes in .NET, rather than deriving identity from credentials. I'm happy to add that here or as a follow-up.

2. No cooldown. This is deliberate and matches MEAI's OrderedFailoverChatClient, which also starts every request at the first client, with no backoff or health tracking. MEAI leaves stateful policies to subclasses through SelectClientAsync and OnRoutingUpdateAsync. I'd rather follow that design with a selection/attempt hook as a follow-up than build in a fixed cooldown policy. In the meantime, a primary that hangs rather than fails fast costs its request timeout on each call, so the mitigation is to set a short timeout on each client: FoundryChatClient(timeout=...), or an AsyncOpenAI(timeout=...) passed as async_client to OpenAIChatClient (a direct timeout parameter for that client is in #9924).

This branch was successfully deployed

1 active deployment
github-app-auth — 7602faca Deployed Nov 14, 2026 by karthik-0306 via add_label #27871
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: [Feature]: Model failover across chat clients that never re-runs tools (parity with .NET FailoverChatClient)

3 participants