Skip to content

Python: [BREAKING] Do not replay an MCP tools/call whose response was lost - #9243

Draft
Yufeng He (he-yufeng) wants to merge 19 commits into
microsoft:mainfrom
he-yufeng:fix/mcp-call-no-replay-without-idempotency
Draft

Yufeng He (he-yufeng) wants to merge 19 commits into
microsoft:mainfrom
he-yufeng:fix/mcp-call-no-replay-without-idempotency

Conversation

@he-yufeng

@he-yufeng Yufeng He (he-yufeng) commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Motivation & Context

When the MCP connection drops during an ordinary tools/call, the request may already have executed server-side before the response was lost. The reconnect-and-retry added for #2884 re-issued the same call unconditionally on ClosedResourceError / session-terminated MCPError, so a non-idempotent operation (a charge, a write, a ticket creation) could run twice, and a successful replay hid the failure from the caller entirely.

Description & Review Guide

  • What are the major changes? _call_tool_with_retries dispatches once. On a connection loss it recovers the connection through the existing _recover_connection_after_loss helpers, then raises ToolExecutionException stating that the remote execution outcome is unknown and the call was not retried, with the connection error as __cause__, the connection error type on the span, and a redacted WARNING log. Server ToolAnnotations stay advisory: idempotentHint and readOnlyHint do not authorize a replay, and this adds no trust or idempotency option. tools/list, get_prompt, and caller-owned connection errors keep their current behavior.
  • What is the impact of these changes? A dropped tools/call now surfaces an explicit failure instead of a transparent retry. For framework-owned connections the connection is already restored when the exception surfaces, so a later, distinct call proceeds normally; caller-supplied Client / ClientSession objects are never recreated. This is a behavioral break on the error path, so the title carries [BREAKING] and packages/core/README.md documents the migration: catch the exception, reconcile the outcome, retry explicitly if safe. No exactly-once guarantee is claimed.
  • What do you want reviewers to focus on? Whether recovery-then-raise is the right ordering for framework-owned connections, and the concurrency expectations now pinned in test_simultaneous_connection_failures_share_one_recovery and test_ambiguous_call_is_not_replayed_after_a_concurrent_identity_change: both callers get the unknown-outcome failure, one replacement connection is created, and neither operation is re-issued.

The earlier revision of this PR gated replay on server hints plus a client trust_server_idempotency_hints opt-in. That option, _is_call_replayable, and the pre-dispatch eligibility hook are all removed here. The production diff is now confined to _call_tool_with_retries; _invoke_tool is back to its main signature.

Related Issue

Refs #9204

The maintainer planning comment on #9204 asks for no replay by default, recovery separated from the failed call, read-only retries preserved, and explicitly no new trust or idempotency setting. This revision follows that scope. #9204 still has no ready-for-implementation label, so this stays a draft and I am not requesting review; the direction question there is still open.

AI Assistance

  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: implementation, tests, documentation, and this PR description. I reviewed the final diff line by line and ran the suites and the local MCP integration tests reported below.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. The [BREAKING] title prefix declares the intentional error-path behavior change described above.
Local results on b6637210d

Python 3.14.2, MCP SDK 2.3.0, agent_framework resolving to this checkout.

  • test_mcp.py + test_mcp_http_auth.py + test_mcp_observability.py + test_security.py, non-integration: 1308 passed, 3 skipped, 2 deselected.
  • Both URL-backed integration tests against the repository's deterministic local MCP server on loopback (LOCAL_MCP_URL, no remote service): test_streamable_http_integration, test_mcp_connection_reset_integration — 2 passed.
  • Red check: with _mcp.py restored to main and these tests unchanged, 17 failed / 760 passed, and test_mcp_connection_reset_integration fails with DID NOT RAISE. Restoring the fix returns both runs to green.
  • Ruff 0.17.4 check on all changed Python files: passed. git diff --check: clean.
  • Ruff 0.17.4 format reports three findings in _mcp.py and one in scripts/local_mcp_streamable_http_server.py; all four are byte-identical on main and untouched here, so they are left alone rather than reformatting unrelated upstream code.
  • Pyright 1.1.414 over all 107 core source files: 2 errors, both the pre-existing PyYAML compose unknown-type findings in _skills.py. _mcp.py has no diagnostics. This is not a clean full-core typing result.

Copilot AI balanced review requested due to automatic review settings October 9, 2026 20:22
@agent-framework-automation agent-framework-automation Bot added documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs python Usage: [Issues, PRs], Target: Python labels Oct 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The annotation snapshot is not restored when connection discovery fails, allowing stale replay guarantees.

1 open finding
What changed in this PR

Prevents unsafe replay of MCP tool calls after connection loss unless the tool advertises read-only or idempotent behavior.

Changes:

  • Captures tool annotations and gates reconnect retries.
  • Adds regression tests for duplicate side effects.
  • Documents the replay policy.
File Description
python/​packages/​core/​agent_framework/​_mcp.py Implements annotation-based replay gating.
python/​packages/​core/​tests/​core/​test_mcp.py Tests replayable and non-replayable calls.
python/​packages/​core/​AGENTS.md Documents the new policy.

🧠 Review effort: Balanced


💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Yufeng He (he-yufeng) added a commit to he-yufeng/agent-framework that referenced this pull request Oct 17, 2026
…otency hints

ToolAnnotations are server-supplied and untrusted by default, so an
annotation alone must not authorize replaying an ambiguous tools/call.
Replay now additionally requires the client-owned
trust_server_idempotency_hints=True opt-in on the MCP tool; without it the
lost-response path stays no-replay and surfaces the unknown outcome.

Review thread: microsoft#9243 (comment)
@theblueskies

Copy link
Copy Markdown

One follow-up worth considering: when the error says the outcome is unknown, the model may simply call the tool again with a new request, which brings back the duplicate this PR removes. Passing a stable per-call key the server can dedupe on (or documenting that pattern for mutating tools) would make that case safe too.

@he-yufeng

Copy link
Copy Markdown
Contributor Author

That is the right next question to ask. The failure mode you describe is real: once the client refuses to replay, the model may simply issue a fresh call, which lands the same mutation twice without the dedupe this PR removed. A stable per-call idempotency key would close it, but the MCP tools/call envelope has no standard field for one today, so it would need either a protocol-level addition or a documented per-server convention for mutating tools. I would rather not grow this PR into protocol design; I can open a tracking issue proposing the idempotency-key pattern for tool calls if that sounds right to you.

Comment thread python/packages/core/AGENTS.md Resolved
…y hint

Fixes microsoft#9204.

When the MCP connection drops during an ordinary tools/call, the request
may already have executed server-side before the response was lost. The
reconnect-and-retry added for microsoft#2884 re-issues the same call
unconditionally on ClosedResourceError / session-terminated McpError, so
a non-idempotent operation (a charge, a write, a ticket creation) can run
twice. The long-running task path already refuses to re-issue an
augmented call before a task_id is known for exactly this reason; the
plain call path now follows the same policy.

_call_tool_with_retries gates the replay on the tool's advertised
ToolAnnotations, captured from tools/list into a _tool_annotations_by_name
snapshot alongside the existing meta/task-support maps. Replay requires
both a server idempotentHint or readOnlyHint and the client-owned
trust_server_idempotency_hints opt-in: annotations are server-supplied and
untrusted by default, so a misannotated server must not get a
non-idempotent call executed twice on its own say-so. Any other tool
raises ToolExecutionException stating the connection was lost and the
remote execution outcome is unknown, with the connection error as the
inner exception.

The gate runs after the reconnect attempt, so a caller-owned connection
still fails with its existing 'cannot be recreated' error and never
reaches the ambiguous-outcome path. tools/list, get_prompt, and the
long-running task lifecycle keep their current retry behavior.

Tests: test_mcp.py covers a hinted tool replaying once after reconnect,
an unhinted tool raising without a replay, a hinted tool without the
client opt-in raising without a replay, a lost-response call executing at
most once, and the annotations snapshot rolling back when discovery
fails after load_tools. The three no-replay cases go red when
the gate is disabled. Full test_mcp.py: 467 passed, 2 skipped.
poe syntax -P core and poe pyright -P core are clean.
@he-yufeng

Copy link
Copy Markdown
Contributor Author

Rebuilt onto current main as a single commit and tightened the gate ordering while resolving the conflicts.

Two things changed under review of the rebase. The session-fixtures conflict showed that the caller-owned reconnect test pins a "caller-owned and cannot be recreated" error and a no-reconnect, no-replay outcome for exactly the ambiguous call this PR gates. The gate now runs after the reconnect attempt instead of before it, so that contract stays intact: a caller-owned connection still fails with its existing error, and a framework-owned one is back up when the unknown-outcome exception surfaces. Replay still requires both the server hint and the trust_server_idempotency_hints opt-in, so the non-idempotent path never re-issues.

Verification on this base: full packages/core/tests/core/test_mcp.py 467 passed 2 skipped, poe syntax -P core clean, poe pyright -P core 0 errors. The three no-replay cases (unhinted tool, hinted tool without client trust, lost-response side effect) go red when the gate is disabled.

The connection-reset integration test marks the query tool read-only
before the simulated drop so the lost response is safe to replay, but the
replay also needs the client-owned trust opt-in, which the default
posture withholds. Without it the second invocation takes the
unknown-outcome path and never reaches the reconnection the test is
about. Opt the tool into trust_server_idempotency_hints for that test
only, keeping the read-only hint it already sets.
@eavanvalkenburg

Copy link
Copy Markdown
Member

Yufeng He (@he-yufeng) When LOCAL_MCP_URL enables the modified reconnect integration test, it fails before exercising replay. A URL-backed MCPStreamableHTTPTool uses the high-level _ClientMCPConnection, whose session is deliberately None, so the test’s session assertion cannot pass and patching tool.session.call_tool targets the wrong lifecycle boundary. Please capture and patch the active framework-owned Client instead, consistent with ADR 0045, so the enabled integration test validates the replay behavior it intends to cover.

@he-yufeng

Copy link
Copy Markdown
Contributor Author

Eduard van Valkenburg (@eavanvalkenburg) Updated in bc22c21da, with current main merged in df44595c0.

Both URL-backed HTTP integration tests now assert the active framework-owned high-level Client and deliberately empty low-level session. The reconnect test captures that Client, patches call_tool with patch.object, and asserts that reconnect replaces both the Client and exit stack.

The local fixture also needed two corrections to exercise the intended replay. It now uses the MCP 2 MCPServer API required by the locked mcp==2.3.0, and its query tool advertises readOnlyHint through tools/list. A test-local annotation-map mutation was not sufficient because reconnect rediscovery removed that hint and correctly blocked replay.

Before the fix, both integration tests failed on assert tool.session is not None. On the merged head, both pass against the real local MCP server with LOCAL_MCP_URL enabled. The non-integration test_mcp.py plus test_mcp_observability.py run also passes all 566 tests, and Ruff is clean. The local full-core typing attempts were not counted as evidence; the hosted typing matrix remains the authoritative gate.

…lay-without-idempotency-rebuild

# Conflicts:
#	python/packages/core/tests/core/test_mcp.py
@he-yufeng

Copy link
Copy Markdown
Contributor Author

Merged current main in 33f0c0fc9. The two textual conflicts now preserve both the new loaded-state contract and the replay annotations. Four additional replay tests also had to move from overriding connect() to _reconnect_without_loading(), because main's recovery path no longer calls the old override; without that migration, recovery tried to build a real session from the mock transport.

Final test_mcp.py verification is 543 collected, 541 passed, 2 skipped, 0 failed. Ruff format/check pass, strict source Pyright reports 0 errors, and all five test-side type checkers pass: Pyright, MyPy, Pyrefly, ty, and zuban.

The pushed head currently has 8 fork workflows marked action_required; the local results above are not a substitute for those hosted runs.

from mcp import MCPError

# A catalog refresh cannot retroactively make the first dispatch safe to repeat.
replay_eligible = self._is_call_replayable(tool_name)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yufeng He (@he-yufeng) This initial replay eligibility is read before request-specific HTTP admission. On a shared MCPStreamableHTTPTool, it can come from principal A’s hinted catalog while principal B’s unhinted mutation executes. Recovery can then switch back to A before the second check and retry B after admission reloads B’s unhinted catalog, executing B twice. Please capture initial eligibility after the first attempt’s identity reconciliation and validate replay eligibility inside the retry admission scope against the catalog actually used for dispatch, with a concurrent cross-identity regression.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eduard van Valkenburg (@eavanvalkenburg) Implemented in ec388cf3d, with current main merged in 3e66e6a50.

The first replay decision is now captured inside _invoke_tool after _http_operation() has reconciled the request identity. On retry, the original and current hints are checked again inside the retry admission before dispatch. A's hinted catalog can no longer authorize B's unhinted call, and a reconnect cannot change replay permission outside that invocation's admission.

The new concurrent regression starts B with A's hinted catalog loaded, drops B after its first dispatch, lets A finish recovery first, and verifies that B is dispatched once with an unknown-outcome error. Before the fix, it returned successfully after two B dispatches.

Verification on the merged head: 739 MCP tests with 0 failures and 5 skips; core Ruff and source Pyright; all five test-side type checkers. The six hosted failures were recovery tests that expected replay without opting in. They now mock replay permission explicitly, while catalog behavior stays covered by the dedicated replay tests.

Capture replay permission after HTTP identity reconciliation and validate it again inside retry admission. A concurrent caller can no longer lend its idempotency hint to another principal's operation.

Add a deterministic cross-identity regression and opt recovery-focused tests into replay explicitly.
@he-yufeng

Copy link
Copy Markdown
Contributor Author

I investigated the Windows failure in test_auth_context_binding_survives_workflow_resume. It is in the declarative approval-resume path, not the core MCP replay code changed here.

On the merged head 3e66e6a50, using Python 3.13 with the real PowerFx and .NET runtime on macOS, I ran the four-parameter case ten times: all 40 executions passed. The initial four skips on Python 3.14 were not counted as validation. Upstream has not changed this test or the related workflow files since the merged head.

I attempted to rerun the failed Windows lanes, but GitHub rejected the rerun with Must have admin rights to Repository. The failing workflow therefore still needs a maintainer-authorized rerun; I have not changed the unrelated test or claimed the Windows lane is green.

@eavanvalkenburg

Copy link
Copy Markdown
Member

Yufeng He (@he-yufeng) Thanks for the contribution. The linked issue #9204 does not currently have ready-for-implementation or explicit maintainer agreement on this implementation direction. Please wait for that direction before we review or merge this PR.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
@he-yufeng

Copy link
Copy Markdown
Contributor Author

Understood, and I will hold. #9204 still carries only python, agents, mcp, reproduced — no ready-for-implementation and no maintainer agreement on the direction, so this stays a draft and I will not request review.

For whenever that direction is settled, the current state of the branch: f27315ae15 is merged up to date with main, all three checks pass, and the Windows failure I reported earlier is in the declarative approval-resume path rather than the MCP replay change here.

If the preferred direction turns out to be different — for example handling this at the transport rather than in the call path — say so on #9204 and I will rework or close this accordingly. I am not going to keep pushing commits into an unagreed design.

…s lost

_call_tool_with_retries dispatched the same tools/call a second time after a
ClosedResourceError or a session-terminated MCPError. The server can commit the
operation before the response is lost, so a non-idempotent tool ran twice and the
caller saw a successful result.

Dispatch once. On connection loss, recover the connection through the existing
recovery helpers and then raise ToolExecutionException stating that the remote
outcome is unknown and the call was not retried, keeping the connection error as
the cause and on the span, with a redacted warning log. A later, distinct call
proceeds on the replacement connection. Caller-owned connections keep their
existing "cannot be recreated" error, and tools/list and get_prompt keep their
reconnect-and-retry behavior.

Server ToolAnnotations stay advisory: idempotentHint and readOnlyHint do not
authorize a replay, and no trust or idempotency option is added.

Refs microsoft#9204
@he-yufeng Yufeng He (he-yufeng) changed the title Python: fix(core): do not replay MCP tools/call without an idempotency hint Python: [BREAKING] Do not replay an MCP tools/call whose response was lost Nov 14, 2026
@agent-framework-automation agent-framework-automation Bot added the breaking change Usage: [PRs], Target: all PRs that introduce changes that are not backward compatible label Nov 14, 2026
@he-yufeng

Copy link
Copy Markdown
Contributor Author

Reworked to the scope in the #9204 planning comment: dispatch once, no trust or idempotency setting.

b6637210d removes trust_server_idempotency_hints from all four constructors, removes _is_call_replayable, and removes the pre-dispatch eligibility hook, so _invoke_tool is back to its main signature. The production diff is now confined to _call_tool_with_retries: recover the connection through the existing helpers, then raise with the outcome unknown, the connection error as cause and span error type, and a redacted warning. Advisory server hints no longer influence anything on this path. tools/list, get_prompt, and caller-owned connection errors are unchanged.

The error-path behavior change is real, so the title now carries [BREAKING] and packages/core/README.md documents the migration next to the existing MCP breaking-change notes: catch the exception, reconcile, retry explicitly if safe. No exactly-once claim.

Coverage follows the plan's list: one effect and one dispatch with the original cause, for both ClosedResourceError and session-terminated MCPError; a parametrized case asserting no replay for unhinted, idempotentHint, readOnlyHint, and both; a later distinct call succeeding on the replacement connection; concurrent callers both getting the failure with one replacement and zero re-dispatch; recovery-budget timeouts and caller-owned connections unchanged; and the observability case now pinning one dispatch, the redacted warning, and error.type from the connection error.

Local results on this head, Python 3.14.2 / MCP SDK 2.3.0: 1308 passed, 3 skipped across test_mcp.py, test_mcp_http_auth.py, test_mcp_observability.py, test_security.py. Both URL-backed integration tests pass against the repository's deterministic local server on loopback. With _mcp.py restored to main and the tests unchanged, 17 fail and the reset integration test fails with DID NOT RAISE; restoring the fix returns both to green. Ruff 0.17.4 check passes; its four format findings are byte-identical on main and left untouched. Pyright 1.1.414 over 107 core files reports only the two pre-existing _skills.py PyYAML findings.

This stays a draft. #9204 still has no ready-for-implementation, and I am not requesting review until the direction there is settled.

This branch was successfully deployed

1 active deployment
github-app-auth — b6637210 Deployed Nov 14, 2026 by he-yufeng via add_label #27857
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking change Usage: [PRs], Target: all PRs that introduce changes that are not backward compatible documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants