Repository navigation
Python: [BREAKING] Do not replay an MCP tools/call whose response was lost - #9243
Yufeng He (he-yufeng) wants to merge 19 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The annotation snapshot is not restored when connection discovery fails, allowing stale replay guarantees.
1 open finding
What changed in this PR
Prevents unsafe replay of MCP tool calls after connection loss unless the tool advertises read-only or idempotent behavior.
Changes:
- Captures tool annotations and gates reconnect retries.
- Adds regression tests for duplicate side effects.
- Documents the replay policy.
| File | Description |
|---|---|
python/packages/core/agent_framework/_mcp.py |
Implements annotation-based replay gating. |
python/packages/core/tests/core/test_mcp.py |
Tests replayable and non-replayable calls. |
python/packages/core/AGENTS.md |
Documents the new policy. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
…otency hints ToolAnnotations are server-supplied and untrusted by default, so an annotation alone must not authorize replaying an ambiguous tools/call. Replay now additionally requires the client-owned trust_server_idempotency_hints=True opt-in on the MCP tool; without it the lost-response path stays no-replay and surfaces the unknown outcome. Review thread: microsoft#9243 (comment)
|
One follow-up worth considering: when the error says the outcome is unknown, the model may simply call the tool again with a new request, which brings back the duplicate this PR removes. Passing a stable per-call key the server can dedupe on (or documenting that pattern for mutating tools) would make that case safe too. |
|
That is the right next question to ask. The failure mode you describe is real: once the client refuses to replay, the model may simply issue a fresh call, which lands the same mutation twice without the dedupe this PR removed. A stable per-call idempotency key would close it, but the MCP tools/call envelope has no standard field for one today, so it would need either a protocol-level addition or a documented per-server convention for mutating tools. I would rather not grow this PR into protocol design; I can open a tracking issue proposing the idempotency-key pattern for tool calls if that sounds right to you. |
…y hint Fixes microsoft#9204. When the MCP connection drops during an ordinary tools/call, the request may already have executed server-side before the response was lost. The reconnect-and-retry added for microsoft#2884 re-issues the same call unconditionally on ClosedResourceError / session-terminated McpError, so a non-idempotent operation (a charge, a write, a ticket creation) can run twice. The long-running task path already refuses to re-issue an augmented call before a task_id is known for exactly this reason; the plain call path now follows the same policy. _call_tool_with_retries gates the replay on the tool's advertised ToolAnnotations, captured from tools/list into a _tool_annotations_by_name snapshot alongside the existing meta/task-support maps. Replay requires both a server idempotentHint or readOnlyHint and the client-owned trust_server_idempotency_hints opt-in: annotations are server-supplied and untrusted by default, so a misannotated server must not get a non-idempotent call executed twice on its own say-so. Any other tool raises ToolExecutionException stating the connection was lost and the remote execution outcome is unknown, with the connection error as the inner exception. The gate runs after the reconnect attempt, so a caller-owned connection still fails with its existing 'cannot be recreated' error and never reaches the ambiguous-outcome path. tools/list, get_prompt, and the long-running task lifecycle keep their current retry behavior. Tests: test_mcp.py covers a hinted tool replaying once after reconnect, an unhinted tool raising without a replay, a hinted tool without the client opt-in raising without a replay, a lost-response call executing at most once, and the annotations snapshot rolling back when discovery fails after load_tools. The three no-replay cases go red when the gate is disabled. Full test_mcp.py: 467 passed, 2 skipped. poe syntax -P core and poe pyright -P core are clean.
b5c2244 to
254b89b
Compare
|
Rebuilt onto current main as a single commit and tightened the gate ordering while resolving the conflicts. Two things changed under review of the rebase. The session-fixtures conflict showed that the caller-owned reconnect test pins a "caller-owned and cannot be recreated" error and a no-reconnect, no-replay outcome for exactly the ambiguous call this PR gates. The gate now runs after the reconnect attempt instead of before it, so that contract stays intact: a caller-owned connection still fails with its existing error, and a framework-owned one is back up when the unknown-outcome exception surfaces. Replay still requires both the server hint and the trust_server_idempotency_hints opt-in, so the non-idempotent path never re-issues. Verification on this base: full packages/core/tests/core/test_mcp.py 467 passed 2 skipped, poe syntax -P core clean, poe pyright -P core 0 errors. The three no-replay cases (unhinted tool, hinted tool without client trust, lost-response side effect) go red when the gate is disabled. |
The connection-reset integration test marks the query tool read-only before the simulated drop so the lost response is safe to replay, but the replay also needs the client-owned trust opt-in, which the default posture withholds. Without it the second invocation takes the unknown-outcome path and never reaches the reconnection the test is about. Opt the tool into trust_server_idempotency_hints for that test only, keeping the read-only hint it already sets.
|
Yufeng He (@he-yufeng) When |
|
Eduard van Valkenburg (@eavanvalkenburg) Updated in Both URL-backed HTTP integration tests now assert the active framework-owned high-level The local fixture also needed two corrections to exercise the intended replay. It now uses the MCP 2 Before the fix, both integration tests failed on |
…lay-without-idempotency-rebuild # Conflicts: # python/packages/core/tests/core/test_mcp.py
|
Merged current main in Final The pushed head currently has 8 fork workflows marked |
| from mcp import MCPError | ||
|
|
||
| # A catalog refresh cannot retroactively make the first dispatch safe to repeat. | ||
| replay_eligible = self._is_call_replayable(tool_name) |
There was a problem hiding this comment.
Yufeng He (@he-yufeng) This initial replay eligibility is read before request-specific HTTP admission. On a shared MCPStreamableHTTPTool, it can come from principal A’s hinted catalog while principal B’s unhinted mutation executes. Recovery can then switch back to A before the second check and retry B after admission reloads B’s unhinted catalog, executing B twice. Please capture initial eligibility after the first attempt’s identity reconciliation and validate replay eligibility inside the retry admission scope against the catalog actually used for dispatch, with a concurrent cross-identity regression.
There was a problem hiding this comment.
Eduard van Valkenburg (@eavanvalkenburg) Implemented in ec388cf3d, with current main merged in 3e66e6a50.
The first replay decision is now captured inside _invoke_tool after _http_operation() has reconciled the request identity. On retry, the original and current hints are checked again inside the retry admission before dispatch. A's hinted catalog can no longer authorize B's unhinted call, and a reconnect cannot change replay permission outside that invocation's admission.
The new concurrent regression starts B with A's hinted catalog loaded, drops B after its first dispatch, lets A finish recovery first, and verifies that B is dispatched once with an unknown-outcome error. Before the fix, it returned successfully after two B dispatches.
Verification on the merged head: 739 MCP tests with 0 failures and 5 skips; core Ruff and source Pyright; all five test-side type checkers. The six hosted failures were recovery tests that expected replay without opting in. They now mock replay permission explicitly, while catalog behavior stays covered by the dedicated replay tests.
Capture replay permission after HTTP identity reconciliation and validate it again inside retry admission. A concurrent caller can no longer lend its idempotency hint to another principal's operation. Add a deterministic cross-identity regression and opt recovery-focused tests into replay explicitly.
…lay-without-idempotency-rebuild
|
I investigated the Windows failure in On the merged head I attempted to rerun the failed Windows lanes, but GitHub rejected the rerun with |
|
Yufeng He (@he-yufeng) Thanks for the contribution. The linked issue #9204 does not currently have |
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
|
Understood, and I will hold. #9204 still carries only For whenever that direction is settled, the current state of the branch: If the preferred direction turns out to be different — for example handling this at the transport rather than in the call path — say so on #9204 and I will rework or close this accordingly. I am not going to keep pushing commits into an unagreed design. |
…s lost _call_tool_with_retries dispatched the same tools/call a second time after a ClosedResourceError or a session-terminated MCPError. The server can commit the operation before the response is lost, so a non-idempotent tool ran twice and the caller saw a successful result. Dispatch once. On connection loss, recover the connection through the existing recovery helpers and then raise ToolExecutionException stating that the remote outcome is unknown and the call was not retried, keeping the connection error as the cause and on the span, with a redacted warning log. A later, distinct call proceeds on the replacement connection. Caller-owned connections keep their existing "cannot be recreated" error, and tools/list and get_prompt keep their reconnect-and-retry behavior. Server ToolAnnotations stay advisory: idempotentHint and readOnlyHint do not authorize a replay, and no trust or idempotency option is added. Refs microsoft#9204
|
Reworked to the scope in the #9204 planning comment: dispatch once, no trust or idempotency setting.
The error-path behavior change is real, so the title now carries Coverage follows the plan's list: one effect and one dispatch with the original cause, for both Local results on this head, Python 3.14.2 / MCP SDK 2.3.0: 1308 passed, 3 skipped across This stays a draft. #9204 still has no |

Motivation & Context
When the MCP connection drops during an ordinary
tools/call, the request may already have executed server-side before the response was lost. The reconnect-and-retry added for #2884 re-issued the same call unconditionally onClosedResourceError/ session-terminatedMCPError, so a non-idempotent operation (a charge, a write, a ticket creation) could run twice, and a successful replay hid the failure from the caller entirely.Description & Review Guide
_call_tool_with_retriesdispatches once. On a connection loss it recovers the connection through the existing_recover_connection_after_losshelpers, then raisesToolExecutionExceptionstating that the remote execution outcome is unknown and the call was not retried, with the connection error as__cause__, the connection error type on the span, and a redacted WARNING log. ServerToolAnnotationsstay advisory:idempotentHintandreadOnlyHintdo not authorize a replay, and this adds no trust or idempotency option.tools/list,get_prompt, and caller-owned connection errors keep their current behavior.tools/callnow surfaces an explicit failure instead of a transparent retry. For framework-owned connections the connection is already restored when the exception surfaces, so a later, distinct call proceeds normally; caller-suppliedClient/ClientSessionobjects are never recreated. This is a behavioral break on the error path, so the title carries[BREAKING]andpackages/core/README.mddocuments the migration: catch the exception, reconcile the outcome, retry explicitly if safe. No exactly-once guarantee is claimed.test_simultaneous_connection_failures_share_one_recoveryandtest_ambiguous_call_is_not_replayed_after_a_concurrent_identity_change: both callers get the unknown-outcome failure, one replacement connection is created, and neither operation is re-issued.The earlier revision of this PR gated replay on server hints plus a client
trust_server_idempotency_hintsopt-in. That option,_is_call_replayable, and the pre-dispatch eligibility hook are all removed here. The production diff is now confined to_call_tool_with_retries;_invoke_toolis back to itsmainsignature.Related Issue
Refs #9204
The maintainer planning comment on #9204 asks for no replay by default, recovery separated from the failed call, read-only retries preserved, and explicitly no new trust or idempotency setting. This revision follows that scope. #9204 still has no
ready-for-implementationlabel, so this stays a draft and I am not requesting review; the direction question there is still open.AI Assistance
AI assistance details: implementation, tests, documentation, and this PR description. I reviewed the final diff line by line and ran the suites and the local MCP integration tests reported below.
Contribution Checklist
[BREAKING]title prefix declares the intentional error-path behavior change described above.Local results on
b6637210dPython 3.14.2, MCP SDK 2.3.0,
agent_frameworkresolving to this checkout.test_mcp.py+test_mcp_http_auth.py+test_mcp_observability.py+test_security.py, non-integration: 1308 passed, 3 skipped, 2 deselected.LOCAL_MCP_URL, no remote service):test_streamable_http_integration,test_mcp_connection_reset_integration— 2 passed._mcp.pyrestored tomainand these tests unchanged, 17 failed / 760 passed, andtest_mcp_connection_reset_integrationfails withDID NOT RAISE. Restoring the fix returns both runs to green.git diff --check: clean._mcp.pyand one inscripts/local_mcp_streamable_http_server.py; all four are byte-identical onmainand untouched here, so they are left alone rather than reformatting unrelated upstream code.composeunknown-type findings in_skills.py._mcp.pyhas no diagnostics. This is not a clean full-core typing result.