Repository navigation
Python: Record cancelled tool calls as errors - #44
adamdougal merged 4 commits into
Conversation
FunctionTool.invoke now records a tool call cancelled while running as an ERROR execute_tool span and adds error.type to its duration metric. error.type is conversation_closed when that is the cancel message, and cancelled for any other cancellation, so per-scope messages like anyio's never become metric values. The cancellation still propagates unchanged. Spec 006 drops "in its own PR first" and says other cancel messages record cancelled. The commit hook also reformats two asserts in test_harness_memory.py, which arrived unformatted from upstream. Design: #30 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A str subclass whose __hash__ raises passed the isinstance check, so the frozenset lookup raised TypeError inside the cancellation handler and replaced the CancelledError. Only an exact str can now be a recorded reason, so no user code runs while the cancellation is recorded. Found in review of #30 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Local checks on
The one failure is |
|
GPT-family review (gpt-6.1-sol) of the full diff found one issue. A |
There was a problem hiding this comment.
🟡 Changes recommended
The cancellation path leaves end_time_stamp unset, allowing span updates and synchronous logging to inflate cancelled-call duration measurements.
1 open issue
What changed in this PR
Updates Python tool telemetry so cancelled calls can be distinguished from successful calls in chat and realtime scenarios, without changing cancellation propagation.
Changes:
- Record cancellation as ERROR spans and duration metrics with low-cardinality
error.typevalues. - Add tests for cancellation reasons, propagation, privacy, exception limits, and middleware interactions.
- Align the realtime specification and include whitespace-only formatting fixes in memory tests.
| File | Description |
|---|---|
python/packages/core/tests/core/test_tool_cancellation_telemetry.py |
Adds cancellation telemetry and behavior coverage. |
python/packages/core/tests/core/test_harness_memory.py |
Reformats two assertions without changing behavior. |
python/packages/core/agent_framework/_tools.py |
Records cancellation telemetry and documents the behavior. |
docs/specs/006-python-realtime-agents.md |
Clarifies cancellation metrics, reason handling, and delivery scope. |
Review effort: Max
Capture the end time first in the cancellation branch, as the failure path does, so span updates and logging don't count toward the recorded duration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…into adamdougal-cancelled-tool-calls-as-errors # Conflicts: # docs/specs/006-python-realtime-agents.md
|
Local checks after merging
The spec 006 conflict with #45 is resolved by keeping both edits: #45 opens the "On stable APIs" paragraph with |
af704eb
into
feature-python-realtime-agents

Motivation & Context
FunctionTool.invokecatches onlyException, andasyncio.CancelledErrordoesn't inherit from it. So a tool callcancelled while it runs ends its
execute_toolspan with status unset and records its duration as a success, andtraces and metrics can't tell an abandoned call from a finished one. Realtime agents need that difference: closing a
live conversation cancels its running tools, and the contact-centre accelerator counts abandoned tools from
error.type(#9). The prototype'shang-up-toolscenario failed on exactly this gap (#10). Chat runs hit it too, forexample when a parallel call is cancelled after a
MiddlewareFailure, or when a caller abandons a run mid-tool.This piece has no realtime dependency. It merges into the fork's
feature-python-realtime-agentsbranch, which goesupstream later as one PR.
Description & Review Guide
What are the major changes?
FunctionTool.invokecatchesasyncio.CancelledErroraround the tool body. It sets ERRORstatus and
error.typeon the span, addserror.typetoagent_framework.function.invocation.duration, logs oneINFO line,
Function <name> cancelled (<error.type>)., and re-raises the cancellation unchanged._cancellation_error_type()in_tools.pypickserror.type: the cancel message when it's a reasonAgent Framework sends, else
cancelled. Today the only such reason isconversation_closed, held in the privateconstant
_CONVERSATION_CLOSED_REASON, which the realtime conversation will import and pass totask.cancel().packages/core/tests/core/test_tool_cancellation_telemetry.py, and a docstring paragraph oninvoke.packages/core/tests/core/test_harness_memory.pyarrived unformatted fromupstream Python: fix(memory): return no results for a non-positive transcript search limit microsoft/agent-framework#9907. The commit hook formats the whole package, which blocks any core commit
until they're fixed, so this PR carries the whitespace-only fix.
error.typetoo, and that any othercancel message records
cancelled.PR.
What is the impact of these changes? The change is additive, and it adds no public API. Cancelled tool calls now
show as ERROR spans with an
error.typeon the duration metric, so error counts in dashboards will include them.Behaviour is unchanged:
max_invocation_exceptions;Function-loop behaviour is unchanged, so spec 004 isn't edited.
What do you want reviewers to focus on?
error.type. anyio, used by FastAPI, Starlette and the MCP SDK, cancelswith messages like
Cancelled via cancel scope 102d90890; reason: deadline exceeded, with a new id per scope.Recorded verbatim, each cancellation would start a new metric series. The ticket said "the cancellation's reason
when it gives one"; Adam chose known reasons only in the
design.
strcan match a known reason. A GPT-family review found that astrsubclass whose__hash__raises turned the cancellation into a
TypeError, so no user code now runs while a cancellation is recorded.Related Issue
Fixes #30. This PR targets
feature-python-realtime-agents, so the ticket is closed by hand when it merges.AI Assistance
AI assistance details: GitHub Copilot wrote the design notes, implementation, tests, the spec 006 edits and this
description. Adam approved the design and the test list before implementation.
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.