Skip to content

Python: Record cancelled tool calls as errors - #44

Merged
adamdougal merged 4 commits into
feature-python-realtime-agentsfrom
adamdougal-cancelled-tool-calls-as-errors
Nov 10, 2026
Merged

adamdougal merged 4 commits into
feature-python-realtime-agentsfrom
adamdougal-cancelled-tool-calls-as-errors

Conversation

@adamdougal

Copy link
Copy Markdown
Owner

Motivation & Context

FunctionTool.invoke catches only Exception, and asyncio.CancelledError doesn't inherit from it. So a tool call
cancelled while it runs ends its execute_tool span with status unset and records its duration as a success, and
traces and metrics can't tell an abandoned call from a finished one. Realtime agents need that difference: closing a
live conversation cancels its running tools, and the contact-centre accelerator counts abandoned tools from
error.type (#9). The prototype's hang-up-tool scenario failed on exactly this gap (#10). Chat runs hit it too, for
example when a parallel call is cancelled after a MiddlewareFailure, or when a caller abandons a run mid-tool.

This piece has no realtime dependency. It merges into the fork's feature-python-realtime-agents branch, which goes
upstream later as one PR.

Description & Review Guide

  • What are the major changes?

    • With telemetry on, FunctionTool.invoke catches asyncio.CancelledError around the tool body. It sets ERROR
      status and error.type on the span, adds error.type to agent_framework.function.invocation.duration, logs one
      INFO line, Function <name> cancelled (<error.type>)., and re-raises the cancellation unchanged.
    • A private _cancellation_error_type() in _tools.py picks error.type: the cancel message when it's a reason
      Agent Framework sends, else cancelled. Today the only such reason is conversation_closed, held in the private
      constant _CONVERSATION_CLOSED_REASON, which the realtime conversation will import and pass to task.cancel().
    • New tests in packages/core/tests/core/test_tool_cancellation_telemetry.py, and a docstring paragraph on invoke.
    • Unrelated reformat: two asserts in packages/core/tests/core/test_harness_memory.py arrived unformatted from
      upstream Python: fix(memory): return no results for a non-positive transcript search limit microsoft/agent-framework#9907. The commit hook formats the whole package, which blocks any core commit
      until they're fixed, so this PR carries the whitespace-only fix.
    • Spec 006 corrections:
      • Telemetry's "Cancelled tools" bullet now says the duration metric carries error.type too, and that any other
        cancel message records cancelled.
      • "Changes to existing packages" drops "in its own PR first", because Write the extended ADR and open the upstream PR #11 moved the fix inside the one upstream
        PR.
      • ADR 0054 is unchanged.
  • What is the impact of these changes? The change is additive, and it adds no public API. Cancelled tool calls now
    show as ERROR spans with an error.type on the duration metric, so error counts in dashboards will include them.
    Behaviour is unchanged:

    • the cancellation still propagates as before;
    • it doesn't count toward max_invocation_exceptions;
    • calls that never ran still get no span.

    Function-loop behaviour is unchanged, so spec 004 isn't edited.

  • What do you want reviewers to focus on?

    • Only reasons Agent Framework sends become error.type. anyio, used by FastAPI, Starlette and the MCP SDK, cancels
      with messages like Cancelled via cancel scope 102d90890; reason: deadline exceeded, with a new id per scope.
      Recorded verbatim, each cancellation would start a new metric series. The ticket said "the cancellation's reason
      when it gives one"; Adam chose known reasons only in the
      design.
    • Only an exact str can match a known reason. A GPT-family review found that a str subclass whose __hash__
      raises turned the cancellation into a TypeError, so no user code now runs while a cancellation is recorded.
    • A cancelled span gets no status description and no exception event, even with sensitive data on.
    • The log line is INFO rather than ERROR, because closing a conversation cancels tools as a matter of course.

Related Issue

Fixes #30. This PR targets feature-python-realtime-agents, so the ticket is closed by hand when it merges.

AI Assistance

  • No material AI assistance was used.
  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: GitHub Copilot wrote the design notes, implementation, tests, the spec 006 edits and this
description. Adam approved the design and the test list before implementation.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

adamdougal and others added 2 commits November 9, 2026 15:22
FunctionTool.invoke now records a tool call cancelled while running as an
ERROR execute_tool span and adds error.type to its duration metric. error.type
is conversation_closed when that is the cancel message, and cancelled for any
other cancellation, so per-scope messages like anyio's never become metric
values. The cancellation still propagates unchanged.

Spec 006 drops "in its own PR first" and says other cancel messages record
cancelled. The commit hook also reformats two asserts in
test_harness_memory.py, which arrived unformatted from upstream.

Design: #30

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A str subclass whose __hash__ raises passed the isinstance check, so the
frozenset lookup raised TypeError inside the cancellation handler and
replaced the CancelledError. Only an exact str can now be a recorded reason,
so no user code runs while the cancellation is recorded.

Found in review of #30

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@adamdougal

Copy link
Copy Markdown
Owner Author

Local checks on 1b4a760b4, Python 3.13.11, run from python/:

Command Result
uv run poe check 386 of 387 tasks passed: syntax 86/86, pyright 43/43, sample checks 215/215, package tests 42/43. The one failure is the flaky test below.
uv run poe test -P core -C Passed: 9535 passed, 23 skipped, 2 xfailed. Core coverage is 92%, with every new line in _tools.py covered.
uv run poe test-typing -P core Passed: mypy, pyrefly, ty, zuban and pyright.
uv run pytest tests/core/test_tool_cancellation_telemetry.py (in packages/core) 15 passed.

The one failure is packages/tools's test_windows_timeout_sentinel_does_not_reuse_exiting_process, which expected exit code 130 but got 124. It's a timing-sensitive ShellSession test that doesn't go through FunctionTool.invoke, and it fails on the base branch too. uv run poe test -P tools failed it in 1 of 3 runs with this change and in 1 of 3 runs without it, and the test passes alone 3 times out of 3. In an earlier full run, foundry_hosting's test_real_sdk_background_polling_pairs_each_step_and_usage once missed its 2 s start wait under load. It runs a workflow with no tools, passes alone and passed in this final run.

@adamdougal

Copy link
Copy Markdown
Owner Author

GPT-family review (gpt-6.1-sol) of the full diff found one issue. A str subclass whose __hash__ raises passed the isinstance check, so the reason lookup raised TypeError and replaced the cancellation. 1b4a760 fixes it by matching only an exact str, with a regression case in the parametrised unhashable-message test. The reviewer re-checked that commit and the full diff and reported no remaining material issues.

@adamdougal
adamdougal marked this pull request as ready for review November 9, 2026 15:49
@adamdougal adamdougal mentioned this pull request Nov 9, 2026
6 of 7 tasks
@adamdougal
adamdougal requested a max review from Copilot November 10, 2026 10:20

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The cancellation path leaves end_time_stamp unset, allowing span updates and synchronous logging to inflate cancelled-call duration measurements.

1 open issue
What changed in this PR

Updates Python tool telemetry so cancelled calls can be distinguished from successful calls in chat and realtime scenarios, without changing cancellation propagation.

Changes:

  • Record cancellation as ERROR spans and duration metrics with low-cardinality error.type values.
  • Add tests for cancellation reasons, propagation, privacy, exception limits, and middleware interactions.
  • Align the realtime specification and include whitespace-only formatting fixes in memory tests.
File Description
python/​packages/​core/​tests/​core/​test_tool_cancellation_telemetry.py Adds cancellation telemetry and behavior coverage.
python/​packages/​core/​tests/​core/​test_harness_memory.py Reformats two assertions without changing behavior.
python/​packages/​core/​agent_framework/​_tools.py Records cancellation telemetry and documents the behavior.
docs/​specs/​006-python-realtime-agents.md Clarifies cancellation metrics, reason handling, and delivery scope.

Review effort: Max

Capture the end time first in the cancellation branch, as the failure path
does, so span updates and logging don't count toward the recorded duration.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…into adamdougal-cancelled-tool-calls-as-errors

# Conflicts:
#	docs/specs/006-python-realtime-agents.md
@adamdougal

Copy link
Copy Markdown
Owner Author

Local checks after merging feature-python-realtime-agents (e72cd2b, with #45 and upstream main) into this branch, on 6d741ceb5, Python 3.13.11:

Command Result
uv run poe check Passed: syntax 86/86, pyright 43/43, sample checks 215/215, package tests 43/43, markdown code lint passed.
uv run poe test-typing -P core Passed: mypy, pyrefly, ty, zuban and pyright.
uv run pytest tests -n auto (in packages/core) 9606 passed, 25 skipped, 2 xfailed.
uv run pytest tests/core/test_tool_cancellation_telemetry.py (in packages/core) 15 passed.

The spec 006 conflict with #45 is resolved by keeping both edits: #45 opens the "On stable APIs" paragraph with Content's kind and service_event_type fields, and this PR drops "in its own PR first".

@adamdougal
adamdougal merged commit af704eb into feature-python-realtime-agents Nov 10, 2026
23 of 25 checks passed

This branch had an error being deployed

1 failed deployment
github-app-auth — 6d741ceb Deployed Nov 10, 2026 by adamdougal via add_label #16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants