Skip to content

Python: [Bug]: Local tool checks ignore the conversation split, so a tool call from an earlier turn passes later turns #10091

Description

Observed Behavior

tool_called_check, tool_calls_present and tool_call_args_match scan item.conversation, the entire conversation, for function calls (_evaluation.py: the _check loop in tool_called_check, and _extract_tool_calls). Every other derived value on EvalItem (query, response) and the Foundry provider (item.split_messages(effective_split)) uses the item's split strategy.

As a result, in multi-turn evaluation a tool call made in an earlier turn satisfies the tool check for a later turn. With EvalItem.per_turn_items(...), a turn in which the agent made no tool call still passes tool_called_check("get_weather"), tool_calls_present and tool_call_args_match, because the call from turn 1 is on the query side of turn 2's split.

This contradicts the per-turn design described in ADR-0023 and in per_turn_items: response_messages "contains the agent's actions up to the next user message", which "lets you evaluate each response independently". ADR-0023 also says: "Changing the split_strategy consistently changes all derived values." For a CI gate, the outcome is a silent false pass on exactly the turns that per-turn evaluation is meant to pinpoint.

Expected Behavior

Local tool checks should inspect the response side of the item's split (item.split_messages()), the same messages that item.response and Foundry's tool evaluators use. With the default LAST_TURN split on a single-turn item, this is identical to today's behaviour. With FULL, the whole trajectory after the first user message still counts.

Steps to Reproduce

  1. Build a two-turn conversation where only turn 1 calls get_weather.
  2. Split it with EvalItem.per_turn_items(...).
  3. Run tool_called_check("get_weather") (or tool_calls_present / tool_call_args_match with expected_tool_calls) on the second item.
  4. Observe that it passes, although item.response for that turn is only "No idea." and the turn made no tool call.

Minimal Reproduction

from agent_framework import Content, EvalItem, Message
from agent_framework._evaluation import ExpectedToolCall, tool_called_check, tool_calls_present

conversation = [
    Message("user", ["Weather in NYC?"]),
    Message("assistant", [Content.from_function_call("c1", "get_weather", arguments={"location": "NYC"})]),
    Message("assistant", ["Sunny."]),
    Message("user", ["And tomorrow?"]),
    Message("assistant", ["No idea."]),
]
_, second = EvalItem.per_turn_items(conversation)
second.expected_tool_calls = [ExpectedToolCall("get_weather")]

print(repr(second.response))                                # 'No idea.'
print(tool_called_check("get_weather")(second).passed)      # True, expected False
print(tool_calls_present(second).passed)                    # True, expected False

Error Messages and Stack Traces

No exception is raised. The checks pass silently.

Proposed Fix

This is a small change, with no new API. I have it implemented locally with regression tests, and I'm happy to open a PR once the direction is agreed:

  • Add a private helper _response_messages(item) that returns item.split_messages()[1]. Use it in tool_called_check and _extract_tool_calls, which serves tool_calls_present and tool_call_args_match.
  • Update the two docstrings that currently say "conversation history" / "in the conversation" to describe the response side of the split.

Locally, the new regression test fails on main and passes with the change. It covers per-turn items for all three checks, plus a FULL split that still counts the whole trajectory. All 323 tests that use these checks or LocalEvaluator / evaluate_agent / evaluate_workflow pass, as do poe syntax, pyright and test-typing for core. Single-turn evaluate_agent results are unchanged, because the query side there contains only the user message.

The alternative is to keep whole-conversation matching and document it. But then local tool checks cannot be combined with per_turn_items or LAST_TURN on multi-turn data, and they disagree with Foundry evaluators run on the same items.

Related: closed PR #8695 touched tool_called_check argument validation, which is a different problem.

AI assistance disclosure: an AI coding assistant helped audit the module, write the reproduction and tests, and draft this report. I reviewed the analysis, and the reproduction above was run against main at the commit below.

Package Versions

agent-framework-core: 1.24.0 (main at fd12308)

Python Version

Python 3.13

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 4, 2026
  2. added
    reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow
    on Nov 4, 2026
  3. github-actions commented on Nov 4, 2026

    @github-actions
    Contributor

    🤖 Automated triage reproduction notes (agent-authored — trust but verify)

    Agent analysis

    The bug is in python/packages/core/agent_framework/_evaluation.py::tool_called_check and _extract_tool_calls, which scan item.conversation rather than the response side of its split. It occurs when a cumulative multi-turn EvalItem has an earlier function call on the query side. Minimal repro: create two turns with a tool call only in turn 1, select the second result from EvalItem.per_turn_items, and run the three local tool checks.

    • Failing test: python/packages/core/tests/core/test_local_eval.py
    • Files examined: python/packages/core/agent_framework/_evaluation.py, python/packages/core/tests/core/test_local_eval.py, python/packages/core/tests/core/test_evaluation.py, docs/decisions/0023-foundry-evals-integration.md, python/packages/core/pyproject.toml
    • Tests run: python/packages/core/tests/core/test_local_eval.py, python/packages/core/tests/core/test_local_eval.py::test_per_turn_tool_checks_ignore_calls_from_prior_turn
    • Reported version: 1.24.0
    • Current version: 1.24.0
  4. added
    agentsUsage: [Issues, PRs], Target: Single agent
    and removed
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 4, 2026
  5. 14H034160212 commented on Nov 5, 2026

    @14H034160212
    Author

    Eduard van Valkenburg (@eavanvalkenburg) I have a fix with tests ready for this one, if scoping to the split is the intended behaviour:

    • tool_called_check and _extract_tool_calls (used by tool_calls_present and tool_call_args_match) read the response side of item.split_messages(), the same messages item.response uses. No API change.
    • Single-turn LAST_TURN items are unchanged, and with ConversationSplit.FULL the whole trajectory still counts (covered by a test).

    Branch: main...14H034160212:agent-framework:fix-10091-tool-check-turns (the per-turn test fails on main and passes with the change; lint, type checks and the core test suite pass).

    Happy to open a PR if this direction works for you.

  6. eavanvalkenburg commented on Nov 9, 2026

    @eavanvalkenburg
    Member

    Thanks for the clear reproduction. We will consider this in core MAF: local tool checks should support consistent per-turn evaluation alongside the item's conversation split. Before approving implementation, we need to settle the scope and compatibility contract because the existing check docstrings describe conversation-wide matching. This does not promise implementation or timing; planning should preserve whole-trajectory evaluation and cover existing callers.

  7. eavanvalkenburg commented on Nov 9, 2026

    @eavanvalkenburg
    Member

    /plan Resolve the local tool-check scope contract before implementation. Core ownership has been accepted for consideration, not delivery approval. Reconcile the conversation-wide check docstrings with ADR-0023, per_turn_items, and Foundry's split-aware evaluation. Assess response-side matching for tool_called_check, tool_calls_present, and tool_call_args_match, including compatibility for existing multi-turn callers and custom splits. Preserve single-turn behavior and explicit FULL whole-trajectory evaluation; cover all three checks with focused regression tests and clarify migration/documentation needs. Review the existing PR #10186 as a proposal, without treating it or this planning request as implementation approval.

  8. github-actions commented on Nov 9, 2026

    @github-actions
    Contributor

    Summary

    Three local checks (tool_called_check, tool_calls_present, tool_call_args_match) read item.conversation, the whole conversation. Everything else on EvalItem follows the item's split: query, response, and the Foundry evaluator (item.split_messages(...) at _foundry_evals.py L986-989). With per_turn_items, each turn's item holds the conversation up to that turn, so earlier tool calls sit on the query side and still satisfy later turns. I read the code only; I did not run anything. I did not read PR #10186's diff or reviews, so this plan is independent of it.

    Confirmed facts

    • The check docstrings say "conversation history" / "in the conversation". That wording was a deliberate whole-conversation contract, not just an oversight.
    • per_turn_items says it exists to "evaluate each response independently". ADR-0023 says changing split_strategy "consistently changes all derived values". Both point to response-side semantics.
    • FULL splits after the first user message, so the response side is the whole trajectory. Scoping to the response side keeps whole-trajectory evaluation for FULL.
    • Single-turn LAST_TURN items: the query side holds only the user message, so results are identical.
    • The Foundry evaluator also falls back to an evaluator-level _conversation_split when item.split_strategy is unset. LocalEvaluator has no such setting, so local checks would use only the item's own strategy or the LAST_TURN default.

    Recommendation

    Code changes are warranted. Make the local tool checks use the response side of item.split_messages(), plus a documentation update. A docs-only "documented limitation" would leave per_turn_items unable to give per-turn tool checks, and would make local and Foundry results disagree on the same item. It would also keep the silent false pass the issue describes.

    Trade-off: this is a behavior change for existing callers. Anyone who passes a multi-turn conversation with the default LAST_TURN and relies on an earlier-turn call counting will start failing, and the failure is correct under the per-turn contract. The framework treats evals as experimental, but the change still needs an explicit note. The alternative of an opt-in flag would keep the false pass as the default, so I don't recommend it.

    Proposed plan (proposal only; nothing changed, not delivery approval)

    1. Decide the contract. Local tool checks inspect the response side of item.split_messages() (item strategy, else LAST_TURN). Use one private helper for all three checks. Calls in the query side never count. FULL still counts the whole trajectory after the first user message.

    2. Custom splits. The response side is whatever the splitter returns. For example, the split_before_memory example in ConversationSplitter puts the memory tool call on the response side, so it counts. A splitter that puts the call on the query side makes the check fail, which matches item.response and Foundry. Document this.

    3. Edge cases to pin down. There is no user message (LAST_TURN returns the whole conversation as the response, so nothing changes). A tool call earlier in the same turn (counts). Maintainers should confirm whether any evaluate_workflow sub-agent items can carry multiple user turns under the default split. I did not trace that path, and it is the main place existing callers could see a change.

    4. Docs. Reword the three docstrings and the ConversationSplit / EvalItem text to say tool checks follow the split, and how to evaluate a whole trajectory (ConversationSplit.FULL or conversation_split=). Add a short migration note for multi-turn callers: use FULL, or pass expected calls per turn. Per repo rules, do not edit python/CHANGELOG.md in the PR; the release process assembles it.

    5. Tests in test_local_eval.py, for each of the three checks:

      • Per-turn item where only turn 1 called the tool: turn 2 fails, turn 1 passes.
      • FULL split counts calls from every turn.
      • Single-turn LAST_TURN unchanged.
      • Custom splitter that moves the call to the response side or the query side.
      • Same-turn calls and multiple calls with arguments (tool_call_args_match).
    6. Done when the new tests pass, existing tests using these checks stay green (the reporter says 323 pass locally; that is unverified by me), and the docs describe the split-aware contract.

    Questions for maintainers (not blocking the direction)

    • Is a changelog or release-note callout at release time enough for the multi-turn behavior change, or do you want a deprecation period?
    • Should LocalEvaluator ever get its own default split like Foundry's _conversation_split? I suggest not now, to keep scope small.

    Generated by Issue Planning for #10091 · copilot · auto · ~37 AIC · ⌖ ~1 AIC · ⊞ 4.8K · ◷

  9. 14H034160212 commented on Nov 10, 2026

    @14H034160212
    Author

    Thanks Eduard van Valkenburg (@eavanvalkenburg), the plan matches what I had in mind. I've updated my branch to follow it:

    • One private helper (_response_function_calls) used by tool_called_check, tool_calls_present and tool_call_args_match, reading the response side of item.split_messages() (item strategy, else LAST_TURN).
    • Docstrings for the three checks, ConversationSplit and EvalItem.split_strategy now say that tool checks follow the split, and that FULL checks the whole trajectory.
    • Tests in test_local_eval.py for all three checks: per-turn item where only turn 1 called the tool, FULL across turns, single-turn LAST_TURN unchanged, a conversation with no user message, a custom splitter that puts the call on the response side or the query side, and two calls with arguments in the current turn. The per-turn, custom-split and current-turn tests fail on main and pass with the change.
    • test_local_eval.py (103), the core test suite (7505 passed) and test_foundry_evals.py (144) pass; ruff, mypy and pyright are clean. No CHANGELOG.md edit.

    Branch: main...14H034160212:agent-framework:fix-10091-tool-check-turns-v2

    I'll open it as a PR once the direction is marked ready for implementation.

    AI assistance: I used Claude (Anthropic) to help with the analysis, implementation and tests; I reviewed the change and ran the checks above myself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

agentsUsage: [Issues, PRs], Target: Single agentpythonUsage: [Issues, PRs], Target: Pythonready-for-implementationMaintainers agree the problem and direction are ready for implementation.reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions