Repository navigation
Description
Observed Behavior
tool_called_check, tool_calls_present and tool_call_args_match scan item.conversation, the entire conversation, for function calls (_evaluation.py: the _check loop in tool_called_check, and _extract_tool_calls). Every other derived value on EvalItem (query, response) and the Foundry provider (item.split_messages(effective_split)) uses the item's split strategy.
As a result, in multi-turn evaluation a tool call made in an earlier turn satisfies the tool check for a later turn. With EvalItem.per_turn_items(...), a turn in which the agent made no tool call still passes tool_called_check("get_weather"), tool_calls_present and tool_call_args_match, because the call from turn 1 is on the query side of turn 2's split.
This contradicts the per-turn design described in ADR-0023 and in per_turn_items: response_messages "contains the agent's actions up to the next user message", which "lets you evaluate each response independently". ADR-0023 also says: "Changing the split_strategy consistently changes all derived values." For a CI gate, the outcome is a silent false pass on exactly the turns that per-turn evaluation is meant to pinpoint.
Expected Behavior
Local tool checks should inspect the response side of the item's split (item.split_messages()), the same messages that item.response and Foundry's tool evaluators use. With the default LAST_TURN split on a single-turn item, this is identical to today's behaviour. With FULL, the whole trajectory after the first user message still counts.
Steps to Reproduce
- Build a two-turn conversation where only turn 1 calls
get_weather. - Split it with
EvalItem.per_turn_items(...). - Run
tool_called_check("get_weather")(ortool_calls_present/tool_call_args_matchwithexpected_tool_calls) on the second item. - Observe that it passes, although
item.responsefor that turn is only"No idea."and the turn made no tool call.
Minimal Reproduction
from agent_framework import Content, EvalItem, Message
from agent_framework._evaluation import ExpectedToolCall, tool_called_check, tool_calls_present
conversation = [
Message("user", ["Weather in NYC?"]),
Message("assistant", [Content.from_function_call("c1", "get_weather", arguments={"location": "NYC"})]),
Message("assistant", ["Sunny."]),
Message("user", ["And tomorrow?"]),
Message("assistant", ["No idea."]),
]
_, second = EvalItem.per_turn_items(conversation)
second.expected_tool_calls = [ExpectedToolCall("get_weather")]
print(repr(second.response)) # 'No idea.'
print(tool_called_check("get_weather")(second).passed) # True, expected False
print(tool_calls_present(second).passed) # True, expected FalseError Messages and Stack Traces
No exception is raised. The checks pass silently.
Proposed Fix
This is a small change, with no new API. I have it implemented locally with regression tests, and I'm happy to open a PR once the direction is agreed:
- Add a private helper
_response_messages(item)that returnsitem.split_messages()[1]. Use it intool_called_checkand_extract_tool_calls, which servestool_calls_presentandtool_call_args_match. - Update the two docstrings that currently say "conversation history" / "in the conversation" to describe the response side of the split.
Locally, the new regression test fails on main and passes with the change. It covers per-turn items for all three checks, plus a FULL split that still counts the whole trajectory. All 323 tests that use these checks or LocalEvaluator / evaluate_agent / evaluate_workflow pass, as do poe syntax, pyright and test-typing for core. Single-turn evaluate_agent results are unchanged, because the query side there contains only the user message.
The alternative is to keep whole-conversation matching and document it. But then local tool checks cannot be combined with per_turn_items or LAST_TURN on multi-turn data, and they disagree with Foundry evaluators run on the same items.
Related: closed PR #8695 touched tool_called_check argument validation, which is a different problem.
AI assistance disclosure: an AI coding assistant helped audit the module, write the reproduction and tests, and draft this report. I reviewed the analysis, and the reproduction above was run against main at the commit below.
Package Versions
agent-framework-core: 1.24.0 (main at fd12308)
Python Version
Python 3.13
Metadata
Metadata
Labels
Type
Projects
- StatusShow more project fieldsNo status
Activity
github-actions commentedon Nov 4, 2026
🤖 Automated triage reproduction notes (agent-authored — trust but verify)
Agent analysis
The bug is in
python/packages/core/agent_framework/_evaluation.py::tool_called_checkand_extract_tool_calls, which scanitem.conversationrather than the response side of its split. It occurs when a cumulative multi-turnEvalItemhas an earlier function call on the query side. Minimal repro: create two turns with a tool call only in turn 1, select the second result fromEvalItem.per_turn_items, and run the three local tool checks.python/packages/core/tests/core/test_local_eval.py1.24.01.24.014H034160212 commentedon Nov 5, 2026
Eduard van Valkenburg (@eavanvalkenburg) I have a fix with tests ready for this one, if scoping to the split is the intended behaviour:
tool_called_checkand_extract_tool_calls(used bytool_calls_presentandtool_call_args_match) read the response side ofitem.split_messages(), the same messagesitem.responseuses. No API change.LAST_TURNitems are unchanged, and withConversationSplit.FULLthe whole trajectory still counts (covered by a test).Branch: main...14H034160212:agent-framework:fix-10091-tool-check-turns (the per-turn test fails on
mainand passes with the change; lint, type checks and the core test suite pass).Happy to open a PR if this direction works for you.
eavanvalkenburg commentedon Nov 9, 2026
Thanks for the clear reproduction. We will consider this in core MAF: local tool checks should support consistent per-turn evaluation alongside the item's conversation split. Before approving implementation, we need to settle the scope and compatibility contract because the existing check docstrings describe conversation-wide matching. This does not promise implementation or timing; planning should preserve whole-trajectory evaluation and cover existing callers.
eavanvalkenburg commentedon Nov 9, 2026
/plan Resolve the local tool-check scope contract before implementation. Core ownership has been accepted for consideration, not delivery approval. Reconcile the conversation-wide check docstrings with ADR-0023, per_turn_items, and Foundry's split-aware evaluation. Assess response-side matching for tool_called_check, tool_calls_present, and tool_call_args_match, including compatibility for existing multi-turn callers and custom splits. Preserve single-turn behavior and explicit FULL whole-trajectory evaluation; cover all three checks with focused regression tests and clarify migration/documentation needs. Review the existing PR #10186 as a proposal, without treating it or this planning request as implementation approval.
github-actions commentedon Nov 9, 2026
Summary
Three local checks (
tool_called_check,tool_calls_present,tool_call_args_match) readitem.conversation, the whole conversation. Everything else onEvalItemfollows the item's split:query,response, and the Foundry evaluator (item.split_messages(...)at_foundry_evals.pyL986-989). Withper_turn_items, each turn's item holds the conversation up to that turn, so earlier tool calls sit on the query side and still satisfy later turns. I read the code only; I did not run anything. I did not read PR #10186's diff or reviews, so this plan is independent of it.Confirmed facts
per_turn_itemssays it exists to "evaluate each response independently". ADR-0023 says changingsplit_strategy"consistently changes all derived values". Both point to response-side semantics.FULLsplits after the first user message, so the response side is the whole trajectory. Scoping to the response side keeps whole-trajectory evaluation forFULL.LAST_TURNitems: the query side holds only the user message, so results are identical._conversation_splitwhenitem.split_strategyis unset.LocalEvaluatorhas no such setting, so local checks would use only the item's own strategy or theLAST_TURNdefault.Recommendation
Code changes are warranted. Make the local tool checks use the response side of
item.split_messages(), plus a documentation update. A docs-only "documented limitation" would leaveper_turn_itemsunable to give per-turn tool checks, and would make local and Foundry results disagree on the same item. It would also keep the silent false pass the issue describes.Trade-off: this is a behavior change for existing callers. Anyone who passes a multi-turn conversation with the default
LAST_TURNand relies on an earlier-turn call counting will start failing, and the failure is correct under the per-turn contract. The framework treats evals as experimental, but the change still needs an explicit note. The alternative of an opt-in flag would keep the false pass as the default, so I don't recommend it.Proposed plan (proposal only; nothing changed, not delivery approval)
Decide the contract. Local tool checks inspect the response side of
item.split_messages()(item strategy, elseLAST_TURN). Use one private helper for all three checks. Calls in the query side never count.FULLstill counts the whole trajectory after the first user message.Custom splits. The response side is whatever the splitter returns. For example, the
split_before_memoryexample inConversationSplitterputs the memory tool call on the response side, so it counts. A splitter that puts the call on the query side makes the check fail, which matchesitem.responseand Foundry. Document this.Edge cases to pin down. There is no user message (
LAST_TURNreturns the whole conversation as the response, so nothing changes). A tool call earlier in the same turn (counts). Maintainers should confirm whether anyevaluate_workflowsub-agent items can carry multiple user turns under the default split. I did not trace that path, and it is the main place existing callers could see a change.Docs. Reword the three docstrings and the
ConversationSplit/EvalItemtext to say tool checks follow the split, and how to evaluate a whole trajectory (ConversationSplit.FULLorconversation_split=). Add a short migration note for multi-turn callers: useFULL, or pass expected calls per turn. Per repo rules, do not editpython/CHANGELOG.mdin the PR; the release process assembles it.Tests in
test_local_eval.py, for each of the three checks:FULLsplit counts calls from every turn.LAST_TURNunchanged.tool_call_args_match).Done when the new tests pass, existing tests using these checks stay green (the reporter says 323 pass locally; that is unverified by me), and the docs describe the split-aware contract.
Questions for maintainers (not blocking the direction)
LocalEvaluatorever get its own default split like Foundry's_conversation_split? I suggest not now, to keep scope small.14H034160212 commentedon Nov 10, 2026
Thanks Eduard van Valkenburg (@eavanvalkenburg), the plan matches what I had in mind. I've updated my branch to follow it:
_response_function_calls) used bytool_called_check,tool_calls_presentandtool_call_args_match, reading the response side ofitem.split_messages()(item strategy, elseLAST_TURN).ConversationSplitandEvalItem.split_strategynow say that tool checks follow the split, and thatFULLchecks the whole trajectory.test_local_eval.pyfor all three checks: per-turn item where only turn 1 called the tool,FULLacross turns, single-turnLAST_TURNunchanged, a conversation with no user message, a custom splitter that puts the call on the response side or the query side, and two calls with arguments in the current turn. The per-turn, custom-split and current-turn tests fail onmainand pass with the change.test_local_eval.py(103), the core test suite (7505 passed) andtest_foundry_evals.py(144) pass; ruff, mypy and pyright are clean. NoCHANGELOG.mdedit.Branch: main...14H034160212:agent-framework:fix-10091-tool-check-turns-v2
I'll open it as a PR once the direction is marked ready for implementation.
AI assistance: I used Claude (Anthropic) to help with the analysis, implementation and tests; I reviewed the change and ran the checks above myself.