Repository navigation
Python: Scope local tool checks to the response side of the conversation split - #10245
Qiming Bao (14H034160212) wants to merge 5 commits into
Conversation
tool_called_check, tool_calls_present and tool_call_args_match scanned item.conversation, so a tool call made in an earlier turn (on the query side of the split) satisfied the check for a later turn. With EvalItem.per_turn_items(), a turn with no tool call still passed. Inspect item.split_messages()[1] instead, the same response messages that item.response and the Foundry tool evaluators use. For single-turn items with the default LAST_TURN split the result is unchanged; with FULL, the whole trajectory after the first user message still counts. Fixes microsoft#10091
Use one helper for tool_called_check, tool_calls_present and tool_call_args_match, document that tool checks follow the split, and add tests for single-turn items, items without a user message, custom splits and calls with arguments in the current turn.
There was a problem hiding this comment.
🟡 Changes recommended
The conversation factory shared by six tests remains a test-class static method instead of a conftest.py fixture or explicitly imported helper, contrary to the repository’s coding standard.
1 open issue
What changed in this PR
This PR fixes #10091 by aligning Python’s local tool checks with EvalItem’s conversation split, preventing earlier-turn tool calls from falsely satisfying later-turn checks.
Changes:
- Adds a shared helper that extracts function calls only from response-side messages.
- Documents split-aware checks and
FULLfor whole-trajectory evaluation. - Adds regression coverage for per-turn isolation, custom splits, argument matching, and unchanged single-turn behavior.
| File | Description |
|---|---|
python/packages/core/tests/core/test_local_eval.py |
Adds six tests covering split-aware behavior across all three tool checks. |
python/packages/core/agent_framework/_evaluation.py |
Centralizes response-side function-call extraction and updates split and check docstrings. |
Review effort: Balanced
| @staticmethod | ||
| def _two_turn_conversation() -> list[Message]: |
There was a problem hiding this comment.
Good point. Fixed in 5ebd65e: the conversation is now the two_turn_conversation fixture in tests/core/conftest.py, and the six tests take it through pytest injection, as CODING_STANDARD.md asks. test_local_eval.py passes, and ruff and pyright are clean.
Follow python/CODING_STANDARD.md: shared setup lives in conftest.py and is consumed through pytest injection.
pyright and mypy match the protocol by parameter name, so splitters named conv(...) did not satisfy ConversationSplitter.
|
Qiming Bao (@14H034160212) Thanks for the contribution. This PR duplicates the older open #10186, which implements the same #10091 behavior in the same core evaluation code and tests. To keep review focused on one implementation, I’m closing this newer PR. |

Motivation & Context
tool_called_check,tool_calls_presentandtool_call_args_matchscanneditem.conversation, the whole conversation, for function calls. Every other derived value onEvalItem(query,response) and the Foundry evaluator follow the item's split strategy.As a result, in multi-turn evaluation a tool call from an earlier turn satisfied the check for a later turn. With
EvalItem.per_turn_items(...), a turn in which the agent made no tool call still passed all three checks, because the earlier call sits on the query side of that turn's split. This is a silent false pass on the turns that per-turn evaluation is meant to isolate, and it contradicts ADR-0023 ("Changing thesplit_strategyconsistently changes all derived values").This implements the plan agreed in #10091.
Description & Review Guide
_response_function_calls, returns the function calls on the response side ofitem.split_messages()(the item's strategy, elseLAST_TURN).tool_called_check,tool_calls_presentandtool_call_args_matchall use it, so they look at the same messages asitem.response.ConversationSplitandEvalItem.split_strategynow say that the tool checks follow the split, and thatConversationSplit.FULLchecks the whole trajectory.LAST_TURNsplit (for example fromevaluate_agent) give the same results, because the query side holds only the user message. WithFULL, every call after the first user message still counts. The behavior change is for multi-turn items: tool calls on the query side, such as calls from earlier turns, no longer count. Callers who want whole-conversation matching can useConversationSplit.FULL. A custom splitter decides which calls count, the same way it decidesitem.response.python/CHANGELOG.md, following the repo rules.Tests. New
TestToolChecksUseConversationSplitinpython/packages/core/tests/core/test_local_eval.py. Each test covers all three checks:test_per_turn_tool_checks_ignore_calls_from_prior_turn: only turn 1 calls the tool. Turn 1 passes and turn 2 fails.test_full_split_counts_whole_trajectory: withFULL, calls from every turn count.test_single_turn_last_turn_unchangedtest_conversation_without_user_message_counts_all_callstest_custom_split_decides_which_calls_count: a custom splitter that puts the call on the response side or on the query side.test_calls_in_the_current_turn_count_and_match_arguments: two calls with arguments in the current turn.The per-turn, custom-split and current-turn tests fail on
mainand pass with this change. The others pass on both and guard the behavior that should not change.After merging the latest
main:test_local_eval.py: 107 passedruff checkandruff format --check: cleanpyrighton_evaluation.py: 0 errorsRelated Issue
Fixes #10091
#10186 also targets this issue. It was opened before the plan in #10091 was posted. This PR follows that plan: one shared helper, docstrings for
ConversationSplitandEvalItem, and tests for all three checks, including custom splits.AI Assistance
AI assistance details: I used Claude (Anthropic) to help with the analysis, implementation, tests and this description. I reviewed the change and ran the tests and checks listed above myself.
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.I left the breaking-change box unticked. There is no API change, but multi-turn results change as described above. Evals are experimental, so I did not add the
breaking changelabel myself. Please add it if you treat this as breaking.