Repository navigation
Python: Use the tool call index for streamed Azure AI Inference tool calls - #14751
Lanre Shittu (Shizoqua) wants to merge 2 commits into
Conversation
…calls The Azure AI Inference connector set the index of each streamed tool call to the choice index, so every parallel tool call in a stream got index 0. Argument deltas after the first one carry only the index, so when the chunks were added together the arguments of the second call were appended to the first call, which then had invalid JSON arguments. Read the index the service sends for each tool call, and fall back to the position in the delta when it is missing.
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the change preserving streamed tool-call indexes.
No open issues 🎉
What changed in this PR
Fixes #14750 by preserving tool-call indexes in the Python Azure AI Inference connector, preventing streamed parallel calls from merging arguments into the wrong function call.
Changes:
- Uses the service-provided tool-call index, falling back to the call’s position in the delta when absent.
- Adds a regression test verifying that accumulated parallel tool calls retain separate indexes and arguments.
| File | Description |
|---|---|
python/tests/unit/connectors/ai/azure_ai_inference/services/test_azure_ai_inference_chat_completion.py |
Adds regression coverage for streamed parallel tool-call accumulation. |
python/semantic_kernel/connectors/ai/azure_ai_inference/services/azure_ai_inference_chat_completion.py |
Uses individual tool-call indexes instead of the choice index. |
Review effort: Balanced
Muhammad Abdullah Butt (1AbdullahButt)
left a comment
There was a problem hiding this comment.
Looks correct. Here's what I checked.
- Why the old index broke parallel calls.
FunctionCallContent.__add__(function_call_content.py:121) raisesContentAdditionExceptionwhen the indexes differ._merge_items_lists(streaming_content_mixin.py:28) uses that exception to decide whether to append a new item or merge into an existing one. With every call atchoice.index(0), the second call's argument-only delta had noidto tell it apart, so it merged into the first call. That's the{"city": "Paris"}{"zone": "Tokyo"}in #14750. - Reading the index.
StreamingChatResponseToolCallUpdateis an azure-coreModel, which is aMutableMappingover the raw JSON. Sotool_call.get("index")reads the service's field even though the typed model doesn't declare it. - The fallback.
idxfromenumerate(choice.delta.tool_calls)gives 0 for the usual one-call delta, which is whatmainproduced. So a service that omitsindexbehaves as before. choice_indexis unchanged. The message keepschoice_index=choice.index, son > 1choices are still kept apart at the message level (streaming_chat_message_content.py:186).git merge-treeagainstmainis clean.
I didn't run the tests. The new test feeds the argument-only deltas without id, which is the case that failed. It would also be worth one case where both calls arrive in a single delta (tool_calls with two entries and no index). That pins the enumerate fallback, so a later change can't quietly put both back at 0.
Reviewed with help from Claude Code.
Cover the fallback where the service sends several tool calls in one delta without an index. They should be indexed by their position.
|
Thanks for the review. I added a test for two tool calls arriving in a single delta without an index. |
Motivation and Context
Fixes #14750
When the Azure AI Inference connector streams parallel tool calls, it set the index of every
FunctionCallContentto the choice index, so all tool calls got index 0. Argument deltas after the first one carry only the tool call index, so when the chunks were added together the arguments of the second call were appended to the first call. The first function was then called with invalid JSON arguments such as{"city": "Paris"}{"zone": "Tokyo"}.Description
_create_streaming_chat_message_contentnow reads the index the service sends for each tool call withtool_call.get("index"). The SDK modelStreamingChatResponseToolCallUpdatehas noindexattribute, but it keeps the raw field. When the index is missing, it falls back to the position of the tool call in the delta. For a single tool call this gives 0, the same as before.Added
test_azure_ai_inference_streaming_parallel_tool_calls_keep_their_index, which streams two parallel tool calls and adds the chunks together. It fails on main with('call_1', 0, 'Weather-get_weather', '{"city": "Paris"}{"zone": "Tokyo"}') != ('call_1', 0, 'Weather-get_weather', '{"city": "Paris"}')and passes with the fix.Contribution Checklist