Skip to content

Commit 94a78d1

Browse files
TaoChenOSUTaoChenOSU
authored
Python: Add on_intermediate_message callback to the Agent abstraction (#11321)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Currently, our agents' `invoke` and `invoke_stream` have inconsistent behavior where the responses may contain different contents. For example, the chat completion agent (`invoke`) may return all content types generated by the agent including function call/result contents that are meant to be internal only, while the assistant agent doesn't return those contents. This creates confusion both for users and developers of SK of what an agent should return when working with agents. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> This PR standardizes the behavior of agents by adding an `on_intermediate_message ` parameter to the `invoke` and `invoke_stream` methods in the `Agent` abstraction. This parameter accepts a callback function that will be called whenever a new message is generated by the agent, except the last one that gets returned to the caller. This allows the caller to inspect what the agent did after a response is returned. In addition, this PR also converges on the behavior of the `invoke` and `invoke_stream` methods (respectively) to return the same content types from all agents, making it easier for users to work with agents ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄
1 parent f00dc80 commit 94a78d1

File tree

25 files changed

+1577
-857
lines changed

25 files changed

+1577
-857
lines changed

‎python/samples/concepts/README.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,8 @@
1010
- [Azure AI Agent with Azure AI Search](./agents/azure_ai_agent/azure_ai_agent_azure_ai_search.py)
1111
- [Azure AI Agent File Manipulation](./agents/azure_ai_agent/azure_ai_agent_file_manipulation.py)
1212
- [Azure AI Agent Prompt Templating](./agents/azure_ai_agent/azure_ai_agent_prompt_templating.py)
13-
- [Azure AI Agent Chat History Callback](./agents/azure_ai_agent/azure_ai_agent_streaming_chat_history_callback.py)
13+
- [Azure AI Agent Message Callback Streaming](./agents/azure_ai_agent/azure_ai_agent_message_callback_streaming.py)
14+
- [Azure AI Agent Message Callback](./agents/azure_ai_agent/azure_ai_agent_message_callback.py)
1415
- [Azure AI Agent Streaming](./agents/azure_ai_agent/azure_ai_agent_streaming.py)
1516
- [Azure AI Agent Structured Outputs](./agents/azure_ai_agent/azure_ai_agent_structured_outputs.py)
1617

@@ -31,6 +32,8 @@
3132
- [Chat Completion Agent as Kernel Function](./agents/chat_completion_agent/chat_completion_agent_as_kernel_function.py)
3233
- [Chat Completion Agent Function Termination](./agents/chat_completion_agent/chat_completion_agent_function_termination.py)
3334
- [Chat Completion Agent Templating](./agents/chat_completion_agent/chat_completion_agent_prompt_templating.py)
35+
- [Chat Completion Agent Message Callback Streaming](./agents/chat_completion_agent/chat_completion_agent_message_callback_streaming.py)
36+
- [Chat Completion Agent Message Callback](./agents/chat_completion_agent/chat_completion_agent_message_callback.py)
3437
- [Chat Completion Agent Summary History Reducer Agent Chat](./agents/chat_completion_agent/chat_completion_agent_summary_history_reducer_agent_chat.py)
3538
- [Chat Completion Agent Summary History Reducer Single Agent](./agents/chat_completion_agent/chat_completion_agent_summary_history_reducer_single_agent.py)
3639
- [Chat Completion Agent Truncate History Reducer Agent Chat](./agents/chat_completion_agent/chat_completion_agent_truncate_history_reducer_agent_chat.py)
@@ -51,9 +54,9 @@
5154
- [OpenAI Assistant Chart Maker](./agents/openai_assistant/openai_assistant_chart_maker.py)
5255
- [OpenAI Assistant File Manipulation Streaming](./agents/openai_assistant/openai_assistant_file_manipulation_streaming.py)
5356
- [OpenAI Assistant File Manipulation](./agents/openai_assistant/openai_assistant_file_manipulation.py)
54-
- [OpenAI Assistant File Manipulation Streaming](./agents/openai_assistant/openai_assistant_file_manipulation_streaming.py)
5557
- [OpenAI Assistant Retrieval](./agents/openai_assistant/openai_assistant_retrieval.py)
56-
- [OpenAI Assistant Streaming Chat History Callback](./agents/openai_assistant/openai_assistant_streaming_chat_history_callback.py)
58+
- [OpenAI Assistant Message Callback Streaming](./agents/openai_assistant/openai_assistant_message_callback_streaming.py)
59+
- [OpenAI Assistant Message Callback](./agents/openai_assistant/openai_assistant_message_callback.py)
5760
- [OpenAI Assistant Streaming](./agents/openai_assistant/openai_assistant_streaming.py)
5861
- [OpenAI Assistant Structured Outputs](./agents/openai_assistant/openai_assistant_structured_outputs.py)
5962
- [OpenAI Assistant Templating Streaming](./agents/openai_assistant/openai_assistant_templating_streaming.py)

‎python/samples/concepts/agents/README.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
This project contains a step by step guide to get started with _Semantic Kernel Agents_ in Python.
44

5-
## PyPI:
5+
## PyPI
66

77
- For the use of Chat Completion agents, the minimum allowed Semantic Kernel pypi version is 1.3.0.
88
- For the use of OpenAI Assistant agents, the minimum allowed Semantic Kernel pypi version is 1.4.0.
@@ -12,7 +12,6 @@ This project contains a step by step guide to get started with _Semantic Kernel
1212
- For the use of Crew.AI as a plugin, the minimum allowed Semantic Kernel pypi version is 1.21.1.
1313
- For the use of OpenAI Responses agents, the minimum allowed Semantic Kernel pypi version is 1.27.0.
1414

15-
1615
## Source
1716

1817
- [Semantic Kernel Agent Framework](../../../semantic_kernel/agents/)

‎python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_message_callback.py‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
# Copyright (c) Microsoft. All rights reserved.
2+
3+
import asyncio
4+
from typing import Annotated
5+
6+
from azure.identity.aio import DefaultAzureCredential
7+
8+
from semantic_kernel.agents import AzureAIAgent, AzureAIAgentSettings, AzureAIAgentThread
9+
from semantic_kernel.contents import FunctionCallContent, FunctionResultContent
10+
from semantic_kernel.contents.chat_message_content import ChatMessageContent
11+
from semantic_kernel.functions import kernel_function
12+
13+
"""
14+
The following sample demonstrates how to create an Azure AI Agent
15+
and use it with functions. In order to answer user questions, the
16+
agent internally uses the functions. These internal steps are returned
17+
to the user as part of the agent's response. Thus, the invoke method
18+
configures a message callback to receive the agent's internal messages.
19+
20+
The agent is configured to use a plugin that provides a list of
21+
specials from the menu and the price of the requested menu item.
22+
"""
23+
24+
25+
# Define a sample plugin for the sample
26+
class MenuPlugin:
27+
"""A sample Menu Plugin used for the concept sample."""
28+
29+
@kernel_function(description="Provides a list of specials from the menu.")
30+
def get_specials(self) -> Annotated[str, "Returns the specials from the menu."]:
31+
return """
32+
Special Soup: Clam Chowder
33+
Special Salad: Cobb Salad
34+
Special Drink: Chai Tea
35+
"""
36+
37+
@kernel_function(description="Provides the price of the requested menu item.")
38+
def get_item_price(
39+
self, menu_item: Annotated[str, "The name of the menu item."]
40+
) -> Annotated[str, "Returns the price of the menu item."]:
41+
return "$9.99"
42+
43+
44+
intermediate_steps: list[ChatMessageContent] = []
45+
46+
47+
async def handle_intermediate_steps(message: ChatMessageContent) -> None:
48+
intermediate_steps.append(message)
49+
50+
51+
async def main() -> None:
52+
ai_agent_settings = AzureAIAgentSettings.create()
53+
54+
async with (
55+
DefaultAzureCredential() as creds,
56+
AzureAIAgent.create_client(
57+
credential=creds,
58+
conn_str=ai_agent_settings.project_connection_string.get_secret_value(),
59+
) as client,
60+
):
61+
AGENT_NAME = "Host"
62+
AGENT_INSTRUCTIONS = "Answer questions about the menu."
63+
64+
# Create agent definition
65+
agent_definition = await client.agents.create_agent(
66+
model=ai_agent_settings.model_deployment_name,
67+
name=AGENT_NAME,
68+
instructions=AGENT_INSTRUCTIONS,
69+
)
70+
71+
# Create the AzureAI Agent
72+
agent = AzureAIAgent(
73+
client=client,
74+
definition=agent_definition,
75+
plugins=[MenuPlugin()], # add the sample plugin to the agent
76+
)
77+
78+
# Create a thread for the agent
79+
# If no thread is provided, a new thread will be
80+
# created and returned with the initial response
81+
thread: AzureAIAgentThread = None
82+
83+
user_inputs = [
84+
"Hello",
85+
"What is the special soup?",
86+
"How much does that cost?",
87+
"Thank you",
88+
]
89+
90+
try:
91+
for user_input in user_inputs:
92+
print(f"# User: '{user_input}'")
93+
async for response in agent.invoke(
94+
messages=user_input,
95+
thread=thread,
96+
on_intermediate_message=handle_intermediate_steps,
97+
):
98+
print(f"# Agent: {response}")
99+
thread = response.thread
100+
finally:
101+
# Cleanup: Delete the thread and agent
102+
await thread.delete() if thread else None
103+
await client.agents.delete_agent(agent.id)
104+
105+
# Print the intermediate steps
106+
print("\nIntermediate Steps:")
107+
for msg in intermediate_steps:
108+
if any(isinstance(item, FunctionResultContent) for item in msg.items):
109+
for fr in msg.items:
110+
if isinstance(fr, FunctionResultContent):
111+
print(f"Function Result:> {fr.result} for function: {fr.name}")
112+
elif any(isinstance(item, FunctionCallContent) for item in msg.items):
113+
for fcc in msg.items:
114+
if isinstance(fcc, FunctionCallContent):
115+
print(f"Function Call:> {fcc.name} with arguments: {fcc.arguments}")
116+
else:
117+
print(f"{msg.role}: {msg.content}")
118+
119+
# Sample output:
120+
# User: 'Hello'
121+
# Agent: Hello! How can I assist you today?
122+
# User: 'What is the special soup?'
123+
# Agent: The special soup is Clam Chowder. Would you like to know more about the menu or anything else?
124+
# User: 'How much does that cost?'
125+
# Agent: The Clam Chowder costs $9.99. If you have any more questions or need further assistance, feel free to ask!
126+
# User: 'Thank you'
127+
# Agent: You're welcome! If you have any more questions in the future, don't hesitate to ask. Have a great day!
128+
#
129+
# Intermediate Steps:
130+
# AuthorRole.ASSISTANT: Hello! How can I assist you today?
131+
# Function Call:> MenuPlugin-get_specials with arguments: {}
132+
# Function Result:>
133+
# Special Soup: Clam Chowder
134+
# Special Salad: Cobb Salad
135+
# Special Drink: Chai Tea
136+
# for function: MenuPlugin-get_specials
137+
# AuthorRole.ASSISTANT: The special soup is Clam Chowder. Would you like to know more about the menu or anything
138+
# else?
139+
# Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item":"Clam Chowder"}
140+
# Function Result:> $9.99 for function: MenuPlugin-get_item_price
141+
# AuthorRole.ASSISTANT: The Clam Chowder costs $9.99. If you have any more questions or need further assistance,
142+
# feel free to ask!
143+
# AuthorRole.ASSISTANT: You're welcome! If you have any more questions in the future, don't hesitate to ask.
144+
# Have a great day!
145+
146+
147+
if __name__ == "__main__":
148+
asyncio.run(main())

‎python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_streaming_chat_history_callback.py‎ renamed to ‎python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_message_callback_streaming.py‎

Lines changed: 36 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,14 +6,15 @@
66
from azure.identity.aio import DefaultAzureCredential
77

88
from semantic_kernel.agents import AzureAIAgent, AzureAIAgentSettings, AzureAIAgentThread
9-
from semantic_kernel.contents import ChatHistory, FunctionCallContent, FunctionResultContent
9+
from semantic_kernel.contents import FunctionCallContent, FunctionResultContent
10+
from semantic_kernel.contents.chat_message_content import ChatMessageContent
1011
from semantic_kernel.functions import kernel_function
1112

1213
"""
1314
The following sample demonstrates how to create an Azure AI Agent
1415
and use it with streaming responses. Additionally, the invoke_stream
15-
configures a chat history callback to receive the conversation history
16-
once the streaming invocation is complete. The agent is configured to use
16+
configures a message callback to receive fully formed messages once
17+
the streaming invocation is complete. The agent is configured to use
1718
a plugin that provides a list of specials from the menu and the price
1819
of the requested menu item.
1920
"""
@@ -38,11 +39,11 @@ def get_item_price(
3839
return "$9.99"
3940

4041

41-
final_chat_history = ChatHistory()
42+
intermediate_steps: list[ChatMessageContent] = []
4243

4344

44-
def handle_stream_completion(history: ChatHistory) -> None:
45-
final_chat_history.messages.extend(history.messages)
45+
async def handle_streaming_intermediate_steps(message: ChatMessageContent) -> None:
46+
intermediate_steps.append(message)
4647

4748

4849
async def main() -> None:
@@ -91,7 +92,7 @@ async def main() -> None:
9192
async for response in agent.invoke_stream(
9293
messages=user_input,
9394
thread=thread,
94-
on_complete=handle_stream_completion,
95+
on_intermediate_message=handle_streaming_intermediate_steps,
9596
):
9697
if first_chunk:
9798
print(f"# {response.role}: ", end="", flush=True)
@@ -104,9 +105,9 @@ async def main() -> None:
104105
await thread.delete() if thread else None
105106
await client.agents.delete_agent(agent.id)
106107

107-
# Print the final chat history
108-
print("\nFinal chat history:")
109-
for msg in final_chat_history.messages:
108+
# Print the intermediate steps
109+
print("\nIntermediate Steps:")
110+
for msg in intermediate_steps:
110111
if any(isinstance(item, FunctionResultContent) for item in msg.items):
111112
for fr in msg.items:
112113
if isinstance(fr, FunctionResultContent):
@@ -118,6 +119,31 @@ async def main() -> None:
118119
else:
119120
print(f"{msg.role}: {msg.content}")
120121

122+
# Sample output:
123+
# User: 'Hello'
124+
# AuthorRole.ASSISTANT: Hello! How can I assist you today?
125+
# User: 'What is the special soup?'
126+
# AuthorRole.ASSISTANT: The special soup is Clam Chowder. Would you like to know more about the menu or any
127+
# specific items?
128+
# User: 'How much does that cost?'
129+
# AuthorRole.ASSISTANT: The Clam Chowder costs $9.99. Would you like to explore anything else on the menu?
130+
# User: 'Thank you'
131+
# AuthorRole.ASSISTANT: You're welcome! If you have any more questions or need assistance in the future, feel
132+
# free to ask. Have a great day!
133+
#
134+
# Intermediate Steps:
135+
# AuthorRole.ASSISTANT: Hello! How can I assist you today?
136+
# Function Call:> MenuPlugin-get_specials with arguments: {}
137+
# Function Result:>
138+
# Special Soup: Clam Chowder
139+
# Special Salad: Cobb Salad
140+
# Special Drink: Chai Tea
141+
# for function: MenuPlugin-get_specials
142+
# Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item":"Clam Chowder"}
143+
# Function Result:> $9.99 for function: MenuPlugin-get_item_price
144+
# AuthorRole.ASSISTANT: You're welcome! If you have any more questions or need assistance in the future, feel
145+
# free to ask. Have a great day!
146+
121147

122148
if __name__ == "__main__":
123149
asyncio.run(main())

0 commit comments

Comments
 (0)