fix: preserve completed tool results in chat context after interruption#4877
Draft
CarlosGarciaPro wants to merge 1 commit intolivekit:mainfrom
Draft
fix: preserve completed tool results in chat context after interruption#4877CarlosGarciaPro wants to merge 1 commit intolivekit:mainfrom
CarlosGarciaPro wants to merge 1 commit intolivekit:mainfrom
Conversation
Fixes livekit#3702 When the user interrupts the agent mid-speech, completed tool call results were being discarded. This caused the LLM to re-call the same tools on the next turn, wasting time and API calls. Now, after an interruption, any tool calls that already completed have their FunctionCall + FunctionCallOutput preserved in the agent's chat context. The LLM can reference these results on the next turn without re-executing. Applied to both the stateless LLM pipeline (_pipeline_reply_task) and the realtime LLM pipeline (_realtime_pipeline_reply_task).
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3702
When a user interrupts the agent mid-speech, any tool calls that had already completed were being discarded from the chat context. On the next turn, the LLM would re-call the same tools — wasting time, API calls, and creating a poor user experience.
This PR preserves completed
FunctionCall+FunctionCallOutputpairs in the agent's_chat_ctxeven after interruption, so the LLM can reference those results on the next turn without re-executing.Problem
In
_pipeline_reply_task(and_realtime_pipeline_reply_task), whenspeech_handle.interruptedis true:cancel_and_wait(exe_task)is called — which internally waits for in-flight tool tasks to finish (via theCancelledErrorhandler in_execute_tools_task)tool_output.outputmay contain completed resultsreturns — those results are thrown awayThis is especially problematic for:
Changes
Stateless LLM pipeline (
_pipeline_reply_task):cancel_and_wait(exe_task), iteratestool_output.outputfnc_callandfnc_call_outintoself._agent._chat_ctxviainsert()_tool_items_added()Realtime LLM pipeline (
_realtime_pipeline_reply_task):_tool_execution_started_cbalready addsfnc_callto the chat context in the realtime path, onlyfnc_call_outis addeditems.append()consistent with the existing realtime path patternBoth paths include debug logging to track when tool results are preserved.
Behavior
Test plan
_chat_ctxon next turnfnc_call_outadded (not duplicatefnc_call)