Skip to content

fix: preserve completed tool results in chat context after interruption#4877

Draft
CarlosGarciaPro wants to merge 1 commit intolivekit:mainfrom
CarlosGarciaPro:fix/preserve-tool-results-on-interruption
Draft

fix: preserve completed tool results in chat context after interruption#4877
CarlosGarciaPro wants to merge 1 commit intolivekit:mainfrom
CarlosGarciaPro:fix/preserve-tool-results-on-interruption

Conversation

@CarlosGarciaPro
Copy link

Summary

Fixes #3702

When a user interrupts the agent mid-speech, any tool calls that had already completed were being discarded from the chat context. On the next turn, the LLM would re-call the same tools — wasting time, API calls, and creating a poor user experience.

This PR preserves completed FunctionCall + FunctionCallOutput pairs in the agent's _chat_ctx even after interruption, so the LLM can reference those results on the next turn without re-executing.

Problem

In _pipeline_reply_task (and _realtime_pipeline_reply_task), when speech_handle.interrupted is true:

  1. cancel_and_wait(exe_task) is called — which internally waits for in-flight tool tasks to finish (via the CancelledError handler in _execute_tools_task)
  2. At this point, tool_output.output may contain completed results
  3. But the code immediately returns — those results are thrown away
  4. On the next user turn, the LLM has no memory of the tool results, so it calls the same tool again

This is especially problematic for:

  • Slow tools (API lookups, database queries) that take several seconds
  • Stateful tools (reservations, payments) where re-execution has side effects
  • Real-time voice agents where latency from duplicate calls degrades UX

Changes

Stateless LLM pipeline (_pipeline_reply_task):

  • After cancel_and_wait(exe_task), iterates tool_output.output
  • For each completed output, inserts both fnc_call and fnc_call_out into self._agent._chat_ctx via insert()
  • Notifies the session via _tool_items_added()

Realtime LLM pipeline (_realtime_pipeline_reply_task):

  • Same pattern, but since _tool_execution_started_cb already adds fnc_call to the chat context in the realtime path, only fnc_call_out is added
  • Uses items.append() consistent with the existing realtime path pattern

Both paths include debug logging to track when tool results are preserved.

Behavior

Scenario Before After
Interrupted, tools completed Results discarded, LLM re-calls Results preserved in chat_ctx
Interrupted, no tools completed Clean return Clean return (unchanged)
Not interrupted Normal flow Normal flow (unchanged)

Test plan

  • Interrupted speech with completed tools → results appear in _chat_ctx on next turn
  • Interrupted speech with no completed tools → clean return, no change
  • Non-interrupted path → behavior unchanged
  • Realtime pipeline: only fnc_call_out added (not duplicate fnc_call)
  • Existing test suite passes without regressions

Fixes livekit#3702

When the user interrupts the agent mid-speech, completed tool call results
were being discarded. This caused the LLM to re-call the same tools on the
next turn, wasting time and API calls.

Now, after an interruption, any tool calls that already completed have their
FunctionCall + FunctionCallOutput preserved in the agent's chat context. The
LLM can reference these results on the next turn without re-executing.

Applied to both the stateless LLM pipeline (_pipeline_reply_task) and the
realtime LLM pipeline (_realtime_pipeline_reply_task).
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tool Call Results Lost During User Interruption Leading to Duplicate Executions

2 participants