Support adaptive interruption for realtime models#2099
Open
rosetta-livekit-bot[bot] wants to merge 2 commits into
Open
Support adaptive interruption for realtime models#2099rosetta-livekit-bot[bot] wants to merge 2 commits into
rosetta-livekit-bot[bot] wants to merge 2 commits into
Conversation
🦋 Changeset detectedLatest commit: 798b150 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Port livekit/agents#6488 to agents-js.
Source diff coverage
livekit-agents/livekit/agents/inference/interruption.py: ported toagents/src/inference/interruption/types.ts,agents/src/inference/interruption/interruption_stream.ts, andagents/src/inference/interruption/ws_transport.ts. Adds the optionalagentEndedoverlap verdict bit to the target event/sentinel types and propagates it for synthetic non-interruption events; remote interruption detections set it to false.livekit-agents/livekit/agents/voice/agent_activity.py: adapted toagents/src/voice/agent_activity.ts. Same behavior mapped to existing TS names and millisecond time units: adaptive interruption can gate realtime models without STT when server turn detection is off; realtime backchannel turns are dropped; confirmed backchannels clear realtime buffered audio; adaptive interruption eligibility no longer rejects all realtime models.livekit-agents/livekit/agents/voice/audio_recognition.py: adapted toagents/src/voice/audio_recognition.ts. Same behavior mapped to the existing TS task/stream structure and millisecond time units: turn-scoped overlap/backchannel state, agent-ended overlap distinction, confirmed-backchannel hook, and end-of-turn backchannel verdict propagation.tests/test_agent_session.py: not applicable as a direct file port. The changed Python assertion covers preserving realtime user transcription item IDs; the target counterpart already exists inagents/src/voice/agent_activity.test.tsand required no change for this PR.tests/test_realtime_adaptive_interruption.py: adapted toagents/src/voice/realtime_adaptive_interruption.test.tsusing Vitest and target test utilities. Ports the source behavior cases without adding unrelated coverage.Verification
pnpm test -- agents/src/voice/realtime_adaptive_interruption.test.ts agents/src/voice/audio_recognition_interruption.test.ts agents/src/voice/audio_recognition_backchannel.test.tspnpm --filter @livekit/agents typecheckpnpm buildpnpm --filter @livekit/agents lint(passes with existing warnings; isolated worktree config used to avoid duplicate parent-worktree plugin discovery)Cue voice E2E
798b1505becfc2a4df86e8d17dac875d906ff4f6sid_67bbfe6fd281(voice, ended/flushed)gpt-realtime, server turn detection disabled, text modality, no STT, Silero VAD/client-side turn detection, adaptive interruption, and Cartesiasonic-3for a deterministic 317-second initial playout.agent_state_changed(.new_state="AS_SPEAKING")at1995ms.6723mswith:{overlapping_speech(.is_interruption=false) or overlapping_speech(.is_interruption!=true)} -> {!agent_state_changed(.new_state="AS_THINKING")}[5s]17526msand18542ms. The bounded window contained zeroAS_THINKINGorconversation_item_addedevents, so the backchannel caused no user-turn commit or reply.agentEnded: false,isInterruption: false,input_audio_buffer.clear, and the server acknowledgementinput_audio_buffer.clearedfor both overlap segments.~/.cue-cli/sessions/sid_67bbfe6fd281/manifest.json~/.cue-cli/sessions/sid_67bbfe6fd281/events.jsonl~/.cue-cli/sessions/sid_67bbfe6fd281/recordings/001_run/result.json~/.cue-cli/sessions/sid_67bbfe6fd281/recordings/002_run/result.json~/.cue-cli/sessions/sid_67bbfe6fd281/recording.wav~/.cue-cli/sessions/sid_67bbfe6fd281/recordings/002_run/recording.wav/tmp/pr-2099-worker-evidence.logfalse, so the persistedoverlapping_speechobject has no printedis_interruptionkey and the literal=.falsearm alone cannot resolve. The!=truearm is the equivalent persisted predicate; the exact worker event confirmsisInterruption: false.Ported from livekit/agents#6488
Original PR description
Enables client-side adaptive interruption (barge-in) for OpenAI-compatible realtime models when the model's own server-side turn detection is disabled — so a backchannel ("uh-huh") while the agent is speaking doesn't trigger a spurious reply, without requiring a separate STT.
The gate. With no STT there's no transcript to gatekeep, so the audio recognition layer tracks a turn-scoped signal — whether the turn's speech overlapped agent speech and was classified a backchannel — and surfaces it on the end-of-turn info. When set, the user turn is dropped and the buffered input audio is cleared, so no reply is generated.
Difference from the STT gate. The STT path is timestamp-windowed: it drops the backchannel portion of a turn and re-emits the rest. Realtime is all-or-nothing —
clear_audio()wipes the whole input buffer and the model buffers the turn as one unit, so there's no per-timestamp split. It's correct for the dominant cases (pure backchannel, real interruption) and drops only on a confirmed backchannel; a slow verdict after the agent has stopped falls back to committing rather than silently dropping a possible real barge-in.STT + realtime. If you want the finer-grained STT gate, pass an STT alongside the realtime model — the realtime drop path is skipped when an STT is present and transcript-based backchanneling takes over.