Skip to content

fix(elevenlabs): don't forward re-sent partial transcripts - #6922

Open
captainbanan wants to merge 1 commit into
livekit:mainfrom
captainbanan:elevenlabs-stt-drop-resent-partials
Open

fix(elevenlabs): don't forward re-sent partial transcripts#6922
captainbanan wants to merge 1 commit into
livekit:mainfrom
captainbanan:elevenlabs-stt-drop-resent-partials

Conversation

@captainbanan

Copy link
Copy Markdown
Contributor

The behaviour

Scribe streams the open segment roughly once a second whether or not its words grew, so a segment it holds open produces a run of byte-identical INTERIM_TRANSCRIPT events. The 1Hz cadence is not a keepalive — it is Scribe's normal partial rate; measured over 6h of production traffic, partials whose text changed arrive with the same median gap (1.01s) as ones that did not. What is abnormal is only that the words stop advancing while the stream keeps ticking.

The framework itself is indifferent: audio_recognition.py stores a partial with a plain assignment (self._audio_interim_transcript = ev.alternatives[0].text), and since the interim → _interrupt_by_audio_activity() path is guarded by self.vad is None, a VAD-based pipeline does nothing with the extra events either.

Application code is not indifferent. Every re-send is emitted as user_input_transcribed, so anything that treats an interim as evidence the user is still talking receives that evidence at 1Hz for as long as the segment stays open — a hold that should last one window lasts the whole stall. We hit this with a reply gate of our own and fixed it on our side, then found a second consumer with the same assumption, which is what prompted moving the fix to where the events are produced.

How often, measured

6h of production traffic, 1599 sessions, 343k interim events:

partials that repeat the previous one 1.95%
sessions with ≥1 repeat 67.8%
sessions with a run ≥5 9.0%
sessions with a run ≥10 3.1%
run length ×2 (a single duplicate) 87.9%
seconds stuck per run: median / p90 / max 1.01s / 2.26s / 33.59s

So 88% of the time this is one harmless extra event. The tail is what bites: the longest runs were 30–35 partials in a row, covering up to 33s of a single segment Scribe never committed. A separate session we traced had 25 identical partials across 24.9s.

The change

Forward a partial only when its text differs from the last one, and clear the stored text whenever the segment it belonged to ends:

ends by why it must clear
a commit with text the normal path — the same words next are a new utterance
an empty commit closes the segment without producing a final
a reconnect a fresh connection transcribes from scratch

Nothing else changes: the first partial of a segment, every advancing partial, START_OF_SPEECH, finals and END_OF_SPEECH are all emitted exactly as before.

Behaviour worth calling out

For pipelines with no VAD (vad is None), a re-sent partial currently keeps calling _interrupt_by_audio_activity() and re-arming the false-interruption resume timer (false_interruption_timeout, default 2.0s). Since re-sends arrive about every second, a paused agent turn can stay paused for as long as Scribe holds the segment open. After this change those re-sends no longer re-arm the timer, so such a turn resumes on schedule. I believe that is the intended behaviour — a re-send carries no new words — but it is a behaviour change for VAD-less setups and not just a noise reduction, so flagging it explicitly.

One residual case this does not cover: if Scribe drops a segment without ever committing it and the next utterance opens with the exact same words, its first partial is read as a re-send and dropped. The plugin cannot see the local VAD, so it has no signal for that boundary; the following partial (or the commit) still comes through, and adding a staleness timer felt worse than the case it would buy.

Tests

tests/test_plugin_elevenlabs_stt.py, four new cases against the existing _process_stream_event harness, each pinned by a mutation only it catches:

test mutation it catches
test_re_sent_partial_transcript_is_dropped drop text != self._last_partial_text
test_advancing_partial_transcripts_are_forwarded over-eager filtering
test_the_same_words_after_a_commit_are_forwarded_again drop the reset on a commit
test_the_same_words_after_an_empty_commit_are_forwarded_again drop the reset on an empty commit

ruff format --check · ruff check · scripts/check_types.py (mypy, 634 source files) · pytest --unit (1996 passed, 5 skipped) · pytest --plugin elevenlabs (29 passed) — all green on main at 5a36124.

Scribe streams the open segment roughly once a second whether or not its
words grew, so a segment it holds open produces a run of byte-identical
INTERIM_TRANSCRIPT events. The framework itself is indifferent to them
(`_audio_interim_transcript` is a plain assignment), but every
`user_input_transcribed` subscriber sees them, and code that reads an
interim as evidence the user is still talking gets that evidence at 1Hz
for as long as the segment stays open.

Forward a partial only when its text differs from the last one, and clear
the stored text whenever the segment ends: on a commit (with or without
text) and on a reconnect, which transcribes from scratch.

Measured over 6h of production traffic (1599 sessions, 343k interim
events): 1.95% of partials repeat the previous one, 67.8% of sessions
contain at least one, and the longest runs were 30-35 partials in a row
covering up to 33s of a single unfinalised segment.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@captainbanan
captainbanan force-pushed the elevenlabs-stt-drop-resent-partials branch from b1a36cd to 07c8a0e Compare August 20, 2026 11:40
@captainbanan
captainbanan marked this pull request as ready for review August 20, 2026 11:47
@captainbanan
captainbanan requested a review from a team as a code owner August 20, 2026 11:47

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant