feat(elevenlabs): stream eleven_v3/eleven_v3_conversational via text-to-dialogue - #2347
feat(elevenlabs): stream eleven_v3/eleven_v3_conversational via text-to-dialogue#2347rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 969893d The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if (ctx.timeoutTimer) { | ||
| clearTimeout(ctx.timeoutTimer); | ||
| } |
There was a problem hiding this comment.
🟡 Response timeout disabled after first audio chunk
When audio arrives, clearTimeout clears the per-context watchdog but leaves ctx.timeoutTimer set, so the arm guard at tts.ts never re-arms it on later content sends. If the provider stalls mid-stream after some audio, the stream waits forever instead of timing out.
| if (ctx.timeoutTimer) { | |
| clearTimeout(ctx.timeoutTimer); | |
| } | |
| if (ctx.timeoutTimer) { | |
| clearTimeout(ctx.timeoutTimer); | |
| ctx.timeoutTimer = undefined; | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
| : { | ||
| context_id: contextId, | ||
| error: data.error, | ||
| 'lk.pii.data': data, | ||
| }, |
There was a problem hiding this comment.
🟨 Provider error logged under untagged key
The non-dialogue error branch logs the provider error string as error: data.error, a key with no dot-delimited pii segment. REVIEW.md requires provider response payloads to use an lk.pii.<name> marker; the sibling dialogue branch already uses 'lk.pii.error'. This value reaches logs unredacted.
Was this helpful? React with 👍 or 👎 to provide feedback.
Ports livekit/agents#6838 to the ElevenLabs JS plugin.
Summary
eleven_v3andeleven_v3_conversationalHTTP and WebSocket synthesis through ElevenLabs text-to-dialogueImplementation gap
The JS streaming input queue does not retain replayable text after a context-level failure. Dialogue provider errors and context timeouts therefore terminate the input/tokenizer tasks and are non-retryable, rather than honoring
maxRetryas the Python stream wrapper can. Porting retry parity would require a core replayable streaming-input redesign outside the ElevenLabs plugin; this PR avoids duplicate or lost text instead of silently attempting an unsafe retry.Source diff coverage
livekit-plugins/livekit-plugins-elevenlabs/livekit/plugins/elevenlabs/models.py: adapted toplugins/elevenlabs/src/models.ts; addseleven_v3_conversationaland the dialogue-model prefix classifier using the TypeScript model union.livekit-plugins/livekit-plugins-elevenlabs/livekit/plugins/elevenlabs/tts.py: adapted toplugins/elevenlabs/src/tts.ts; preserves text-to-dialogue HTTP and multiplexed WebSocket routing, request framing, supported-setting filtering and warnings, pronunciation dictionaries, snake-case response parsing, error handling, late-audio suppression, immediate context close, and per-context keep-alives within the target's private-field connection architecture.tests/test_plugin_elevenlabs_tts.py: adapted toplugins/elevenlabs/src/tts.test.ts; ports the source model classification, endpoint/body, warning, WebSocket framing, alignment/finality, error, late-audio, close, and keep-alive scenarios with Vitest and local HTTP/WebSocket doubles.Target-only support files: generated API report
plugins/elevenlabs/etc/agents-plugin-elevenlabs.api.mdand minor changeset.changeset/bright-dialogue-streams.md.Validation
pnpm test plugins/elevenlabs(36 passed, 2 credential-gated skipped)pnpm buildpnpm lintpnpm format:checkpnpm --filter @livekit/agents-plugin-elevenlabs api:checkcue-cliruntime validation was not run becauseELEVEN_API_KEYis unavailable in the environment. The source-equivalent dialogue protocol behavior is covered by local HTTP/WebSocket tests.Ported from livekit/agents#6838
Original PR description
Summary
Adds streaming support for eleven_v3 / eleven_v3_conversational by routing them through ElevenLabs' text-to-dialogue API — the regular text-to-speech streaming websocket (multi-stream-input) rejects these models outright today, so SynthesizeStream currently fails for them.
Quickstart guide here
Websocket documentation here
Testing