fix(cartesia): reuse one TTS websocket across generations#2085
fix(cartesia): reuse one TTS websocket across generations#2085sunnyysetia wants to merge 4 commits into
Conversation
🦋 Changeset detectedLatest commit: 65283c5 The changes in this PR will be included in the next version bump. This PR includes changesets to release 38 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
`SynthesizeStream.run()` opened a fresh Cartesia WebSocket for every synthesis and force-terminated it in `finally`, so each turn repaid the TCP/TLS and WebSocket handshake. Cartesia documents the opposite: keep one socket open and run many generations over it, because a new connection costs tens to low hundreds of milliseconds per turn. The Python plugin already does this with a pooled connection, and the JS plugin even carried a `so connection can be reused` comment that was never wired to a pool. Hold a `ConnectionPool<WebSocket>` on the `TTS` instance, the same pattern the fishaudio, inworld, and xai plugins use, and run each generation inside `pool.withConnection`. A generation no longer closes the socket on `done`, so the pool hands the same socket to the next turn and only the first turn pays the connect. The chunk-timeout watchdog now poisons and discards its socket instead of closing a pooled one, and the socket is invalidated only when a field that is sent at handshake time changes (apiKey, apiVersion, baseUrl); model, voice, encoding, sample rate, and the generation controls travel in-band per message, so a pooled socket serves new values without reconnecting. Adds `TTS.prewarm()` to open the socket before the first turn and a `TTS.close()` that drains the pool. New tests cover socket reuse across turns, prewarm, discard-on-failure, idle-close replacement, and close.
f480c6b to
72d40b8
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f480c6bb29
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| async #connectWebSocket(timeoutMs: number): Promise<WebSocket> { | ||
| const wsUrl = this.#opts.baseUrl.replace(/^http/, 'ws'); |
There was a problem hiding this comment.
When a stream is created and TTS.updateOptions() changes apiKey, apiVersion, or baseUrl before that stream's pool checkout opens a socket, this helper reads the mutable TTS.#opts instead of the stream's snapshotted options. The stream still sends its old per-generation payload from SynthesizeStream.#opts, so the request can be sent over a WebSocket authenticated to a different key/version or pointed at a different host; before this change the URL and headers were built from the stream's own options. Please keep handshake options with the stream/pool entry or cancel/version in-flight connects on option changes.
Useful? React with 👍 / 👎.
| if (ws.readyState !== WebSocket.OPEN) { | ||
| throw new APIConnectionError({ message: 'Cartesia pooled websocket is not open' }); |
There was a problem hiding this comment.
When Cartesia or a proxy closes a pooled socket while it is idle, there are no generation listeners attached, so the pool keeps that closed WebSocket in available. The next synthesis hits this readyState check and spends a normal TTS retry just to discard it; with maxRetry: 0 the turn fails before any transcript is sent, whereas the old per-turn connection would have opened a fresh socket. Please drop closed idle sockets before returning them from the pool or attach an idle close handler that removes them.
Useful? React with 👍 / 👎.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
… opts Addresses review feedback on the pooling change. - Attach an idle close and error handler when a pooled socket is created, so a socket that closes between turns is removed from the pool instead of being handed to the next turn, where the readyState guard would spend a retry, or fail at maxRetry:0, just to discard it. - Re-check the handshake inputs (apiKey, apiVersion, baseUrl) after connect and reconnect if one changed mid-connect, so a pooled socket is never built on stale credentials. Strengthens the idle-close test to assert recovery at maxRetry:0.
|
Thanks, both are fair. Addressed in 00780c4: Idle-closed sockets burning a retry. When a pooled socket is created it now gets an idle Handshake options read live during connect. |
Addresses review feedback. If the WebSocket closed or errored before the generation's done message, recvTask ended its loop and returned normally, so the turn was treated as finished: the agent went silent for the rest of it and the dead socket could be handed back to the pool. Track completion and capture an early close or error as a pending error, then throw it after the receive loop. The turn now fails and the framework retries, and withConnection removes the dead socket instead of pooling it. Adds a test that drops the socket after one chunk and asserts fail-over plus a fresh reconnect.
4cc0cd8 to
e590394
Compare
|
Addressed the newer feedback in the latest commits (e590394): Mid-generation socket drop (Devin). Good catch. Idle-closed socket / readyState guard (Codex, re-flag). The common idle-close path no longer reaches that guard: a pooled socket now carries a |
2 participants
Problem
plugins/cartesia/src/tts.tsopens a new Cartesia WebSocket for every streaming synthesis and force-terminates it when the generation ends.SynthesizeStream.run()connects per call:closes it on the normal
donepath:and tears it down again in
finally:The
TTSinstance holds no connection state, so every turn repays the TCP, TLS, and WebSocket handshake. Cartesia recommends the opposite: keep one socket open and run many generations over it, because a fresh connection costs tens to low hundreds of milliseconds per turn (https://docs.cartesia.ai/use-the-api/compare-tts-endpoints). The Python plugin already follows that advice with a pooled connection (livekit-plugins-cartesia,utils.ConnectionPool,max_session_duration=300), and the JSrecvTaskeven carried a// IMPORTANT: Remove listeners so connection can be reusedcomment that was never wired to a pool.Everything Cartesia needs per generation (model, voice, encoding, sample rate, speed, emotion, volume, language) already travels in-band on each message, and a fresh
context_idis generated perrun(), so the socket is safe to reuse as-is. Only the auth and version headers are set at handshake time.Fix
Hold a
ConnectionPool<WebSocket>on theTTSinstance, the same patternplugins/fishaudio,plugins/inworld, andplugins/xaialready use, and run each generation insidepool.withConnection:done; it just breaks the receive loop, sowithConnectionreturns the socket to the pool and the next turn skips the handshake.APITimeoutError, sowithConnectionremoves that socket from the pool instead of returning a stuck one.updateOptionsinvalidates the pool only when a handshake input actually changes (apiKey,apiVersion,baseUrl). Model, voice, and the generation controls are sent per message, so a pooled socket serves the new values without reconnecting.TTS.prewarm()to open the socket before the first turn andTTS.close()to drain the pool.maxSessionDurationis 300000 ms withmarkRefreshedOnGet, matching the Python plugin.This mirrors the Python plugin's behaviour and keeps the existing IPv4 happy-eyeballs connect retry, chunk-timeout, and error-propagation logic intact.
Verification
Added a local-WebSocketServer test suite in
plugins/cartesia/src/tts.test.tsthat needs no API key (mirrorsplugins/fishaudio/src/tts.test.ts):connectionCount === 1)vitest run plugins/cartesia/src/tts.test.ts: 5 passed, 1 skipped (the key-gated integration test). Build,typecheck,lint, andprettierare clean.api:checkfor this plugin fails, but it fails identically on a clean checkout ofmain(there is no checked-inetc/*.api.mdreport for the plugin), so it is unrelated to this change. Added aminorchangeset.I also measured the actual patched plugin against direct Cartesia Sonic from a hosted us-east worker, 11 sequential turns, first-audio latency:
Pooling removes about 102 ms at p50 and collapses the tail, because the reconnect p95 is a handshake landing on a turn while the pooled path has already paid it once. A prewarmed first turn measured 274 ms, within the warm band.