feat(synthesia): add interactive avatar plugin - #2486
feat(synthesia): add interactive avatar plugin#2486rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 48c1513 The changes in this PR will be included in the next version bump. This PR includes changesets to release 40 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 7 potential issues.
4 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| const audioOutput = this.audioOutput; | ||
| this.audioOutput = undefined; | ||
| await audioOutput?.aclose(); |
There was a problem hiding this comment.
🔴 Closed avatar tail breaks speech
When aclose() runs before the agent session ends, audioOutput remains installed after closing. Pending playout stalls, and later speech fails against the closed output.
Learn more
The Synthesia output becomes the leaf of agentSession.output.audio, either directly or beneath recorder and transcription wrappers. Closing the DataStreamAudioOutput does not remove that leaf or settle its counted playback segments. Its captureFrame rejects every later frame once closed, while waitForPlayout() can remain blocked because no further playback RPC reaches the deleted handler. This path runs during explicit avatar closure and automatic teardown after the avatar disconnects.
Example: An avatar disconnects while one reply is playing. Automatic teardown closes the data-stream output. The current reply remains in playout, and the next generated reply rejects its first frame instead of continuing without avatar audio.
Recommended fix: Retain enough ownership information to remove or restore the exact tail during teardown. Before closing it, detach it from AgentOutput, settle interrupted segments through every retained wrapper, and ensure subsequent captures no longer route to the closed sink.
Was this helpful? React with 👍 or 👎 to provide feedback.
| this.room.off(RoomEvent.ConnectionStateChanged, this.onRoomConnectionStateChanged); | ||
| this.startTask?.cancel(); |
There was a problem hiding this comment.
🔴 Pre-connect capture never cancels
Closing after a disconnected-room captureFrame() leaves startTask waiting forever. roomConnectedFuture ignores cancellation after its resolving listener is removed.
Learn more
A capture made before room connection creates startTask, and _start first awaits roomConnectedFuture. The abort signal is only passed to later participant and track waits. Closing removes the connection-state listener before aborting, so a disconnected room can no longer resolve that future. The pending capture retains the task, output, and room indefinitely.
Example: Construct the output with room.isConnected === false, call captureFrame(), then call aclose() before connection. aclose() returns, but the capture promise never resolves or rejects.
Recommended fix: Make the room-connection wait abortable and use cancelAndWait() during closure. Remove the room listener only after cancellation has released the wait, or explicitly reject the connection future on close.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if (this.streamWriter) { | ||
| await this.streamWriter.close(); | ||
| this.streamWriter = undefined; | ||
| } | ||
| if ( | ||
| DataStreamAudioOutput._playbackFinishedHandlers[this.destinationIdentity] === | ||
| this.playbackFinishedHandler | ||
| ) { | ||
| delete DataStreamAudioOutput._playbackFinishedHandlers[this.destinationIdentity]; |
There was a problem hiding this comment.
🔴 Stream failure aborts avatar cleanup
When streamWriter.close() rejects, aclose() skips RPC-handler cleanup. The rejection also prevents closeImpl() from removing lifecycle listeners and the avatar participant.
Learn more
Stream closure can reject during a disconnect or transport failure. The handler deletions follow the awaited close without a finally, so they do not run after rejection. The caller closeImpl similarly awaits this method before unregistering its room listeners and calling the base avatar cleanup. One stream error therefore skips every later cleanup stage.
Example: The avatar's data connection drops while a byte stream is open. streamWriter.close() rejects. The Synthesia room listeners remain registered, and the base session never attempts to remove the avatar participant.
Recommended fix: Make each cleanup stage independent with try/finally or Promise.allSettled. Always clear the writer reference and RPC handlers, and make AvatarSession.closeImpl() run listener removal and super.aclose() even when output closure fails; preserve the first error after cleanup finishes.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if (current.pendingPlayoutSegments > 0) { | ||
| current.onPlaybackFinished({ playbackPosition: 0, interrupted: true }); | ||
| } |
There was a problem hiding this comment.
🔴 Tail swap strands queued segments
With multiple pending segments, replaceAudioTail() settles only one after clearing the old leaf. Remaining segments lose their completion source, so playout waits never finish.
Learn more
pendingPlayoutSegments is a count, not a boolean. A wrapper can have several flushed segments still queued at its leaf. Clearing and detaching that leaf prevents all of their real finish events, but the replacement emits only one synthetic finish. The wrapper's playback count therefore remains behind its capture count.
Example: A wrapper has two flushed segments pending when an avatar starts late. The swap clears both from the previous sink but reports one interruption. A caller waiting for the second segment remains blocked because the detached sink can no longer report it.
Recommended fix: Snapshot the pending count before mutation and reconcile every abandoned segment in order. Use wrapper-specific settlement paths where required so transcription and recorder queues receive one interrupted completion per abandoned segment.
Was this helpful? React with 👍 or 👎 to provide feedback.
| throw new SynthesiaError(`avatar swap RPC failed: ${String(cause)}`, { | ||
| type: ErrorType.CONNECTION, | ||
| cause, | ||
| }); | ||
| } | ||
|
|
||
| let response: unknown; | ||
| try { | ||
| response = JSON.parse(raw) as unknown; | ||
| } catch (cause) { | ||
| throw new SynthesiaError('avatar swap returned a malformed response', { cause }); | ||
| } | ||
| const result = isRecord(response) ? response.avatar_id : undefined; | ||
| if (!isRecord(response) || response.error || typeof result !== 'string') { | ||
| const detail = isRecord(response) ? response.error : undefined; | ||
| throw new SynthesiaError(`avatar swap failed: ${detail || raw}`); |
There was a problem hiding this comment.
🟨 Swap errors expose provider content
A failed swapAvatar() embeds the RPC error or raw response in SynthesiaError. Automatic error logging can expose unredactable provider content and cause chains.
Was this helpful? React with 👍 or 👎 to provide feedback.
| private beginTeardown() { | ||
| if (this.teardownPromise || this.state === State.CLOSED) return; | ||
| this.teardownPromise = this.aclose(); | ||
| void this.teardownPromise.catch((error) => log().error({ error }, 'avatar teardown failed')); |
There was a problem hiding this comment.
🟨 Teardown logs untagged exception content
Background teardown logs error and its cause chain under an untagged key. Swap and transport exceptions can contain provider payloads or credentials.
Was this helpful? React with 👍 or 👎 to provide feedback.
| const response = await this.fetch(url, { | ||
| method: 'POST', | ||
| headers: { Authorization: this.#apiKey, 'Content-Type': 'application/json' }, | ||
| body: JSON.stringify(payload), | ||
| signal: AbortSignal.timeout(connOptions.timeoutMs), |
There was a problem hiding this comment.
🟨 Custom API URL permits plaintext credentials
An http:// apiUrl sends the Synthesia key and minted LiveKit token without transport encryption. Environment or configuration mistakes expose both credentials in transit.
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
Ports livekit/agents#7216 to Node.js by adding
@livekit/agents-plugin-synthesia.swapAvatar()callsSource diff coverage
File-by-file classification for livekit/agents#7216
livekit-agents/pyproject.toml: Not applicable. agents-js consumers install independently published plugins; it has no Python-style package extras.livekit-plugins/livekit-plugins-synthesia/README.md: Adapted toplugins/synthesia/README.md; npm installation and TypeScript usage replace pip/Python syntax.livekit-plugins/livekit-plugins-synthesia/livekit/plugins/synthesia/__init__.py: Adapted toplugins/synthesia/src/index.ts; exports and plugin registration use agents-js conventions.livekit-plugins/livekit-plugins-synthesia/livekit/plugins/synthesia/api.py: Ported toplugins/synthesia/src/api.ts;fetchand millisecond connection options replace aiohttp and Python seconds.livekit-plugins/livekit-plugins-synthesia/livekit/plugins/synthesia/avatar.py: Ported toplugins/synthesia/src/avatar.ts; rtc-node events/RPC, livekit-server-sdk tokens, and agents-js avatar/data-stream APIs replace Python equivalents. Required output-chain and close support is adapted inagents/src/voice/io.ts,agents/src/voice/transcription/synchronizer.ts, andagents/src/voice/avatar/datastream_io.ts.livekit-plugins/livekit-plugins-synthesia/livekit/plugins/synthesia/errors.py: Ported toplugins/synthesia/src/errors.ts;ErrorTypeandSynthesiaError extends APIErrorretain the source metadata and retryability behavior.livekit-plugins/livekit-plugins-synthesia/livekit/plugins/synthesia/log.py: Ported toplugins/synthesia/src/log.tsusing the agents-js child logger.livekit-plugins/livekit-plugins-synthesia/livekit/plugins/synthesia/py.typed: Not applicable. The TypeScript package emits native declaration files.livekit-plugins/livekit-plugins-synthesia/livekit/plugins/synthesia/types.py: Ported toplugins/synthesia/src/types.ts; Python timeout seconds are adapted to standard JS milliseconds.livekit-plugins/livekit-plugins-synthesia/livekit/plugins/synthesia/version.py: Adapted toplugins/synthesia/package.jsonand the build-time package version.livekit-plugins/livekit-plugins-synthesia/pyproject.toml: Adapted toplugins/synthesia/package.json,tsconfig.json,tsup.config.ts, andapi-extractor.jsonfollowing neighboring avatar packages.pyproject.toml: Not applicable.pnpm-workspace.yamlalready includesplugins/*automatically.tests/test_plugin_synthesia.py: Adapted toplugins/synthesia/src/api.test.ts,avatar.test.ts,errors.test.ts, andindex.test.ts; registration/config, error taxonomy, HTTP mapping/retries, tokens, swaps, lifecycle, cleanup, and README usage remain covered. Required output-tail infrastructure cases are adapted inagents/src/voice/io.test.ts.uv.lock: Adapted to theplugins/synthesiaimporter inpnpm-lock.yamlusing dependency versions already locked by agents-js.Validation
pnpm test agents: 2613 passed, 5 skippedpnpm test plugins/synthesia: 89 passedpnpm build: 41/41 packages passedpnpm format:check: passedpnpm --filter @livekit/agents-plugin-synthesia lint: passedpnpm --filter @livekit/agents api:check: passedpnpm --filter @livekit/agents-plugin-synthesia api:check: passedpnpm lint: blocked by unrelatedplugins/openai/src/ws/llm.ts:127(@typescript-eslint/no-misused-promises); agents lint otherwise exits with existing warnings and Synthesia is cleancue-cli: attempted against the configured LiveKit project, but the worker was rejected with HTTP 401 before registration, so framework-event runtime validation could not runSource: livekit/agents#7216
Ported from livekit/agents#7216
Original PR description
Summary
Adds
livekit-plugins-synthesia, a plugin that attaches a Synthesia interactive avatar to a LiveKit voice agent:synthesia.AvatarSessionextendslivekit.agents.voice.avatar.AvatarSession. Callstart()beforeAgentSession.start()to dispatch the hosted avatar worker into the room and wire the agent's speech to it over a data stream.swap_avatar()switches between up to five precomputed avatars mid-session.SynthesiaError(anAPIError) with atype: ErrorTypefield identifying the failure (auth, unknown avatar, quota, rate limit, concurrency limit, timeout, connection, etc.), plusretryable,retry_after,status, andrequest_id.What's included
livekit-plugins/livekit-plugins-synthesia/— the plugin packageuvworkspace member (pyproject.toml) and as alivekit-agents[synthesia]extra (livekit-agents/pyproject.toml), withuv.lockregenerated accordinglytests/test_plugin_synthesia.py— a single combined unit-test module (plugin registration, config validation, error taxonomy, the HTTP client, the avatar session lifecycle, and a README-mirroring usage example), taggedpytest.mark.unitandpytest.mark.plugin("synthesia")per the existing test-category conventionTest plan
ruff check/ruff format --checkcleanmypy(uv run mypy -p livekit.plugins.synthesia) clean under the repo's strict configpytest tests/test_plugin_synthesia.py— 165 passedpytest --list-categoriesthat the new test module is picked up under bothunitandplugincategories, so it runs in the standardmake unit-testsCI gateuv sync --all-extras --devsucceeds with the new package in the workspace