feat: expose the assistant audio element for playback volume control - #169
Merged
Conversation
## Problem The SDK creates an `<audio>` element per remote track in `buildAudioPlayer` and discards the reference, so there is no supported way to reach it. There is no way to set playback volume at all: `volume-level` only reports measured loudness, and `setMuted` affects the user's microphone. Consumers work around this by querying the DOM for `audio[data-participant-id]`, which relies on an internal detail and can match an orphaned element from a previous call (#161). Requested in #37. ## What this adds - `setVolume(volume)` for assistant playback volume, 0 to 1. The value is remembered, so it can be set before the track arrives and it survives across calls and mid-call track replacement. Out-of-range values are clamped and non-finite values ignored, since the DOM rejects both rather than saturating. - `getAudioPlayer()` returning the element, or null before its track arrives. - An `audio` event carrying the element the moment it exists, mirroring how `video` already emits one line above in the same handler. - An `error` of type `audio-start-failed` when the browser blocks playback, which is usually autoplay policy and is recoverable by prompting for a gesture. Previously this surfaced only as an unhandled rejection. Pairing a getter with an event matches `getLocalAudioLevel` alongside `local-volume-level`. ## Lifecycle correctness `buildAudioPlayer` awaits `play()`, which pends for the whole of media startup, so several players can be in flight, they can finish out of order, and the call or the participant can go away underneath them. Each build now takes a sequence number and records its participant while pending, so that: - a player built for a call that has since ended or been replaced is discarded - a teardown invalidates only the departing participant's builds, so an unrelated participant leaving cannot cost us the assistant's audio - a build that started before the one currently attached cannot overwrite it - a superseded element is removed when its replacement is adopted, which matters because renegotiation reuses the participant id and both elements would otherwise match the selector teardown uses - the reference is dropped in `stop()`, `cleanup()` and on `participant-left`, so `setVolume` can never write to a torn-down player Consumer emits are routed through `emitToConsumer`, because a listener that threw previously propagated out of the `track-started` handler and skipped `sendAppMessage('playable')`, stalling the call. This applied to the existing `video` emit too. `handleTrackStarted` is extracted so the initial and reconnect paths share it rather than being two byte-identical copies. ## Not addressed The orphaned elements themselves (#161). This ensures the new API never hands back an orphan, but does not stop them accumulating. Relatedly, `playable` is still not sent when playback genuinely fails, which is unchanged behavior and a question about what that signal means server-side. ## Testing 46 tests, `tsc`, `npm run build` and `npm run test:example` all pass. No dependency, lockfile, tsconfig, packaging or CI changes. The two functions that touch the DOM are reached through thin private wrappers so the lifecycle logic around them is covered in the existing node test environment without adding a DOM dependency. Every fix above was mutation tested: reverting it fails the suite. Still uncovered, as on main: the DOM calls themselves. Also verified by hand on two live calls, confirming a volume set before the call is applied to the real element and persists into the next call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The change raised comment density in vapi.ts from 8% to 12% and used multi-line paragraph blocks where the file's own style is short single lines. Removed comments that restate the code (teardown nulling fields, the guard condition in handleTrackStarted, method docs that repeat the method name) and comments duplicated between production and tests. Compressed the rest, keeping only what a reader cannot get from the code: why builds are tracked individually, why the emit is guarded and not routed to the error event, why volume is set before play(), why the DOMException fixture is not a real DOMException. vapi.ts is now 10%, and the test file 6%, against main's 8%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Thank you for testing, and will add |
shubham-vapiai
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat: expose the assistant audio element for playback volume control
Problem
The SDK creates an
<audio>element per remote track inbuildAudioPlayerand discards the reference, so there is no supported way to reach it. There is
no way to set playback volume at all:
volume-levelonly reports measuredloudness, and
setMutedaffects the user's microphone.Consumers work around this by querying the DOM for
audio[data-participant-id], which relies on an internal detail and can matchan orphaned element from a previous call (#161). Requested in #37.
What this adds
setVolume(volume)for assistant playback volume, 0 to 1. The value isremembered, so it can be set before the track arrives and it survives across
calls and mid-call track replacement. Out-of-range values are clamped and
non-finite values ignored, since the DOM rejects both rather than saturating.
getAudioPlayer()returning the element, or null before its track arrives.audioevent carrying the element the moment it exists, mirroring howvideoalready emits one line above in the same handler.errorof typeaudio-start-failedwhen the browser blocks playback,which is usually autoplay policy and is recoverable by prompting for a
gesture. Previously this surfaced only as an unhandled rejection.
Pairing a getter with an event matches
getLocalAudioLevelalongsidelocal-volume-level.Lifecycle correctness
buildAudioPlayerawaitsplay(), which pends for the whole of media startup,so several players can be in flight, they can finish out of order, and the call
or the participant can go away underneath them. Each build now takes a sequence
number and records its participant while pending, so that:
unrelated participant leaving cannot cost us the assistant's audio
matters because renegotiation reuses the participant id and both elements
would otherwise match the selector teardown uses
stop(),cleanup()and onparticipant-left,so
setVolumecan never write to a torn-down playerConsumer emits are routed through
emitToConsumer, because a listener thatthrew previously propagated out of the
track-startedhandler and skippedsendAppMessage('playable'), stalling the call. This applied to the existingvideoemit too.handleTrackStartedis extracted so the initial and reconnect paths share itrather than being two byte-identical copies.
For existing apps
Nothing breaks at runtime and no existing API changed. For callers who never
touch
setVolumeorgetAudioPlayer, the observable differences are threefixes and one new event.
Fixed: a throwing
videolistener no longer stalls the call.emit('video', track)sat one statement beforesendAppMessage('playable'),so a listener that threw skipped the playable signal and left the call sitting
there. Both emits are now guarded, so the throw is logged and the call
continues. Reaches
tavuscalls and video recording.Fixed: a blocked
play()no longer surfaces only as an unhandled rejection.It used to reject out of the un-awaited
track-startedhandler. It is nowcaught and reported. If you were catching this via a global unhandled-rejection
reporter, listen for the event below instead.
Fixed: fewer stray
<audio>elements. Three cases that previously strandedone now remove it: a superseded element on renegotiation, a player built for a
call that ended mid-build, and a player whose
play()failed. Relevant only tocode that queries the DOM for
audio[data-participant-id], where the firstmatch could previously be a dead element. That code should move to
getAudioPlayer(). This does not fix the main leak, one element per endedcall (#161).
New:
errorcan now carry{ type: 'audio-start-failed', error }when thebrowser blocks playback, usually autoplay policy. Existing
errorlistenerswill see this type for the first time, so apps that surface every
errorverbatim to users may want to filter it.
No dependency, lockfile, tsconfig, packaging or CI changes.
Not addressed
The orphaned elements themselves (#161). This ensures the new API never hands
back an orphan, but does not stop them accumulating. Relatedly,
playableisstill not sent when playback genuinely fails, which is unchanged behavior and a
question about what that signal means server-side.
Testing
46 tests,
tsc,npm run buildandnpm run test:exampleall pass. Nodependency, lockfile, tsconfig, packaging or CI changes.
The two functions that touch the DOM are reached through thin private wrappers
so the lifecycle logic around them is covered in the existing node test
environment without adding a DOM dependency. Every fix above was mutation
tested: reverting it fails the suite. Still uncovered, as on main: the DOM
calls themselves.
Also verified by hand on two live calls, confirming a volume set before the
call is applied to the real element and persists into the next call.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com