Skip to content

feat: expose the assistant audio element for playback volume control - #169

Merged
veltson-vapi merged 3 commits into
mainfrom
feat/expose-audio-player
Aug 14, 2026
Merged

feat: expose the assistant audio element for playback volume control#169
veltson-vapi merged 3 commits into
mainfrom
feat/expose-audio-player

Conversation

@veltson-vapi

@veltson-vapi veltson-vapi commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

feat: expose the assistant audio element for playback volume control

Problem

The SDK creates an <audio> element per remote track in buildAudioPlayer
and discards the reference, so there is no supported way to reach it. There is
no way to set playback volume at all: volume-level only reports measured
loudness, and setMuted affects the user's microphone.

Consumers work around this by querying the DOM for
audio[data-participant-id], which relies on an internal detail and can match
an orphaned element from a previous call (#161). Requested in #37.

What this adds

  • setVolume(volume) for assistant playback volume, 0 to 1. The value is
    remembered, so it can be set before the track arrives and it survives across
    calls and mid-call track replacement. Out-of-range values are clamped and
    non-finite values ignored, since the DOM rejects both rather than saturating.
  • getAudioPlayer() returning the element, or null before its track arrives.
  • An audio event carrying the element the moment it exists, mirroring how
    video already emits one line above in the same handler.
  • An error of type audio-start-failed when the browser blocks playback,
    which is usually autoplay policy and is recoverable by prompting for a
    gesture. Previously this surfaced only as an unhandled rejection.

Pairing a getter with an event matches getLocalAudioLevel alongside
local-volume-level.

Lifecycle correctness

buildAudioPlayer awaits play(), which pends for the whole of media startup,
so several players can be in flight, they can finish out of order, and the call
or the participant can go away underneath them. Each build now takes a sequence
number and records its participant while pending, so that:

  • a player built for a call that has since ended or been replaced is discarded
  • a teardown invalidates only the departing participant's builds, so an
    unrelated participant leaving cannot cost us the assistant's audio
  • a build that started before the one currently attached cannot overwrite it
  • a superseded element is removed when its replacement is adopted, which
    matters because renegotiation reuses the participant id and both elements
    would otherwise match the selector teardown uses
  • the reference is dropped in stop(), cleanup() and on participant-left,
    so setVolume can never write to a torn-down player

Consumer emits are routed through emitToConsumer, because a listener that
threw previously propagated out of the track-started handler and skipped
sendAppMessage('playable'), stalling the call. This applied to the existing
video emit too.

handleTrackStarted is extracted so the initial and reconnect paths share it
rather than being two byte-identical copies.

For existing apps

Nothing breaks at runtime and no existing API changed. For callers who never
touch setVolume or getAudioPlayer, the observable differences are three
fixes and one new event.

Fixed: a throwing video listener no longer stalls the call.
emit('video', track) sat one statement before sendAppMessage('playable'),
so a listener that threw skipped the playable signal and left the call sitting
there. Both emits are now guarded, so the throw is logged and the call
continues. Reaches tavus calls and video recording.

Fixed: a blocked play() no longer surfaces only as an unhandled rejection.
It used to reject out of the un-awaited track-started handler. It is now
caught and reported. If you were catching this via a global unhandled-rejection
reporter, listen for the event below instead.

Fixed: fewer stray <audio> elements. Three cases that previously stranded
one now remove it: a superseded element on renegotiation, a player built for a
call that ended mid-build, and a player whose play() failed. Relevant only to
code that queries the DOM for audio[data-participant-id], where the first
match could previously be a dead element. That code should move to
getAudioPlayer(). This does not fix the main leak, one element per ended
call (#161).

New: error can now carry { type: 'audio-start-failed', error } when the
browser blocks playback, usually autoplay policy. Existing error listeners
will see this type for the first time, so apps that surface every error
verbatim to users may want to filter it.

No dependency, lockfile, tsconfig, packaging or CI changes.

Not addressed

The orphaned elements themselves (#161). This ensures the new API never hands
back an orphan, but does not stop them accumulating. Relatedly, playable is
still not sent when playback genuinely fails, which is unchanged behavior and a
question about what that signal means server-side.

Testing

46 tests, tsc, npm run build and npm run test:example all pass. No
dependency, lockfile, tsconfig, packaging or CI changes.

The two functions that touch the DOM are reached through thin private wrappers
so the lifecycle logic around them is covered in the existing node test
environment without adding a DOM dependency. Every fix above was mutation
tested: reverting it fails the suite. Still uncovered, as on main: the DOM
calls themselves.

Also verified by hand on two live calls, confirming a volume set before the
call is applied to the real element and persists into the next call.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

veltsonbastien and others added 2 commits August 11, 2026 15:04
## Problem

The SDK creates an `<audio>` element per remote track in `buildAudioPlayer`
and discards the reference, so there is no supported way to reach it. There is
no way to set playback volume at all: `volume-level` only reports measured
loudness, and `setMuted` affects the user's microphone.

Consumers work around this by querying the DOM for
`audio[data-participant-id]`, which relies on an internal detail and can match
an orphaned element from a previous call (#161). Requested in #37.

## What this adds

- `setVolume(volume)` for assistant playback volume, 0 to 1. The value is
  remembered, so it can be set before the track arrives and it survives across
  calls and mid-call track replacement. Out-of-range values are clamped and
  non-finite values ignored, since the DOM rejects both rather than saturating.
- `getAudioPlayer()` returning the element, or null before its track arrives.
- An `audio` event carrying the element the moment it exists, mirroring how
  `video` already emits one line above in the same handler.
- An `error` of type `audio-start-failed` when the browser blocks playback,
  which is usually autoplay policy and is recoverable by prompting for a
  gesture. Previously this surfaced only as an unhandled rejection.

Pairing a getter with an event matches `getLocalAudioLevel` alongside
`local-volume-level`.

## Lifecycle correctness

`buildAudioPlayer` awaits `play()`, which pends for the whole of media startup,
so several players can be in flight, they can finish out of order, and the call
or the participant can go away underneath them. Each build now takes a sequence
number and records its participant while pending, so that:

- a player built for a call that has since ended or been replaced is discarded
- a teardown invalidates only the departing participant's builds, so an
  unrelated participant leaving cannot cost us the assistant's audio
- a build that started before the one currently attached cannot overwrite it
- a superseded element is removed when its replacement is adopted, which
  matters because renegotiation reuses the participant id and both elements
  would otherwise match the selector teardown uses
- the reference is dropped in `stop()`, `cleanup()` and on `participant-left`,
  so `setVolume` can never write to a torn-down player

Consumer emits are routed through `emitToConsumer`, because a listener that
threw previously propagated out of the `track-started` handler and skipped
`sendAppMessage('playable')`, stalling the call. This applied to the existing
`video` emit too.

`handleTrackStarted` is extracted so the initial and reconnect paths share it
rather than being two byte-identical copies.

## Not addressed

The orphaned elements themselves (#161). This ensures the new API never hands
back an orphan, but does not stop them accumulating. Relatedly, `playable` is
still not sent when playback genuinely fails, which is unchanged behavior and a
question about what that signal means server-side.

## Testing

46 tests, `tsc`, `npm run build` and `npm run test:example` all pass. No
dependency, lockfile, tsconfig, packaging or CI changes.

The two functions that touch the DOM are reached through thin private wrappers
so the lifecycle logic around them is covered in the existing node test
environment without adding a DOM dependency. Every fix above was mutation
tested: reverting it fails the suite. Still uncovered, as on main: the DOM
calls themselves.

Also verified by hand on two live calls, confirming a volume set before the
call is applied to the real element and persists into the next call.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The change raised comment density in vapi.ts from 8% to 12% and used
multi-line paragraph blocks where the file's own style is short single lines.

Removed comments that restate the code (teardown nulling fields, the guard
condition in handleTrackStarted, method docs that repeat the method name) and
comments duplicated between production and tests. Compressed the rest, keeping
only what a reader cannot get from the code: why builds are tracked
individually, why the emit is guarded and not routed to the error event, why
volume is set before play(), why the DOMException fixture is not a real
DOMException.

vapi.ts is now 10%, and the test file 6%, against main's 8%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@VapiAI VapiAI deleted a comment from bajajcodes Aug 14, 2026

@shubham-vapiai shubham-vapiai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@veltson-vapi

  • tested on my local it works
  • can you add an section in PR description for existing apps, listing behavior that changes even for callers who never touch setVolume/getAudioPlayer

@veltson-vapi

veltson-vapi commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Thank you for testing, and will add

@veltson-vapi
veltson-vapi merged commit a43e421 into main Aug 14, 2026
4 checks passed
@veltson-vapi
veltson-vapi deleted the feat/expose-audio-player branch August 14, 2026 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants