Conversation
GPT-Live speaks only while its input clock runs, and that clock is the audio the client appends. A session with no microphone (a text simulation, a text-mode console, muted input) never appended anything, so every ask timed out with "the model did not start speaking when asked". GPTLiveSession now appends 100 ms of silence whenever nothing has been pushed for 200 ms, once the session has started. A live microphone is never padded.
The guard from #2481 treated the missing audio as fatal. The GPT-Live plugin now supplies the input clock itself, so a text simulation runs a duplex model like any other.
🦋 Changeset detectedLatest commit: a1bbe66 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| private readonly silenceTimer = setInterval(() => { | ||
| if (!this.sessionStarted || this.closing) return; | ||
| if (Date.now() - this.lastAudioAt < INPUT_IDLE_MS) return; | ||
| this.appendAudio(SILENCE_100MS); |
There was a problem hiding this comment.
🟡 Idle padding resets microphone resampling
After a non-24 kHz microphone pauses, appendAudio treats padding as a source-rate change. It closes the microphone resampler, so resumed speech can lose buffered samples or start with audio artifacts.
Learn more
The padding frame uses the provider's 24 kHz wire format, while real microphone frames can use any sample rate. Passing padding through appendAudio changes inputRate, closes inputResampler, and replaces its state. The next real frame changes the rate back and creates another resampler. Audio resamplers retain filter and partial-sample state across frames, so these synthetic rate changes can discard input and create discontinuities.
Example: A 48 kHz microphone stops for 300 ms between words. The first padding tick closes its 48-to-24 kHz resampler. The next word creates a fresh resampler instead of continuing the microphone stream, so its leading samples can be clipped or distorted.
Recommended fix: Send generated 24 kHz silence directly to the output byte stream or wire queue without changing inputRate or inputResampler. Keep microphone resampling state exclusively owned by real pushAudio frames, and add a pause/resume test using 48 kHz input.
Was this helpful? React with 👍 or 👎 to provide feedback.
| // a duplex model has no text modality, and the adapter resolves each reply from the audio | ||
| // the model produces; without audio a text simulation would only time out on turn one | ||
| if (this.agentSession._textOnly && this.llm instanceof DuplexRealtimeAdapter) { | ||
| throw new Error( | ||
| 'a DuplexModel speaks only through audio, so it cannot run under a text simulation; run `lk agent simulate audio` instead', | ||
| ); | ||
| } |
There was a problem hiding this comment.
🟡 Unsupported duplex text sessions time out
A DuplexModel without idle-input support now starts in text simulations. With no audio channel advancing it, generateReply times out instead of rejecting startup.
Learn more
Text simulations disable audio input, while DuplexModel is a public base class with no capability declaring no-audio operation. GPT-Live now compensates internally, but the removed check covered every duplex implementation. A custom or future duplex provider without its own clock padding therefore enters a mode it cannot service, and generateReply waits ten seconds for output before failing.
Example: An application supplies a custom audio-only DuplexModel and runs lk agent simulate text. Startup now succeeds, but the provider receives no input audio and each typed turn fails after the reply timeout. Previously startup rejected immediately with a useful mode error.
Recommended fix: Add an explicit duplex capability for text/no-input-audio operation. Retain the startup rejection unless that capability is set, and enable it for GPTLiveModel.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Problem
GPT-Live speaks only while its input clock runs, and that clock is the audio the client appends. A session with no microphone never appends anything, so every ask times out after 10 seconds with "the model did not start speaking when asked". That is what happens under a text simulation, in
lk agent console --text, and whenever input audio is disabled. #2481 refused to start in the first case; the other two still hang.Measured against the live service through the Python plugin: with nothing pushed the ask times out; with 100 ms silence frames pushed at 10 Hz the reply opens in 1.6 s with transcript and audio.
session.input_audio.mutealone does not do it.Change
GPTLiveSessionappends 100 ms of silence whenever nothing has been pushed for 200 ms, once the session has started. A live microphone is never padded: the timer only fills gaps.pushAudiosplit into the timestamp and a sharedappendAudio.Tests
gpt_live_model.test.ts: an idle session appends silence; a live microphone is not padded. The test server'sevents()ignores silent appends by default so the existing sequence assertions stay exact; the two tests that assert on silent input opt back in. The #2481 test is removed with the guard. 148 tests pass across the GPT-Live and duplex adapter files. The idle test fails when the timer is disabled.What text mode still cannot do well
GPT-Live has no typed user-input channel; the plugin delivers typed turns as
session.commentary.appendwith an instruction to reply. In the Python end-to-end run the voice model delegated to the backend, and so reached the tools, in one of three text-mode scenarios versus three of three in audio mode. Text mode is now usable for exercising conversation flow, not for asserting tool behavior. That limit is the protocol's, not this change's.Python counterpart: livekit/agents#7239