Repository navigation
Conversation
GenerateSpeechAsync used the SDK defaults: 100s per attempt and 3 retries.
In production some /v1/audio/speech requests never return, so a failed call
blocked the caller for about 7 minutes ("Retry failed after 4 tries").
AudioSynthesisProvider now waits at most 30s per attempt and keeps 3 retries.
Both can be set per model under Audio.Synthesis (TimeoutSeconds, MaxRetries).
ProviderHelper.GetClient takes the timeout and retry count as optional
arguments; other callers keep the SDK defaults.
Verified against a local server that never answers: 4 attempts about 30s
apart, failing after 125.7s (was ~400s); a hang followed by a good response
recovers on the retry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Say mode (Twilio:SayGeneratedSpeech, off by default)
- Generated speech is saved as a text file and spoken with <Say voice=SayVoice>
instead of pre-generating an mp3 with the audio synthesizer: the opening line
(no longer synthesized before dialing), replies, hang-up, transfer, voicemail
and tool-call indications.
- Prerecorded prompts (twilio/{name}.mp3) are spoken from Twilio:PromptTexts
so the whole call uses one voice; prompts without text keep playing.
- TwilioService.SaveSpeechAsync / AppendSpeech are the single places that
write and play speech. With the switch off, behaviour is unchanged.
ConversationRelay (Twilio:ConversationRelay, off by default)
- Outbound calls for enabled agents (AgentIds, empty = all) are answered with
<Connect><ConversationRelay>: Twilio transcribes and speaks, and exchanges
text with TwilioConversationRelayMiddleware at /twilio/relay/{agent}/{conv}.
- ConversationRelaySession runs one agent turn at a time (later words are
merged into the next turn), speaks tool-call indications, re-prompts on
caller silence, hangs up after MaxIdlePrompts or AiResponseTimeoutSeconds,
and ends with handoffData hangup/transfer.
- twilio/relay/action dials the CSR on transfer, hangs up otherwise, and
reconnects a failed session up to MaxReconnects times.
- The WebSocket handshake is checked against X-Twilio-Signature.
- The caller-message handling used by the webhook queue moved to
TwilioConversationProcessor so both flows share it.
Tested with in-process harnesses: 24 TwiML checks for Say mode and 28 checks
driving the relay WebSocket (turns, merging, DTMF, indications, silence,
timeouts, signature, TwiML, action callback).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… them With Twilio:ConversationRelay:StreamReplies on (off by default), a relay turn asks BotSharp to stream (use_stream_message, and language English when unset) and forwards each OnReceiveLlmStreamMessage piece to Twilio as a text token with last=false, so Twilio starts speaking with the first words instead of after the whole reply. - Each model response is closed with last=true when the next one starts, before a tool-call indication is spoken, and at the end of the turn. - The final reply is not spoken again; only the part not streamed is (text set by a tool, or a reply rewritten after the model wrote it). - When the caller cuts in, the rest of the streamed reply is dropped; when the turn times out, nothing streamed afterwards reaches the call. - TwilioConversationProcessor takes AgentTurnCallbacks (indication, stream start, stream text); the webhook queue passes only the indication, so that flow does not stream. Needs a chat provider that streams (OpenAI, Azure OpenAI, Anthropic, Google); non-English conversations get the whole reply as before. Tested with in-process harnesses: relay harness 42/42 (13 new: pieces and close, first words before the model finishes, tool-call boundary, remainder, rewritten reply, tool-set reply, interrupt, timeout, and the processor's state and event wiring against stubbed BotSharp services); Say harness passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Stop dequeuing caller input once the call is gone: when the relay socket closes, a turn already running finishes (so the conversation is saved) but words queued during it no longer start another agent turn. - Keep each keypad entry's Digits: inputs are queued with their digits, and a keypad entry ends the merge of words said while the agent was busy, so two entries during one turn become two turns instead of losing the first. - Decide and end the silence hang-up atomically, so caller input arriving at that moment cannot start a turn on a call being hung up; skip the "say it again" prompt when the caller spoke since it was decided. - Drop the unused conversation lookup on every caller message (moved from the webhook queue, never read). - Escape agent and conversation ids in the relay action query and as single path segments of the WebSocket URL; the signature check rebuilds the same escaped URL. - Move ConversationRelayHandoff to its own file and wrap lines over 125 characters. Tested with the relay harness, 49/49: seven new checks for these fixes, six of which fail on the previous code (the hang-up race itself cannot be forced, its check guards the outcome). Say harness passes. Downstream app: build clean, unit and integration suites at their known baselines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inbound calls for agents listed in ConversationRelay.AgentIds now connect to ConversationRelay (streamed replies) instead of the Gather flow. The welcome prompts queued by session hooks become the relay greeting through their Twilio:PromptTexts text. Calls that arrive with an intent keep the Gather flow, which already has their first message queued. ConversationRelay.ResponseWatchdogSeconds (off by default) says one short filler from ResponseWatchdogPrompts (default hold-on-1..5) when the agent has said nothing that long after the caller finished, at most once per turn. ResponseWatchdogAgentIds limits it to some agents. The filler is decided while holding the send lock, so it never lands inside a streamed reply, and a turn whose reply is already in never gets one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the same twice ResponseWatchdogSeconds becomes ResponseWatchdogIntervals: seconds of silence before each filler while the agent works on a reply. The first counts from when the caller finished, each next one from when whatever was said last (a filler, an indication, a streamed reply) ends; the last value repeats until the reply comes. The first filler of a turn comes from ResponseWatchdogPrompts (default hold-on-1..5), later ones from ResponseWatchdogLongPrompts (default hold-on-long-1..10), and no sentence is repeated in a turn while unused ones remain. Still nothing while a reply is streaming, and nothing once the turn's reply is in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Code Review by QodoSorry, something went wrongWe weren't able to complete the code review on our side. Please try again manually by commenting/agentic_review on this PR.
|
Contributor
PR Summary by QodoAdd Twilio ConversationRelay calls and a response watchdog
AI Description
Diagram
High-Level Assessment
Files changed (22)
|
…tool call
A line the model says before a slow tool ("Okay, that's everything I need")
stayed an open utterance until the next response started streaming, so the
response watchdog never filled the silence while the tool ran. The relay now
closes it as soon as the response turns out to be a tool call
(AgentTurnCallbacks.OnStreamEndedWithToolCall), and the watchdog counts from
the end of that line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reply cut off by max_output_tokens is replaced by a fixed notice, so the log is the only record of what the model wrote. The streaming path logged nothing, and the non-streaming paths left out the tool calls, which is where a cut-off usually happens. All three now log the text and the tool calls with their arguments as far as the model got. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The inbound ConversationRelay greeting could only come from the prompt texts behind the speech paths the session hooks queue, which are fixed. A hook that writes ConversationalVoiceResponse.Text (e.g. a greeting with the caller's name) now has that said instead; without it the greeting is built as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A streamed piece, an idle "say it again" prompt or a spoken line was checked first and sent after waiting for the send lock, so the decision could go stale meanwhile: - a streamed piece queued behind a filler could go out after the caller cut in or the session ended, leaving one fragment of the reply; - an idle prompt could be sent after caller input had started a turn, landing before or over that turn's reply. SendIfAsync now takes the send lock first and evaluates the condition under _sync right before sending; streaming bookkeeping changes only when the piece is actually sent. Every nonempty partial prompt (partialPrompts) now counts as caller activity, so a caller mid-sentence is never asked to repeat or treated as silent. Tests (onebrain host): a reply queued behind a filler goes out whole, a cut-in on a streamed reply stops the rest, and a caller sending partial prompts past the idle timeout is not prompted. The first and last fail on the previous code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
An inbound phone agent that creates work orders needs the same streamed ConversationRelay experience #1459 gives outbound
calls, and callers should not sit in silence while the model thinks or a slow tool (a ~13s create-work-order API) runs.
What
Inbound calls on ConversationRelay (
TwilioInboundController)ConversationRelay.AgentIdsnow get ConversationRelay TwiML ontwilio/inbound, instead of the Gather flow.SpeechPathsbecome the welcome greeting throughtheir
Twilio:PromptTextstext (TwilioService.GetPromptText).Intentkeep the Gather flow, which already has their first message queued.Enabled: trueand an emptyAgentIds, inbound calls of every agent switch to ConversationRelay.Response watchdog (
ConversationRelaySession, off by default)ResponseWatchdogIntervals, e.g.[1.5, 3, 5]: seconds of silence before each filler while the agent works on a reply.The first counts from when the caller finished, each next one from when whatever was said last (a filler, a tool
indication, a streamed reply) ends; the last value repeats until the reply comes.
ResponseWatchdogPrompts(defaulthold-on-1..5), later ones fromResponseWatchdogLongPrompts(defaulthold-on-long-1..10); no sentence repeats in a turn while unused ones remain.ResponseWatchdogAgentIdslimits it to some agents.the turn's reply is in.
Testing
ConversationRelaySessionover a fake WebSocket): slow reply gets one filler,fast reply none, unlisted agent none, an indication keeps it quiet, a filler never splits a streamed reply, and a long
wait gets a different filler at each interval. Repeated runs are stable.
twilio/inboundplus the relay WebSocket in text against a dev environment: greeting fromthe prompt text, fillers at the configured intervals with different sentences, and the expected handoff on transfer.
🤖 Generated with Claude Code