new user simulator - #214
Draft
tara-bogavelli wants to merge 67 commits into
Draft
Conversation
eva.backend and eva.role were added as scaffolding for a symmetric provider abstraction that was rejected in favor of a tick-based cascade design. Nothing imports them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…o, read-only tick Addresses code review on 6c545af: consume caller audio only after a successful adapter call, delegate padding to split_tick_audio instead of duplicating it, document the single-caller precondition and enqueue concatenation semantics, and make tick read-only. Adds tests for simultaneous speech, tick numbering, utterance contiguity, and adapter-failure safety.
Passing state=None on every audioop.ratecv call caused per-frame filter warm-up loss (2 bytes/call, ~0.3% drift over a conversation) and boundary discontinuities. Threading state through per direction eliminates both; drops the byte-count clamp/pad that masked the drift instead of fixing it.
- Replace the sleep(0)-based drain (which drains 0 frames against a real websockets.recv(), which always suspends at least once) with a background receive task feeding an adapter-owned buffer, per review. - Surface receive-loop errors from run_tick instead of degrading to silent silence forever; log the exception. - Scope stop()'s CancelledError suppression to the task's own cancellation so an outer cancel of stop() itself still propagates. - Emit user_speech_start/stop events matching audio_bridge.py's payload shape, needed for user-turn timestamps and latency metrics. - Use deadline-based pacing for outbound frames instead of a fixed post-frame sleep, and drop the sleep after the last frame. - Guard the empty-audio case in _send_tick_audio explicitly instead of an unreachable fallback that raised ValueError at len(mulaw) == 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Renders caller text to PCM16 via Cartesia Sonic HTTP API, with stable per-persona voice selection for later phrase-cache reuse. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds user_simulator.cascade_turn_contract, the JSON output contract the cascade STT->LLM->TTS pipeline layers on top of the existing persona/goal prompt assembled by _build_prompt(). Hanging up remains a tool call (end_call), not a JSON field, per the openai_realtime pattern. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the CascadeSimulatorConfig dispatch branch so EVA_USER_SIMULATOR__PROVIDER=cascade selects CascadeUserSimulator end to end. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A silent tick (outgoing_audio=None) returned immediately with no I/O, so the tick loop spun at CPU speed instead of pacing at 200ms/tick, racing thousands of ticks ahead of the assistant within ~400ms and producing a bogus timeout. run_tick now sleeps out the remainder of TICK_DURATION_MS after all other work, so a speaking tick (which already spends ~200ms in per-frame outbound pacing) does not double-sleep. Bump simulation_version: this is a user-simulator behavior change.
A silent tick (outgoing_audio=None) sent nothing at all instead of silence, leaving gaps in the outbound stream. The assistant's STT/VAD expects a continuous stream like a real phone line, so gaps caused turn detection to misfire and transcripts to merge into single malformed entries. run_tick now always calls _send_tick_audio with either the real audio or a synthesized tick of SILENCE_BYTE, framed and paced identically to speech. Bump simulation_version: this is a user-simulator behavior change.
Flip agent/caller roles when building the caller LLM's message list — it is itself the "assistant" in its own frame, so an untouched history read its own prior turns as its own output and echoed them back. Also feed Scribe on every tick instead of only while the assistant speaks, since a continuous PCM stream is what keeps the session from timing out; feed() now raises once the socket closes instead of repeating a swallowed warning every 200ms.
Live runs showed the caller re-sending its turn nearly verbatim when the assistant's LLM+TTS round trip exceeded WAIT_TO_RESPOND_SELF_MS: both silence thresholds are satisfied a fixed time after the caller stops talking regardless of whether a reply ever arrived. Add a strict has-the-assistant-replied gate, set when the caller starts a turn and cleared only when the assistant produces speech, combined with (not replacing) the existing thresholds. Bump simulation_version to 2.0.12.
Scribe closes cleanly (code 1000) around 397s of continuous audio, which would otherwise truncate long conversations. Detect ConnectionClosedOK in the receive loop and transparently reconnect, preserving committed transcript and dropping only the in-flight partial. Caps consecutive clean-close reconnects to distinguish this from a genuine hard failure (auth, quota, repeated immediate closes), which still fails loudly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Default CascadeSimulatorConfig.llm to "user-llm" instead of "gpt-5.5" so the caller's model/params can differ from the assistant under test without touching the shared gpt-5.5 deployment (preserves comparability with the ElevenLabs baseline). Drop llm_params, which was never wired to anything.
…e repeats The cascade simulator regenerated its turn from unchanged history whenever a transcript failed to arrive, producing verbatim repeats the assistant noticed and complained about. Reading the buffer was optimistic: no wait, no acknowledgement that the commit landed, no fallback, no check that anything was heard at all. _collect_heard_text now waits a bounded number of ticks for the final, falls back to the in-flight partial, and — if nothing arrived — tells the model it missed the utterance so it asks for a repeat instead of re-prompting on identical history. Both outcomes are logged as events so transcript loss can be measured per provider. STT moves to the LiveKit ElevenLabs plugin used standalone (no room). Omitting server_vad selects commit_strategy=manual, so the tick scheduler still owns the turn boundary, and flush() replaces our hand-rolled commit flag; measured flush->final is 0.15s. This deletes the reconnect and idle-close machinery ScribeStreamingSTT needed. Also drops the cascade-specific turn-contract prompt. It demanded "a single JSON object and nothing else", which suppressed the end_call tool call entirely and left conversations hanging until the assistant server's idle timeout. The per-domain user_simulator prompt already covers end_call, and the tool description is now shared with the OpenAI Realtime provider. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The metrics processor numbers turns from audio_start(simulated_user) events (processor.py:117,433). BotToBotAudioBridge emitted them from inside its send loop; the cascade simulator replaced that bridge and never re-emitted them, so every conversation_trace entry collapsed onto turn_id 0 on S2S frameworks and latency_assistant_turns came out empty on every framework. Runs looked healthy because the artifacts all existed and the verification passed --metrics '[]', so nothing ever computed the metrics that were missing. TickScheduler gains caller_spoke_this_tick, which flips on the ticks audio actually enters and leaves the wire — distinct from caller_is_speaking, which is true from the moment an utterance is queued. The simulator emits the boundary events from that transition for the user and from has_assistant_speech for the assistant. These boundaries are authored rather than detected. The bridge inferred end-of-audio from a silence threshold and then back-dated the stamp to undo the lag (~600ms per its own comment), whereas the playout queue drains on a known tick. Latency measured against these stamps is therefore more accurate but not directly comparable to earlier ElevenLabs-driven runs. Verified live with metrics enabled: latency_assistant_turns 0 -> 8 populated turns, trace turn_ids 0..8 instead of all zero, strict role alternation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An assistant that stops replying was being treated as a transcript problem: the simulator logged a transcript_missed event, told the model it had misheard, and released the awaiting-reply gate so it could speak again. It usually repeated itself instead, producing loops of up to 14 identical utterances and minutes of dead air. Root-caused against the matrix runs: the assistant's STT drops 100% of sub-second user utterances (16/16; 0 transcribed), so a terse reply like "Aisle." never reaches its LLM and it has nothing to answer. The simulator was correctly hearing nothing. Nothing was wrong with our own transcription, and working around the assistant's STT is not this component's job. EVA already models this. conversation_valid_end treats inactivity_timeout with the user speaking last as a definitive terminal state, and ElevenLabsUserSimulator ends the call after 12 keep-alives without activity. Cascade now does the same at the same two-minute threshold, so both providers record the same end state. Removes transcript_missed, MISSED_UTTERANCE_DIRECTIVE and the ASSISTANT_UNRESPONSIVE_MS gate release. The bounded wait and partial-transcript fallback stay: those cover our own STT finalizing late, which is our concern. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_ticks_assistant_silent was never initialised, so the first silent tick raised AttributeError and failed the run. The unit tests missed it because they build bare instances with __new__ and set the counters by hand, testing around the gap; both counters are now class-level defaults and a test asserts they exist without manual setup. Also restores the wait-counter reset inside _collect_heard_text, which a scripted edit removed by mistake, leaving the counter climbing across turns. Verified live: assistant stalls, conversation ends inactivity_timeout, and conversation_valid_end scores 1.0 as agent_timeout_on_user_turn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…caller - _play_interruption discarded extract_turn's end_call flag, so a caller that decided to hang up mid-assistant-turn emitted nothing and the conversation ran on to the inactivity timeout. Observed live: six empty interruptions after the caller said goodbye, with the assistant looping. A hang-up is also never treated as stale. - Slip was computed from tick deltas, but run_tick is only pumped by _run, so no tick can advance during the generation await: 525/525 interruptions reported slip_ms=0 and the staleness check never engaged. Measured on the wall clock now. - The self-correction gate reseeded random.Random(0) every conversation, whose first draw below SELF_CORRECTION_RATE is #26 while a conversation runs ~7 turns, so no correction ever armed. Seeded per record id instead.
The interrupt decision prompt's YES criteria ('has the user heard enough to have a
response ready?') describe ordinary conversational readiness, so it answered YES at
the end of every assistant turn. Combined with the tick loop only checking the
interrupt path while the assistant speaks, every caller utterance became an
interruption and normal turn-taking was bypassed: logged barge-ins were 10-50s apart
and carried ordinary replies, each with a spurious 'Actually-'/'Hold on-' opener.
INTERRUPT_RATE is rolled once when an assistant turn starts, and the eligibility is
cleared on firing, so a turn carries at most one barge-in. Uses its own RNG stream so
enabling interruptions cannot shift the self-correction draws.
The decision only saw the transcript, so it could not tell whether the caller still had anything left to accomplish — it authorized barge-ins on turns where the caller's next move was to hang up, producing an opener with no speech behind it. It now receives the goal, must/nice-to-have criteria, option-evaluation steps, and the resolution, failure and escalation conditions, with the field meanings explained in the prompt template so the explanations stay static across calls. Placed ahead of the conversation history so the block is prompt-cacheable. edge_cases and information_required are omitted: they describe how to answer questions, not whether the goal is finished.
Three fixes, all confirmed against live conversation data. 1. inactivity_timeout measured cumulative, not contiguous, silence. The check was only reached on silent ticks because the speech branch continued first, making its reset line unreachable in production, so scattered quiet ticks accumulated over the whole call. Measured: one conversation was killed for '120s of assistant silence' after 73.7s of actual silence. Now checked on every tick. This has been ending healthy conversations since Plan 1, and inflated inactivity_timeout in every run including baseline. 2. An assistant turn boundary was any single 200ms quiet tick, so a pause between sentences re-armed the interruption cap mid-utterance and one turn could collect several barge-ins. Now uses the same 1s threshold the turn gate already applies. 3. The barge-in opener was emitted before the content existed, to hide ~1s of generation latency. A hang-up or a stale drop then left an orphaned 'Actually-' on the wire with nothing behind it. Content is now generated first, so dropping costs no audio at all.
The rate gate was added when the decision looked like it always said YES. It does not: it says YES about 14% of checks, once per assistant turn, at the point the caller is ready to reply. The defect was where it fired, not how often, so a random dice roll suppressed decisions that were correct while leaving the misjudgement in place. Giving the decision the caller's goal fixes the judgement at its source, so the gate became a second suppressor stacked on a fix. The per-turn eligibility flag stays, since that is the one-barge-in-per-assistant-turn cap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…g mode Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two faults the unit tests could not see, both found by running it: - With all pacing removed nothing waited for the provider, so the tick loop exhausted its whole budget in 160ms and every call ended at tick zero. Ticks now wait up to one tick for a full tick of assistant audio, returning the moment it is available, so nothing already generated is held back. - Emitting nothing on a silent tick starves the provider's VAD of the trailing silence that ends the caller's turn, so the assistant never replies. The freeze comes from not ticking while the caller thinks, not from sending nothing inside a tick; every tick now sends a full tick, unpaced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the session's defects (cumulative-silence inactivity timeout, 200ms turn boundaries, opener-before-content), the rate-gate add-then-remove reasoning, and the headline measurement: 35 barge-ins fired, 4 kept, 31 dropped for lateness, median slip 1902ms against a 1500ms budget. Corrects an earlier claim that the interrupt path affects metric turn numbering - it does not; nothing under src/eva/metrics reads caller_turn.
tara-bogavelli
marked this pull request as draft
August 21, 2026 00:03
worker.py always passes paced_output, but only OpenAIRealtimeAssistantServer took it (via **kwargs). The four servers with explicit __init__ signatures raised TypeError on construction, which is why running --framework pipecat failed and the argument had to be commented out at the call site. Add the parameter to pipecat, gemini_live, elevenlabs and smallest_hydra, and forward it to the base class. Also guard the silent-ignore case. Accepting paced_output=False and then pacing anyway would leave a tick-driven caller believing the assistant was unpaced, so AbstractAssistantServer now declares supports_unpaced_output and rejects a request it cannot honor. Only a server whose outbound relay throttle we own can drop it: openai_realtime sets it True (grok_voice inherits), the rest keep the False default. No current framework is affected, since each receives the value it can honor.
current_text() appends "[CURRENTLY SPEAKING, INCOMPLETE]" because the listener check prompts are tuned around that marker — their few-shot examples all end in it, and it tells the check to judge only the complete sentences. That makes it the wrong accessor for anything that records what was said. Add a marker-free heard_text() alongside it so prompt input and transcript content stop sharing one accessor.
The interrupt and backchannel checks collapsed to a bool, so a check that ran and answered NO was indistinguishable from one that never ran or one whose call raised. That gap made a run with zero interruptions unexplainable from its artifacts. Have _check return a CheckTrace carrying whether it ran, the raw reply, its latency, and any error, and hang both traces off ListenerVerdict. The verdict fields are unchanged, so callers reading should_interrupt/should_backchannel behave exactly as before.
user_simulator_events.jsonl only records actions the caller took, so a listener check that declined leaves no trace at all. A run with no interruptions looks identical to one where the interrupt path was never reachable. DecisionLog writes user_simulator_decisions.jsonl, one JSON object per row. Rows are written and flushed as they happen rather than buffered until the end: a conversation that dies mid-run is exactly the one worth having a trace for, and the event log's save-at-exit is why failed attempts currently leave no diagnostics. The file is only created on first write, so a run that traces nothing leaves none behind.
…ions Three changes to the cascade caller that cannot be separated into their own commits: they overlap in simulator.py, including the import block and the whole of _play_interruption, so splitting them at line level would leave commits that do not import. Remove self-correction entirely. Gone are the enable_self_correction config field, SELF_CORRECTION_DELAY_MS and _RATE, should_fire_self_correction, correction_rng, _maybe_arm_self_correction, _fire_self_correction, _drop_stale_correction, the armed-correction state, the tick-loop firing branch, and the cascade_self_correction prompt. --user-simulator.enable-self-correction is now a hard CLI error rather than a silently ignored flag. extract_correction survives as extract_optional_line: speculative generation uses it to parse its candidate, so deleting it would have broken that path. Its old name described only the removed feature. Replace the barge-in slip budget with a staleness test about the assistant. MAX_INTERRUPT_SLIP_MS and enforces_slip_budget are gone; should_drop_interrupt now asks whether the assistant is still speaking and still in the same turn. A line is a real interruption whenever the assistant is mid-utterance, however long generation took, and the 1500ms cap was discarding well-placed barge-ins on the real-time path — a live pipecat run had one die at slip_ms 1798 with the assistant still talking. "Still speaking" alone is not enough, though: over a couple of seconds one assistant turn can end and the next begin, landing the line as a non-sequitur against speech the caller never heard. A new _assistant_turn_index, captured at decision time, rules that out. slip_ms is still measured and traced, since it says whether speculative generation is earning its keep, but it no longer gates anything. Write the decision trace. Every tick records its speech state, including the audio RMS — has_assistant_speech is true for any non-zero bytes, silence included, so a transport that pads reads as continuous speech and this makes that measurable. Every listener check records its inputs and verdict, every speculative candidate records whether it was produced or declined and why, and every barge-in records its outcome with both turn indices. Also stop leaking the "[CURRENTLY SPEAKING, INCOMPLETE]" prompt marker into conversation history: _play_interruption now banks heard_text() instead of current_text(). The marker is prompt scaffolding for the checks, not something the assistant said.
gate interruption calls on STT update to reduce usage save stalls as failed records instead of exceptions make phrase cache globally shared, reduce generation cost
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.