new user simulator

.env.example

@@ -196,6 +196,15 @@ EVA_MODEL_LIST='[
       "aws_secret_access_key": "os.environ/AWS_SECRET_ACCESS_KEY",
       "max_parallel_requests": 5
     }
+  },
+  {
+    "model_name": "user-llm",
+    "litellm_params": {
+      "model": "openai/gpt-5.5",
+      "api_key": "os.environ/OPENAI_API_KEY",
+      "max_parallel_requests": 5,
+      "reasoning_effort": "low"
+    }
   }
 ]'
 

README.md

@@ -343,7 +343,7 @@ output/<run_id>/
 | **🎯 EVA-A · Accuracy** | **✨ EVA-X · Experience** |
 |---|---|
 | *Did the agent complete the task correctly?* | *Was the conversational experience high quality?* |
-|  **Task Completion** · Deterministic |  **Turn Taking** · LLM Judge `BETA` |
+|  **Task Completion** · Deterministic |  **Turn Taking** · Deterministic |
 |  **Agent Speech Fidelity** · Audio LLM Judge `BETA` |  **Conciseness** · LLM Judge |
 |  **Faithfulness** · LLM Judge |  **Conversation Progression** · LLM Judge |
 

configs/caller_phrases.yaml

@@ -0,0 +1,69 @@
+de:
+  backchannels:
+  - mhm
+  - aha
+  - ja
+  barge_in_openers:
+  - Moment—
+  - Entschuldigung—
+  - Warte—
+  - Also—
+en:
+  backchannels:
+  - uh-huh
+  - mm-hmm
+  barge_in_openers:
+  - Wait—
+  - Sorry—
+  - Hold on—
+  - Actually—
+es:
+  backchannels:
+  - ajá
+  - mmm
+  - ya
+  barge_in_openers:
+  - Espera
+  - Perdona
+  - Un momento
+  - Bueno
+fr:
+  backchannels:
+  - hum hum
+  - oui
+  - d’accord
+  barge_in_openers:
+  - Attendez
+  - Pardon
+  - Juste
+  - En fait
+fr-CA:
+  backchannels:
+  - hum hum
+  - ouais
+  - OK
+  barge_in_openers:
+  - Attends
+  - Excuse
+  - Juste
+  - En fait
+hi:
+  backchannels:
+  - हूँ
+  - अच्छा
+  - हाँ
+  barge_in_openers:
+  - रुकिए
+  - सुनिए
+  - माफ़ कीजिए
+  - असल में
+ko:
+  backchannels:
+  - 네
+  - 음
+  - 아
+  barge_in_openers:
+  - 잠깐만요
+  - 죄송한데
+  - 아니요
+  - 근데

configs/prompts/simulation.yaml

@@ -507,3 +507,146 @@ user_simulator:
     For languages that use non-Latin scripts, spell out characters using their standard phonetic names in your language.
 
     IMPORTANT: Before ending the conversation, confirm with the agent that there are no outstanding actions. The end_call tool should only be called in a turn that is a brief goodbye — never in the same turn where you are providing the agent with data, an identifier, a request to transfer to a live agent, an approval to proceed, or any kind of additional information.
+
+  interruption_decision: |
+    You are analyzing a conversation to decide if the user should interrupt the agent.
+
+    This is what the user called to accomplish, so you can judge what they still have left
+    to do. The fields mean:
+
+    - GOAL: what the user is trying to achieve overall.
+    - MUST HAVE: non-negotiable requirements. The user will never accept an outcome that
+      fails any of these, and will not hang up until all of them are met.
+    - NICE TO HAVE: things the user wants but will give up if necessary.
+    - HOW THEY EVALUATE OPTIONS: the steps the user follows when the agent presents choices.
+    - RESOLVED WHEN: the success condition. Once this is met the user says a brief goodbye
+      and ends the call, so there is nothing left worth interrupting for.
+    - FAILED WHEN: the failure condition. This also ends the call.
+    - ESCALATION: how the user handles being transferred to a live agent.
+
+    <user_goal>
+    {user_goal}
+    </user_goal>
+
+    Conversation history (most recent at bottom):
+
+    <conversation_history>
+    {conversation_history}
+    </conversation_history>
+
+    The agent is CURRENTLY speaking (you can see their ongoing speech in the conversation above).
+
+    Based on the conversation so far, should the user interrupt the agent NOW?
+
+    Consider:
+    - Has the user heard enough to understand what the agent is asking or saying?
+    - Has the user heard enough to have a response, question, or correction ready?
+    - Did the agent just complete the sentence which has all the pertinent information the user was looking for?
+    - Do NOT repeatedly interrupt the agent if it has spoken only a few words (roughly fewer than 5 words in English, or the equivalent short fragment in whatever language the conversation is in — word counts are not comparable across languages).
+    - Are the user's request(s) basically accomplished and the user is likely to hang up on the next turn (if so it should not interrupt)?
+    - Is this a logical point in the conversation to interrupt?
+
+    Respond with ONLY "YES" if the user should interrupt now, or "NO" if they should keep listening.
+
+  backchannel_decision: |
+    You simulate a natural listener who occasionally makes a brief continuer sound to show they're following along.
+
+    The conversation may be in any language. Judge the FUNCTION of what is said, not the
+    presence of any particular word: every language has these sounds, and the caller draws
+    its own from a vocabulary configured for the conversation's language. The English
+    examples below illustrate the judgement, not the words to look for.
+
+    <conversation_history>
+    {conversation_history}
+    </conversation_history>
+
+    The agent is still speaking [CURRENTLY SPEAKING, INCOMPLETE]. Ignore the trailing incomplete word/phrase — focus only on the COMPLETE sentences delivered so far in the agent's current turn.
+
+    Continuers (English: "uh-huh", "mm-hmm", "yeah"; every language has equivalents) are brief
+    sounds that mean "I'm listening, keep going." They:
+    - Happen naturally during extended speech
+    - Show engagement without interrupting
+    - Are NOT responses to specific content — just signals of attention
+
+    Say YES if:
+    - The agent has completed at least 2 full, substantive sentences in their current turn
+      (Short phrases like "Thanks for your patience" or "Let me check on that" don't count as substantive)
+    - The user hasn't spoken or backchanneled recently (check the last 3 exchanges for ANY brief
+      acknowledgement sound from the user, in whatever language the conversation is in)
+    - It would feel natural to briefly signal "I'm still here"
+
+    Say NO if:
+    - The agent just started speaking (fewer than 2 substantive sentences)
+    - The user spoke OR backchanneled within the last 2-3 exchanges
+    - The agent's current turn contains or ends with a question
+    - The agent is wrapping up or about to finish their thought
+
+    Frequency guidance:
+    - Continuers are occasional, not constant
+    - Even when conditions seem right, real listeners only backchannel sometimes
+    - Aim for roughly 1 continuer per 4-6 sentences of extended agent speech
+    - When in doubt, say NO — silence is also natural
+    - Too few continuers is better than too many
+
+    Examples (English, for illustration — apply the same judgement in any language):
+
+    AGENT: "Hi there! How can I hel [CURRENTLY SPEAKING, INCOMPLETE]"
+    → NO (just started)
+
+    AGENT: "Thanks for your patience. [CURRENTLY SPEAKING, INCOMPLETE]"
+    → NO (only 1 short sentence, not substantive enough)
+
+    AGENT: "Sure, I can help with that. First I'll need to verify your account. Could you provide your email or your name and zi [CURRENTLY SPEAKING, INCOMPLETE]"
+    → NO (agent is asking a question)
+
+    AGENT: "No problem. We can use your name and zip code instead. Let me look that up for you. I'll check our system now and see if I can fin [CURRENTLY SPEAKING, INCOMPLETE]"
+    → YES (3 substantive sentences, agent explaining process)
+
+    AGENT: "I found your order. It includes a keyboard, thermostat, and headphones. The order was delivered last Tuesday. Now for the exchange, we have a few opti [CURRENTLY SPEAKING, INCOMPLETE]"
+    → YES (extended explanation with specific details)
+
+    [If the user made a brief acknowledgement sound 2 exchanges ago]
+    AGENT: "...and those are the available options. Now I'll need your input on which [CURRENTLY SPEAKING, INCOMPLETE]"
+    → NO (user backchanneled recently, don't do it again so soon)
+
+    Respond with ONLY "YES" or "NO".
+
+  cascade_next_interruption: |
+    You just said this line to the agent:
+
+    <what_you_just_said>
+    {utterance}
+    </what_you_just_said>
+
+    Write the single line you would say to cut the agent off mid-reply, if its
+    answer turns out to need correcting or clarifying.
+
+    Rules:
+    - Write it as a natural interruption, not a full turn. Short.
+    - It must follow from your own goal, not from any specific thing the agent
+      might say — you have not heard the reply yet.
+    - Write it in the same language as the line above; the caller does not switch
+      languages mid-call.
+    - Reply with ONLY the spoken line, nothing else. No quotes, no labels.
+    - If you would have no reason to interrupt whatever the agent says next,
+      reply with exactly NONE.
+
+  cascade_relevance_gate: |
+    The caller prepared this line a moment ago, before hearing the rest of what
+    the agent is currently saying:
+
+    <prepared_line>
+    {candidate}
+    </prepared_line>
+
+    Here is what the agent has said so far in its current turn:
+
+    <heard_so_far>
+    {heard}
+    </heard_so_far>
+
+    Would saying the prepared line right now still make sense as a natural
+    interruption? Answer NO if it has become irrelevant, already been addressed,
+    or would read as a non-sequitur.
+
+    Respond with ONLY "YES" or "NO".

pyproject.toml

@@ -35,6 +35,8 @@ dependencies = [
     "azure-cognitiveservices-speech>=1.31.0",
     "cartesia>=1.0.0",
     "assemblyai>=0.17.0",
+    "livekit-agents>=1.6.8",
+    "livekit-plugins-elevenlabs>=1.6.8",
     "setuptools>=65.0.0",
     "fastapi>=0.100.0",
     "uvicorn>=0.23.0",

scripts/add_culture_data.py

@@ -103,6 +103,7 @@
 REPO_ROOT = Path(__file__).resolve().parent.parent
 DATA_DIR = REPO_ROOT / "data"
 INITIAL_MESSAGES_PATH = REPO_ROOT / "configs" / "agents" / "initial_messages.yaml"
+CALLER_PHRASES_PATH = REPO_ROOT / "configs" / "caller_phrases.yaml"
 WER_CONFIGS_DIR = REPO_ROOT / "src" / "eva" / "utils" / "wer_normalization" / "configs"
 
 DEFAULT_MODEL = "gpt-5.5-2026-04-23"
@@ -353,6 +354,63 @@ def _update_initial_messages(language: str, message: str) -> None:
     INITIAL_MESSAGES_PATH.write_text(yaml.safe_dump(existing, allow_unicode=True, sort_keys=True), encoding="utf-8")
 
 
+async def _translate_caller_phrases(
+    language: str, language_name: str, llm: LLMClient, overwrite: bool = False
+) -> dict[str, list[str]] | None:
+    """Generate the simulated caller's out-of-turn vocabulary for ``language``.
+
+    These are the continuers and barge-in openers the caller speaks *as audio*, so
+    they must be things a native speaker actually says, not translations of the
+    English ones — "uh-huh" has no word-for-word equivalent in most languages.
+
+    Returns None when the language already has phrases and ``overwrite`` is unset.
+    """
+    existing: dict[str, Any] = {}
+    if CALLER_PHRASES_PATH.exists():
+        existing = yaml.safe_load(CALLER_PHRASES_PATH.read_text(encoding="utf-8")) or {}
+    if language in existing and not overwrite:
+        return None
+
+    english = existing.get("en", {})
+    prompt = (
+        f"You are localising a simulated phone caller for {language_name}.\n\n"
+        "Produce two short lists of things the caller says out loud:\n\n"
+        "1. backchannels — brief continuer sounds meaning 'I am listening, keep going', "
+        "said while the other person is still talking. Give the sounds a native speaker "
+        "actually makes, not translations of English ones.\n"
+        "2. barge_in_openers — the very first word or two of an interruption, said just "
+        "before the interrupting sentence. Keep them to one or two words so they sound "
+        "like a real cut-in.\n\n"
+        "These are spoken aloud by a text-to-speech voice, so use ordinary spelling with "
+        "no stage directions, no parentheses and no transliteration hints. Give 2-3 "
+        "backchannels and 3-4 openers.\n\n"
+        f"For reference, the English set is: {json.dumps(english, ensure_ascii=False)}\n\n"
+        'Return JSON: {"backchannels": ["..."], "barge_in_openers": ["..."]}'
+    )
+    text, _ = await llm.generate_text(
+        [{"role": "user", "content": prompt}],
+        response_format={"type": "json_object"},
+    )
+    data = extract_and_load_json(text)
+    backchannels = [str(p).strip() for p in (data.get("backchannels") or []) if str(p).strip()]
+    openers = [str(p).strip() for p in (data.get("barge_in_openers") or []) if str(p).strip()]
+    if not backchannels or not openers:
+        raise ValueError(f"Caller phrase generation returned an incomplete result: {data!r}")
+    return {"backchannels": backchannels, "barge_in_openers": openers}
+
+
+def _update_caller_phrases(language: str, phrases: dict[str, list[str]]) -> None:
+    """Merge one language's phrase set into configs/caller_phrases.yaml."""
+    existing: dict[str, Any] = {}
+    if CALLER_PHRASES_PATH.exists():
+        existing = yaml.safe_load(CALLER_PHRASES_PATH.read_text(encoding="utf-8")) or {}
+    if existing.get(language) == phrases:
+        return
+    existing[language] = phrases
+    CALLER_PHRASES_PATH.parent.mkdir(parents=True, exist_ok=True)
+    CALLER_PHRASES_PATH.write_text(yaml.safe_dump(existing, allow_unicode=True, sort_keys=True), encoding="utf-8")
+
+
 async def _translate_aliases(
     name_to_base: dict[str, list[str]],
     language_name: str,
@@ -680,6 +738,16 @@ async def amain(args: argparse.Namespace) -> int:
         _update_initial_messages(args.language, initial_message)
         logger.info(f"Updated {INITIAL_MESSAGES_PATH}")
 
+    logger.info(f"Generating caller out-of-turn phrases for {args.language_name}")
+    caller_phrases = await _translate_caller_phrases(args.language, args.language_name, llm, args.overwrite_all)
+    if caller_phrases is None:
+        logger.info(f"Caller phrases for {args.language} already present — skipping")
+    elif args.dry_run:
+        logger.info(f"[dry-run] would write caller phrases for {args.language}: {caller_phrases}")
+    else:
+        _update_caller_phrases(args.language, caller_phrases)
+        logger.info(f"Updated {CALLER_PHRASES_PATH}: {caller_phrases}")
+
     for domain in domains:
         logger.info(f"=== Domain: {domain} ===")
         if domain == "airline":

src/eva/__init__.py

@@ -7,7 +7,7 @@
 
 # Bump simulation_version when changes affect benchmark outputs (agent code,
 # user simulator, orchestrator, simulation prompts, agent configs, tool mocks).
-simulation_version = "2.0.2"
+simulation_version = "2.0.34"
 
 # Bump metrics_version when changes affect metric computation (metrics code,
 # judge prompts, pricing tables, postprocessor).

src/eva/assistant/base_server.py

@@ -41,6 +41,15 @@ class AbstractAssistantServer(ABC):
     5. Populate the AuditLog with conversation events
     """
 
+    supports_unpaced_output: bool = False
+    """Whether this server can honor ``paced_output=False``.
+
+    Only a server whose outbound relay throttle we own can drop it. A server whose
+    pacing lives inside a third-party runtime cannot, and must reject the request
+    rather than accept it and keep pacing, which would leave a tick-driven caller
+    believing the assistant was unpaced.
+    """
+
     def __init__(
         self,
         current_date_time: str,
@@ -52,6 +61,7 @@ def __init__(
         port: int,
         conversation_id: str,
         language: str = "en",
+        paced_output: bool = True,
     ):
         """Initialize the assistant server.
 
@@ -65,10 +75,20 @@ def __init__(
             port: Port to listen on
             conversation_id: Unique ID for this conversation
             language: BCP 47 language tag for STT/TTS/S2S services (e.g. 'en', 'fr', 'es-MX')
+            paced_output: Whether to emit audio at real-time cadence. True for callers
+                that infer turn boundaries from silence timing (the ElevenLabs simulator).
+                False for a tick-driven caller, which buffers whatever arrives and
+                releases it one tick at a time, so pacing here would only add latency.
         """
         self.current_date_time = current_date_time
         self.pipeline_config = pipeline_config
         self.language = language
+        if not paced_output and not self.supports_unpaced_output:
+            raise ValueError(
+                f"{type(self).__name__} cannot honor paced_output=False: its output pacing is not ours to remove. "
+                "Tick-driving this framework requires leaving pacing on."
+            )
+        self.paced_output = paced_output
         self.initial_message = get_initial_message(language)
         self.agent: AgentConfig = agent
         self.agent_config_path = agent_config_path