Add AWS/Bedrock user simulator (Nova Sonic) + optional Bedrock agent STT/TTS & judges#183
Open
nevo-biton wants to merge 3 commits into
Open
Add AWS/Bedrock user simulator (Nova Sonic) + optional Bedrock agent STT/TTS & judges#183nevo-biton wants to merge 3 commits into
nevo-biton wants to merge 3 commits into
Conversation
Adds a native, audio-native S2S caller (`provider="bedrock_s2s"`) — the AWS analog of the OpenAI Realtime caller. EVA streams the agent's audio straight to Amazon Nova Sonic over Bedrock's bidirectional streaming API and streams the caller's spoken reply back; Nova Sonic also emits both transcripts itself (agent ASR + caller text), so there is no separate STT/LLM/TTS cascade. - `BedrockS2SUserSimulator` mirrors `OpenAIRealtimeUserSimulator`: same `BotToBotAudioBridge`, same `_build_prompt()` (sent as the Nova SYSTEM text), same `end_call` tool. Turn-taking uses Nova's native endpointing plus half-duplex suppression (gated on the bridge's is_caller_playing()). - Audio: μ-law 8 kHz -> LPCM 16 kHz in; Nova output pinned to 16 kHz to match the bridge (no resample) out. - `BedrockS2SSimulatorConfig` added to the `UserSimulatorConfig` union; `factory.create_user_simulator` gains a guarded `bedrock_s2s` branch. Robustness (found and fixed via a live smoke against a stub agent): - Keepalive silence frames so Nova doesn't drop the session on its ~55s no-input timeout when the agent goes quiet / at end-of-call. - Signal the bridge with notify_user_utterance_complete() at caller turn end, or is_caller_playing() wedges True and starves the input stream (a freeze). - Buffer the caller's text and flush once per turn (Nova may only send the SPECULATIVE preview, not a FINAL, for short turns). - end_call terminates promptly (Nova may invoke it as a pure tool with no trailing audio END_TURN); _finish_caller_response is idempotent. The experimental `aws-sdk-bedrock-runtime` client (pre-1.0, Python >=3.12) is isolated as an optional extra (`eva[bedrock-s2s]`), imported lazily behind a factory guard, with a scoped mypy override for its untyped smithy deps — so EVA's 3.11 floor and base install are unaffected. Tests are fully mocked (no AWS creds / no SDK required). Live-validated end-to-end against a real Nova Sonic session (agent ASR, caller speech + text, end_call -> clean hang-up). `simulation_version` bumped; no metrics touched.
Completes the all-AWS agent-under-test pipeline. The agent LLM already runs on Bedrock via EVA's LiteLLM router (a bedrock/... deployment in EVA_MODEL_LIST); this adds the missing STT and TTS so a cascade agent runs entirely on AWS (Amazon Transcribe -> Bedrock -> Amazon Polly). - services.py: `aws_transcribe` (aliases: aws, transcribe, amazon_transcribe) and `aws_polly` (aliases: aws, polly, amazon_polly) branches wiring pipecat's `AWSTranscribeSTTService` / `AWSPollyTTSService`. Credentials + region resolve from params then the standard AWS env chain via a shared `_aws_credentials` helper; voice/engine/language forwarded through the pipecat Settings. - config.py: AWS Transcribe/Polly authenticate via the env chain and have no api_key/model, so `_validate_service_params` skips the required-keys check for them (`_AWS_ENV_CREDENTIALED_STT_TTS`), and `_param_alias` / `pipeline_parts` fall back to the provider name when params are absent. - runner.py: `resolved_models` no longer assumes stt_params/tts_params carry a "model" (None for AWS) — falls back to the provider name. - No new dependencies: pipecat's Transcribe uses `websockets` (core) and Polly uses `aiobotocore` (already pulled by `aioboto3`). Not using pipecat's [aws] extra because it pins aiobotocore>=3, conflicting with aioboto3's 2.x. - `.env.example`: all-AWS cascade example; `simulation_version` bumped. Tests: STT/TTS creation (aliases, voice/engine, no api_key/model) and a config test that a full all-AWS cascade validates. Live-validated end-to-end: an itsm conversation ran with Transcribe (STT) -> Bedrock Claude (LLM) -> Polly (TTS) against the bedrock_s2s Nova Sonic caller (completed=True; STT/LLM/TTS latencies all recorded).
Lets the validation judges run on Amazon Bedrock so an all-AWS pipeline validates cleanly. The text judge (user_behavioral_fidelity) already honors the JUDGE_MODEL env var, but two things blocked Bedrock: - The audio judge (user_speech_fidelity) needs an audio-capable model; no Bedrock model accepts audio input (Nova understanding = image/video/doc only; audio is Nova Sonic streaming). ValidationRunner now skips the audio judge when its model isn't a deployment in EVA_MODEL_LIST — so AWS-only validation runs the Bedrock-capable judges instead of erroring on a model that can never run. - The LLM judges send service_tier="flex" (OpenAI/Azure-only); Bedrock rejects it and litellm's drop_params doesn't strip it. LLMClient now drops service_tier when the judge model doesn't resolve to an OpenAI/Azure deployment. Config: set JUDGE_MODEL to a Bedrock model_name in EVA_MODEL_LIST (documented in .env.example). Live-validated: an all-AWS itsm run now passes validation (user_behavioral_fidelity on Bedrock score=1.0; audio judge skipped; 100% success). simulation_version bumped (orchestrator change). No metric outputs change for existing OpenAI/Gemini setups.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed in #182 — addresses the User Simulator Options roadmap item and extends AWS/Bedrock across the pipeline. Happy to split into separate PRs if you prefer (caller / agent STT+TTS / judges).
What
A full AWS/Bedrock EVA pipeline — caller, agent-under-test, and validation judges can all run on AWS. Three commits:
bedrock_s2suser simulator — an audio-native Amazon Nova Sonic caller (Bedrock bidirectional streaming), the AWS analog ofOpenAIRealtimeUserSimulator(reusesBotToBotAudioBridge,_build_prompt(), theend_calltool). This is the roadmap "User Simulator Options" item.model_list).Configure with
EVA_USER_SIMULATOR__PROVIDER=bedrock_s2s,EVA_MODEL__STT=aws_transcribe,EVA_MODEL__TTS=aws_polly, anEVA_MODEL__LLMpointing at abedrock/...deployment, andJUDGE_MODEL=<a bedrock model_name>. AWS creds + region come from the standard env chain.Live-validated end-to-end ✅
A real
itsmconversation ran entirely on AWS — thebedrock_s2sNova Sonic caller vs a pipecat agent (Transcribe → Bedrock Claude → Polly) — and passed validation on Bedrock (user_behavioral_fidelityscore=1.0, audio judge auto-skipped, 100% success). Per-stage latencies confirmed each AWS service ran.Commit 1 —
bedrock_s2scallerAudio-native S2S over Nova Sonic bidirectional streaming; mirrors
OpenAIRealtimeUserSimulator. Turn-taking uses Nova's native endpointing + half-duplex suppression, with a ~55s keepalive,notify_user_utterance_complete()at turn end, buffered caller-text flush (SPECULATIVE→FINAL), and promptend_calltermination.Commit 2 — Amazon Transcribe / Polly assistant STT+TTS
create_stt_service/create_tts_servicegainaws_transcribe/aws_pollybranches wiring pipecat'sAWSTranscribeSTTService/AWSPollyTTSService; creds/region via a shared helper._validate_service_paramsskipsapi_key/modelfor AWS (env-credentialed). No new dependencies — useswebsockets(core) +aiobotocore(viaaioboto3); avoids pipecat's[aws]extra, which pinsaiobotocore>=3and conflicts withaioboto32.x.Commit 3 — Bedrock validation judges
Audio judge (
user_speech_fidelity) auto-skips when its model isn't a deployment inEVA_MODEL_LIST(no Bedrock model accepts audio input).LLMClientdrops the OpenAI-onlyservice_tierparam for non-OpenAI/Azure judge models (Bedrock rejects it and litellm'sdrop_paramsdoesn't strip it).JUDGE_MODELpoints the text judges at a Bedrock deployment.Caveats (please review)
bedrock_s2scaller needs the experimentalaws-sdk-bedrock-runtime(pre-1.0, Python ≥3.12), isolated as an optionaleva[bedrock-s2s]extra behind a lazy factory guard + scoped mypy override — EVA's 3.11 floor and base install are unaffected.Tests
Fully mocked (no AWS creds / no experimental SDK needed): the caller, the AWS STT/TTS branches + config validation, the audio-judge-skip guard, and
service_tierprovider resolution.ruffclean; full unit suite green (1866 passed).simulation_versionbumped; no metric-output change for existing OpenAI/Gemini setups.