Add agent stt sdk - #127
Open
giorgosHadji wants to merge 19 commits into
Open
Conversation
speechmatics-agent-stt is an extension of the RT SDK for the Voice Agent Service. The service works in segments rather than word groups and reports speech and turn events, so the client consumes AddSegment/AddPartialSegment and accumulates the session transcript from them. The SDK runs no VAD or turn detection of its own. Either the service's VAD closes turns (VADMode.SERVER), or the host framework's does (VADMode.CLIENT) by calling finalize(), which sends ForceEndOfUtterance stamped with the audio position at the moment of the call. This is what Pipecat and LiveKit need, and replaces the voice SDK's in-process VAD and smart-turn models. - endpoint is the RT URL plus /agent, optionally plus a service profile - TranscriptionConfig adds vad_mode, vad_config and emit_sentences, and leaves model unset so it cannot conflict with the profile's locked operating_point - every server message is kept in client.events, including unmodelled ones - 16 kHz raw PCM defaults, as the service requires Verified against the real service with a stub transcriber behind it. Nothing in the voice SDK or Pipecat is touched; see sdk/agent_stt/PLAN.md for the Pipecat migration plan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Engine silence-based end of utterance is off for the Agent STT service, so TurnDetectionMode.FIXED has nothing to map onto and is removed rather than aliased. Records the settings that go with it, and notes in the SDK README that a turn ends only via the service VAD or finalize(). SMART_TURN stays open: it ran an in-process model, which this SDK does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The service has no smart-turn endpoint yet and this SDK loads no models, so the mode cannot be honoured. It costs nothing: any host-side endpointing, including Pipecat's own turn analyzer, reaches the service through finalize(). Makes VADMode.CLIENT explicit that it is agnostic about what produced the end-of-speech signal - VAD, turn model, or push-to-talk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A typed entry for transcription_config.additional_vocab, so callers can pass words with pronunciation hints instead of raw dicts. Both forms are accepted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The SDK previously left `model` unset, on the reasoning that the service's default profile pins and locks `operating_point: enhanced`, so sending a model name alongside it would put both keys in the merged StartRecognition. That reasoning held for a direct connection to the service websocket, but it is not how requests actually reach it: they go through a proxy first, which resolves the Agent STT model name onto the engine's operating point. The transcriber therefore never sees a name it has no notion of, and sending `linden-1` is safe. Replace the re-exported RT `Model` with an Agent STT one. The RT models (`enhanced`, `standard`) are not Agent STT models, so leaving them reachable from this package only invited configs the service would reject. Default resolution uses the same `_UNSET` sentinel pattern as `rt.TranscriptionConfig` rather than a plain default, because the deprecated `operating_point` has to suppress the model rather than collide with it: passing `operating_point` leaves `model` unsent, and passing both still raises. `DEFAULT_MODEL` is a separate constant and the docstrings name it rather than the value, so linden-2 is one enum member plus, if it becomes the default, one line. The end-to-end verification recorded in PLAN.md predates this and ran against the service with a stub transcriber and no proxy, so it exercised neither the model default nor its resolution. Noted there as still needing a run against the real proxy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
giorgosHadji
commented
Aug 18, 2026
giorgosHadji
commented
Aug 18, 2026
giorgosHadji
commented
Aug 18, 2026
Contributor
Author
There was a problem hiding this comment.
Will let it so people can view it and will remove once reviewing is done
giorgosHadji
commented
Aug 18, 2026
| @@ -0,0 +1,148 @@ | |||
| # Speechmatics Agent STT SDK | |||
Contributor
Author
There was a problem hiding this comment.
Needs public docs ref here too
giorgosHadji
commented
Aug 18, 2026
giorgosHadji
commented
Aug 18, 2026
|
|
||
| ## Audio | ||
|
|
||
| The service requires **16 kHz raw PCM**, `pcm_s16le` or `pcm_f32le`, which is what the client |
Contributor
Author
There was a problem hiding this comment.
and mulaw I believe
giorgosHadji
marked this pull request as draft
August 18, 2026 11:20
giorgosHadji
commented
Aug 18, 2026
giorgosHadji
commented
Aug 18, 2026
giorgosHadji
commented
Aug 18, 2026
added 9 commits
August 18, 2026 13:14
giorgosHadji
marked this pull request as ready for review
August 18, 2026 13:01
giorgosHadji
commented
Aug 18, 2026
added 2 commits
August 18, 2026 14:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add agent stt sdk which will communicate with our AgentSTT offering - add docs here.
This MR doesn't remove/deprecate the voice sdk - that will happen in follow up MRs.
How to test this:
Get an api key, and point to rt preview (SPEECHMATICS_RT_URL=wss://preview.rt.speechmatics.com/v2)
Install locally the agent stt sdk - >
python -m pip install -e sdk/agent_stt[dev]Then run for a file based test :
python examples/agent_stt/realtime_file/main.pyor you can use an interactive script (using microphone):
python .\examples\agent_stt\microphone\main.py