From abc665047923ef5cc41c4f4b10bd7ac504b41f72 Mon Sep 17 00:00:00 2001 From: nickrobin Date: Mon, 17 Aug 2026 11:12:18 -0700 Subject: [PATCH 1/2] docs: update AssemblyAI transcriber docs for Universal 3.5 Pro and multilingual support Addresses AssemblyAI team feedback on docs.vapi.ai: - assembly-ai provider page: add Universal 3.5 Pro model (modes, prompt, agentContext, languageCodes, keyterms), correct the keyterms caveat - custom-keywords: retitle provider-neutral, add AssemblyAI keyterms section - multilingual table: AssemblyAI now full auto-detection, 18 languages - voice-pipeline-configuration: drop stale English-only claims for Assembly Co-Authored-By: Claude Fable 5 --- fern/customization/custom-keywords.mdx | 53 +++++++++++++++++-- fern/customization/multilingual.mdx | 2 +- .../voice-pipeline-configuration.mdx | 4 +- fern/providers/transcriber/assembly-ai.mdx | 31 +++++++++-- 4 files changed, 79 insertions(+), 11 deletions(-) diff --git a/fern/customization/custom-keywords.mdx b/fern/customization/custom-keywords.mdx index 0327cb3b7..0731668b7 100644 --- a/fern/customization/custom-keywords.mdx +++ b/fern/customization/custom-keywords.mdx @@ -1,11 +1,13 @@ --- -title: Deepgram Keywords and Keyterm Prompting +title: Keywords and keyterm prompting subtitle: Boost STT accuracy for domain words and phrases slug: customization/custom-keywords --- -Vapi allows you to improve the accuracy of your transcriptions by leveraging Deepgram's keyword boosting feature. This is particularly useful when dealing with specialized terminology or uncommon proper nouns. By providing specific keywords to the Deepgram model, you can enhance transcription quality directly through Vapi. +Vapi allows you to improve the accuracy of your transcriptions by leveraging keyword boosting and keyterm prompting. This is particularly useful when dealing with specialized terminology or uncommon proper nouns. Both [Deepgram](#deepgram-keywords-and-keyterm-prompting) and [AssemblyAI](#assemblyai-keyterms-prompting) transcribers support this through Vapi. + +## Deepgram keywords and keyterm prompting ### Why Use Keyword Boosting? @@ -26,9 +28,9 @@ Keyword boosting is beneficial for: - Keywords is available on Deepgram Nova-2, Nova-1, Enhanced, and Base speech-to-text models. - For Nova-3 models, use Keyterm Prompting instead of Keywords. -## Enabling Keyword Boosting in Vapi +### Enabling Keyword Boosting in Vapi -### API Call Integration +#### API Call Integration To enable keyword boosting, add the `keywords` parameter to your assistant's `transcriber` configuration when using the Deepgram provider. You can also supply `keyterm` to boost recall for phrases. @@ -118,4 +120,45 @@ For more details, see: - Deepgram Keyterm Prompting: [developers.deepgram.com/docs/keyterm](https://developers.deepgram.com/docs/keyterm) - API reference: Deepgram transcriber `keywords` and `keyterm` in the [API reference](https://api.vapi.ai/api#:~:text=DeepgramTranscriber) -By following these guidelines, you can effectively utilize Deepgram's keyword boosting feature within your Vapi assistant, ensuring enhanced transcription accuracy for specialized terminology and uncommon proper nouns. \ No newline at end of file +By following these guidelines, you can effectively utilize Deepgram's keyword boosting feature within your Vapi assistant, ensuring enhanced transcription accuracy for specialized terminology and uncommon proper nouns. + +## AssemblyAI keyterms prompting + +AssemblyAI's Universal-Streaming keyterms prompting boosts recognition of domain-specific words and phrases. Add the `keytermsPrompt` parameter to your assistant's `transcriber` configuration when using the `assembly-ai` provider. + +- Up to 100 keyterms per session, each up to 50 characters. +- Keyterms can be single words or multi-word phrases — no intensifiers needed. +- Supported with the `universal-streaming-english` and `universal-3-5-pro` speech models. Not supported with `universal-streaming-multilingual`. +- Keyterms prompting adds $0.04/hour to transcription cost on `universal-streaming-english`. + +### Example + +```bash +curl \ + --request POST \ + --header 'Authorization: Bearer ' \ + --header 'Content-Type: application/json' \ + --data '{ + "name": "Emma", + "model": { + "model": "gpt-4o", + "provider": "openai" + }, + "transcriber": { + "provider": "assembly-ai", + "speechModel": "universal-streaming-english", + "keytermsPrompt": [ + "order number", + "account ID", + "PCI compliance" + ] + }, + "firstMessage": "Hi, I am Emma, what is your name?" + }' \ + https://api.vapi.ai/assistant +``` + +### Additional resources + +- AssemblyAI Keyterms Prompting: [assemblyai.com/docs/streaming/keyterms-prompting](https://www.assemblyai.com/docs/streaming/keyterms-prompting) +- API reference: [`AssemblyAITranscriber` fields](/api-reference/assistants/create#request.body.transcriber.AssemblyAITranscriber) \ No newline at end of file diff --git a/fern/customization/multilingual.mdx b/fern/customization/multilingual.mdx index 1be7a6efb..d280427ce 100644 --- a/fern/customization/multilingual.mdx +++ b/fern/customization/multilingual.mdx @@ -461,7 +461,7 @@ Validate your configuration with different languages and scenarios. | **Deepgram** | ✅ Full auto-detection | 100+ | **Recommended**: Nova 2/Nova 3 with "Multi" language setting | | **Google STT** | ✅ Full auto-detection | 125+ | Latest models with "Multilingual" language setting | | **Gladia** | ✅ Full auto-detection | 110+ | Supports automatic language detection and code-switching | -| **Assembly AI** | ❌ English only | English | No multilingual support | +| **Assembly AI** | ✅ Full auto-detection | 18 | Universal 3.5 Pro (`universal-3-5-pro`) supports automatic language detection and code-switching | | **Azure STT** | ❌ Single language | 100+ | Many languages, but no auto-detection | | **OpenAI Whisper** | ❌ Single language | 90+ | Many languages, but no auto-detection | | **Speechmatics** | ❌ Single language | 50+ | Many languages, but no auto-detection | diff --git a/fern/customization/voice-pipeline-configuration.mdx b/fern/customization/voice-pipeline-configuration.mdx index d3a3bb4d6..0bde1356a 100644 --- a/fern/customization/voice-pipeline-configuration.mdx +++ b/fern/customization/voice-pipeline-configuration.mdx @@ -222,7 +222,7 @@ Uses AI models to analyze speech patterns, context, and audio cues to predict wh **Audio-text based providers:** - **deepgram-flux**: Deepgram's latest transcriber model with built-in conversational speech recognition. Use `flux-general-en` for English-only conversations or `flux-general-multi` for multilingual conversations. - - **assembly**: Transcriber with built-in end-of-turn detection (English only) + - **assembly**: Transcriber with built-in end-of-turn detection (English, multilingual, and Universal 3.5 Pro models)
@@ -232,7 +232,7 @@ Uses AI models to analyze speech patterns, context, and audio cues to predict wh **When to use smart endpointing:** - **Deepgram Flux**: English and Multi-lingual conversations using Deepgram as a transcriber. -- **Assembly**: Best used when Assembly is already your transcriber provider for English conversations with integrated end-of-turn detection +- **Assembly**: Best used when Assembly is already your transcriber provider, with integrated end-of-turn detection - **LiveKit**: English conversations where Deepgram is not the transcriber of choice. - **Vapi**: Non-English conversations with default stop speaking plan settings - **Krisp**: Non-English conversations with a robustly configured stop speaking plan diff --git a/fern/providers/transcriber/assembly-ai.mdx b/fern/providers/transcriber/assembly-ai.mdx index 536c33051..b32628fae 100644 --- a/fern/providers/transcriber/assembly-ai.mdx +++ b/fern/providers/transcriber/assembly-ai.mdx @@ -11,7 +11,7 @@ Universal-Streaming is AssemblyAI's purpose-built speech-to-text model that deli ## Speech-to-text -Set `transcriber.provider` to `assembly-ai`. AssemblyAI's Universal-Streaming model is selected automatically, so no `model` field is required. +Set `transcriber.provider` to `assembly-ai`. By default the `universal-streaming-english` speech model is used; set `transcriber.speechModel` to choose a different model. ```bash curl -X PATCH "https://api.vapi.ai/assistant/ASSISTANT_ID" \ @@ -25,6 +25,30 @@ curl -X PATCH "https://api.vapi.ai/assistant/ASSISTANT_ID" \ }' ``` +### Universal 3.5 Pro + +Universal 3.5 Pro (`universal-3-5-pro`) is AssemblyAI's most accurate voice-agent model, with automatic language detection and code-switching across 18 languages. + +```bash +curl -X PATCH "https://api.vapi.ai/assistant/ASSISTANT_ID" \ + -H "Authorization: Bearer $VAPI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "transcriber": { + "provider": "assembly-ai", + "speechModel": "universal-3-5-pro" + } + }' +``` + +Universal 3.5 Pro supports additional configuration: + +- `mode` — trade off accuracy and latency: `max_accuracy`, `balanced`, or `min_latency`. +- `prompt` — additional context for the transcription model (up to 1,750 characters). +- `agentContext` — context about your voice agent that guides the transcription model (up to 1,750 characters). +- `languageCodes` — language codes used to steer automatic language detection, e.g. `["en", "es"]`. +- `keytermsPrompt` — domain-specific terms to boost recognition accuracy. + For additional API configuration options, review the [`AssemblyAITranscriber` fields](/api-reference/assistants/create#request.body.transcriber.AssemblyAITranscriber) in the Create Assistant API reference. ### Configure in the dashboard @@ -54,14 +78,15 @@ This guide details how to setup AssemblyAI as a transcriber for your assistant. ### Supported languages -Vapi supports AssemblyAI's English and multilingual Universal-Streaming modes. +Vapi supports AssemblyAI's English and multilingual Universal-Streaming modes, plus the Universal 3.5 Pro model. | Mode | Speech model | Language value | Supported languages | | --- | --- | --- | --- | | English | `universal-streaming-english` | `en` | English | | Multilingual | `universal-streaming-multilingual` | `multi` | English, French, German, Italian, Portuguese, and Spanish | +| Universal 3.5 Pro | `universal-3-5-pro` | Automatic | 18 languages with automatic language detection and code-switching — see [AssemblyAI's supported languages](https://www.assemblyai.com/docs/streaming/multilingual-transcription#supported-languages) | -Keyterms prompting is not supported with the multilingual model. +Keyterms prompting is supported with the `universal-streaming-english` and `universal-3-5-pro` models. It is not supported with `universal-streaming-multilingual`. ## Related From 3eecec9f8e81a86fce3f49708ca883639573626d Mon Sep 17 00:00:00 2001 From: Stephen Smith Date: Mon, 17 Aug 2026 12:49:39 -0700 Subject: [PATCH 2/2] docs: align AssemblyAI updates with provider standards --- fern/customization/custom-keywords.mdx | 4 +- fern/customization/multilingual.mdx | 4 +- .../voice-pipeline-configuration.mdx | 4 +- fern/providers/transcriber/assembly-ai.mdx | 51 +++++++++++++++---- 4 files changed, 47 insertions(+), 16 deletions(-) diff --git a/fern/customization/custom-keywords.mdx b/fern/customization/custom-keywords.mdx index 0731668b7..0fb5e0a0f 100644 --- a/fern/customization/custom-keywords.mdx +++ b/fern/customization/custom-keywords.mdx @@ -160,5 +160,5 @@ curl \ ### Additional resources -- AssemblyAI Keyterms Prompting: [assemblyai.com/docs/streaming/keyterms-prompting](https://www.assemblyai.com/docs/streaming/keyterms-prompting) -- API reference: [`AssemblyAITranscriber` fields](/api-reference/assistants/create#request.body.transcriber.AssemblyAITranscriber) \ No newline at end of file +- AssemblyAI Keyterms Prompting: [AssemblyAI's prompting and keyterms guide](https://www.assemblyai.com/docs/streaming/prompting-and-keyterms) +- API reference: [`AssemblyAITranscriber` fields](/api-reference/assistants/create#request.body.transcriber.AssemblyAITranscriber) diff --git a/fern/customization/multilingual.mdx b/fern/customization/multilingual.mdx index d280427ce..ccd910194 100644 --- a/fern/customization/multilingual.mdx +++ b/fern/customization/multilingual.mdx @@ -461,7 +461,7 @@ Validate your configuration with different languages and scenarios. | **Deepgram** | ✅ Full auto-detection | 100+ | **Recommended**: Nova 2/Nova 3 with "Multi" language setting | | **Google STT** | ✅ Full auto-detection | 125+ | Latest models with "Multilingual" language setting | | **Gladia** | ✅ Full auto-detection | 110+ | Supports automatic language detection and code-switching | -| **Assembly AI** | ✅ Full auto-detection | 18 | Universal 3.5 Pro (`universal-3-5-pro`) supports automatic language detection and code-switching | +| **AssemblyAI** | ✅ Full auto-detection | 18 | Set `speechModel` to `universal-3-5-pro` for automatic language detection and code-switching | | **Azure STT** | ❌ Single language | 100+ | Many languages, but no auto-detection | | **OpenAI Whisper** | ❌ Single language | 90+ | Many languages, but no auto-detection | | **Speechmatics** | ❌ Single language | 50+ | Many languages, but no auto-detection | @@ -482,7 +482,7 @@ Validate your configuration with different languages and scenarios. **Solutions:** - - Use Deepgram (Nova 2/Nova 3 with "Multi"), Google STT (with "Multilingual"), or Gladia (automatic language detection) + - Use AssemblyAI Universal 3.5 Pro (`speechModel: "universal-3-5-pro"`), Deepgram (Nova 2/Nova 3 with "Multi"), Google STT (with "Multilingual"), or Gladia (automatic language detection) - Ensure high-quality audio input for better detection accuracy - Test with native speakers of target languages - Consider provider-specific language combinations for optimal results diff --git a/fern/customization/voice-pipeline-configuration.mdx b/fern/customization/voice-pipeline-configuration.mdx index 0bde1356a..eb4106010 100644 --- a/fern/customization/voice-pipeline-configuration.mdx +++ b/fern/customization/voice-pipeline-configuration.mdx @@ -222,7 +222,7 @@ Uses AI models to analyze speech patterns, context, and audio cues to predict wh **Audio-text based providers:** - **deepgram-flux**: Deepgram's latest transcriber model with built-in conversational speech recognition. Use `flux-general-en` for English-only conversations or `flux-general-multi` for multilingual conversations. - - **assembly**: Transcriber with built-in end-of-turn detection (English, multilingual, and Universal 3.5 Pro models) + - **assembly**: Transcriber with built-in end-of-turn detection for Universal-Streaming English, Universal-Streaming Multilingual, and Universal 3.5 Pro
@@ -232,7 +232,7 @@ Uses AI models to analyze speech patterns, context, and audio cues to predict wh **When to use smart endpointing:** - **Deepgram Flux**: English and Multi-lingual conversations using Deepgram as a transcriber. -- **Assembly**: Best used when Assembly is already your transcriber provider, with integrated end-of-turn detection +- **AssemblyAI**: Use when AssemblyAI is already your transcriber provider and you want integrated end-of-turn detection - **LiveKit**: English conversations where Deepgram is not the transcriber of choice. - **Vapi**: Non-English conversations with default stop speaking plan settings - **Krisp**: Non-English conversations with a robustly configured stop speaking plan diff --git a/fern/providers/transcriber/assembly-ai.mdx b/fern/providers/transcriber/assembly-ai.mdx index b32628fae..86dc16a95 100644 --- a/fern/providers/transcriber/assembly-ai.mdx +++ b/fern/providers/transcriber/assembly-ai.mdx @@ -1,7 +1,7 @@ --- title: AssemblyAI -subtitle: Use AssemblyAI Universal-Streaming for real-time speech-to-text in Vapi. -description: Configure AssemblyAI Universal-Streaming as your Vapi assistant's transcriber through the dashboard or API. +subtitle: Use AssemblyAI for real-time speech-to-text in Vapi. +description: Configure AssemblyAI speech models, multilingual transcription, language steering, and prompting for a Vapi assistant. slug: providers/transcriber/assembly-ai --- @@ -9,6 +9,14 @@ slug: providers/transcriber/assembly-ai Universal-Streaming is AssemblyAI's purpose-built speech-to-text model that delivers ultra-fast, immutable transcripts in ~300ms with intelligent endpointing and superior accuracy for voice agents. It eliminates common pain points like misheard account numbers, awkward pauses, and premature cutoffs, enabling more natural and successful voice interactions. +AssemblyAI provides real-time speech-to-text and turn detection for Vapi voice agents. + +## Supported capability + +| Capability | Provider value | +| --- | --- | +| Speech-to-text | `assembly-ai` | + ## Speech-to-text Set `transcriber.provider` to `assembly-ai`. By default the `universal-streaming-english` speech model is used; set `transcriber.speechModel` to choose a different model. @@ -25,6 +33,8 @@ curl -X PATCH "https://api.vapi.ai/assistant/ASSISTANT_ID" \ }' ``` +For additional API configuration options, review the [`AssemblyAITranscriber` fields](/api-reference/assistants/create#request.body.transcriber.AssemblyAITranscriber) in the Create Assistant API reference. + ### Universal 3.5 Pro Universal 3.5 Pro (`universal-3-5-pro`) is AssemblyAI's most accurate voice-agent model, with automatic language detection and code-switching across 18 languages. @@ -43,13 +53,11 @@ curl -X PATCH "https://api.vapi.ai/assistant/ASSISTANT_ID" \ Universal 3.5 Pro supports additional configuration: -- `mode` — trade off accuracy and latency: `max_accuracy`, `balanced`, or `min_latency`. -- `prompt` — additional context for the transcription model (up to 1,750 characters). -- `agentContext` — context about your voice agent that guides the transcription model (up to 1,750 characters). -- `languageCodes` — language codes used to steer automatic language detection, e.g. `["en", "es"]`. -- `keytermsPrompt` — domain-specific terms to boost recognition accuracy. - -For additional API configuration options, review the [`AssemblyAITranscriber` fields](/api-reference/assistants/create#request.body.transcriber.AssemblyAITranscriber) in the Create Assistant API reference. +- `mode`: Trade off accuracy and latency with `max_accuracy`, `balanced`, or `min_latency`. +- `prompt`: Provide additional context for the transcription model, up to 1,750 characters. +- `agentContext`: Provide context about the assistant that helps the transcription model interpret caller speech, up to 1,750 characters. +- `languageCodes`: Steer automatic language detection with language codes such as `["en", "es"]`. +- `keytermsPrompt`: Boost recognition of domain-specific terms. ### Configure in the dashboard @@ -84,7 +92,30 @@ Vapi supports AssemblyAI's English and multilingual Universal-Streaming modes, p | --- | --- | --- | --- | | English | `universal-streaming-english` | `en` | English | | Multilingual | `universal-streaming-multilingual` | `multi` | English, French, German, Italian, Portuguese, and Spanish | -| Universal 3.5 Pro | `universal-3-5-pro` | Automatic | 18 languages with automatic language detection and code-switching — see [AssemblyAI's supported languages](https://www.assemblyai.com/docs/streaming/multilingual-transcription#supported-languages) | +| Universal 3.5 Pro | `universal-3-5-pro` | Automatic | 18 languages with automatic language detection and code-switching | + +Universal 3.5 Pro accepts the following `languageCodes` values for language steering. Omit `languageCodes` to keep automatic detection and code-switching across all supported languages. See [AssemblyAI's language steering guide](https://www.assemblyai.com/blog/fix-language-steering-streaming-transcription) for provider guidance. + +| Language | Language code | +| --- | --- | +| Arabic | `ar` | +| Danish | `da` | +| Dutch | `nl` | +| English | `en` | +| Finnish | `fi` | +| French | `fr` | +| German | `de` | +| Hebrew | `he` | +| Hindi | `hi` | +| Italian | `it` | +| Japanese | `ja` | +| Mandarin Chinese | `zh` | +| Norwegian | `no` | +| Portuguese | `pt` | +| Spanish | `es` | +| Swedish | `sv` | +| Turkish | `tr` | +| Vietnamese | `vi` | Keyterms prompting is supported with the `universal-streaming-english` and `universal-3-5-pro` models. It is not supported with `universal-streaming-multilingual`.