Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 48 additions & 5 deletions fern/customization/custom-keywords.mdx
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
---
title: Deepgram Keywords and Keyterm Prompting
title: Keywords and keyterm prompting
subtitle: Boost STT accuracy for domain words and phrases
slug: customization/custom-keywords
---


Vapi allows you to improve the accuracy of your transcriptions by leveraging Deepgram's keyword boosting feature. This is particularly useful when dealing with specialized terminology or uncommon proper nouns. By providing specific keywords to the Deepgram model, you can enhance transcription quality directly through Vapi.
Vapi allows you to improve the accuracy of your transcriptions by leveraging keyword boosting and keyterm prompting. This is particularly useful when dealing with specialized terminology or uncommon proper nouns. Both [Deepgram](#deepgram-keywords-and-keyterm-prompting) and [AssemblyAI](#assemblyai-keyterms-prompting) transcribers support this through Vapi.

## Deepgram keywords and keyterm prompting

### Why Use Keyword Boosting?

Expand All @@ -26,9 +28,9 @@ Keyword boosting is beneficial for:
- Keywords is available on Deepgram Nova-2, Nova-1, Enhanced, and Base speech-to-text models.
- For Nova-3 models, use Keyterm Prompting instead of Keywords.

## Enabling Keyword Boosting in Vapi
### Enabling Keyword Boosting in Vapi

### API Call Integration
#### API Call Integration

To enable keyword boosting, add the `keywords` parameter to your assistant's `transcriber` configuration when using the Deepgram provider. You can also supply `keyterm` to boost recall for phrases.

Expand Down Expand Up @@ -118,4 +120,45 @@ For more details, see:
- Deepgram Keyterm Prompting: [developers.deepgram.com/docs/keyterm](https://developers.deepgram.com/docs/keyterm)
- API reference: Deepgram transcriber `keywords` and `keyterm` in the [API reference](https://api.vapi.ai/api#:~:text=DeepgramTranscriber)

By following these guidelines, you can effectively utilize Deepgram's keyword boosting feature within your Vapi assistant, ensuring enhanced transcription accuracy for specialized terminology and uncommon proper nouns.
By following these guidelines, you can effectively utilize Deepgram's keyword boosting feature within your Vapi assistant, ensuring enhanced transcription accuracy for specialized terminology and uncommon proper nouns.

## AssemblyAI keyterms prompting

AssemblyAI's Universal-Streaming keyterms prompting boosts recognition of domain-specific words and phrases. Add the `keytermsPrompt` parameter to your assistant's `transcriber` configuration when using the `assembly-ai` provider.

- Up to 100 keyterms per session, each up to 50 characters.
- Keyterms can be single words or multi-word phrases — no intensifiers needed.
- Supported with the `universal-streaming-english` and `universal-3-5-pro` speech models. Not supported with `universal-streaming-multilingual`.
- Keyterms prompting adds $0.04/hour to transcription cost on `universal-streaming-english`.

### Example

```bash
curl \
--request POST \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data '{
"name": "Emma",
"model": {
"model": "gpt-4o",
"provider": "openai"
},
"transcriber": {
"provider": "assembly-ai",
"speechModel": "universal-streaming-english",
"keytermsPrompt": [
"order number",
"account ID",
"PCI compliance"
]
},
"firstMessage": "Hi, I am Emma, what is your name?"
}' \
https://api.vapi.ai/assistant
```

### Additional resources

- AssemblyAI Keyterms Prompting: [AssemblyAI's prompting and keyterms guide](https://www.assemblyai.com/docs/streaming/prompting-and-keyterms)
- API reference: [`AssemblyAITranscriber` fields](/api-reference/assistants/create#request.body.transcriber.AssemblyAITranscriber)
4 changes: 2 additions & 2 deletions fern/customization/multilingual.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -461,7 +461,7 @@ Validate your configuration with different languages and scenarios.
| **Deepgram** | ✅ Full auto-detection | 100+ | **Recommended**: Nova 2/Nova 3 with "Multi" language setting |
| **Google STT** | ✅ Full auto-detection | 125+ | Latest models with "Multilingual" language setting |
| **Gladia** | ✅ Full auto-detection | 110+ | Supports automatic language detection and code-switching |
| **Assembly AI** | ❌ English only | English | No multilingual support |
| **AssemblyAI** | ✅ Full auto-detection | 18 | Set `speechModel` to `universal-3-5-pro` for automatic language detection and code-switching |
| **Azure STT** | ❌ Single language | 100+ | Many languages, but no auto-detection |
| **OpenAI Whisper** | ❌ Single language | 90+ | Many languages, but no auto-detection |
| **Speechmatics** | ❌ Single language | 50+ | Many languages, but no auto-detection |
Expand All @@ -482,7 +482,7 @@ Validate your configuration with different languages and scenarios.
<AccordionGroup>
<Accordion title="Language detection is inaccurate">
**Solutions:**
- Use Deepgram (Nova 2/Nova 3 with "Multi"), Google STT (with "Multilingual"), or Gladia (automatic language detection)
- Use AssemblyAI Universal 3.5 Pro (`speechModel: "universal-3-5-pro"`), Deepgram (Nova 2/Nova 3 with "Multi"), Google STT (with "Multilingual"), or Gladia (automatic language detection)
- Ensure high-quality audio input for better detection accuracy
- Test with native speakers of target languages
- Consider provider-specific language combinations for optimal results
Expand Down
4 changes: 2 additions & 2 deletions fern/customization/voice-pipeline-configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ Uses AI models to analyze speech patterns, context, and audio cues to predict wh

**Audio-text based providers:**
- **deepgram-flux**: Deepgram's latest transcriber model with built-in conversational speech recognition. Use `flux-general-en` for English-only conversations or `flux-general-multi` for multilingual conversations.
- **assembly**: Transcriber with built-in end-of-turn detection (English only)
- **assembly**: Transcriber with built-in end-of-turn detection for Universal-Streaming English, Universal-Streaming Multilingual, and Universal 3.5 Pro

<hr />
</Tab>
Expand All @@ -232,7 +232,7 @@ Uses AI models to analyze speech patterns, context, and audio cues to predict wh
**When to use smart endpointing:**

- **Deepgram Flux**: English and Multi-lingual conversations using Deepgram as a transcriber.
- **Assembly**: Best used when Assembly is already your transcriber provider for English conversations with integrated end-of-turn detection
- **AssemblyAI**: Use when AssemblyAI is already your transcriber provider and you want integrated end-of-turn detection
- **LiveKit**: English conversations where Deepgram is not the transcriber of choice.
- **Vapi**: Non-English conversations with default stop speaking plan settings
- **Krisp**: Non-English conversations with a robustly configured stop speaking plan
Expand Down
68 changes: 62 additions & 6 deletions fern/providers/transcriber/assembly-ai.mdx
Original file line number Diff line number Diff line change
@@ -1,17 +1,25 @@
---
title: AssemblyAI
subtitle: Use AssemblyAI Universal-Streaming for real-time speech-to-text in Vapi.
description: Configure AssemblyAI Universal-Streaming as your Vapi assistant's transcriber through the dashboard or API.
subtitle: Use AssemblyAI for real-time speech-to-text in Vapi.
description: Configure AssemblyAI speech models, multilingual transcription, language steering, and prompting for a Vapi assistant.
slug: providers/transcriber/assembly-ai
---

## Universal-Streaming

Universal-Streaming is AssemblyAI's purpose-built speech-to-text model that delivers ultra-fast, immutable transcripts in ~300ms with intelligent endpointing and superior accuracy for voice agents. It eliminates common pain points like misheard account numbers, awkward pauses, and premature cutoffs, enabling more natural and successful voice interactions.

AssemblyAI provides real-time speech-to-text and turn detection for Vapi voice agents.

## Supported capability

| Capability | Provider value |
| --- | --- |
| Speech-to-text | `assembly-ai` |

## Speech-to-text

Set `transcriber.provider` to `assembly-ai`. AssemblyAI's Universal-Streaming model is selected automatically, so no `model` field is required.
Set `transcriber.provider` to `assembly-ai`. By default the `universal-streaming-english` speech model is used; set `transcriber.speechModel` to choose a different model.

```bash
curl -X PATCH "https://api.vapi.ai/assistant/ASSISTANT_ID" \
Expand All @@ -27,6 +35,30 @@ curl -X PATCH "https://api.vapi.ai/assistant/ASSISTANT_ID" \

For additional API configuration options, review the [`AssemblyAITranscriber` fields](/api-reference/assistants/create#request.body.transcriber.AssemblyAITranscriber) in the Create Assistant API reference.

### Universal 3.5 Pro

Universal 3.5 Pro (`universal-3-5-pro`) is AssemblyAI's most accurate voice-agent model, with automatic language detection and code-switching across 18 languages.

```bash
curl -X PATCH "https://api.vapi.ai/assistant/ASSISTANT_ID" \
-H "Authorization: Bearer $VAPI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"transcriber": {
"provider": "assembly-ai",
"speechModel": "universal-3-5-pro"
}
}'
```

Universal 3.5 Pro supports additional configuration:

- `mode`: Trade off accuracy and latency with `max_accuracy`, `balanced`, or `min_latency`.
- `prompt`: Provide additional context for the transcription model, up to 1,750 characters.
- `agentContext`: Provide context about the assistant that helps the transcription model interpret caller speech, up to 1,750 characters.
- `languageCodes`: Steer automatic language detection with language codes such as `["en", "es"]`.
- `keytermsPrompt`: Boost recognition of domain-specific terms.

### Configure in the dashboard

This guide details how to setup AssemblyAI as a transcriber for your assistant.
Expand Down Expand Up @@ -54,14 +86,38 @@ This guide details how to setup AssemblyAI as a transcriber for your assistant.

### Supported languages

Vapi supports AssemblyAI's English and multilingual Universal-Streaming modes.
Vapi supports AssemblyAI's English and multilingual Universal-Streaming modes, plus the Universal 3.5 Pro model.

| Mode | Speech model | Language value | Supported languages |
| --- | --- | --- | --- |
| English | `universal-streaming-english` | `en` | English |
| Multilingual | `universal-streaming-multilingual` | `multi` | English, French, German, Italian, Portuguese, and Spanish |

Keyterms prompting is not supported with the multilingual model.
| Universal 3.5 Pro | `universal-3-5-pro` | Automatic | 18 languages with automatic language detection and code-switching |

Universal 3.5 Pro accepts the following `languageCodes` values for language steering. Omit `languageCodes` to keep automatic detection and code-switching across all supported languages. See [AssemblyAI's language steering guide](https://www.assemblyai.com/blog/fix-language-steering-streaming-transcription) for provider guidance.

| Language | Language code |
| --- | --- |
| Arabic | `ar` |
| Danish | `da` |
| Dutch | `nl` |
| English | `en` |
| Finnish | `fi` |
| French | `fr` |
| German | `de` |
| Hebrew | `he` |
| Hindi | `hi` |
| Italian | `it` |
| Japanese | `ja` |
| Mandarin Chinese | `zh` |
| Norwegian | `no` |
| Portuguese | `pt` |
| Spanish | `es` |
| Swedish | `sv` |
| Turkish | `tr` |
| Vietnamese | `vi` |

Keyterms prompting is supported with the `universal-streaming-english` and `universal-3-5-pro` models. It is not supported with `universal-streaming-multilingual`.

## Related

Expand Down
Loading