diff --git a/spellcheck/dictionary.txt b/spellcheck/dictionary.txt index 17b18320a9ec..2a7f01f375e8 100644 --- a/spellcheck/dictionary.txt +++ b/spellcheck/dictionary.txt @@ -628,6 +628,8 @@ polyfill polyfills polymorphism postfix +postprocessing +Postprocessing postroll postrolls Pre diff --git a/theolive/contribution/enhancements/speech.mdx b/theolive/contribution/enhancements/speech.mdx index ddc73aa9076b..fb61f063fa78 100644 --- a/theolive/contribution/enhancements/speech.mdx +++ b/theolive/contribution/enhancements/speech.mdx @@ -18,7 +18,7 @@ As long as the feature is not enabled on your account, requests that configure a ## How it works -You enable the speech enhancement on an **ingest** by adding an entry with `"kind": "speech"` to its `enhancements` list. You tell us which language is spoken in the stream (`sourceLanguage`) and which languages you want translated captions in (`targetTextTracks`). The audio is transcribed, translated into the requested target languages and injected back into the stream as text tracks, aligned with what is being said. Optionally, a **dictionary** with domain-specific terms can be attached to improve the speech recognition. +You enable the speech enhancement on an **ingest** by adding an entry with `"kind": "speech"` to its `enhancements` list. You tell us which language is spoken in the stream (`sourceLanguage`) and which languages you want translated captions in (`targetTextTracks`). The audio is transcribed, translated into the requested target languages and injected back into the stream as text tracks, aligned with what is being said. Optionally, a **speech dictionary** with domain-specific terms can be attached to improve the speech recognition, a **postprocessing dictionary** with find/replace rules can correct the transcript before it is translated, and **speaker changes** can be marked in the text tracks. ### What your viewers get @@ -28,7 +28,7 @@ You enable the speech enhancement on an **ingest** by adding an entry with `"kin ### Dashboard -You don't need the API to set this up: the ingest form in the dashboard has a **Speech** toggle where you can pick the source language, target text tracks, AI context, latency mode, caption position and dictionary. +You don't need the API to set this up: the ingest form in the dashboard has a **Speech** toggle where you can pick the source language, target text tracks, AI context, latency mode, caption position, dictionary, postprocessing dictionary and speaker change detection. Both kinds of dictionaries are managed on the **Streaming > Enhancements > Speech** page. ## Quick start @@ -36,8 +36,9 @@ You don't need the API to set this up: the ingest form in the dashboard has a ** 2. Look up the supported languages: - [`GET /v2/enhancements/speech/supported-source-languages`](../../api/get-supported-source-languages.api.mdx) - [`GET /v2/enhancements/speech/supported-target-languages`](../../api/get-supported-target-languages.api.mdx) -3. (Optional) Create a dictionary with domain-specific terms: [`POST /v2/enhancements/speech/dictionaries`](../../api/create-speech-dictionary.api.mdx). -4. Add a `speech` entry to the `enhancements` list when creating or updating an ingest. +3. (Optional) Create a speech dictionary with domain-specific terms: [`POST /v2/enhancements/speech/dictionaries`](../../api/create-speech-dictionary.api.mdx). +4. (Optional) Create a postprocessing dictionary with find/replace rules for the transcript: `POST /v2/enhancements/speech/postprocessing-dictionaries`. +5. Add a `speech` entry to the `enhancements` list when creating or updating an ingest. ## Enabling the speech enhancement on an ingest @@ -59,7 +60,9 @@ The speech enhancement is configured per ingest, either when creating the ingest "latencyMode": "fast", "extraPrompt": "this is a live horse race", "position": { "bottom": 10 }, - "dictionary": "my-dictionary-id" + "dictionary": "my-dictionary-id", + "postprocessingDictionary": "my-postprocessing-dictionary-id", + "diarization": { "enabled": true, "speakerLabel": ">> " } } ] } @@ -71,21 +74,23 @@ Use a `region` id as returned by [`GET /v2/regions`](../../api/get-regions.api.m The full schema is documented in the API reference (`SpeechEnhancement` on [create ingest](../../api/create-channel-ingest.api.mdx) and [update ingest](../../api/update-ingest.api.mdx)). In short: -| Field | Type | Required | Description | -| ------------------ | ---------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `sourceLanguage` | `string` | yes | Language spoken in the stream (must be a supported source language) | -| `targetTextTracks` | `string[]` | yes | Languages to translate into (1 to **5**, must be supported target languages). A caption track in the source language is always included, so don't list the source language or a variant of it (e.g. `en-US` for `en`) | -| `sourceAudioTrack` | `number` | no | PID of the audio track to use as translation source (non-negative integer) | -| `dictionary` | `string` | no | ID of a dictionary that helps the speech recognition. The dictionary's `language` must match the enhancement's `sourceLanguage`. Set to `null` to disconnect the dictionary | -| `latencyMode` | `string` | no | Trade-off between caption delay and translation quality: `fastest`, `fast` (default), `balanced` or `highest`. `highest` waits longest for the best result | -| `extraPrompt` | `string` | no | Extra context for the AI, applied to every translation (at most 1000 characters), e.g. `"this is a live horse race"` or `"remove profane language from the output"`. Set to `null` to remove it | -| `position` | `object` | no | Position of the rendered captions. `position.bottom` is an integer from 0 to 90: the margin from the bottom of the viewport, in percent of the viewport height | +| Field | Type | Required | Description | +| -------------------------- | ---------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `sourceLanguage` | `string` | yes | Language spoken in the stream (must be a supported source language) | +| `targetTextTracks` | `string[]` | yes | Languages to translate into (1 to **5**, must be supported target languages). A caption track in the source language is always included, so don't list the source language or a variant of it (e.g. `en-US` for `en`) | +| `sourceAudioTrack` | `number` | no | PID of the audio track to use as translation source (non-negative integer) | +| `dictionary` | `string` | no | ID of a [speech dictionary](#speech-dictionaries) that helps the speech recognition. The dictionary's `language` must match the enhancement's `sourceLanguage`. Set to `null` to disconnect the dictionary | +| `postprocessingDictionary` | `string` | no | ID of a [postprocessing dictionary](#postprocessing-dictionaries) whose find/replace rules are applied to the transcript before translation. Its `language` must match the enhancement's `sourceLanguage`. Set to `null` to disconnect it | +| `latencyMode` | `string` | no | Trade-off between caption delay and translation quality: `fastest`, `fast` (default), `balanced` or `highest`. `highest` buffers the output the longest (tens of seconds) and gives the best transcript and translations | +| `extraPrompt` | `string` | no | Extra context for the AI, applied to every translation (at most 1000 characters), e.g. `"this is a live horse race"` or `"remove profane language from the output"`. Set to `null` to remove it | +| `diarization` | `object` | no | [Speaker change detection](#speaker-changes-diarization) in the text tracks: `enabled` (`boolean`, default `false`) and `speakerLabel` (`string` of at most 10 characters, default `">> "`). Missing keys take their default | +| `position` | `object` | no | Position of the rendered captions. `position.bottom` is an integer from 0 to 90: the margin from the bottom of the viewport, in percent of the viewport height | ### Updating and disabling - On update, the `enhancements` list is replaced as a whole: always send the full list you want to keep. - To disable the speech enhancement, update the ingest with an `enhancements` list without a `speech` entry (e.g. `[]`). -- To disconnect a dictionary, resend the `speech` entry without the `dictionary` field (or with `"dictionary": null`). +- To disconnect a dictionary, resend the `speech` entry without the `dictionary` field (or with `"dictionary": null`). The same goes for `postprocessingDictionary`. - `captions` and `enhancements` can't be changed while an engine of the ingest is running: the update fails with `409 engine_playing`. Stop the channel (or its engines) first. ### Connection details @@ -123,6 +128,15 @@ Both return a simple list: ## Dictionaries +Two kinds of dictionaries help you get the source-language transcript right. Since the transcript is the input of every translation, a correction made here carries over into all target text tracks: + +- A [**speech dictionary**](#speech-dictionaries) is a list of terms the speech recognition keeps an eye out for while it listens. It is a hint: it makes the right spelling more likely but doesn't guarantee it. +- A [**postprocessing dictionary**](#postprocessing-dictionaries) is a list of find/replace rules applied to the transcript _after_ speech recognition and _before_ translation. Rules are applied reliably, every time they match. + +Use them together for the best results: the speech dictionary to help the recognition hear the right term, the postprocessing dictionary to fix what still comes out wrong, to standardize formatting or to remove unwanted words. + +### Speech dictionaries + A speech dictionary is a **flat list of terms** — names, proper nouns, acronyms and specialized terminology — that helps the speech recognition correctly caption the source audio of a speech enhancement. It does **not** translate terms: the source audio is first captioned in the source language (aided by the dictionary) and then translated into the target languages. Connect a dictionary to an ingest via the `dictionary` field of the speech enhancement entry; the dictionary's `language` must match that enhancement's `sourceLanguage`. @@ -145,21 +159,81 @@ Example create request: } ``` -### Writing good dictionary terms +#### Writing good dictionary terms - Keep terms short: each entry is a single term of at most **6 words**. One- to three-word phrases give the best results. - Add terms the way they are actually spoken. An entry like `"Initial Public Offering (IPO)"` won't match anything — a speaker says either the full phrase or just "IPO", so add those as two separate terms. - When a term is pronounced differently than its spelling suggests, append one or more **phonetic "sounds like" hints** after a pipe, separated by commas: `"D&I|dee-and-eye,dee-n-eye"`. - Terms containing symbols, digits or punctuation (hashtags, abbreviations, hyphenated words) should always get a phonetic hint, since their spelling alone gives the speech recognition little to work with. -- Dictionary terms are hints, not overrides: when the speech recognition is confident it heard a different word, it captions that instead. +- Dictionary terms are hints, not overrides: when the speech recognition is confident it heard a different word, it captions that instead. Use a [postprocessing dictionary](#postprocessing-dictionaries) when a term must always come out a certain way. -### Dictionary rules +#### Dictionary rules - `language` must be a valid IETF BCP 47 language tag (e.g. `"en"`, `"en-US"`) and must match the `sourceLanguage` of the speech enhancement it is connected to. - Terms are cleaned up automatically where possible: whitespace is normalized and empty and duplicate entries are dropped. Terms that cannot be fixed automatically are rejected with `invalid_value`, describing the offending entry: terms containing the forbidden characters `;` or `/`, terms with more than 6 words, more than one `|` separator, or a missing term before the `|`. - On update, the `terms` list is replaced as a whole. - A dictionary cannot be deleted while it is connected to an ingest that is in use by a running engine; the delete fails with `409 speech_dictionary_in_use`. Once deleted, the dictionary is automatically disconnected from all ingests that referenced it. +### Postprocessing dictionaries + +A postprocessing dictionary is a list of **find/replace rules** applied to the source-language transcript after speech recognition and before translation. Typical uses: fixing recurring misspellings of names, standardizing formatting, removing filler words, and censoring or replacing profanity. + +Connect a postprocessing dictionary to an ingest via the `postprocessingDictionary` field of the speech enhancement entry; its `language` must match that enhancement's `sourceLanguage`. + +| Method | Path | Description | +| -------- | ---------------------------------------------------------- | -------------------------------------------------------------------------- | +| `GET` | `/v2/enhancements/speech/postprocessing-dictionaries` | List dictionaries (supports `cursor`, `limit` and `name` query parameters) | +| `POST` | `/v2/enhancements/speech/postprocessing-dictionaries` | Create a dictionary | +| `GET` | `/v2/enhancements/speech/postprocessing-dictionaries/{id}` | Get a single dictionary | +| `PATCH` | `/v2/enhancements/speech/postprocessing-dictionaries/{id}` | Update a dictionary | +| `DELETE` | `/v2/enhancements/speech/postprocessing-dictionaries/{id}` | Delete a dictionary | + +Example create request: + +```json +{ + "name": "news-desk", + "language": "en", + "terms": ["OptiView|[O|o]ptiview,Opti-view", "New York|NewYork,Newyork", "|basically,literally", "d***|damn"] +} +``` + +#### Rule format + +Every entry is one rule of the form `replacement|pattern,pattern`. It looks like a speech dictionary term, but the meaning is different: + +- Everything after the **first** `|` is a comma-separated list of **patterns**. Every word of the transcript that matches one of the patterns is replaced. The `|` and at least one pattern are mandatory. +- A pattern matches an **entire word** and cannot contain spaces. Regular expressions are supported, e.g. `[O|o]ptiview`; a `|` inside a pattern is fine since only the first `|` separates the replacement. Because patterns are comma-separated, a pattern cannot contain a comma. +- The **replacement** (before the `|`) may contain spaces (`New York|NewYork`) and may be empty to remove the matched word altogether (`|basically,literally`). +- A dictionary holds at most **1000** rules. Very large dictionaries or complex regular expressions can delay or break the caption output: test a dictionary with real content before using it on a production stream. + +#### Postprocessing dictionary rules + +- `language` must be a valid IETF BCP 47 language tag and must match the `sourceLanguage` of the speech enhancement it is connected to. +- Rules are cleaned up automatically where possible: whitespace is normalized, spaces around `|` and `,` are removed, and empty and duplicate entries are dropped. Rules that cannot be fixed are rejected with `invalid_value`, describing the offending entry: a missing `|` separator or pattern, a space inside a pattern, more than 1000 rules, or a rule the captioning service rejects (e.g. an invalid regular expression; the per-rule messages are included in the error). +- On update, the `terms` list is replaced as a whole. +- A postprocessing dictionary cannot be deleted while it is connected to an ingest that is in use by a running engine; the delete fails with `409 speech_postprocessing_dictionary_in_use`. Once deleted, the dictionary is automatically disconnected from all ingests that referenced it. + +## Speaker changes (diarization) + +When several people talk in the stream, the text tracks can mark where a new speaker takes over. Enable it with the `diarization` object of the speech enhancement: + +```json +{ + "kind": "speech", + "sourceLanguage": "en", + "targetTextTracks": ["fr"], + "diarization": { "enabled": true, "speakerLabel": "- " } +} +``` + +| Field | Type | Default | Description | +| -------------- | --------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `enabled` | `boolean` | `false` | Whether speaker changes are detected and marked in the text tracks. Enabling it can slightly increase the caption latency | +| `speakerLabel` | `string` | `">> "` | Text placed in front of a caption line that starts a new speaker (at most 10 characters), e.g. `">> "` or `"- "`. An empty string marks the change with a new line only | + +Speakers are not identified by name: every change of speaker starts a new caption line prefixed with the label. The label is applied to the source-language track and to every translated track. Keys you leave out take their default, so `{ "enabled": true }` is enough to get the `">> "` label. + ## Usage Usage of the speech enhancement is reported in **minutes**: captioned minutes (transcription of the source audio) and translated minutes (one minute per target language per minute of audio). Use these to reconcile what you are billed: @@ -184,12 +258,14 @@ All of these take `start` and `end` dates (`YYYY-MM-DD`, interpreted as UTC) and ## Errors -| Code | Meaning | -| --------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `translations_not_enabled` | Live captions & translations is not enabled on your account — contact your account team | -| `speech_enhancement_not_supported` | The ingest's `region` does not support speech enhancements | -| `invalid_value` | Invalid configuration (wrong ingest type, missing `region`, missing/unsupported languages, source language in `targetTextTracks`, invalid dictionary terms) | -| `too_many_values` | More than 5 target text tracks were provided | -| `engine_playing` (409) | `captions` or `enhancements` were changed while an engine of the ingest is running | -| `speech_dictionary_in_use` (409) | The dictionary is connected to an ingest with a running engine and cannot be deleted | -| `speech_dictionary_create_failed` / `speech_dictionary_update_failed` / `speech_dictionary_delete_failed` | The dictionary could not be synchronized with the captioning provider — retry, or contact support if the problem persists | +| Code | Meaning | +| ------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `translations_not_enabled` | Live captions & translations is not enabled on your account — contact your account team | +| `speech_enhancement_not_supported` | The ingest's `region` does not support speech enhancements | +| `invalid_value` | Invalid configuration (wrong ingest type, missing `region`, missing/unsupported languages, source language in `targetTextTracks`, invalid dictionary terms or postprocessing rules, dictionary language not matching `sourceLanguage`) | +| `too_many_values` | More than 5 target text tracks were provided | +| `engine_playing` (409) | `captions` or `enhancements` were changed while an engine of the ingest is running | +| `speech_dictionary_in_use` (409) | The speech dictionary is connected to an ingest with a running engine and cannot be deleted | +| `speech_dictionary_create_failed` / `speech_dictionary_update_failed` / `speech_dictionary_delete_failed` | The speech dictionary could not be synchronized with the captioning provider — retry, or contact support if the problem persists | +| `speech_postprocessing_dictionary_in_use` (409) | The postprocessing dictionary is connected to an ingest with a running engine and cannot be deleted | +| `speech_postprocessing_dictionary_create_failed` / `speech_postprocessing_dictionary_update_failed` / `speech_postprocessing_dictionary_delete_failed` | The postprocessing dictionary could not be synchronized with the captioning provider — retry, or contact support if the problem persists |