diff --git a/explainers/speech-recognition-result-timestamps.md b/explainers/speech-recognition-result-timestamps.md index 0200cc5..2aca9d1 100644 --- a/explainers/speech-recognition-result-timestamps.md +++ b/explainers/speech-recognition-result-timestamps.md @@ -4,122 +4,172 @@ ### Problem -The Web Speech API currently does not expose the start and end timestamps of the source audio corresponding to a given transcription result (`SpeechRecognitionResult`). This limitation creates two major challenges for API clients and end users: +The Web Speech API currently does not expose the start and end timestamps of the source audio corresponding to a given transcription result (`SpeechRecognitionResult`). This limitation creates a major challenge for API clients requiring timeline association use cases: + - **Subtitling & Closed Captions:** Web applications cannot automatically generate synchronized subtitle tracks (e.g. WebVTT / SRT cues) because they lack the exact `[startTime, endTime]` boundaries for each phrase. + - **Interactive Meeting Transcripts ("Click-to-Seek"):** In recorded meetings, video lectures, and podcast players, applications cannot offer "click-to-seek" navigation—where clicking on a sentence or word in the transcript jumps media playback to that exact moment. + - **Live WebRTC Video Sync & Lip-Sync:** In real-time video conferencing (e.g. Google Meet), web applications cannot reliably synchronize live captions or translated subtitles with incoming video frames. Because DOM events conflate speech timing with processing and main-thread queuing delays, subtitles either lag behind speaker lip movement or disappear prematurely. + - **Text-Based Media Editing:** Web-based podcast and video editors cannot allow users to cut, splice, or re-time media segments by editing transcript text without knowing the corresponding audio boundaries. -- **Timeline Association:** Developers cannot readily associate transcribed text with specific segments of the audio source, making it difficult to map generated captions to media timelines or audio tracks. -- **Latency Tracking & Backend Failover:** With the adoption of on-device Automatic Speech Recognition (ASR) to improve privacy and reduce server costs, processing performance becomes heavily dependent on local client hardware resources. The Web Speech API acts as a "black box" regarding local processing delays. Developers cannot programmatically calculate transcription latency or detect when on-device models fall behind real-time. This leads to poor user experiences (e.g. caption lag during live video conferencing) and deprives applications of the signal needed to seamlessly fail over to high-performance cloud backends. +--- ### Proposed Solution -We propose extending the `SpeechRecognitionResult` interface to include optional (nullable) `audioStartTime` and `audioEndTime` attributes. +We propose extending the `SpeechRecognitionResult` interface with `speechStartTime` and `speechEndTime` attributes. #### Web IDL Definition ```webidl partial interface SpeechRecognitionResult { - // Start timestamp of the audio segment in milliseconds (relative to the start of the audio stream) - readonly attribute DOMHighResTimeStamp? audioStartTime; + // Start timestamp of the speech segment in seconds relative to the start of the audio stream (0.0s). + readonly attribute double speechStartTime; - // End timestamp of the audio segment in milliseconds (relative to the start of the audio stream) - readonly attribute DOMHighResTimeStamp? audioEndTime; + // End timestamp of the speech segment in seconds relative to the start of the audio stream. + readonly attribute double speechEndTime; }; ``` -### Proposed Behavior & Example Usage +### Choice of Naming & Representation -The `audioStartTime` and `audioEndTime` properties represent the audio duration bounds (in milliseconds) corresponding to the transcribed segment. If the underlying recognition engine backend does not support segment timestamps, these attributes return `null`. +1. **Mirroring `speechstart` and `speechend`:** + * The names `speechStartTime` and `speechEndTime` mirror the existing `speechstart` and `speechend` events in the Web Speech API, clearly communicating that these timestamps bound the acoustic speech segment corresponding to the transcript hypothesis. -Developers can programmatically compute processing latency by comparing `audioEndTime` against the standard DOM event generation timestamp (`Event.timeStamp`): +2. **Seconds as `double` (Consistency with Adjacent Media APIs):** + * In adjacent W3C media specifications, media-local timelines are universally represented in **seconds** as a `double`: + * **Web Audio API:** [`BaseAudioContext.currentTime`](https://webaudio.github.io/web-audio-api/#dom-baseaudiocontext-currenttime) (seconds) + * **HTML Media Elements:** [`HTMLMediaElement.currentTime`](https://html.spec.whatwg.org/multipage/media.html#dom-media-currenttime) (seconds) + * **AudioParam Scheduling:** [`AudioParam.setValueAtTime()`](https://webaudio.github.io/web-audio-api/#dom-audioparam-setvalueattime) (seconds) + * Using seconds ensures seamless interoperability when routing audio between media elements, Web Audio graphs, and `SpeechRecognition`, avoiding repetitive unit conversions ($1000\times / \div 1000$). -```javascript -const recognition = new SpeechRecognition(); -recognition.continuous = true; -recognition.interimResults = true; +3. **Semantic Inaccuracy of `DOMHighResTimeStamp`:** + * Under [W3C High Resolution Time Level 3](https://www.w3.org/TR/hr-time-3/#sec-domhighrestimestamp), `DOMHighResTimeStamp` represents milliseconds relative to document uptime (`performance.timeOrigin`). Because `speechStartTime` and `speechEndTime` represent media-local stream offsets ($t = 0.0\text{s}$ at stream start), using `DOMHighResTimeStamp` would be semantically inaccurate. -recognition.onresult = (event) => { - const result = event.results[event.resultIndex]; +--- - if (result.audioEndTime !== null && result.audioEndTime !== undefined) { - // Calculate on-device processing latency - const processingLatencyMs = event.timeStamp - result.audioEndTime; +### Proposed Behavior & Examples - // Trigger seamless failover to cloud backend if latency breaches acceptable threshold - if (processingLatencyMs > 1500) { - console.warn(`ASR processing lag detected (${processingLatencyMs}ms). Transitioning to cloud provider.`); - switchToCloudBackend(); - } - } -}; +The `speechStartTime` and `speechEndTime` properties represent the exact acoustic bounds (in seconds) of the recognized speech, measured from the start of the audio stream consumed by the recognizer ($t = 0.0\text{s}$, marked by `audiostart`). -recognition.start(); -``` +#### Example 1: Automated Subtitling & "Click-to-Seek" Navigation -## Converting Stream Timestamps to Document Time Origin +In this example, speech recognition results from an audio/video element are converted into WebVTT subtitle cues and interactive "click-to-seek" transcript links: -`audioStartTime` and `audioEndTime` are defined as media-local offsets in milliseconds relative to the start of the audio stream ($t = 0.0\text{ms}$). +```javascript +const mediaElement = document.querySelector('video'); +const track = mediaElement.addTextTrack('captions', 'English', 'en'); +track.mode = 'showing'; -For real-time applications such as **live translation**, **subtitling overlays**, and **audio-visual sync**, developers often need to map these stream offsets to the document's global timeline (`DOMHighResTimeStamp` / `performance.now()`). +const recognition = new SpeechRecognition(); +recognition.continuous = true; +recognition.interimResults = false; -### Pattern: Capturing the Audio Timeline Origin +// Transcribe audio track from HTMLMediaElement +const audioStream = mediaElement.captureStream(); +recognition.start(audioStream.getAudioTracks()[0]); -To convert stream-relative timestamps to document time coordinates: -1. Record the baseline timestamp when the `audiostart` event fires (`event.timeStamp` is a `DOMHighResTimeStamp` relative to `timeOrigin`). -2. Add the result's `audioStartTime` and `audioEndTime` offsets to that baseline. +recognition.onresult = (event) => { + for (let i = event.resultIndex; i < event.results.length; ++i) { + const result = event.results[i]; + if (!result.isFinal) continue; + + const transcriptText = result[0].transcript; + + // 1. Create a synchronized WebVTT caption cue + const cue = new VTTCue(result.speechStartTime, result.speechEndTime, transcriptText); + track.addCue(cue); + + // 2. Build interactive transcript element ("Click-to-Seek") + const transcriptItem = document.createElement('p'); + transcriptItem.textContent = `[${result.speechStartTime.toFixed(1)}s] ${transcriptText}`; + transcriptItem.onclick = () => { + mediaElement.currentTime = result.speechStartTime; + mediaElement.play(); + }; + document.getElementById('transcript-container').appendChild(transcriptItem); + } +}; +``` + +--- -$$\text{absoluteStartTime} = \text{audioOrigin} + \text{result.audioStartTime}$$ -$$\text{absoluteEndTime} = \text{audioOrigin} + \text{result.audioEndTime}$$ +#### Example 2: Live Video Conferencing (Live Display + Recording Timeline Association) -### Measuring Live Translation Latency Example +In zero-delay live conferencing (such as Google Meet), captions are displayed immediately upon arrival during the live call to minimize perceived lag. Simultaneously, `speechStartTime` and `speechEndTime` are preserved in the meeting transcript so that the exported recording video track can be rendered with frame-accurate subtitle alignment: -In live speech translation workflows, measuring both **Speech-to-Text (STT) latency** and **Machine Translation (MT) end-to-end latency** is essential: +*(See the [interactive live meeting and recording demo](https://alan33d.github.io/web-speech-demos/live_meeting_recording_demo.html) for a working implementation of this pattern).* ```javascript const recognition = new SpeechRecognition(); recognition.continuous = true; recognition.interimResults = true; -let audioOriginTime = 0; +// 1. Live Call: Display captions immediately as results arrive +let hideTimeout = null; +const liveSubtitleOverlay = document.getElementById('live-captions'); -// 1. Capture the audio stream's time origin on the document timeline -recognition.onaudiostart = (event) => { - audioOriginTime = event.timeStamp; -}; +// 2. Meeting Transcript: Archive finalized results with exact audio boundaries +const meetingTranscript = []; -recognition.onresult = async (event) => { +recognition.onresult = (event) => { const result = event.results[event.resultIndex]; - if (result.audioEndTime === null || result.audioEndTime === undefined) return; - - // 2. Convert stream offsets to document time origin coordinates - const absoluteAudioStart = audioOriginTime + result.audioStartTime; - const absoluteAudioEnd = audioOriginTime + result.audioEndTime; - - // 3. Compute ASR recognition latency - const asrLatencyMs = event.timeStamp - absoluteAudioEnd; - - // 4. Perform live translation - const text = result[0].transcript; - const translationStartTime = performance.now(); - const translatedText = await translateService.translate(text, 'es'); - const translationEndTime = performance.now(); + const transcriptText = result[0].transcript; + + // Render live captions immediately on screen + liveSubtitleOverlay.textContent = transcriptText; + liveSubtitleOverlay.classList.toggle('interim', !result.isFinal); + + // Keep final captions on screen for a 3-second reading persistence window + clearTimeout(hideTimeout); + if (result.isFinal) { + hideTimeout = setTimeout(() => { + liveSubtitleOverlay.textContent = ''; + }, 3000); + + // Archive the finalized sentence with exact acoustic timestamps + meetingTranscript.push({ + text: transcriptText, + startTime: result.speechStartTime, + endTime: result.speechEndTime + }); + } +}; - // 5. Total end-to-end latency from speaker utterance to translated subtitle - const totalE2ELatencyMs = translationEndTime - absoluteAudioEnd; +// 3. Post-Meeting Replay: Synchronize caption cues with the recorded meeting video +function initializeRecordingCaptions(recordedVideoElement) { + const captionTrack = recordedVideoElement.addTextTrack('captions', 'Meeting Captions'); + captionTrack.mode = 'showing'; - console.log(`ASR Processing Time: ${asrLatencyMs.toFixed(1)}ms`); - console.log(`Total Live Translation Delay: ${totalE2ELatencyMs.toFixed(1)}ms`); -}; + for (const entry of meetingTranscript) { + captionTrack.addCue(new VTTCue(entry.startTime, entry.endTime, entry.text)); + } +} ``` + --- + ### Security and Privacy Considerations #### Fingerprinting Risk -Exposing sub-millisecond or precise micro-architectural timing information enables hardware profiling (measuring CPU execution speed, thermal throttling, and system load), creating a tracking vector for cross-origin user fingerprinting. +Exposing sub-millisecond or precise micro-architectural timing information enables hardware profiling (measuring CPU execution speed, thermal throttling, and system load), creating a potential tracking vector for cross-origin user fingerprinting. #### Mitigation Strategy -To mitigate fingerprinting vectors, browser implementations MUST apply timestamp fuzzing and precision reduction before exposing timing attributes to web scripts. We propose mirroring the precision capping strategy used in [`HTMLMediaElement.currentTime`](https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/currentTime), rounding exposed timestamps to **2ms** precision (or matching the site-wide timer resolution policy). +To mitigate potential side-channel and fingerprinting vectors: +* **Precedent:** Implementations (such as Chromium) follow the security posture established by [`HTMLMediaElement.currentTime`](https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/currentTime), coarsening raw engine timestamps to discrete resolution buckets (e.g. 2ms resolution) in non-isolated execution contexts. +* **Specification Precedent:** In accordance with [W3C High Resolution Time Level 3](https://www.w3.org/TR/hr-time-3/#privacy-security), the specification does not mandate a hardcoded quantization value, allowing user agents to adjust timer resolution or introduce jitter according to their security and cross-origin isolation policies. + +--- ### Alternatives Considered -- **Browser-Generated Warning Events (`onprocessinglag`):** Simple for web applications to catch, but fails to accommodate varying latency thresholds across different use cases (e.g. real-time meeting captioning requires <200ms latency, while dictation tools tolerate multi-second delays). -- **Internal Processing Queue Metric (`queueDepth`):** Directly exposes engine backlogs, but is difficult to standardize across fragmented engine architectures, model types, and buffering strategies. -- **Binary Status Flag (`isRealTime`):** Simple boolean check, but lacks numerical precision for applications seeking to track progressive latency degradation trendlines. +- **Splitting `result` into `resultstart` and `resultend` Events:** + Splitting the single `result` event into separate `resultstart` and `resultend` DOM events (each carrying individual `event.timeStamp` values) was considered. However, localizing timestamps directly within `SpeechRecognitionResult` and `SpeechRecognitionResultList` is architecturally superior: + 1. **Fragile Client State Management:** Requiring developers to correlate separate, asynchronous start and end events across streaming interim updates introduces a complex, race-prone state machine into web applications. + 2. **Decoupling from Cumulative Results (`SpeechRecognitionResultList`):** In continuous recognition mode (`continuous = true`), the API maintains a cumulative list of all results generated during the session. If timing information is only emitted on transient events, historical results stored in `event.results` would have no localized timestamps. Developers would be forced to manually maintain a secondary timeline mapping in application code. + 3. **Unnecessary Event Overhead for Limited Utility:** Adding yet another pair of events that client applications must process adds event-loop overhead and API surface complexity with limited practical use cases compared to self-contained result properties. + 4. **Self-Contained Data Model:** Attaching `speechStartTime` and `speechEndTime` directly to `SpeechRecognitionResult` ensures that each transcript hypothesis is self-contained and atomically bound to its exact temporal audio segment, allowing results to be passed, cached, and manipulated independently of DOM event lifecycles. + +- **Using Existing VAD (`speechstart` / `speechend`) and `result` Event Timestamps:** + Relying on existing Voice Activity Detection (VAD) events (`speechstart` / `speechend`) and `result` event timestamps was considered. While `speechend.timeStamp` (or the proposed start and end timestamps on `SpeechRecognitionResult`) could potentially be used for basic processing latency tracking, existing events are fundamentally insufficient for timeline association: + 1. **Cannot Capture Start Times in Continuous Recognition:** Neither VAD events nor `result` events capture the acoustic start timestamp of interim or final results that occur in the middle of a continuous recognition session. In continuous recognition mode (`continuous = true`), `speechstart` fires only once at the beginning of the entire session, leaving all subsequent phrases without an acoustic start boundary. + 2. **No Start Timestamp on `speechend` or `result` Events:** `speechend` only marks when speech activity ended, and `result.timeStamp` only indicates when the DOM event was dispatched. Neither provides the acoustic start time needed to compute utterance duration or construct `[startTime, endTime]` subtitle cues. + 3. **No Support for Interim Results:** While a speaker is actively talking mid-sentence, `speechend` cannot fire, leaving streaming captions without timing information. + 4. **Endpointer Trailing Silence Skew:** Voice Activity Detection (VAD) endpointers only fire `speechend` after observing 500ms–1500ms of trailing silence. This introduces non-speech padding into the timestamp, degrading audio alignment.