diff --git a/explainers/speech-recognition-result-timestamps.md b/explainers/speech-recognition-result-timestamps.md index ff85d63..b9a260e 100644 --- a/explainers/speech-recognition-result-timestamps.md +++ b/explainers/speech-recognition-result-timestamps.md @@ -4,60 +4,67 @@ ### Problem -The Web Speech API currently does not expose the start and end timestamps of the source audio corresponding to a given transcription result (`SpeechRecognitionResult`). This limitation creates two major challenges for API clients and end users: +The Web Speech API currently does not expose the start and end timestamps of the source audio corresponding to a given transcription result (`SpeechRecognitionResult`). This limitation creates a major challenge for API clients requiring timeline association use cases: + - **Subtitling & Closed Captions:** Web applications cannot automatically generate synchronized subtitle tracks (e.g. WebVTT / SRT cues) because they lack the exact `[startTime, endTime]` boundaries for each phrase. + - **Interactive Meeting Transcripts ("Click-to-Seek"):** In recorded meetings, video lectures, and podcast players, applications cannot offer "click-to-seek" navigation—where clicking on a sentence or phrase in the transcript jumps media playback to that exact moment. + - **Live WebRTC Video Sync & Lip-Sync:** In real-time video conferencing (e.g. Google Meet), web applications cannot reliably synchronize live captions or translated subtitles with incoming video frames. Because DOM events conflate speech timing with processing and main-thread queuing delays, subtitles either lag behind speaker lip movement or disappear prematurely. + - **Text-Based Media Editing:** Web-based podcast and video editors cannot allow users to cut, splice, or re-time media segments by editing transcript text without knowing the corresponding audio boundaries. -- **Timeline Association:** Developers cannot readily associate transcribed text with specific segments of the audio source, making it difficult to map generated captions to media timelines, audio tracks, or video frames. -- **Latency Tracking & Backend Failover:** With the adoption of on-device Automatic Speech Recognition (ASR) to improve privacy and reduce server costs, processing performance becomes heavily dependent on local client hardware resources. The Web Speech API acts as a "black box" regarding local processing delays. Developers cannot programmatically calculate transcription latency or detect when on-device models fall behind real-time. This leads to poor user experiences (e.g. caption lag during live video conferencing) and deprives applications of the signal needed to seamlessly fail over to high-performance cloud backends. +--- ### Proposed Solution -We propose extending the `SpeechRecognitionResult` interface to include `audioStartTime` and `audioEndTime` attributes. +We propose extending the `SpeechRecognitionResult` interface with `speechStartTime` and `speechEndTime` attributes. #### Web IDL Definition ```webidl partial interface SpeechRecognitionResult { - // Start timestamp of the audio segment in seconds relative to the start of the audio stream (0.0s). - readonly attribute double audioStartTime; + // Start timestamp of the speech segment in seconds relative to the start of the audio stream (0.0s). + readonly attribute double speechStartTime; - // End timestamp of the audio segment in seconds relative to the start of the audio stream. - readonly attribute double audioEndTime; + // End timestamp of the speech segment in seconds relative to the start of the audio stream. + readonly attribute double speechEndTime; }; ``` -### Choice of Time Representation: Seconds as `double` +### Choice of Naming & Representation -The timestamps `audioStartTime` and `audioEndTime` are defined as `double` representing **seconds**, rather than `DOMHighResTimeStamp` (milliseconds). This design choice is based on the following considerations: +1. **Mirroring `speechstart` and `speechend`:** + * The names `speechStartTime` and `speechEndTime` mirror the existing `speechstart` and `speechend` events in the Web Speech API, clearly communicating that these timestamps bound the acoustic speech segment corresponding to the transcript hypothesis. -1. **Consistency with Adjacent Web Audio & Media APIs:** - * In adjacent W3C media specifications, media-local stream timelines are universally represented in **seconds** as a `double`: +2. **Seconds as `double` (Consistency with Adjacent Media APIs):** + * In adjacent W3C media specifications, media-local timelines are universally represented in **seconds** as a `double`: * **Web Audio API:** [`BaseAudioContext.currentTime`](https://webaudio.github.io/web-audio-api/#dom-baseaudiocontext-currenttime) (seconds) * **HTML Media Elements:** [`HTMLMediaElement.currentTime`](https://html.spec.whatwg.org/multipage/media.html#dom-media-currenttime) (seconds) * **AudioParam Scheduling:** [`AudioParam.setValueAtTime()`](https://webaudio.github.io/web-audio-api/#dom-audioparam-setvalueattime) (seconds) - * Using seconds ensures seamless interoperability when developers route audio between `