Repository navigation
perf(desktop-gpui): caption long recordings in minute-long Parakeet chunks (up to 44% faster, no 6m40s limit) - #2539
Conversation
|
@unhingedpanda is attempting to deploy a commit to the Cap Software Inc Team on Vercel. A member of the Team first needs to authorize it. |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| std::fs::write(path, dump).expect("write words"); | ||
| } | ||
| } | ||
| Err(error) => println!("parakeet_seconds={seconds:.2} error={error}"), |
There was a problem hiding this comment.
benchmark_parakeet_transcription passes even when process_with_parakeet fails. A missing WAV, model error, or failed long recording prints an error but still gives cargo test a successful exit. Keep the timing output, then fail the test so developers can tell a broken run from a successful one.
| Err(error) => println!("parakeet_seconds={seconds:.2} error={error}"), | |
| Err(error) => panic!("parakeet_seconds={seconds:.2} error={error}"), |
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/src/transcription.rs
Line: 2243
Comment:
**Failed benchmark looks successful**
`benchmark_parakeet_transcription` passes even when `process_with_parakeet` fails. A missing WAV, model error, or failed long recording prints an error but still gives `cargo test` a successful exit. Keep the timing output, then fail the test so developers can tell a broken run from a successful one.
```suggestion
Err(error) => panic!("parakeet_seconds={seconds:.2} error={error}"),
```
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
|
@greptileai review |
|
Thanks for this! It now conflicts with main after today's merges. Could you merge or rebase onto the latest main? Once it's up to date I'll run the full engine verify and real-capture suites on it and merge if they pass. |
…-captions # Conflicts: # apps/desktop-gpui/src/transcription.rs
|
Merged the latest main (2b2bf84). The only conflict was in |
|
@greptileai review |
richiemcilroy
left a comment
richiemcilroy
left a comment
There was a problem hiding this comment.
Reproduced on an M4 Max with Parakeet TDT 0.6b v3 int8: a 6.5 min scripted recording goes from 17.2 s / 6.8 GB to 9.3 s / 2.4 GB, and a 7m53s one that fails on main transcribes in 11.1 s. WER against the script is within noise (6.9% vs 7.5%, differences spread across the recording, not at the cuts). Thanks!
|
@richiemcilroy Thanks for building such a great piece of software and making it free. Glad to finally contribute, I've been an avid Cap user for a long time. Would both #2536 and #2539 be eligible for sponsorship? |
2 participants
Generating captions with the default "Recommended" model (Parakeet TDT 0.6b v3 int8) fails on any recording longer than 6m40s. Below that limit it gets slower per minute as the recording grows, and on real speech it drops whole sentences. The GPUI app passed the whole recording to Parakeet in one encoder pass. That pass fails with an ONNX broadcast error past 400 s (5,000 encoder frames). Up to that point its time and memory grow faster than the audio's length.
This PR feeds Parakeet one minute of audio at a time. Each cut lands on the quietest 100 ms in the last 5 s before the minute mark, so cuts usually fall between words. Word timestamps are shifted back onto the recording's timeline. The model configuration, including its 4-thread default, is unchanged. Whisper models are untouched.
Results
Machine: Apple M5 Pro (15 cores), 24 GB, release build. Each number is the median of 3 runs, and before and after runs alternate. The 60-minute "after" figure is a single run. Time includes model load. Peak memory comes from
/usr/bin/time -l.After the change, time grows linearly at about 1.45 s per recorded minute. Peak memory stays flat at about 2.3 GB, most of which is the loaded model (1.8 GB right after load).
Correctness
Real speech. I used a 6m22s screen recording of someone talking through a demo. It is short enough that the old code still runs. The old single pass returned 534 words and the chunked pass returned 634. Most of the difference is whole sentences that the single pass left out, at 35 s, 68 s, 86 s, 101 s and 257 s. I transcribed a 12 s clip around each of four of those spots on its own, and every one contains the sentence. So the single pass was dropping real speech. Within 3 s of the six cut points, three short spans differ, all near the first cut. One is a small miss ("compose" became "compost"). The other five cuts show no differences.
Scripted speech. This audio is macOS
sayreading plain prose sentences from Cap's docs, which gives a known script to score against. It is written as 16 kHz mono s16le, the same formatextract_project_audiowrites. Text-to-speech pauses cleanly at every sentence, so it is easier to cut than real speech. Word error rate against the script:Word counts differ by a few words between builds (for example 1,002 against 998 at 6 min). The WER table above covers those differences.
Other checks
normalize_caption_wordsandcaption_word_chunksas before.extract_project_audiowrites. Empty audio returns "No speech detected" instead of reaching parakeet-rs, which indexes the first sample.Reproduce
The ignored
benchmark_parakeet_transcriptiontest times the realprocess_with_parakeetcall. For the baseline, I ran the same test on unchangedmain(3d19c5cfa).Set
CAP_PARAKEET_BENCH_WORDS=/path/out.tsvto dump the words with their timestamps.Notes
The PR appears safe to merge; no new actionable issue was found.Summary
Parakeet now reads recordings in chunks of at most one minute, choosing quiet cut points and shifting word times onto the recording’s timeline.
Reviews (4) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." · Reviewed by Greptile