Repository navigation
Desktop meeting endpoints: AI notes, Ask and cloud transcription - #4322
richiemcilroy wants to merge 151 commits into
Conversation
One row per (user, meeting) written up free, so signed-in people without Pro get their first few meeting notes without a separate counter.
checkFirewallRateLimitUnits charges several units against a firewall rule in one round trip, and consumeRollingBudget keeps an in-process rolling window that reports when it frees up. Adds the four rl_meetings_* rules.
/api/desktop/meetings/* for the desktop meeting recorder. Notes and Ask stream over SSE from the AI Gateway (provider chain off Vercel), with model and kill switches in MEETINGS_AI_CONFIG or an Edge Config item. Cloud transcription is only offered when ASSEMBLY_BASE_URL points at Cap's own speech service, never AssemblyAI directly.
The meeting upgrade sheet leaves the plan to the server, so the default can change without an app release.
|
Sandchest reviewed Sandchest reviews every change for vulnerabilities an attacker could use · 22 credits · Review details |
Each free meeting takes a numbered slot with a unique (userId, slot) key, so concurrent claims cannot pass the limit. A meeting is marked written when its notes finish; only a first write-up that failed gives the meeting back, and a second request for a meeting still being written waits. Rewrites and questions per free meeting are capped. Covered against MySQL in the Infra Database job.
Ask checks askEnabled and its own model chain instead of the notes switch. A free meeting can be rewritten after all of them are used. Speech calls are abandoned when the deadline passes or the app disconnects. Off Vercel, long transcripts are condensed before the single notes pass.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
@greptileai review |
The meeting is marked written before the done event, so Ask works as soon as notes finish. A run that delivered any notes keeps its free meeting even if the stream then fails or is dropped; only a run that showed nothing gives it back. The speech copy is deleted from the submission itself, so a submission that completes after the request gave up is still removed.
|
@greptileai review |
# Conflicts: # .github/workflows/infra-database.yml
Ask accepts the desktop vocabulary and adds it as its own delimited block so names and terms are spelled the same way as in the transcript. The notes prompt now tells the model to use speaker labels exactly as the transcript writes them (for example Person 2 or You), so a later rename in the app maps each label to a name with a precise replace.
|
@greptileai review |
The encoder's weights moved to their side file inside the first meeting's model load, so a fresh install showed no captions until the split was done. On a small machine that was most of the wait: on 2 vCPU test boxes the first caption came 9.7 s (Linux) and 12-43 s (Windows) into the meeting. The download now starts the split on a low-priority thread once the files are in place. The bytes written are the same, and the load still calls the split, so a model downloaded before this, or a meeting that starts while the split runs, waits on the same lock as before. A meeting already running when a download finishes leaves the split to the next load, as today. Fresh install, meeting 90 s after the download: first caption 3.2 s on Linux 2 vCPU and 4.8-5.8 s on Windows 2 vCPU.
|
@greptileai review |
| .spawn(move || { | ||
| lower_thread_priority(); | ||
| split_parakeet_encoder(&dir); |
There was a problem hiding this comment.
Captions wait for another model
Downloading best can delay captions that use an already-installed best-max. The new background split holds the global SPLITTING lock while rewriting the roughly 900 MB encoder. Loading best-max waits for that lock before checking whether it needs splitting, even though its weights are already separate.
On a slow disk, captions can wait over a minute for work on an unrelated model. Skip already-external models before taking the lock, or use a separate lock for each model directory.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/src/transcription.rs
Line: 1917-1919
Comment:
**Captions wait for another model**
Downloading `best` can delay captions that use an already-installed `best-max`. The new background split holds the global `SPLITTING` lock while rewriting the roughly 900 MB encoder. Loading `best-max` waits for that lock before checking whether it needs splitting, even though its weights are already separate.
On a slow disk, captions can wait over a minute for work on an unrelated model. Skip already-external models before taking the lock, or use a separate lock for each model directory.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in c42c34d: the split lock is now one per model folder (folder_lock, keyed on the canonical path), so splitting best after its download never makes a best-max load wait. Test a_split_of_one_model_never_holds_up_another; full cap-gpui suite green.
Splitting the encoder right after a download held one lock shared by every model, so loading an installed model (best-max, say) could wait a minute on a slow disk for the split of another (best). Each model folder now has its own lock.
|
@greptileai review |
…nd note excerpt Meetings carry the app they ran in (a browser call resolves to the service its join link points at) and the calendar event they were taken for. Plain notes get their first lines as an excerpt, read once per scan from small files only. The demo seed covers every meeting app and, with CAP_GPUI_LIBRARY_SEED_GAPS=1, missing pictures and empty notes; without it the seed is unchanged.
…aper notes Meeting rows and cards show the icon of the meeting app installed on this computer, read off the main thread once per app and scale (NSWorkspace on macOS, the freedesktop icon theme on Linux, a meeting glyph elsewhere or when the app is missing). Calendar meetings get a calendar leaf on cards and Up next, notes without a picture become paper cards with whole lines of their text, and a missing picture shows its kind's tile instead of an empty well.
A search keystroke walks every item. Each item kept its folded title, app and every person as separate strings, so one keystroke ran a substring search per part and followed a pointer per part: at 2,000 items that is about 10,000 searcher setups and as many scattered reads, and any growth of the item spread them further. The folded parts now live in one string with a NUL before every part after the title, read by one contains per item. Folded parts never hold a NUL, so a match cannot run from one part into the next and results are unchanged. The item also gets 40 bytes smaller.
|
@greptileai review |
| /// Looked up and found. | ||
| Image(Arc<RenderImage>), | ||
| /// Still looking: draw nothing yet (the slot keeps its size). | ||
| Pending, | ||
| /// Not installed, or no icon: draw the meeting glyph. | ||
| Missing, |
There was a problem hiding this comment.
The new comments describe obvious code instead of explaining a hidden constraint. For example, “Looked up and found” repeats what Icon::Image means. AGENTS.md explicitly bans comments that narrate code or restate types. Remove these before merging; keep explanations of platform constraints and non-obvious trade-offs. The same pattern appears above glyph_tile and leaf_face.
Context Used: AGENTS.md (source)
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/src/library_window/app_icons.rs
Line: 27-32
Comment:
**Comments repeat the code**
The new comments describe obvious code instead of explaining a hidden constraint. For example, “Looked up and found” repeats what `Icon::Image` means. `AGENTS.md` explicitly bans comments that narrate code or restate types. Remove these before merging; keep explanations of platform constraints and non-obvious trade-offs. The same pattern appears above `glyph_tile` and `leaf_face`.
**Context Used:** AGENTS.md ([source](https://github.com/capsoftware/cap/blob/main/AGENTS.md))
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
Fixed in 1f3f8b6: removed the comments that restated the code (Icon variants, get/take_wanted, glyph_tile, leaf_face). Kept the ones that explain a constraint: Pending keeps its slot size so rows do not shift, AppKit premultiplied vs straight alpha, the Exec field holding only the binary name, and the icon lookup order.
|
@greptileai review |
… computers apart On a computer too slow for live local captions, the live transcript with Cap Cloud defaults off until the user turns it on. Every other computer and engine keeps the existing choice, and settings files only gain the new field once someone chooses.
Every Mac without Apple silicon (it runs the lighter Whisper model), and Windows or Linux below 4 logical cores. Apple silicon never counts: a 3-core M1 keeps up. CAP_GPUI_MEETING_SLOW_DEVICE=1 or 0 overrides it, and tests always see a capable computer unless they set otherwise.
With the live transcript off there, a Cloud meeting runs the local model for nothing while the cloud answers: no captions and no local lines before the cloud's, as with no local model installed. The model still writes the words when the cloud fails or refuses the account, so nothing is lost. Every other computer and mode runs exactly as before.
… and settings A short note under the engine choice says this computer may be too slow to show the transcript live and that Cap Cloud keeps up instead, with Get Cap Pro or sign in when Cap Cloud isn't theirs yet. The live transcript switch shows and changes the slow computer's Cap Cloud choice. Translated into every catalog.
…the others An Intel Mac now writes notes with Whisper base: Whisper small falls minutes behind there, base keeps up. Windows and Linux below 4 cores keep Parakeet where it writes the language and use base in place of small elsewhere. Base downloads from the same model release, and the disk scan only looks for it on those computers, so capable ones are unchanged.
…ap Cloud is recommended The model row shows the one compatible model with a short note and Sign in, Upgrade to Cap Pro or Use Cap Cloud, by account. With Cap Cloud picked, only the live transcript note shows. Whisper small from before is offered for deletion and never removed unasked. Translated into every catalog.
The page posts /mic and /started at nearly the same moment, each handled on its own server thread. Windows appends by seeking to the end and then writing, so two unlocked writers can land on the same offset: the 'mic on' line overwrote the 'started ' prefix of the other, the system audio offset was lost, and the local meeting scored 74.5% words lost with every word actually there. One lock around each line now.
To-magazine and to-list cost differently, so the mixed p50 sat between two clusters and swung between runs. The window bench now also reports each direction, and the switch bench toggles notes too.
The switch bench now reports the frame that shows the loaded note on its own, and a search bench types into the sidebar search, timing each key's frame and the frame its hits land in.
CAP_GPUI_LIBRARY_BENCH=scroll-all scrolls Screenshots and Notes to the end and back in list and magazine, a draw per step, then pauses with a settled marker so a harness can read the resident size afterwards.
…ames GPUI keeps a shaped line only while consecutive frames draw it, so switching between list and magazine or between sections shaped every card's text again, most of that frame. KeptText measures, truncates, clamps and paints like a plain string child but looks its lines up in a cache the window owns, trimmed to the most recently drawn half past 1,024 lines. The cache drops with the window, and its key carries the font, size, colour, wrap and truncation, so a theme or locale change never reuses old lines.

Server side of the desktop meeting recorder:
/api/desktop/meetings/*for capabilities, cloud transcription of meeting clips, AI notes and Ask.sequenceDiagram participant D as Cap desktop participant W as /api/desktop/meetings participant S as Cap speech service (ASSEMBLY_BASE_URL) participant G as AI Gateway D->>W: GET /capabilities W-->>D: { transcription, notes, notesTrialRemaining?, limits } loop during the meeting (Pro, optional) D->>W: POST /transcribe (16 kHz mono WAV clip) W->>S: upload + transcript (AssemblyAI API shape) W-->>D: { text, language, durationMs, words } end D->>W: POST /notes { transcript, meetingId, ... } W->>G: stream (condense long transcripts in chunks first) W-->>D: SSE title, delta..., done D->>W: POST /ask { question, transcript, notesMarkdown } W-->>D: SSE delta..., doneEndpoints
All behind the existing desktop bearer auth (
withAuth, scoped to/meetingsonly).GET /capabilitiesCache-Control: no-storePOST /transcribeaudio/wav16 kHz mono s16, max 120 s / 4 MB, keywords inx-cap-keywordsPOST /notesPOST /askASSEMBLY_BASE_URLpoints at Cap's own speech service. Unset, or anassemblyai.comhost, meanstranscription: falsein capabilities and503 not_configuredfrom/transcribe, so the app stays on its local model (apps/web/lib/meetings/speech.ts).(userId, meetingId)in the newmeeting_notes_trial_usestable (migration0062). A failed write gives the meeting back; Ask works only on those meetings.AI_GATEWAY_API_KEYelsewhere), falling back to the existing provider chain on self-hosted installs. Prompts wrap meeting material in a per-request random boundary (Ask uses an HMAC of the material so follow-ups reuse the cached prefix).POST /api/desktop/subscribenow defaults to the monthly plan when nopriceIdis sent, so the in-app upgrade sheet does not hard-code a price.Limits
Per user, checked before the body is read or the provider is called (
apps/web/lib/meetings/budgets.ts):rl_meetings_transcriberl_meetings_audio(1 unit per started 10 s)rl_meetings_notesrl_meetings_askDeadlines: transcribe 90 s, notes 270 s, ask 120 s.
Kill switches
Layered: defaults <
MEETINGS_AI_CONFIG(JSON env) < Edge Config itemmeetingsAiwhenEDGE_CONFIGis set (read every 30 s, never blocks a warm instance).{ "notesEnabled": false, "askEnabled": false, "transcriptionEnabled": false, "notesModels": ["openai/gpt-6-luna"], "trialMeetings": 3, "reasoningEffort": "none" }Existing routes
route.tsonly mounts/meetings; its auth middleware does not run for other desktop routes.How to test
Or point the desktop app's server URL at the deployment, turn on Meeting Mode and end a meeting: notes write themselves and Ask answers from the note.


Confidence Score: 5/5
The PR appears safe to merge, with no new findings or outstanding numbered findings.Summary
Adds desktop meeting recording, calendar links, notes, Ask, and cloud transcription.
Reviews (12) · Last reviewed commit: "chore(desktop): drop Library comments th..." · Reviewed by Greptile