Skip to content

Desktop meeting endpoints: AI notes, Ask and cloud transcription - #4322

Open
richiemcilroy wants to merge 151 commits into
mainfrom
feat/meeting-notes-api
Open

richiemcilroy wants to merge 151 commits into
mainfrom
feat/meeting-notes-api

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Server side of the desktop meeting recorder: /api/desktop/meetings/* for capabilities, cloud transcription of meeting clips, AI notes and Ask.

sequenceDiagram
    participant D as Cap desktop
    participant W as /api/desktop/meetings
    participant S as Cap speech service (ASSEMBLY_BASE_URL)
    participant G as AI Gateway
    D->>W: GET /capabilities
    W-->>D: { transcription, notes, notesTrialRemaining?, limits }
    loop during the meeting (Pro, optional)
        D->>W: POST /transcribe (16 kHz mono WAV clip)
        W->>S: upload + transcript (AssemblyAI API shape)
        W-->>D: { text, language, durationMs, words }
    end
    D->>W: POST /notes { transcript, meetingId, ... }
    W->>G: stream (condense long transcripts in chunks first)
    W-->>D: SSE title, delta..., done
    D->>W: POST /ask { question, transcript, notesMarkdown }
    W-->>D: SSE delta..., done
Loading

Endpoints

All behind the existing desktop bearer auth (withAuth, scoped to /meetings only).

Route Gate Body Reply
GET /capabilities signed in none JSON, Cache-Control: no-store
POST /transcribe Pro, cloud transcription configured audio/wav 16 kHz mono s16, max 120 s / 4 MB, keywords in x-cap-keywords JSON words with ms timings
POST /notes Pro, or a free meeting left JSON transcript (max 400k chars) SSE
POST /ask Pro, or a meeting written up free JSON question + meeting material SSE
  • Cloud transcription is only offered when ASSEMBLY_BASE_URL points at Cap's own speech service. Unset, or an assemblyai.com host, means transcription: false in capabilities and 503 not_configured from /transcribe, so the app stays on its local model (apps/web/lib/meetings/speech.ts).
  • Clips go out as ordinary audio with no priority extension fields, so they queue behind live captions on the speech service.
  • Signed-in people without Pro get their first 3 meetings written up free, counted per (userId, meetingId) in the new meeting_notes_trial_uses table (migration 0062). A failed write gives the meeting back; Ask works only on those meetings.
  • Notes and Ask run on the Vercel AI Gateway (OIDC on Vercel, AI_GATEWAY_API_KEY elsewhere), falling back to the existing provider chain on self-hosted installs. Prompts wrap meeting material in a per-request random boundary (Ask uses an HMAC of the material so follow-ups reuse the cached prefix).
  • POST /api/desktop/subscribe now defaults to the monthly plan when no priceId is sent, so the in-app upgrade sheet does not hard-code a price.

Limits

Per user, checked before the body is read or the provider is called (apps/web/lib/meetings/budgets.ts):

Rule Firewall (suggested) In-process fallback
rl_meetings_transcribe 1200 / 1 h 1200 / 1 h
rl_meetings_audio (1 unit per started 10 s) 720 / 1 h 3600 / 24 h
rl_meetings_notes 60 / 1 h 60 / 1 h
rl_meetings_ask 120 / 1 h 120 / 1 h

Deadlines: transcribe 90 s, notes 270 s, ask 120 s.

Kill switches

Layered: defaults < MEETINGS_AI_CONFIG (JSON env) < Edge Config item meetingsAi when EDGE_CONFIG is set (read every 30 s, never blocks a warm instance).

{ "notesEnabled": false, "askEnabled": false, "transcriptionEnabled": false,
  "notesModels": ["openai/gpt-6-luna"], "trialMeetings": 3, "reasoningEffort": "none" }

Existing routes

  • route.ts only mounts /meetings; its auth middleware does not run for other desktop routes.
  • No new queries or awaits on existing routes. The subscribe default is a synchronous plan lookup.

How to test

curl -H "Authorization: Bearer $KEY" "$HOST/api/desktop/meetings/capabilities"
curl -N -H "Authorization: Bearer $KEY" -H "content-type: application/json" \
  -d '{"transcript":"Alex: ship Friday. Sam: I will write the notes.","meetingId":"m1","title":""}' \
  "$HOST/api/desktop/meetings/notes"

Or point the desktop app's server URL at the deployment, turn on Meeting Mode and end a meeting: notes write themselves and Ask answers from the note.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge, with no new findings or outstanding numbered findings.

Summary

Adds desktop meeting recording, calendar links, notes, Ask, and cloud transcription.

  • Cap desktop can record meetings, transcribe them, and create AI notes.
  • Meeting endpoints use per-user limits and a free notes trial.
  • Desktop upgrades choose the monthly plan when no price is sent.
  • CI can run meeting transcription checks on Linux and Windows.

Reviews (12) · Last reviewed commit: "chore(desktop): drop Library comments th..." · Reviewed by Greptile

One row per (user, meeting) written up free, so signed-in people without
Pro get their first few meeting notes without a separate counter.
checkFirewallRateLimitUnits charges several units against a firewall rule
in one round trip, and consumeRollingBudget keeps an in-process rolling
window that reports when it frees up. Adds the four rl_meetings_* rules.
/api/desktop/meetings/* for the desktop meeting recorder. Notes and Ask
stream over SSE from the AI Gateway (provider chain off Vercel), with
model and kill switches in MEETINGS_AI_CONFIG or an Edge Config item.
Cloud transcription is only offered when ASSEMBLY_BASE_URL points at
Cap's own speech service, never AssemblyAI directly.
The meeting upgrade sheet leaves the plan to the server, so the default
can change without an app release.
@sandchest-com

sandchest-com Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Sandchest

Sandchest reviewed fd779df: nothing exploitable found in this change.

Sandchest reviews every change for vulnerabilities an attacker could use · 22 credits · Review details

Comment thread apps/web/lib/meetings/trial.ts Resolved

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 5 potential issues (5 medium).

View 2 other findings in Capy.

Open in Capy

Comment thread apps/web/lib/meetings/trial.ts Resolved
Comment thread apps/web/lib/meetings/ai-config.ts Resolved
Each free meeting takes a numbered slot with a unique (userId, slot)
key, so concurrent claims cannot pass the limit. A meeting is marked
written when its notes finish; only a first write-up that failed gives
the meeting back, and a second request for a meeting still being
written waits. Rewrites and questions per free meeting are capped.
Covered against MySQL in the Infra Database job.
Ask checks askEnabled and its own model chain instead of the notes
switch. A free meeting can be rewritten after all of them are used.
Speech calls are abandoned when the deadline passes or the app
disconnects. Off Vercel, long transcripts are condensed before the
single notes pass.
@vercel

vercel Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Skipped Skipped Nov 22, 2026 11:05pm UTC

@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 2 other findings in Capy.

Open in Capy

The meeting is marked written before the done event, so Ask works as
soon as notes finish. A run that delivered any notes keeps its free
meeting even if the stream then fails or is dropped; only a run that
showed nothing gives it back. The speech copy is deleted from the
submission itself, so a submission that completes after the request
gave up is still removed.
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 17, 2026 19:18 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 3 other findings in Capy.

Open in Capy

# Conflicts:
#	.github/workflows/infra-database.yml

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

Ask accepts the desktop vocabulary and adds it as its own delimited block
so names and terms are spelled the same way as in the transcript.

The notes prompt now tells the model to use speaker labels exactly as the
transcript writes them (for example Person 2 or You), so a later rename in
the app maps each label to a name with a precise replace.

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

The encoder's weights moved to their side file inside the first meeting's
model load, so a fresh install showed no captions until the split was done.
On a small machine that was most of the wait: on 2 vCPU test boxes the first
caption came 9.7 s (Linux) and 12-43 s (Windows) into the meeting.

The download now starts the split on a low-priority thread once the files are
in place. The bytes written are the same, and the load still calls the split,
so a model downloaded before this, or a meeting that starts while the split
runs, waits on the same lock as before. A meeting already running when a
download finishes leaves the split to the next load, as today.

Fresh install, meeting 90 s after the download: first caption 3.2 s on Linux
2 vCPU and 4.8-5.8 s on Windows 2 vCPU.

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

Comment on lines +1917 to +1919
.spawn(move || {
lower_thread_priority();
split_parakeet_encoder(&dir);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Captions wait for another model

Downloading best can delay captions that use an already-installed best-max. The new background split holds the global SPLITTING lock while rewriting the roughly 900 MB encoder. Loading best-max waits for that lock before checking whether it needs splitting, even though its weights are already separate.

On a slow disk, captions can wait over a minute for work on an unrelated model. Skip already-external models before taking the lock, or use a separate lock for each model directory.

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/src/transcription.rs
Line: 1917-1919

Comment:
**Captions wait for another model**

Downloading `best` can delay captions that use an already-installed `best-max`. The new background split holds the global `SPLITTING` lock while rewriting the roughly 900 MB encoder. Loading `best-max` waits for that lock before checking whether it needs splitting, even though its weights are already separate.

On a slow disk, captions can wait over a minute for work on an unrelated model. Skip already-external models before taking the lock, or use a separate lock for each model directory.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c42c34d: the split lock is now one per model folder (folder_lock, keyed on the canonical path), so splitting best after its download never makes a best-max load wait. Test a_split_of_one_model_never_holds_up_another; full cap-gpui suite green.

Splitting the encoder right after a download held one lock shared by
every model, so loading an installed model (best-max, say) could wait a
minute on a slow disk for the split of another (best). Each model folder
now has its own lock.
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

…nd note excerpt

Meetings carry the app they ran in (a browser call resolves to the service
its join link points at) and the calendar event they were taken for. Plain
notes get their first lines as an excerpt, read once per scan from small
files only. The demo seed covers every meeting app and, with
CAP_GPUI_LIBRARY_SEED_GAPS=1, missing pictures and empty notes; without it
the seed is unchanged.
…aper notes

Meeting rows and cards show the icon of the meeting app installed on this
computer, read off the main thread once per app and scale (NSWorkspace on
macOS, the freedesktop icon theme on Linux, a meeting glyph elsewhere or
when the app is missing). Calendar meetings get a calendar leaf on cards and
Up next, notes without a picture become paper cards with whole lines of
their text, and a missing picture shows its kind's tile instead of an empty
well.
A search keystroke walks every item. Each item kept its folded title, app
and every person as separate strings, so one keystroke ran a substring
search per part and followed a pointer per part: at 2,000 items that is
about 10,000 searcher setups and as many scattered reads, and any growth of
the item spread them further. The folded parts now live in one string with
a NUL before every part after the title, read by one contains per item.
Folded parts never hold a NUL, so a match cannot run from one part into the
next and results are unchanged. The item also gets 40 bytes smaller.

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 3 other findings in Capy.

Open in Capy

@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

Comment on lines +27 to +32
/// Looked up and found.
Image(Arc<RenderImage>),
/// Still looking: draw nothing yet (the slot keeps its size).
Pending,
/// Not installed, or no icon: draw the meeting glyph.
Missing,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Comments repeat the code

The new comments describe obvious code instead of explaining a hidden constraint. For example, “Looked up and found” repeats what Icon::Image means. AGENTS.md explicitly bans comments that narrate code or restate types. Remove these before merging; keep explanations of platform constraints and non-obvious trade-offs. The same pattern appears above glyph_tile and leaf_face.

Context Used: AGENTS.md (source)

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/src/library_window/app_icons.rs
Line: 27-32

Comment:
**Comments repeat the code**

The new comments describe obvious code instead of explaining a hidden constraint. For example, “Looked up and found” repeats what `Icon::Image` means. `AGENTS.md` explicitly bans comments that narrate code or restate types. Remove these before merging; keep explanations of platform constraints and non-obvious trade-offs. The same pattern appears above `glyph_tile` and `leaf_face`.

**Context Used:** AGENTS.md ([source](https://github.com/capsoftware/cap/blob/main/AGENTS.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1f3f8b6: removed the comments that restated the code (Icon variants, get/take_wanted, glyph_tile, leaf_face). Kept the ones that explain a constraint: Pending keeps its slot size so rows do not shift, AppKit premultiplied vs straight alpha, the Exec field holding only the binary name, and the icon lookup order.

@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

… computers apart

On a computer too slow for live local captions, the live transcript with
Cap Cloud defaults off until the user turns it on. Every other computer
and engine keeps the existing choice, and settings files only gain the
new field once someone chooses.
Every Mac without Apple silicon (it runs the lighter Whisper model), and
Windows or Linux below 4 logical cores. Apple silicon never counts: a
3-core M1 keeps up. CAP_GPUI_MEETING_SLOW_DEVICE=1 or 0 overrides it, and
tests always see a capable computer unless they set otherwise.
With the live transcript off there, a Cloud meeting runs the local model
for nothing while the cloud answers: no captions and no local lines
before the cloud's, as with no local model installed. The model still
writes the words when the cloud fails or refuses the account, so nothing
is lost. Every other computer and mode runs exactly as before.
… and settings

A short note under the engine choice says this computer may be too slow
to show the transcript live and that Cap Cloud keeps up instead, with
Get Cap Pro or sign in when Cap Cloud isn't theirs yet. The live
transcript switch shows and changes the slow computer's Cap Cloud choice.
Translated into every catalog.
…the others

An Intel Mac now writes notes with Whisper base: Whisper small falls
minutes behind there, base keeps up. Windows and Linux below 4 cores keep
Parakeet where it writes the language and use base in place of small
elsewhere. Base downloads from the same model release, and the disk scan
only looks for it on those computers, so capable ones are unchanged.
…ap Cloud is recommended

The model row shows the one compatible model with a short note and Sign
in, Upgrade to Cap Pro or Use Cap Cloud, by account. With Cap Cloud picked,
only the live transcript note shows. Whisper small from before is offered
for deletion and never removed unasked. Translated into every catalog.

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 3 other findings in Capy.

Open in Capy

The page posts /mic and /started at nearly the same moment, each handled on
its own server thread. Windows appends by seeking to the end and then
writing, so two unlocked writers can land on the same offset: the
'mic on' line overwrote the 'started ' prefix of the other, the system
audio offset was lost, and the local meeting scored 74.5% words lost with
every word actually there. One lock around each line now.

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

To-magazine and to-list cost differently, so the mixed p50 sat between two clusters and swung between runs. The window bench now also reports each direction, and the switch bench toggles notes too.
The switch bench now reports the frame that shows the loaded note on its own, and a search bench types into the sidebar search, timing each key's frame and the frame its hits land in.
CAP_GPUI_LIBRARY_BENCH=scroll-all scrolls Screenshots and Notes to the end and back in list and magazine, a draw per step, then pauses with a settled marker so a harness can read the resident size afterwards.
…ames

GPUI keeps a shaped line only while consecutive frames draw it, so switching between list and magazine or between sections shaped every card's text again, most of that frame. KeptText measures, truncates, clamps and paints like a plain string child but looks its lines up in a cache the window owns, trimmed to the most recently drawn half past 1,024 lines. The cache drops with the window, and its key carries the font, size, colour, wrap and truncation, so a theme or locale change never reuses old lines.

This branch was successfully deployed

1 active (outdated) and 1 inactive deployments
Preview – cap-status — c35bfb58 Deployed Nov 22, 2026 by vercel[bot]
Preview – cap-web — 9b319ba4 Deployed Nov 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants