Make dictation latency measurable and gated, and cut what the measurements justified

[r3dbars]

Why

The app's dictation start path is 3-4x over its own latency budget in production, and the check that would have caught it was dark: the budgets in performance-budget.rb only evaluated inside if required_samples.positive?, --require-dictation-fast-start-samples defaults to 0, and build.sh never passed --events at all. On top of that, every stage timing on that path was being written to events.jsonl as [redacted-sensitive-value], so the tail could not be attributed to a stage even by hand.

Measured on this machine over 176-203 real dictations (v1.1.56):

metric	p50	p95	p99	budget (p95)
request_to_recording_ms	107	1089	1685	250
start_to_first_sample_ms	198	814	1097	350
dictation fast-start	—	711	991	250
stop → text in editor	267	450	741	750
decode	148	307	451	—
launch → interactive	238	268	282	400 (target)
Home list @ 10k captures	183	—	—	750

Launch, the Home loader, and the transcription model are all comfortably inside budget. Dictation start is not, and at p95 that means losing most of a second of speech.

Product Impact

• Affects: dictation
• Lane: dictation reliability
• Why this matters: start latency is the difference between the app catching your first words and eating them. It was unmeasurable and ungated; now it is both.

What changed

Five commits, each independently revertible.

• Keep numeric measurements readable in the on-disk event log. The local sanitizer drops any key containing audio by substring, which swallowed audio_engine_start_ms, audio_tap_install_ms, audio_input_total_ms and seven more in 197/197 events. A bare number under a _ms/_s/_hz/_bytes key now survives; anything non-numeric still redacts, so audio_device_ms stays redacted. Local sink only — Sentry, Analytics, and reliability packets sanitize mergedContext separately through their own allowlists.
• Cut two measured costs. wavData appended 2 bytes per sample through a generic helper (4.8M round trips at the 5-minute cap); one contiguous Int16 block is 21x faster (2.58 → 0.12 ms at 10s, 82.0 → 3.7 ms at 5 min) and byte-identical. AccessibilityBridge made 5 unbounded AX round trips on the main actor gating the overlay's first pixel; now capped at 2s on the application element.
• Add repeatable benchmarks with p99 — bench-launch-latency.sh, latency-percentiles.py, bench-all.sh.
• Actually enforce the latency budgets in build.sh.
• Strip and prune the release bundle — 88.3 → 65.9 MiB.

On the gate's thresholds

Turning it on at the real targets fails immediately — that is the finding. So build.sh uses ratchet ceilings set just above today's measured latency (fast-start 900ms, request-to-recording 1300ms, start-to-first-sample 1000ms). It catches any further regression, which is strictly more than the nothing it caught before. The real targets stay as the defaults in performance-budget.rb, the gap is written down in one comment, and the ceilings may only ever move down. This matches how build.sh already overrides --max-app-mb and --max-launch-interactive-ms for its surface.

Two supporting changes were needed to make that safe:

• A budget now scores itself once the log carries 20+ samples; below that it is skipped, not failed — a thin log means "not enough data to judge", not "regression".
• --min-transcription-samples was an assertion, not a floor, so any sparse log hard-failed with Only 2 transcription samples, need at least 10. That would have broken build.sh on a fresh checkout or after a quiet fortnight. It is now a scoring floor, with --require-transcription-samples added for callers that want the assertion.

A failing run now prints the full summary on stdout before the reasons on stderr (needs an explicit flush — stdout is block-buffered when piped). Scoped to a 14-day window since events.jsonl is append-only across app versions. No log means no scoring, so CI and fresh checkouts stay green. TRANSCRIPTED_SKIP_RUNTIME_BUDGET=1 opts out.

How I checked it

• scripts/dev/agent-preflight.sh
• Selected checks from .agents/test-matrix.yml for the files changed
• bash build.sh --no-open
• bash run-tests.sh — 12,223 passed, 0 failed
• Performance budget passed
• bash run-integration-smoke.sh — n/a, no Sources/Meeting/ or Sources/TranscriptedCore/ changes
• swift test — n/a, no Package.swift or core seam changes
• Also run: bash build-deps.sh --force, SKIP_NOTARIZATION=1 bash build-beta.sh, python3 scripts/dev/check-build-source-lists.py, python3 scripts/dev/check-analytics-emitters.py, ruby -c, bash -n on both build scripts
• Manual check — gate behavior matrix, all five as expected: passes on the ratchet; fails when a ceiling drops below measured; skips on an empty log; skips on 2 samples; fails on 2 samples with --require-transcription-samples 10
• Manual check — byte-identity of the WAV encoder over 600,432 adversarial floats (signaling NaN, ±inf, subnormals, out-of-range, −0.0, every round-half tie), zero mismatches
• Manual check — release bundle: strip -x 30,363,584 → 17,045,424 bytes, app and dSYM LC_UUID identical, codesign --verify --deep --strict clean, no leftover Headers/Modules

Caveat on the beta build: run as a thin build (REQUIRE_BUNDLED_PARAKEET_MODELS=0 BUNDLE_DIARIZER_MODELS=0) because this machine has no local Parakeet/diarizer model cache. Strip, framework prune, signing, and dSYM pairing are all verified; model bundling is not.

Risk Review

• Privacy / local-first behavior reviewed — the sanitizer change is the only privacy-relevant edit. It is scoped to the on-disk sink, gated on both a measurement-suffixed key and a bare-numeric value, and covered by 11 new tests including negative cases (device labels, file paths, transcript text, unit-suffixed numbers, exponent notation, grouped digits under _ms keys). Off-device payloads are untouched.
• Storage path or migration impact reviewed — none.
• Public-facing copy stays concrete and matches current product scope — no user-facing copy changed.
• Release/update impact reviewed — build-beta.sh changes are release path. LC_UUID preserved so Sentry symbolication and the dSYM-UUID gate still pass; helper binaries deliberately not stripped.
• Agent PRs link the issue/workpad and stay draft until human review — no linked issue; opened as draft.
• UI changes include sanitized .agent-review/visuals/ evidence — n/a, no UI changes.
• No private transcripts, audio, tokens, personal paths, or customer data are included — the measurements quoted are aggregate percentiles only.

Notes

Six of the eight highest-ranked candidates were rejected under adversarial review, and that is worth as much as what landed:

• Cache the mic TCC check — ranked refactor(ui): Split FloatingPanel.swift into 10 organized files #2 on a measured 15.5ms per call, twice per start. That measurement was taken from a Terminal child process; re-measured inside a LaunchServices-launched bundle the real steady-state cost is ~2.4µs. A ~6000x methodology error; the entire prize was ~7µs against a stale-denied bug.
• Start hands-free dictation on key-down — nominally the largest win (80-250ms). Reverts 654aa15b under the defaults that motivated it and breaks two pinning tests. Hands-free is a toggle: copying push-to-talk's still-held gate makes a quick tap fail to stop a running dictation; dropping it spawns phantom dictations whenever ⌥M is held past 140ms. The "win" is the user's own finger dwell, and a 140ms disambiguation timer is slower than a typical 70-120ms tap.
• AXUIElementSetMessagingTimeout on the paste-confirmation call — that one is on the system-wide element, where it sets the process-global default rather than bounding one call; and the query is a guard, so failing it fast routes into the branch that leaves the clipboard borrowed, turning rare stalls into systematic clipboard clobbering on slow Electron/JetBrains targets.
• Drain audio before mic teardown — stop_to_mic_stop is 38% of felt stop latency and produces nothing for the user, but the 50ms sleep inside it guards a documented fatal CoreAudio assertion (isSink || tap != nullptr) on a path no CI job exercises.
• WAL + synchronous=OFF on the Home metadata cache — justified by a 200-row benchmark; the real cold-load workload is ~11 rows (meetingLimit = 10) on a background task.
• Async runtime-diagnostics writes — coalescing lets a pending write land after markCleanShutdown() and rewrite cleanShutdown: false, poisoning a live release-health metric.

Known measurement gaps, not addressed here:

• Stop latency is emitted only on the success branch, so every reported p99 (including the table above) is survivor-biased — the true tail is worse. Fixing it belongs in a separate dictation_stop_abandoned event, not by widening dictation_stop_latency_measured, which the release gate selects on.
• stop_to_mic_stop_ms fuses a queue hop, the engine stop, the 50ms sleep, and a 1.5s CoreAudio restore into one field; paste_prepare_ms fuses the AX capture with the pasteboard snapshot. Those are the two largest p99 sources on the stop path and each is reported as a single number.
• build.sh's launch smoke reported 758ms on builds where every isolated measurement landed at 220-290ms; I could not reproduce it across fresh HOME, fresh bundle copy, fresh code signature, or system load. swift-ci.yml:270 already documents a 48% swing on identical commits and suggests min-of-N sampling — bench-launch-latency.sh now does exactly that, so swapping the smoke for it is a small follow-up.

Next step worth more than any speculative optimization here: run a few dozen dictations on a build with the sanitizer fix, then read the start-path stage breakdown. It is measurable for the first time.

Agent handoff

COORD_DONE: GREEN | this PR | unredacted start-path timings, 21x WAV encode, bounded overlay AX query, live latency gate in build.sh with ratchet ceilings, release strip+prune (88.3->65.9 MiB), 3 new benchmark harnesses | no GitHub cleanup needed | decisions: ratchet ceilings vs hard-failing on the real targets, and whether to ship the survivor-bias fix as a separate event | checks: agent-preflight, build.sh --no-open, run-tests.sh (12223 pass), build-deps.sh --force, build-beta.sh thin, check-build-source-lists, check-analytics-emitters, ruby -c, bash -n, gate behavior matrix, WAV byte-identity over 600k adversarial floats | smallest next action: collect dictation samples on a build with the sanitizer fix and read the start-path stage breakdown