FE-1654: Test Realtime rephrasing of completed Brunch responses

[kostandinang]

🌟 What is the purpose of this PR?

Test whether OpenAI Realtime can keep Voice conversational while Brunch remains the authoritative domain agent. Brunch streams its complete response to the screen independently; after the entire turn and its tool continuations succeed, Realtime speaks a concise, faithful rephrasing. During work, Realtime may deliver only application-selected, event-backed acknowledgements or progress notices.

This is a draft sibling of #9622, based on #9585, not stacked on the separate-Brunch-authored-speech experiment. One short live query now completes through Realtime → Brunch → spoken rephrasing, with the returned audio inspected. The original acknowledgement-only failure did not reproduce; broader fidelity, naturalness, queueing and interruption evaluation remains outstanding.

🔗 Related links

• Parent relay experiment: FE-1630: Optimize and measure the Brunch Voice relay #9585
• Comparison/control: FE-1657: Separate Brunch-authored Voice speech from on-screen responses #9622
• Experiment contract and local verification
• FE-1654 — Test Realtime rephrasing of completed Brunch responses (internal)

🚫 Blocked by

• FE-1630: Optimize and measure the Brunch Voice relay #9585 must land, or the sibling must be reconciled with its replacement, before targeting main.
• Human comparison must evaluate fidelity, qualifications, fabricated claims/progress, and naturalness alongside audio timing.
• Deployed end-to-end verification remains outstanding.

🔍 What does this change?

Realtime becomes a bounded spoken rephrasing layer over completed Brunch responses. It has no domain tools or autonomous response path. Brunch continues to own the full visible answer, reasoning, questions, and tool calls. Realtime speech is not inserted as another canonical chat message.

Additional finalized speech is retained in FIFO order while Brunch works. A compact follow-up count confirms retention without displaying a separate transcript list. Discard removes waiting inputs; failure holds them for explicit Resume. Speaking over audio interrupts playback only, while Stop cancels active work and withdraws queued inputs.

Rephrasing is answer-first and conversational. Missing or unfinished completed-response text produces a visible error, and acknowledgement requests are excluded from substantive speech timing.

🏗️ Agent notes

Status and imperative

Draft experiment, not accepted. Test conversational fluidity without transferring domain authority from Brunch. A concise spoken answer is the normal path; Read full response remains optional.

Throughline and contracts

Realtime transcription → existing panel FIFO → AI SDK/Flue admission → Brunch and browser-tool continuations → canonical screen stream → positively completed correlated response snapshot → application-requested Realtime rephrasing → audio.

• Brunch owns domain claims, questions, conclusions, tools, and workpieces. Realtime re-expresses supplied completed content only. A marked Brunch question is delivered exactly.
• VAD does not automatically create or interrupt responses. The application requests all speech. Fixed notices correspond to actual receipt, actual queue retention, or admitted continuation; notices are deduplicated/coalesced and obsolete ones dropped. No timer-generated or inferred progress.
• Rephrasing receives the complete ordered canonical source and any valid marked question through an isolated conversation: "none" request. No reasoning, partial output, raw tool results, history, or queued user text enters that request. Source content is data, not instructions.
• The prompt preserves quantities, negation, uncertainty, consequential qualifications, later corrections, and proposed/attempted/completed/validated distinctions. Prefer 2–4 sentences; fidelity takes priority over length. Prompting is not a semantic guarantee.
• Every root and causally linked continuation must settle successfully, including textless continuations, before final speech. UI ready or an individual message completion is insufficient. Failed, cancelled, stopped, or unproven work cannot trigger a final paraphrase, and late events cannot revive invalidated generations.
• A completed explanation of a rejected/no-op domain tool result may be spoken; this differs from a failed agent execution.
• FIFO follows capture/commit identity rather than asynchronous transcription completion. Deduplicate identities, not equal words. Freeze A's response before admitting B; queue draining does not wait for audio. Audio is serialized and correlated separately.
• Microphone capture stays available during work and playback. Barge-in cancels audio only. Stop revokes Voice completion synchronously, cancels work, and withdraws unadmitted input. Failed/ambiguous admissions hold the queue for explicit recovery. The queue is session-local, not durable.
• Disconnect does not abort Brunch. Reopening does not autoplay responses completed while disconnected or historical responses. Reload does not resend queue contents.
• Audio failure, truncation, or an oversized source leaves canonical text intact and reports failure. Never silently truncate source or automatically read the report. Exact Read full response and Repeat question remain optional.

Proof and experiment measurements

Automated checks, UI inspection and the single live diagnostic are recorded below. They do not establish general fidelity or naturalness.

Measure acknowledgement latency, silence during Brunch work, completion-to-substantive-audio latency, qualification fidelity, fabricated claims/progress, queued-turn correctness and conversational naturalness. Include queue wait and missing answers. Use synchronized audio/screen recording; provider-buffer timestamps are only proxies. Pin models, prompts and fixtures across #9585, #9622 and this variant. #9622 also changes Brunch generation, so differences cannot all be attributed to Realtime.

Constraints

No Realtime domain tools or independent domain reasoning. Preserve typed-mode behavior, exact user text, turn identity and idempotency. No new endpoint, conversation store or Flue upgrade; the parent’s local Flue 2.0.3 context patch is retained. Operational logs contain metadata, not transcript, source or audio.

Fog-line

The acknowledgement-only report remains unexplained. Long answers, tool continuations, overlapping speech and real microphone echo need live evaluation. Runtime completion gates do not guarantee faithful wording. Deployed behavior is unverified.

Stop or reorient

Reassess on premature speech, fabricated progress, qualification loss, lost/reordered turns, playback after cancellation or typed-mode changes. Do not expand Realtime’s authority to compensate for poor naturalness. Further paid trials require approval.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

• modifies an npm-publishable library and includes a patch changeset for @hashintel/petrinaut

📜 Does this require a change to the docs?

The changes in this PR:

• require changes to docs which are made as part of this PR

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

• do not affect the execution graph

⚠️ Known issues

• The reported “Got that” without a final answer did not reproduce in the short live test. Its cause remains unconfirmed; no root-cause fix is claimed.
• One short, no-tool response does not establish fidelity or naturalness for long answers, tool continuations or interruptions. Rephrasing can still lose qualifications or invent content; real microphone echo is untested.
• Waiting for all Brunch continuations preserves authority but leaves reasoning latency. Event-backed notices cannot eliminate silence.
• The queue is session-local. Stop suppresses playback, but if Flue already settled before browser work was stopped, that stopped state has an inherited persistence limitation.
• Deployed verification and maintainer-captured user-guide screenshots remain outstanding. Existing build/lint warnings remain.

🐾 Next steps

Compare the three variants using short and long answers, tool continuations, queued turns and interruptions. Judge Brunch’s correctness against tool evidence separately from spoken fidelity against its canonical response.

🛡 What tests cover this?

Latest local verification: 1,239 tests passed — Petrinaut 808, website 378, transport 52, Brunch Voice context 1.

All 12 build/test/typecheck/lint tasks passed for Petrinaut, the website and transport:

CI=1 yarn exec turbo run build test:unit lint:tsc lint:eslint \
  --filter @hashintel/petrinaut \
  --filter @apps/petrinaut-website \
  --filter @hashintel/brunch-agent-transport-aisdk \
  --only --force --output-logs errors-only
yarn workspace @apps/brunch-agent exec vitest run test/voice-context.test.ts

Coverage includes complete-turn gating, failed/textless continuations, Stop races, FIFO ordering and recovery, no historical autoplay, missing/unfinished response errors and acknowledgement-excluded speech timing. Prompt/payload tests cover source isolation, exact marked questions and no domain tools. Browser checks exercise Resume/Discard; queued, recovery and listening states were visually inspected. Changed-file formatting and commit hooks passed.

Live diagnostic — 10 September 2026

Fresh local Chrome conversation with real WebRTC, Realtime and Brunch. Synthetic spoken input: “In two sentences, what is a Petri net? Do not change the model.”

• Brunch completed one submission and displayed its full two-sentence answer. Realtime spoke “Okay, I hear you” followed by a complete three-sentence rephrasing. Recorded audio inspection found no truncation, added claims or lost qualifications in this sample. The screen retained one canonical answer and returned to Listening.
• Provider-buffer timing proxies: 1.27s from user speech-end to acknowledgement; 0.66s from complete Brunch response to substantive audio. The gap after acknowledgement was 2.45s, including 1.79s while Brunch still worked. These are not first-audible measurements.
• Test used a 2,048-output-token paraphrase ceiling (540 used). It covers one short, no-tool answer, not live queueing, continuations or interruptions.

❓ How to test this?

1. Check out ka/realtime-voice-rephrasing, build workspace dependencies and run the checks above. This branch targets FE-1630: Optimize and measure the Brunch Voice relay #9585, not FE-1657: Separate Brunch-authored Voice speech from on-screen responses #9622.
2. For provider-free UI review, run yarn exec storybook dev -p 6108 --ci --no-open in libs/@hashintel/petrinaut. Inspect editor-ai-assistant--voice-queued-turns, --voice-queue-recovery, and --voice-session-listening; verify the compact count, Resume/Discard, and no empty indicator.
3. Only after paid trial approval, start the existing Brunch/website development stack with Voice enabled and open /?brunch-fixture=crew-reservation-v1.
4. Ask “What does reserving a dispatch crew mean here?” and “Give me a detailed analysis of this model, including assumptions, possible bottlenecks, missing constraints, and what still needs validation. Do not change the model.” Check immediate canonical streaming and one final paraphrase only after all continuations succeed.
5. Speak follow-ups during work, interrupt playback, Stop, trigger a failed continuation, resume/discard held inputs, and reopen. Check FIFO admission, preserved capture, no final paraphrase for failed/stopped work, optional exact replay, and no history autoplay. Record actual audio alongside the screen for the seven measurements.

📹 Demo

Inspected local Chrome/Storybook fixtures. These show UI layout only; no live provider or acoustic result is implied.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
petrinaut	Ready	Preview	Sep 10, 2026 9:09am UTC
petrinaut-docs	Ready	Preview	Sep 10, 2026 9:09am UTC

2 Skipped Deployments

Project	Deployment	Actions	Updated
hash	Ignored	Preview	Sep 10, 2026 9:09am UTC
hashdotdesign-tokens	Ignored	Preview	Sep 10, 2026 9:09am UTC