feat: add realtime chat support with OpenAI and ElevenLabs adapters#300
Open
feat: add realtime chat support with OpenAI and ElevenLabs adapters#300
Conversation
# Conflicts: # packages/typescript/ai-openai/src/index.ts # pnpm-lock.yaml
Contributor
📝 WalkthroughWalkthroughAdds a provider-agnostic realtime voice chat subsystem: server token adapters, OpenAI and ElevenLabs client adapters, a RealtimeClient library, a React hook (useRealtimeChat), shared realtime types, example/demo pages, and docs describing flows and token lifecycle. Changes
Sequence Diagram(s)sequenceDiagram
participant Browser as Client (Browser)
participant Server as App Server
participant Provider as Provider API (OpenAI / ElevenLabs)
Browser->>Server: POST /api/realtime-token (provider, agentId?, tools, config)
Server->>Provider: Request session/signed URL (provider API)
Provider-->>Server: Token/session + config
Server-->>Browser: RealtimeToken { provider, token, expiresAt, config }
Browser->>Browser: RealtimeAdapter.connect(token)
Browser->>Provider: Establish realtime session (WebRTC / WebSocket)
Provider-->>Browser: Connection established / events
Browser->>Browser: startAudioCapture() → stream audio
Provider-->>Browser: transcript / message / tool_call events
Browser->>Browser: execute local tool (if tool_call)
Browser->>Provider: sendToolResult()
sequenceDiagram
participant React as UI Component
participant Hook as useRealtimeChat
participant Client as RealtimeClient
participant Adapter as RealtimeAdapter
participant Connection as RealtimeConnection
React->>Hook: init useRealtimeChat(options)
Hook->>Client: construct RealtimeClient(options)
React->>Hook: connect()
Hook->>Client: connect()
Client->>Hook: getToken() (server call)
Client->>Adapter: adapter.connect(token)
Adapter->>Connection: establish media/events
Connection-->>Client: onConnect
Client-->>Hook: update state (status, mode, messages)
React->>Hook: startListening()
Hook->>Client: startListening()
Client->>Connection: startAudioCapture()
Connection-->>Client: transcript events → messages
Client-->>Hook: propagate messages to React
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Co-Authored-By: Warp <agent@warp.dev>
|
View your CI Pipeline Execution ↗ for commit 18ce05c
☁️ Nx Cloud last updated this comment at |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 17
Note
Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.
🟡 Minor comments (10)
packages/typescript/ai-elevenlabs/README.md-48-62 (1)
48-62:⚠️ Potential issue | 🟡 Minor
messagesis destructured but never used in the React example.
messagesis included in the destructuring on Line 48 but does not appear in the JSX. Either remove it to avoid misleading readers about the API surface, or add a minimal rendering example (e.g., a message list) to demonstrate its intended use.📝 Option A — remove unused destructuring
- const { status, mode, messages, connect, disconnect } = useRealtimeChat({ + const { status, mode, connect, disconnect } = useRealtimeChat({🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/README.md` around lines 48 - 62, The example destructures messages from useRealtimeChat (symbols: messages, useRealtimeChat, elevenlabsRealtime, connect, disconnect, status, mode) but never uses it; either remove messages from the destructuring to avoid implying it's used, or add a minimal rendering of it (e.g., map over messages to output a list of message.text or message.content) inside the JSX so the example demonstrates its intended use—update the return JSX accordingly and keep connect/disconnect and status/mode logic unchanged.packages/typescript/ai-elevenlabs/README.md-7-9 (1)
7-9:⚠️ Potential issue | 🟡 Minor
@tanstack/ai-reactis missing from the installation instructions.The React section (Lines 44–63) imports
useRealtimeChatfrom@tanstack/ai-react, but that package is not listed in thenpm installcommand. Users following the README sequentially will encounter a missing module error.📝 Proposed fix
-npm install `@tanstack/ai-elevenlabs` `@tanstack/ai` `@tanstack/ai-client` +npm install `@tanstack/ai-elevenlabs` `@tanstack/ai` `@tanstack/ai-client` `@tanstack/ai-react`🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/README.md` around lines 7 - 9, Update the installation command in README.md to include the missing React package: add `@tanstack/ai-react` to the npm install list so the React example that imports useRealtimeChat from `@tanstack/ai-react` will not fail; locate the npm install snippet in the README (the block that currently lists `@tanstack/ai-elevenlabs`, `@tanstack/ai`, `@tanstack/ai-client`) and append `@tanstack/ai-react` to that list.packages/typescript/ai-elevenlabs/package.json-39-39 (1)
39-39:⚠️ Potential issue | 🟡 Minor
test:libscript will run Vitest in watch mode — should bevitest run.The
ai-openaipackage uses"vitest run"for itstest:libscript, which exits after one pass. Using bare"vitest"here defaults to watch mode and will hang in CI.Proposed fix
- "test:lib": "vitest", + "test:lib": "vitest run",🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/package.json` at line 39, Update the package.json "test:lib" npm script so it runs Vitest in single-run mode instead of watch mode; locate the "test:lib" script entry and change its value from "vitest" to "vitest run" so CI will exit after one pass (match the ai-openai package pattern).packages/typescript/ai-elevenlabs/package.json-50-54 (1)
50-54:⚠️ Potential issue | 🟡 MinorMissing
vitedev dependency.The
buildscript usesvite build, butviteis not listed indevDependencies. Other packages likeai-openaiinclude it. Without it, the build will fail ifviteisn't hoisted from a sibling.Proposed fix
"devDependencies": { "@tanstack/ai": "workspace:*", "@tanstack/ai-client": "workspace:*", - "@vitest/coverage-v8": "4.0.14" + "@vitest/coverage-v8": "4.0.14", + "vite": "^7.2.7" }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/package.json` around lines 50 - 54, package.json's build script calls "vite build" but "vite" is missing from devDependencies; add "vite" to the devDependencies section (use the same version string used in sibling packages like ai-openai to keep hoisting consistent), then reinstall so "vite" is available for the build script; update package.json's "devDependencies" to include "vite" and run your package manager to restore node_modules.packages/typescript/ai-elevenlabs/src/realtime/token.ts-81-82 (1)
81-82:⚠️ Potential issue | 🟡 MinorValidate
signed_urlbefore using it — silentundefinedtoken if the response shape changes.
response.json()returnsany; if the ElevenLabs API returns a 2xx with an unexpected shape (field rename, error envelope, rate-limit wrapper),signedUrlisundefinedand propagates astoken: undefinedwithout an immediate, readable error.🔧 Proposed fix
- const data = await response.json() - const signedUrl = data.signed_url as string + const data = await response.json() + const signedUrl: string | undefined = data?.signed_url + if (!signedUrl) { + throw new Error( + `ElevenLabs signed URL response missing 'signed_url' field: ${JSON.stringify(data)}`, + ) + }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/src/realtime/token.ts` around lines 81 - 82, Validate that data.signed_url is present and a non-empty string before assigning to signedUrl and using it as token: after const data = await response.json() check typeof data?.signed_url === 'string' && data.signed_url.trim() !== '' and if not, throw or return a clear error (include response.status and JSON body) so callers don't receive a silent undefined token; update the code path that builds the token to rely on this validated signedUrl variable.docs/architecture/realtime-chat.md-19-63 (1)
19-63:⚠️ Potential issue | 🟡 MinorAdd a language specifier to the fenced code block.
The ASCII architecture diagram code block lacks a language identifier, which triggers a markdownlint warning (MD040). Use
textorplaintextto satisfy the linter.-``` +```text ┌─────────────────────────────────────────────────────────────────┐🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/architecture/realtime-chat.md` around lines 19 - 63, The fenced ASCII diagram block (the triple backticks before the diagram showing "React Application", "RealtimeClient", "Provider Adapters", etc.) is missing a language specifier and triggers MD040; update the opening fence from ``` to ```text (or ```plaintext) so the diagram is treated as plain text by the linter and the MD040 warning is resolved.packages/typescript/ai-openai/src/realtime/token.ts-1-7 (1)
1-7:⚠️ Potential issue | 🟡 MinorFix import order per ESLint rule.
ESLint reports that
../utilsshould be imported before the type import of@tanstack/ai.Proposed fix
+import { getOpenAIApiKeyFromEnv } from '../utils' import type { RealtimeToken, RealtimeTokenAdapter, Tool } from '@tanstack/ai' -import { getOpenAIApiKeyFromEnv } from '../utils' import type {🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/token.ts` around lines 1 - 7, Reorder the top-level imports so that the runtime import getOpenAIApiKeyFromEnv from '../utils' appears before the type-only import from '@tanstack/ai'; specifically move the import of getOpenAIApiKeyFromEnv to come before the import type { RealtimeToken, RealtimeTokenAdapter, Tool } line in this file (token.ts) and keep the OpenAIRealtimeModel/OpenAIRealtimeSessionResponse/OpenAIRealtimeTokenOptions type imports unchanged.examples/ts-react-chat/src/routes/api.realtime-token.ts-5-11 (1)
5-11:⚠️ Potential issue | 🟡 MinorFix import ordering per ESLint rules.
ESLint reports two issues: members in the
@/lib/realtime-toolsimport should be sorted alphabetically, and thezodimport should precede@/lib/realtime-tools.Proposed fix
+import * as z from 'zod' import { getCurrentTimeToolDef, getWeatherToolDef, - setReminderToolDef, searchKnowledgeToolDef, + setReminderToolDef, } from '@/lib/realtime-tools' -import * as z from 'zod'🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/ts-react-chat/src/routes/api.realtime-token.ts` around lines 5 - 11, Move the external import for zod above the local module and alphabetize the named exports from '@/lib/realtime-tools'; specifically place "import * as z from 'zod'" before the other import and reorder the members to: getCurrentTimeToolDef, getWeatherToolDef, searchKnowledgeToolDef, setReminderToolDef so the import list is sorted per ESLint.packages/typescript/ai-openai/src/realtime/adapter.ts-124-130 (1)
124-130:⚠️ Potential issue | 🟡 MinorRemove debug
console.logstatements before merging.Multiple
console.logcalls are left in (lines 125, 128, 178, 180, 514). These will pollute the browser console for every user of the library. Gate them behind thedebugoption or remove them entirely.Also applies to: 178-180, 512-514
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 124 - 130, Remove or gate the stray console.log calls in the realtime adapter: locate the pc.ontrack handler and other places where console.log is used (e.g., the lines printing '[Realtime] ontrack event:', output analyser creation, and the other occurrences around the 178-180 and 512-514 regions) and either delete those console.log statements or wrap them with the library's debug flag (e.g., if (options?.debug) { ... }) or use the existing logger; ensure references like pc.ontrack, setupOutputAudioAnalysis and outputAnalyser remain unchanged and only the logging behavior is modified.packages/typescript/ai-openai/src/realtime/adapter.ts-378-405 (1)
378-405:⚠️ Potential issue | 🟡 Minor
disconnectdoes not null out analyser/source nodes, risking use-after-close.After
audioContext.close()at line 400,inputAnalyser,outputAnalyser,inputSource, andoutputSourcestill reference closed nodes. IfgetAudioVisualization()getters are called after disconnect (e.g., due to a pending animation frame), calling methods on these closed nodes could throw.Null out audio nodes on disconnect
if (audioContext) { await audioContext.close() audioContext = null + inputAnalyser = null + outputAnalyser = null + inputSource = null + outputSource = null }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 378 - 405, The disconnect() method leaves references to closed audio nodes which can cause use-after-close errors; after awaiting audioContext.close() in disconnect(), explicitly set inputAnalyser, outputAnalyser, inputSource, and outputSource to null (and any related visualization state used by getAudioVisualization()) and ensure any pending animation frames/handlers that may reference them are cancelled or guarded so subsequent calls cannot invoke methods on closed nodes; update the disconnect() implementation to null out these symbols (inputAnalyser, outputAnalyser, inputSource, outputSource, and any visualizer handles) immediately after closing audioContext.
🧹 Nitpick comments (14)
packages/typescript/ai-elevenlabs/README.md (1)
65-72: Consider documenting browser/microphone prerequisites.The package targets realtime voice conversations over WebRTC/WebSocket, which requires microphone access and a browser environment. The Requirements section currently only covers the ElevenLabs account and agent ID; a note about browser microphone permissions (and that this is a browser-only API) would help developers avoid unexpected failures, especially in SSR contexts.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/README.md` around lines 65 - 72, Update the README's Requirements section (near the "## Environment Variables" and "## Requirements" headings) to include a short note that this package is browser-only and requires microphone access/permissions for realtime WebRTC/WebSocket voice conversations; mention that ELEVENLABS_API_KEY is for server-side token generation and warn about SSR contexts where microphone/browser APIs are unavailable to prevent unexpected runtime failures.packages/typescript/ai-elevenlabs/vite.config.ts (2)
18-18:'node_modules/'incoverage.excludeis redundant.The v8 coverage provider already excludes
node_modulesby default.♻️ Proposed cleanup
exclude: [ - 'node_modules/', 'dist/',🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/vite.config.ts` at line 18, Remove the redundant 'node_modules/' entry from the coverage.exclude array in vite.config.ts (the array used to configure the v8 coverage provider); locate the coverage config (symbol: coverage.exclude) and delete that string element so the default v8 behavior remains and the exclusion list is not duplicated.
8-8:dir: './'is redundant.
'./'is the default when the config file lives at the project root. Removing it avoids confusion about whether it's intentionally overriding a parent-level setting.♻️ Proposed cleanup
test: { name: packageJson.name, - dir: './', watch: false,🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/vite.config.ts` at line 8, Remove the redundant property "dir: './'" from the Vite config (in packages/typescript/ai-elevenlabs/vite.config.ts) — locate the object that contains dir: './' and delete that key so the config uses the default project-root directory behavior; no other changes needed.packages/typescript/ai-client/src/realtime-client.ts (2)
314-336: Token refresh could spin ifgetTokenreturns already-near-expired tokens.If
getToken()consistently returns tokens withexpiresAtless thanTOKEN_REFRESH_BUFFER_MSin the future,refreshInis0andscheduleTokenRefresh→refreshToken→scheduleTokenRefreshcreates a tight loop of immediate refreshes.Consider adding a minimum delay or a retry-backoff to guard against this.
Proposed fix
+ const MIN_REFRESH_INTERVAL_MS = 5_000 + private scheduleTokenRefresh(): void { if (!this.token) return const timeUntilExpiry = this.token.expiresAt - Date.now() - const refreshIn = Math.max(0, timeUntilExpiry - TOKEN_REFRESH_BUFFER_MS) + const refreshIn = Math.max(MIN_REFRESH_INTERVAL_MS, timeUntilExpiry - TOKEN_REFRESH_BUFFER_MS) this.tokenRefreshTimeout = setTimeout(() => { this.refreshToken() }, refreshIn) }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-client/src/realtime-client.ts` around lines 314 - 336, The token refresh can spin if getToken() returns tokens already expiring within TOKEN_REFRESH_BUFFER_MS; modify scheduleTokenRefresh() and refreshToken() to enforce a minimum delay/backoff: when computing refreshIn (currently using token.expiresAt - Date.now() - TOKEN_REFRESH_BUFFER_MS) clamp it with a MIN_REFRESH_INTERVAL_MS (e.g., a few seconds) and a MAX_REFRESH_INTERVAL_MS if desired, and when refreshToken() catches an error or receives a token that would schedule immediate refresh (refreshIn <= 0), schedule the next attempt using an increasing retryBackoff (exponential or capped linear) stored on the instance (reset on success). Update uses of tokenRefreshTimeout, scheduleTokenRefresh, refreshToken, TOKEN_REFRESH_BUFFER_MS, and getToken to implement this guard.
63-63: Remove debugconsole.logstatements from library code.Lines 63, 105, and 254 contain debug logging that will appear in every consumer's console. Library packages should either omit these or use a conditional debug logger.
Proposed fix
- console.log(`[RealtimeClient #${this.instanceId}] Created`) ... - console.log(`[RealtimeClient #${this.instanceId}] Connection established:`, !!this.connection) ... - console.log(`[RealtimeClient #${this.instanceId}] audio getter, connection:`, !!this.connection)Also applies to: 105-105, 254-254
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-client/src/realtime-client.ts` at line 63, Remove the stray debug console.log statements from the RealtimeClient class (the creation log that prints `[RealtimeClient #${this.instanceId}] Created` and the other console.log calls around the same class at the other locations). Replace them with either no-op removal or routed debug logging using the module's logger/debug facility (e.g., use an existing logger on the class or a conditional debug utility tied to env/flag) so library consumers don't get unsolicited console output; update references inside the RealtimeClient constructor/initialization and the methods where those console.logs appear to call the chosen logger (or remove entirely) instead of console.log.examples/ts-react-chat/package.json (1)
20-20: Dependency is out of alphabetical order.
@tanstack/ai-elevenlabsshould be placed between@tanstack/ai-clientand@tanstack/ai-geminito maintain the sorted order of the other@tanstackdependencies.Proposed fix
"@tanstack/ai-client": "workspace:*", + "@tanstack/ai-elevenlabs": "workspace:*", "@tanstack/ai-gemini": "workspace:*", "@tanstack/ai-grok": "workspace:*", "@tanstack/ai-ollama": "workspace:*", "@tanstack/ai-openai": "workspace:*", - "@tanstack/ai-elevenlabs": "workspace:*", "@tanstack/ai-openrouter": "workspace:*",🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/ts-react-chat/package.json` at line 20, The dependency "@tanstack/ai-elevenlabs" is out of alphabetical order in package.json; move the line with "@tanstack/ai-elevenlabs": "workspace:*" so it appears between "@tanstack/ai-client" and "@tanstack/ai-gemini" to keep all "@tanstack" entries sorted; update the dependencies block in package.json accordingly and run a quick yarn/npm install to ensure lockfile consistency.packages/typescript/ai/src/index.ts (1)
82-104: Consider a dedicated/realtimesubpath export to keep the main barrel lean.All 20 realtime symbols (including the runtime
realtimeTokenfunction) are added to the root entry point. Consumers who never use realtime features still have to parse/consider these exports; tree-shakers help for values but some bundlers treat the whole barrel as one chunk. A@tanstack/ai/realtimesubpath (viapackage.jsonexports) would let realtime-only code be cleanly separated, consistent with the existing/adapterspattern in the monorepo.Based on learnings: "Applies to
packages/typescript/*/src/index.ts: Export tree-shakeable adapters with clear subpath exports inpackage.json(e.g.,@tanstack/ai/adapters,@tanstack/ai-openai/adapters) to minimize bundle size."🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai/src/index.ts` around lines 82 - 104, Remove the realtime exports from the root barrel (the list starting with realtimeToken and the Realtime* types in packages/typescript/ai/src/index.ts) and re-export them from a dedicated subpath; create a new entry re-export file (e.g., packages/typescript/ai/src/realtime/index.ts) that exports realtimeToken and all Realtime* types, then update package.json "exports" to add a "./realtime" subpath pointing to the built realtime entry so consumers can import "@tanstack/ai/realtime"; ensure the build output includes the new realtime entry so the new subpath resolves at runtime.examples/ts-react-chat/src/routes/realtime.tsx (1)
134-138: Variableerrorshadows the outererrorfromuseRealtimeChat.The
errordeclared in the catch block at line 136 shadows the destructurederrorfromuseRealtimeChatat line 116. This works due to block scoping but harms readability. Consider renaming toerrBodyorerrorData.Proposed fix
if (!response.ok) { - const error = await response.json() - throw new Error(error.error || 'Failed to get token') + const errBody = await response.json() + throw new Error(errBody.error || 'Failed to get token') }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/ts-react-chat/src/routes/realtime.tsx` around lines 134 - 138, The response error variable shadows the outer `error` from `useRealtimeChat`; in the fetch failure branch inside the token retrieval logic rename the local `error` to a non-conflicting name like `errBody` or `errorData` (where `const error = await response.json()` is declared) and update the subsequent throw to use that new name (e.g., `throw new Error(errorData.error || 'Failed to get token')`) so `useRealtimeChat`'s `error` remains unshadowed and readability is preserved.packages/typescript/ai-elevenlabs/src/realtime/adapter.ts (2)
58-59: Shared mutableUint8Arrayfallback buffers can be corrupted by callers.
emptyFrequencyDataandemptyTimeDomainDataare allocated once and returned by reference fromgetAudioVisualization(). If any consumer writes to the returned array (e.g., applying a transform in-place), subsequent callers will see the mutated data. The same pattern exists in the OpenAI adapter.A low-risk concern for now, but worth noting. A defensive approach would be to return a new copy each time, or document that the returned arrays must not be mutated.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/src/realtime/adapter.ts` around lines 58 - 59, The shared fallback buffers emptyFrequencyData and emptyTimeDomainData used by getAudioVisualization() are returned by reference and can be mutated by callers; change getAudioVisualization() to return fresh copies instead of the shared Uint8Array instances (e.g., construct a new Uint8Array from the fallback buffer or use .slice()) so each caller receives an independent buffer, and apply the same defensive copying approach in the OpenAI adapter where the same pattern exists.
148-168: Stub methods silently degrade — consider throwing or returning a sentinel to surface unsupported operations.
sendText,sendToolResult, andupdateSessionallconsole.warnbut otherwise succeed silently. A caller using the genericRealtimeConnectioninterface has no programmatic way to know the operation was a no-op. If these are genuinely unsupported, throwing a clear error (or returning afalse/Promise.reject) would prevent subtle bugs where the caller assumes the action succeeded.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/src/realtime/adapter.ts` around lines 148 - 168, These stub methods (sendText, sendToolResult, updateSession) currently only console.warn and silently succeed; update them to surface unsupported operations to callers by throwing a clear Error (e.g., new Error('ElevenLabs realtime does not support sendText')) or returning a rejected Promise with that Error to match the method's async contract; ensure the behavior is consistent with the RealtimeConnection interface (adjust signatures to return Promise.reject if these are async) and include descriptive messages that reference the operation name so callers can programmatically detect failures.packages/typescript/ai-react/src/use-realtime-chat.ts (2)
200-206:setVADModeonly updates local state — the TODO for propagating to the session is still open.Calling
setVADModewhile connected will show the new mode in the UI but won't actually change the server-side VAD behavior, leading to a confusing discrepancy.Would you like me to open an issue to track implementing the session update propagation?
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-react/src/use-realtime-chat.ts` around lines 200 - 206, setVADMode currently only calls setVADModeState and doesn't propagate the change to the active session, causing UI/server mismatch; modify the useCallback to include relevant dependencies (e.g., session and isConnected) and, when connected, call the session API to update the server-side config (for example session.updateConfig or session.send/update method to set vadMode to newMode) before/after calling setVADModeState, handling errors appropriately so the server and UI stay in sync.
110-114: Explicitly capture and unsubscribe fromonStateChangesubscription for better cleanup handling.The return value from
onStateChange()is discarded at lines 110-114. While the current code is safe becausedestroy()clears all state change callbacks on unmount, explicitly storing and calling the unsubscribe function provides more robust cleanup:const unsubscribe = clientRef.current.onStateChange((state) => { setPendingUserTranscript(state.pendingUserTranscript) setPendingAssistantTranscript(state.pendingAssistantTranscript) })Then clean it up in the unmount effect or store it for explicit cleanup. This makes the subscription lifecycle explicit and less dependent on the internal implementation of
destroy().🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-react/src/use-realtime-chat.ts` around lines 110 - 114, The onStateChange subscription return value is discarded; explicitly capture and unsubscribe it to ensure proper cleanup: when calling clientRef.current.onStateChange(...) assign the returned unsubscribe function to a variable (e.g., unsubscribe) and then call that unsubscribe during the component cleanup (or when calling destroy()), ensuring you still update setPendingUserTranscript and setPendingAssistantTranscript from the callback and not rely solely on clientRef.current.destroy() to clear callbacks.packages/typescript/ai-openai/src/realtime/adapter.ts (2)
516-537:calculateLevelallocates a newUint8Array(fftSize)on every invocation — hot-path allocation concern.
calculateLevelis called via theinputLevelandoutputLevelgetters, which the React hook polls on every animation frame (~60 fps). Each call allocates anew Uint8Array(2048), creating ~240 KB/s of short-lived garbage (2 analyzers × 2048 bytes × 60 fps). The frequency and time-domain methods (lines 550–576) have the same pattern.Consider pre-allocating reusable buffers:
Pre-allocate buffers for visualization
+ // Pre-allocated buffers for visualization (avoid GC pressure in hot path) + const inputLevelBuffer = new Uint8Array(2048) + const outputLevelBuffer = new Uint8Array(2048) + function calculateLevel(analyser: AnalyserNode): number { - const data = new Uint8Array(analyser.fftSize) + const data = analyser === inputAnalyser ? inputLevelBuffer : outputLevelBuffer analyser.getByteTimeDomainData(data)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 516 - 537, The calculateLevel function is allocating a new Uint8Array on every frame; fix by pre-allocating and reusing buffers per AnalyserNode instead of creating new arrays each call: add a module-scoped Map<AnalyserNode, Uint8Array> (or attach a Symbol-keyed property) to cache a buffer for each analyser, on first use create Uint8Array(analyser.fftSize) and store it, then in calculateLevel (and the frequency/time-domain helper methods used by the inputLevel/outputLevel getters) retrieve and reuse that cached buffer for analyser.getByteTimeDomainData / getByteFrequencyData to eliminate per-frame allocations. Ensure buffer size matches analyser.fftSize if it can change (recreate and replace cache entry when size differs).
260-299:response.donehandler doesn't handle missingresponse.outputgracefully.At line 267,
response.outputis cast toArray<...>without a null check. The fallbackoutput || []on line 277 only guards the iteration, but ifresponseitself is undefined/null (e.g., malformed server event), line 266 will throw. Also, if there arefunction_calltype items in output, they're silently skipped — consider whether tool-call parts should be included in the message.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 260 - 299, The handler for 'response.done' accesses event.response.output without guarding against a missing response; update the block around currentMode/emit/currentMessageId to safely handle missing or non-array data by null-checking event.response (or defaulting to an empty object) and validating response.output as an array before iterating (e.g., const response = event.response ?? {}; const output = Array.isArray(response.output) ? response.output : []), and while extracting parts extend the item-type handling to also capture 'function_call' (or tool-call) items into message.parts (e.g., add a branch for item.type === 'function_call' that pushes a part with a clear type/name/arguments) so tool calls aren't silently skipped; leave emit('message_complete', { message }) and currentMessageId reset logic unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@examples/ts-react-chat/src/routes/realtime.tsx`:
- Line 108: The adapter variable (created with openaiRealtime() or
elevenlabsRealtime()) is recreated on every render; memoize it (e.g., with
React's useMemo or a useRef + effect) so it only changes when provider changes,
and ensure the RealtimeClient that consumes it is recreated or reconnected when
that memoized adapter changes. Locate references to adapter, provider,
openaiRealtime, elevenlabsRealtime, and the RealtimeClient instantiation and
update them so the adapter is stable across renders and is replaced only when
provider changes, triggering a clean disconnect/reconnect of the RealtimeClient.
In `@packages/typescript/ai-client/src/realtime-client.ts`:
- Around line 1-8: There are two separate type imports from the same module
(`@tanstack/ai`) — one importing AudioVisualization, RealtimeMessage,
RealtimeMode, RealtimeStatus, RealtimeToken and another importing AnyClientTool
— which triggers import/no-duplicates; fix it by merging them into a single
type-only import (e.g., import type { AudioVisualization, RealtimeMessage,
RealtimeMode, RealtimeStatus, RealtimeToken, AnyClientTool } from
'@tanstack/ai') and remove the duplicate import statement so all types come from
one import declaration.
- Around line 383-402: The handler registered in the
this.connection.on('tool_call', ...) callback currently ignores requests when
this.clientTools.get(toolName) returns undefined, causing the provider to wait
forever; update the callback so that if the tool is not found (i.e., tool is
falsy or lacks an execute method) you immediately call
this.connection?.sendToolResult(toolCallId, JSON.stringify({ error: `tool
"${toolName}" not registered` })) (and optionally log via processLogger or
this.logger) before returning, otherwise proceed to call tool.execute as before;
ensure you reference toolCallId, toolName, this.clientTools.get, and
sendToolResult in the change.
- Around line 281-284: The destroy() method currently calls async disconnect()
without awaiting, causing stateChangeCallbacks.clear() to run before disconnect
completes; change destroy to await the disconnect call (e.g., make destroy async
and await this.disconnect()) so updateState and any final `'idle'` notifications
run before clearing; if you cannot change the API to async, instead chain the
Promise (return this.disconnect().finally(() =>
this.stateChangeCallbacks.clear())) or at minimum call
this.disconnect().catch(() => {}).finally(() =>
this.stateChangeCallbacks.clear()) to avoid unhandled rejections and ensure
cleanup happens after disconnect; reference methods: destroy(), disconnect(),
updateState, stateChangeCallbacks, and connection.disconnect().
- Around line 163-169: In startListening(), await the Promise returned by
this.connection.startAudioCapture() and only call this.updateState({ mode:
'listening' }) after the await succeeds; if startAudioCapture() rejects, catch
the error (e.g., in a try/catch), log or surface it via your logger or
this.updateState to reflect failure, and avoid flipping the mode to 'listening'
on error — modify the startListening() implementation to use async/await with
try/catch around this.connection.startAudioCapture(), referencing
startListening(), this.connection.startAudioCapture(), and this.updateState().
In `@packages/typescript/ai-client/src/realtime-types.ts`:
- Around line 1-11: Merge the two imports from '@tanstack/ai' into a single
import statement: include AudioVisualization, RealtimeEvent,
RealtimeEventHandler, RealtimeMessage, RealtimeMode, RealtimeSessionConfig,
RealtimeStatus, RealtimeToken and AnyClientTool together in one import to
resolve the import/no-duplicates ESLint error (locate the current separate
imports in realtime-types.ts and combine them into one).
- Around line 21-72: Change all interface method signatures in RealtimeAdapter
and RealtimeConnection to function property signatures (e.g., connect: (token:
RealtimeToken) => Promise<RealtimeConnection>) instead of shorthand method
forms; update every lifecycle, audio I/O, text input, tool result, session
management, events, and audio visualization members (disconnect,
startAudioCapture, stopAudioCapture, sendText, sendToolResult, updateSession,
interrupt, on, getAudioVisualization) to the form methodName: (args...) =>
ReturnType so the types are checked as function properties and satisfy
`@typescript-eslint/method-signature-style`.
In `@packages/typescript/ai-elevenlabs/src/realtime/adapter.ts`:
- Around line 79-125: The initial emits inside the Conversation.startSession
callbacks (specifically the onConnect handler that calls emit('status_change', {
status: 'connected' }) and emit('mode_change', { mode: 'listening' })) run
synchronously during await and get dropped before the caller can register
handlers; wrap those initial emit calls in queueMicrotask() (or equivalent
microtask deferral) inside the onConnect callback so the connection object is
returned and handlers can be attached before the events fire; update the
onConnect handler in the Conversation.startSession options to defer the
emit(...) calls via queueMicrotask while leaving other callbacks
(onDisconnect/onModeChange/onMessage/onError) unchanged.
In `@packages/typescript/ai-elevenlabs/src/realtime/token.ts`:
- Around line 9-29: getElevenLabsApiKey currently reads from window.env which
can leak secrets if bundled to the browser; remove the browser fallback so
getElevenLabsApiKey only reads server-safe sources (process.env or explicit
injection), and update elevenlabsRealtimeToken to accept an explicit apiKey
(e.g., via ElevenLabsRealtimeTokenOptions) or use a safe runtime global like
globalThis.ELEVENLABS_API_KEY for edge runtimes; ensure getElevenLabsApiKey
throws the same clear error when no server-side key is provided and update
callers (elevenlabsRealtimeToken) to pass the api key explicitly rather than
relying on window.env.
In `@packages/typescript/ai-openai/src/index.ts`:
- Around line 103-121: The package is missing dedicated subpath exports for the
realtime adapter (and other adapters) so tree-shaking fails; add subpath export
entries in package.json for "./realtime" (and similarly for "./text",
"./summarize", "./image", "./video", "./tts", "./transcription") that map
import/require/types to the built outputs (e.g., "import":
"./dist/esm/realtime/index.js", "require": "./dist/cjs/realtime/index.js",
"types": "./dist/types/realtime/index.d.ts") so the top-level exported symbols
openaiRealtimeToken, openaiRealtime and the exported types OpenAIRealtimeVoice,
OpenAIRealtimeModel, OpenAIRealtimeTokenOptions, OpenAIRealtimeOptions,
OpenAITurnDetection, OpenAISemanticVADConfig, OpenAIServerVADConfig resolve via
the "./realtime" subpath and become tree-shakeable.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts`:
- Around line 132-180: Wrap the SDP fetch/answer negotiation (the fetch to
OPENAI_REALTIME_URL, response handling, and pc.setRemoteDescription) in a
try/catch/finally so that if any of fetch, awaiting the response, or
pc.setRemoteDescription fails you explicitly clean up resources: stop all tracks
on localStream (localStream.getTracks().forEach(t => t.stop())), close the
RTCPeerConnection instance pc (pc.close()), close the dataChannel if open
(dataChannel.close()), and tear down any input audio analysis state created by
setupInputAudioAnalysis/inputAnalyser (undo whatever setupInputAudioAnalysis
does or call a new cleanup function). Ensure
setupInputAudioAnalysis(localStream) and the console logs only run after
successful negotiation (or move setup into the try after successful
setRemoteDescription), and rethrow the original error after cleanup so callers
still receive the failure.
In `@packages/typescript/ai-openai/src/realtime/types.ts`:
- Around line 19-23: Replace the restricted preview-only union in
OpenAIRealtimeModel with the new production names and a flexible string pattern:
update the OpenAIRealtimeModel type to include 'gpt-realtime' and
'gpt-realtime-mini' (instead of the four preview names) and add the permissive
fallback (string & {}) so unknown/future model identifiers are accepted while
preserving IDE autocomplete for known models; modify the type definition for
OpenAIRealtimeModel accordingly.
In `@packages/typescript/ai-react/src/realtime-types.ts`:
- Around line 1-8: There are two separate imports from '@tanstack/ai' causing an
import/no-duplicates ESLint error; merge AnyClientTool into the existing import
that brings in RealtimeMessage, RealtimeMode, RealtimeStatus, and RealtimeToken
so all types from '@tanstack/ai' are imported in one statement (leave the
separate import type { RealtimeAdapter } from '@tanstack/ai-client' as-is).
Update the import that references RealtimeMessage, RealtimeMode, RealtimeStatus,
RealtimeToken to also include AnyClientTool.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts`:
- Around line 120-139: The updateLevels loop in useEffect calls
setInputLevel/setOutputLevel every requestAnimationFrame causing ~60fps
re-renders; modify updateLevels to throttle updates to ~15fps (or ~66ms) and
only call setInputLevel/setOutputLevel when the new value differs by a small
threshold (e.g., delta > 0.01) to avoid unnecessary renders. Concretely, inside
the updateLevels function (which uses animationFrameRef and clientRef), track
the last update timestamp (or frame count) and only perform the React state
updates when enough time has elapsed, and compute a small difference check
against previousLevel refs before calling setInputLevel/setOutputLevel; keep the
existing cleanup of animationFrameRef and the status === 'connected' start
condition.
- Around line 74-115: The RealtimeClient is being created with getToken,
adapter, and tools captured once (optionsRef.current.getToken,
optionsRef.current.adapter, optionsRef.current.tools) which freezes stale
values; change the RealtimeClient construction so those fields are passed as
thin wrapper functions that read from optionsRef.current at invocation time
(e.g., getToken: (...args) => optionsRef.current.getToken?.(...args), adapter:
(...args) => optionsRef.current.adapter?.(...args) or adapter: () =>
optionsRef.current.adapter, and similarly for tools) so the client uses
up-to-date values from optionsRef when methods are called; keep existing
callbacks (onMessage/onError/etc.), clientRef.current and the onStateChange
subscription unchanged.
In `@packages/typescript/ai/src/realtime/types.ts`:
- Around line 56-61: Convert all shorthand interface method signatures to
function-property form: in RealtimeTokenAdapter change "generateToken():
Promise<RealtimeToken>" to a property-style signature like "generateToken: () =>
Promise<RealtimeToken>"; do the same for every shorthand method in the
AudioVisualization interface (replace e.g. "start(): void" with "start: () =>
void", "stop(): void" with "stop: () => void", etc.), ensuring return types and
parameter lists are preserved exactly.
---
Nitpick comments:
In `@examples/ts-react-chat/package.json`:
- Line 20: The dependency "@tanstack/ai-elevenlabs" is out of alphabetical order
in package.json; move the line with "@tanstack/ai-elevenlabs": "workspace:*" so
it appears between "@tanstack/ai-client" and "@tanstack/ai-gemini" to keep all
"@tanstack" entries sorted; update the dependencies block in package.json
accordingly and run a quick yarn/npm install to ensure lockfile consistency.
In `@examples/ts-react-chat/src/routes/realtime.tsx`:
- Around line 134-138: The response error variable shadows the outer `error`
from `useRealtimeChat`; in the fetch failure branch inside the token retrieval
logic rename the local `error` to a non-conflicting name like `errBody` or
`errorData` (where `const error = await response.json()` is declared) and update
the subsequent throw to use that new name (e.g., `throw new
Error(errorData.error || 'Failed to get token')`) so `useRealtimeChat`'s `error`
remains unshadowed and readability is preserved.
In `@packages/typescript/ai-client/src/realtime-client.ts`:
- Around line 314-336: The token refresh can spin if getToken() returns tokens
already expiring within TOKEN_REFRESH_BUFFER_MS; modify scheduleTokenRefresh()
and refreshToken() to enforce a minimum delay/backoff: when computing refreshIn
(currently using token.expiresAt - Date.now() - TOKEN_REFRESH_BUFFER_MS) clamp
it with a MIN_REFRESH_INTERVAL_MS (e.g., a few seconds) and a
MAX_REFRESH_INTERVAL_MS if desired, and when refreshToken() catches an error or
receives a token that would schedule immediate refresh (refreshIn <= 0),
schedule the next attempt using an increasing retryBackoff (exponential or
capped linear) stored on the instance (reset on success). Update uses of
tokenRefreshTimeout, scheduleTokenRefresh, refreshToken,
TOKEN_REFRESH_BUFFER_MS, and getToken to implement this guard.
- Line 63: Remove the stray debug console.log statements from the RealtimeClient
class (the creation log that prints `[RealtimeClient #${this.instanceId}]
Created` and the other console.log calls around the same class at the other
locations). Replace them with either no-op removal or routed debug logging using
the module's logger/debug facility (e.g., use an existing logger on the class or
a conditional debug utility tied to env/flag) so library consumers don't get
unsolicited console output; update references inside the RealtimeClient
constructor/initialization and the methods where those console.logs appear to
call the chosen logger (or remove entirely) instead of console.log.
In `@packages/typescript/ai-elevenlabs/README.md`:
- Around line 65-72: Update the README's Requirements section (near the "##
Environment Variables" and "## Requirements" headings) to include a short note
that this package is browser-only and requires microphone access/permissions for
realtime WebRTC/WebSocket voice conversations; mention that ELEVENLABS_API_KEY
is for server-side token generation and warn about SSR contexts where
microphone/browser APIs are unavailable to prevent unexpected runtime failures.
In `@packages/typescript/ai-elevenlabs/src/realtime/adapter.ts`:
- Around line 58-59: The shared fallback buffers emptyFrequencyData and
emptyTimeDomainData used by getAudioVisualization() are returned by reference
and can be mutated by callers; change getAudioVisualization() to return fresh
copies instead of the shared Uint8Array instances (e.g., construct a new
Uint8Array from the fallback buffer or use .slice()) so each caller receives an
independent buffer, and apply the same defensive copying approach in the OpenAI
adapter where the same pattern exists.
- Around line 148-168: These stub methods (sendText, sendToolResult,
updateSession) currently only console.warn and silently succeed; update them to
surface unsupported operations to callers by throwing a clear Error (e.g., new
Error('ElevenLabs realtime does not support sendText')) or returning a rejected
Promise with that Error to match the method's async contract; ensure the
behavior is consistent with the RealtimeConnection interface (adjust signatures
to return Promise.reject if these are async) and include descriptive messages
that reference the operation name so callers can programmatically detect
failures.
In `@packages/typescript/ai-elevenlabs/vite.config.ts`:
- Line 18: Remove the redundant 'node_modules/' entry from the coverage.exclude
array in vite.config.ts (the array used to configure the v8 coverage provider);
locate the coverage config (symbol: coverage.exclude) and delete that string
element so the default v8 behavior remains and the exclusion list is not
duplicated.
- Line 8: Remove the redundant property "dir: './'" from the Vite config (in
packages/typescript/ai-elevenlabs/vite.config.ts) — locate the object that
contains dir: './' and delete that key so the config uses the default
project-root directory behavior; no other changes needed.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts`:
- Around line 516-537: The calculateLevel function is allocating a new
Uint8Array on every frame; fix by pre-allocating and reusing buffers per
AnalyserNode instead of creating new arrays each call: add a module-scoped
Map<AnalyserNode, Uint8Array> (or attach a Symbol-keyed property) to cache a
buffer for each analyser, on first use create Uint8Array(analyser.fftSize) and
store it, then in calculateLevel (and the frequency/time-domain helper methods
used by the inputLevel/outputLevel getters) retrieve and reuse that cached
buffer for analyser.getByteTimeDomainData / getByteFrequencyData to eliminate
per-frame allocations. Ensure buffer size matches analyser.fftSize if it can
change (recreate and replace cache entry when size differs).
- Around line 260-299: The handler for 'response.done' accesses
event.response.output without guarding against a missing response; update the
block around currentMode/emit/currentMessageId to safely handle missing or
non-array data by null-checking event.response (or defaulting to an empty
object) and validating response.output as an array before iterating (e.g., const
response = event.response ?? {}; const output = Array.isArray(response.output) ?
response.output : []), and while extracting parts extend the item-type handling
to also capture 'function_call' (or tool-call) items into message.parts (e.g.,
add a branch for item.type === 'function_call' that pushes a part with a clear
type/name/arguments) so tool calls aren't silently skipped; leave
emit('message_complete', { message }) and currentMessageId reset logic
unchanged.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts`:
- Around line 200-206: setVADMode currently only calls setVADModeState and
doesn't propagate the change to the active session, causing UI/server mismatch;
modify the useCallback to include relevant dependencies (e.g., session and
isConnected) and, when connected, call the session API to update the server-side
config (for example session.updateConfig or session.send/update method to set
vadMode to newMode) before/after calling setVADModeState, handling errors
appropriately so the server and UI stay in sync.
- Around line 110-114: The onStateChange subscription return value is discarded;
explicitly capture and unsubscribe it to ensure proper cleanup: when calling
clientRef.current.onStateChange(...) assign the returned unsubscribe function to
a variable (e.g., unsubscribe) and then call that unsubscribe during the
component cleanup (or when calling destroy()), ensuring you still update
setPendingUserTranscript and setPendingAssistantTranscript from the callback and
not rely solely on clientRef.current.destroy() to clear callbacks.
In `@packages/typescript/ai/src/index.ts`:
- Around line 82-104: Remove the realtime exports from the root barrel (the list
starting with realtimeToken and the Realtime* types in
packages/typescript/ai/src/index.ts) and re-export them from a dedicated
subpath; create a new entry re-export file (e.g.,
packages/typescript/ai/src/realtime/index.ts) that exports realtimeToken and all
Realtime* types, then update package.json "exports" to add a "./realtime"
subpath pointing to the built realtime entry so consumers can import
"@tanstack/ai/realtime"; ensure the build output includes the new realtime entry
so the new subpath resolves at runtime.
| const messagesEndRef = useRef<HTMLDivElement>(null) | ||
|
|
||
| // Get the appropriate adapter based on provider | ||
| const adapter = provider === 'openai' ? openaiRealtime() : elevenlabsRealtime() |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
fd -g 'use-realtime-chat.ts' packages --exec cat {}Repository: TanStack/ai
Length of output: 6771
🏁 Script executed:
cat -n examples/ts-react-chat/src/routes/realtime.tsx | sed -n '100,120p'Repository: TanStack/ai
Length of output: 703
🏁 Script executed:
rg 'adapter' packages/typescript/ai-react/src -A 3 -B 1 -t tsRepository: TanStack/ai
Length of output: 2446
🏁 Script executed:
rg 'optionsRef.current.adapter' packages/typescript/ai-react/src -A 5 -B 5Repository: TanStack/ai
Length of output: 1161
🏁 Script executed:
grep -A 20 'new RealtimeClient' packages/typescript/ai-react/src/use-realtime-chat.tsRepository: TanStack/ai
Length of output: 799
🏁 Script executed:
rg 'class RealtimeClient' packages -A 50 | head -100Repository: TanStack/ai
Length of output: 4265
🏁 Script executed:
rg 'adapter' packages/typescript/ai-client/src -t ts | grep -i 'set\|change\|update'Repository: TanStack/ai
Length of output: 37
🏁 Script executed:
rg 'this.options.adapter' packages/typescript/ai-client/src/realtime-client.ts -A 2 -B 2Repository: TanStack/ai
Length of output: 257
🏁 Script executed:
rg 'options.adapter' packages/typescript/ai-client/src/realtime-client.ts -A 2 -B 2Repository: TanStack/ai
Length of output: 257
🏁 Script executed:
grep -n 'adapter' packages/typescript/ai-client/src/realtime-client.ts | head -20Repository: TanStack/ai
Length of output: 256
🏁 Script executed:
cat -n packages/typescript/ai-client/src/realtime-client.ts | sed -n '95,115p'Repository: TanStack/ai
Length of output: 853
🏁 Script executed:
rg 'disconnect|reconnect' packages/typescript/ai-client/src/realtime-client.ts -A 3 -B 1 | head -40Repository: TanStack/ai
Length of output: 672
🏁 Script executed:
cat -n packages/typescript/ai-react/src/use-realtime-chat.ts | sed -n '40,90p'Repository: TanStack/ai
Length of output: 2359
🏁 Script executed:
rg 'useEffect.*adapter|useEffect.*provider' examples/ts-react-chat/src/routes/realtime.tsxRepository: TanStack/ai
Length of output: 37
Memoize the adapter to prevent unnecessary recreation on each render.
The adapter is instantiated on every render. When provider changes, users must manually disconnect and reconnect to switch to the new adapter; otherwise the RealtimeClient uses the stale adapter reference from its initial creation.
Proposed fix
+import { useEffect, useMemo, useRef, useState } from 'react'
-import { useEffect, useRef, useState } from 'react'
...
- const adapter = provider === 'openai' ? openaiRealtime() : elevenlabsRealtime()
+ const adapter = useMemo(
+ () => provider === 'openai' ? openaiRealtime() : elevenlabsRealtime(),
+ [provider],
+ )📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| const adapter = provider === 'openai' ? openaiRealtime() : elevenlabsRealtime() | |
| const adapter = useMemo( | |
| () => provider === 'openai' ? openaiRealtime() : elevenlabsRealtime(), | |
| [provider], | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@examples/ts-react-chat/src/routes/realtime.tsx` at line 108, The adapter
variable (created with openaiRealtime() or elevenlabsRealtime()) is recreated on
every render; memoize it (e.g., with React's useMemo or a useRef + effect) so it
only changes when provider changes, and ensure the RealtimeClient that consumes
it is recreated or reconnected when that memoized adapter changes. Locate
references to adapter, provider, openaiRealtime, elevenlabsRealtime, and the
RealtimeClient instantiation and update them so the adapter is stable across
renders and is replaced only when provider changes, triggering a clean
disconnect/reconnect of the RealtimeClient.
Comment on lines
96
to
122
| try { | ||
| // Fetch token from server | ||
| this.token = await this.options.getToken() | ||
|
|
||
| // Schedule token refresh | ||
| this.scheduleTokenRefresh() | ||
|
|
||
| // Connect via adapter | ||
| this.connection = await this.options.adapter.connect(this.token) | ||
| console.log(`[RealtimeClient #${this.instanceId}] Connection established:`, !!this.connection) | ||
|
|
||
| // Subscribe to connection events | ||
| this.subscribeToConnectionEvents() | ||
|
|
||
| // Start audio capture if configured | ||
| if (this.options.autoCapture) { | ||
| await this.connection.startAudioCapture() | ||
| } | ||
|
|
||
| this.updateState({ status: 'connected', mode: 'listening' }) | ||
| this.options.onConnect?.() | ||
| } catch (error) { | ||
| const err = error instanceof Error ? error : new Error(String(error)) | ||
| this.updateState({ status: 'error', error: err }) | ||
| this.options.onError?.(err) | ||
| throw err | ||
| } |
Contributor
There was a problem hiding this comment.
Partial failure in connect(): if startAudioCapture throws, the connection is left open.
When autoCapture is enabled and startAudioCapture() fails (e.g., mic permission denied), the catch block sets status to 'error' but doesn't disconnect the already-established connection. This leaks the open WebRTC/WebSocket session.
Proposed fix — disconnect on partial failure
} catch (error) {
const err = error instanceof Error ? error : new Error(String(error))
+ // Clean up partial connection
+ if (this.connection) {
+ await this.connection.disconnect().catch(() => {})
+ this.connection = null
+ }
this.updateState({ status: 'error', error: err })
this.options.onError?.(err)
throw err
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| try { | |
| // Fetch token from server | |
| this.token = await this.options.getToken() | |
| // Schedule token refresh | |
| this.scheduleTokenRefresh() | |
| // Connect via adapter | |
| this.connection = await this.options.adapter.connect(this.token) | |
| console.log(`[RealtimeClient #${this.instanceId}] Connection established:`, !!this.connection) | |
| // Subscribe to connection events | |
| this.subscribeToConnectionEvents() | |
| // Start audio capture if configured | |
| if (this.options.autoCapture) { | |
| await this.connection.startAudioCapture() | |
| } | |
| this.updateState({ status: 'connected', mode: 'listening' }) | |
| this.options.onConnect?.() | |
| } catch (error) { | |
| const err = error instanceof Error ? error : new Error(String(error)) | |
| this.updateState({ status: 'error', error: err }) | |
| this.options.onError?.(err) | |
| throw err | |
| } | |
| try { | |
| // Fetch token from server | |
| this.token = await this.options.getToken() | |
| // Schedule token refresh | |
| this.scheduleTokenRefresh() | |
| // Connect via adapter | |
| this.connection = await this.options.adapter.connect(this.token) | |
| console.log(`[RealtimeClient #${this.instanceId}] Connection established:`, !!this.connection) | |
| // Subscribe to connection events | |
| this.subscribeToConnectionEvents() | |
| // Start audio capture if configured | |
| if (this.options.autoCapture) { | |
| await this.connection.startAudioCapture() | |
| } | |
| this.updateState({ status: 'connected', mode: 'listening' }) | |
| this.options.onConnect?.() | |
| } catch (error) { | |
| const err = error instanceof Error ? error : new Error(String(error)) | |
| // Clean up partial connection | |
| if (this.connection) { | |
| await this.connection.disconnect().catch(() => {}) | |
| this.connection = null | |
| } | |
| this.updateState({ status: 'error', error: err }) | |
| this.options.onError?.(err) | |
| throw err | |
| } |
Comment on lines
+163
to
+169
| startListening(): void { | ||
| if (!this.connection || this.state.status !== 'connected') { | ||
| return | ||
| } | ||
| this.connection.startAudioCapture() | ||
| this.updateState({ mode: 'listening' }) | ||
| } |
Contributor
There was a problem hiding this comment.
startListening() ignores the Promise from startAudioCapture().
startAudioCapture() returns Promise<void> (e.g., requesting mic permissions). The unawaited call means permission failures or errors are silently swallowed, and mode is set to 'listening' even if capture fails.
Proposed fix
- startListening(): void {
+ async startListening(): Promise<void> {
if (!this.connection || this.state.status !== 'connected') {
return
}
- this.connection.startAudioCapture()
- this.updateState({ mode: 'listening' })
+ try {
+ await this.connection.startAudioCapture()
+ this.updateState({ mode: 'listening' })
+ } catch (error) {
+ const err = error instanceof Error ? error : new Error(String(error))
+ this.updateState({ error: err })
+ this.options.onError?.(err)
+ }
}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-client/src/realtime-client.ts` around lines 163 - 169,
In startListening(), await the Promise returned by
this.connection.startAudioCapture() and only call this.updateState({ mode:
'listening' }) after the await succeeds; if startAudioCapture() rejects, catch
the error (e.g., in a try/catch), log or surface it via your logger or
this.updateState to reflect failure, and avoid flipping the mode to 'listening'
on error — modify the startListening() implementation to use async/await with
try/catch around this.connection.startAudioCapture(), referencing
startListening(), this.connection.startAudioCapture(), and this.updateState().
Comment on lines
+281
to
+284
| destroy(): void { | ||
| this.disconnect() | ||
| this.stateChangeCallbacks.clear() | ||
| } |
Contributor
There was a problem hiding this comment.
destroy() does not await disconnect() — final state update and cleanup race.
disconnect() is async (awaits connection.disconnect() and then calls updateState). Since destroy() is synchronous and doesn't await the returned Promise, stateChangeCallbacks.clear() on line 283 executes immediately — before disconnect() finishes. This means:
- State-change listeners never receive the final
'idle'status fromdisconnect(). - If
connection.disconnect()rejects, it becomes an unhandled Promise rejection.
Proposed fix — make destroy async, or guard the clear
- destroy(): void {
- this.disconnect()
- this.stateChangeCallbacks.clear()
+ async destroy(): Promise<void> {
+ await this.disconnect()
+ this.stateChangeCallbacks.clear()
}If callers cannot await (e.g., React cleanup), chain instead:
- destroy(): void {
- this.disconnect()
- this.stateChangeCallbacks.clear()
+ destroy(): void {
+ this.disconnect().catch(() => {}).finally(() => {
+ this.stateChangeCallbacks.clear()
+ })
}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-client/src/realtime-client.ts` around lines 281 - 284,
The destroy() method currently calls async disconnect() without awaiting,
causing stateChangeCallbacks.clear() to run before disconnect completes; change
destroy to await the disconnect call (e.g., make destroy async and await
this.disconnect()) so updateState and any final `'idle'` notifications run
before clearing; if you cannot change the API to async, instead chain the
Promise (return this.disconnect().finally(() =>
this.stateChangeCallbacks.clear())) or at minimum call
this.disconnect().catch(() => {}).finally(() =>
this.stateChangeCallbacks.clear()) to avoid unhandled rejections and ensure
cleanup happens after disconnect; reference methods: destroy(), disconnect(),
updateState, stateChangeCallbacks, and connection.disconnect().
Comment on lines
+19
to
+23
| export type OpenAIRealtimeModel = | ||
| | 'gpt-4o-realtime-preview' | ||
| | 'gpt-4o-realtime-preview-2024-10-01' | ||
| | 'gpt-4o-mini-realtime-preview' | ||
| | 'gpt-4o-mini-realtime-preview-2024-12-17' |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
OpenAI realtime API models 2025 2026 latest gpt-4o-realtime
💡 Result:
As of February 19, 2026, OpenAI’s latest Realtime API models are:
gpt-realtime(general availability realtime model; text+audio over WebRTC/WebSocket/SIP) [1]gpt-realtime-mini(lower-cost realtime model; text+audio over WebRTC/WebSocket/SIP) [2]
The older GPT‑4o realtime naming you mentioned is not gpt-4o-realtime; it exists as the preview line:
gpt-4o-realtime-preview(and dated snapshots like...-2025-06-03,...-2024-12-17) [3]gpt-4o-mini-realtime-preview[4]
But these preview models are scheduled for shutdown on 2026-03-24, with recommended replacements:
gpt-4o-realtime-preview*→gpt-realtimegpt-4o-mini-realtime-preview→gpt-realtime-mini[4]
Sources:
[1] gpt-realtime model docs [1]
[2] gpt-realtime-mini model docs [2]
[3] gpt-4o-realtime-preview model docs [3]
[4] OpenAI API deprecations notice (shutdown/replacements) [4]
🏁 Script executed:
cat packages/typescript/ai-openai/src/realtime/types.tsRepository: TanStack/ai
Length of output: 3083
Update realtime model type to include production models and allow flexible versioning — preview models are being deprecated.
The OpenAIRealtimeModel union is restricted to four preview models that OpenAI is deprecating on 2026-03-24. Replace them with the new production models (gpt-realtime, gpt-realtime-mini) and adopt a flexible pattern to support future updates without library changes:
Suggested fix
export type OpenAIRealtimeModel =
+ | 'gpt-realtime'
+ | 'gpt-realtime-mini'
| 'gpt-4o-realtime-preview'
| 'gpt-4o-realtime-preview-2024-10-01'
| 'gpt-4o-mini-realtime-preview'
| 'gpt-4o-mini-realtime-preview-2024-12-17'
+ | (string & {})The | (string & {}) pattern preserves IDE autocomplete for known models while allowing arbitrary model identifiers for rapid API updates (same pattern used in the FAL adapter for 600+ models).
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-openai/src/realtime/types.ts` around lines 19 - 23,
Replace the restricted preview-only union in OpenAIRealtimeModel with the new
production names and a flexible string pattern: update the OpenAIRealtimeModel
type to include 'gpt-realtime' and 'gpt-realtime-mini' (instead of the four
preview names) and add the permissive fallback (string & {}) so unknown/future
model identifiers are accepted while preserving IDE autocomplete for known
models; modify the type definition for OpenAIRealtimeModel accordingly.
Comment on lines
+74
to
+115
| if (!clientRef.current) { | ||
| clientRef.current = new RealtimeClient({ | ||
| getToken: optionsRef.current.getToken, | ||
| adapter: optionsRef.current.adapter, | ||
| tools: optionsRef.current.tools, | ||
| autoPlayback: optionsRef.current.autoPlayback, | ||
| autoCapture: optionsRef.current.autoCapture, | ||
| vadMode: optionsRef.current.vadMode, | ||
| onStatusChange: (newStatus) => { | ||
| setStatus(newStatus) | ||
| }, | ||
| onModeChange: (newMode) => { | ||
| setMode(newMode) | ||
| optionsRef.current.onModeChange?.(newMode) | ||
| }, | ||
| onMessage: (message) => { | ||
| setMessages((prev) => [...prev, message]) | ||
| optionsRef.current.onMessage?.(message) | ||
| }, | ||
| onError: (err) => { | ||
| setError(err) | ||
| optionsRef.current.onError?.(err) | ||
| }, | ||
| onConnect: () => { | ||
| setError(null) | ||
| optionsRef.current.onConnect?.() | ||
| }, | ||
| onDisconnect: () => { | ||
| optionsRef.current.onDisconnect?.() | ||
| }, | ||
| onInterrupted: () => { | ||
| setPendingAssistantTranscript(null) | ||
| optionsRef.current.onInterrupted?.() | ||
| }, | ||
| }) | ||
|
|
||
| // Subscribe to state changes for transcripts | ||
| clientRef.current.onStateChange((state) => { | ||
| setPendingUserTranscript(state.pendingUserTranscript) | ||
| setPendingAssistantTranscript(state.pendingAssistantTranscript) | ||
| }) | ||
| } |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
find . -type f -name "*realtime-client*" | head -20Repository: TanStack/ai
Length of output: 109
🏁 Script executed:
rg -l "class RealtimeClient" --type=ts --type=jsRepository: TanStack/ai
Length of output: 107
🏁 Script executed:
rg -n "class RealtimeClient" -A 50 --type=ts --type=jsRepository: TanStack/ai
Length of output: 4418
🏁 Script executed:
rg -n "async connect" -A 30 packages/typescript/ai-client/src/realtime-client.tsRepository: TanStack/ai
Length of output: 1166
🏁 Script executed:
rg -n "getToken" -B 2 -A 5 packages/typescript/ai-client/src/realtime-client.tsRepository: TanStack/ai
Length of output: 944
🏁 Script executed:
cat -n packages/typescript/ai-react/src/use-realtime-chat.ts | head -150Repository: TanStack/ai
Length of output: 5823
🏁 Script executed:
rg -n "optionsRef" packages/typescript/ai-react/src/use-realtime-chat.tsRepository: TanStack/ai
Length of output: 711
🏁 Script executed:
rg -n "clientTools" packages/typescript/ai-client/src/realtime-client.ts | head -20Repository: TanStack/ai
Length of output: 248
🏁 Script executed:
cat -n packages/typescript/ai-client/src/realtime-client.ts | sed -n '200,250p'Repository: TanStack/ai
Length of output: 1749
🏁 Script executed:
rg -n "onClientToolExecute\|sendToolResult" packages/typescript/ai-client/src/realtime-client.ts -A 5Repository: TanStack/ai
Length of output: 37
🏁 Script executed:
cat -n packages/typescript/ai-client/src/realtime-client.ts | sed -n '375,395p'Repository: TanStack/ai
Length of output: 904
getToken, adapter, and tools are captured by value at client creation time and never updated.
Lines 76–78 pass optionsRef.current.getToken, optionsRef.current.adapter, and optionsRef.current.tools to the RealtimeClient constructor. These are evaluated once (when clientRef.current is first set) and frozen for the lifetime of the hook. If the parent component changes getToken (e.g., after auth refresh) or adapter or tools, the client will still use the stale values.
The event callbacks correctly read from optionsRef.current at invocation time — getToken, adapter, and tools should follow the same pattern:
Proposed fix
clientRef.current = new RealtimeClient({
- getToken: optionsRef.current.getToken,
- adapter: optionsRef.current.adapter,
- tools: optionsRef.current.tools,
+ getToken: (...args) => optionsRef.current.getToken(...args),
+ adapter: optionsRef.current.adapter, // adapter is likely stable, but worth noting
+ tools: optionsRef.current.tools, // same concern if tools change📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if (!clientRef.current) { | |
| clientRef.current = new RealtimeClient({ | |
| getToken: optionsRef.current.getToken, | |
| adapter: optionsRef.current.adapter, | |
| tools: optionsRef.current.tools, | |
| autoPlayback: optionsRef.current.autoPlayback, | |
| autoCapture: optionsRef.current.autoCapture, | |
| vadMode: optionsRef.current.vadMode, | |
| onStatusChange: (newStatus) => { | |
| setStatus(newStatus) | |
| }, | |
| onModeChange: (newMode) => { | |
| setMode(newMode) | |
| optionsRef.current.onModeChange?.(newMode) | |
| }, | |
| onMessage: (message) => { | |
| setMessages((prev) => [...prev, message]) | |
| optionsRef.current.onMessage?.(message) | |
| }, | |
| onError: (err) => { | |
| setError(err) | |
| optionsRef.current.onError?.(err) | |
| }, | |
| onConnect: () => { | |
| setError(null) | |
| optionsRef.current.onConnect?.() | |
| }, | |
| onDisconnect: () => { | |
| optionsRef.current.onDisconnect?.() | |
| }, | |
| onInterrupted: () => { | |
| setPendingAssistantTranscript(null) | |
| optionsRef.current.onInterrupted?.() | |
| }, | |
| }) | |
| // Subscribe to state changes for transcripts | |
| clientRef.current.onStateChange((state) => { | |
| setPendingUserTranscript(state.pendingUserTranscript) | |
| setPendingAssistantTranscript(state.pendingAssistantTranscript) | |
| }) | |
| } | |
| if (!clientRef.current) { | |
| clientRef.current = new RealtimeClient({ | |
| getToken: (...args) => optionsRef.current.getToken(...args), | |
| adapter: optionsRef.current.adapter, // adapter is likely stable, but worth noting | |
| tools: optionsRef.current.tools, // same concern if tools change | |
| autoPlayback: optionsRef.current.autoPlayback, | |
| autoCapture: optionsRef.current.autoCapture, | |
| vadMode: optionsRef.current.vadMode, | |
| onStatusChange: (newStatus) => { | |
| setStatus(newStatus) | |
| }, | |
| onModeChange: (newMode) => { | |
| setMode(newMode) | |
| optionsRef.current.onModeChange?.(newMode) | |
| }, | |
| onMessage: (message) => { | |
| setMessages((prev) => [...prev, message]) | |
| optionsRef.current.onMessage?.(message) | |
| }, | |
| onError: (err) => { | |
| setError(err) | |
| optionsRef.current.onError?.(err) | |
| }, | |
| onConnect: () => { | |
| setError(null) | |
| optionsRef.current.onConnect?.() | |
| }, | |
| onDisconnect: () => { | |
| optionsRef.current.onDisconnect?.() | |
| }, | |
| onInterrupted: () => { | |
| setPendingAssistantTranscript(null) | |
| optionsRef.current.onInterrupted?.() | |
| }, | |
| }) | |
| // Subscribe to state changes for transcripts | |
| clientRef.current.onStateChange((state) => { | |
| setPendingUserTranscript(state.pendingUserTranscript) | |
| setPendingAssistantTranscript(state.pendingAssistantTranscript) | |
| }) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts` around lines 74 - 115,
The RealtimeClient is being created with getToken, adapter, and tools captured
once (optionsRef.current.getToken, optionsRef.current.adapter,
optionsRef.current.tools) which freezes stale values; change the RealtimeClient
construction so those fields are passed as thin wrapper functions that read from
optionsRef.current at invocation time (e.g., getToken: (...args) =>
optionsRef.current.getToken?.(...args), adapter: (...args) =>
optionsRef.current.adapter?.(...args) or adapter: () =>
optionsRef.current.adapter, and similarly for tools) so the client uses
up-to-date values from optionsRef when methods are called; keep existing
callbacks (onMessage/onError/etc.), clientRef.current and the onStateChange
subscription unchanged.
Comment on lines
+120
to
+139
| useEffect(() => { | ||
| function updateLevels() { | ||
| if (clientRef.current?.audio) { | ||
| setInputLevel(clientRef.current.audio.inputLevel) | ||
| setOutputLevel(clientRef.current.audio.outputLevel) | ||
| } | ||
| animationFrameRef.current = requestAnimationFrame(updateLevels) | ||
| } | ||
|
|
||
| if (status === 'connected') { | ||
| updateLevels() | ||
| } | ||
|
|
||
| return () => { | ||
| if (animationFrameRef.current) { | ||
| cancelAnimationFrame(animationFrameRef.current) | ||
| animationFrameRef.current = null | ||
| } | ||
| } | ||
| }, [status]) |
Contributor
There was a problem hiding this comment.
Animation frame loop triggers React state updates at ~60 fps, causing excessive re-renders.
setInputLevel / setOutputLevel are called on every requestAnimationFrame tick (~60 fps). Each call triggers a React re-render of the component and all its children, even when the audio levels haven't meaningfully changed. For a real-time audio UI this will cause significant performance degradation.
Consider one of:
- Use refs for levels and let the visualization consumer read them directly (e.g., via a getter or a ref-forwarding API).
- Throttle updates to ~10-15 fps, which is visually sufficient for level meters.
- Only update state when the value changes beyond a threshold.
Option 2: Throttle to ~15 fps
useEffect(() => {
+ let lastUpdate = 0
function updateLevels() {
- if (clientRef.current?.audio) {
- setInputLevel(clientRef.current.audio.inputLevel)
- setOutputLevel(clientRef.current.audio.outputLevel)
+ const now = performance.now()
+ if (now - lastUpdate > 66 && clientRef.current?.audio) { // ~15fps
+ setInputLevel(clientRef.current.audio.inputLevel)
+ setOutputLevel(clientRef.current.audio.outputLevel)
+ lastUpdate = now
}
animationFrameRef.current = requestAnimationFrame(updateLevels)
}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts` around lines 120 -
139, The updateLevels loop in useEffect calls setInputLevel/setOutputLevel every
requestAnimationFrame causing ~60fps re-renders; modify updateLevels to throttle
updates to ~15fps (or ~66ms) and only call setInputLevel/setOutputLevel when the
new value differs by a small threshold (e.g., delta > 0.01) to avoid unnecessary
renders. Concretely, inside the updateLevels function (which uses
animationFrameRef and clientRef), track the last update timestamp (or frame
count) and only perform the React state updates when enough time has elapsed,
and compute a small difference check against previousLevel refs before calling
setInputLevel/setOutputLevel; keep the existing cleanup of animationFrameRef and
the status === 'connected' start condition.
@tanstack/ai
@tanstack/ai-anthropic
@tanstack/ai-client
@tanstack/ai-devtools-core
@tanstack/ai-elevenlabs
@tanstack/ai-fal
@tanstack/ai-gemini
@tanstack/ai-grok
@tanstack/ai-ollama
@tanstack/ai-openai
@tanstack/ai-openrouter
@tanstack/ai-preact
@tanstack/ai-react
@tanstack/ai-react-ui
@tanstack/ai-solid
@tanstack/ai-solid-ui
@tanstack/ai-svelte
@tanstack/ai-vue
@tanstack/ai-vue-ui
@tanstack/preact-ai-devtools
@tanstack/react-ai-devtools
@tanstack/solid-ai-devtools
commit: |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 13
🧹 Nitpick comments (6)
packages/typescript/ai-elevenlabs/package.json (2)
46-49:peerDependenciesuseworkspace:^whiledevDependenciesuseworkspace:*for the same packagesBoth
@tanstack/aiand@tanstack/ai-clientappear inpeerDependenciesasworkspace:^(lines 47–48) and indevDependenciesasworkspace:*(lines 51–52). The inconsistency is confusing and deviates from the established convention.♻️ Proposed fix
"peerDependencies": { - "@tanstack/ai": "workspace:^", - "@tanstack/ai-client": "workspace:^" + "@tanstack/ai": "workspace:*", + "@tanstack/ai-client": "workspace:*" },Based on learnings: "Use
workspace:*protocol for internal package dependencies in package.json (e.g.,"tanstack/ai": "workspace:*")"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/package.json` around lines 46 - 49, Update the peerDependencies entries in package.json so they use the same workspace protocol as devDependencies: change "@tanstack/ai": "workspace:^" and "@tanstack/ai-client": "workspace:^" to use "workspace:*" under the peerDependencies block so the package names and protocols are consistent with devDependencies and the repo convention.
23-28: Consider adding a"default"fallback condition to the exports mapThe current exports only declare
"types"and"import". Bundlers and tools that do not recognise the"import"condition (e.g., certain Jest or CJS-first configurations) have no fallback and will fail to resolve the package.♻️ Proposed fix
"exports": { ".": { "types": "./dist/esm/index.d.ts", - "import": "./dist/esm/index.js" + "import": "./dist/esm/index.js", + "default": "./dist/esm/index.js" } },🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/package.json` around lines 23 - 28, The exports map currently only provides "types" and the "import" condition which can break resolution in CJS-first runtimes; update the package.json "exports" object to include a "default" fallback entry that points to the CommonJS bundle (e.g., the package's CJS runtime entry such as the compiled ./dist CJS file) so tools that don't support the "import" condition can resolve the package; ensure the "types" key remains and that the "default" path matches your distributed CJS output file.packages/typescript/ai-client/src/realtime-client.ts (1)
63-63: Debugconsole.logcalls should be removed from library code.Lines 63, 105–108, and 257–260 contain
[RealtimeClient #...]debug logs. These will appear in every consumer's production console.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-client/src/realtime-client.ts` at line 63, Remove the debug console.log calls that emit `[RealtimeClient #${this.instanceId}]` from the RealtimeClient class (the constructor and other locations that currently log at lines where instanceId is used), replacing them with either no-op or a proper logger call at debug/trace level if a logging facility exists; specifically eliminate the console.log statements in the RealtimeClient that print instance creation and other internal state so library consumers don't get production console output, or switch them to use the project's logger (e.g., logger.debug) if consistent logging is required.packages/typescript/ai-openai/src/realtime/adapter.ts (1)
175-180: Severalconsole.logdebug statements should be removed before release.Lines 122–126 (
[Realtime] ontrack event), 175–180 ([Realtime] Setting up input audio analysis), and 519–525 ([Realtime] getAudioVisualization called) are debug artifacts that will pollute consumers' production consoles. ThegetAudioVisualizationgetter is called on every animation frame, so line 519–525 will log at ~60 fps when audio is active.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 175 - 180, Remove the debug console.log statements that were left in the realtime adapter: delete the logs emitted inside the ontrack handler (the "[Realtime] ontrack event" messages), remove the "[Realtime] Setting up input audio analysis" and "[Realtime] Input analyser created:" logs around the call to setupInputAudioAnalysis(localStream) and inputAnalyser, and remove any logging inside the getAudioVisualization getter (which is called every animation frame). Keep behavior unchanged—only remove the console.log calls so production consoles are not spammed.examples/ts-react-chat/src/lib/realtime-tools.ts (1)
116-139:setTimeoutinsidesetReminderClientis untracked — leaks if the component unmounts.
setTimeoutstores no reference and is never cancelled. If the page/component is torn down before the timer fires, thealertwill still trigger. For a demo this is acceptable, but callers should be aware.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/ts-react-chat/src/lib/realtime-tools.ts` around lines 116 - 139, The setReminderClient implementation uses setTimeout without storing or exposing the timer id, which can leak and still call alert after a component/page unmounts; modify the function (the setReminderClient created from setReminderToolDef) to capture the timeout id returned by setTimeout, return alongside the success payload a cancel/clear function or token (e.g., { cancel: () => clearTimeout(id) }) so callers can cancel the scheduled reminder, and update any docs/comments to advise callers to call that cancel method (or call clearTimeout with the stored id) during unmount/cleanup to prevent the alert from firing after teardown.packages/typescript/ai/src/realtime/types.ts (1)
210-243: Consider derivingRealtimeEventfromRealtimeEventPayloadsto keep them in sync.
RealtimeEventis a manually maintained string union that must exactly mirror the keys ofRealtimeEventPayloads. Deriving one from the other eliminates the risk of them drifting apart.Proposed fix
-export type RealtimeEvent = - | 'status_change' - | 'mode_change' - | 'transcript' - | 'audio_chunk' - | 'tool_call' - | 'message_complete' - | 'interrupted' - | 'error' - /** * Event payloads for realtime events */ export interface RealtimeEventPayloads { status_change: { status: RealtimeStatus } mode_change: { mode: RealtimeMode } transcript: { role: 'user' | 'assistant' transcript: string isFinal: boolean } audio_chunk: { data: ArrayBuffer; sampleRate: number } tool_call: { toolCallId: string; toolName: string; input: unknown } message_complete: { message: RealtimeMessage } interrupted: { messageId?: string } error: { error: Error } } +/** + * Events emitted by the realtime connection + */ +export type RealtimeEvent = keyof RealtimeEventPayloads🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai/src/realtime/types.ts` around lines 210 - 243, Replace the manually-maintained string union RealtimeEvent with a derived union from the keys of RealtimeEventPayloads to keep them in sync: change the declaration of RealtimeEvent to be based on keyof RealtimeEventPayloads (ensuring it still exports the same literal union type), and keep RealtimeEventHandler<TEvent extends RealtimeEvent> using the mapped lookup RealtimeEventPayloads[TEvent] as before so handlers continue to type-check against the payloads.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@examples/ts-react-chat/src/routes/api.realtime-token.ts`:
- Around line 5-11: Reorder the imports to satisfy ESLint import-order rules:
move the `import * as z from 'zod'` line above the `@/lib/realtime-tools` import
group, and within the named imports from `@/lib/realtime-tools` ensure
`searchKnowledgeToolDef` appears alphabetically before `setReminderToolDef` (so
the named list reads e.g. getCurrentTimeToolDef, getWeatherToolDef,
searchKnowledgeToolDef, setReminderToolDef).
In `@examples/ts-react-chat/src/routes/realtime.tsx`:
- Around line 136-140: The error handling uses response.json() which will throw
if the server returns non-JSON; modify the error path where you check if
(!response.ok) to safely attempt response.json() inside a try/catch and fall
back to await response.text() if parsing fails, then throw a new Error that
includes response.status (and/or statusText) plus the parsed error message or
raw text; keep the successful return as return response.json(). Ensure you
reference the same response variable in this block so the thrown error contains
meaningful HTTP status and body content instead of a parse exception.
In `@packages/typescript/ai-client/src/realtime-client.ts`:
- Around line 89-92: The connect() method can be invoked concurrently causing
two fetches/connections; protect it by making the status check atomic: when
entering connect() check this.state.status and if not 'connected' immediately
set this.state.status = 'connecting' (or acquire a simple mutex/Promise lock)
before any async work so other callers will see 'connecting' and return/wait;
ensure any early returns or errors restore status to 'idle' and that successful
completion sets status = 'connected'. Update the connect() implementation to use
this new status transition (or an explicit lock) around token fetch and socket
open to prevent parallel connect attempts.
- Around line 379-385: The assistant transcript handler currently replaces
pendingAssistantTranscript on each non-final delta, losing earlier tokens;
modify the logic in the transcript event handler (the block that calls
updateState and touches pendingAssistantTranscript) to accumulate chunks instead
of overwriting: when isFinal is false, set pendingAssistantTranscript to
(existing pendingAssistantTranscript || '') + transcript (handling
null/undefined), and when isFinal is true clear pendingAssistantTranscript (or
let message_complete handle finalization) so the transcript grows across deltas;
ensure you reference updateState and pendingAssistantTranscript in the change.
In `@packages/typescript/ai-elevenlabs/package.json`:
- Around line 43-45: Update the deprecated dependency "@11labs/client" in
package.json to the maintained "@elevenlabs/client" (pin to a modern release
like "^0.14.0"), and replace all import/require occurrences of "@11labs/client"
in the codebase with "@elevenlabs/client"; also audit call sites for SDK
breaking changes (notably replace any use of the removed prompt.tools field with
prompt.tool_ids and update any changed method/option names per the ElevenLabs
changelog) so the package compiles and tests pass.
In `@packages/typescript/ai-elevenlabs/README.md`:
- Around line 7-9: The install command in the README is missing the React
package required by the example; update the npm install line to include
`@tanstack/ai-react` so the React example that imports useRealtimeChat (and other
React hooks from `@tanstack/ai-react`) works correctly — ensure the install
command adds `@tanstack/ai-react` alongside `@tanstack/ai-elevenlabs`, `@tanstack/ai`,
and `@tanstack/ai-client`.
In `@packages/typescript/ai-elevenlabs/src/realtime/token.ts`:
- Around line 64-66: The URL built for the fetch call to
`${ELEVENLABS_API_URL}/convai/conversation/get_signed_url?agent_id=${agentId}`
uses raw agentId which can contain characters that break the URL; fix this in
the token generation logic by URL-encoding agentId (e.g., via encodeURIComponent
or equivalent) when constructing the query string so the fetch to the
get_signed_url endpoint always targets the correct resource while keeping
ELEVENLABS_API_URL and the rest of the path unchanged.
- Around line 81-82: The code unsafely casts data.signed_url to string
(signedUrl = data.signed_url as string) without validation; update the
token-fetching logic (the response/json handling and variable signedUrl) to
validate that data.signed_url exists and is a non-empty string (e.g., typeof
data.signed_url === 'string' && data.signed_url.trim() !== '') and if not, throw
or return a clear error (with context) instead of silently casting, so
downstream callers of the token/signedUrl get a guaranteed valid value or an
actionable error.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts`:
- Around line 497-502: interrupt() is emitting 'interrupted' optimistically then
the server's conversation.item.truncated handler also emits 'interrupted',
causing duplicate notifications; introduce a clientInterruptPending boolean that
interrupt() sets to true before emitting (and include currentMessageId), then in
the conversation.item.truncated handler check clientInterruptPending and
suppress the emit('interrupted', ...) for the same messageId if the flag is set,
finally clear clientInterruptPending when the server event arrives (or on other
flows like response.cancel acknowledgement) so future interruptions behave
normally; reference interrupt(), the conversation.item.truncated handler,
currentMessageId, and emit('interrupted', ...) when adding the flag logic.
- Around line 116-118: The onerror handler for dataChannel currently stringifies
the Event, producing "[object Event]"; instead, extract a useful message from
the event (handle RTCErrorEvent by checking event.error?.message or event.error,
fallback to event.message or event.type) and include that in the emitted Error,
or attach the original event as the Error's cause/metadata; update the
dataChannel.onerror callback (the code that calls emit('error', { error: new
Error(`Data channel error: ${error}`) }) ) to build the Error with the extracted
message and/or include the original event as the cause so logs contain
actionable info.
In `@packages/typescript/ai-openai/src/realtime/token.ts`:
- Around line 114-118: The code assumes sessionData.tools is an array and calls
sessionData.tools.map which will throw if tools is null/undefined; update the
logic that builds tools (the variable named tools and the sessionData.tools.map
usage in token.ts) to first coerce or guard the value (e.g., treat
non-array/null as an empty array using a nullish/Array.isArray check) and then
map to the Tool shape so tools is always an Array<Tool> (empty when no tools
provided).
In `@packages/typescript/ai-react/src/use-realtime-chat.ts`:
- Around line 153-157: The cleanup effect currently calls
clientRef.current?.destroy() but leaves clientRef.current set, so after React
StrictMode unmount/remount the destroyed client exists and onStateChange
subscriptions aren't re-registered causing
pendingUserTranscript/pendingAssistantTranscript to never update; update the
cleanup in the useEffect to call clientRef.current?.destroy() and then null out
clientRef.current (or set to undefined) so subsequent mounts detect no client
and recreate it and re-subscribe via onStateChange.
- Around line 13-15: The fallback arrays emptyFrequencyData and
emptyTimeDomainData are sized at 128 but must match the adapter's FFT output
(frequencyBinCount = 1024 and time domain length = 2048) to avoid consumers
seeing length changes; update emptyFrequencyData to new Uint8Array(1024) and
emptyTimeDomainData to new Uint8Array(2048). Keep the time-domain default values
(e.g., fill(128)) as before so that getInputFrequencyData() returns the
same-shaped arrays whether connected or not.
---
Duplicate comments:
In `@examples/ts-react-chat/src/routes/realtime.tsx`:
- Around line 108-109: The adapter is recreated on every render because it
directly calls openaiRealtime() or elevenlabsRealtime(); wrap the adapter
creation in useMemo so it returns a stable reference based on provider. Replace
the direct assignment to adapter with a memoized value using React's useMemo and
key the memo on provider (referencing adapter, openaiRealtime, and
elevenlabsRealtime) so the hook consuming adapter receives a stable object
between renders.
In `@packages/typescript/ai-client/src/realtime-client.ts`:
- Around line 96-126: The connect() flow can leave this.connection open if
this.options.autoCapture is true and connection.startAudioCapture() throws;
update the catch branch in connect() to perform cleanup: if this.connection
exists call its disconnect() (or close) and null out this.connection,
unsubscribe/cleanup any connection event handlers created by
subscribeToConnectionEvents(), and cancel any scheduled token refresh started by
scheduleTokenRefresh(); ensure you still set state to error, call
this.options.onError, and rethrow the error so the connection/permission
indicator is not left active when startAudioCapture fails.
- Around line 163-172: The startListening function calls
connection.startAudioCapture() without awaiting it so any Promise rejection
(e.g., mic permission denied) is swallowed and updateState({ mode: 'listening'
}) runs regardless; change startListening to await
this.connection.startAudioCapture(), wrap the await in try/catch, only call
this.updateState({ mode: 'listening' }) on success, and on error log or
propagate the error (e.g., via processLogger or this.updateState to an error
status) so failures are not silently ignored; reference the startListening
method and connection.startAudioCapture() and updateState() calls when making
the change.
- Around line 393-409: When this.clientTools.get(toolName) returns undefined (or
the retrieved tool lacks an execute method) the provider never responds; update
the branch around the retrieval of tool (this.clientTools.get(toolName)) to send
a failure tool result back using this.connection?.sendToolResult(toolCallId,
JSON.stringify({ error: ... })) including the toolName and a clear message
(e.g., "tool not registered" or "tool has no execute method") so callers aren’t
left waiting; keep the existing try/catch for actual execution errors in execute
and only short-circuit with the error response when the tool is missing or
invalid.
- Around line 287-290: The destroy() implementation calls the async disconnect()
without awaiting it, causing stateChangeCallbacks.clear() to run too early and
any rejection from connection.disconnect() to be an unhandled promise; change
destroy() to be async (or return a Promise) and await this.disconnect(), then
clear stateChangeCallbacks after the await, and wrap the await in a try/catch to
handle/log any error from connection.disconnect() (reference methods: destroy(),
disconnect(), and the stateChangeCallbacks collection).
In `@packages/typescript/ai-elevenlabs/src/realtime/token.ts`:
- Around line 9-29: The getElevenLabsApiKey function currently falls back to
reading window.env, which can leak the server API key if this server-only module
is bundled client-side; remove the browser fallback so getElevenLabsApiKey only
reads process.env.ELEVENLABS_API_KEY (and throws if missing) and delete the
block that checks/casts window and window.env; keep the existing error throw
path and ensure the function is only using the process check (symbol:
getElevenLabsApiKey).
In `@packages/typescript/ai-openai/src/index.ts`:
- Around line 103-118: The realtime adapter exports (openaiRealtimeToken,
openaiRealtime and the OpenAIRealtime* types) were added but the package-level
subpath exports for the realtime folder are missing, breaking tree-shaking and
subpath imports; update the package.json "exports" to add a "./realtime" subpath
(and corresponding "./realtime/*" if needed) pointing to the built JS and type
entry files so consumers can import openaiRealtimeToken/openaiRealtime and the
OpenAIRealtime types directly and tree-shake unused code; ensure the paths map
to the compiled output and type declaration files consistent with how other
adapter subpaths are declared.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts`:
- Around line 150-172: Wrap the SDP exchange and remote-description steps (the
fetch to OPENAI_REALTIME_URL, sdpResponse.ok handling, and
pc.setRemoteDescription) in a try/catch (or try/catch/finally) so any failure
cleans up resources: if an error occurs, stop all tracks on localStream (for
each track call stop()), close dataChannel if it exists (dataChannel.close()),
and close the RTCPeerConnection (await pc.close()); also null out or clear
references as needed to avoid dangling state. Ensure checks for undefined/null
on localStream, dataChannel, and pc before calling stop/close, and rethrow the
original error after cleanup so callers still see the failure.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts`:
- Around line 87-89: The client is being constructed with stale snapshots of
getToken, adapter, and tools; instead pass through optionsRef to always read the
latest values. Update the client construction (where getToken, adapter, tools
are set) to use wrappers that call optionsRef.current (e.g., getToken: (...args)
=> optionsRef.current.getToken?.(...args), and similarly resolve adapter and
tools via optionsRef.current when invoked) so the realtime client uses
up-to-date references; keep the unique names getToken, adapter, tools and
optionsRef to locate and modify the assignment.
- Around line 131-150: The animation loop in updateLevels currently calls
setInputLevel/setOutputLevel on every requestAnimationFrame (via
animationFrameRef) causing excessive re-renders; modify updateLevels (used with
clientRef.current.audio and status === 'connected') to only call
setInputLevel/setOutputLevel at a lower rate (e.g., throttle to ~15 fps using a
lastUpdateRef timestamp) or only when the level delta exceeds a small threshold
(e.g., Math.abs(new - prev) > 0.01), and keep scheduling requestAnimationFrame
as before; ensure you read and update a ref for last input/output values (and
last update time) so you avoid state updates when values haven't meaningfully
changed.
---
Nitpick comments:
In `@examples/ts-react-chat/src/lib/realtime-tools.ts`:
- Around line 116-139: The setReminderClient implementation uses setTimeout
without storing or exposing the timer id, which can leak and still call alert
after a component/page unmounts; modify the function (the setReminderClient
created from setReminderToolDef) to capture the timeout id returned by
setTimeout, return alongside the success payload a cancel/clear function or
token (e.g., { cancel: () => clearTimeout(id) }) so callers can cancel the
scheduled reminder, and update any docs/comments to advise callers to call that
cancel method (or call clearTimeout with the stored id) during unmount/cleanup
to prevent the alert from firing after teardown.
In `@packages/typescript/ai-client/src/realtime-client.ts`:
- Line 63: Remove the debug console.log calls that emit `[RealtimeClient
#${this.instanceId}]` from the RealtimeClient class (the constructor and other
locations that currently log at lines where instanceId is used), replacing them
with either no-op or a proper logger call at debug/trace level if a logging
facility exists; specifically eliminate the console.log statements in the
RealtimeClient that print instance creation and other internal state so library
consumers don't get production console output, or switch them to use the
project's logger (e.g., logger.debug) if consistent logging is required.
In `@packages/typescript/ai-elevenlabs/package.json`:
- Around line 46-49: Update the peerDependencies entries in package.json so they
use the same workspace protocol as devDependencies: change "@tanstack/ai":
"workspace:^" and "@tanstack/ai-client": "workspace:^" to use "workspace:*"
under the peerDependencies block so the package names and protocols are
consistent with devDependencies and the repo convention.
- Around line 23-28: The exports map currently only provides "types" and the
"import" condition which can break resolution in CJS-first runtimes; update the
package.json "exports" object to include a "default" fallback entry that points
to the CommonJS bundle (e.g., the package's CJS runtime entry such as the
compiled ./dist CJS file) so tools that don't support the "import" condition can
resolve the package; ensure the "types" key remains and that the "default" path
matches your distributed CJS output file.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts`:
- Around line 175-180: Remove the debug console.log statements that were left in
the realtime adapter: delete the logs emitted inside the ontrack handler (the
"[Realtime] ontrack event" messages), remove the "[Realtime] Setting up input
audio analysis" and "[Realtime] Input analyser created:" logs around the call to
setupInputAudioAnalysis(localStream) and inputAnalyser, and remove any logging
inside the getAudioVisualization getter (which is called every animation frame).
Keep behavior unchanged—only remove the console.log calls so production consoles
are not spammed.
In `@packages/typescript/ai/src/realtime/types.ts`:
- Around line 210-243: Replace the manually-maintained string union
RealtimeEvent with a derived union from the keys of RealtimeEventPayloads to
keep them in sync: change the declaration of RealtimeEvent to be based on keyof
RealtimeEventPayloads (ensuring it still exports the same literal union type),
and keep RealtimeEventHandler<TEvent extends RealtimeEvent> using the mapped
lookup RealtimeEventPayloads[TEvent] as before so handlers continue to
type-check against the payloads.
Comment on lines
+5
to
+11
| import { | ||
| getCurrentTimeToolDef, | ||
| getWeatherToolDef, | ||
| setReminderToolDef, | ||
| searchKnowledgeToolDef, | ||
| } from '@/lib/realtime-tools' | ||
| import * as z from 'zod' |
Contributor
There was a problem hiding this comment.
ESLint reports two import-ordering violations.
Static analysis flags:
searchKnowledgeToolDef(line 9) should appear alphabetically beforesetReminderToolDef.- The
zodimport (line 11) should come before the@/lib/realtime-toolsimport (lines 5–10).
✏️ Proposed fix
+import * as z from 'zod'
import { createFileRoute } from '@tanstack/react-router'
import { realtimeToken } from '@tanstack/ai'
import { openaiRealtimeToken } from '@tanstack/ai-openai'
import { elevenlabsRealtimeToken } from '@tanstack/ai-elevenlabs'
import {
getCurrentTimeToolDef,
getWeatherToolDef,
+ searchKnowledgeToolDef,
setReminderToolDef,
- searchKnowledgeToolDef,
} from '@/lib/realtime-tools'
-import * as z from 'zod'🧰 Tools
🪛 ESLint
[error] 9-9: Member 'searchKnowledgeToolDef' of the import declaration should be sorted alphabetically.
(sort-imports)
[error] 11-11: zod import should occur before import of @/lib/realtime-tools
(import/order)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@examples/ts-react-chat/src/routes/api.realtime-token.ts` around lines 5 - 11,
Reorder the imports to satisfy ESLint import-order rules: move the `import * as
z from 'zod'` line above the `@/lib/realtime-tools` import group, and within the
named imports from `@/lib/realtime-tools` ensure `searchKnowledgeToolDef`
appears alphabetically before `setReminderToolDef` (so the named list reads e.g.
getCurrentTimeToolDef, getWeatherToolDef, searchKnowledgeToolDef,
setReminderToolDef).
Comment on lines
+136
to
+140
| if (!response.ok) { | ||
| const error = await response.json() | ||
| throw new Error(error.error || 'Failed to get token') | ||
| } | ||
| return response.json() |
Contributor
There was a problem hiding this comment.
Error-path response.json() will throw if the server returns non-JSON (e.g. plain-text 500).
If the token endpoint returns a non-JSON error body, response.json() throws an unrelated parse error that masks the real HTTP failure.
Proposed fix
if (!response.ok) {
- const error = await response.json()
- throw new Error(error.error || 'Failed to get token')
+ let message = 'Failed to get token'
+ try {
+ const body = await response.json()
+ if (body.error) message = body.error
+ } catch {
+ // non-JSON error body – fall through with default message
+ }
+ throw new Error(message)
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if (!response.ok) { | |
| const error = await response.json() | |
| throw new Error(error.error || 'Failed to get token') | |
| } | |
| return response.json() | |
| if (!response.ok) { | |
| let message = 'Failed to get token' | |
| try { | |
| const body = await response.json() | |
| if (body.error) message = body.error | |
| } catch { | |
| // non-JSON error body – fall through with default message | |
| } | |
| throw new Error(message) | |
| } | |
| return response.json() |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@examples/ts-react-chat/src/routes/realtime.tsx` around lines 136 - 140, The
error handling uses response.json() which will throw if the server returns
non-JSON; modify the error path where you check if (!response.ok) to safely
attempt response.json() inside a try/catch and fall back to await
response.text() if parsing fails, then throw a new Error that includes
response.status (and/or statusText) plus the parsed error message or raw text;
keep the successful return as return response.json(). Ensure you reference the
same response variable in this block so the thrown error contains meaningful
HTTP status and body content instead of a parse exception.
Comment on lines
+89
to
+92
| async connect(): Promise<void> { | ||
| if (this.state.status === 'connected') { | ||
| return | ||
| } |
Contributor
There was a problem hiding this comment.
Concurrent connect() calls are not guarded — two callers can both proceed past the 'connected' check.
The guard only returns early when status === 'connected'; concurrent invocations starting from 'idle' both pass through, fetch two tokens, and open two connections.
🛡️ Proposed fix
- if (this.state.status === 'connected') {
+ if (this.state.status === 'connected' || this.state.status === 'connecting') {
return
}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-client/src/realtime-client.ts` around lines 89 - 92,
The connect() method can be invoked concurrently causing two
fetches/connections; protect it by making the status check atomic: when entering
connect() check this.state.status and if not 'connected' immediately set
this.state.status = 'connecting' (or acquire a simple mutex/Promise lock) before
any async work so other callers will see 'connecting' and return/wait; ensure
any early returns or errors restore status to 'idle' and that successful
completion sets status = 'connected'. Update the connect() implementation to use
this new status transition (or an explicit lock) around token fetch and socket
open to prevent parallel connect attempts.
Comment on lines
+379
to
+385
| } else { | ||
| // Assistant transcripts - just update pending, message_complete handles final | ||
| this.updateState({ | ||
| pendingAssistantTranscript: isFinal ? null : transcript, | ||
| }) | ||
| } | ||
| }), |
Contributor
There was a problem hiding this comment.
Streaming assistant transcript overwrites on each delta instead of accumulating.
The OpenAI adapter emits individual delta chunks via the transcript event (isFinal: false). The current code replaces pendingAssistantTranscript with each new chunk rather than concatenating, so users see only the last word/token rather than the growing sentence.
🛡️ Proposed fix
} else {
// Assistant transcripts - just update pending, message_complete handles final
this.updateState({
- pendingAssistantTranscript: isFinal ? null : transcript,
+ pendingAssistantTranscript: isFinal
+ ? null
+ : (this.state.pendingAssistantTranscript ?? '') + transcript,
})
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| } else { | |
| // Assistant transcripts - just update pending, message_complete handles final | |
| this.updateState({ | |
| pendingAssistantTranscript: isFinal ? null : transcript, | |
| }) | |
| } | |
| }), | |
| } else { | |
| // Assistant transcripts - just update pending, message_complete handles final | |
| this.updateState({ | |
| pendingAssistantTranscript: isFinal | |
| ? null | |
| : (this.state.pendingAssistantTranscript ?? '') + transcript, | |
| }) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-client/src/realtime-client.ts` around lines 379 - 385,
The assistant transcript handler currently replaces pendingAssistantTranscript
on each non-final delta, losing earlier tokens; modify the logic in the
transcript event handler (the block that calls updateState and touches
pendingAssistantTranscript) to accumulate chunks instead of overwriting: when
isFinal is false, set pendingAssistantTranscript to (existing
pendingAssistantTranscript || '') + transcript (handling null/undefined), and
when isFinal is true clear pendingAssistantTranscript (or let message_complete
handle finalization) so the transcript grows across deltas; ensure you reference
updateState and pendingAssistantTranscript in the change.
Comment on lines
+43
to
+45
| "dependencies": { | ||
| "@11labs/client": "^0.2.0" | ||
| }, |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
@elevenlabs/client npm latest version changelog
💡 Result:
Latest @elevenlabs/client version: 0.14.0 (released 2026-02-04). [1][2]
Changelog (v0.14.0):
- Reduced audio chunk length from 250ms → 100ms for lower-latency agent conversations. [1][3]
- Normalized
textOnlyoption handling between the top-level option and the overrides object (top-level takes precedence if both are provided). [1][3] - Dependency update:
@elevenlabs/types@0.5.0. [1]
Sources:
[1] GitHub releases (elevenlabs/packages) (github.com)
[2] jsDelivr package page (jsdelivr.com)
[3] ElevenLabs docs changelog (elevenlabs.io)
Citations:
- 1: https://github.com/elevenlabs/packages/releases
- 2: https://www.jsdelivr.com/package/npm/%40elevenlabs/client?utm_source=openai
- 3: https://elevenlabs.io/docs/changelog
🌐 Web query:
@11labs/client npm package status deprecated
💡 Result:
Yes — @11labs/client is deprecated on npm. The npm page shows the deprecation notice with the author message: “This package is no longer maintained. Please use @elevenlabs/client for the latest version.” [1]
🌐 Web query:
@elevenlabs/client migration guide from @11labs/client breaking changes
💡 Result:
Migration: @11labs/client → @elevenlabs/client
1) Swap the package
npm uninstall `@11labs/client`
npm install `@elevenlabs/client`@11labs/client is deprecated and explicitly points to @elevenlabs/client as the maintained replacement. [1]
2) Update imports
Before
import { Conversation } from "@11labs/client";After
import { Conversation } from "@elevenlabs/client";The core usage pattern remains:
const conversation = await Conversation.startSession({ agentId: "...", connectionType: "webrtc" });[2]
3) Re-check “client tools” / agent tool wiring (API breaking change, if you manage agents)
If you’re configuring ConvAI agent tools via the HTTP API, the legacy agent field prompt.tools was removed in July 2025; you must use prompt.tool_ids (and prompt.built_in_tools for built-ins). [3]
4) Review SDK release notes for behavioral changes
ElevenLabs publishes SDK releases in their changelog (look for “ConvAI packages” entries mentioning @elevenlabs/client). [4]
Sources: [1] [2] [3] [4]
@11labs/client is deprecated — migrate to @elevenlabs/client
@11labs/client is deprecated on npm with an explicit notice directing users to @elevenlabs/client. The actively maintained package is at v0.14.0, released 2026-02-04. Using the stale @11labs/client@^0.2.0 pins consumers to an unmaintained dependency and misses significant feature and stability improvements from 12+ versions of releases.
Note: If agent tool configuration is involved, be aware that the legacy prompt.tools API field was removed in July 2025 and must be replaced with prompt.tool_ids (see ElevenLabs SDK changelog for ConvAI packages).
📦 Proposed fix
"dependencies": {
- "@11labs/client": "^0.2.0"
+ "@elevenlabs/client": "^0.14.0"
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| "dependencies": { | |
| "@11labs/client": "^0.2.0" | |
| }, | |
| "dependencies": { | |
| "@elevenlabs/client": "^0.14.0" | |
| }, |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-elevenlabs/package.json` around lines 43 - 45, Update
the deprecated dependency "@11labs/client" in package.json to the maintained
"@elevenlabs/client" (pin to a modern release like "^0.14.0"), and replace all
import/require occurrences of "@11labs/client" in the codebase with
"@elevenlabs/client"; also audit call sites for SDK breaking changes (notably
replace any use of the removed prompt.tools field with prompt.tool_ids and
update any changed method/option names per the ElevenLabs changelog) so the
package compiles and tests pass.
Comment on lines
+116
to
+118
| dataChannel.onerror = (error) => { | ||
| emit('error', { error: new Error(`Data channel error: ${error}`) }) | ||
| } |
Contributor
There was a problem hiding this comment.
dataChannel.onerror passes an Event object to new Error(...) — produces "[object Event]" in the error message.
The onerror handler receives an Event (or RTCErrorEvent), not an Error. Template-literal interpolation of a DOM event yields [object Event], making the error message useless.
🛡️ Proposed fix
- dataChannel.onerror = (error) => {
- emit('error', { error: new Error(`Data channel error: ${error}`) })
+ dataChannel.onerror = (event) => {
+ const rtcError = event as RTCErrorEvent
+ const message = rtcError.error?.message ?? 'Data channel error'
+ emit('error', { error: new Error(message) })
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| dataChannel.onerror = (error) => { | |
| emit('error', { error: new Error(`Data channel error: ${error}`) }) | |
| } | |
| dataChannel.onerror = (event) => { | |
| const rtcError = event as RTCErrorEvent | |
| const message = rtcError.error?.message ?? 'Data channel error' | |
| emit('error', { error: new Error(message) }) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 116 -
118, The onerror handler for dataChannel currently stringifies the Event,
producing "[object Event]"; instead, extract a useful message from the event
(handle RTCErrorEvent by checking event.error?.message or event.error, fallback
to event.message or event.type) and include that in the emitted Error, or attach
the original event as the Error's cause/metadata; update the dataChannel.onerror
callback (the code that calls emit('error', { error: new Error(`Data channel
error: ${error}`) }) ) to build the Error with the extracted message and/or
include the original event as the cause so logs contain actionable info.
Comment on lines
+497
to
+502
| interrupt() { | ||
| sendEvent({ type: 'response.cancel' }) | ||
| currentMode = 'listening' | ||
| emit('mode_change', { mode: 'listening' }) | ||
| emit('interrupted', { messageId: currentMessageId ?? undefined }) | ||
| }, |
Contributor
There was a problem hiding this comment.
interrupt() and the conversation.item.truncated server event both emit 'interrupted' for the same message.
interrupt() immediately emits 'interrupted' (line 501), and later the server sends conversation.item.truncated which also emits 'interrupted' (line 309). The RealtimeClient listener runs twice for the same interruption: the message is marked interrupted: true twice and onInterrupted fires twice. Consider only emitting optimistically in interrupt() and suppressing the duplicate from conversation.item.truncated when a client-side interrupt was already issued (e.g., track a clientInterruptPending flag).
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 497 -
502, interrupt() is emitting 'interrupted' optimistically then the server's
conversation.item.truncated handler also emits 'interrupted', causing duplicate
notifications; introduce a clientInterruptPending boolean that interrupt() sets
to true before emitting (and include currentMessageId), then in the
conversation.item.truncated handler check clientInterruptPending and suppress
the emit('interrupted', ...) for the same messageId if the flag is set, finally
clear clientInterruptPending when the server event arrives (or on other flows
like response.cancel acknowledgement) so future interruptions behave normally;
reference interrupt(), the conversation.item.truncated handler,
currentMessageId, and emit('interrupted', ...) when adding the flag logic.
Comment on lines
+114
to
+118
| const tools: Array<Tool> = sessionData.tools.map((t) => ({ | ||
| name: t.name, | ||
| description: t.description, | ||
| inputSchema: t.parameters, | ||
| })) |
Contributor
There was a problem hiding this comment.
sessionData.tools.map() will throw if the OpenAI response omits tools.
When no tools are configured in the token options, OpenAI's sessions endpoint may return tools: null or omit the field entirely. Calling .map() on that would throw a TypeError at runtime.
🛡️ Proposed fix
- const tools: Array<Tool> = sessionData.tools.map((t) => ({
+ const tools: Array<Tool> = (sessionData.tools ?? []).map((t) => ({
name: t.name,
description: t.description,
inputSchema: t.parameters,
}))📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| const tools: Array<Tool> = sessionData.tools.map((t) => ({ | |
| name: t.name, | |
| description: t.description, | |
| inputSchema: t.parameters, | |
| })) | |
| const tools: Array<Tool> = (sessionData.tools ?? []).map((t) => ({ | |
| name: t.name, | |
| description: t.description, | |
| inputSchema: t.parameters, | |
| })) |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-openai/src/realtime/token.ts` around lines 114 - 118,
The code assumes sessionData.tools is an array and calls sessionData.tools.map
which will throw if tools is null/undefined; update the logic that builds tools
(the variable named tools and the sessionData.tools.map usage in token.ts) to
first coerce or guard the value (e.g., treat non-array/null as an empty array
using a nullish/Array.isArray check) and then map to the Tool shape so tools is
always an Array<Tool> (empty when no tools provided).
Comment on lines
+13
to
+15
| // Empty frequency data for when client is not connected | ||
| const emptyFrequencyData = new Uint8Array(128) | ||
| const emptyTimeDomainData = new Uint8Array(128).fill(128) |
Contributor
There was a problem hiding this comment.
Fallback empty-array sizes (128) don't match the adapter's actual output sizes (1024 / 2048).
When the client is not connected, getInputFrequencyData() returns a 128-element array; when connected, the OpenAI adapter returns 1024 elements (frequencyBinCount = fftSize/2). Visualization consumers that assume a fixed array length will misbehave when switching between the disconnected fallback and live data.
🛡️ Proposed fix — match the adapter's fftSize
-const emptyFrequencyData = new Uint8Array(128)
-const emptyTimeDomainData = new Uint8Array(128).fill(128)
+const emptyFrequencyData = new Uint8Array(1024) // frequencyBinCount = fftSize/2 = 1024
+const emptyTimeDomainData = new Uint8Array(2048).fill(128) // fftSize = 2048🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts` around lines 13 - 15,
The fallback arrays emptyFrequencyData and emptyTimeDomainData are sized at 128
but must match the adapter's FFT output (frequencyBinCount = 1024 and time
domain length = 2048) to avoid consumers seeing length changes; update
emptyFrequencyData to new Uint8Array(1024) and emptyTimeDomainData to new
Uint8Array(2048). Keep the time-domain default values (e.g., fill(128)) as
before so that getInputFrequencyData() returns the same-shaped arrays whether
connected or not.
Comment on lines
+153
to
+157
| useEffect(() => { | ||
| return () => { | ||
| clientRef.current?.destroy() | ||
| } | ||
| }, []) |
Contributor
There was a problem hiding this comment.
React StrictMode: destroy() clears stateChangeCallbacks but clientRef is not nulled — the subscription is permanently lost.
In React 18 Strict Mode the cleanup effect runs, calling destroy() which clears stateChangeCallbacks. On re-mount, if (!clientRef.current) evaluates to false (the destroyed client still exists), so the onStateChange subscription is never re-established. pendingUserTranscript / pendingAssistantTranscript will never update for the rest of that session.
🛡️ Proposed fix
return () => {
clientRef.current?.destroy()
+ clientRef.current = null
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| useEffect(() => { | |
| return () => { | |
| clientRef.current?.destroy() | |
| } | |
| }, []) | |
| useEffect(() => { | |
| return () => { | |
| clientRef.current?.destroy() | |
| clientRef.current = null | |
| } | |
| }, []) |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts` around lines 153 -
157, The cleanup effect currently calls clientRef.current?.destroy() but leaves
clientRef.current set, so after React StrictMode unmount/remount the destroyed
client exists and onStateChange subscriptions aren't re-registered causing
pendingUserTranscript/pendingAssistantTranscript to never update; update the
cleanup in the useEffect to call clientRef.current?.destroy() and then null out
clientRef.current (or set to undefined) so subsequent mounts detect no client
and recreate it and re-subscribe via onStateChange.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds realtime voice/text chat capabilities:
@tanstack/ai-client@tanstack/ai-openai@tanstack/ai-elevenlabspackageuseRealtimeChathook in@tanstack/ai-react@tanstack/aiexamples/ts-react-chatCo-Authored-By: Warp agent@warp.dev
Summary by CodeRabbit
New Features
Documentation
Examples