fix(core): Improve Vercel AI SDK instrumentation attributes#19717
Open
RulaKhaled wants to merge 7 commits intodevelopfrom
Open
fix(core): Improve Vercel AI SDK instrumentation attributes#19717RulaKhaled wants to merge 7 commits intodevelopfrom
RulaKhaled wants to merge 7 commits intodevelopfrom
Conversation
Contributor
size-limit report 📦
|
Contributor
node-overhead report 🧳Note: This is a synthetic benchmark with a minimal express app and does not necessarily reflect the real-world performance impact in an application.
|
RulaKhaled
changed the title
RulaKhaled
changed the title
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
Comment on lines
+135
to
+144
| function hasVercelImageData(part: NonNullable<unknown>): part is { type: 'image'; image: string; mimeType: string } { | ||
| return ( | ||
| 'type' in part && | ||
| part.type === 'image' && | ||
| 'image' in part && | ||
| typeof part.image === 'string' && | ||
| 'mimeType' in part && | ||
| typeof part.mimeType === 'string' | ||
| ); | ||
| } |
There was a problem hiding this comment.
Bug: The hasVercelImageData function incorrectly identifies HTTP/HTTPS URLs as media to be stripped, causing telemetry data loss when the URL is replaced with a placeholder.
Severity: MEDIUM
Suggested Fix
Update the hasVercelImageData function to verify that the image string is a data URL (e.g., by checking if it startsWith('data:')) or a base64 string, rather than just checking if its type is string. This will prevent it from incorrectly redacting standard HTTP/HTTPS URLs.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.
Location: packages/core/src/tracing/ai/mediaStripping.ts#L135-L144
Potential issue: The function `hasVercelImageData` checks if the `image` property is a
string, but does not differentiate between a URL and a base64-encoded data string. This
causes the media stripping logic in `stripInlineMediaFromSingleMessage` to incorrectly
redact HTTP/HTTPS URLs by replacing them with `[Blob substitute]`. This behavior leads
to the loss of valuable telemetry data, as the URL reference to the image is removed,
preventing users from seeing which images were part of the AI operation. The intended
behavior is to only strip binary data like base64 strings or data URLs, while preserving
standard URLs.
Did we get this right? 👍 / 👎 to inform future reviews.
Comment on lines
+316
to
+319
| // eslint-disable-next-line @typescript-eslint/no-dynamic-delete | ||
| delete attributes[AI_RESPONSE_TEXT_ATTRIBUTE]; | ||
| // eslint-disable-next-line @typescript-eslint/no-dynamic-delete | ||
| delete attributes[AI_RESPONSE_TOOL_CALLS_ATTRIBUTE]; |
There was a problem hiding this comment.
Bug: Source attributes AI_RESPONSE_TEXT_ATTRIBUTE and AI_RESPONSE_TOOL_CALLS_ATTRIBUTE are deleted even if buildOutputMessages fails to process them, leading to data loss in telemetry.
Severity: MEDIUM
Suggested Fix
The deletion of AI_RESPONSE_TEXT_ATTRIBUTE and AI_RESPONSE_TOOL_CALLS_ATTRIBUTE should be conditional. Only delete these attributes if the buildOutputMessages function successfully generates and sets the GEN_AI_OUTPUT_MESSAGES_ATTRIBUTE.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.
Location: packages/core/src/tracing/vercel-ai/index.ts#L316-L319
Potential issue: In `processEndedVercelAiSpan`, the attributes
`AI_RESPONSE_TEXT_ATTRIBUTE` and `AI_RESPONSE_TOOL_CALLS_ATTRIBUTE` are unconditionally
deleted after `buildOutputMessages` is called. However, `buildOutputMessages` may not
generate any output if it receives an empty string for `responseText` and invalid JSON
for `responseToolCalls`. In this scenario, the source attributes are deleted without
being captured in the `gen_ai.output.messages` attribute, resulting in silent and
permanent loss of telemetry data.
Did we get this right? 👍 / 👎 to inform future reviews.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: V6 tests missing new output messages attribute assertions
- Added explicit
GEN_AI_OUTPUT_MESSAGES_ATTRIBUTEassertions (and import) across the v6 span expectations sogen_ai.output.messagesis now validated for text and tool-call outputs.
- Added explicit
Or push these changes by commenting:
@cursor push 8e0d6cceb7
Preview (8e0d6cceb7)
diff --git a/dev-packages/node-integration-tests/suites/tracing/vercelai/v6/test.ts b/dev-packages/node-integration-tests/suites/tracing/vercelai/v6/test.ts
--- a/dev-packages/node-integration-tests/suites/tracing/vercelai/v6/test.ts
+++ b/dev-packages/node-integration-tests/suites/tracing/vercelai/v6/test.ts
@@ -4,6 +4,7 @@
import {
GEN_AI_INPUT_MESSAGES_ATTRIBUTE,
GEN_AI_OPERATION_NAME_ATTRIBUTE,
+ GEN_AI_OUTPUT_MESSAGES_ATTRIBUTE,
GEN_AI_REQUEST_AVAILABLE_TOOLS_ATTRIBUTE,
GEN_AI_REQUEST_MODEL_ATTRIBUTE,
GEN_AI_RESPONSE_FINISH_REASONS_ATTRIBUTE,
@@ -97,6 +98,8 @@
'vercel.ai.settings.maxRetries': 2,
'vercel.ai.streaming': false,
[GEN_AI_INPUT_MESSAGES_ATTRIBUTE]: '[{"role":"user","content":"Where is the second span?"}]',
+ [GEN_AI_OUTPUT_MESSAGES_ATTRIBUTE]:
+ '[{"role":"assistant","parts":[{"type":"text","content":"Second span here!"}],"finish_reason":"stop"}]',
[GEN_AI_RESPONSE_MODEL_ATTRIBUTE]: 'mock-model-id',
[GEN_AI_USAGE_INPUT_TOKENS_ATTRIBUTE]: 10,
[GEN_AI_USAGE_OUTPUT_TOKENS_ATTRIBUTE]: 20,
@@ -129,6 +132,8 @@
'vercel.ai.response.id': expect.any(String),
'vercel.ai.response.timestamp': expect.any(String),
[GEN_AI_INPUT_MESSAGES_ATTRIBUTE]: expect.any(String),
+ [GEN_AI_OUTPUT_MESSAGES_ATTRIBUTE]:
+ '[{"role":"assistant","parts":[{"type":"text","content":"Second span here!"}],"finish_reason":"stop"}]',
[GEN_AI_RESPONSE_FINISH_REASONS_ATTRIBUTE]: ['stop'],
[GEN_AI_USAGE_INPUT_TOKENS_ATTRIBUTE]: 10,
[GEN_AI_USAGE_OUTPUT_TOKENS_ATTRIBUTE]: 20,
@@ -231,6 +236,8 @@
'vercel.ai.prompt': '[{"role":"user","content":"Where is the first span?"}]',
'vercel.ai.request.headers.user-agent': expect.any(String),
[GEN_AI_INPUT_MESSAGES_ATTRIBUTE]: '[{"role":"user","content":"Where is the first span?"}]',
+ [GEN_AI_OUTPUT_MESSAGES_ATTRIBUTE]:
+ '[{"role":"assistant","parts":[{"type":"text","content":"First span here!"}],"finish_reason":"stop"}]',
'vercel.ai.response.finishReason': 'stop',
'vercel.ai.settings.maxRetries': 2,
'vercel.ai.streaming': false,
@@ -257,6 +264,8 @@
'vercel.ai.request.headers.user-agent': expect.any(String),
[GEN_AI_INPUT_MESSAGES_ATTRIBUTE]:
'[{"role":"user","content":[{"type":"text","text":"Where is the first span?"}]}]',
+ [GEN_AI_OUTPUT_MESSAGES_ATTRIBUTE]:
+ '[{"role":"assistant","parts":[{"type":"text","content":"First span here!"}],"finish_reason":"stop"}]',
'vercel.ai.response.finishReason': 'stop',
'vercel.ai.response.id': expect.any(String),
'vercel.ai.response.model': 'mock-model-id',
@@ -289,6 +298,8 @@
'vercel.ai.prompt': '[{"role":"user","content":"Where is the second span?"}]',
'vercel.ai.request.headers.user-agent': expect.any(String),
[GEN_AI_INPUT_MESSAGES_ATTRIBUTE]: '[{"role":"user","content":"Where is the second span?"}]',
+ [GEN_AI_OUTPUT_MESSAGES_ATTRIBUTE]:
+ '[{"role":"assistant","parts":[{"type":"text","content":"Second span here!"}],"finish_reason":"stop"}]',
'vercel.ai.response.finishReason': 'stop',
'vercel.ai.settings.maxRetries': 2,
'vercel.ai.streaming': false,
@@ -324,6 +335,8 @@
'vercel.ai.response.id': expect.any(String),
'vercel.ai.response.timestamp': expect.any(String),
[GEN_AI_INPUT_MESSAGES_ATTRIBUTE]: expect.any(String),
+ [GEN_AI_OUTPUT_MESSAGES_ATTRIBUTE]:
+ '[{"role":"assistant","parts":[{"type":"text","content":"Second span here!"}],"finish_reason":"stop"}]',
[GEN_AI_RESPONSE_FINISH_REASONS_ATTRIBUTE]: ['stop'],
[GEN_AI_USAGE_INPUT_TOKENS_ATTRIBUTE]: 10,
[GEN_AI_USAGE_OUTPUT_TOKENS_ATTRIBUTE]: 20,
@@ -346,6 +359,8 @@
'vercel.ai.prompt': '[{"role":"user","content":"What is the weather in San Francisco?"}]',
'vercel.ai.request.headers.user-agent': expect.any(String),
[GEN_AI_INPUT_MESSAGES_ATTRIBUTE]: '[{"role":"user","content":"What is the weather in San Francisco?"}]',
+ [GEN_AI_OUTPUT_MESSAGES_ATTRIBUTE]:
+ '[{"role":"assistant","parts":[{"type":"tool_call","id":"call-1","name":"getWeather","arguments":"{\\"location\\":\\"San Francisco\\"}"}],"finish_reason":"tool-calls"}]',
'vercel.ai.response.finishReason': 'tool-calls',
'vercel.ai.settings.maxRetries': 2,
'vercel.ai.streaming': false,
@@ -371,6 +386,8 @@
'vercel.ai.pipeline.name': 'generateText.doGenerate',
'vercel.ai.request.headers.user-agent': expect.any(String),
[GEN_AI_INPUT_MESSAGES_ATTRIBUTE]: expect.any(String),
+ [GEN_AI_OUTPUT_MESSAGES_ATTRIBUTE]:
+ '[{"role":"assistant","parts":[{"type":"tool_call","id":"call-1","name":"getWeather","arguments":"{\\"location\\":\\"San Francisco\\"}"}],"finish_reason":"tool-calls"}]',
'vercel.ai.prompt.toolChoice': expect.any(String),
[GEN_AI_REQUEST_AVAILABLE_TOOLS_ATTRIBUTE]: EXPECTED_AVAILABLE_TOOLS_JSON,
'vercel.ai.response.finishReason': 'tool-calls',| 'vercel.ai.prompt': '[{"role":"user","content":"Where is the second span?"}]', | ||
| 'vercel.ai.request.headers.user-agent': expect.any(String), | ||
| 'vercel.ai.response.finishReason': 'stop', | ||
| [GEN_AI_RESPONSE_TEXT_ATTRIBUTE]: expect.any(String), |
There was a problem hiding this comment.
V6 tests missing new output messages attribute assertions
Medium Severity
The v6 tests remove GEN_AI_RESPONSE_TEXT_ATTRIBUTE and GEN_AI_RESPONSE_TOOL_CALLS_ATTRIBUTE assertions but never add corresponding GEN_AI_OUTPUT_MESSAGES_ATTRIBUTE assertions (it's not even imported). The v4 and v5 tests properly add the new attribute assertions. Since v6 tests use expect.objectContaining throughout, the new gen_ai.output.messages behavior is completely unverified for v6, even though the v6 mock model does emit ai.response.text. This violates the rule to check that tests actually test newly added behavior.
Additional Locations (1)
Triggered by project rule: PR Review Guidelines for Cursor Bot
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces some attributes and fixes to Vercel AI SDK:
Closes #19574