fix(core): infer Anthropic tool_result image MIME type#2036
fix(core): infer Anthropic tool_result image MIME type#2036BABTUNA wants to merge 1 commit intobrowserbase:mainfrom
Conversation
Use parsed data URL MIME/data for computer tool_result images in both success and error paths, with PNG fallback for malformed payloads. Adds focused unit coverage for JPEG parsing, fallback behavior, and error-path payload handling. Refs browserbase#2035.
|
|
This PR is from an external contributor and must be approved by a stagehand team member with write access before CI can run. |
why
Anthropic CUA computer tool results currently hardcode
media_type: "image/png"and strip only a PNG prefix from screenshot data URLs.That is brittle if screenshot input is non-PNG (for example JPEG/WebP), and can produce mismatched metadata or malformed image payloads.
Fixes #2035.
what changed
AnthropicCUAClientto extract:media_typefrom image data URLspackages/core/tests/unit/anthropic-cua-client.test.tsfor:test plan
npm.cmd exec prettier -- --check packages/core/lib/v3/agent/AnthropicCUAClient.ts packages/core/tests/unit/anthropic-cua-client.test.tsnode node_modules/vitest/vitest.mjs run --config .tmp-vitest-unit-config.mjsfrompackages/core(temporary local config targetingtests/unit/anthropic-cua-client.test.ts)Summary by cubic
Infer MIME type and base64 data from screenshot data URLs for Anthropic CUA
tool_resultimages instead of forcing PNG. Prevents mismatched metadata and malformed image payloads. Fixes #2035.AnthropicCUAClientto extractmedia_typeand base64 data from image data URLs.computertool_result paths, with a PNG fallback for malformed inputs.Written for commit 9aa0587. Summary will update on new commits. Review in cubic