Skip to content

fix(core): infer Anthropic tool_result image MIME type#2036

Open
BABTUNA wants to merge 1 commit intobrowserbase:mainfrom
BABTUNA:fix-anthropic-cua-tool-result-mime
Open

fix(core): infer Anthropic tool_result image MIME type#2036
BABTUNA wants to merge 1 commit intobrowserbase:mainfrom
BABTUNA:fix-anthropic-cua-tool-result-mime

Conversation

@BABTUNA
Copy link
Copy Markdown
Contributor

@BABTUNA BABTUNA commented Apr 24, 2026

why

Anthropic CUA computer tool results currently hardcode media_type: "image/png" and strip only a PNG prefix from screenshot data URLs.

That is brittle if screenshot input is non-PNG (for example JPEG/WebP), and can produce mismatched metadata or malformed image payloads.

Fixes #2035.

what changed

  • Added a small internal parser in AnthropicCUAClient to extract:
    • media_type from image data URLs
    • base64 payload without the data URL prefix
  • Applied parsed MIME/data in both computer-tool image paths:
    • normal success tool result
    • error fallback tool result
  • Kept a PNG fallback path for malformed/unexpected screenshot strings.
  • Added focused unit tests in packages/core/tests/unit/anthropic-cua-client.test.ts for:
    • JPEG MIME parsing in success path
    • PNG fallback on non-data URL input
    • MIME/data handling in error path retry screenshot payload

test plan

  • npm.cmd exec prettier -- --check packages/core/lib/v3/agent/AnthropicCUAClient.ts packages/core/tests/unit/anthropic-cua-client.test.ts
  • node node_modules/vitest/vitest.mjs run --config .tmp-vitest-unit-config.mjs from packages/core (temporary local config targeting tests/unit/anthropic-cua-client.test.ts)

Summary by cubic

Infer MIME type and base64 data from screenshot data URLs for Anthropic CUA tool_result images instead of forcing PNG. Prevents mismatched metadata and malformed image payloads. Fixes #2035.

  • Bug Fixes
    • Added a parser in AnthropicCUAClient to extract media_type and base64 data from image data URLs.
    • Used parsed values in both success and error computer tool_result paths, with a PNG fallback for malformed inputs.
    • Added unit tests for JPEG parsing, fallback behavior, and error-path screenshot payloads.

Written for commit 9aa0587. Summary will update on new commits. Review in cubic

Use parsed data URL MIME/data for computer tool_result images in both success and error paths, with PNG fallback for malformed payloads. Adds focused unit coverage for JPEG parsing, fallback behavior, and error-path payload handling. Refs browserbase#2035.
@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented Apr 24, 2026

⚠️ No Changeset found

Latest commit: 9aa0587

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions
Copy link
Copy Markdown
Contributor

This PR is from an external contributor and must be approved by a stagehand team member with write access before CI can run.
Approving the latest commit mirrors it into an internal PR owned by the approver.
If new commits are pushed later, the internal PR stays open but is marked stale until someone approves the latest external commit and refreshes it.

@github-actions github-actions Bot added external-contributor Tracks PRs mirrored from external contributor forks. external-contributor:awaiting-approval Waiting for a stagehand team member to approve the latest external commit. labels Apr 24, 2026
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contributor:awaiting-approval Waiting for a stagehand team member to approve the latest external commit. external-contributor Tracks PRs mirrored from external contributor forks.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

core(cua): Anthropic tool_result image handling hardcodes PNG media type

1 participant