Skip to content

Python: Decode base64 image, audio and blob content from MCP servers - #14675

Open
Jae Birdsall (jaeminbird) wants to merge 1 commit into
microsoft:mainfrom
jaeminbird:fix-mcp-base64-content
Open

Jae Birdsall (jaeminbird) wants to merge 1 commit into
microsoft:mainfrom
jaeminbird:fix-mcp-base64-content

Conversation

@jaeminbird

Copy link
Copy Markdown
Contributor

Motivation and Context

MCP sends image/audio data and blob blob as base64, but the MCP connector builds ImageContent/AudioContent/BinaryContent without data_format="base64". So .data holds the base64 text, data_uri becomes data:image/png;,iVBOR... (no ;base64), and blob resources lose their mimeType (text/plain). Tool results, prompts and sampling messages with images reach the model broken; the Google AI connector sends the base64 text as image bytes.

Description

Pass data_format="base64" for the three types, and the resource mimeType for blobs.

  • New test covers image, audio and a blob resource via call_tool and get_prompt; fails before, passes after.
  • urllib.request.urlopen(content.data_uri) now returns the original bytes (before: the base64 text).
  • tests/unit/connectors/mcp and tests/unit/contents pass; ruff and mypy clean.

Contribution Checklist

Copilot AI balanced review requested due to automatic review settings October 29, 2026 06:19
@jaeminbird
Jae Birdsall (jaeminbird) requested a review from a team as a code owner October 29, 2026 06:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@semantic-kernel-automation semantic-kernel-automation Bot added the python Pull requests for the Python Semantic Kernel label Oct 29, 2026
@jaeminbird

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree

@eavanvalkenburg

Copy link
Copy Markdown
Member

This is approved, but GitHub now marks its merge-queue entry UNMERGEABLE. I confirmed a conflict with #14676 in python/tests/unit/connectors/mcp/test_mcp.py: both PRs append tests at the same location. Please rebase after #14676 lands and keep both sets of regression tests.

@jaeminbird

Copy link
Copy Markdown
Contributor Author

Rebased onto main after #14676 (a68f006). Both sets of tests are kept in test_mcp.py, and tests/unit/connectors/mcp passes (69 tests).

@jaeminbird

Copy link
Copy Markdown
Contributor Author

Rebased again onto current main (4c8d27c), which also picks up #14095 in mcp.py. Both sets of regression tests from #14676 and this PR are kept in test_mcp.py; tests/unit/connectors/mcp passes (73 tests) and ruff/mypy are clean. Thanks Eduard van Valkenburg (@eavanvalkenburg) for tracking down the conflict.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I applied gh pr diff onto main (df1cb33) in a detached worktree. It applied cleanly. I ran everything below with Python 3.13.3 in python/, with the real mcp types.

Probe: I converted mcp.types.ImageContent, AudioContent and an EmbeddedResource with a BlobResourceContents (all carrying the base64 of 512 bytes, bytes(range(256)) * 2) with _mcp_content_types_to_kernel_content, then read .data, .mime_type and .data_uri (main, then PR):

content .data equals the 512 raw bytes .mime_type .data_uri starts with urllib.request.urlopen(data_uri).read() equals the raw bytes
image False, then True image/png, same data:image/png;,AAEC..., then data:image/png;base64,AAEC... False, then True
audio False, then True audio/wav, same data:audio/wav;,..., then data:audio/wav;base64,... False, then True
blob resource (application/pdf) False, then True text/plain, then application/pdf data:text/plain;,..., then data:application/pdf;base64,... False, then True

The reverse direction, _kernel_content_to_mcp_content_types, hands back the original base64 string on both trees for all three. So a server-side round trip is not broken by the change.

Tests:

  • tests/unit/connectors/mcp plus tests/unit/contents with the PR applied: 511 passed. Main gives 508 passed.
  • The PR's test_mcp.py against unpatched main source: 3 failed (test_mcp_base64_content_is_decoded for image, audio and blob_resource), 70 passed.

One behavior change to weigh, which is not a blocker. Decoding is now strict: the content classes reject data that is not strict base64. I converted an MCP ImageContent carrying each of these data strings:

  • Standard base64: converts on both trees.
  • Base64 wrapped at 76 columns with newlines (the MIME style), base64 without padding, the URL-safe alphabet, and the text hello world!!: main converts all four and keeps the text bytes. The PR raises ContentInitializationError: Invalid base64 data. for all four.
  • Empty string: converts on both.

Through the plugin, with a mocked ClientSession, a newline-wrapped image makes both call_tool and get_prompt raise FunctionExecutionException on the PR, and both succeed on main (with undecoded data). MCP says image and audio data is base64, and main's result for that input was wrong bytes anyway. Still, a server that wraps lines would now fail the whole tool call. If that matters, a lenient decode (strip whitespace before decoding) or a fallback would avoid it. I did not try either, so I can not say what it costs.

I did not run the Google AI connector or any other connector that consumes data_uri. The statement in the description about the Google AI connector is read only.

Approving, with that note.

@jaeminbird

Copy link
Copy Markdown
Contributor Author

Eduard van Valkenburg (@eavanvalkenburg) Rebased onto current main again (b42ccc8, now signed); tests/unit/connectors/mcp passes (73 tests). Could you requeue it when you get a chance?

This branch was successfully deployed

1 active deployment
github-app-auth — b42ccc8a Deployed Nov 10, 2026 by jaeminbird via add_label #29654
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Pull requests for the Python Semantic Kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants