Repository navigation
Python: Decode base64 image, audio and blob content from MCP servers - #14675
Jae Birdsall (jaeminbird) wants to merge 1 commit into
Conversation
|
@microsoft-github-policy-service agree |
7581050 to
a68f006
Compare
a68f006 to
4c8d27c
Compare
|
Rebased again onto current main (4c8d27c), which also picks up #14095 in |
PRABHU KIRAN VANDRANKI (VANDRANKI)
left a comment
There was a problem hiding this comment.
I applied gh pr diff onto main (df1cb33) in a detached worktree. It applied cleanly. I ran everything below with Python 3.13.3 in python/, with the real mcp types.
Probe: I converted mcp.types.ImageContent, AudioContent and an EmbeddedResource with a BlobResourceContents (all carrying the base64 of 512 bytes, bytes(range(256)) * 2) with _mcp_content_types_to_kernel_content, then read .data, .mime_type and .data_uri (main, then PR):
| content | .data equals the 512 raw bytes |
.mime_type |
.data_uri starts with |
urllib.request.urlopen(data_uri).read() equals the raw bytes |
|---|---|---|---|---|
| image | False, then True | image/png, same |
data:image/png;,AAEC..., then data:image/png;base64,AAEC... |
False, then True |
| audio | False, then True | audio/wav, same |
data:audio/wav;,..., then data:audio/wav;base64,... |
False, then True |
blob resource (application/pdf) |
False, then True | text/plain, then application/pdf |
data:text/plain;,..., then data:application/pdf;base64,... |
False, then True |
The reverse direction, _kernel_content_to_mcp_content_types, hands back the original base64 string on both trees for all three. So a server-side round trip is not broken by the change.
Tests:
tests/unit/connectors/mcpplustests/unit/contentswith the PR applied: 511 passed. Main gives 508 passed.- The PR's
test_mcp.pyagainst unpatched main source: 3 failed (test_mcp_base64_content_is_decodedfor image, audio and blob_resource), 70 passed.
One behavior change to weigh, which is not a blocker. Decoding is now strict: the content classes reject data that is not strict base64. I converted an MCP ImageContent carrying each of these data strings:
- Standard base64: converts on both trees.
- Base64 wrapped at 76 columns with newlines (the MIME style), base64 without padding, the URL-safe alphabet, and the text
hello world!!: main converts all four and keeps the text bytes. The PR raisesContentInitializationError: Invalid base64 data.for all four. - Empty string: converts on both.
Through the plugin, with a mocked ClientSession, a newline-wrapped image makes both call_tool and get_prompt raise FunctionExecutionException on the PR, and both succeed on main (with undecoded data). MCP says image and audio data is base64, and main's result for that input was wrong bytes anyway. Still, a server that wraps lines would now fail the whole tool call. If that matters, a lenient decode (strip whitespace before decoding) or a fallback would avoid it. I did not try either, so I can not say what it costs.
I did not run the Google AI connector or any other connector that consumes data_uri. The statement in the description about the Google AI connector is read only.
Approving, with that note.
|
Eduard van Valkenburg (@eavanvalkenburg) Rebased onto current main again (b42ccc8, now signed); |
4c8d27c to
b42ccc8
Compare
Motivation and Context
MCP sends image/audio
dataand blobblobas base64, but the MCP connector buildsImageContent/AudioContent/BinaryContentwithoutdata_format="base64". So.dataholds the base64 text,data_uribecomesdata:image/png;,iVBOR...(no;base64), and blob resources lose their mimeType (text/plain). Tool results, prompts and sampling messages with images reach the model broken; the Google AI connector sends the base64 text as image bytes.Description
Pass
data_format="base64"for the three types, and the resource mimeType for blobs.call_toolandget_prompt; fails before, passes after.urllib.request.urlopen(content.data_uri)now returns the original bytes (before: the base64 text).tests/unit/connectors/mcpandtests/unit/contentspass; ruff and mypy clean.Contribution Checklist