Repository navigation
Python: fix(contents): accept RFC 6838 media types and parameters in data URIs - #14663
Chen Yufeiyang (feiiiiii5) wants to merge 4 commits into
Conversation
The data URI parser only allowed letters and hyphens in the media type and treated everything after the first semicolon as the data format, so image/svg+xml was rejected and charset=utf-8;base64 left the base64 payload undecoded. Parse the media type and parameters with the RFC 6838 token character set and keep the optional data format as a token.
There was a problem hiding this comment.
🟡 Changes recommended
The revised parser rejects valid parameter characters and shorthand URIs that omit the media type.
2 open issues
What changed in this PR
This PR updates Python’s data URI parser to accept additional media-type characters and correctly separate parameters from the base64 marker.
Changes:
- Expands the parser’s media-type and parameter character classes.
- Adds regression tests for SVG, vendor media types, and charset parameters before base64.
- Tests
ImageContentconstruction from an SVG data URI.
| File | Description |
|---|---|
python/tests/unit/contents/test_image_content.py |
Adds SVG data URI construction coverage. |
python/tests/unit/contents/test_data_uri.py |
Adds media-type and parameter parsing cases. |
python/semantic_kernel/contents/utils/data_uri.py |
Revises the data URI parsing expression. |
Review effort: Balanced
| # Parameters use the RFC 2045 token alphabet, which is broader than a | ||
| # media type's restricted-name alphabet. RFC 2397 allows parameters even | ||
| # when the default text/plain media type is omitted. | ||
| parameter_token = r"[a-zA-Z0-9!#$%&'*+.^_`|~-]+" |
There was a problem hiding this comment.
The current pre-commit job fails on this assignment with Bandit B105:hardcoded_password_string (https://github.com/microsoft/semantic-kernel/actions/runs/40984502209). This is a false positive: the value is a regex, not a credential, but the variable name parameter_token triggers the check. Please rename it to something like parameter_pattern so the existing hook passes without weakening the security checks.
There was a problem hiding this comment.
Done — renamed the local regex variable to parameter_pattern (both the definition and the two references in the f-string). Bandit B105 no longer fires; I left the token alphabet and behavior unchanged and did not add # nosec. Verified locally with the same pre-commit hook (bandit passes) and pytest tests/unit/contents/test_data_uri.py (48 passed).
The local regex variable was named parameter_token, which Bandit flags as B105 (possible hardcoded password) even though it is a character-class pattern. Rename it to parameter_pattern instead of suppressing the check. Same behavior; pre-commit bandit hook and tests/unit/contents/test_data_uri.py pass.
PRABHU KIRAN VANDRANKI (VANDRANKI)
left a comment
There was a problem hiding this comment.
I applied gh pr diff onto main (df1cb33) in a detached worktree. It applied cleanly. I ran everything below with Python 3.13.3 in python/.
Tests:
tests/unit/contents,tests/unit/prompt_templateandtests/unit/template_enginewith the PR applied: 1140 passed. Main gives 1127 passed.- The PR's
test_data_uri.pyandtest_image_content.pyagainst unpatched main source: 13 failed, 49 passed. All 13 failures are new tests.
Direct checks (main, then PR):
| input | main | PR |
|---|---|---|
data:image/svg+xml;base64,PHN2Zy8+ |
ContentInitializationError |
mime image/svg+xml, bytes b'<svg/>' |
ImageContent(data_uri=<that svg URI>) |
ContentInitializationError |
mime image/svg+xml, data b'<svg/>' |
data:text/plain;charset=utf-8;base64,SGVsbG8= |
data_format='charset=utf-8;base64', bytes b'SGVsbG8=' (not decoded) |
params {'charset': 'utf-8'}, format base64, bytes b'Hello' |
data:text/plain;name=a.txt;base64,SGVsbG8= |
not decoded, same as above | params {'name': 'a.txt'}, bytes b'Hello' |
data:text/plain,Hello (no ;) |
ContentInitializationError |
bytes b'Hello' |
To look for regressions I generated 1920 data URIs from 8 media types (including none and TEXT/PLAIN), 8 parameter strings, 5 format strings and 6 payloads, parsed each with DataUri.from_data_uri on both trees and compared.
- Among the 480 sane inputs (valid parameters,
base64or none with a payload that matches, expected result worked out by hand): main gives the expected mime type, parameters, format and bytes for 42 of them. The PR gives it for all 480. None of the 480 is right on main and wrong on the PR. - 543 of the 1920 parse on main but raise on the PR. I put them in buckets by what the input contains. The
;;base64bucket has 93, thea==bparameter bucket has 114, and thecharset="utf-8"bucket has 150. On main each of those was accepted by putting the whole text intodata_format, so the decoded bytes were wrong. The 186 left over are the remainder. I only looked at five of them, and those werebase64markers with payloads that are not base64, such asa,bandHello.
One limit, not a blocker. A quoted parameter value, as in data:text/plain;charset="utf-8";base64,SGVsbG8=, is valid in RFC 2045, and the PR raises ContentInitializationError for it. Main did not parse it correctly either: it returned b'SGVsbG8=' undecoded. Raising is safer than returning wrong bytes. If you want quoted values, the parameter pattern would need a quoted-string branch. I did not test that.
No blockers from me. Approving.

Motivation and Context
Valid RFC 2397 data URIs are rejected or silently mis-decoded when the media type contains
+(for exampleimage/svg+xml) or when a parameter precedesbase64(for exampledata:text/plain;charset=utf-8;base64,...).ImageContent(data_uri="data:image/svg+xml;base64,PHN2Zy8+")raisesContentInitializationError: Invalid data uri format.DataUri.from_data_uri("data:text/plain;charset=utf-8;base64,SGVsbG8=")does not raise, but returns the base64 text as bytes instead of decoding it, becausecharset=utf-8;base64is treated as one data-format string. Both are valid data URIs, and the second is a silent data-corruption path.Description
The parser now uses RFC 6838/RFC 2045 token characters for the media type and parameters, and keeps the optional data format as a token. This accepts
image/svg+xml, vendor types such asapplication/vnd.ms-excel, and;charset=utf-8;base64without treating the parameter as part of the base64 marker.Tests added for the media-type
+, a parameter beforebase64, a vendor media type, andImageContentconstruction from an SVG data URI. On the base commit483ffb4, the four new tests fail; on this branch the related contents/prompt/template/text suites are 1,111 passed.ruff checkandruff format --checkare clean.Contribution Checklist
ruff checkandruff format --checkpass on the changed files.