Request builds incorrectly if data values are passed as bytes

[harrylloyd-bl]

OS platform: Windows
Python version: 3.12.0
Installed dependencies and versions (python -m pip freeze):
Conda environment: conda_env.log
httpx - 0.27.0

Code snippet

data = {  # sample data with bytes for the oauth_body_hash and oauth_signature
    'oauth_body_hash': b'myoauthbodyhash',  # bytes
    'oauth_signature': b'myoauthsignature'  # bytes
}


with httpx.Client() as c:
    req = c.build_request(
        method="POST",
        url="https://example.org",
        data=data
    )

    # break the request back down
    deconstruct_req = req.content.decode("utf-8").split("&")
    deconstructed_params = {p.split("=")[0]: p.split("=")[1] for p in deconstruct_req}

    assert deconstructed_params["oauth_signature"] == data["oauth_signature"]
    assert deconstructed_params["oauth_body_hash"] == data["oauth_body_hash"]

Error traceback
This fails because the oauth_signature and oauth_body_hash have the b'' byte object markers included when the body of the url is encoded

• Result: req.content = b"oauth_body_hash=b%27myoauthbodyhash%27&oauth_signature=b%27oauthsignature%27"
• Expected result: req.content = b"oauth_body_hash=myoauthbodyhash&oauth_signature=myoauthsignature"

Does the issue exist on HTTP/1.1, or HTTP/2, or both? HTTP/1.1, unsure about /2
Does the issue exist with Client, AsyncClient, or both? Both
When using AsyncClient does the issue exist when using asyncio or trio, or both? Both

This is loosely related to #3115

This will apply to any parameter in data, I came across it because the oauth lib I'm using returns the signature and body hash as bytes. If it's best practice that data should be passed as str not bytes then I'm happy to have this closed, but the RequestData typing is currently Mapping[str, Any], so that might be updated to exclude bytes.

The issue arises because Request calls _content.encode_urlencoded_data which uses _utils.primitive_value_to_str to stringify the values in data. _utils.primitive_value_to_str takes type PrimitiveData = Optional[Union[str, int, float, bool]], so isn't expecting bytes. Aside from decoding any bytes values in data before they're passed to build_request the fix that I've used is to add an elif to _utils.primitive_value_to_str to return the utf-8 decoded string if type == bytes. The typing for _utils.primitive_value_to_str would need to change though.

[ketpatil77]

For the OAuth values in your example, normalize the base64/ASCII bytes to strings before giving them to data=. HTTPX 0.28.1 still stringifies bytes in that mapping rather than treating them as already-decoded field values:

from urllib.parse import parse_qs
import httpx

def oauth_request(data):
    normalized = {
        key: value.decode("ascii") if isinstance(value, bytes) else value
        for key, value in data.items()
    }
    request = httpx.Request("POST", "https://example.org", data=normalized)
    decoded = parse_qs(request.content.decode("ascii"), keep_blank_values=True)
    return request, decoded

request, fields = oauth_request({"oauth_signature": b"abc+/=", "oauth_body_hash": b"hash"})
assert fields["oauth_signature"] == ["abc+/="]
assert fields["oauth_body_hash"] == ["hash"]

There are two separate issues in the original assertions: the decoded form value is a str while the input is bytes, and splitting on '&'/'=' does not perform percent/plus decoding and can mishandle '=' inside a value. parse_qs handles form decoding and repeated keys.

Strict ASCII decoding is appropriate for these ASCII OAuth outputs; do not silently UTF-8-decode arbitrary binary form data. For genuinely binary values, choose the endpoint's documented representation, such as multipart files or explicit base64. The HTTP request body is serialized before HTTP/1.1 versus HTTP/2 transport, so switching protocol does not change this conversion. Checked the original bytes-repr behavior and the example above. Form data.

Validation: HTTPX 0.28.1, Python 3.11.9.