Get Response Content on DecodeError

[aaronst]

Sometimes a server tells us content is encoded a certain way, even though it's not. This causes a DecodeError when we try to make a request. Is there a way to access the raw response bytes after a DecodeError occurs?

[ketpatil77]

For malformed HTTP Content-Encoding (such as a body labelled gzip that is not gzip), capture through streaming iter_raw() before asking HTTPX to decode it. A DecodingError from an already consumed non-streaming get is not a retained raw-body container.

def capture_encoded_body(client, url, limit=1024 * 1024):
    if limit < 0:
        raise ValueError("limit must be non-negative")
    with client.stream("GET", url) as response:
        chunks, size = [], 0
        for chunk in response.iter_raw():
            size += len(chunk)
            if size > limit:
                raise ValueError("encoded body exceeds capture limit")
            chunks.append(chunk)
        return response.status_code, dict(response.headers), b"".join(chunks)

Here “raw” means the content-encoded entity bytes; HTTP transfer framing has already been handled. You can inspect/store the returned bytes and decide how to decode them yourself. Requesting Accept-Encoding: identity may avoid compression if the server honors it, but it is not a recovery mechanism for the failed response.

This helper performs a new GET. Do not automatically replay a failed POST or another non-idempotent operation to recover bytes, and do not assume a second request returns the same representation. If preserving the exact failing response matters, capture it this way on the original request. This concerns HTTP compression decoding, not a mistaken text charset; for charset issues, preserve response.content before decoding text.

Validated a stream labelled gzip containing invalid gzip bytes: normal .read() raises httpx.DecodingError, while iter_raw() preserves those bytes; oversized capture is rejected. Raw response streaming.

Validation: HTTPX 0.28.1, Python 3.11.9.