Skip to content

Fix binary responses, header case-insensitivity, Content-Type leak; add timeout - #157

Merged
FlorianREGAZ merged 8 commits into
mainfrom
fix/response-correctness
Oct 10, 2026
Merged

FlorianREGAZ merged 8 commits into
mainfrom
fix/response-correctness

Conversation

@FlorianREGAZ

Copy link
Copy Markdown
Owner

Five response and request fixes, one commit each.

Binary downloads arrive intact (#143)

Requests now send isByteResponse: true. The library then returns the raw body as a data:<mime>;base64,<payload> URI (an empty body comes back as data:text/plain; charset=utf-8;base64,), and build_response decodes it into .content. The library guesses that mime type from the bytes, so it is ignored. .text decodes .content the way requests does: with the charset from the Content-Type header, else UTF-8, using errors="replace". The charset is exposed as response.encoding and can be set to decode with another one.

Fixes #143

Fewer copies of large bodies (#135, Python side)

.content is now the only stored copy of the body. .text is decoded each time it is read. The reply is decoded and parsed without binding the intermediate copies to names, and build_response pops the data URI and frees it before decoding. For a 36 MB JSON body, tracemalloc peak went from 192 MB to 95 MB and retained memory from 72 MB to 36 MB, on both Python 3.10 and 3.12. Memory held by the Go library is not affected.

Fixes #135

Response headers are case-insensitive (#129)

build_response used to replace the CaseInsensitiveDict with a plain dict. It now builds the CaseInsensitiveDict directly, so response.headers["content-type"] works.

Fixes #129

Inferred Content-Type no longer sticks to the session (#117)

execute_request used to write the Content-Type it inferred from json=/data= into self.headers. After one json= POST, every later request claimed application/json. The header now goes only into that request's copy of the headers, and only when neither the session nor the request sets a Content-Type (matched case-insensitively). self.headers is never modified.

Fixes #117

timeout= like requests (#108)

get/post/... and execute_request accept timeout= in seconds, as an int or a float. timeout_seconds= still works, and passing both raises ValueError. The value goes to the library as timeoutMilliseconds, so fractions of a second work. timeoutSeconds is no longer sent: it only takes whole seconds, and the library rejects requests that set both. session.timeout_seconds can also be a float. The docstring and README document all of this.

Fixes #108

Behaviour changes

  • response.content is now the exact bytes the server sent. Before, it was the library's UTF-8 text encoded again.
  • response.text is a read-only property and can no longer be assigned. When the header gives no charset it is decoded as UTF-8, with replacement characters for bytes that don't decode. Before, the library converted bodies to UTF-8 itself and also read <meta charset> from HTML. Pages that declare a non-UTF-8 charset only in <meta> now need response.encoding = "...".
  • New response.encoding attribute.
  • session.headers no longer gains a Content-Type after a json=/data= request. Code that relied on that leak has to set the header itself.
  • Passing Content-Type: None in a request's headers now suppresses the inferred Content-Type. None values in session.headers are dropped even when the request passes no headers.
  • The payload sends timeoutMilliseconds instead of timeoutSeconds.
  • Still unchanged: the library fixes a session's timeout on its first request, so a different timeout/timeout_seconds on a later request of the same session has no effect. The existing xfail test now covers both names, and the docstring and README mention it. Changing this would mean recreating the native session, which drops its connections and its cookie jar. I left that for a separate change.

Testing

  • New tests/integration/test_local_server.py: a stdlib ThreadingHTTPServer started in a thread. Real requests go through the native library for bytes(range(256)), a PNG, an 8 MiB random body, an empty body, JSON, UTF-8 and ISO-8859-1 bodies, mixed-case response headers, request-header echo for the Content-Type leak, and a slow endpoint for fractional timeouts.
  • tests/integration/test_httpbin.py: the binary-body xfail test used /base64, which httpbin can only return as UTF-8 text. It now downloads /image/png and compares the bytes with httpbin's file, and it no longer needs the xfail.
  • Unit tests: data URI decoding, charset handling, encoding override, a tracemalloc check that build_response doesn't hold the data URI, the payload and the bytes all at once, the Content-Type xfail removed plus new cases, and timeout conversion and conflict.
  • Locally on macOS arm64 with the v1.16.0 dylib: uv run ruff check, uv run ruff format --check, uv run mypy, uv lock --check pass. uv run pytest gives 147 passed, 7 xfailed (coverage 92%). uv run pytest -m network gives 9 passed, 1 xfailed. scripts/smoke_test.py passes.

🤖 Generated with Claude Code

FlorianREGAZ and others added 8 commits October 10, 2026 13:19
build_response copied the headers into a plain dict, which replaced the
CaseInsensitiveDict that Response starts with, so response.headers
["content-type"] raised KeyError. Build the CaseInsensitiveDict directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The library converted every body to UTF-8 text, and Response.content was
that text encoded again, so images and other binary downloads came back
corrupted. Requests now set isByteResponse, which makes the library send
the raw bytes as a base64 data URI, and build_response decodes them into
.content.

.text is now a property that decodes .content like requests does: with
the charset from the Content-Type header, else UTF-8, replacing bytes
that don't decode. The charset is exposed as .encoding and can be set to
decode with another one. The data URI's own mime type is ignored because
the library sniffs it from the bytes.

The httpbin test for this asked /base64 for PNG bytes, which httpbin can
only return as UTF-8 text; it now downloads /image/png. A stdlib server
covers exact bytes, empty and large bodies, and charsets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A response used to keep its body twice (.text and .content), and while
it was built the raw reply, its decoded string, the parsed body, .text
and .content were all alive at once. With a 36 MB JSON body,
tracemalloc showed a 192 MB peak and 72 MB retained.

.content is now the only stored copy (.text is decoded on access since
the previous commit), the reply is decoded and parsed without binding
the intermediate copies, and build_response pops the data URI from the
reply and drops it before decoding. Measured the same way, on Python
3.10 and 3.12: 95 MB peak, 36 MB retained. Memory the Go library holds
is not affected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
execute_request wrote the Content-Type it inferred from json= or data=
into self.headers. After one json= POST, every later request on the
session claimed to be application/json, form posts included.

The label now goes into the per-request copy of the headers, and only if
neither the session nor the request set a Content-Type (in any case).
Setting it to None for a request sends none. Request headers are always
merged into a copy, so None values in session headers are dropped too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Requests take timeout= now, in seconds as an int or float, next to the
existing timeout_seconds=. Passing both raises ValueError instead of
picking one silently. session.timeout_seconds can be a float as well.

The timeout goes to the library as timeoutMilliseconds, so fractions of
a second work; timeoutSeconds only takes whole seconds and the library
refuses requests that set both, so it is no longer sent.

The library still fixes a session's timeout on its first request (the
existing xfail test, now run for both names). The docstring and README
say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Windows reports a client that gave up as ConnectionAbortedError, which
the handler didn't catch. Catch ConnectionError, the base of all three,
so the timeout tests don't print tracebacks from the server thread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The library used to pick the charset of text responses itself, including
from <meta charset> tags. Since .text is now decoded in Python, pages that
only declare GBK or Shift_JIS in the document would turn into mojibake.
Sniff BOMs, <meta> and <?xml?> declarations in the first 1024 bytes when
the Content-Type header names no charset, and decode GB2312 as GBK like
browsers do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FlorianREGAZ
FlorianREGAZ force-pushed the fix/response-correctness branch from a6f5287 to 64e2f7a Compare October 10, 2026 11:19
@FlorianREGAZ
FlorianREGAZ merged commit 4855d70 into main Oct 10, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant