Repository navigation
Fix binary responses, header case-insensitivity, Content-Type leak; add timeout - #157
Merged
Merged
Conversation
build_response copied the headers into a plain dict, which replaced the CaseInsensitiveDict that Response starts with, so response.headers ["content-type"] raised KeyError. Build the CaseInsensitiveDict directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The library converted every body to UTF-8 text, and Response.content was that text encoded again, so images and other binary downloads came back corrupted. Requests now set isByteResponse, which makes the library send the raw bytes as a base64 data URI, and build_response decodes them into .content. .text is now a property that decodes .content like requests does: with the charset from the Content-Type header, else UTF-8, replacing bytes that don't decode. The charset is exposed as .encoding and can be set to decode with another one. The data URI's own mime type is ignored because the library sniffs it from the bytes. The httpbin test for this asked /base64 for PNG bytes, which httpbin can only return as UTF-8 text; it now downloads /image/png. A stdlib server covers exact bytes, empty and large bodies, and charsets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A response used to keep its body twice (.text and .content), and while it was built the raw reply, its decoded string, the parsed body, .text and .content were all alive at once. With a 36 MB JSON body, tracemalloc showed a 192 MB peak and 72 MB retained. .content is now the only stored copy (.text is decoded on access since the previous commit), the reply is decoded and parsed without binding the intermediate copies, and build_response pops the data URI from the reply and drops it before decoding. Measured the same way, on Python 3.10 and 3.12: 95 MB peak, 36 MB retained. Memory the Go library holds is not affected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
execute_request wrote the Content-Type it inferred from json= or data= into self.headers. After one json= POST, every later request on the session claimed to be application/json, form posts included. The label now goes into the per-request copy of the headers, and only if neither the session nor the request set a Content-Type (in any case). Setting it to None for a request sends none. Request headers are always merged into a copy, so None values in session headers are dropped too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Requests take timeout= now, in seconds as an int or float, next to the existing timeout_seconds=. Passing both raises ValueError instead of picking one silently. session.timeout_seconds can be a float as well. The timeout goes to the library as timeoutMilliseconds, so fractions of a second work; timeoutSeconds only takes whole seconds and the library refuses requests that set both, so it is no longer sent. The library still fixes a session's timeout on its first request (the existing xfail test, now run for both names). The docstring and README say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Windows reports a client that gave up as ConnectionAbortedError, which the handler didn't catch. Catch ConnectionError, the base of all three, so the timeout tests don't print tracebacks from the server thread. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The library used to pick the charset of text responses itself, including from <meta charset> tags. Since .text is now decoded in Python, pages that only declare GBK or Shift_JIS in the document would turn into mojibake. Sniff BOMs, <meta> and <?xml?> declarations in the first 1024 bytes when the Content-Type header names no charset, and decode GB2312 as GBK like browsers do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FlorianREGAZ
force-pushed
the
fix/response-correctness
branch
from
October 10, 2026 11:19
a6f5287 to
64e2f7a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five response and request fixes, one commit each.
Binary downloads arrive intact (#143)
Requests now send
isByteResponse: true. The library then returns the raw body as adata:<mime>;base64,<payload>URI (an empty body comes back asdata:text/plain; charset=utf-8;base64,), andbuild_responsedecodes it into.content. The library guesses that mime type from the bytes, so it is ignored..textdecodes.contentthe way requests does: with the charset from theContent-Typeheader, else UTF-8, usingerrors="replace". The charset is exposed asresponse.encodingand can be set to decode with another one.Fixes #143
Fewer copies of large bodies (#135, Python side)
.contentis now the only stored copy of the body..textis decoded each time it is read. The reply is decoded and parsed without binding the intermediate copies to names, andbuild_responsepops the data URI and frees it before decoding. For a 36 MB JSON body, tracemalloc peak went from 192 MB to 95 MB and retained memory from 72 MB to 36 MB, on both Python 3.10 and 3.12. Memory held by the Go library is not affected.Fixes #135
Response headers are case-insensitive (#129)
build_responseused to replace theCaseInsensitiveDictwith a plain dict. It now builds theCaseInsensitiveDictdirectly, soresponse.headers["content-type"]works.Fixes #129
Inferred Content-Type no longer sticks to the session (#117)
execute_requestused to write the Content-Type it inferred fromjson=/data=intoself.headers. After onejson=POST, every later request claimedapplication/json. The header now goes only into that request's copy of the headers, and only when neither the session nor the request sets a Content-Type (matched case-insensitively).self.headersis never modified.Fixes #117
timeout=like requests (#108)get/post/... andexecute_requestaccepttimeout=in seconds, as an int or a float.timeout_seconds=still works, and passing both raisesValueError. The value goes to the library astimeoutMilliseconds, so fractions of a second work.timeoutSecondsis no longer sent: it only takes whole seconds, and the library rejects requests that set both.session.timeout_secondscan also be a float. The docstring and README document all of this.Fixes #108
Behaviour changes
response.contentis now the exact bytes the server sent. Before, it was the library's UTF-8 text encoded again.response.textis a read-only property and can no longer be assigned. When the header gives no charset it is decoded as UTF-8, with replacement characters for bytes that don't decode. Before, the library converted bodies to UTF-8 itself and also read<meta charset>from HTML. Pages that declare a non-UTF-8 charset only in<meta>now needresponse.encoding = "...".response.encodingattribute.session.headersno longer gains aContent-Typeafter ajson=/data=request. Code that relied on that leak has to set the header itself.Content-Type: Nonein a request'sheadersnow suppresses the inferred Content-Type.Nonevalues insession.headersare dropped even when the request passes no headers.timeoutMillisecondsinstead oftimeoutSeconds.timeout/timeout_secondson a later request of the same session has no effect. The existing xfail test now covers both names, and the docstring and README mention it. Changing this would mean recreating the native session, which drops its connections and its cookie jar. I left that for a separate change.Testing
tests/integration/test_local_server.py: a stdlibThreadingHTTPServerstarted in a thread. Real requests go through the native library forbytes(range(256)), a PNG, an 8 MiB random body, an empty body, JSON, UTF-8 and ISO-8859-1 bodies, mixed-case response headers, request-header echo for the Content-Type leak, and a slow endpoint for fractional timeouts.tests/integration/test_httpbin.py: the binary-body xfail test used/base64, which httpbin can only return as UTF-8 text. It now downloads/image/pngand compares the bytes with httpbin's file, and it no longer needs the xfail.encodingoverride, a tracemalloc check thatbuild_responsedoesn't hold the data URI, the payload and the bytes all at once, the Content-Type xfail removed plus new cases, and timeout conversion and conflict.uv run ruff check,uv run ruff format --check,uv run mypy,uv lock --checkpass.uv run pytestgives 147 passed, 7 xfailed (coverage 92%).uv run pytest -m networkgives 9 passed, 1 xfailed.scripts/smoke_test.pypasses.🤖 Generated with Claude Code