iter_bytes w/ chunk_size ignores last non-full chunk

[rvernica]

When using iter_bytes with chunk_size any data that is not filling up a full chunk gets lost.

I'm porting a Python library from requests to httpx. I'm replacing a requests.get(stream=True... with httpx.stream("GET"....

In the requests version I'm using:

                        for chunk in resp.iter_content(chunk_size=8 * 1024):
                            buf.write(chunk)

which I replaced with:

                        for chunk in resp.iter_bytes(chunk_size=8 * 1024):
                            buf.write(chunk)

The endpoint returns 10920 bytes, and the requests block reads them all. The httpx block only reads 8192 bytes.

I changed the chunk_size to 1024 and the httpx block only reads 10240 bytes.

If the chunks are not full, the leftover data is lost.

[lovelydinosaur]

Are you able to minimally demo this?...

[rvernica]

I'm trying to get a minimal reproduction of the issues.

Right now I have this extra debugging.

iter_bytes with no chunk_size

Fetching https://localhost:8239/api/v1/sessions/1f8a221c-c04a-4a3d-b34d-924b10ec9730/queries/0.1743725136130127976
Headers: {}
Response: 200:OK
iter_bytes: Current: 30, Accumulated: 30
iter_bytes: Current: 1500, Accumulated: 1530
iter_bytes: Current: 1500, Accumulated: 3030
iter_bytes: Current: 1500, Accumulated: 4530
iter_bytes: Current: 1500, Accumulated: 6030
iter_bytes: Current: 1500, Accumulated: 7530
iter_bytes: Current: 1500, Accumulated: 9030
iter_bytes: Current: 1500, Accumulated: 10530
iter_bytes: Current: 390, Accumulated: 10920
Exception: RemoteProtocolError:peer closed connection without sending complete message body (incomplete chunked read)

Fetching https://localhost:8239/api/v1/sessions/1f8a221c-c04a-4a3d-b34d-924b10ec9730/queries/0.1743725136130127976
Headers: {'Range': 'bytes=10920-'}
Response: 206:Partial Content
iter_bytes: Current: 1500, Accumulated: 1500
iter_bytes: Current: 20, Accumulated: 1520
Exception: RemoteProtocolError:peer closed connection without sending complete message body (incomplete chunked read)

iter_bytes with chunk_size = 8 * 1024

Fetching https://localhost:8239/api/v1/sessions/67497027-3f7d-42ad-8da5-45661408e378/queries/0.1743725343349192715
Headers: {}
Response: 200:OK
iter_bytes: Current: 8192, Accumulated: 8192
Exception: RemoteProtocolError:peer closed connection without sending complete message body (incomplete chunked read)

Fetching https://localhost:8239/api/v1/sessions/67497027-3f7d-42ad-8da5-45661408e378/queries/0.1743725343349192715
Headers: {'Range': 'bytes=8192-'}
Response: 416:Requested Range Not Satisfiable

There are 2,728 bytes that the request with chunk_size misses and the range of the subsequent request is off and not satisfiable.

The data and the behavior of the URLs are identical even if the URLs are different.

[rvernica]

Side question: why would the library raise a RemoteProtocolError peer closed connection without sending a complete message body (incomplete chunked read) when the transfer-encoding: chunked header is present in the response?

[karpetrosyan]

Can you provide the code you're running? I noticed that you're sending a range request, which isn't how httpx streams data.

I tried to reproduce this scenario locally, but I'm getting all the data without any issues. Here's an example:

run with uv run -s path.py

# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "fastapi==0.115.12",
#     "httpx==0.28.1",
#     "uvicorn===0.34.0",
# ]
# ///
from time import sleep

import fastapi
import uvicorn

import threading
from fastapi.responses import StreamingResponse
import httpx


app = fastapi.FastAPI()

async def streaming_response():
    for i in range(10):
        yield f"a" * 1000

@app.get("/")
async def root():
    return StreamingResponse(
        content=streaming_response()
    )


t = threading.Thread(
    target=lambda: uvicorn.run(app, host="0.0.0.0", port=8000),
)

t.start()
sleep(1)

with httpx.stream(
    "GET",
    "http://localhost:8000/",
) as response:
    print(response.headers)
    for chunk in response.iter_bytes(8*1024):
        print(len(chunk))

Result:

Headers({'date': 'Fri, 04 Apr 2025 18:25:46 GMT', 'server': 'uvicorn', 'transfer-encoding': 'chunked'})
8192
1808

[rvernica]

Unfortunately, I can't share the server code.

What is wrong with httpx streaming requests and ranges?

[rvernica]

@tomchristie @karpetrosyan I have a small demo to show that when using a chunk_size, data gets lost. Included below, you will find the server code, the client code, example runs proving the point,t as well as a client implementation using requests, which works fine.

Server

# pip install flask
# python server.py
# curl --trace-ascii /dev/stdin http://127.0.0.1:5000/stream

import time
import threading
import socket
from flask import Flask, Response
from werkzeug.serving import WSGIRequestHandler

app = Flask(__name__)

CHUNK_DATA = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus dignissim sem ut erat sagittis interdum."
N_CHUNKS = 3
CHUNK_SIZE = 48
CHUNK_SLEEP = 2  # seconds
RESPONSE_TIMEOUT = 5  # seconds


class TimeoutWSGIRequestHandler(WSGIRequestHandler):
    """Custom request handler that allows access to the socket"""

    def handle_one_request(self):
        """Override to enforce the connection time length"""
        timer = threading.Timer(
            RESPONSE_TIMEOUT, terminate_connection, args=[self.connection]
        )
        timer.daemon = True
        timer.start()
        return super().handle_one_request()


def terminate_connection(connection):
    """Enfore connection time length by terminating the connection"""
    print(f"Connection timeout {RESPONSE_TIMEOUT} seconds. Closing connection.")
    connection.shutdown(socket.SHUT_RDWR)
    connection.close()


def generator():
    for i in range(N_CHUNKS):
        print(f"Sleep {CHUNK_SLEEP} and send {CHUNK_SIZE} bytes")
        time.sleep(CHUNK_SLEEP)  # Simulate delay for chunk generation
        yield CHUNK_DATA[i * CHUNK_SIZE : (i + 1) * CHUNK_SIZE]


@app.route("/stream")
def stream_random_data():
    return Response(generator(), status=200)


if __name__ == "__main__":
    # Run with our custom request handler
    app.run(
        debug=True, port=5000, threaded=True, request_handler=TimeoutWSGIRequestHandler
    )

Client

# pip install httpx
# python client.py
# python client.py 64

import httpx
import sys


def stream_data(url: str):
    bytes_count = 0
    with httpx.Client() as client:
        try:
            with client.stream("GET", url) as response:
                for chunk in response.iter_bytes(
                    chunk_size=int(sys.argv[1]) if len(sys.argv) > 1 else None
                ):
                    bytes_count += len(chunk)
                    buf = chunk.decode("utf-8")
                    print(f"Buffer size: {len(buf)}")
                    print(f"Buffer content: {buf}")
        except httpx.RemoteProtocolError:
            print("Connection closed by the server.")
    print(f"Total bytes received: {bytes_count}")


def main():
    url = "http://localhost:5000/stream"
    stream_data(url)


if __name__ == "__main__":
    main()

Results

Use cURL against the server:

> curl --trace-ascii /dev/stdin http://127.0.0.1:5000/stream
...
<= Recv data, 54 bytes (0x36)
0000: 30
0004: Lorem ipsum dolor sit amet, consectetur adipisci
<= Recv data, 54 bytes (0x36)
0000: 30
0004: ng elit. Phasellus dignissim sem ut erat sagitti
<= Recv data, 0 bytes (0x0)
== Info: transfer closed with outstanding read data remaining
== Info: closing connection #0
curl: (18) transfer closed with outstanding read data remaining
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus dignissim sem ut erat sagitti%

Notice the last two words erat sagitti

Use httpx without a specified chunk size:

> python client.py
Buffer size: 48
Buffer content: Lorem ipsum dolor sit amet, consectetur adipisci
Buffer size: 48
Buffer content: ng elit. Phasellus dignissim sem ut erat sagitti
Connection closed by the server.
Total bytes received: 96

Notice the total bytes received are 96, and the last two words are erat sagitti

Use httpx with a specified chunk size. This is a broken behavior:

> python client.py 64
Buffer size: 64
Buffer content: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasell
Connection closed by the server.
Total bytes received: 64

Only 64 bytes are received and the last two words don't match with previous runs. The server transmitted the last few words, dignissim sem ut erat sagitti, but they are lost.

Client using requests

# pip install requests
# python client-requests.py
# python client-requests.py 32
# python client-requests.py 64

import requests
import sys


def stream_data(url: str):
    bytes_count = 0
    try:
        with requests.get(url, stream=True) as response:
            chunk_size = int(sys.argv[1]) if len(sys.argv) > 1 else None
            for chunk in response.iter_content(
                chunk_size=chunk_size, decode_unicode=True
            ):
                if chunk:  # filter out keep-alive new chunks
                    bytes_count += len(chunk.encode("utf-8"))  # count actual bytes
                    print(f"Buffer size: {len(chunk)}")
                    print(f"Buffer content: {chunk}")
    except requests.exceptions.RequestException as e:
        print(f"Connection error: {e}")
    print(f"Total bytes received: {bytes_count}")


def main():
    url = "http://localhost:5000/stream"
    stream_data(url)


if __name__ == "__main__":
    main()

Notice that with various chunk sizes, all the data makes it through:

> python client-requests.py
Buffer size: 48
Buffer content: Lorem ipsum dolor sit amet, consectetur adipisci
Buffer size: 48
Buffer content: ng elit. Phasellus dignissim sem ut erat sagitti
Connection error: Response ended prematurely
Total bytes received: 96

> python client-requests.py 32
Buffer size: 32
Buffer content: Lorem ipsum dolor sit amet, cons
Buffer size: 16
Buffer content: ectetur adipisci
Buffer size: 32
Buffer content: ng elit. Phasellus dignissim sem
Buffer size: 16
Buffer content:  ut erat sagitti
Connection error: Response ended prematurely
Total bytes received: 96

> python client-requests.py 64
Buffer size: 48
Buffer content: Lorem ipsum dolor sit amet, consectetur adipisci
Buffer size: 48
Buffer content: ng elit. Phasellus dignissim sem ut erat sagitti
Connection error: Response ended prematurely
Total bytes received: 96

[rvernica]

@tomchristie @karpetrosyan Any thoughts on this? Shall I submit a Bug Report?

[rvernica]

Bug report #3553