Inconsistent results when streaming to unidiff (does not happen with urlopen or requests)

[AdamWill]

In an LLM code review project we've been looking at different ways of parsing diffs. Today I was looking at different ways of reading diffs from a forge (GitHub, GitLab etc.) into unidiff, a diff parser. I encountered some weird behavior which I first reported to unidiff but now think is probably an httpx issue.

Here's the reproducer:

#!/bin/python

import hashlib
import urllib.request

import httpx
import requests
import unidiff

diffurl = "https://forge.fedoraproject.org/releng/tickets/pulls/12599.diff"

def check(patches):
    md5 = hashlib.md5(str(patches).encode("utf-8")).hexdigest()
    if md5 != "c28ad8dd59833ff6d8f2f7d680cf49ca":
        with open(f"/var/home/adamw/diff-{md5}.diff", "w", encoding="utf-8") as fh:
            fh.write(str(patches))
            print("DUMPED " + md5)
    else:
        print("OK")

def naive():
    diff = httpx.get(diffurl).text
    patches = unidiff.PatchSet(diff)
    check(patches)

def sophisticated_httpx():
    with httpx.stream("GET", diffurl) as r:
        patches = unidiff.PatchSet(r.iter_bytes(), encoding="utf-8")
    check(patches)

def sophisticated_requests():
    with requests.get(diffurl, stream=True) as r:
        patches = unidiff.PatchSet(r.raw, encoding="utf-8")
    check(patches)

def sophisticated_urlopen():
    req = urllib.request.Request(diffurl)
    req.add_header("Accept", "text/plain")
    with urllib.request.urlopen(req) as r:
        patches = unidiff.PatchSet(r, encoding="utf-8")
    check(patches)

for i in range(50):
#    sophisticated_requests()
    sophisticated_httpx()

as you can see, that implements various ways of getting the same diff file, feeding it to unidiff, and checking that it generates the expected result.

If you run the 50-iteration loop for naive, sophisticated_requests, or sophisticated_urlopen, they pass: they print OK 50 times. If you run it for sophisticated_httpx, it will produce some OKs but also fail in various interesting ways, documented in detail in the issue I reported to unidiff - it either misses some content, or crashes.

All three 'sophisticated' methods use the same amount of memory, indicating they all are actually streaming the content to unidiff. But urlopen and requests work reliably, while httpx doesn't. I'm not sure why this is.

[AdamWill]

This is with python3-httpx-0.28.1-9.fc43.noarch on Fedora.

[Doribelove]

PatchSet expects its iterable to yield one complete diff line per item, but r.iter_bytes() and r.iter_text() yield chunks with arbitrary boundaries. A chunk can contain several lines, or end halfway through one. requests.Response.raw and the urlopen() response are file-like iterators, so iterating them yields lines instead. That difference explains why the three streaming examples are not equivalent.

I reproduced the corruption without a network connection by passing the same synthetic diff through different chunk boundaries. There is no need for a nested-diff parsing bug: the parser is being given chunks as if they were lines.

For your checksum comparison, preserve line endings too. iter_lines() strips them, so feeding that directly to PatchSet changes str(patches). This adapter keeps LF/CRLF endings and an unterminated final line while still consuming the response incrementally:

import httpx
import unidiff


def diff_lines(response):
    pending = ""
    for chunk in response.iter_text():
        parts = (pending + chunk).split("\n")
        pending = parts.pop()
        for line in parts:
            yield line + "\n"
    if pending:
        yield pending


def sophisticated_httpx():
    with httpx.stream("GET", diffurl) as r:
        r.raise_for_status()
        r.encoding = "utf-8"
        patches = unidiff.PatchSet(diff_lines(r))
    check(patches)

Here diffurl and check are the ones in your example. No encoding= argument is needed on PatchSet, since iter_text() has already decoded the bytes, including characters split across byte chunks.

Tested locally with HTTPX 0.28.1, unidiff 0.7.5, and Python 3.10.12: 514 controlled chunk layouts across LF, CRLF, and missing-final-newline inputs all matched str(PatchSet(full_text)) exactly. Direct chunk iteration produced truncated parses or exceptions. This test used a synthetic diff containing nested diff text and UTF-8 characters, rather than downloading the Fedora diff.

The relevant contracts are in HTTPX's streaming documentation and unidiff's line-by-line parser.

Prepared with AI assistance; the reproduction and adapter were executed locally.