Digest auth not re-used after connection close.

[catsmanac]

Not sure if it is as intended, a failure or a new feature request, so starting as discussion for now.

We are using HTTPX client for communication with a server requiring digest auth on 1 endpoint. We observe connection failures during use. #1467 requested for re-use Digest and was implemented in 0.23.2.

What we observe

• When a get from the endpoint is requested, the Envoy returns a 401 with the digest information included.
• HTTPX then repeats the get request on the same connection with the digest now included.
• The server replies with TCP FIN,ACK to the message and closes the connection using RST.
• HTTPX signals a RemoteProtocolError.
• HTTPX repeats the get, new connection is build, but Digest is not included and same sequence of messages occurs. The nonce being incremented by the server.

The included capture file capture-pyenphase.pcapng contains 85 packets. The initial request for the endpoint is packet 19, performing a get on the existing connection which doesn't require the digest. That connection is then closed and a retry is in packet 25. Server replies with the 401 and digest in packet 27. Client repeats the get request in packet 29 and includes the digest.

Right after receiving the get the server closes the connection. The client reacts by opening a new connection and issuing the get request again. The Digest Auth is not included this time and the whole sequence repeats.

A similar test with curl, capture-curl.pcapng, shows same server behavior closing connection in packet 17. Curl however retries the get on a new connection in packet 21 including the Digest information received from the server.

capture-files.zip

The client uses httpx.Limits(keepalive_expiry=KEEP_ALIVE_TIMEOUT) with the timeout being 15 sec.

When utilizing limits=httpx.Limits(max_keepalive_connections=0) all is working fine and the issue does not occur.

The code employs httpx.DigestAuth and tenacity retry

httpx.DigestAuth(self.local_username, self.local_password)

@retry(
    retry=retry_if_exception_type(
        (
            httpx.NetworkError,
            httpx.TimeoutException,
            httpx.RemoteProtocolError,
            orjson.JSONDecodeError,
        )
    ),
    wait=wait_random_exponential(multiplier=2, max=5),
    stop=stop_after_delay(MAX_REQUEST_DELAY)
    | stop_after_attempt(MAX_REQUEST_ATTEMPTS),
    reraise=True,
)
async def request(

We can not change the server behavior as it is a commercial available device.

The question is if HTTPX behavior is as expected or if it should behave like CURL by re-using the Digest received when opening the new connection?

[ketpatil77]

Digest challenge state belongs to the DigestAuth object, not to the TCP connection. One thing worth checking in the retry wrapper is whether it constructs a fresh DigestAuth/AsyncClient on each attempt. Retain the same auth object while retrying the request:

import httpx

def make_client(username, password, **kwargs):
    auth = httpx.DigestAuth(username, password)
    return httpx.AsyncClient(auth=auth, **kwargs)

async def read_device(client, url):
    # A retry wrapper can call this again with the SAME client.
    response = await client.get(url)
    response.raise_for_status()
    return response

Use the returned client as an async context manager. On 0.28.1, after receiving a Digest challenge, the auth instance retains it even if the authenticated attempt subsequently raises RemoteProtocolError. A later request through that same instance can send Digest preemptively. A new auth instance has no such state. Reuse an instance only within the intended protection space/device; do not share its cached challenge across unrelated origins.

I tested a controlled sequence: unauthenticated request, SHA-256 Digest challenge, protocol failure during the authenticated attempt, then another request. The last request carried Digest when the auth object was reused. MockTransport tests state retention, not TCP reconnection or your Envoy/device; I have not inspected the capture attachment. If your retry already retains the same object, this check does not explain the failure, and reproducing the actual server's nonce/challenge behavior is still needed. Disabling keep-alive changes the connection pattern but does not establish the root cause. Authentication API.

Validation: HTTPX 0.28.1, Python 3.11.9.