Testing: Certificates incompatible with urllib3 2.4.0 and Python 3.13

[amotl]

Problem

urllib3 introduced a slightly breaking change for Python 3.13, which started tripping the test suite on nightly builds.

crate.client.exceptions.ConnectionError: Server not available, exception: 
HTTPSConnectionPool(host='localhost', port=65534): Max retries exceeded with url: / 
(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] 
certificate verify failed: Missing Authority Key Identifier (_ssl.c:1028)')))

Details

@jvanasco explains the situation...

The default certificate does not have an AKID - which is considered to be RFC non-compliant. It was generated before support was introduced in 2019. This breaks against the default Python 3.13 ssl context with urllib3 2.4.0 now, which aligned their ssl_context with cPython. It was released yesterday.

Solution

.... and also provides a solution:

The current version of minica generates certs that are compliant, so please regenerate [testing certificates] with current minica.

Thanks!

References

• create_urllib3_context() should set ssl.VERIFY_X509_PARTIAL_CHAIN and ssl.VERIFY_X509_STRICT urllib3/urllib3#3571
• https://urllib3.readthedocs.io/en/stable/changelog.html

[amotl]

This comment by @bdraco (thanks!) also seems relevant in this context:

Thats likely due to the openssl version requiring the certificate to meet newest CAB forum requirements and comply with the RFC https://www.rfc-editor.org/rfc/rfc5280#section-4.2.1.9

• ClientConnectorCertificateError / SSLCertVerificationError after update (regression) aio-libs/aiohttp#9869 (comment)

[amotl]

For people using aiohttp, the solution to configure a more relaxed SSL client context as provided by @szelenka-cisco (thanks!) is included in the same discussion:

• ClientConnectorCertificateError / SSLCertVerificationError after update (regression) aio-libs/aiohttp#9869 (comment)

It seems to be revolving around disabling FIPS mode?

[amotl]

Reading about matters a bit closer ¹, I think CA certificates MUST adhere to stronger requirements about their certificates now, like providing Authority Key Identifier and Subject Key Identifier elements.

Clients are starting to validate those requirements now, as the error message Missing Authority Key Identifier conveys.

Footnotes

1. ... but still too quick that my assessments can also be wrong. So, please correct me where I am wrong. ↩

[jvanasco]

Since I got some messages after being cross-referenced to this, hopefully I can help a bit. To clarify:

• Python introduced some changes to the basic validation in which, amongst other things, AKID became required.
• urllib3 aligned their behavior starting in the 2.4 release for py3.13 and up

IMHO the easiest solutions right now are to either:

1- pin urllib3<2.4 which will let things just work as-is.
2- regenerate your cert. the new version of minica is compatible. i generated one here if you want to use it for your tests: https://github.com/aptise/peter_sslers/tree/main/tests/test_configuration/pebble/test-alt/certs

(I discovered the issue so quickly, because CI tests against the default pebble install failed, but ones against my alternate passed)

The other ticket cross-referenced is from a tangential issue - verification failed due to a different alignment within the SSL context. The one you're getting hit with is the new requirement of VERIFY_X509_STRICT (though VERIFY_X509_PARTIAL_CHAIN also changed).

You can also reimplement how the SSL context is created via create_urllib3_context. The relevant code for the py3.13 change is here: https://github.com/urllib3/urllib3/blob/main/src/urllib3/util/ssl_.py#L333-L340

You could just generate that default context, catch py313 to undo the changes, and pass that context explicitly. I found version pinning to be easiest.

IMHO the best solution would be updating the cert and adapting code to enforce the py313 changes on all platforms. That latter bit is a pain. I should file a request against urrlib3 over this.

[amotl]

Dear @jvanasco. Thank you so much for sharing your excellent elaborations and suggestions.

For the sake of completeness: We have been tripped by this beyond the Python driver for the CrateDB database, but also with crate-operator on Python 3.13, that's why I relayed information about how to configure the SSL context for aiohttp: In this case, SSL certificates employed by the AKS K8s HTTP API don't use AKID attributes yet. As we can't do anything about it, we need to apply one of the workarounds as outlined in your suggestions.

We summarized them like this:

This means either do not use Python 3.13 at all, adjust its default SSL context to be more relaxed, for example before making connections using its built-in urllib library, or, in many other cases, just downgrade to urllib3<2.4 until relevant CA certificates have been replaced by modern ones.

We may update this statement to also include advises about aiohttp and httpx, accompanied by further details from your pen about adjusting Python's vanilla SSL context. 🌻

Within this repository, we will use minica to re-generate testing certificates, as you suggested. Until then, we added a build-time constraint d32b74a to fix the build. We don't expect too many woes on runtime aspects, as we think most of relevant public CA certificates have been upgraded already, so we guess this issue is mostly hitting people who are still using old self-signed certificates.

[amotl]

IMHO the best solution would be updating the cert and adapting code to enforce the py313 changes on all platforms. That latter bit is a pain. I should file a request against urllib3 over this.

I think it is a Python-wide consensus to enforce stronger constraints starting with Python 3.13 only. I am inclined to believe there have been plenty of discussions around this very detail, and I don't disagree too much with that decision.

[jvanasco]

In this case, SSL certificates employed by the AKS K8s HTTP API don't use AKID attributes yet.

Oh gosh, that's messy.

Your users are probably getting hit by ssl.create_default_context which is used by the stdlib's urllib and asyncio - and where those urllib3 changes were backported from::

• docs: https://docs.python.org/3/library/ssl.html#ssl.create_default_context
• source: https://github.com/python/cpython/blob/14305a83d381ccdcae814abd9e2c28dda066720b/Lib/ssl.py#L682

AFAIK, in those situations you would need to either pin python under 3.13 OR explicitly create and pass-in a context: ssl.SSLContext var to the various request functions in urllib and asyncio

I think it is a Python-wide consensus to enforce stronger constraints starting with Python 3.13 only. I am inclined to believe there have been plenty of discussions around this very detail, and I don't disagree too much with that decision.

I agree with you here. There is a slight nuance that may not have been conveyed in my above comment - I intended to address the lack of an easy way to explicitly opt-in earlier Python versions to the new stronger constraints, which was not part of the discussion on this urllib3 change. I just offered to write a PR to accomplish this: the stricter behavior would apply to py>=3.13 OR the presence of an environment variable.

[amotl]

Thanks. Yeah, with CrateDB Operator talking to AKS, we will stick with using Python 3.12 for now. Others that use urllib3, we will just downgrade to urllib3<2.4.

On another detail when sampling other popular HTTP libraries for Python, just looking for "Missing Authority Key Identifier" on the httpx repository, we discovered encode/httpx#3434, where cryptography's changelog for version 44.0.0 - 2024-11-27 says:

Relax the Authority Key Identifier requirements on root CA certificates during X.509 verification to allow fields permitted by RFC 5280 but forbidden by the CA/Browser BRs.

So it looks like httpx and possibly other libraries that use the cryptography package yield yet another kind of behaviour according to its default configuration in this regard, which seems to take a more relaxed approach, regardless of Python versions?

[amotl]

Regarding the vanilla Python SSL context, because you referred to ssl.create_default_context(), do you think this configuration, strictly using ssl.CERT_REQUIRED, would be the recommended way to take a more relaxed approach which skips the newer defaults ssl.VERIFY_X509_PARTIAL_CHAIN and ssl.VERIFY_X509_STRICT altogether?

ctx = ssl.create_default_context()
ctx.verify_flags = ssl.CERT_REQUIRED

[jvanasco]

I am not familiar enough with the current flags to comment on that.

I would probably just remove the two new flags, with something like...

ctx = ssl.create_default_context()
if sys.version_info >= (3, 13):
    # Added in version 3.4
    ctx.verify_flags = ctx.verify_flags & ~ssl.VERIFY_X509_STRICT
    # Added in version 3.10.
    ctx.verify_flags = ctx.verify_flags & ~ssl.VERIFY_X509_PARTIAL_CHAIN

Background on the change is here:

• Discussion https://discuss.python.org/t/ssl-changing-the-default-sslcontext-verify-flags/30230
• Issue ssl.create_default_context(): add VERIFY_X509_STRICT and VERIFY_X509_PARTIAL_CHAIN to the default verify_flags python/cpython#107361
• PR gh-107361: strengthen default SSL context flags python/cpython#112389

In terms of the 5280 stuff::

Of relevance, the py>=3.13 security policy enforces the requirement of AKIDs.

The "relax" PR in cryptography just allows for additional fields the CA/B Forum prohibits, but the RFC permits and have been "out in the wild" across trusted root programs for a while.

The RFC 5280 stuff is tied to this PR, and basically the inverse of our shared problem :: https://github.com/pyca/cryptography/pull/11462/files