RFC: AWS-LC as TLS Backend for HAProxy

[hoffmaen]

Link to the document for quick review: rfc-draft-haproxy-awslc.md

Summary

This PR adds the RFC for AWS-LC as TLS Backend for the haproxy-boshrelease.

AWS-LC is independent of the version provided via the BOSH stemcell and can be kept up to date interdependently. The OpenSSL that is currently used via the stemcell is on an outdated version that has severe performance impacts in high-load scenarios. These would be improved in newer OpenSSL versions, but not resolved. HAProxy explicitely recommends against OpenSSL in production (haproxy/haproxy#3086).

AWS-LC alleviates these issues and is proposed to be added as an optional release. For operators nothing changes, unless they want to opt into using AWS-LC.

AWS-LC is FIPS 140-3 compliant.

[beyhan]

Hi @hoffmaen,

thank you for this pr. Is it intentionally in draft status?

[Soha-Albaghdady]

LGTM, proposing some minor formatting and restructuring

[beyhan]

@hoffmaen could you please ask in your WG (meeting) about feedback.

[b1tamara]

LGTM

[rkoster]

Code Review

PR #1501: RFC: AWS-LC as TLS Backend for HAProxy

Summary

This RFC proposes adding AWS-LC as an optional TLS backend for the haproxy-boshrelease to address OpenSSL 3.0.2 TLS-handshake performance and the lack of post-quantum key exchange. The problem statement is strong: it is well-evidenced (benchmarks, HAProxy upstream recommendation in haproxy#3086, FIPS 140-3 validation) and the non-breaking, opt-in framing with OpenSSL-as-default is the right compatibility posture. The core AWS-LC value proposition is convincing. The main gap is the packaging design: the RFC introduces a 7-tarball matrix (openssl, openssl-patched, awslc, awslc-patched, awslc-fips, awslc-fips-patched, multi) but only justifies 4 of the 7. The three '-patched' variants are never explained, and there is no analysis of the CI/compilation cost this matrix imposes despite it roughly doubling the build surface. There are also smaller correctness/process issues: the implementation link 404s (the branch lives on a fork as PR #925), the RFC metadata omits fields required by the template, and the document still carries Draft status with an empty RFC Pull Request field.

Critical Issues - Must Fix

1. The 7-variant release matrix is under-justified: the three '-patched' variants have no stated rationale

   • File: toc/rfc/rfc-draft-haproxy-awslc.md
   • The Release Variants table defines seven tarballs: openssl, openssl-patched, awslc, awslc-patched, awslc-fips, awslc-fips-patched, and multi. The RFC justifies openssl (backward compat), awslc (performance), awslc-fips (FIPS compliance), and multi (migration safety net). It does NOT justify the '-patched' axis at all. 'custom HAProxy patches' appears in the table and Build Architecture sections but the RFC never states what these patches are, why they are needed, which users need them, or why they must ship as separate first-class release variants rather than being folded into the base builds or omitted. This single unexplained axis doubles the variant count from 3 to 6 and doubles the number of HAProxy binaries in the multi release from 3 to 6. Reviewers cannot evaluate the proposal's scope without knowing what problem the patched variants solve.
   • Fix: Add a subsection explaining the custom HAProxy patches: what they change, why they cannot be upstreamed or made default, and who requires them. Then justify (or drop) each patched variant explicitly. If the patches are not required for the initial rollout, consider deferring all three '-patched' variants to a follow-up and shipping only openssl / awslc / awslc-fips / multi.

2. No analysis of CI and compilation cost for the expanded build matrix

   • File: toc/rfc/rfc-draft-haproxy-awslc.md
   • The proposal significantly expands what haproxy-boshrelease must compile and test: the multi release alone builds 6 HAProxy binaries plus aws-lc, aws-lc-fips, cmake, and a Go toolchain (confirmed against the implementation's packages-multi directory). Six additional single-variant tarballs are also produced. The RFC gives only two isolated build-time figures (~15 min OpenSSL, ~25 min AWS-LC) and no accounting of the total CI impact: how many release tarballs are built and tested per pipeline run, how compilation time and cost scale across the full matrix, what the storage/artifact footprint is, or how the test matrix (BOSH deployment tests per variant) grows. For a shared community release this recurring cost is a first-order concern, not an implementation detail.
   • Fix: Add a 'Costs' or 'CI / Build Impact' section quantifying the added compilation time, pipeline duration, artifact count/size, and test-matrix growth for the full set of variants. Use this to justify the matrix size, and explicitly weigh it against dropping the patched variants and/or the multi release.

Important Issues - Should Fix

1. Implementation branch link is broken (404)

   • File: toc/rfc/rfc-draft-haproxy-awslc.md
   • The References section links to https://github.com/cloudfoundry/haproxy-boshrelease/tree/aws-lc-multi as the implementation branch. That branch does not exist in cloudfoundry/haproxy-boshrelease and the URL returns 404. The implementation actually lives on the fork sap-contributions/haproxy-boshrelease (branch aws-lc-multi), proposed upstream as PR [ARI] Remove @kulhadia as Autoscaler reviever #925 (Add Support for AWS-LC haproxy-boshrelease#925). Reviewers following the link cannot see the code.
   • Suggestion: Update the link to the fork branch (https://github.com/sap-contributions/haproxy-boshrelease/tree/aws-lc-multi) and/or reference the upstream PR Add Support for AWS-LC haproxy-boshrelease#925.

2. RFC metadata omits required template fields

   • File: toc/rfc/rfc-draft-haproxy-awslc.md
   • The Meta block is missing the 'Related RFCs' field present in rfc-template.md, and the 'RFC Pull Request' field is left empty. rfc-0023-add-cf-supports-for-fips-stemcell may be relevant to the FIPS discussion and worth cross-linking under Related RFCs.
   • Suggestion: Add the 'Related RFCs' field (linking rfc-0023 if applicable, or leaving empty per template) and fill in 'RFC Pull Request' with this PR's URL, as the RFC README process step requires.

3. Runtime binary selection via filesystem-path probing is described but its failure modes are not

   • File: toc/rfc/rfc-draft-haproxy-awslc.md
   • Runtime Selection resolves the binary by checking whether /var/vcap/packages/haproxy-<ssl_variant> exists, else falling back to /var/vcap/packages/haproxy. The RFC does not specify behaviour when ssl_variant is set to a valid value on a single-variant (non-multi) release where that package is absent: does it silently fall back to the wrong backend, or fail fast? Silent fallback could mask a misconfiguration where an operator believes they are running AWS-LC FIPS but are actually on OpenSSL.
   • Suggestion: Specify the validation/error behaviour when ssl_variant names a variant not present in the deployed release. Prefer failing fast with a clear error over silent fallback, and state this in the RFC.

4. No testing strategy for the variants, and the reference implementation adds zero test coverage

   • File: toc/rfc/rfc-draft-haproxy-awslc.md
   • The RFC defines 7 variants with materially different TLS behaviour (no DHE, no X448, no FFDH in AWS-LC; FIPS-mode claims; PQ named groups) and a runtime ssl_variant selector, but says nothing about how any of this is verified. haproxy-boshrelease already has good coverage - rspec template specs and a Go/Ginkgo acceptance suite (https_frontend_test.go, mtls_frontend_test.go, strict_sni_test.go, tcp_frontend_test.go). The linked implementation branch (aws-lc-multi, 36 files changed) modifies NONE of these: zero *_spec.rb and zero *_test.go changes; the only acceptance-tests edits are README.md and run-local.sh. So nothing verifies that awslc negotiates the expected ciphers, that the documented DHE->ECDHE fallback actually works, that SNI/mTLS still pass on the new binaries, that the ssl_variant switch selects the right binary, or that FIPS is genuinely active. These are the exact claims the RFC relies on.
   • Suggestion: Add a Testing section stating which variants receive automated acceptance coverage and what it asserts (cipher/curve negotiation, DHE->ECDHE fallback, SNI + mTLS regression, ssl_variant selection resolves the correct binary, FIPS-active assertion for the FIPS variants), and how that test matrix scales in the pipeline. The implementation PR should extend the existing rspec/acceptance suites accordingly.

5. cf-deployment integration and testing expectation is undefined

   • File: toc/rfc/rfc-draft-haproxy-awslc.md
   • The only cf-deployment reference is the line-18 disclaimer that this is not about the TCP Router. haproxy-boshrelease is not in the cf-deployment base manifest; it is opt-in via operations/use-haproxy.yml, which pins a single haproxy release version and adds one haproxy instance group. The RFC does not state whether the AWS-LC variants are expected to be exercised at the cf-deployment level - e.g. via a companion ops-file to use-haproxy.yml or an ssl_variant wiring - nor who owns that integration. Without this, it is unclear how an operator using the standard cf-deployment ops-file path would adopt or validate AWS-LC end to end.
   • Suggestion: State the cf-deployment expectation explicitly: either add/describe a companion ops-file (e.g. use-haproxy-awslc.yml) naming which variant it wires and who maintains it, or declare cf-deployment integration testing out of scope with a rationale and point operators at the haproxy-boshrelease-level path instead.

Minor Issues

1. Filename still uses the rfc-draft- prefix in toc/rfc/rfc-draft-haproxy-awslc.md
2. Status is Draft while the PR is out of draft and under active review in toc/rfc/rfc-draft-haproxy-awslc.md

Positive Notes

• Strong, well-evidenced problem statement: benchmark tables for both non-FIPS and FIPS, the HAProxy upstream recommendation (haproxy#3086), and FIPS 140-3 validation together make a compelling case.
• Correct compatibility posture: OpenSSL remains the default and AWS-LC is strictly opt-in, so existing deployments see no change.
• The cipher/curve differences section (no DHE, no X448, no FFDH) is thorough and honestly addresses the main client-compatibility risk, correctly noting TLS 1.2-minimum policy filters out affected legacy clients.
• Including a concrete, phased, reversible migration path with realistic rollback timing (~2 min property switch) shows good operational thinking.
• Alternatives Considered addresses both other TLS libraries (WolfSSL, BoringSSL) and other proxies (Envoy, nginx) with clear reasons for rejection.

Recommendation

Request changes - critical issues need to be addressed before merging.

[rkoster]

Requesting changes - see the detailed review comment above. Main blockers: the -patched variant axis is unjustified and there is no CI/build-cost analysis for the expanded matrix. Important: broken implementation link, missing testing strategy (impl branch changes no tests), and undefined cf-deployment integration.

[rkoster]

The '-patched' axis (openssl-patched, awslc-patched, awslc-fips-patched) is introduced here but never explained anywhere in the RFC. What are the 'custom HAProxy patches', why are they needed, and who needs them? This axis doubles the variant count from 3 to 6 (and doubles the multi release's HAProxy binaries), which is the main driver of the added build/CI cost. Please justify each patched variant explicitly, or defer them to a follow-up if they are not required for the initial rollout.

[hoffmaen]

Thanks for the detailed review.
The RFC now has a "Custom HAProxy Patches (the -patched variants)" subsection under Release Variants. It documents what the patch does (relaxes RFC 6455 Sec-WebSocket-Key enforcement in the H1 mux), why it's required in some scenarios (legacy clients opening long-lived WebSocket connections without a spec-compliant key), why it can't be the default (it makes HAProxy intentionally non-compliant with RFC 6455), and why it ships as a separate variant rather than a runtime toggle (BOSH properties are deploy-time, not compile-time, and BOSH won't recompile on a property change, so a compile-time patch toggle isn't expressible as a property). The patch is a long-standing carried requirement, not new to any HAProxy version.

[rkoster]

This build architecture makes the CI cost concrete: the multi release compiles 6 HAProxy binaries plus aws-lc, aws-lc-fips, cmake, and a Go toolchain, and six additional single-variant tarballs are produced on top. The RFC gives only ~15 min / ~25 min per-variant figures but never totals the pipeline time, artifact count/size, or per-variant BOSH test-matrix growth. For a shared community release this recurring cost is a first-order design concern. Please add a Costs / CI-impact section quantifying it and use it to justify the matrix size.

[hoffmaen]

Added a "Costs and CI Impact" section. The key correction: a BOSH release tarball only ships source + blobs. The actual HAProxy/AWS-LC compilation happens on the target VM at deploy time, and for the single-variant releases each operator compiles only the one variant they deploy. So extra variants impose no compile cost on operators who don't use them (the multi release is the only thing that compiles everything, and it's explicitly a migration/testing artifact). The incremental cost is confined to the Concourse release-build pipeline: all the extra variants together add a few minutes of pipeline wall-clock plus the disk to store the extra tarballs. Given we cut 9 releases in H1 2026, that recurring cost is negligible. The acceptance-test matrix is per-backend (awslc, awslc-fips), so it doesn't grow with the -patched axis.

[rkoster]

What happens on a single-variant release if an operator sets ha_proxy.ssl_variant to a valid-but-absent variant? As written, the fallback to /var/vcap/packages/haproxy could silently run a different TLS backend than the operator intended (e.g. they believe they are on awslc-fips but are actually on openssl). Recommend failing fast with a clear error instead of silent fallback, and stating that behaviour here.

[hoffmaen]

Good catch - the silent fallback is now documented as a configuration error that must fail fast.

[rkoster]

These behavioural claims (DHE absent, clients transparently fall back to ECDHE, existing mixed cipher configs keep working) are exactly the kind of thing that needs an automated test - but the implementation branch (aws-lc-multi) changes no acceptance or rspec tests at all (the existing suite has https_frontend_test.go, mtls_frontend_test.go, strict_sni_test.go). Please add a Testing section covering which variants get automated coverage and what it asserts: cipher/curve negotiation, DHE->ECDHE fallback, SNI + mTLS regression, ssl_variant selecting the correct binary, and a FIPS-active assertion for the FIPS variants. Separately, please clarify the cf-deployment expectation - haproxy is opt-in there via operations/use-haproxy.yml (one pinned release), so state whether a companion ops-file / ssl_variant wiring is planned or whether cf-deployment integration is out of scope.

[hoffmaen]

Regarding cf-deployment integration: Added a short cf-deployment Integration subsection stating this is out of scope. haproxy-boshrelease is opt-in in cf-deployment via use-haproxy.yml (auto-bumped release version); operators adopting AWS-LC manage their own release pinning outside that flow, so no cf-deployment change is required here. The RFC notes that first-class support could later be added via a variant selector in use-haproxy.yml that still tracks automatic version bumps, but that is a cf-deployment-side change and out of scope for this RFC.

[hoffmaen]

Added a Testing section. The core approach is to run the existing acceptance suite (HTTPS over TLS 1.2/1.3, ALPN, mTLS, strict SNI, WebSocket, + ~20 feature tests) against the AWS-LC-linked binary - since those already drive the full TLS data path, they're the real regression guarantee that swapping the TLS library preserves behaviour, and they directly cover the SNI + mTLS points. The one new test is an rspec spec for the ssl_variant binary selector.

We deliberately diverged on a few of the specific asks:

• DHE→ECDHE fallback: removed, because the original "fallback" claim was wrong — there is no fallback. Under the shipped default cipher order (ECDHE ahead of DHE, via SSL_OP_CIPHER_SERVER_PREFERENCE), an ECDHE-capable client already negotiates ECDHE, so a client still on DHE has no ECDHE alternative and simply fails on AWS-LC. The RFC now states this accurately, so there's no fallback behaviour to test.
• Per-cipher/curve negotiation tests: not added — the suite's style is functional (drive a real client, assert success), not primitive-level assertions; the differences are documented and bounded by the recommended config instead.
• FIPS-active acceptance run: not added — each acceptance run recompiles the release (~20–30 min for an AWS-LC variant), so a second full run for awslc-fips roughly doubles cost for little signal given FIPS is a library property; FIPS activation is verified out-of-band during release validation.

Note the implementation branch is still draft and will follow the RFC, so the test changes land on the implementation PR rather than being present now.