From 4321230373fa6d0c3d5a3b782eb7543c5ea1bcdf Mon Sep 17 00:00:00 2001 From: Michael Harp Date: Fri, 22 May 2026 11:15:41 -0400 Subject: [PATCH 1/4] openvox 8x: document custom CA for HTTPS-inspecting proxies Add a new page covering how to add a proxy CA to OpenVox's vendored cert bundle so that gem installs and puppet module installs work behind Squid SSL Bump and similar intercepting proxies. Co-Authored-By: Claude Sonnet 4.6 Signed-off-by: Michael Harp --- _data/nav/openvox_8x.yml | 2 + docs/_openvox_8x/ssl_custom_ca_proxy.md | 144 ++++++++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 docs/_openvox_8x/ssl_custom_ca_proxy.md diff --git a/_data/nav/openvox_8x.yml b/_data/nav/openvox_8x.yml index d21a63c9a..0b755337b 100644 --- a/_data/nav/openvox_8x.yml +++ b/_data/nav/openvox_8x.yml @@ -457,6 +457,8 @@ link: "ssl_attributes_extensions.html" - text: Regenerating all certificates in a deployment link: "ssl_regenerate_certificates.html" + - text: Adding a custom CA for HTTPS-inspecting proxies + link: "ssl_custom_ca_proxy.html" - text: Details about OpenVox's internals items: - text: Agent/server HTTPS communications diff --git a/docs/_openvox_8x/ssl_custom_ca_proxy.md b/docs/_openvox_8x/ssl_custom_ca_proxy.md new file mode 100644 index 000000000..a95e67506 --- /dev/null +++ b/docs/_openvox_8x/ssl_custom_ca_proxy.md @@ -0,0 +1,144 @@ +--- +layout: default +title: "SSL configuration: Adding a custom CA for HTTPS-inspecting proxies" +--- + +Some network environments use HTTPS-inspecting proxies (such as Squid with SSL Bump) that +re-sign outbound TLS connections using a local CA. OpenVox's Ruby runtime uses its own +bundled CA certificate store rather than the system trust store, so the proxy CA must be +added explicitly before OpenVox will trust connections the proxy intercepts. + +Common symptoms: `gem install` or module downloads (`puppet module install`) fail with +certificate verification errors even though the system CA trust store already includes the +proxy CA. + +## How OpenVox validates outbound TLS + +OpenVox ships its own Ruby runtime with a vendored CA bundle compiled in at +`/opt/puppetlabs/puppet/ssl/cert.pem`. This path is OpenVox's `OpenSSL::X509::DEFAULT_CERT_FILE` +and is what Ruby uses — not the OS trust store — when validating TLS certificates for +outbound connections such as gem downloads and `puppet module install`. + +## Quick fix: append the CA to the bundle + +Appending your proxy CA to `cert.pem` works immediately and requires no additional +configuration: + +```console +cat /path/to/proxy-ca.pem >> /opt/puppetlabs/puppet/ssl/cert.pem +``` + +The downside is that `cert.pem` is owned by the `openvox-agent` package and may be +overwritten during upgrades, removing your CA and breaking the proxy again. + +## Persistent fix: `ssl_trust_store` (module downloads only) + +For `puppet module install` and https file sources, the cleanest option is the +`ssl_trust_store` setting in `puppet.conf`. OpenVox loads this file as an additional trust +store on top of the built-in bundle, so your proxy CA survives upgrades and no environment +variables are required: + +```console +puppet config set ssl_trust_store /etc/ssl/certs/proxy-ca.pem +``` + +**Important:** `ssl_trust_store` only applies to Puppet's own outbound HTTPS requests. It +does not affect gem installs performed via the `puppet_gem` package provider, because those +run `gem` as a subprocess that does not read `puppet.conf`. Use the `SSL_CERT_FILE` approach +below if you also need gem installs to work. + +## Persistent fix: `SSL_CERT_FILE` pointing at a merged bundle + +The `SSL_CERT_FILE` environment variable overrides OpenSSL's default cert path and works +for both `puppet module install` and gem installs. Create a merged bundle that combines the +original Mozilla certs with your proxy CA: + +```console +cat /opt/puppetlabs/puppet/ssl/cert.pem /path/to/proxy-ca.pem \ + > /etc/ssl/certs/puppet-custom-bundle.pem +``` + +**Make it permanent for the Puppet agent service** by adding the variable to the service +environment. On systemd systems, create a drop-in: + +```console +mkdir -p /etc/systemd/system/puppet.service.d +cat > /etc/systemd/system/puppet.service.d/ssl_cert_file.conf <<'EOF' +[Service] +Environment=SSL_CERT_FILE=/etc/ssl/certs/puppet-custom-bundle.pem +EOF +systemctl daemon-reload +systemctl restart puppet +``` + +For one-off commands, export the variable in the same shell: + +```console +SSL_CERT_FILE=/etc/ssl/certs/puppet-custom-bundle.pem puppet agent -t +``` + +## Managing with Puppet + +Use [puppetlabs/concat](https://forge.puppet.com/modules/puppetlabs/concat) to assemble +and maintain the merged bundle. The `file:///` source scheme reads `cert.pem` directly from +the local filesystem at catalog apply time, so the bundle automatically picks up the fresh +Mozilla certs after an `openvox-agent` upgrade: + +```puppet +concat { '/etc/ssl/certs/puppet-custom-bundle.pem': + ensure => present, + owner => 'root', + group => 'root', + mode => '0644', +} + +concat::fragment { 'openvox-mozilla-bundle': + target => '/etc/ssl/certs/puppet-custom-bundle.pem', + source => 'file:///opt/puppetlabs/puppet/ssl/cert.pem', + order => '01', +} + +concat::fragment { 'proxy-ca': + target => '/etc/ssl/certs/puppet-custom-bundle.pem', + content => lookup('profile::proxy_ca_cert'), + order => '02', +} + +file { '/etc/systemd/system/puppet.service.d/ssl_cert_file.conf': + ensure => file, + owner => 'root', + group => 'root', + mode => '0644', + content => "[Service]\nEnvironment=SSL_CERT_FILE=/etc/ssl/certs/puppet-custom-bundle.pem\n", + notify => Exec['systemd-daemon-reload'], +} + +exec { 'systemd-daemon-reload': + command => '/bin/systemctl daemon-reload', + refreshonly => true, +} +``` + +Store the proxy CA certificate as a multiline string in Hiera: + +```yaml +profile::proxy_ca_cert: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- +``` + +Because `concat::fragment` reads `cert.pem` on every Puppet run, the merged bundle stays +current after upgrades with no additional steps. + +## Verifying the configuration + +Confirm Ruby can reach an intercepted host: + +```console +/opt/puppetlabs/puppet/bin/ruby -rnet/http -ruri \ + -e 'Net::HTTP.get(URI("https://forgeapi.puppet.com")); puts "OK"' +``` + +A successful response prints `OK`. A certificate verification error means the proxy CA is +still not trusted by OpenVox's Ruby environment. From 002087f1b797b09ce44a98c3eac9f8f7af3a7202 Mon Sep 17 00:00:00 2001 From: Michael Harp Date: Fri, 22 May 2026 11:55:40 -0400 Subject: [PATCH 2/4] openvox 8x: incorporate review feedback on custom CA proxy doc - Promote certs/+rehash as the preferred quick fix (survives upgrades, no environment variable management needed) per Sharpie's review - Note Windows caveat for openssl rehash - Update Puppet example to lead with file+exec rehash; keep concat approach as the SSL_CERT_FILE alternative - Retain direct cert.pem append as a one-liner fallback Co-Authored-By: Claude Sonnet 4.6 Signed-off-by: Michael Harp --- docs/_openvox_8x/ssl_custom_ca_proxy.md | 73 ++++++++++++++++++------- 1 file changed, 52 insertions(+), 21 deletions(-) diff --git a/docs/_openvox_8x/ssl_custom_ca_proxy.md b/docs/_openvox_8x/ssl_custom_ca_proxy.md index a95e67506..0124b6e9c 100644 --- a/docs/_openvox_8x/ssl_custom_ca_proxy.md +++ b/docs/_openvox_8x/ssl_custom_ca_proxy.md @@ -19,17 +19,29 @@ OpenVox ships its own Ruby runtime with a vendored CA bundle compiled in at and is what Ruby uses — not the OS trust store — when validating TLS certificates for outbound connections such as gem downloads and `puppet module install`. -## Quick fix: append the CA to the bundle +## Quick fix: add the CA to the `certs/` directory -Appending your proxy CA to `cert.pem` works immediately and requires no additional -configuration: +Copy your proxy CA into OpenVox's `certs/` directory and run `openssl rehash` to generate +the fingerprint symlinks that OpenSSL uses to look up certificates: ```console -cat /path/to/proxy-ca.pem >> /opt/puppetlabs/puppet/ssl/cert.pem +cp /path/to/proxy-ca.pem /opt/puppetlabs/puppet/ssl/certs/proxy-ca.pem +/opt/puppetlabs/puppet/bin/openssl rehash /opt/puppetlabs/puppet/ssl/certs/ ``` -The downside is that `cert.pem` is owned by the `openvox-agent` package and may be -overwritten during upgrades, removing your CA and breaking the proxy again. +This directory is OpenVox Ruby's `DEFAULT_CERT_DIR` and is included by `set_default_paths` +on every connection. The directory is empty by default — the `openvox-agent` package does +not place any files there — so user-added files survive package upgrades. + +> **Note:** `openssl rehash` is not supported on Windows as of OpenVox 8. Use the +> `SSL_CERT_FILE` approach below on Windows nodes. + +If you prefer a one-liner that skips rehash, appending directly to `cert.pem` also works, +but that file is replaced on upgrade: + +```console +cat /path/to/proxy-ca.pem >> /opt/puppetlabs/puppet/ssl/cert.pem +``` ## Persistent fix: `ssl_trust_store` (module downloads only) @@ -79,10 +91,41 @@ SSL_CERT_FILE=/etc/ssl/certs/puppet-custom-bundle.pem puppet agent -t ## Managing with Puppet +### `certs/` + rehash (simplest) + +Deploy the CA with a `file` resource and trigger `openssl rehash` on change: + +```puppet +file { '/opt/puppetlabs/puppet/ssl/certs/proxy-ca.pem': + ensure => file, + owner => 'root', + group => 'root', + mode => '0644', + content => lookup('profile::proxy_ca_cert'), + notify => Exec['rehash-puppet-ssl-certs'], +} + +exec { 'rehash-puppet-ssl-certs': + command => '/opt/puppetlabs/puppet/bin/openssl rehash /opt/puppetlabs/puppet/ssl/certs/', + refreshonly => true, +} +``` + +Store the proxy CA certificate as a multiline string in Hiera: + +```yaml +profile::proxy_ca_cert: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- +``` + +### `SSL_CERT_FILE` merged bundle (covers gem installs on Windows or when rehash is unavailable) + Use [puppetlabs/concat](https://forge.puppet.com/modules/puppetlabs/concat) to assemble -and maintain the merged bundle. The `file:///` source scheme reads `cert.pem` directly from -the local filesystem at catalog apply time, so the bundle automatically picks up the fresh -Mozilla certs after an `openvox-agent` upgrade: +the merged bundle. The `file:///` source scheme reads `cert.pem` from the local filesystem +at catalog apply time, so the bundle automatically picks up fresh Mozilla certs after an +`openvox-agent` upgrade: ```puppet concat { '/etc/ssl/certs/puppet-custom-bundle.pem': @@ -119,18 +162,6 @@ exec { 'systemd-daemon-reload': } ``` -Store the proxy CA certificate as a multiline string in Hiera: - -```yaml -profile::proxy_ca_cert: | - -----BEGIN CERTIFICATE----- - ... - -----END CERTIFICATE----- -``` - -Because `concat::fragment` reads `cert.pem` on every Puppet run, the merged bundle stays -current after upgrades with no additional steps. - ## Verifying the configuration Confirm Ruby can reach an intercepted host: From 2b789aac7b2e3afb69d617ba22bddf63cc59461f Mon Sep 17 00:00:00 2001 From: Michael Harp Date: Fri, 22 May 2026 11:58:36 -0400 Subject: [PATCH 3/4] openvox 8x: remove concat approach from managing-with-puppet section The certs/+rehash file+exec example covers the recommended approach without needing an extra module dependency. SSL_CERT_FILE manual steps remain documented for Windows edge cases. Co-Authored-By: Claude Sonnet 4.6 Signed-off-by: Michael Harp --- docs/_openvox_8x/ssl_custom_ca_proxy.md | 43 ------------------------- 1 file changed, 43 deletions(-) diff --git a/docs/_openvox_8x/ssl_custom_ca_proxy.md b/docs/_openvox_8x/ssl_custom_ca_proxy.md index 0124b6e9c..bf7e5e77d 100644 --- a/docs/_openvox_8x/ssl_custom_ca_proxy.md +++ b/docs/_openvox_8x/ssl_custom_ca_proxy.md @@ -91,8 +91,6 @@ SSL_CERT_FILE=/etc/ssl/certs/puppet-custom-bundle.pem puppet agent -t ## Managing with Puppet -### `certs/` + rehash (simplest) - Deploy the CA with a `file` resource and trigger `openssl rehash` on change: ```puppet @@ -120,47 +118,6 @@ profile::proxy_ca_cert: | -----END CERTIFICATE----- ``` -### `SSL_CERT_FILE` merged bundle (covers gem installs on Windows or when rehash is unavailable) - -Use [puppetlabs/concat](https://forge.puppet.com/modules/puppetlabs/concat) to assemble -the merged bundle. The `file:///` source scheme reads `cert.pem` from the local filesystem -at catalog apply time, so the bundle automatically picks up fresh Mozilla certs after an -`openvox-agent` upgrade: - -```puppet -concat { '/etc/ssl/certs/puppet-custom-bundle.pem': - ensure => present, - owner => 'root', - group => 'root', - mode => '0644', -} - -concat::fragment { 'openvox-mozilla-bundle': - target => '/etc/ssl/certs/puppet-custom-bundle.pem', - source => 'file:///opt/puppetlabs/puppet/ssl/cert.pem', - order => '01', -} - -concat::fragment { 'proxy-ca': - target => '/etc/ssl/certs/puppet-custom-bundle.pem', - content => lookup('profile::proxy_ca_cert'), - order => '02', -} - -file { '/etc/systemd/system/puppet.service.d/ssl_cert_file.conf': - ensure => file, - owner => 'root', - group => 'root', - mode => '0644', - content => "[Service]\nEnvironment=SSL_CERT_FILE=/etc/ssl/certs/puppet-custom-bundle.pem\n", - notify => Exec['systemd-daemon-reload'], -} - -exec { 'systemd-daemon-reload': - command => '/bin/systemctl daemon-reload', - refreshonly => true, -} -``` ## Verifying the configuration From 964db7ba1ccc4c99a2b448c69e8b81ca5d675fd7 Mon Sep 17 00:00:00 2001 From: Michael Harp Date: Fri, 22 May 2026 12:35:00 -0400 Subject: [PATCH 4/4] openvox 8x: add symlink alternative for certs/ approach MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add ln -s as an integration pattern in the Puppet section for nodes where the CA is already deployed by puppet/trusted_ca into the OS trust store. Remove symlink from the manual quick fix section — cp is simpler when deploying the cert fresh. Co-Authored-By: Claude Sonnet 4.6 Signed-off-by: Michael Harp --- docs/_openvox_8x/ssl_custom_ca_proxy.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/_openvox_8x/ssl_custom_ca_proxy.md b/docs/_openvox_8x/ssl_custom_ca_proxy.md index bf7e5e77d..5dee0aaf5 100644 --- a/docs/_openvox_8x/ssl_custom_ca_proxy.md +++ b/docs/_openvox_8x/ssl_custom_ca_proxy.md @@ -118,6 +118,23 @@ profile::proxy_ca_cert: | -----END CERTIFICATE----- ``` +If the certificate is already deployed on the node at a known path (for example, by the +[puppet/trusted_ca](https://forge.puppet.com/modules/puppet/trusted_ca) module into the OS +trust store), use a symlink instead to avoid keeping a second copy: + +```puppet +file { '/opt/puppetlabs/puppet/ssl/certs/proxy-ca.pem': + ensure => link, + target => '/etc/pki/ca-trust/source/anchors/proxy-ca.pem', + notify => Exec['rehash-puppet-ssl-certs'], +} + +exec { 'rehash-puppet-ssl-certs': + command => '/opt/puppetlabs/puppet/bin/openssl rehash /opt/puppetlabs/puppet/ssl/certs/', + refreshonly => true, +} +``` + ## Verifying the configuration