From bd960d69504bec7a13cd09af02d72f8d245245e3 Mon Sep 17 00:00:00 2001 From: Mehdi ABAAKOUK Date: Wed, 2 Sep 2026 10:30:32 +0200 Subject: [PATCH] docs(enterprise): document trusting a private CA The enterprise docs never said how to make an on-premise deployment trust a private or self-signed certificate authority, and installation.mdx told operators to disable Redis TLS verification instead, which is the anti-pattern this was raised about. Add an enterprise page for MERGIFYENGINE_EXTRA_CA_BUNDLE: what belongs in the bundle, how to mount it, how to confirm it took effect, and the places where the setting alone is not enough. PostgreSQL needs sslmode=verify-full before libpq consults the roots at all, REDIS_SSL_VERIFY_MODE_CERT_NONE overrides the bundle, SSL_CERT_DIR reaches only part of the engine, and the bundle is read once per process so rotating the CA needs a restart. The PostgreSQL and Redis caveats come before the verification steps on purpose: a connectivity check against a connection that verifies nothing reports ok either way. installation.mdx now leads its self-signed Redis section with the bundle and keeps REDIS_SSL_VERIFY_MODE_CERT_NONE only as the fallback for a certificate you cannot obtain, while still telling a truly self-signed deployment to put the server certificate itself in the bundle, and pointing managed-Redis users at the CA their provider publishes. Its vague PostgreSQL "configure the connection options accordingly" becomes the concrete sslmode. The section heading is unchanged so its existing anchor keeps resolving. Troubleshooting and requirements link the page from the two places an operator hits the wall: a connectivity check that fails on certificate verification, and an egress proxy that re-signs the subscription calls. Fixes MRGFY-8831 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018XJzBXXpAFeRk3pLsADvjw Change-Id: Ib0ef33d61614b688512cfe24897e121790072193 --- src/content/docs/enterprise.mdx | 3 + .../docs/enterprise/advanced-features.mdx | 3 + src/content/docs/enterprise/custom-ca.mdx | 148 ++++++++++++++++++ src/content/docs/enterprise/installation.mdx | 27 +++- .../enterprise/manual-installation-legacy.mdx | 4 +- src/content/docs/enterprise/requirements.mdx | 7 +- .../docs/enterprise/troubleshooting.mdx | 3 + src/content/enterpriseNavItems.ts | 1 + 8 files changed, 187 insertions(+), 9 deletions(-) create mode 100644 src/content/docs/enterprise/custom-ca.mdx diff --git a/src/content/docs/enterprise.mdx b/src/content/docs/enterprise.mdx index ac5a874f71..63a5131c74 100644 --- a/src/content/docs/enterprise.mdx +++ b/src/content/docs/enterprise.mdx @@ -12,6 +12,9 @@ Enterprise, from requirements to installation, integrations, maintenance, and tr [Manual installation](/enterprise/manual-installation-legacy) if you create the GitHub App yourself instead of using the installer. +- Read [Trusting a Private CA](/enterprise/custom-ca) if any of your infrastructure presents + certificates that are not publicly trusted. + - Explore [Advanced features](/enterprise/advanced-features) such as Datadog telemetry, CI Insights, and Slack integrations. diff --git a/src/content/docs/enterprise/advanced-features.mdx b/src/content/docs/enterprise/advanced-features.mdx index dca067f7c2..c2798d2dce 100644 --- a/src/content/docs/enterprise/advanced-features.mdx +++ b/src/content/docs/enterprise/advanced-features.mdx @@ -118,6 +118,9 @@ MERGIFYENGINE_CI_TRACES_DONE_BUCKET="-mergify-ci-traces-done" MERGIFYENGINE_AWS_ENDPOINT_URL_S3=https://my-s3-domain.example.com:1234/ ``` +A self-hosted endpoint whose certificate is not publicly trusted needs its certificate authority +too: see [Trusting a Private CA](/enterprise/custom-ca). + #### Option A: IAM role discovery (recommended) Leave `MERGIFYENGINE_AWS_ACCESS_KEY_ID` and `MERGIFYENGINE_AWS_SECRET_ACCESS_KEY` unset and Mergify diff --git a/src/content/docs/enterprise/custom-ca.mdx b/src/content/docs/enterprise/custom-ca.mdx new file mode 100644 index 0000000000..67a62c6938 --- /dev/null +++ b/src/content/docs/enterprise/custom-ca.mdx @@ -0,0 +1,148 @@ +--- +title: 'Trusting a Private CA' +description: 'Make an on-premise deployment trust certificates issued by your own certificate authority.' +--- + +Mergify checks the TLS certificates it is presented against a set of public certificate authorities +(CAs). When a server presents a certificate that traces back to none of them, verification fails and +the connection is refused. + +Point the `MERGIFYENGINE_EXTRA_CA_BUNDLE` environment variable at a PEM file and Mergify trusts what +it holds on top of the public certificate authorities it already ships. It is the trust store for +every outbound connection the engine opens. You need it when: + +- Your GitHub Enterprise Server presents a self-signed certificate, or one issued by your internal + PKI. This covers the API calls as well as the `git` clones and pushes the merge queue makes. + +- Your Redis, PostgreSQL, or object storage endpoints do the same. Object storage means both Amazon + S3 and Google Cloud Storage. Managed providers usually publish their certificate authority as a + downloadable PEM file, and that file is what goes here. + +- An egress proxy intercepts outbound TLS and re-signs it with a corporate root. + +Your certificates are added to the public ones rather than replacing them, so calls to Mergify's own +[subscription endpoints](/enterprise/requirements/#external-network-access) keep working. + +Two dependencies need one more setting before they check anything against the bundle: +[PostgreSQL](#postgresql-needs-sslmode-too), which verifies no certificate at all by default, and +[Redis](#redis-and-certificate-verification). + +## Prepare the Bundle + +The file is a PEM bundle: one or more `-----BEGIN CERTIFICATE-----` blocks in a single file. What +belongs in it depends on how the server certificate was issued: + +- **Issued by a certificate authority**, whether your internal PKI or a managed provider's. Use the + root certificate and any intermediates, obtained from whoever runs that authority rather than + harvested from the connection you are trying to verify. + +- **Self-signed.** The certificate is its own issuer, so there is no separate authority to ask for. + Put the server certificate itself in the bundle. + +If what you were given is DER-encoded, convert it first: + +```sh +openssl x509 -inform der -in mycompany-ca.crt -out mycompany-ca.pem +``` + +## Configure the Engine + +Mount the file into the container and name it in the environment. Add these two options to your +[normal-mode command](/enterprise/installation/#start-mergify-in-normal-mode): + +```sh +-v /etc/pki/mycompany-ca.pem:/etc/mergify/ca.pem:ro \ +-e MERGIFYENGINE_EXTRA_CA_BUNDLE=/etc/mergify/ca.pem \ +``` + +The container runs as an unprivileged user, so the mounted file has to be readable by it: + +```sh +chmod 644 /etc/pki/mycompany-ca.pem +``` + +Mergify loads the bundle while it starts and refuses to boot when the file is missing, unreadable, +or not valid PEM, so a wrong path is a startup error naming the setting rather than a handshake +failure hours later. + +If you split the deployment across several containers, mount the file into each one and set the +variable on all of them. Every process reads the bundle for itself. + +## PostgreSQL Needs `sslmode` Too + +The bundle gives PostgreSQL the certificates to check against, but it does not decide whether they +get checked. That is `sslmode` in the DSN, and its default, `prefer`, encrypts the connection +without verifying the certificate at all. Ask for verification explicitly: + +```ini +MERGIFYENGINE_DATABASE_URL=postgresql://postgres:password@db.mycompany.com:5432/postgres?sslmode=verify-full +``` + +`verify-full` also requires the certificate to name the host you connect by. When it does not, a +container alias for instance, use `sslmode=verify-ca`, which checks the certificate against the +bundle without pinning the hostname. Either way the engine opens the database while it starts, so a +mismatch is a boot failure rather than something you find later. + +A `?sslrootcert=` already in the DSN takes precedence, and PostgreSQL then verifies against that +file alone. + +## Redis and Certificate Verification + +Connect with `rediss://` and Mergify verifies the Redis certificate against the bundle like any +other. A `?ssl_ca_certs=` in `MERGIFYENGINE_REDIS_URL` points Redis at that file instead. + +:::caution + `MERGIFYENGINE_REDIS_SSL_VERIFY_MODE_CERT_NONE=1` turns Redis certificate verification off + entirely and overrides the bundle. The connection stays encrypted, but Mergify can no longer tell + your Redis apart from anything else answering at that address. Keep it only while you cannot + obtain the certificate, and unset it once you can; Mergify logs a line when both are configured. +::: + +## Verify It Took Effect + +Finish the two sections above first. A connectivity check against a PostgreSQL or Redis that is not +verifying anything reports `ok` without a certificate having been checked. + +Restart the container and look for the bundle in the startup logs: + +```sh +docker logs mergify-engine | grep "Extra CA bundle" +``` + +Each service logs one line naming the bundle it loaded. If the grep finds nothing, read the logs +unfiltered before suspecting your `-e` flag: a bundle the engine cannot read or parse stops the +container before that line, with a different error naming the setting. + +Then check that Mergify reaches each dependency: + +```sh +docker exec -u root -it mergify-engine /bin/bash +mergify-admin connectivity-check +``` + +Every configured dependency should report `ok`; `skipped` means it is not configured at all. A check +that fails on certificate verification means that server's issuer is still missing from the bundle. +See [Troubleshooting](/enterprise/troubleshooting/#checking-third-party-connectivity) for the full +output format. + +## Rotating the Certificate Authority + +The bundle is read once, when the process starts. Replacing the file on disk changes nothing until +the containers restart. + +To rotate without downtime, add the incoming certificate to the bundle alongside the outgoing one, +restart Mergify, switch your servers over, and only then drop the old certificate and restart again. + +## Other Trust Store Variables + +`SSL_CERT_FILE` and `SSL_CERT_DIR` replace the public certificate authorities Mergify would +otherwise use, and `MERGIFYENGINE_EXTRA_CA_BUNDLE` is added on top of whichever you set. + +Use `SSL_CERT_FILE` with a PEM bundle. `SSL_CERT_DIR` reaches the GitHub and Redis connections, but +not the ones that take a single bundle file: PostgreSQL, object storage, Sentry and `git` fall back +to the public authorities Mergify ships, plus your bundle. Your replacement trust store is then only +partly in effect, and Mergify logs a line saying so at startup. + +`AWS_CA_BUNDLE`, if you already have one set, keeps object storage on that file and the bundle is +not applied there. `REQUESTS_CA_BUNDLE` and `CURL_CA_BUNDLE` do nothing at all: Mergify clears them +before it builds any client. diff --git a/src/content/docs/enterprise/installation.mdx b/src/content/docs/enterprise/installation.mdx index ff33321ff4..1b9367b421 100644 --- a/src/content/docs/enterprise/installation.mdx +++ b/src/content/docs/enterprise/installation.mdx @@ -163,21 +163,32 @@ Redis server provides TLS termination. To connect to such an instance you need t #### Optional: use Redis with self-signed TLS certificate -Some Redis server providers set it up with self-signed certificates. In that case, to connect to -such an instance you need to: +Some Redis server providers set it up with self-signed certificates, or with certificates from their +own certificate authority. In that case, to connect to such an instance you need to: - Use `rediss://` instead of `redis://` in the `MERGIFYENGINE_REDIS_URL` environment variable. -- Set `MERGIFYENGINE_REDIS_SSL_VERIFY_MODE_CERT_NONE=1` in your environment variables. Mergify will - connect to Redis using encryption but will not verify the server certificate. +- Add the certificate that signed it, or the server certificate itself when it is self-signed, to + the bundle described in [Trusting a Private CA](/enterprise/custom-ca). Managed providers usually + publish theirs as a downloadable PEM file. Mergify then verifies the Redis certificate like any + other. + +:::caution + `MERGIFYENGINE_REDIS_SSL_VERIFY_MODE_CERT_NONE=1` is the fallback for when you cannot obtain the + certificate at all. Mergify keeps encrypting the connection but stops verifying the server + certificate, so it can no longer tell your Redis apart from anything else answering at that + address. It also overrides the bundle, so unset it once you hold the certificate. +::: ### PostgreSQL Provision a PostgreSQL instance, create a database/user for Mergify, and ensure the container can reach it over the network. Capture the DSN you will pass through `MERGIFYENGINE_DATABASE_URL` (for example `postgresql://postgres:password@postgres:5432/postgres`). -If your provider enforces TLS, configure the connection options accordingly. The first start also -creates a few extensions in that database, so check the +If your provider enforces TLS, add the matching `sslmode` to the DSN; a certificate issued by your +own certificate authority needs `sslmode=verify-full` and the bundle described in +[Trusting a Private CA](/enterprise/custom-ca). The first start also creates a few extensions in +that database, so check the [PostgreSQL requirements](/enterprise/requirements/#postgresql-requirements) before you lock down its privileges. @@ -206,7 +217,9 @@ each authentication mode. ::: The example below includes the CI Insights and Test Insights environment variables. Omit them if you -skipped the previous step. +skipped the previous step. If any of the services Mergify connects to present certificates that are +not publicly trusted, see [Trusting a Private CA](/enterprise/custom-ca) for the two options to add +to this command. ```sh docker run \ diff --git a/src/content/docs/enterprise/manual-installation-legacy.mdx b/src/content/docs/enterprise/manual-installation-legacy.mdx index c98fdb89e6..bd4ff7e89d 100644 --- a/src/content/docs/enterprise/manual-installation-legacy.mdx +++ b/src/content/docs/enterprise/manual-installation-legacy.mdx @@ -128,7 +128,9 @@ docker pull registry.mergify.com/enterprise:@@ENTERPRISE_VERSION@@ > A subscription token is required. Contact Mergify if you do not have one. > -> Ensure Redis (with persistence/TLS if needed) and Postgres are ready. +> Ensure Redis (with persistence/TLS if needed) and Postgres are ready. If they, or your GitHub +> Enterprise Server, present certificates that are not publicly trusted, see +> [Trusting a Private CA](/enterprise/custom-ca). ```sh docker run \ diff --git a/src/content/docs/enterprise/requirements.mdx b/src/content/docs/enterprise/requirements.mdx index c688f79ac3..0283932831 100644 --- a/src/content/docs/enterprise/requirements.mdx +++ b/src/content/docs/enterprise/requirements.mdx @@ -56,6 +56,9 @@ docker run --name some-redis-with-tls -d \ bitnami/redis ``` +That example makes Redis present a certificate signed by a CA you created. Give that CA to Mergify +as described in [Trusting a Private CA](/enterprise/custom-ca), so it can verify the connection. + ## Mergify Requirements The Mergify container needs at least: @@ -90,4 +93,6 @@ Two outbound HTTPS calls must be allowed so Mergify can validate and report lice - `POST https://subscription.mergify.com/on-premise/report` Both calls carry your subscription token and are made by the Mergify container itself, so no other -component needs egress to that host. +component needs egress to that host. If a proxy intercepts outbound TLS and re-signs it with your +own certificate authority, Mergify has to trust that authority for these calls to succeed: see +[Trusting a Private CA](/enterprise/custom-ca). diff --git a/src/content/docs/enterprise/troubleshooting.mdx b/src/content/docs/enterprise/troubleshooting.mdx index 75639ad177..b126a3b0f5 100644 --- a/src/content/docs/enterprise/troubleshooting.mdx +++ b/src/content/docs/enterprise/troubleshooting.mdx @@ -37,6 +37,9 @@ object_storage: skipped github_server: ok ``` +A check that fails on certificate verification means the server presents a certificate Mergify does +not trust. See [Trusting a Private CA](/enterprise/custom-ca). + ### API healthcheck Add a shared token: diff --git a/src/content/enterpriseNavItems.ts b/src/content/enterpriseNavItems.ts index 5f12a316ee..4ff52376b8 100644 --- a/src/content/enterpriseNavItems.ts +++ b/src/content/enterpriseNavItems.ts @@ -5,6 +5,7 @@ const enterpriseNavItems: NavItem[] = [ { title: 'Architecture', path: '/enterprise/architecture', icon: 'lucide:layout-grid' }, { title: 'Requirements', path: '/enterprise/requirements', icon: 'lucide:clipboard-list' }, { title: 'Installation', path: '/enterprise/installation', icon: 'lucide:wrench' }, + { title: 'Private CA', path: '/enterprise/custom-ca', icon: 'lucide:shield-check' }, { title: 'Advanced Features', path: '/enterprise/advanced-features',