merge queue: checking #12658 on main (82fa6ed), stacked on #12659

src/content/docs/api/usage.mdx

@@ -17,9 +17,9 @@ The API is entirely documented in the [API Reference](/api).
 
 ## Authentication
 
-The Mergify API supports two authentication methods, both using Bearer
-tokens: **Application Keys** (generated from your dashboard)
-and **GitHub Personal Access Tokens**.
+The Mergify API supports three authentication methods, all using Bearer
+tokens: **Application Keys** (generated from your dashboard), **GitHub
+Personal Access Tokens**, and **Mergify User Tokens**.
 
 ### Creating an Application Key
 
@@ -118,13 +118,23 @@ curl -H "Accept: application/json" \
   organizations.
 :::
 
+### Using a Mergify User Token
+
+Mergify also issues user tokens of its own, recognizable by their `mut_`
+prefix and obtained through the OAuth 2.0 device authorization grant. Send one
+as a Bearer token, the same way as the credentials above.
+
+A user token identifies the person it was issued to. It reaches exactly what
+its owner's dashboard session reaches, and nothing on GitHub directly, so
+holding one grants no access its owner does not already have.
+
 :::caution
-  Four endpoints refuse a PAT and need an application key. `GET /application`
-  describes the key it was called with, so it accepts either scope. The `PUT` on
-  `/repos/{owner}/{repository}/commits/{sha}/scopes` and the `PUT` and `POST`
-  on `/repos/{owner}/{repository}/pulls/{number}/scopes` need a `ci` key.
-  Everything else takes a PAT or an `admin` key; each endpoint in the [API
-  Reference](/api) lists the keys it accepts.
+  Four endpoints refuse a PAT and a user token alike, and need an application
+  key. `GET /application` describes the key it was called with, so it accepts either
+  scope. The `PUT` on `/repos/{owner}/{repository}/commits/{sha}/scopes` and the
+  `PUT` and `POST` on `/repos/{owner}/{repository}/pulls/{number}/scopes` need a
+  `ci` key. Everything else takes a PAT, a user token, or an `admin` key; each
+  endpoint in the [API Reference](/api) lists the credentials it accepts.
 :::
 
 ### Revoking an Application Key

src/content/docs/enterprise.mdx

@@ -12,6 +12,9 @@ Enterprise, from requirements to installation, integrations, maintenance, and tr
   [Manual installation](/enterprise/manual-installation-legacy) if you create the GitHub App
   yourself instead of using the installer.
 
+- Read [Trusting a Private CA](/enterprise/custom-ca) if any of your infrastructure presents
+  certificates that are not publicly trusted.
+
 - Explore [Advanced features](/enterprise/advanced-features) such as Datadog telemetry, CI Insights,
   and Slack integrations.
 

src/content/docs/enterprise/advanced-features.mdx

@@ -118,6 +118,9 @@ MERGIFYENGINE_CI_TRACES_DONE_BUCKET="<mycompany>-mergify-ci-traces-done"
 MERGIFYENGINE_AWS_ENDPOINT_URL_S3=https://my-s3-domain.example.com:1234/
 ```
 
+A self-hosted endpoint whose certificate is not publicly trusted needs its certificate authority
+too: see [Trusting a Private CA](/enterprise/custom-ca).
+
 #### Option A: IAM role discovery (recommended)
 
 Leave `MERGIFYENGINE_AWS_ACCESS_KEY_ID` and `MERGIFYENGINE_AWS_SECRET_ACCESS_KEY` unset and Mergify

src/content/docs/enterprise/custom-ca.mdx

@@ -0,0 +1,148 @@
+---
+title: 'Trusting a Private CA'
+description: 'Make an on-premise deployment trust certificates issued by your own certificate authority.'
+---
+
+Mergify checks the TLS certificates it is presented against a set of public certificate authorities
+(CAs). When a server presents a certificate that traces back to none of them, verification fails and
+the connection is refused.
+
+Point the `MERGIFYENGINE_EXTRA_CA_BUNDLE` environment variable at a PEM file and Mergify trusts what
+it holds on top of the public certificate authorities it already ships. It is the trust store for
+every outbound connection the engine opens. You need it when:
+
+- Your GitHub Enterprise Server presents a self-signed certificate, or one issued by your internal
+  PKI. This covers the API calls as well as the `git` clones and pushes the merge queue makes.
+
+- Your Redis, PostgreSQL, or object storage endpoints do the same. Object storage means both Amazon
+  S3 and Google Cloud Storage. Managed providers usually publish their certificate authority as a
+  downloadable PEM file, and that file is what goes here.
+
+- An egress proxy intercepts outbound TLS and re-signs it with a corporate root.
+
+Your certificates are added to the public ones rather than replacing them, so calls to Mergify's own
+[subscription endpoints](/enterprise/requirements/#external-network-access) keep working.
+
+Two dependencies need one more setting before they check anything against the bundle:
+[PostgreSQL](#postgresql-needs-sslmode-too), which verifies no certificate at all by default, and
+[Redis](#redis-and-certificate-verification).
+
+## Prepare the Bundle
+
+The file is a PEM bundle: one or more `-----BEGIN CERTIFICATE-----` blocks in a single file. What
+belongs in it depends on how the server certificate was issued:
+
+- **Issued by a certificate authority**, whether your internal PKI or a managed provider's. Use the
+  root certificate and any intermediates, obtained from whoever runs that authority rather than
+  harvested from the connection you are trying to verify.
+
+- **Self-signed.** The certificate is its own issuer, so there is no separate authority to ask for.
+  Put the server certificate itself in the bundle.
+
+If what you were given is DER-encoded, convert it first:
+
+```sh
+openssl x509 -inform der -in mycompany-ca.crt -out mycompany-ca.pem
+```
+
+## Configure the Engine
+
+Mount the file into the container and name it in the environment. Add these two options to your
+[normal-mode command](/enterprise/installation/#start-mergify-in-normal-mode):
+
+```sh
+-v /etc/pki/mycompany-ca.pem:/etc/mergify/ca.pem:ro \
+-e MERGIFYENGINE_EXTRA_CA_BUNDLE=/etc/mergify/ca.pem \
+```
+
+The container runs as an unprivileged user, so the mounted file has to be readable by it:
+
+```sh
+chmod 644 /etc/pki/mycompany-ca.pem
+```
+
+Mergify loads the bundle while it starts and refuses to boot when the file is missing, unreadable,
+or not valid PEM, so a wrong path is a startup error naming the setting rather than a handshake
+failure hours later.
+
+If you split the deployment across several containers, mount the file into each one and set the
+variable on all of them. Every process reads the bundle for itself.
+
+## PostgreSQL Needs `sslmode` Too
+
+The bundle gives PostgreSQL the certificates to check against, but it does not decide whether they
+get checked. That is `sslmode` in the DSN, and its default, `prefer`, encrypts the connection
+without verifying the certificate at all. Ask for verification explicitly:
+
+```ini
+MERGIFYENGINE_DATABASE_URL=postgresql://postgres:password@db.mycompany.com:5432/postgres?sslmode=verify-full
+```
+
+`verify-full` also requires the certificate to name the host you connect by. When it does not, a
+container alias for instance, use `sslmode=verify-ca`, which checks the certificate against the
+bundle without pinning the hostname. Either way the engine opens the database while it starts, so a
+mismatch is a boot failure rather than something you find later.
+
+A `?sslrootcert=` already in the DSN takes precedence, and PostgreSQL then verifies against that
+file alone.
+
+## Redis and Certificate Verification
+
+Connect with `rediss://` and Mergify verifies the Redis certificate against the bundle like any
+other. A `?ssl_ca_certs=` in `MERGIFYENGINE_REDIS_URL` points Redis at that file instead.
+
+:::caution
+  `MERGIFYENGINE_REDIS_SSL_VERIFY_MODE_CERT_NONE=1` turns Redis certificate verification off
+  entirely and overrides the bundle. The connection stays encrypted, but Mergify can no longer tell
+  your Redis apart from anything else answering at that address. Keep it only while you cannot
+  obtain the certificate, and unset it once you can; Mergify logs a line when both are configured.
+:::
+
+## Verify It Took Effect
+
+Finish the two sections above first. A connectivity check against a PostgreSQL or Redis that is not
+verifying anything reports `ok` without a certificate having been checked.
+
+Restart the container and look for the bundle in the startup logs:
+
+```sh
+docker logs mergify-engine | grep "Extra CA bundle"
+```
+
+Each service logs one line naming the bundle it loaded. If the grep finds nothing, read the logs
+unfiltered before suspecting your `-e` flag: a bundle the engine cannot read or parse stops the
+container before that line, with a different error naming the setting.
+
+Then check that Mergify reaches each dependency:
+
+```sh
+docker exec -u root -it mergify-engine /bin/bash
+mergify-admin connectivity-check
+```
+
+Every configured dependency should report `ok`; `skipped` means it is not configured at all. A check
+that fails on certificate verification means that server's issuer is still missing from the bundle.
+See [Troubleshooting](/enterprise/troubleshooting/#checking-third-party-connectivity) for the full
+output format.
+
+## Rotating the Certificate Authority
+
+The bundle is read once, when the process starts. Replacing the file on disk changes nothing until
+the containers restart.
+
+To rotate without downtime, add the incoming certificate to the bundle alongside the outgoing one,
+restart Mergify, switch your servers over, and only then drop the old certificate and restart again.
+
+## Other Trust Store Variables
+
+`SSL_CERT_FILE` and `SSL_CERT_DIR` replace the public certificate authorities Mergify would
+otherwise use, and `MERGIFYENGINE_EXTRA_CA_BUNDLE` is added on top of whichever you set.
+
+Use `SSL_CERT_FILE` with a PEM bundle. `SSL_CERT_DIR` reaches the GitHub and Redis connections, but
+not the ones that take a single bundle file: PostgreSQL, object storage, Sentry and `git` fall back
+to the public authorities Mergify ships, plus your bundle. Your replacement trust store is then only
+partly in effect, and Mergify logs a line saying so at startup.
+
+`AWS_CA_BUNDLE`, if you already have one set, keeps object storage on that file and the bundle is
+not applied there. `REQUESTS_CA_BUNDLE` and `CURL_CA_BUNDLE` do nothing at all: Mergify clears them
+before it builds any client.

src/content/docs/enterprise/installation.mdx

@@ -163,21 +163,32 @@ Redis server provides TLS termination. To connect to such an instance you need t
 
 #### Optional: use Redis with self-signed TLS certificate
 
-Some Redis server providers set it up with self-signed certificates. In that case, to connect to
-such an instance you need to:
+Some Redis server providers set it up with self-signed certificates, or with certificates from their
+own certificate authority. In that case, to connect to such an instance you need to:
 
 - Use `rediss://` instead of `redis://` in the `MERGIFYENGINE_REDIS_URL` environment variable.
 
-- Set `MERGIFYENGINE_REDIS_SSL_VERIFY_MODE_CERT_NONE=1` in your environment variables. Mergify will
-  connect to Redis using encryption but will not verify the server certificate.
+- Add the certificate that signed it, or the server certificate itself when it is self-signed, to
+  the bundle described in [Trusting a Private CA](/enterprise/custom-ca). Managed providers usually
+  publish theirs as a downloadable PEM file. Mergify then verifies the Redis certificate like any
+  other.
+
+:::caution
+  `MERGIFYENGINE_REDIS_SSL_VERIFY_MODE_CERT_NONE=1` is the fallback for when you cannot obtain the
+  certificate at all. Mergify keeps encrypting the connection but stops verifying the server
+  certificate, so it can no longer tell your Redis apart from anything else answering at that
+  address. It also overrides the bundle, so unset it once you hold the certificate.
+:::
 
 ### PostgreSQL
 
 Provision a PostgreSQL instance, create a database/user for Mergify, and ensure the container can
 reach it over the network. Capture the DSN you will pass through
 `MERGIFYENGINE_DATABASE_URL` (for example `postgresql://postgres:password@postgres:5432/postgres`).
-If your provider enforces TLS, configure the connection options accordingly. The first start also
-creates a few extensions in that database, so check the
+If your provider enforces TLS, add the matching `sslmode` to the DSN; a certificate issued by your
+own certificate authority needs `sslmode=verify-full` and the bundle described in
+[Trusting a Private CA](/enterprise/custom-ca). The first start also creates a few extensions in
+that database, so check the
 [PostgreSQL requirements](/enterprise/requirements/#postgresql-requirements) before you lock down
 its privileges.
 
@@ -206,7 +217,9 @@ each authentication mode.
 :::
 
 The example below includes the CI Insights and Test Insights environment variables. Omit them if you
-skipped the previous step.
+skipped the previous step. If any of the services Mergify connects to present certificates that are
+not publicly trusted, see [Trusting a Private CA](/enterprise/custom-ca) for the two options to add
+to this command.
 
 ```sh
 docker run \

src/content/docs/enterprise/manual-installation-legacy.mdx

@@ -128,7 +128,9 @@ docker pull registry.mergify.com/enterprise:@@ENTERPRISE_VERSION@@
 
 > A subscription token is required. Contact Mergify if you do not have one.
 >
-> Ensure Redis (with persistence/TLS if needed) and Postgres are ready.
+> Ensure Redis (with persistence/TLS if needed) and Postgres are ready. If they, or your GitHub
+> Enterprise Server, present certificates that are not publicly trusted, see
+> [Trusting a Private CA](/enterprise/custom-ca).
 
 ```sh
 docker run \

src/content/docs/enterprise/requirements.mdx

@@ -56,6 +56,9 @@ docker run --name some-redis-with-tls -d \
     bitnami/redis
 ```
 
+That example makes Redis present a certificate signed by a CA you created. Give that CA to Mergify
+as described in [Trusting a Private CA](/enterprise/custom-ca), so it can verify the connection.
+
 ## Mergify Requirements
 
 The Mergify container needs at least:
@@ -90,4 +93,6 @@ Two outbound HTTPS calls must be allowed so Mergify can validate and report lice
 - `POST https://subscription.mergify.com/on-premise/report`
 
 Both calls carry your subscription token and are made by the Mergify container itself, so no other
-component needs egress to that host.
+component needs egress to that host. If a proxy intercepts outbound TLS and re-signs it with your
+own certificate authority, Mergify has to trust that authority for these calls to succeed: see
+[Trusting a Private CA](/enterprise/custom-ca).

src/content/docs/enterprise/troubleshooting.mdx

@@ -37,6 +37,9 @@ object_storage: skipped
 github_server: ok
 ```
 
+A check that fails on certificate verification means the server presents a certificate Mergify does
+not trust. See [Trusting a Private CA](/enterprise/custom-ca).
+
 ### API healthcheck
 
 Add a shared token:

src/content/enterpriseNavItems.ts

@@ -5,6 +5,7 @@ const enterpriseNavItems: NavItem[] = [
   { title: 'Architecture', path: '/enterprise/architecture', icon: 'lucide:layout-grid' },
   { title: 'Requirements', path: '/enterprise/requirements', icon: 'lucide:clipboard-list' },
   { title: 'Installation', path: '/enterprise/installation', icon: 'lucide:wrench' },
+  { title: 'Private CA', path: '/enterprise/custom-ca', icon: 'lucide:shield-check' },
   {
     title: 'Advanced Features',
     path: '/enterprise/advanced-features',