From dc3463586e8243c9236f74e949f0f4fdcadc502f Mon Sep 17 00:00:00 2001 From: as51340 Date: Thu, 28 May 2026 21:21:05 +0200 Subject: [PATCH 1/2] feat: Intra-cluster TLS --- .../how-high-availability-works.mdx | 63 ++++++++++++++++++- .../query-privileges.mdx | 2 + .../role-based-access-control.mdx | 1 + pages/database-management/configuration.mdx | 3 + pages/database-management/ssl-encryption.mdx | 28 +++++++++ pages/release-notes.mdx | 10 +++ 6 files changed, 106 insertions(+), 1 deletion(-) diff --git a/pages/clustering/high-availability/how-high-availability-works.mdx b/pages/clustering/high-availability/how-high-availability-works.mdx index 454f92fc9..6ce13f8b3 100644 --- a/pages/clustering/high-availability/how-high-availability-works.mdx +++ b/pages/clustering/high-availability/how-high-availability-works.mdx @@ -118,7 +118,8 @@ That means, you cannot run any data queries on the coordinator directly (we will talk more about routing data queries in the next sections). However, system information queries such as `SHOW CONFIG`, `SHOW LICENSE INFO`, `SHOW BUILD INFO` and `SHOW STORAGE INFO` are supported on coordinators, as -well as `SET DATABASE SETTING` and `RELOAD SSL`. +well as `SET DATABASE SETTING`, `RELOAD BOLT_SERVER TLS` and +`RELOAD INTRA_CLUSTER TLS`. Since coordinators do not store user data, the following restrictions apply: @@ -272,6 +273,66 @@ in the cluster to ensure high availability, with timeouts. | `WalFilesRpc` | Main | Replica | proportional | | `CurrentWalRpc` | Main | Replica | proportional | +## Intra-cluster TLS + +By default, the communication between instances in a high-availability cluster +is unencrypted. To secure it, Memgraph supports **intra-cluster TLS**, which +encrypts all internal cluster traffic using mutual TLS (mTLS). When enabled, +TLS protects the communication on: + +- the **management server** (health checks between the leader coordinator and + the instances), +- the **replication server** (data replication between MAIN and REPLICA + instances), and +- the **coordinator server** (synchronization and log replication between + coordinators). + + +Intra-cluster TLS is independent of [Bolt SSL/TLS](/database-management/ssl-encryption). +Bolt encryption secures client-to-instance connections and is configured +separately with the `--bolt-cert-file` and `--bolt-key-file` flags, while +intra-cluster TLS secures the internal cluster communication described above. + + +### Enabling intra-cluster TLS + +Intra-cluster TLS is enabled by setting the following three flags on every +instance (coordinators and data instances) in the cluster: + +| Flag | Description | +| --------------------- | ---------------------------------------------------------------------------------------------------- | +| `--cluster-cert-file` | Certificate file used for intra-cluster TLS communication. | +| `--cluster-key-file` | Key file used for intra-cluster TLS communication. | +| `--cluster-ca-file` | File storing the certificate of the Certificate Authority you trust for intra-cluster TLS communication. | + +All three flags must be set together. If only some of them are provided, the +instance refuses to start to avoid running in a partially-configured TLS state. +When all three are empty, intra-cluster TLS is disabled and communication is +unencrypted. Because mTLS is used, every instance must present a certificate +that is trusted by the configured Certificate Authority, and all instances in +the cluster must be started with the TLS flags. + +### Reloading intra-cluster TLS certificates at runtime + +You can rotate the intra-cluster TLS certificates without restarting the +cluster by running the `RELOAD INTRA_CLUSTER TLS` Cypher query. Replace the +certificate and key files on disk at the paths configured with +`--cluster-cert-file`, `--cluster-key-file`, and `--cluster-ca-file`, then run: + +```cypher +RELOAD INTRA_CLUSTER TLS; +``` + +`RELOAD INTRA_CLUSTER TLS` reloads both the client and server connections, so +new connections will start using the new certificates. + +This is the intra-cluster counterpart of `RELOAD BOLT_SERVER TLS`, which +reloads the Bolt server certificates. Both queries require the `RELOAD_TLS` +[privilege](/database-management/authentication-and-authorization/role-based-access-control). +For more details on reloading certificates, see the +[SSL encryption](/database-management/ssl-encryption#reload-ssl-certificates-at-runtime) +page. + ## Automatic failover Automatic failover is driven by periodic health checks performed by the leader diff --git a/pages/database-management/authentication-and-authorization/query-privileges.mdx b/pages/database-management/authentication-and-authorization/query-privileges.mdx index 68133a469..b86967997 100644 --- a/pages/database-management/authentication-and-authorization/query-privileges.mdx +++ b/pages/database-management/authentication-and-authorization/query-privileges.mdx @@ -146,6 +146,8 @@ Memgraph's privilege system controls access to various database operations throu | `SHOW VERSION` | `STATS` | `SHOW VERSION` | | `SHOW TRANSACTIONS` | `TRANSACTION_MANAGEMENT` | `SHOW TRANSACTIONS` | | `TERMINATE TRANSACTIONS` | `TRANSACTION_MANAGEMENT` | `TERMINATE TRANSACTIONS 'transaction_id'` | +| `RELOAD BOLT_SERVER TLS` | `RELOAD_TLS` | `RELOAD BOLT_SERVER TLS` | +| `RELOAD INTRA_CLUSTER TLS` | `RELOAD_TLS` | `RELOAD INTRA_CLUSTER TLS` | ## Replication operations diff --git a/pages/database-management/authentication-and-authorization/role-based-access-control.mdx b/pages/database-management/authentication-and-authorization/role-based-access-control.mdx index ab2246994..f0c447432 100644 --- a/pages/database-management/authentication-and-authorization/role-based-access-control.mdx +++ b/pages/database-management/authentication-and-authorization/role-based-access-control.mdx @@ -249,6 +249,7 @@ of the following commands: | Privilege to set limits and monitor resource usage per user (via [user profiles](/database-management/authentication-and-authorization/user-profiles)) or per database (via [tenant profiles](/database-management/tenant-profiles)). | `PROFILE_RESTRICTION` | | Privilege to manage [server-side parameters](/database-management/server-side-parameters) (`SET`, `UNSET`, `SHOW`). | `SERVER_SIDE_PARAMETERS` | | Privilege to manage [server-side descriptions](/database-management/server-side-descriptions) (`SET`, `DELETE`, `SHOW`). | `SERVER_SIDE_DESCRIPTIONS` | +| Privilege to reload [SSL/TLS certificates](/database-management/ssl-encryption#reload-ssl-certificates-at-runtime) at runtime (`RELOAD BOLT_SERVER TLS`, `RELOAD INTRA_CLUSTER TLS`). | `RELOAD_TLS` | | Privileges to specific labels. | `ALL LABELS` | | Privileges to specific relationships types. | `ALL EDGE TYPES` | diff --git a/pages/database-management/configuration.mdx b/pages/database-management/configuration.mdx index 7c49244f8..b437ea674 100644 --- a/pages/database-management/configuration.mdx +++ b/pages/database-management/configuration.mdx @@ -437,6 +437,9 @@ This section contains the list of flags that are used to configure highly availa | ~~`--instance-down-timeout-sec`~~ | **Deprecated in 3.10.** This flag is ignored. Use `SET COORDINATOR SETTING 'instance_down_timeout_sec' TO ''` instead. See [Coordinator runtime settings](/clustering/high-availability/best-practices#coordinator-runtime-settings). | `[uint32]` | | `--nuraft-log-file` | Path to the file where NuRaft logs are saved. | `[string]` | | `--coordinator-hostname` | Coordinator's instance hostname. Used only in `SHOW INSTANCES` query. | `[string]` | +| `--cluster-cert-file` | Certificate file used for [intra-cluster TLS](/clustering/high-availability/how-high-availability-works#intra-cluster-tls) communication. Must be set together with `--cluster-key-file` and `--cluster-ca-file`. | `[string]` | +| `--cluster-key-file` | Key file used for [intra-cluster TLS](/clustering/high-availability/how-high-availability-works#intra-cluster-tls) communication. Must be set together with `--cluster-cert-file` and `--cluster-ca-file`. | `[string]` | +| `--cluster-ca-file` | File storing the certificate of the Certificate Authority you trust for [intra-cluster TLS](/clustering/high-availability/how-high-availability-works#intra-cluster-tls) communication. Must be set together with `--cluster-cert-file` and `--cluster-key-file`. | `[string]` | ### Query diff --git a/pages/database-management/ssl-encryption.mdx b/pages/database-management/ssl-encryption.mdx index 780b62c8e..185c6dd50 100644 --- a/pages/database-management/ssl-encryption.mdx +++ b/pages/database-management/ssl-encryption.mdx @@ -182,6 +182,34 @@ Running `RELOAD BOLT_SERVER TLS` on a Memgraph instance that was started without SSL enabled will return an error. + +**Breaking change in Memgraph 3.11:** `RELOAD BOLT_SERVER TLS` now requires the +`RELOAD_TLS` +[privilege](/database-management/authentication-and-authorization/role-based-access-control). +Previously no privilege was needed. If you use authorization, grant `RELOAD_TLS` +to the users or roles that perform certificate rotation: +`GRANT RELOAD_TLS TO user;`. + + +### Reload intra-cluster TLS certificates + +In a [high-availability](/clustering/high-availability) cluster, the internal +communication between instances can be secured with +[intra-cluster TLS](/clustering/high-availability/how-high-availability-works#intra-cluster-tls). +To rotate those certificates at runtime, replace the certificate and key files +on disk (at the paths configured with `--cluster-cert-file`, +`--cluster-key-file`, and `--cluster-ca-file`) and run: + +```cypher +RELOAD INTRA_CLUSTER TLS; +``` + +`RELOAD INTRA_CLUSTER TLS` reloads both the client and server connections, so +new connections will start using the new certificates. + +Like `RELOAD BOLT_SERVER TLS`, this command requires the `RELOAD_TLS` privilege +and cannot be executed inside an explicit (multi-command) transaction. + ## How to set up SSL encryption Memgraph uses SSL (Secure Sockets Layer) protocol for establishing an diff --git a/pages/release-notes.mdx b/pages/release-notes.mdx index 5bacc4b14..90aceb24c 100644 --- a/pages/release-notes.mdx +++ b/pages/release-notes.mdx @@ -84,6 +84,11 @@ guide. you wrote custom procedures that relied on the `transaction_id` parameter passed by the `cross_database` module, switch to `graph.start_timestamp`. [#4167](https://github.com/memgraph/memgraph/pull/4167) +- `RELOAD BOLT_SERVER TLS` now requires the `RELOAD_TLS` privilege; previously it + required no privilege. If you use authorization, grant `RELOAD_TLS` to the + users or roles that rotate certificates (`GRANT RELOAD_TLS TO user;`), + otherwise the command will fail with an authorization error. + [#4154](https://github.com/memgraph/memgraph/pull/4154) {

✨ New features

} @@ -115,6 +120,11 @@ guide. communication on the management, replication, and coordinator servers. Bolt TLS remains independently configured. [#4140](https://github.com/memgraph/memgraph/pull/4140) +- Added the `RELOAD INTRA_CLUSTER TLS` Cypher query to rotate intra-cluster TLS + certificates at runtime without restarting the cluster. The new `RELOAD_TLS` + privilege now governs both `RELOAD INTRA_CLUSTER TLS` and + `RELOAD BOLT_SERVER TLS`. + [#4154](https://github.com/memgraph/memgraph/pull/4154) - `SHOW TRANSACTIONS` now includes `start_time` (UTC timestamp) and `elapsed_ms` columns, making it easier to identify long-running transactions without calculating elapsed time manually. From b99d23bfcb2cc733c34757ba336467087e5a2d9d Mon Sep 17 00:00:00 2001 From: as51340 Date: Fri, 29 May 2026 13:43:26 +0200 Subject: [PATCH 2/2] docs: Document charts-side of the reload feature --- .../setup-ha-cluster-k8s.mdx | 85 ++++++++++++++++++- 1 file changed, 81 insertions(+), 4 deletions(-) diff --git a/pages/clustering/high-availability/setup-ha-cluster-k8s.mdx b/pages/clustering/high-availability/setup-ha-cluster-k8s.mdx index ca4325d0a..b87265ab2 100644 --- a/pages/clustering/high-availability/setup-ha-cluster-k8s.mdx +++ b/pages/clustering/high-availability/setup-ha-cluster-k8s.mdx @@ -527,6 +527,76 @@ When a coordinator has `tls.bolt.enabled: true`, the cluster-setup job that registers coordinators and data instances automatically uses `--use-ssl` when connecting to coordinator 1. +### Intra-cluster SSL/TLS + +Independently of Bolt TLS, each data instance and coordinator can encrypt the +internal communication between cluster members (coordinator-to-coordinator and +coordinator-to-data traffic). When `tls.intraCluster.enabled` is `true`, the +chart mounts a pre-existing Kubernetes Secret containing the certificate, +private key and CA bundle at `/etc/memgraph/intra_cluster_tls/` and +auto-appends `--cluster-cert-file=/etc/memgraph/intra_cluster_tls/tls.crt`, +`--cluster-key-file=/etc/memgraph/intra_cluster_tls/tls.key` and +`--cluster-ca-file=/etc/memgraph/intra_cluster_tls/ca.crt` to the instance's +args. + +To enable intra-cluster TLS, first create a Kubernetes Secret holding the +certificate, private key and CA bundle for each instance in the release +namespace. For example, for `data-0`: + +```bash +kubectl create secret generic intra-tls-data-0-secret \ + --from-file=tls.crt=path/to/tls.crt \ + --from-file=tls.key=path/to/tls.key \ + --from-file=ca.crt=path/to/ca.crt +``` + +Then enable `tls.intraCluster` on each instance that should encrypt internal +traffic: + +```yaml +data: + - id: "0" + tls: + intraCluster: + enabled: true + secretName: intra-tls-data-0-secret + certSecretPath: tls.crt + keySecretPath: tls.key + caSecretPath: ca.crt + - id: "1" + tls: + intraCluster: + enabled: true + secretName: intra-tls-data-1-secret + certSecretPath: tls.crt + keySecretPath: tls.key + caSecretPath: ca.crt + +coordinators: + - id: "1" + tls: + intraCluster: + enabled: true + secretName: intra-tls-coord-1-secret + - id: "2" + tls: + intraCluster: + enabled: true + secretName: intra-tls-coord-2-secret + - id: "3" + tls: + intraCluster: + enabled: true + secretName: intra-tls-coord-3-secret +``` + +`certSecretPath`, `keySecretPath` and `caSecretPath` are the keys inside the +Secret holding the certificate, private key and CA bundle respectively +(default `tls.crt`, `tls.key` and `ca.crt`). The chart fails the install if +`tls.intraCluster.enabled` is `true` but `tls.intraCluster.secretName` is +empty. Enable it on every instance that participates in encrypted intra-cluster +communication. + ## Setting up the cluster @@ -1064,9 +1134,10 @@ coordinators: The chart auto-appends `--bolt-port`, `--management-port`, `--coordinator-port`, `--coordinator-id`, `--coordinator-hostname`, `--data-directory`, `--log-level`, -`--also-log-to-stderr`, `--log-file`, `--bolt-cert-file` and `--bolt-key-file` +`--also-log-to-stderr`, `--log-file`, `--bolt-cert-file`, `--bolt-key-file`, +`--cluster-cert-file`, `--cluster-key-file` and `--cluster-ca-file` from `ports.*`, `commonArgs.{data,coordinators}.logging.*` and the per-instance -`tls.bolt.*` block. Setting any of these in `data[].args` or +`tls.bolt.*` / `tls.intraCluster.*` blocks. Setting any of these in `data[].args` or `coordinators[].args` causes `helm install` to fail with a template error. @@ -1328,15 +1399,21 @@ following parameters: | `tls.bolt.secretName` | Name of a pre-existing Kubernetes Secret holding the Bolt TLS certificate and private key. Required when `tls.bolt.enabled=true`. | `bolt-tls-secret` | | `tls.bolt.certSecretPath` | Key inside the Secret holding the TLS certificate. | `tls.crt` | | `tls.bolt.keySecretPath` | Key inside the Secret holding the TLS private key. | `tls.key` | +| `tls.intraCluster.enabled` | Enable TLS on internal cluster communication for this instance. The chart auto-appends `--cluster-cert-file` / `--cluster-key-file` / `--cluster-ca-file` and mounts the certificate Secret at `/etc/memgraph/intra_cluster_tls`. | `false` | +| `tls.intraCluster.secretName` | Name of a pre-existing Kubernetes Secret holding the intra-cluster TLS certificate, private key and CA bundle. Required when `tls.intraCluster.enabled=true`. | `intra-tls--secret` | +| `tls.intraCluster.certSecretPath` | Key inside the Secret holding the TLS certificate. | `tls.crt` | +| `tls.intraCluster.keySecretPath` | Key inside the Secret holding the TLS private key. | `tls.key` | +| `tls.intraCluster.caSecretPath` | Key inside the Secret holding the CA bundle. | `ca.crt` | | `args` | Per-instance Memgraph CLI flags. Append-only — see the note below for flags the chart manages. | `["--storage-snapshot-on-exit=false"]` for data, `[]` for coordinators | The `args` field accepts any Memgraph CLI flag **except** the following, which the chart appends automatically and rejects when set per-instance: `--bolt-port`, `--management-port`, `--coordinator-port`, `--coordinator-id`, `--coordinator-hostname`, `--data-directory`, `--log-level`, -`--also-log-to-stderr`, `--log-file`, `--bolt-cert-file` and `--bolt-key-file`. +`--also-log-to-stderr`, `--log-file`, `--bolt-cert-file`, `--bolt-key-file`, +`--cluster-cert-file`, `--cluster-key-file` and `--cluster-ca-file`. Configure those through `ports.*`, `commonArgs.{data,coordinators}.logging.*` -and the per-instance `tls.bolt.*` block instead. +and the per-instance `tls.bolt.*` / `tls.intraCluster.*` blocks instead. For all available database settings, refer to the [configuration settings docs](/database-management/configuration).