Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,8 @@ That means, you cannot run any data queries on the coordinator directly (we
will talk more about routing data queries in the next sections). However,
system information queries such as `SHOW CONFIG`, `SHOW LICENSE INFO`,
`SHOW BUILD INFO` and `SHOW STORAGE INFO` are supported on coordinators, as
well as `SET DATABASE SETTING` and `RELOAD SSL`.
well as `SET DATABASE SETTING`, `RELOAD BOLT_SERVER TLS` and
`RELOAD INTRA_CLUSTER TLS`.
</Callout>

Since coordinators do not store user data, the following restrictions apply:
Expand Down Expand Up @@ -272,6 +273,66 @@ in the cluster to ensure high availability, with timeouts.
| `WalFilesRpc` | Main | Replica | proportional |
| `CurrentWalRpc` | Main | Replica | proportional |

## Intra-cluster TLS

By default, the communication between instances in a high-availability cluster
is unencrypted. To secure it, Memgraph supports **intra-cluster TLS**, which
encrypts all internal cluster traffic using mutual TLS (mTLS). When enabled,
TLS protects the communication on:

- the **management server** (health checks between the leader coordinator and
the instances),
- the **replication server** (data replication between MAIN and REPLICA
instances), and
- the **coordinator server** (synchronization and log replication between
coordinators).

<Callout type="info">
Intra-cluster TLS is independent of [Bolt SSL/TLS](/database-management/ssl-encryption).
Bolt encryption secures client-to-instance connections and is configured
separately with the `--bolt-cert-file` and `--bolt-key-file` flags, while
intra-cluster TLS secures the internal cluster communication described above.
</Callout>

### Enabling intra-cluster TLS

Intra-cluster TLS is enabled by setting the following three flags on every
instance (coordinators and data instances) in the cluster:

| Flag | Description |
| --------------------- | ---------------------------------------------------------------------------------------------------- |
| `--cluster-cert-file` | Certificate file used for intra-cluster TLS communication. |
| `--cluster-key-file` | Key file used for intra-cluster TLS communication. |
| `--cluster-ca-file` | File storing the certificate of the Certificate Authority you trust for intra-cluster TLS communication. |

All three flags must be set together. If only some of them are provided, the
instance refuses to start to avoid running in a partially-configured TLS state.
When all three are empty, intra-cluster TLS is disabled and communication is
unencrypted. Because mTLS is used, every instance must present a certificate
that is trusted by the configured Certificate Authority, and all instances in
the cluster must be started with the TLS flags.

### Reloading intra-cluster TLS certificates at runtime

You can rotate the intra-cluster TLS certificates without restarting the
cluster by running the `RELOAD INTRA_CLUSTER TLS` Cypher query. Replace the
certificate and key files on disk at the paths configured with
`--cluster-cert-file`, `--cluster-key-file`, and `--cluster-ca-file`, then run:

```cypher
RELOAD INTRA_CLUSTER TLS;
```

`RELOAD INTRA_CLUSTER TLS` reloads both the client and server connections, so
new connections will start using the new certificates.

This is the intra-cluster counterpart of `RELOAD BOLT_SERVER TLS`, which
reloads the Bolt server certificates. Both queries require the `RELOAD_TLS`
[privilege](/database-management/authentication-and-authorization/role-based-access-control).
For more details on reloading certificates, see the
[SSL encryption](/database-management/ssl-encryption#reload-ssl-certificates-at-runtime)
page.

## Automatic failover

Automatic failover is driven by periodic health checks performed by the leader
Expand Down
85 changes: 81 additions & 4 deletions pages/clustering/high-availability/setup-ha-cluster-k8s.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -527,6 +527,76 @@ When a coordinator has `tls.bolt.enabled: true`, the cluster-setup job
that registers coordinators and data instances automatically uses
`--use-ssl` when connecting to coordinator 1.

### Intra-cluster SSL/TLS

Independently of Bolt TLS, each data instance and coordinator can encrypt the
internal communication between cluster members (coordinator-to-coordinator and
coordinator-to-data traffic). When `tls.intraCluster.enabled` is `true`, the
chart mounts a pre-existing Kubernetes Secret containing the certificate,
private key and CA bundle at `/etc/memgraph/intra_cluster_tls/` and
auto-appends `--cluster-cert-file=/etc/memgraph/intra_cluster_tls/tls.crt`,
`--cluster-key-file=/etc/memgraph/intra_cluster_tls/tls.key` and
`--cluster-ca-file=/etc/memgraph/intra_cluster_tls/ca.crt` to the instance's
args.

To enable intra-cluster TLS, first create a Kubernetes Secret holding the
certificate, private key and CA bundle for each instance in the release
namespace. For example, for `data-0`:

```bash
kubectl create secret generic intra-tls-data-0-secret \
--from-file=tls.crt=path/to/tls.crt \
--from-file=tls.key=path/to/tls.key \
--from-file=ca.crt=path/to/ca.crt
```

Then enable `tls.intraCluster` on each instance that should encrypt internal
traffic:

```yaml
data:
- id: "0"
tls:
intraCluster:
enabled: true
secretName: intra-tls-data-0-secret
certSecretPath: tls.crt
keySecretPath: tls.key
caSecretPath: ca.crt
- id: "1"
tls:
intraCluster:
enabled: true
secretName: intra-tls-data-1-secret
certSecretPath: tls.crt
keySecretPath: tls.key
caSecretPath: ca.crt

coordinators:
- id: "1"
tls:
intraCluster:
enabled: true
secretName: intra-tls-coord-1-secret
- id: "2"
tls:
intraCluster:
enabled: true
secretName: intra-tls-coord-2-secret
- id: "3"
tls:
intraCluster:
enabled: true
secretName: intra-tls-coord-3-secret
```

`certSecretPath`, `keySecretPath` and `caSecretPath` are the keys inside the
Secret holding the certificate, private key and CA bundle respectively
(default `tls.crt`, `tls.key` and `ca.crt`). The chart fails the install if
`tls.intraCluster.enabled` is `true` but `tls.intraCluster.secretName` is
empty. Enable it on every instance that participates in encrypted intra-cluster
communication.


## Setting up the cluster

Expand Down Expand Up @@ -1064,9 +1134,10 @@ coordinators:
<Callout type="info">
The chart auto-appends `--bolt-port`, `--management-port`, `--coordinator-port`,
`--coordinator-id`, `--coordinator-hostname`, `--data-directory`, `--log-level`,
`--also-log-to-stderr`, `--log-file`, `--bolt-cert-file` and `--bolt-key-file`
`--also-log-to-stderr`, `--log-file`, `--bolt-cert-file`, `--bolt-key-file`,
`--cluster-cert-file`, `--cluster-key-file` and `--cluster-ca-file`
from `ports.*`, `commonArgs.{data,coordinators}.logging.*` and the per-instance
`tls.bolt.*` block. Setting any of these in `data[].args` or
`tls.bolt.*` / `tls.intraCluster.*` blocks. Setting any of these in `data[].args` or
`coordinators[].args` causes `helm install` to fail with a template error.
</Callout>

Expand Down Expand Up @@ -1328,15 +1399,21 @@ following parameters:
| `tls.bolt.secretName` | Name of a pre-existing Kubernetes Secret holding the Bolt TLS certificate and private key. Required when `tls.bolt.enabled=true`. | `bolt-tls-secret` |
| `tls.bolt.certSecretPath` | Key inside the Secret holding the TLS certificate. | `tls.crt` |
| `tls.bolt.keySecretPath` | Key inside the Secret holding the TLS private key. | `tls.key` |
| `tls.intraCluster.enabled` | Enable TLS on internal cluster communication for this instance. The chart auto-appends `--cluster-cert-file` / `--cluster-key-file` / `--cluster-ca-file` and mounts the certificate Secret at `/etc/memgraph/intra_cluster_tls`. | `false` |
| `tls.intraCluster.secretName` | Name of a pre-existing Kubernetes Secret holding the intra-cluster TLS certificate, private key and CA bundle. Required when `tls.intraCluster.enabled=true`. | `intra-tls-<instance>-secret` |
| `tls.intraCluster.certSecretPath` | Key inside the Secret holding the TLS certificate. | `tls.crt` |
| `tls.intraCluster.keySecretPath` | Key inside the Secret holding the TLS private key. | `tls.key` |
| `tls.intraCluster.caSecretPath` | Key inside the Secret holding the CA bundle. | `ca.crt` |
| `args` | Per-instance Memgraph CLI flags. Append-only — see the note below for flags the chart manages. | `["--storage-snapshot-on-exit=false"]` for data, `[]` for coordinators |

The `args` field accepts any Memgraph CLI flag **except** the following, which
the chart appends automatically and rejects when set per-instance:
`--bolt-port`, `--management-port`, `--coordinator-port`, `--coordinator-id`,
`--coordinator-hostname`, `--data-directory`, `--log-level`,
`--also-log-to-stderr`, `--log-file`, `--bolt-cert-file` and `--bolt-key-file`.
`--also-log-to-stderr`, `--log-file`, `--bolt-cert-file`, `--bolt-key-file`,
`--cluster-cert-file`, `--cluster-key-file` and `--cluster-ca-file`.
Configure those through `ports.*`, `commonArgs.{data,coordinators}.logging.*`
and the per-instance `tls.bolt.*` block instead.
and the per-instance `tls.bolt.*` / `tls.intraCluster.*` blocks instead.

For all available database settings, refer to the [configuration settings
docs](/database-management/configuration).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,8 @@ Memgraph's privilege system controls access to various database operations throu
| `SHOW VERSION` | `STATS` | `SHOW VERSION` |
| `SHOW TRANSACTIONS` | `TRANSACTION_MANAGEMENT` | `SHOW TRANSACTIONS` |
| `TERMINATE TRANSACTIONS` | `TRANSACTION_MANAGEMENT` | `TERMINATE TRANSACTIONS 'transaction_id'` |
| `RELOAD BOLT_SERVER TLS` | `RELOAD_TLS` | `RELOAD BOLT_SERVER TLS` |
| `RELOAD INTRA_CLUSTER TLS` | `RELOAD_TLS` | `RELOAD INTRA_CLUSTER TLS` |

## Replication operations

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,7 @@ of the following commands:
| Privilege to set limits and monitor resource usage per user (via [user profiles](/database-management/authentication-and-authorization/user-profiles)) or per database (via [tenant profiles](/database-management/tenant-profiles)). | `PROFILE_RESTRICTION` |
| Privilege to manage [server-side parameters](/database-management/server-side-parameters) (`SET`, `UNSET`, `SHOW`). | `SERVER_SIDE_PARAMETERS` |
| Privilege to manage [server-side descriptions](/database-management/server-side-descriptions) (`SET`, `DELETE`, `SHOW`). | `SERVER_SIDE_DESCRIPTIONS` |
| Privilege to reload [SSL/TLS certificates](/database-management/ssl-encryption#reload-ssl-certificates-at-runtime) at runtime (`RELOAD BOLT_SERVER TLS`, `RELOAD INTRA_CLUSTER TLS`). | `RELOAD_TLS` |
| Privileges to specific labels. | `ALL LABELS` |
| Privileges to specific relationships types. | `ALL EDGE TYPES` |

Expand Down
3 changes: 3 additions & 0 deletions pages/database-management/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -437,6 +437,9 @@ This section contains the list of flags that are used to configure highly availa
| ~~`--instance-down-timeout-sec`~~ | **Deprecated in 3.10.** This flag is ignored. Use `SET COORDINATOR SETTING 'instance_down_timeout_sec' TO '<value>'` instead. See [Coordinator runtime settings](/clustering/high-availability/best-practices#coordinator-runtime-settings). | `[uint32]` |
| `--nuraft-log-file` | Path to the file where NuRaft logs are saved. | `[string]` |
| `--coordinator-hostname` | Coordinator's instance hostname. Used only in `SHOW INSTANCES` query. | `[string]` |
| `--cluster-cert-file` | Certificate file used for [intra-cluster TLS](/clustering/high-availability/how-high-availability-works#intra-cluster-tls) communication. Must be set together with `--cluster-key-file` and `--cluster-ca-file`. | `[string]` |
| `--cluster-key-file` | Key file used for [intra-cluster TLS](/clustering/high-availability/how-high-availability-works#intra-cluster-tls) communication. Must be set together with `--cluster-cert-file` and `--cluster-ca-file`. | `[string]` |
| `--cluster-ca-file` | File storing the certificate of the Certificate Authority you trust for [intra-cluster TLS](/clustering/high-availability/how-high-availability-works#intra-cluster-tls) communication. Must be set together with `--cluster-cert-file` and `--cluster-key-file`. | `[string]` |


### Query
Expand Down
28 changes: 28 additions & 0 deletions pages/database-management/ssl-encryption.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,34 @@ Running `RELOAD BOLT_SERVER TLS` on a Memgraph instance that was started
without SSL enabled will return an error.
</Callout>

<Callout type="warning">
**Breaking change in Memgraph 3.11:** `RELOAD BOLT_SERVER TLS` now requires the
`RELOAD_TLS`
[privilege](/database-management/authentication-and-authorization/role-based-access-control).
Previously no privilege was needed. If you use authorization, grant `RELOAD_TLS`
to the users or roles that perform certificate rotation:
`GRANT RELOAD_TLS TO user;`.
</Callout>

### Reload intra-cluster TLS certificates

In a [high-availability](/clustering/high-availability) cluster, the internal
communication between instances can be secured with
[intra-cluster TLS](/clustering/high-availability/how-high-availability-works#intra-cluster-tls).
To rotate those certificates at runtime, replace the certificate and key files
on disk (at the paths configured with `--cluster-cert-file`,
`--cluster-key-file`, and `--cluster-ca-file`) and run:

```cypher
RELOAD INTRA_CLUSTER TLS;
```

`RELOAD INTRA_CLUSTER TLS` reloads both the client and server connections, so
new connections will start using the new certificates.

Like `RELOAD BOLT_SERVER TLS`, this command requires the `RELOAD_TLS` privilege
and cannot be executed inside an explicit (multi-command) transaction.

## How to set up SSL encryption

Memgraph uses SSL (Secure Sockets Layer) protocol for establishing an
Expand Down
10 changes: 10 additions & 0 deletions pages/release-notes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,11 @@ guide.
you wrote custom procedures that relied on the `transaction_id` parameter
passed by the `cross_database` module, switch to `graph.start_timestamp`.
[#4167](https://github.com/memgraph/memgraph/pull/4167)
- `RELOAD BOLT_SERVER TLS` now requires the `RELOAD_TLS` privilege; previously it
required no privilege. If you use authorization, grant `RELOAD_TLS` to the
users or roles that rotate certificates (`GRANT RELOAD_TLS TO user;`),
otherwise the command will fail with an authorization error.
[#4154](https://github.com/memgraph/memgraph/pull/4154)

{<h4 className="custom-header">✨ New features</h4>}

Expand Down Expand Up @@ -115,6 +120,11 @@ guide.
communication on the management, replication, and coordinator servers. Bolt
TLS remains independently configured.
[#4140](https://github.com/memgraph/memgraph/pull/4140)
- Added the `RELOAD INTRA_CLUSTER TLS` Cypher query to rotate intra-cluster TLS
certificates at runtime without restarting the cluster. The new `RELOAD_TLS`
privilege now governs both `RELOAD INTRA_CLUSTER TLS` and
`RELOAD BOLT_SERVER TLS`.
[#4154](https://github.com/memgraph/memgraph/pull/4154)
- `SHOW TRANSACTIONS` now includes `start_time` (UTC timestamp) and
`elapsed_ms` columns, making it easier to identify long-running transactions
without calculating elapsed time manually.
Expand Down