diff --git a/pages/database-management/upgrades.mdx b/pages/database-management/upgrades.mdx index f3c9fdd7f..137d65852 100644 --- a/pages/database-management/upgrades.mdx +++ b/pages/database-management/upgrades.mdx @@ -65,6 +65,20 @@ For a detailed, step-by-step ISSU procedure using Helm charts, see [In-Service Software Upgrade (ISSU)](/clustering/high-availability/setup-ha-cluster-k8s#in-service-software-upgrade-issu). +## Roll back an upgrade + +Memgraph recovers from durability files written by earlier versions, but files +written by a newer version aren't guaranteed to load in an older one. To go +back to the previous version, restore the backup you created before upgrading: + +1. Stop the instance. +2. Install the previous version. +3. Replace the data directory with the backup and start the instance. Changes + made after the backup was taken are lost. + +For HA clusters, see +[Rollback](/clustering/high-availability/setup-ha-cluster-k8s#rollback). + ## [Version-specific guides](/database-management/upgrades/specific-versions) Certain Memgraph releases introduce breaking changes or require additional diff --git a/pages/deployment/environments/docker.mdx b/pages/deployment/environments/docker.mdx index fb06f2fd3..8c6849f45 100644 --- a/pages/deployment/environments/docker.mdx +++ b/pages/deployment/environments/docker.mdx @@ -294,9 +294,8 @@ periodic snapshot creation and create snapshots manually when you need to. ### Dump database -Another approach is to dump the database into a file. That is useful if you want -to move the data between different Memgraph versions that might have -incompatible data formats are used for snapshots or WAL files. +Another approach is to dump the database into a file of Cypher queries, which +you can run on another Memgraph instance to recreate the data. Here are the steps to dump the database with Memgraph running in a Docker container: @@ -329,10 +328,11 @@ import](/data-migration/best-practices#cypher-queries-best-practices). ### Upgrading Memgraph -Between different versions of Memgraph, we might break the -compatibility of the configurations, snapshots, and WAL files. That is done very -rarely, but it is helpful to keep an eye on the [release notes](/release-notes) -to see if there are any breaking changes. +Memgraph recovers from snapshot and WAL files written by earlier versions, so +the new version starts on your existing data directory. Follow the [upgrade +guide](/database-management/upgrades) and check the [release +notes](/release-notes) for breaking changes, such as removed configuration +flags. ## Set up a cluster diff --git a/pages/deployment/environments/linux.mdx b/pages/deployment/environments/linux.mdx index 5453d8e60..058219b19 100644 --- a/pages/deployment/environments/linux.mdx +++ b/pages/deployment/environments/linux.mdx @@ -198,9 +198,8 @@ UNLOCK DATA DIRECTORY; ### Dump database -Another approach is to dump the database into a file. This is useful if you want -to move the data between different Memgraph versions that might have -incompatible data formats are used for snapshots or wal files. +Another approach is to dump the database into a file of Cypher queries, which +you can run on another Memgraph instance to recreate the data. To dump the database, you can use the `mgconsole` CLI tool. Here is an example of how to dump the database into a file: @@ -237,11 +236,11 @@ import](/data-migration/best-practices#cypher-queries-best-practices). ### Upgrading Memgraph -Keep in mind that between different versions of Memgraph, we might break the -compatibility of the configurations, snapshots, and WAL files. This is done very -rarely, but it is useful to keep an eye on the [release -notes](/release-notes) to see if there are any breaking -changes. +Memgraph recovers from snapshot and WAL files written by earlier versions, so +the new version starts on your existing data directory. Follow the [upgrade +guide](/database-management/upgrades) and check the [release +notes](/release-notes) for breaking changes, such as removed configuration +flags. ## Set up a cluster diff --git a/pages/fundamentals/data-durability.mdx b/pages/fundamentals/data-durability.mdx index 848592a86..1c115d76c 100644 --- a/pages/fundamentals/data-durability.mdx +++ b/pages/fundamentals/data-durability.mdx @@ -1,6 +1,6 @@ --- title: Data durability -description: Learn about data durability - an important aspect of making sure you keep your data secure, regardless of Memgraph's in-memory storage. +description: Configure snapshots and write-ahead logging (WAL), see how Memgraph recovers on startup, and handle recovery failures. --- import { Callout } from 'nextra/components' @@ -84,12 +84,13 @@ By default, WAL files are located at `/var/lib/memgraph/wal`.

WAL file lifecycle

-**Older WAL files are deleted automatically after a snapshot is created** since -the snapshot contains the full database state up to that point. Only WAL files -containing changes after the latest snapshot are retained. +Memgraph deletes older WAL files when it deletes old snapshots, not after every +snapshot. Once a new snapshot brings the number of snapshots to +`--storage-snapshot-retention-count`, Memgraph deletes the WAL files older than +the oldest snapshot it keeps, except the last WAL file written before that +snapshot. -To control WAL file cleanup indirectly, you can limit the number of snapshots -via `--storage-snapshot-retention-count`. +To keep more snapshots, raise `--storage-snapshot-retention-count`. **It is not possible to use WAL files exclusively** without snapshots. Memgraph enforces periodic snapshots when WAL is enabled and will fail to start if WAL is @@ -194,7 +195,7 @@ snapshot creation behavior by [defining the relevant flags](/database-management/configuration#storage). Alternatively, you can make one directly by running the following query: -```opencypher +```cypher CREATE SNAPSHOT; ``` @@ -209,7 +210,7 @@ directory. The `CREATE SNAPSHOT` query will return the path of the newly created snapshot file. To query which snapshots currently exist in the data directory, execute: -```opencypher +```cypher SHOW SNAPSHOTS; ``` @@ -230,14 +231,26 @@ files store incremental changes and need to be replayed sequentially. `IN_MEMORY_TRANSACTIONAL` mode supports periodic snapshot creation. The interval can be set at startup via the `--storage-snapshot-interval` flag or at run-time -via the database settings: +via the database settings. To create a snapshot every 1200 seconds: -```opencypher +```cypher SET DATABASE SETTING "storage.snapshot.interval" TO "1200"; -SET DATABASE SETTING "storage.snapshot.interval" TO "* * 12 * * *"; +``` + +To pause periodic snapshots, set the interval to an empty string: + +```cypher SET DATABASE SETTING "storage.snapshot.interval" TO ""; ``` +With [Memgraph Enterprise](/database-management/enabling-memgraph-enterprise), +you can set the interval as a CRON expression instead. To create a snapshot +every day at 12:00:00: + +```cypher +SET DATABASE SETTING "storage.snapshot.interval" TO "0 0 12 * * *"; +``` + Changing the configuration settings depends on the way you are using Memgraph, so please refer to the [configuration @@ -250,9 +263,7 @@ in seconds. Interval can also be defined as a 6-field CRON expression (seconds, hour, day of month, month, day of week). Standard 5-field cron references (e.g. [crontab.guru](https://crontab.guru/)) describe the last five fields; Memgraph adds seconds as the first field. By setting the value to an empty string, the background process is -paused and any currently active snapshot creation will finish. Please note that -defining the interval via a CRON expression is an [Enterprise -feature](/database-management/enabling-memgraph-enterprise). +paused and any currently active snapshot creation will finish. If the database is started in or migrated into `IN_MEMORY_ANALYTICAL` mode, the background thread will pause and no snapshots will be created as long as that @@ -265,9 +276,15 @@ new writes have been committed since the last snapshot. If the periodic snapshot is skipped it will be logged on INFO level. - -Snapshots and WAL files are presently not compatible between Memgraph versions. - +### Compatibility between versions + +Memgraph recovers from snapshot and WAL files written by Memgraph 2.0 or any +later version, so after an [upgrade](/database-management/upgrades) you can +start the new version on your existing data directory. + +Files written by a newer version aren't guaranteed to load in an older one. To +go back to an earlier version, [restore the backup you took before +upgrading](/database-management/upgrades#roll-back-an-upgrade). ### Parallelized execution @@ -454,6 +471,9 @@ deletes them when [`--storage-backup-dir-enabled`](/database-management/configur is off), leaving a clean single-snapshot directory. As a result, the database **recovers cleanly on the next restart and does not re-enter the broken state**. +The database then holds only the data in that snapshot. Changes made after the +snapshot was taken, including those in a damaged WAL file, are lost. + `RECOVER SNAPSHOT` is available in both Community and Enterprise editions, so single-tenant deployments can recover a corrupt default database. diff --git a/pages/help-center/errors/durability.mdx b/pages/help-center/errors/durability.mdx index d28291bb2..8a34112e5 100644 --- a/pages/help-center/errors/durability.mdx +++ b/pages/help-center/errors/durability.mdx @@ -1,6 +1,6 @@ --- title: Durability errors -description: Discover how to maintain the integrity of your graph database by ensuring durability and data reliability. Explore the docs to get to know how to optimize durability settings. +description: Fix durability errors, from missing snapshot or WAL files and a wrong data directory to recovery failures caused by damaged files. --- import { Callout } from 'nextra/components' @@ -48,4 +48,37 @@ be restarted. The default directory is `/var/lib/memgraph`. To learn about all the configuration options, check out the [reference guide](/database-management/configuration). +## Why doesn't Memgraph start after a recovery failure? + +On startup, Memgraph recovers each database from its snapshot and WAL files. If +a snapshot or WAL file it needs for recovery can't be loaded, usually because a +disk issue damaged it, recovery stops and Memgraph doesn't start. WAL files +that the newest snapshot already covers aren't read, so damage in them doesn't +stop recovery. + +To start the instance and recover the database: + +1. Restart Memgraph with `--storage-allow-recovery-failure=true`. The database + that failed comes up empty, in a broken state, with its files left as they + were. Other databases on the instance work normally. +2. Switch to the broken database and load a good snapshot from its + `snapshots` directory: + + ```cypher template + USE DATABASE DATABASE_NAME; + RECOVER SNAPSHOT "SNAPSHOT_PATH"; + ``` + + `DATABASE_NAME` is the database that failed (`memgraph` by default), and + `SNAPSHOT_PATH` is the full path to the snapshot file. In Memgraph + Community, skip `USE DATABASE`. + +The database then holds only the data in that snapshot. Changes made after the +snapshot was taken, including those in the damaged WAL file, are lost. If you +have a backup of the whole data directory, you can restore it instead. + +The `--storage-allow-recovery-failure` flag works only with in-memory storage +modes. For details, see [recovery failure +handling](/fundamentals/data-durability#recovery-failure-handling). + \ No newline at end of file diff --git a/pages/help-center/errors/snapshots.mdx b/pages/help-center/errors/snapshots.mdx index ebe9a6cdb..db055eb60 100644 --- a/pages/help-center/errors/snapshots.mdx +++ b/pages/help-center/errors/snapshots.mdx @@ -25,9 +25,18 @@ persistence. Memgraph will try to load the newest snapshot file on startup. ## What to do with corrupt snapshots? -Because snapshots are read-only, any modifications will result in corrupt -files that won't be loaded. The solution is to delete the snapshot files and to -start Memgraph again. +If Memgraph can't load a snapshot it needs for recovery, it doesn't start. You +have two options: + +- **Delete the corrupt snapshot file** and start Memgraph again. Memgraph + recovers from the newest remaining snapshot and the WAL files written after + it. If the corrupt file is the only snapshot, deleting it loses the data it + holds. +- **Load a good snapshot.** Start Memgraph with + `--storage-allow-recovery-failure=true`, then load a good snapshot with + `RECOVER SNAPSHOT`. Changes made after that snapshot was taken are lost. See + [recovery failure + handling](/fundamentals/data-durability#recovery-failure-handling). ## Why is data lost when Memgraph is restarted?