From 75e864ccbdbf234e84e2dbd4b703391644a2d79a Mon Sep 17 00:00:00 2001 From: Jianjun Liao Date: Mon, 10 Mar 2025 11:34:43 +0800 Subject: [PATCH 1/6] draft Signed-off-by: Jianjun Liao --- br/br-checkpoint-restore.md | 76 ++++++++++++++++++++++++++++++++++++- 1 file changed, 74 insertions(+), 2 deletions(-) diff --git a/br/br-checkpoint-restore.md b/br/br-checkpoint-restore.md index b94ecc47361a4..c0557eb9c8f2f 100644 --- a/br/br-checkpoint-restore.md +++ b/br/br-checkpoint-restore.md @@ -65,7 +65,11 @@ After a restore failure, avoid writing, deleting, or creating tables in the clus Cross-major-version checkpoint recovery is not recommended. For clusters where `br` recovery fails using the Long-Term Support (LTS) versions prior to v8.5.0, recovery cannot be continued with v8.5.0 or later LTS versions, and vice versa. -## Implementation details +## Implementation details: store checkpoint data in the downstream cluster + +> **Note:** +> +> Starting from v9.0.0, BR stores checkpoint data in the downstream cluster by default. You can configure the parameter `--checkpoint-storage` to make BR store checkpoint data in the external storage. Checkpoint restore operations are divided into two parts: snapshot restore and PITR restore. @@ -81,8 +85,76 @@ If the restore fails and you try to restore backup data with different checkpoin [PITR (Point-in-time recovery)](/br/br-pitr-guide.md) consists of snapshot restore and log restore phases. -During the initial restore, `br` first enters the snapshot restore phase. This phase follows the same process as the preceding [snapshot restore](#snapshot-restore-1): BR records the checkpoint data, the upstream cluster ID, and BackupTS of the backup data (that is, the start time point `start-ts` of log restore) in the `__TiDB_BR_Temporary_Snapshot_Restore_Checkpoint` database. If restore fails during this phase, you cannot adjust the `start-ts` of log restore when resuming checkpoint restore. +During the initial restore, `br` first enters the snapshot restore phase. BR records the checkpoint data, the upstream cluster ID, BackupTS of the backup data (that is, the start time point `start-ts` of log restore) and the restored time point `restored-ts` of log restore in the `__TiDB_BR_Temporary_Snapshot_Restore_Checkpoint` database. If restore fails during this phase, you cannot adjust the `start-ts` and `restored-ts` of log restore when resuming checkpoint restore. + +> **Note:** +> +> Starting from v9.0.0, because PITR restore introduces the table filter feature, the result of the snapshot restore phase in PITR restore is affected by the `restored-ts` of log restore. Therefore, if it fails in the snapshot restore phase, the next retry needs to use the same `restored-ts`. When entering the log restore phase during the initial restore, `br` creates a `__TiDB_BR_Temporary_Log_Restore_Checkpoint` database in the target cluster. This database records checkpoint data, the upstream cluster ID, and the restore time range (`start-ts` and `restored-ts`). If restore fails during this phase, you need to specify the same `start-ts` and `restored-ts` as recorded in the checkpoint database when retrying. Otherwise, `br` will report an error and prompt that the current specified restore time range or upstream cluster ID is different from the checkpoint record. If the restore cluster has been cleaned, you can manually delete the `__TiDB_BR_Temporary_Log_Restore_Checkpoint` database and retry with a different backup. Before entering the log restore phase during the initial restore, `br` constructs a mapping of upstream and downstream cluster database and table IDs at the `restored-ts` time point. This mapping is persisted in the system table `mysql.tidb_pitr_id_map` to prevent duplicate allocation of database and table IDs. Deleting data from `mysql.tidb_pitr_id_map` might lead to inconsistent PITR restore data. + +## Implementation details: store checkpoint data in the external storage + +> **Note:** +> +> Starting from v9.0.0, BR stores checkpoint data in the downstream cluster by default. You can configure the parameter `--checkpoint-storage` to make BR store checkpoint data in the external storage. +> +> For example: `./br restore full -s "s3://backup-bucket/backup-prefix" --checkpoint-storage "s3://temp-bucket/checkpoints"` + +The checkpoint data stored in the external storage has the following directory format: + +- Root path `restore-{downstream-cluster-ID}` uses the downstream cluster ID `{downstream-cluster-ID}` to distinguish different restore clusters. +- Path `restore-{downstream-cluster-ID}/log` stores log file checkpoint data during log restore phase. +- Path `restore-{downstream-cluster-ID}/sst` stores checkpoint data of the SST files that are not backed up by log backup during log restore phase. +- Path `restore-{downstream-cluster-ID}/snapshot` stores SST file checkpoint data during snapshot restore phase. + +``` +. +`-- restore-{downstream-cluster-ID} + |-- log + | |-- checkpoint.meta + | |-- data + | | |-- {uuid}.cpt + | | |-- {uuid}.cpt + | | `-- {uuid}.cpt + | |-- ingest_index.meta + | `-- progress.meta + |-- snapshot + | |-- checkpoint.meta + | |-- checksum + | | |-- {uuid}.cpt + | | |-- {uuid}.cpt + | | `-- {uuid}.cpt + | `-- data + | |-- {uuid}.cpt + | |-- {uuid}.cpt + | `-- {uuid}.cpt + `-- sst + `-- checkpoint.meta +``` + +Checkpoint restore operations are divided into two parts: snapshot restore and PITR restore. + +### Snapshot restore + +During the initial restore, `br` creates a `restore-{downstream-cluster-ID}/snapshot` path in the target cluster. The path records checkpoint data, the upstream cluster ID, and the BackupTS of the backup data. + +If the restore fails, you can retry it using the same command, and `br` will automatically read the checkpoint information from the specified external storage path and resume from the last restore point. + +If the restore fails and you try to restore backup data with different checkpoint information to the same cluster, `br` reports an error. It indicates that the current upstream cluster ID or BackupTS is different from the checkpoint record. If the restore cluster has been cleaned, you can manually clean up the checkpoint data in the external storage or specify another external storage path to store checkpoint data, and retry with a different backup. + +### PITR restore + +[PITR (Point-in-time recovery)](/br/br-pitr-guide.md) consists of snapshot restore and log restore phases. + +During the initial restore, `br` first enters the snapshot restore phase. BR records the checkpoint data, the upstream cluster ID, BackupTS of the backup data (that is, the start time point `start-ts` of log restore) and the restored time point `restored-ts` of log restore in the `restore-{downstream-cluster-ID}/snapshot` path. If restore fails during this phase, you cannot adjust the `start-ts` and `restored-ts` of log restore when resuming checkpoint restore. + +> **Note:** +> +> Starting from v9.0.0, because PITR restore introduces the table filter feature, the result of the snapshot restore phase in PITR restore is affected by the `restored-ts` of log restore. Therefore, if it fails in the snapshot restore phase, the next retry needs to use the same `restored-ts`. + +When entering the log restore phase during the initial restore, `br` creates a `restore-{downstream-cluster-ID}/log` path in the target cluster. This path records checkpoint data, the upstream cluster ID, and the restore time range (`start-ts` and `restored-ts`). If restore fails during this phase, you need to specify the same `start-ts` and `restored-ts` as recorded in the checkpoint database when retrying. Otherwise, `br` will report an error and prompt that the current specified restore time range or upstream cluster ID is different from the checkpoint record. If the restore cluster has been cleaned, you can manually clean up the checkpoint data in the external storage or specify another external storage path to store checkpoint data, and retry with a different backup. + +Before entering the log restore phase during the initial restore, `br` constructs a mapping of upstream and downstream cluster database and table IDs at the `restored-ts` time point. This mapping is persisted in the system table `mysql.tidb_pitr_id_map` to prevent duplicate allocation of database and table IDs. Deleting data from `mysql.tidb_pitr_id_map` might lead to inconsistent PITR restore data. From da875acfbcb8f41136b09e3becc434e8d216097d Mon Sep 17 00:00:00 2001 From: Jianjun Liao Date: Mon, 10 Mar 2025 14:43:13 +0800 Subject: [PATCH 2/6] commit some suggestions Signed-off-by: Jianjun Liao --- br/br-checkpoint-restore.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/br/br-checkpoint-restore.md b/br/br-checkpoint-restore.md index c0557eb9c8f2f..1b6c5b9009a96 100644 --- a/br/br-checkpoint-restore.md +++ b/br/br-checkpoint-restore.md @@ -87,10 +87,6 @@ If the restore fails and you try to restore backup data with different checkpoin During the initial restore, `br` first enters the snapshot restore phase. BR records the checkpoint data, the upstream cluster ID, BackupTS of the backup data (that is, the start time point `start-ts` of log restore) and the restored time point `restored-ts` of log restore in the `__TiDB_BR_Temporary_Snapshot_Restore_Checkpoint` database. If restore fails during this phase, you cannot adjust the `start-ts` and `restored-ts` of log restore when resuming checkpoint restore. -> **Note:** -> -> Starting from v9.0.0, because PITR restore introduces the table filter feature, the result of the snapshot restore phase in PITR restore is affected by the `restored-ts` of log restore. Therefore, if it fails in the snapshot restore phase, the next retry needs to use the same `restored-ts`. - When entering the log restore phase during the initial restore, `br` creates a `__TiDB_BR_Temporary_Log_Restore_Checkpoint` database in the target cluster. This database records checkpoint data, the upstream cluster ID, and the restore time range (`start-ts` and `restored-ts`). If restore fails during this phase, you need to specify the same `start-ts` and `restored-ts` as recorded in the checkpoint database when retrying. Otherwise, `br` will report an error and prompt that the current specified restore time range or upstream cluster ID is different from the checkpoint record. If the restore cluster has been cleaned, you can manually delete the `__TiDB_BR_Temporary_Log_Restore_Checkpoint` database and retry with a different backup. Before entering the log restore phase during the initial restore, `br` constructs a mapping of upstream and downstream cluster database and table IDs at the `restored-ts` time point. This mapping is persisted in the system table `mysql.tidb_pitr_id_map` to prevent duplicate allocation of database and table IDs. Deleting data from `mysql.tidb_pitr_id_map` might lead to inconsistent PITR restore data. @@ -151,10 +147,6 @@ If the restore fails and you try to restore backup data with different checkpoin During the initial restore, `br` first enters the snapshot restore phase. BR records the checkpoint data, the upstream cluster ID, BackupTS of the backup data (that is, the start time point `start-ts` of log restore) and the restored time point `restored-ts` of log restore in the `restore-{downstream-cluster-ID}/snapshot` path. If restore fails during this phase, you cannot adjust the `start-ts` and `restored-ts` of log restore when resuming checkpoint restore. -> **Note:** -> -> Starting from v9.0.0, because PITR restore introduces the table filter feature, the result of the snapshot restore phase in PITR restore is affected by the `restored-ts` of log restore. Therefore, if it fails in the snapshot restore phase, the next retry needs to use the same `restored-ts`. - When entering the log restore phase during the initial restore, `br` creates a `restore-{downstream-cluster-ID}/log` path in the target cluster. This path records checkpoint data, the upstream cluster ID, and the restore time range (`start-ts` and `restored-ts`). If restore fails during this phase, you need to specify the same `start-ts` and `restored-ts` as recorded in the checkpoint database when retrying. Otherwise, `br` will report an error and prompt that the current specified restore time range or upstream cluster ID is different from the checkpoint record. If the restore cluster has been cleaned, you can manually clean up the checkpoint data in the external storage or specify another external storage path to store checkpoint data, and retry with a different backup. Before entering the log restore phase during the initial restore, `br` constructs a mapping of upstream and downstream cluster database and table IDs at the `restored-ts` time point. This mapping is persisted in the system table `mysql.tidb_pitr_id_map` to prevent duplicate allocation of database and table IDs. Deleting data from `mysql.tidb_pitr_id_map` might lead to inconsistent PITR restore data. From bd75bcdab52f105cdff81c658143fb6c6855f6df Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Thu, 20 Mar 2025 10:29:43 +0800 Subject: [PATCH 3/6] Apply suggestions from code review --- br/br-checkpoint-restore.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/br/br-checkpoint-restore.md b/br/br-checkpoint-restore.md index 1b6c5b9009a96..db18d5c577ce1 100644 --- a/br/br-checkpoint-restore.md +++ b/br/br-checkpoint-restore.md @@ -99,12 +99,12 @@ Before entering the log restore phase during the initial restore, `br` construct > > For example: `./br restore full -s "s3://backup-bucket/backup-prefix" --checkpoint-storage "s3://temp-bucket/checkpoints"` -The checkpoint data stored in the external storage has the following directory format: +The checkpoint data stored in the external storage has the following directory formats: - Root path `restore-{downstream-cluster-ID}` uses the downstream cluster ID `{downstream-cluster-ID}` to distinguish different restore clusters. -- Path `restore-{downstream-cluster-ID}/log` stores log file checkpoint data during log restore phase. -- Path `restore-{downstream-cluster-ID}/sst` stores checkpoint data of the SST files that are not backed up by log backup during log restore phase. -- Path `restore-{downstream-cluster-ID}/snapshot` stores SST file checkpoint data during snapshot restore phase. +- Path `restore-{downstream-cluster-ID}/log` stores log file checkpoint data during the log restore phase. +- Path `restore-{downstream-cluster-ID}/sst` stores checkpoint data of the SST files that are not backed up by log backup during the log restore phase. +- Path `restore-{downstream-cluster-ID}/snapshot` stores checkpoint data during the snapshot restore phase. ``` . @@ -137,7 +137,7 @@ Checkpoint restore operations are divided into two parts: snapshot restore and P During the initial restore, `br` creates a `restore-{downstream-cluster-ID}/snapshot` path in the target cluster. The path records checkpoint data, the upstream cluster ID, and the BackupTS of the backup data. -If the restore fails, you can retry it using the same command, and `br` will automatically read the checkpoint information from the specified external storage path and resume from the last restore point. +If the restore fails, you can retry it using the same command. `br` will automatically read the checkpoint information from the specified external storage path and resume from the last restore point. If the restore fails and you try to restore backup data with different checkpoint information to the same cluster, `br` reports an error. It indicates that the current upstream cluster ID or BackupTS is different from the checkpoint record. If the restore cluster has been cleaned, you can manually clean up the checkpoint data in the external storage or specify another external storage path to store checkpoint data, and retry with a different backup. @@ -149,4 +149,4 @@ During the initial restore, `br` first enters the snapshot restore phase. BR rec When entering the log restore phase during the initial restore, `br` creates a `restore-{downstream-cluster-ID}/log` path in the target cluster. This path records checkpoint data, the upstream cluster ID, and the restore time range (`start-ts` and `restored-ts`). If restore fails during this phase, you need to specify the same `start-ts` and `restored-ts` as recorded in the checkpoint database when retrying. Otherwise, `br` will report an error and prompt that the current specified restore time range or upstream cluster ID is different from the checkpoint record. If the restore cluster has been cleaned, you can manually clean up the checkpoint data in the external storage or specify another external storage path to store checkpoint data, and retry with a different backup. -Before entering the log restore phase during the initial restore, `br` constructs a mapping of upstream and downstream cluster database and table IDs at the `restored-ts` time point. This mapping is persisted in the system table `mysql.tidb_pitr_id_map` to prevent duplicate allocation of database and table IDs. Deleting data from `mysql.tidb_pitr_id_map` might lead to inconsistent PITR restore data. +Before entering the log restore phase during the initial restore, `br` constructs a mapping of the database and table IDs in the upstream and downstream clusters at the `restored-ts` time point. This mapping is persisted in the system table `mysql.tidb_pitr_id_map` to prevent duplicate allocation of database and table IDs. Deleting data from `mysql.tidb_pitr_id_map` might lead to inconsistent PITR restore data. From b320612d8fcf192181bf07d745c2881ff009b4fc Mon Sep 17 00:00:00 2001 From: houfaxin Date: Tue, 8 Apr 2025 16:13:36 +0800 Subject: [PATCH 4/6] fix links --- br/br-checkpoint-restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-checkpoint-restore.md b/br/br-checkpoint-restore.md index db18d5c577ce1..0ce175be136af 100644 --- a/br/br-checkpoint-restore.md +++ b/br/br-checkpoint-restore.md @@ -15,7 +15,7 @@ If your TiDB cluster is large and cannot afford to restore again after a failure ## Implementation principles -The implementation of checkpoint restore is divided into two parts: snapshot restore and log restore. For more information, see [Implementation details](#implementation-details). +The implementation of checkpoint restore is divided into two parts: snapshot restore and log restore. For more information, see [Implementation details: store checkpoint data in the downstream cluster](#implementation-details-store-checkpoint-data-in-the-downstream-cluster) and [Implementation details: store checkpoint data in the external storage](#implementation-details-store-checkpoint-data-in-the-external-storage). ### Snapshot restore From 9592d078a98aa3d525fec28fcb92f895e678b6dd Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Thu, 10 Apr 2025 10:35:50 +0800 Subject: [PATCH 5/6] Apply suggestions from code review Co-authored-by: Aolin --- br/br-checkpoint-restore.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/br/br-checkpoint-restore.md b/br/br-checkpoint-restore.md index 0ce175be136af..8a734e98c46c7 100644 --- a/br/br-checkpoint-restore.md +++ b/br/br-checkpoint-restore.md @@ -69,7 +69,7 @@ Cross-major-version checkpoint recovery is not recommended. For clusters where ` > **Note:** > -> Starting from v9.0.0, BR stores checkpoint data in the downstream cluster by default. You can configure the parameter `--checkpoint-storage` to make BR store checkpoint data in the external storage. +> Starting from v9.0.0, BR stores checkpoint data in the downstream cluster by default. You can specify an external storage for checkpoint data using the `--checkpoint-storage` parameter. Checkpoint restore operations are divided into two parts: snapshot restore and PITR restore. @@ -95,13 +95,15 @@ Before entering the log restore phase during the initial restore, `br` construct > **Note:** > -> Starting from v9.0.0, BR stores checkpoint data in the downstream cluster by default. You can configure the parameter `--checkpoint-storage` to make BR store checkpoint data in the external storage. +> Starting from v9.0.0, BR stores checkpoint data in the downstream cluster by default. You can specify an external storage for checkpoint data using the `--checkpoint-storage` parameter. For example: > -> For example: `./br restore full -s "s3://backup-bucket/backup-prefix" --checkpoint-storage "s3://temp-bucket/checkpoints"` +> ```shell +> ./br restore full -s "s3://backup-bucket/backup-prefix" --checkpoint-storage "s3://temp-bucket/checkpoints" +> ``` -The checkpoint data stored in the external storage has the following directory formats: +In the external storage, the directory structure of the checkpoint data is as follows: -- Root path `restore-{downstream-cluster-ID}` uses the downstream cluster ID `{downstream-cluster-ID}` to distinguish different restore clusters. +- Root path `restore-{downstream-cluster-ID}` uses the downstream cluster ID `{downstream-cluster-ID}` to distinguish between different restore clusters. - Path `restore-{downstream-cluster-ID}/log` stores log file checkpoint data during the log restore phase. - Path `restore-{downstream-cluster-ID}/sst` stores checkpoint data of the SST files that are not backed up by log backup during the log restore phase. - Path `restore-{downstream-cluster-ID}/snapshot` stores checkpoint data during the snapshot restore phase. From c73e6765d910758bd01664463fa2bbdae62dc4d0 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Thu, 25 Dec 2025 14:36:36 +0800 Subject: [PATCH 6/6] Update v9.0.0 to v8.5.5 --- br/br-checkpoint-restore.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/br/br-checkpoint-restore.md b/br/br-checkpoint-restore.md index 8a734e98c46c7..6ce79ad7aee13 100644 --- a/br/br-checkpoint-restore.md +++ b/br/br-checkpoint-restore.md @@ -69,7 +69,7 @@ Cross-major-version checkpoint recovery is not recommended. For clusters where ` > **Note:** > -> Starting from v9.0.0, BR stores checkpoint data in the downstream cluster by default. You can specify an external storage for checkpoint data using the `--checkpoint-storage` parameter. +> Starting from v8.5.5, BR stores checkpoint data in the downstream cluster by default. You can specify an external storage for checkpoint data using the `--checkpoint-storage` parameter. Checkpoint restore operations are divided into two parts: snapshot restore and PITR restore. @@ -95,7 +95,7 @@ Before entering the log restore phase during the initial restore, `br` construct > **Note:** > -> Starting from v9.0.0, BR stores checkpoint data in the downstream cluster by default. You can specify an external storage for checkpoint data using the `--checkpoint-storage` parameter. For example: +> Starting from v8.5.5, BR stores checkpoint data in the downstream cluster by default. You can specify an external storage for checkpoint data using the `--checkpoint-storage` parameter. For example: > > ```shell > ./br restore full -s "s3://backup-bucket/backup-prefix" --checkpoint-storage "s3://temp-bucket/checkpoints"