From f162b5b70f79b625ccc051b4bf3744a37a8616b4 Mon Sep 17 00:00:00 2001 From: Aolin Zhang Date: Wed, 18 May 2022 10:58:30 +0800 Subject: [PATCH 1/6] add Optimized batch update of hotspot indexes --- releases/release-6.0.0-dmr.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 7d7e77ed9ca9f..71a9fc4eb149d 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -111,6 +111,12 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. [User document](/functions-and-operators/expressions-pushed-down.md#add-to-the-blocklist), [#30738](https://github.com/pingcap/tidb/issues/30738) +- Optimized batch update of hotspot indexe + + When writing data in batches to TiDB, the single index, auto-incremented by hotspot, causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. + + [User document](/functions-and-operators/tidb-functions.md#tidbshard), [#31040](https://github.com/pingcap/tidb/issues/31040) + - Support dynamic pruning mode for partitioned tables in TiFlash MPP engine (experimental) In this mode, TiDB can read and compute the data on partitioned tables using the MPP engine of TiFlash, which greatly improves the query performance of partitioned tables. From c31a7d98a884c2ef5abc51aafafca9ec96909a0d Mon Sep 17 00:00:00 2001 From: Aolin Date: Fri, 20 May 2022 15:01:01 +0800 Subject: [PATCH 2/6] Apply suggestions from code review Co-authored-by: TonsnakeLin <87681388+TonsnakeLin@users.noreply.github.com> --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 71a9fc4eb149d..4cc0f7a9ce3f8 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -111,7 +111,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. [User document](/functions-and-operators/expressions-pushed-down.md#add-to-the-blocklist), [#30738](https://github.com/pingcap/tidb/issues/30738) -- Optimized batch update of hotspot indexe +- Optimization of hotspot index When writing data in batches to TiDB, the single index, auto-incremented by hotspot, causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. From c7acb1421075ef4e677a3d9cbf1eb27ffe89fb62 Mon Sep 17 00:00:00 2001 From: Aolin Zhang Date: Mon, 23 May 2022 09:37:38 +0800 Subject: [PATCH 3/6] update Optimization of hotspot index --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 4cc0f7a9ce3f8..28d3b1c00c751 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -113,7 +113,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. - Optimization of hotspot index - When writing data in batches to TiDB, the single index, auto-incremented by hotspot, causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. + When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. [User document](/functions-and-operators/tidb-functions.md#tidbshard), [#31040](https://github.com/pingcap/tidb/issues/31040) From 1f37d0be991cb6c7c39020feb4580b36763fcd7c Mon Sep 17 00:00:00 2001 From: Aolin Date: Tue, 24 May 2022 14:20:07 +0800 Subject: [PATCH 4/6] Apply suggestions from code review Co-authored-by: cfzjywxk --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 28d3b1c00c751..922b962ea0c1f 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -113,7 +113,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. - Optimization of hotspot index - When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. + When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance fallback. [User document](/functions-and-operators/tidb-functions.md#tidbshard), [#31040](https://github.com/pingcap/tidb/issues/31040) From 0b648e53e26c1065de134c9e80031b3e596e2a1a Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 25 May 2022 16:12:46 +0800 Subject: [PATCH 5/6] Apply suggestions from code review --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 922b962ea0c1f..10fc11f9eb4b7 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -115,7 +115,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance fallback. - [User document](/functions-and-operators/tidb-functions.md#tidbshard), [#31040](https://github.com/pingcap/tidb/issues/31040) + [User document](/functions-and-operators/tidb-functions.md#tidb_shard), [#31040](https://github.com/pingcap/tidb/issues/31040) - Support dynamic pruning mode for partitioned tables in TiFlash MPP engine (experimental) From e39f18fd5d22f01a8170cf093c4b41ce8aa2d94f Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 25 May 2022 17:49:52 +0800 Subject: [PATCH 6/6] Apply suggestions from code review Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 10fc11f9eb4b7..4b2882d361c14 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -113,7 +113,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. - Optimization of hotspot index - When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance fallback. + Writing monotonically increasing data in batches to the secondary index causes an index hotspot and affects the overall write throughput. Since v6.0.0, TiDB supports scattering the index hotspot using the `tidb_shard` function to improve the write performance. Currently, `tidb_shard` only takes effect on the unique secondary index. This application-friendly solution does not require modifying the original query conditions. You can use this solution in the scenarios of high write throughput, point queries, and batch point queries. Note that using the data that has been scattered by range queries in the application might cause a performance regression. Therefore, do not use this function in such cases without verification. [User document](/functions-and-operators/tidb-functions.md#tidb_shard), [#31040](https://github.com/pingcap/tidb/issues/31040)