From 26cdb6878608d00474c5f7519d00df9d917cafe8 Mon Sep 17 00:00:00 2001 From: Aolin Zhang Date: Wed, 18 May 2022 10:58:30 +0800 Subject: [PATCH 1/6] add Optimized batch update of hotspot indexes --- releases/release-6.0.0-dmr.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 7d7e77ed9ca9f..71a9fc4eb149d 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -111,6 +111,12 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. [User document](/functions-and-operators/expressions-pushed-down.md#add-to-the-blocklist), [#30738](https://github.com/pingcap/tidb/issues/30738) +- Optimized batch update of hotspot indexe + + When writing data in batches to TiDB, the single index, auto-incremented by hotspot, causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. + + [User document](/functions-and-operators/tidb-functions.md#tidbshard), [#31040](https://github.com/pingcap/tidb/issues/31040) + - Support dynamic pruning mode for partitioned tables in TiFlash MPP engine (experimental) In this mode, TiDB can read and compute the data on partitioned tables using the MPP engine of TiFlash, which greatly improves the query performance of partitioned tables. From c0df39cc9c65741c2811320df7ebb4403bfcb569 Mon Sep 17 00:00:00 2001 From: Aolin Date: Fri, 20 May 2022 15:01:01 +0800 Subject: [PATCH 2/6] Apply suggestions from code review Co-authored-by: TonsnakeLin <87681388+TonsnakeLin@users.noreply.github.com> --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 71a9fc4eb149d..4cc0f7a9ce3f8 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -111,7 +111,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. [User document](/functions-and-operators/expressions-pushed-down.md#add-to-the-blocklist), [#30738](https://github.com/pingcap/tidb/issues/30738) -- Optimized batch update of hotspot indexe +- Optimization of hotspot index When writing data in batches to TiDB, the single index, auto-incremented by hotspot, causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. From 629e01c2b53cb2b3ac6660030fe26cc43a8223ce Mon Sep 17 00:00:00 2001 From: Aolin Zhang Date: Mon, 23 May 2022 09:37:38 +0800 Subject: [PATCH 3/6] update Optimization of hotspot index --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 4cc0f7a9ce3f8..28d3b1c00c751 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -113,7 +113,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. - Optimization of hotspot index - When writing data in batches to TiDB, the single index, auto-incremented by hotspot, causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. + When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. [User document](/functions-and-operators/tidb-functions.md#tidbshard), [#31040](https://github.com/pingcap/tidb/issues/31040) From 83c53d9de6721566668ffdec03eb5f3aaa2d8790 Mon Sep 17 00:00:00 2001 From: Aolin Date: Tue, 24 May 2022 14:20:07 +0800 Subject: [PATCH 4/6] Apply suggestions from code review Co-authored-by: cfzjywxk --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 28d3b1c00c751..922b962ea0c1f 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -113,7 +113,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. - Optimization of hotspot index - When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. + When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance fallback. [User document](/functions-and-operators/tidb-functions.md#tidbshard), [#31040](https://github.com/pingcap/tidb/issues/31040) From 5065d23b740d9a25b4d077d541ce2491379401e0 Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 25 May 2022 16:12:46 +0800 Subject: [PATCH 5/6] Apply suggestions from code review --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 922b962ea0c1f..10fc11f9eb4b7 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -115,7 +115,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance fallback. - [User document](/functions-and-operators/tidb-functions.md#tidbshard), [#31040](https://github.com/pingcap/tidb/issues/31040) + [User document](/functions-and-operators/tidb-functions.md#tidb_shard), [#31040](https://github.com/pingcap/tidb/issues/31040) - Support dynamic pruning mode for partitioned tables in TiFlash MPP engine (experimental) From e075e62f74255d3b36be15a2cb55b469ecc3317c Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 25 May 2022 17:49:52 +0800 Subject: [PATCH 6/6] Apply suggestions from code review Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 10fc11f9eb4b7..4b2882d361c14 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -113,7 +113,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. - Optimization of hotspot index - When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance fallback. + Writing monotonically increasing data in batches to the secondary index causes an index hotspot and affects the overall write throughput. Since v6.0.0, TiDB supports scattering the index hotspot using the `tidb_shard` function to improve the write performance. Currently, `tidb_shard` only takes effect on the unique secondary index. This application-friendly solution does not require modifying the original query conditions. You can use this solution in the scenarios of high write throughput, point queries, and batch point queries. Note that using the data that has been scattered by range queries in the application might cause a performance regression. Therefore, do not use this function in such cases without verification. [User document](/functions-and-operators/tidb-functions.md#tidb_shard), [#31040](https://github.com/pingcap/tidb/issues/31040)