From f87242c7ee85828b795636b3fe01813a4c5e12f1 Mon Sep 17 00:00:00 2001 From: Aolin Zhang Date: Wed, 18 May 2022 10:58:30 +0800 Subject: [PATCH 1/6] add Optimized batch update of hotspot indexes --- releases/release-6.0.0-dmr.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 7d7e77ed9ca9f..71a9fc4eb149d 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -111,6 +111,12 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. [User document](/functions-and-operators/expressions-pushed-down.md#add-to-the-blocklist), [#30738](https://github.com/pingcap/tidb/issues/30738) +- Optimized batch update of hotspot indexe + + When writing data in batches to TiDB, the single index, auto-incremented by hotspot, causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. + + [User document](/functions-and-operators/tidb-functions.md#tidbshard), [#31040](https://github.com/pingcap/tidb/issues/31040) + - Support dynamic pruning mode for partitioned tables in TiFlash MPP engine (experimental) In this mode, TiDB can read and compute the data on partitioned tables using the MPP engine of TiFlash, which greatly improves the query performance of partitioned tables. From fa57436438b66824683d8b1a80cdfd50b1f583ea Mon Sep 17 00:00:00 2001 From: Aolin Date: Fri, 20 May 2022 15:01:01 +0800 Subject: [PATCH 2/6] Apply suggestions from code review Co-authored-by: TonsnakeLin <87681388+TonsnakeLin@users.noreply.github.com> --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 71a9fc4eb149d..4cc0f7a9ce3f8 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -111,7 +111,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. [User document](/functions-and-operators/expressions-pushed-down.md#add-to-the-blocklist), [#30738](https://github.com/pingcap/tidb/issues/30738) -- Optimized batch update of hotspot indexe +- Optimization of hotspot index When writing data in batches to TiDB, the single index, auto-incremented by hotspot, causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. From b089981b53c5206f84705003f21bea17b8d48722 Mon Sep 17 00:00:00 2001 From: Aolin Zhang Date: Mon, 23 May 2022 09:37:38 +0800 Subject: [PATCH 3/6] update Optimization of hotspot index --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 4cc0f7a9ce3f8..28d3b1c00c751 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -113,7 +113,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. - Optimization of hotspot index - When writing data in batches to TiDB, the single index, auto-incremented by hotspot, causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. + When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. [User document](/functions-and-operators/tidb-functions.md#tidbshard), [#31040](https://github.com/pingcap/tidb/issues/31040) From 666395e75eb2b1079688aba8a44da22db37d0538 Mon Sep 17 00:00:00 2001 From: Aolin Date: Tue, 24 May 2022 14:20:07 +0800 Subject: [PATCH 4/6] Apply suggestions from code review Co-authored-by: cfzjywxk --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 28d3b1c00c751..922b962ea0c1f 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -113,7 +113,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. - Optimization of hotspot index - When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance rollback. + When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance fallback. [User document](/functions-and-operators/tidb-functions.md#tidbshard), [#31040](https://github.com/pingcap/tidb/issues/31040) From 7db3c802f178931d76aa93a3e7ccd9be7524db12 Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 25 May 2022 16:12:46 +0800 Subject: [PATCH 5/6] Apply suggestions from code review --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 922b962ea0c1f..10fc11f9eb4b7 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -115,7 +115,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance fallback. - [User document](/functions-and-operators/tidb-functions.md#tidbshard), [#31040](https://github.com/pingcap/tidb/issues/31040) + [User document](/functions-and-operators/tidb-functions.md#tidb_shard), [#31040](https://github.com/pingcap/tidb/issues/31040) - Support dynamic pruning mode for partitioned tables in TiFlash MPP engine (experimental) From 4ad84118f083b5016684627c7b61920c93161648 Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 25 May 2022 17:49:52 +0800 Subject: [PATCH 6/6] Apply suggestions from code review Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- releases/release-6.0.0-dmr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.0.0-dmr.md b/releases/release-6.0.0-dmr.md index 10fc11f9eb4b7..4b2882d361c14 100644 --- a/releases/release-6.0.0-dmr.md +++ b/releases/release-6.0.0-dmr.md @@ -113,7 +113,7 @@ TiDB v6.0.0 is a DMR, and its version is 6.0.0-DMR. - Optimization of hotspot index - When writing monotonically increasing data in batches to the secondary index, it makes a hotspot index and causes a performance bottleneck easily, affecting the overall write throughput. In v6.0.0, TiDB supports sharding the index to different nodes through the `tidb_shard` function, which improves the write performance. Currently, `tidb_shard` only affects the secondary unique index. This is an application-friendly solution and does not require modifying the original query conditions. You can use the solution under the scenarios of high write throughput, point queries, and batch point queries. Note that you need to use this feature after verifying it because using the data that is sharded by range query might cause performance fallback. + Writing monotonically increasing data in batches to the secondary index causes an index hotspot and affects the overall write throughput. Since v6.0.0, TiDB supports scattering the index hotspot using the `tidb_shard` function to improve the write performance. Currently, `tidb_shard` only takes effect on the unique secondary index. This application-friendly solution does not require modifying the original query conditions. You can use this solution in the scenarios of high write throughput, point queries, and batch point queries. Note that using the data that has been scattered by range queries in the application might cause a performance regression. Therefore, do not use this function in such cases without verification. [User document](/functions-and-operators/tidb-functions.md#tidb_shard), [#31040](https://github.com/pingcap/tidb/issues/31040)