Skip to content

planner,expression,util: scatter hotspot index in certain scenes by expression index #30655

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
TonsnakeLin wants to merge 15 commits into from
Closed

planner,expression,util: scatter hotspot index in certain scenes by expression index #30655

TonsnakeLin wants to merge 15 commits into from

Conversation

@TonsnakeLin
Copy link
Contributor

@TonsnakeLin TonsnakeLin commented Dec 13, 2021

What problem does this PR solve?

It propose a solution to solve the problem that hot indexes hinder the scalability of TiDB in certain case. The detail design please refer the doc here.

Check List

Tests

  • Unit test
  • Integration test

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains experimental features

Release note

None

TonsnakeLin and others added 15 commits November 10, 2021 14:05
@TonsnakeLin
ddl: Do not consider the clustered index when checking the length of …
bfc9682 
…the secondary index
@TonsnakeLin
Merge branch 'master' of https://github.com/TonsnakeLin/tidb
e1c3410 
ddl: Do not consider the clustered index when checking the length of the secondary index
@TonsnakeLin
Merge branch 'master' into master
662871c
@TonsnakeLin
Merge branch 'master' into master
dce8ef4
@TonsnakeLin
ddl: Do not consider the clustered index when checking the length of the
68a32a0 
secondary index(#29660)
close #29658
@ti-chi-bot
Merge branch 'master' into master
4918824
@ti-chi-bot
Merge branch 'master' into master
4d6c1b0
@TonsnakeLin
Merge branch 'pingcap:master' into master
665924d
@TonsnakeLin
fix the bug TiDB doesn't log any slow log if log level higher than
8f12990 
warn(#30461)
close #30309

Signed-off-by: TonsnakeLin <lpbgytong@163.com>
@TonsnakeLin
Merge branch 'master' of https://github.com/TonsnakeLin/tidb
ea25b28
@TonsnakeLin
fix the bug TiDB doesn't log any slow log(#30461)
5b02862 
close #30309

Signed-off-by: TonsnakeLin <lpbgytong@163.com>
@ti-chi-bot
Merge branch 'master' into master
4a002d7
@TonsnakeLin
add tidb_shard() expression to the where clause implicitly when where…
097115d 
… clause uses all the rest field

Signed-off-by: TonsnakeLin <lpbgytong@163.com>
@TonsnakeLin
remove some tab space for code aligment
d73437f 
Signed-off-by: TonsnakeLin <lpbgytong@163.com>
@TonsnakeLin
add a function (path *AccessPath) IsUkShardIndex()
5de2ba1 
Signed-off-by: TonsnakeLin <lpbgytong@163.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Dec 13, 2021

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Dec 13, 2021
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Dec 13, 2021

@TonsnakeLin: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 13, 2021
@TonsnakeLin
Copy link
Contributor Author

TonsnakeLin commented Dec 13, 2021

@cfzjywxk @youjiali1995 @sticnarf @MyonKeminta @tiancaiamao

@TonsnakeLin
Copy link
Contributor Author

TonsnakeLin commented Dec 13, 2021

@cfzjywxk @youjiali1995 @sticnarf @MyonKeminta @tiancaiamao
I don't known why so many commits I pushed some days ago still appeared in this draft PR.
In actually, The relative commit was since Dec 10, 2021

@sticnarf
Copy link
Contributor

sticnarf commented Dec 13, 2021 •
edited
Loading

@TonsnakeLin Every time you create a new pull request, I suggest you use a separate branch. The branch should be based on the latest master and only contain your related changes.

@tiancaiamao
Copy link
Contributor

tiancaiamao commented Dec 13, 2021

@TonsnakeLin Every time you create a new pull request, I suggest you use a separate branch. The branch should be based on the latest master and only contain your related changes.

That's the develop model in TiKV? @sticnarf Here in TiDB I suggest small PRs (less than 300 lines) with test case for each PR.

The master branch is active and a separate branch with too much change will conflicts a lot.

@tiancaiamao
Copy link
Contributor

tiancaiamao commented Dec 13, 2021

The detail design please refer the doc here.

For design document please use github https://github.com/pingcap/tidb/tree/master/docs/design

Google docs is for discussing internally and your link is not a public one. @TonsnakeLin

@sticnarf
Copy link
Contributor

sticnarf commented Dec 13, 2021

@TonsnakeLin Every time you create a new pull request, I suggest you use a separate branch. The branch should be based on the latest master and only contain your related changes.

That's the develop model in TiKV? @sticnarf Here in TiDB I suggest small PRs (less than 300 lines) with test case for each PR.

The master branch is active and a separate branch with too much change will conflicts a lot.

Ah, I don't mean the development model. I just find @TonsnakeLin seems to reuse his other branches to create this PR, and that leads to the problem.

@TonsnakeLin
Copy link
Contributor Author

TonsnakeLin commented Dec 13, 2021

Got it, I have opened the privilege for every one and you can refer the doc first to understand my code.
I will make a design doc as you said later

@TonsnakeLin
Copy link
Contributor Author

TonsnakeLin commented Dec 13, 2021

@TonsnakeLin Every time you create a new pull request, I suggest you use a separate branch. The branch should be based on the latest master and only contain your related changes.

That's the develop model in TiKV? @sticnarf Here in TiDB I suggest small PRs (less than 300 lines) with test case for each PR.
The master branch is active and a separate branch with too much change will conflicts a lot.

Ah, I don't mean the development model. I just find @TonsnakeLin seems to reuse his other branches to create this PR, and that leads to the problem.

Thanks every much! yes, so I deleted my repository of tidb and fork it again, and made a new branch to create a new drat PR, as bellow.
#30659
Please review the code for the new draft PR, Thanks to @cfzjywxk @youjiali1995 @sticnarf
@MyonKeminta @tiancaiamao

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tiancaiamao tiancaiamao Awaiting requested review from tiancaiamao

@youjiali1995 youjiali1995 Awaiting requested review from youjiali1995

@cfzjywxk cfzjywxk Awaiting requested review from cfzjywxk

Assignees

No one assigned

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@TonsnakeLin @ti-chi-bot @sticnarf @tiancaiamao