redo: create redo dispatcher with correct start-ts #3696
Conversation
|
Skipping CI for Draft Pull Request. |
ti-chi-bot
bot
added
do-not-merge/needs-linked-issue
release-note
Summary of ChangesHello @wk989898, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request is currently a work-in-progress, focusing on refining the initialization logic for redo dispatchers by temporarily adjusting the Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
ti-chi-bot
bot
added
the
size/L
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request appears to be a work-in-progress, with changes primarily focused on testing and debugging. In downstreamadapter/dispatchermanager/dispatcher_manager_redo.go, logic for fetching recovery info for redo dispatchers has been commented out. This is a critical change as the surrounding comments indicate this logic prevents a potential panic. This should be addressed before merging. In tests/integration_tests/run_heavy_it_in_ci.sh, the integration test suite for MySQL has been altered to repeatedly run a single test case. This seems to be for debugging and should be reverted to restore the full test suite. Given the WIP nature of this PR, these changes are likely temporary, but it's important to highlight them to ensure they are not merged into the main branch in their current state.
|
/test pull-cdc-mysql-integration-heavy |
2 similar comments
|
/test pull-cdc-mysql-integration-heavy |
|
/test pull-cdc-mysql-integration-heavy |
…o redo-start-ts
|
/test pull-cdc-mysql-integration-heavy |
ti-chi-bot
bot
added
size/XXL
ti-chi-bot
bot
removed
the
do-not-merge/work-in-progress
|
/test all |
|
/test all |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request refactors the creation of redo dispatchers to ensure they use the correct start-ts. The main change involves removing the logic that aligned the redo dispatcher's start-ts with the MySQL sink's recovery state. This change is propagated throughout the system, with updates to protobuf messages to carry separate redo-related information (RedoCheckpointTs, RedoSchemas), and refactoring in the maintainer and dispatcher orchestrator for cleaner separation of default and redo mode handling. Additionally, several integration tests for consistent replication have been enabled, which is a positive sign of increased confidence in this feature. The changes generally look good and improve code clarity. I have one comment regarding a potentially misleading comment.
| // UpdateMeta used to update redo meta log. | ||
| // The checkpoint-ts is always less than the resolved-ts because it represents the minimum checkpoint-ts of the global event dispatcher. | ||
| // The event dispatcher does not advance until the resolved-ts exceeds the event's commit-ts. | ||
| // only for redo table trigger event dispatcher. |
There was a problem hiding this comment.
The comment on lines 110-111 states that checkpoint-ts is always less than resolved-ts. However, with the changes in this PR (specifically the removal of start-ts alignment logic in dispatcher_manager_redo.go), it seems possible for event dispatchers to start at a later timestamp than redo dispatchers, especially with a MySQL sink.
This would mean checkpoint-ts (from event dispatchers) could be greater than resolved-ts (from redo dispatchers), at least initially. This contradicts the comment.
Could you please clarify if this understanding is correct? If so, this comment is misleading and should be updated to reflect the new logic and explain why checkpoint-ts > resolved-ts is now acceptable. For example, is it only an initial state that gets resolved quickly?
downstreamadapter/dispatcherorchestrator/dispatcher_orchestrator.go
Outdated
Show resolved
Hide resolved
|
|
||
| // Step 1: Determine start timestamp and update DDL dispatcher | ||
| startTs := c.determineStartTs(allNodesResp) | ||
| startTs, redoStartTs := c.determineStartTs(allNodesResp) |
There was a problem hiding this comment.
Should we need to make a common function for bootstrap for redoMode or normalMode? I see a lot of similar logic here
|
/test all |
|
/test all |
ti-chi-bot
bot
added
needs-1-more-lgtm
wk989898
added
do-not-merge/hold
| // UpdateMeta used to update redo meta log. | ||
| // The checkpoint-ts is always less than the resolved-ts because it represents the minimum checkpoint-ts of the global event dispatcher. | ||
| // The event dispatcher does not advance until the resolved-ts exceeds the event's commit-ts. | ||
| // It is possible that the checkpoint-ts is greater than resolved-ts in the beginning because the event dispatcher query the start-ts from mysql sink. |
There was a problem hiding this comment.
The explanation of why the checkpoint-ts may greater than the resolved-ts at the beginning is not clear.
Why query the start-ts from the mysql sink may cause the above case.
There was a problem hiding this comment.
I confirm that the checkpoint-ts is always less than or equal resolved-ts.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: 3AceShowHand, hongyunyan The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
|
@wk989898: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
3 participants
What problem does this PR solve?
Issue Number: close #3726 close #3689
What is changed and how it works?
Before this PR, table trigger redo dispatcher and table trigger event dispatcher used the same start-ts to init schemastore. But actually, the event dispatcher will get true start-ts by querying the downstream, and the event dispatcher start-ts may be greater than the redo dispatcher start-ts. If the redo dispatcher uses this start-ts, it may meet "no version found" error. because some tables are dropped or truncated. That makes some redo dispatchers blocked and checkpointTs lag increases.
In this PR, table trigger redo dispatcher start-ts is split from the event dispatcher start-ts. It's more like two independent responses sent to the maintainer.
Check List
Tests
Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note