Partition export per part local backoff policy#1984
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4d576fefe4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| manifest.retry_initial_backoff_ms = json->getValue<size_t>("retry_initial_backoff_ms"); | ||
| manifest.retry_max_backoff_ms = json->getValue<size_t>("retry_max_backoff_ms"); |
There was a problem hiding this comment.
When a node upgrades with an export created by the previous code still under /exports, that metadata.json does not contain these two new fields because older manifests only wrote max_retries. ExportReplicatedMergeTreePartitionManifest::fromJsonString is used by the manifest updater before it can clean up or kill tasks, and getValue throws on missing keys, so one old in-flight export makes poll fail repeatedly and prevents this table's export state from progressing until the ZooKeeper entry is removed manually. Please read these fields with defaults when absent, similar to the existing allow_lossy_cast compatibility handling.
Useful? React with 👍 / 👎.
arthurpassos
changed the base branch from
rw_lock_for_partition_export
to
antalya-26.3
arthurpassos
changed the title
|
@codex review, do not worry about backwards compatibility |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
arthurpassos
added
antalya
port-antalya
| DECLARE(Bool, export_merge_tree_partition_force_export, false, R"( | ||
| Ignore existing partition export and overwrite the zookeeper entry | ||
| )", 0) \ | ||
| DECLARE(UInt64, export_merge_tree_partition_max_retries, 3, R"( |
There was a problem hiding this comment.
In common case removed settings are not removed completely, added in OBSOLETE_SETTINGS list for backward compatibility. Why not now?
| if (initial > headroom) | ||
| return max_backoff_seconds; | ||
|
|
||
| return std::min(initial << shift, max_backoff_seconds); |
There was a problem hiding this comment.
Up to 2^64-1 seconds (~585 billion years)? :)
| backoff.attempts, manifest.retry_initial_backoff_seconds, manifest.retry_max_backoff_seconds); | ||
| const auto now = time(nullptr); | ||
| /// Clamp so a huge configured back-off cannot overflow time_t (now is a normal wall-clock value). | ||
| const size_t headroom = static_cast<size_t>(std::numeric_limits<time_t>::max() - now); |
There was a problem hiding this comment.
time_t is signed, size_t is unsigned. If we allow all possible ranges, may get something unexpected. Mya be set hardcoded limit in billion years?
3 participants
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Implements a local backoff policy for export partition - helps with transient and local failures.,
It is controlled by
export_merge_tree_partition_retry_initial_backoff_secondsandexport_merge_tree_partition_retry_max_backoff_seconds.Removes
export_merge_tree_partition_max_retriesas it would conflict with a local backoff policy implementation. We already have a task time out anyways. The only limitation of relying on the task time out is that non retryable errors would be retried until it times out. Not ideal, so I implemented an err classification method that will classify between retryable and non retryable. Non retryable will fail the task immediately.Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: