[feature](fe) Add resource group success quorum check by deardeng · Pull Request #66751 · apache/doris

deardeng · 2026-08-14T02:43:36Z

What problem does this PR solve?

Issue Number: None

Related PR: #66680

Problem Summary: Load transaction commits only enforced the ordinary replica quorum and could therefore succeed without a configured minimum number of successful replicas in each availability resource group. Add a mutable FE configuration and enforce the per-resource group success floor in the centralized transaction commit check. Clamp each resource group requirement to the partition declared replica allocation so backend liveness changes and transient extra replicas do not weaken or inflate the commit requirement.

Release note

Add the mutable FE configuration resource_group_succ_quorum to require a minimum number of successful load replicas per availability zone.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

### What problem does this PR solve? Issue Number: None Related PR: apache#66680 Problem Summary: Load transaction commits only enforced the ordinary replica quorum and could therefore succeed without a configured minimum number of successful replicas in each availability zone. Add a mutable FE configuration and enforce the per-AZ success floor in the centralized transaction commit check. Clamp each AZ requirement to the partition declared replica allocation so backend liveness changes and transient extra replicas do not weaken or inflate the commit requirement. ### Release note Add the mutable FE configuration cross_az_succ_quorum to require a minimum number of successful load replicas per availability zone. ### Check List (For Author) - Test: Unit Test - DatabaseTransactionMgrTest cross-AZ quorum, unavailable replica, and extra replica cases - Behavior changed: Yes (when cross_az_succ_quorum is configured, FE rejects commits that do not meet the per-AZ success floor) - Does this need documentation: No

hello-stephen · 2026-08-14T02:43:43Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

deardeng · 2026-08-14T02:43:58Z

run buildall

deardeng · 2026-08-14T02:44:04Z

/review

github-actions

Requesting changes because the opt-in cross-AZ durability policy is not end-to-end safe yet.

Blocking findings:

The FE adds a distribution-specific check after both BE writer generations have already finalized commit infos using only the ordinary scalar quorum, so a healthy slower required-AZ replica can be omitted permanently.
Startup and mutable config updates accept malformed or duplicate policy entries, while the lazy parser silently drops or overwrites them and enforces a weaker policy.
Commit classification reads a backend location tag that MODIFY BACKEND mutates without safe publication, so it can enforce the policy against stale topology.

Critical checkpoint conclusions:

Goal and proof: The local FE check expresses the requested floor, but the three findings mean the end-to-end goal is not accomplished; direct FE tests with fabricated commit infos do not prove the writer-to-commit path.
Scope and clarity: The three-file patch is locally focused and readable, but the required planner/BE producer handoff and validation boundary are missing.
Concurrency and locks: Config cache publication is safe for the supported replace-on-update path. Table locks stabilize partition/index metadata, but not backend tag publication; no additional lock-order or deadlock issue was found.
Lifecycle and statics: The process-static cache has a coherent replacement lifecycle, with no circular initialization or resource-release issue.
Dynamic configuration: The master observes replacement arrays promptly, but malformed values fail open. Master-only failover behavior otherwise matches adjacent mutable-config semantics.
Compatibility: No persisted or symbol format changes are introduced, but current and older BEs receive no AZ-aware completion criterion or capability fence, so rolling deployment is unsafe when the feature is enabled.
Parallel paths: Ordinary non-cloud transaction sources, local 2PC at precommit, and each subtransaction reach checkCommitStatus; final 2PC correctly preserves the prepared decision. Cloud uses a distinct meta-service transaction model. Auto-partition is exposed to the same writer handoff defect.
Conditions: Allocation clamping and the zero-requirement skip match the declared-allocation policy; dead, bad, missing, decommissioned, extra, and loaded schema-change replicas yielded no separate defect.
Test coverage: The added tests cover direct success/failure, clamping, unavailable/bad replicas, and an extra replica. They miss end-to-end v1/v2 delayed-remote behavior, startup/admin rejection, and the backend-topology concurrency boundary.
Test results: Compile, CheckStyle, and several static checks are green; FE UT and regression/performance jobs were still pending at review time. No local tests were run because this workflow is review-only.
Observability: Failure logs include transaction, tablet, table, partition, version, AZ, and replica detail. No separate logging/metrics blocker was found.
Transactions and persistence: The check runs before commit or precommit state advancement; no edit-log payload is added, and prepared-state failover semantics remain intact.
Writes and crashes: Existing atomic state transitions remain in place and no new leak/crash path was found, but the missing producer contract can turn a healthy load into a terminal commit failure.
FE/BE variables: The AZ requirement is not passed or translated into an equivalent requirement on either initial or auto-partition writer paths; this is the first blocking finding.
Performance: Parsing is amortized once per config replacement and backend tags are cached once per commit; no distinct material performance problem was found.
Other issues and user focus: No additional user focus was supplied, and the full-PR sweep found no valuable issue beyond the three inline findings.

### What problem does this PR solve? Issue Number: None Related PR: apache#66751 Problem Summary: Backend location tags can be updated by MODIFY BACKEND while transaction commit threads read them. The plain locationTag reference had no Java memory-model publication edge, so readers could observe a stale availability-zone tag. Publish replacement Tag instances through a volatile reference and add a deterministic regression test for the required publication semantics. ### Release note None ### Check List (For Author) - Test: Unit Test - BackendTest#testLocationTagIsSafelyPublished - Behavior changed: No (ensures existing backend location updates are visible across threads) - Does this need documentation: No

ignore

deardeng · 2026-08-14T03:54:39Z

run buildall

hello-stephen · 2026-08-14T05:30:35Z

FE UT Coverage Report

Increment line coverage 89.66% (26/29) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-08-14T07:30:36Z

FE Regression Coverage Report

Increment line coverage 0.22% (1/459) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Replace availability-zone terminology with the existing resource group concept for load success quorum configuration. Keep Config focused on the raw configuration declaration and move transaction-specific parsing, caching, and logging into DatabaseTransactionMgr without changing runtime behavior. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.transaction.DatabaseTransactionMgrTest (19 tests passed) - mvn checkstyle:check -pl fe-common,fe-core (0 violations) - Behavior changed: No - Does this need documentation: No

deardeng · 2026-08-17T08:11:25Z

run buildall

gavinchou · 2026-08-17T12:28:49Z


+    @ConfField(mutable = true, masterOnly = true, description = "Minimum number of successfully written replicas "
+            + "required in each resource group for a load job.")
+    public static volatile String[] resource_group_succ_quorum = {};


usage example?

how to configure it?

see fix be pr, it has 2 groovy case #66827

github-actions Bot previously requested changes Aug 14, 2026

View reviewed changes

Comment thread fe/fe-common/src/main/java/org/apache/doris/common/Config.java Outdated

Comment thread fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java

Comment thread fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java Outdated

deardeng mentioned this pull request Aug 17, 2026

[feature](be) Support cross-AZ success quorum waiting #66827

Open

16 tasks

deardeng changed the title ~~[feature](fe) Add cross-AZ success quorum check~~ [feature](fe) Add resource group success quorum check Aug 17, 2026

gavinchou reviewed Aug 17, 2026

View reviewed changes

Navigation Menu

Uh oh!

[feature](fe) Add resource group success quorum check - #66751

[feature](fe) Add resource group success quorum check#66751
deardeng wants to merge 3 commits into
apache:masterfrom
deardeng:impl-quorum-load-fe

deardeng commented Aug 14, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Aug 14, 2026

Uh oh!

deardeng commented Aug 14, 2026

Uh oh!

deardeng commented Aug 14, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deardeng commented Aug 14, 2026

Uh oh!

hello-stephen commented Aug 14, 2026

Uh oh!

hello-stephen commented Aug 14, 2026

Uh oh!

deardeng commented Aug 17, 2026

Uh oh!

gavinchou Aug 17, 2026

Uh oh!

gavinchou Aug 17, 2026

Uh oh!

deardeng Aug 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

deardeng commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Aug 14, 2026

Uh oh!

deardeng commented Aug 14, 2026

Uh oh!

deardeng commented Aug 14, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deardeng commented Aug 14, 2026

Uh oh!

hello-stephen commented Aug 14, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Aug 14, 2026

FE Regression Coverage Report

Uh oh!

deardeng commented Aug 17, 2026

Uh oh!

gavinchou Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

gavinchou Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

deardeng Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

deardeng commented Aug 14, 2026 •

edited

Loading