Camel Zookeeper Cluster Split brain issue when leader is isolated - #25529
Camel Zookeeper Cluster Split brain issue when leader is isolated#25529mustafaAlmonayer wants to merge 1 commit into
Conversation
|
First of all, thank you all for this magnificent framework — it has been a pleasure to build on, and I'm grateful for the chance to give something back. I owe you a couple of apologies up front, and I'd rather be transparent about them than have them surface in review. All of the test cases here, including the integration tests, were carried out manually by running the application against a real three-node ZooKeeper ensemble. I have very limited experience writing automated tests in Apache Camel, so I wasn't able to translate those scenarios into proper automated coverage. I'm sorry about that — I know it's a serious gap, especially for a change in this area, and I would very much welcome any pointers on the right harness and patterns to use. The reason I moved quickly is the nature of the defect itself: it undermines the core purpose of the ZooKeeper cluster service by allowing a split-brain condition in which more than one instance holds leadership and stays active at the same time. I also tried to open a JIRA issue before raising this PR, but my ASF account request is still awaiting confirmation, and I didn't want to sit on the fix while waiting. Apologies for not following the usual process. If someone else is able to approve the request — or file the ticket on my behalf — I'll link it here and update the PR title right away. Thank you again for taking the time to look at this, and for all the work that goes into maintaining Camel. I'm glad to make any changes you'd like. |
ZooKeeperClusterView did not release leadership when the ZooKeeper
connection entered an error state, and never re-entered the election
afterwards.
Two separate problems:
CuratorLocalMember#isLeader delegated to LeaderSelector#hasLeadership.
Curator clears that flag only after takeLeadership() has returned, so
a listener querying getLocalMember().isLeader() from inside the
leadership-changed event - which is what ClusteredRoutePolicy does -
still observed true and kept its routes running on a node that had
already lost leadership.
LeaderSelector was started without autoRequeue(), so once
takeLeadership() returned the member left the election permanently.
The only re-entry path was the requeue() call in doStart(), which
requires the view to be restarted. After a connection error the
cluster could be left with no leader at all until the affected
processes were restarted manually.
The view now tracks leadership in its own volatile flag, cleared before
the leadership-lost event is fired, and enables autoRequeue() so the
member rejoins the election. The leadership-lost event reports no leader
explicitly instead of re-reading it from ZooKeeper, where the not yet
released lock node could still name the local member.