Skip to content

HDDS-14966. OM UI shows wrong Leader Readiness values #10036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Russole wants to merge 2 commits into
base: master
Choose a base branch
from
+10 −5
Open

HDDS-14966. OM UI shows wrong Leader Readiness values #10036

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
86227e3
HDDS-14966. SCM Web UI Leader Readiness column does not appear to be …
Russole Apr 3, 2026
fd69796
Replace OM role strings with RaftPeerRole enum
Russole Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_PORT_DEFAULT;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_SERVICE_IDS_KEY;
import static org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.ReadConsistencyProto.READ_CONSISTENCY_UNSPECIFIED;
import static org.apache.ratis.proto.RaftProtos.RaftPeerRole.FOLLOWER;
import static org.apache.ratis.proto.RaftProtos.RaftPeerRole.LEADER;

import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
Expand Down Expand Up @@ -1079,7 +1081,7 @@ public static boolean isBucketSnapshotIndicator(String key) {
}

public static List<List<String>> format(
List<ServiceInfo> nodes, int port, String leaderId, String leaderReadiness) {
List<ServiceInfo> nodes, int port, String leaderId) {
List<List<String>> omInfoList = new ArrayList<>();
// Ensuring OM's are printed in correct order
List<ServiceInfo> omNodes = nodes.stream()
Expand All @@ -1089,8 +1091,11 @@ public static List<List<String>> format(
for (ServiceInfo info : omNodes) {
// Printing only the OM's running
if (info.getNodeType() == HddsProtos.NodeType.OM) {
String role = info.getOmRoleInfo().getNodeId().equals(leaderId)
? "LEADER" : "FOLLOWER";
String nodeId = info.getOmRoleInfo().getNodeId();
String role = nodeId.equals(leaderId) ?
LEADER.name() : FOLLOWER.name();
String leaderReadiness = nodeId.equals(leaderId)
? "LEADER_AND_READY" : "NOT_LEADER";
Comment on lines +1097 to +1098
Copy link
Copy Markdown
Contributor

@adoroszlai adoroszlai Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leaderReadiness should reflect readiness, but currently LEADER_AND_NOT_READY cannot appear in the UI in any case. If we can get correct information about the leader's readiness, we should. Otherwise we should remove the column, since it's redundant.

Copy link
Copy Markdown
Contributor Author

@Russole Russole Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @adoroszlai for the review — good catch.
Currently, the UI does not have access to the actual leader readiness for each OM, so the value is derived from the node’s role. As a result, LEADER_AND_NOT_READY cannot be reflected.

We can either extend this to fetch the real leader readiness from the leader node, or consider removing the column if it is considered redundant.

This change focuses on fixing the inconsistent UI behavior across leader and follower nodes while keeping the existing semantics unchanged.

Copy link
Copy Markdown
Contributor

@adoroszlai adoroszlai Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should consider reverting HDDS-11726 then. BTW, the problem was also reported in the original PR: #7628 (comment)

Copy link
Copy Markdown
Contributor Author

@Russole Russole Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @adoroszlai , based on the current implementation of OzoneManagerRatisServer, getLeaderStatus() only reflects the local OM's Raft server state (via the local division), and there is no mechanism to retrieve readiness for other OM nodes.

This means the UI currently does not have access to per-node leader readiness in an HA setup.

So while the original issue comes from applying the local status to all rows, the underlying limitation is that per-OM readiness is not available from the current API.

Given that, we may need to either extend the backend to expose per-node readiness, or consider removing the column if that information cannot be accurately represented.

List<String> omInfo = new ArrayList<>();
omInfo.add(info.getHostname());
omInfo.add(info.getOmRoleInfo().getNodeId());
Expand Down
4 changes: 2 additions & 2 deletions hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3217,7 +3217,7 @@ public List<List<String>> getRatisRoles() {
if (null == omRatisServer) {
return getRatisRolesException("Server is shutting down");
}
String leaderReadiness = omRatisServer.getLeaderStatus().name();

final RaftPeerId leaderId = omRatisServer.getLeaderId();
if (leaderId == null) {
LOG.error(NO_LEADER_ERROR_MESSAGE);
Expand All @@ -3231,7 +3231,7 @@ public List<List<String>> getRatisRoles() {
LOG.error("Failed to getServiceList", e);
return getRatisRolesException("IO-Exception Occurred, " + e.getMessage());
}
return OmUtils.format(serviceList, port, leaderId.toString(), leaderReadiness);
return OmUtils.format(serviceList, port, leaderId.toString());
}

/**
Expand Down