Skip to content

HDDS-15535. Container Balancer should validate configuration and report startup failures to user#10812

Draft
sreejasahithi wants to merge 2 commits into
apache:masterfrom
sreejasahithi:HDDS-15535
Draft

HDDS-15535. Container Balancer should validate configuration and report startup failures to user#10812
sreejasahithi wants to merge 2 commits into
apache:masterfrom
sreejasahithi:HDDS-15535

Conversation

@sreejasahithi

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Currently, Container Balancer could start with configuration that made balancing impossible or misleading, with no clear reporting to the user.
This change adds startup time validation and ensures configuration errors are returned to the client with a useful message.
Problem :
Users could start the balancer with invalid or self-defeating configuration, for example:

  • non existent container IDs
  • all include-datanodes covered by exclude-datanodes, or all include-containers covered by exclude-containers
  • fewer than two eligible healthy in-service datanodes after applying include/exclude filters
  • max-datanodes-percentage-to-involve-per-iterationthat rounds down to fewer than two datanodes
  • per iteration size limits where max-size-entering-target or max-size-leaving-source exceeds max-size-moved-max-per- iteration
  • max-datanodes-percentage-to-involve-per-iteration set to 0, which cannot support any balancing iteration

This patch adds validation in ContainerBalancer before the balancer task starts. Failures throw InvalidContainerBalancerConfigurationException, so ozone admin containerbalancer start reports the reason to the user.

Note: The datanode check validates that at least two eligible datanodes exist and that the configured percentage allows at least two datanodes to be involved per iteration. It does not validate whether those datanodes can actually form a valid source or target, because that would require reusing much of the balancer iteration logic. Failures of that kind may still appear via ozone admin containerbalancer status (stop reason / message).

What is the link to the Apache JIRA

HDDS-15535

How was this patch tested?

Added tests
Manually tested in docker ozone cluster:

bash-5.1$ ozone admin containerbalancer start --include-datanodes ozone-balancer-datanode1-1.ozone-balancer_default,ozone-balancer-datanode2-1.ozone-balancer_default --exclude-datanodes ozone-balancer-datanode2-1.ozone-balancer_default,ozone-balancer-datanode1-1.ozone-balancer_default  -t 0.001
Failed to start Container Balancer. include-datanodes is a subset of exclude-datanodes, no datanode can participate in balancing.
bash-5.1$
bash-5.1$ ozone admin containerbalancer start --include-containers 1,2,3,4 --exclude-containers 1,2,3,4 -t 0.001
Failed to start Container Balancer. include-containers is a subset of exclude-containers, no container can be selected for balancing.
bash-5.1$
bash-5.1$ ozone admin containerbalancer start --max-datanodes-percentage-to-involve-per-iteration 20 -t 0.001
Failed to start Container Balancer. max-datanodes-percentage-to-involve-per-iteration=20 allows at most 1 datanode(s) per iteration with 6 eligible datanode(s), but at least 2 are required for a source and target datanode pair.
bash-5.1$
bash-5.1$ ozone admin containerbalancer start --max-datanodes-percentage-to-involve-per-iteration 0 -t 0.001
Max Datanodes Percentage To Involve Per Iteration must be greater than zero.
bash-5.1$

Green CI : https://github.com/sreejasahithi/ozone/actions/runs/29627948987

@sreejasahithi

sreejasahithi commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Open question for reviewers:
Should include-* and exclude-* be mutually exclusive for both datanodes and containers, or should we keep allowing both together?
According to me they could be made mutually exclusive.
The only case I can think of where both will be used at same time would be a scenario like:
automation generates include - e.g. “all nodes in rack A should be included” -> dn1,dn2,dn3,dn4,dn5. But user decides to skip a node from participating in balancing, so user adds --exclude-datanodes dn3 instead of editing the script for changing the include list.

@sreejasahithi

Copy link
Copy Markdown
Contributor Author

@ashishkumar50 could you please review this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant