HDDS-13108. Refactor StorageVolume to use SlidingWindow by ptlrs · Pull Request #8843 · apache/ozone

ptlrs · 2025-07-22T06:37:23Z

Please describe your PR in detail:

This PR uses the new sliding window implementation.
It migrates all existing checks to detect a failed volume to use the new time-based sliding window utility.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13108

How was this patch tested?

CI:https://github.com/ptlrs/ozone/actions/runs/16436635030

… scanner failures

… tests

…e-failed-volume-checks-to-one-sliding-window

…g window mechanism

ptlrs · 2025-07-22T18:03:33Z

Hi @errose28 @Tejaskriya @adoroszlai can you please review this PR?

Tejaskriya

Thanks for working on this @ptlrs , please find a suggestion below

...ner-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java

ptlrs · 2025-07-31T06:38:04Z

Thanks for the review @Tejaskriya. I have added the configuration.

Tejaskriya

Looks good to me, @errose28 could you please take a look?

github-actions · 2025-11-11T00:05:52Z

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

github-actions · 2025-11-25T00:05:10Z

Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it.

ptlrs · 2026-02-09T19:29:33Z

Hi @errose28, could you please reopen this PR?

…e-failed-volume-checks-to-one-sliding-window # Conflicts: # hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java # hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java

ptlrs · 2026-02-12T08:48:18Z

Hi @errose28, the conflicts have been resolved for this PR. Could you please take a look.

github-actions · 2026-03-06T00:13:27Z

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

…failed-volume-checks-to-one-sliding-window

adoroszlai

Thanks @ptlrs for the patch.

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java

...test/java/org/apache/hadoop/ozone/container/common/volume/TestStorageVolumeHealthChecks.java

...c/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java

...ner-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java

…k interval

…stomizable sliding window timestamps

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/SlidingWindow.java

Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

ptlrs · 2026-03-27T18:24:43Z

Hi @errose28 @ChenSammi @adoroszlai I have updated this PR, could you please take another look?

adoroszlai

Thanks @ptlrs for updating the patch, LGTM.

...test/java/org/apache/hadoop/ozone/container/common/volume/TestStorageVolumeHealthChecks.java

...c/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java

ChenSammi · 2026-03-30T06:39:11Z

...c/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java

  )
  private Duration diskCheckTimeout = DISK_CHECK_TIMEOUT_DEFAULT;

+  @Config(key = "hdds.datanode.disk.check.sliding.window.timeout",


With this sliding window introduced, "hdds.datanode.disk.check.io.test.count" property function is half removed. We should consider deprecate "hdds.datanode.disk.check.io.test.count" and introduce a new boolean property with name, like "hdds.datanode.disk.check.io.test.enabled".

To not break existing users, it's recommend to add new property "hdds.datanode.disk.check.io.test.enabled", instead of change the current property "hdds.datanode.disk.check.io.test.count" to "hdds.datanode.disk.check.io.test.enabled".

Ok, I reverted the change which removed the config and updated the deprecated config list.

…ust configurations for volume health checks

…ibility

ptlrs added 4 commits July 11, 2025 22:15

HDDS-13108. Transition to use the generic sliding window for counting…

b1df449

… scanner failures

HDDS-13108. Remove unused SlidingWindow implementation and associated…

625d52a

… tests

Merge remote-tracking branch 'upstream/master' into HDDS-13108-Migrat…

f108d3c

…e-failed-volume-checks-to-one-sliding-window

HDDS-13108. Replace count-based failure checks with time-based slidin…

33b9a4b

…g window mechanism

ptlrs marked this pull request as draft July 22, 2025 06:38

errose28 added the scanners Changes related to datanode container and volume scanners label Jul 22, 2025

Tejaskriya self-requested a review July 23, 2025 08:31

errose28 self-requested a review July 23, 2025 18:46

Tejaskriya reviewed Jul 29, 2025

View reviewed changes

...ner-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java Outdated Show resolved Hide resolved

HDDS-13108. Add configuration for sliding window timeout in disk checks

e81f59b

ptlrs requested a review from Tejaskriya July 31, 2025 06:38

Tejaskriya reviewed Aug 4, 2025

View reviewed changes

github-actions bot added the stale label Nov 11, 2025

github-actions bot closed this Nov 25, 2025

errose28 reopened this Feb 9, 2026

github-actions bot removed the stale label Feb 10, 2026

HDDS-13108. Remove unused imports from HddsVolume

17fa938

github-actions bot added the stale label Mar 6, 2026

Merge remote-tracking branch 'origin/master' into HDDS-13108-Migrate-…

4d8b6fe

…failed-volume-checks-to-one-sliding-window

adoroszlai reviewed Mar 9, 2026

View reviewed changes

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java Outdated Show resolved Hide resolved

...test/java/org/apache/hadoop/ozone/container/common/volume/TestStorageVolumeHealthChecks.java Outdated Show resolved Hide resolved

adoroszlai removed the stale label Mar 9, 2026

adoroszlai changed the title ~~HDDS-13108. Migrate failed volume checks to one sliding window~~ HDDS-13108. Refactor StorageVolume to use SlidingWindow Mar 9, 2026

checkstyle

0b9e02b

ChenSammi reviewed Mar 11, 2026

View reviewed changes

...c/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java Show resolved Hide resolved

ChenSammi reviewed Mar 11, 2026

View reviewed changes

...ner-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java Show resolved Hide resolved

ChenSammi reviewed Mar 11, 2026

View reviewed changes

...ner-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/StorageVolume.java Show resolved Hide resolved

ptlrs added 3 commits March 24, 2026 19:39

HDDS-13108. Enforce minimum sliding window timeout based on disk chec…

158819e

…k interval

HDDS-13108. Add Clock support to StorageVolume and its Builder for cu…

10d0dcd

…stomizable sliding window timestamps

HDDS-13108. Update sliding window checks and fix test clock adjustments

af0e4f8

adoroszlai reviewed Mar 25, 2026

View reviewed changes

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/SlidingWindow.java Outdated Show resolved Hide resolved

ptlrs and others added 2 commits March 25, 2026 08:30

Update SlidingWindow.java

1ec9e0e

Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

HDDS-13108. Fix Javadoc for MonotonicClock in SlidingWindow

0de347c

ptlrs requested review from ChenSammi and adoroszlai March 27, 2026 18:24

adoroszlai approved these changes Mar 27, 2026

View reviewed changes

...test/java/org/apache/hadoop/ozone/container/common/volume/TestStorageVolumeHealthChecks.java Show resolved Hide resolved

ptlrs mentioned this pull request Mar 29, 2026

HDDS-14871. DataNode: tolerate per-volume health-check latch timeouts before marking volumes failed. #9954

Open

ChenSammi marked this pull request as ready for review March 30, 2026 04:25

ChenSammi reviewed Mar 30, 2026

View reviewed changes

...c/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java Outdated Show resolved Hide resolved

ChenSammi reviewed Mar 30, 2026

View reviewed changes

HDDS-13108. Replace IO test count with disk check enable flag and adj…

9e20c2d

…ust configurations for volume health checks

ptlrs requested a review from ChenSammi March 31, 2026 03:13

HDDS-13108. Revert change to remove IO test count for backward compat…

b6d2478

…ibility

Conversation

ptlrs commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Please describe your PR in detail:

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

ptlrs commented Jul 22, 2025

Uh oh!

Tejaskriya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ptlrs commented Jul 31, 2025

Uh oh!

Tejaskriya left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 11, 2025

Uh oh!

github-actions bot commented Nov 25, 2025

Uh oh!

ptlrs commented Feb 9, 2026

Uh oh!

ptlrs commented Feb 12, 2026

Uh oh!

github-actions bot commented Mar 6, 2026

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ptlrs commented Mar 27, 2026

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ChenSammi Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

ptlrs Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

ChenSammi Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

ptlrs Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ptlrs commented Jul 22, 2025 •

edited

Loading