HDDS-13661. Fix flaky TestKeyDeletingService#testPurgeKeysRequestBatching

[arunsarin85]

What changes were proposed in this pull request?

Fixed two race conditions in TestKeyDeletingService$RequestBatching#testPurgeKeysRequestBatching that caused intermittent

1. Removed background thread interference
2. Ensured deleted keys are visible in RocksDB before the service runs
3. Removed the @flaky("HDDS-13661") annotation

Please describe your PR in detail:
The test testPurgeKeysRequestBatching validates that KeyDeletingService correctly batches PurgeKeysRequests to stay within the Ratis byte limit (introduced by HDDS-13517). It creates and deletes 50 keys, then triggers the service manually via runPeriodicalTaskNow(), asserting that all 50 keys are purged across the captured batches.

Root cause 1 — background thread race:
In the intermittent failure scenario, the background thread could pick up all 50 keys but only submit 7 of the 8 batches before getAllValues() is called, while the 8th batch (containing the 50th key) is still in-flight. The fix sets OZONE_BLOCK_DELETING_SERVICE_INTERVAL to 1 day so the background thread never fires during the test window.
Root cause 2 — DoubleBuffer visibility:
inserts om.awaitDoubleBufferFlush() after createAndDeleteKeys() but before resume(), ensuring all 50 entries are committed to RocksDB before the service runs.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13661

How was this patch tested?

• The flaky test was reproduced locally (intermittent expected: <50> but was: <49>) before the fix was applied.
• After the fix, the test was stress-run 20 consecutive times using a local stress-run script (run-flaky-test.sh) with no failures:

[adoroszlai]

Thanks @arunsarin85 for working on this. Can you please trigger a run of flaky-test-check for TestKeyDeletingService with and without this change (i.e. branch HDDS-13661 and master)?

[arunsarin85]

Thanks @arunsarin85 for working on this. Can you please trigger a run of flaky-test-check for TestKeyDeletingService with and without this change (i.e. branch HDDS-13661 and master)?

Hi @adoroszlai ,

Ran testPurgeKeysRequestBatching 100 times locally on master (without the fix) - all passed. Looks like the race window is too narrow to trigger reliably on local machine, but opens up under the load and timing conditions of CI environments.

full_log_TestKeyDeletingService$RequestBatching_testPurgeKeysRequestBatching_20260403_111721.txt

[adoroszlai]

Ran testPurgeKeysRequestBatching 100 times locally on master (without the fix) - all passed. Looks like the race window is too narrow to trigger reliably on local machine, but opens up under the load and timing conditions of CI environments.

Please use:
https://github.com/arunsarin85/ozone/actions/workflows/intermittent-test-check.yml

[peterxcli]

I think we should suspend the background worker instead.