HDDS-15966. Fix flaky TestScmHAFinalization#testSnapshotFinalization - #11200
Open
errose28 wants to merge 6 commits into
Open
HDDS-15966. Fix flaky TestScmHAFinalization#testSnapshotFinalization#11200errose28 wants to merge 6 commits into
errose28 wants to merge 6 commits into
Conversation
errose28
added
test
zdu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This failure is unique to the ZDU branch. 200 runs on master passed, but 200 runs on the base ZDU branch (
f535d5dabec700be68733241461b1f483d0c359e) showed some failures. Two types of failures showed up in this run on the ZDU branch:Failure 1: DB is closed during read (6/200 times)
Problem
As soon as the stopped SCM is restarted and expected to install a snapshot, we begin polling its DB to see if the finalization key has been written yet as an indication that it has finalized. Snapshot install requires closing the DB and opening a new one, so there is a window where this call can fail before snapshot install finished.
Solution
Ignore transient errors while waiting for the DB status. If the DB is successfully installed but still produces errors for other reasons, the test will still fail with a timeout and log the stack traces.
Failure 2: Follower SCM fails to exit safemode (1/200 times)
Problem
Test logs show the SCMs had the following roles:
The test failed waiting for scm2. It is the only one whose logs show up in the finalization polling loop:
However, the finalization condition currently checks for both finalization and safemode exit. The test logs further show that scm2 did finalize, but failed to exit safemode, which stalled the test:
Digging further, we see that the pipeline safemode rule is the only one that did not validate:
So the test failure was caused by the pipeline safemode rule failing to validate in 1 minute on an SCM that is not the main one under test.
Root Cause
The order of operations to cause the failure is:
ALLOCATEDpipelines on all SCMs via Ratis, and instructs Datanodes to create them.ALLOCATED.PipelineManager#openPipeline, which delegates to a@Replicatemethod that will throughNotLeaderException.OPEN_PIPELINEevent from firing.PipelineManager#openPipelinecall succeeds, firing theOPEN_PIPELINEevent and moving it out of safemode.ALLOCATEDtoOPEN, but this does not trigger any reprocessing of the active pipelines to affect the safemode rule.This is supported by another test run which reproduced the issue with additional logging added. Here we can see the Ratis three pipelines reach the call to open the pipeline on the scm2, but do not make it past that. Only the
Attempting to openmessages are presentThe new logs show the cause is the
NotLeaderException, which only occurs on scm2:Follow-up Questions and Answers
Why does it always affect scm2 (the follower installing the snapshot)?
The leader SCM is able to submit the ratis request to move the pipeline state, so every report of a pipeline with a leader and existing datanodes will trigger the
OPEN_PIPELINEevent on the SCM leader. The follower installing the snapshot is not started until later in the test, so the SCM DB and Datanode pipeline state have converged by the time it loads the snapshot. No Datanode reports on this follower are dropped due to pipelines still beALLOCATEDin SCM. This leaves scm2 as the only node able to hit the issue.Why doesn't this happen in other mini ozone cluster tests?
Standard mini ozone cluster tests only wait for the leader to exit safemode, so if followers hit this issue it does not affect test execution unless the test adds an extra assertion on follower safemode status like this one. This is an existing issue that occurs with very low probability and was just exposed due to this test change on the ZDU branch.
Why didn't this happen on the master version of this test?
The non-rolling upgrade framework on master bypasses the pipeline safemode rule while the cluster is finalizing, so it passes trivially regardless of the pipeline reports since all pipelines are expected to be closed while Datanodes finalize.
Is this a production issue?
Not really, because a real cluster would still converge to the expected value in 1-2 minutes. Within this window there could be a case where the only the leader is out of safemode and the cluster goes back in safemode because a follower is elected leader, but this was a known issue anyways and can only happen briefly right after startup. It may be helpful to add better logging for this case though.
Solution
If safemode exit for all SCMs was actually a requirement in this test, we could reduce the pipeline reporting interval. However, safemode exit is no longer required for SCM finalization in the new upgrade framework. It is only required to instruct OMs to finalize which is not covered by this test. This means we can just remove the safemode exit check from the assertion.
What is the link to the Apache JIRA
HDDS-15966
How was this patch tested?
TestScmHAFinalizationsuite passed 400 times on my fork