When decommission is attempted again, clear any existing transferred_ranges to force re-streaming and avoid consistency violation - #5003
Open
Jollyplum wants to merge 1 commit into
Conversation
Jollyplum
force-pushed
the
mbyrd/CASSANDRA-16290/cassandra-4.0
branch
from
August 6, 2026 21:35
4fc35c5 to
d07e94f
Compare
| PendingRangeCalculatorService.instance.blockUntilFinished(); | ||
|
|
||
| // This check needs to happen above startLeaving | ||
| boolean resumingInFlightDecommission = tokenMetadata.isLeaving(FBUtilities.getBroadcastAddressAndPort()) |
Contributor
There was a problem hiding this comment.
Why don't you just check operationMode == Mode.LEAVING ?
Contributor
Author
There was a problem hiding this comment.
Yes, so on 4.0/4.1 this is redundant the guards above guarantee we're in one of NORMAL/LEAVING.
on 5.0 however we have DECOMMISSION_FAILED which is likely a common decommission resume scenario.
The problem then becomes:
- decommission streams some ranges, fails late → transferred_ranges populated
- restart to abort → mode NORMAL, pending ranges gone, transferred_ranges rows still on disk
- nodetool decommission → fail due to not enough live nodes → DECOMMISSION_FAILED and crucially since we failed early enough, isLeaving == false
- Then nodetool decommission --force → guard true (mode ≠ NORMAL) → reset skipped, stale rows honoured → CASSANDRA-16290 data loss
I wanted to keep the patch as close as possibly between 4.0/4.1/5.0, though there are plenty of test quirks and other deltas.
We could though simplify the 4.0 and 4.1 variants if desired.
Or maybe a more succinct comment expounding a bit more on the purpose of tokenMetadata.isLeaving on 5.0 and stating here for symmetry on 4.0/4.1
Jollyplum
force-pushed
the
mbyrd/CASSANDRA-16290/cassandra-4.0
branch
from
August 7, 2026 15:39
d07e94f to
f08b8b5
Compare
…ranges to force re-streaming and avoid consistency violation Patch by Matt Byrd; reviewed by Caleb Rackliffe and Sam Tunnicliffe for CASSANDRA-16290
Jollyplum
force-pushed
the
mbyrd/CASSANDRA-16290/cassandra-4.0
branch
from
August 7, 2026 23:14
f08b8b5 to
84598f3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Patch by Matt Byrd; reviewed by Caleb Rackliffe and Sam Tunnicliffe for CASSANDRA-16290