MINOR: Add compaction replay tests for the group coordinator - #23220
Open
izzyharker wants to merge 7 commits into
Open
MINOR: Add compaction replay tests for the group coordinator#23220izzyharker wants to merge 7 commits into
izzyharker wants to merge 7 commits into
Conversation
Adds GroupMetadataManagerCompactionReplayTest, which catches "compaction
makes a partition fail to load" bugs by replaying every realistic compacted
variant of a real group-coordinator record log through a fresh coordinator.
For each scenario it captures the records the coordinator writes (one batch
per write), then sweeps the clean/dirty compaction boundary across every batch
boundary, with tombstones retained and dropped, replaying each variant and
asserting it loads without throwing. Offset-commit records are replayed through
an OffsetMetadataManager sharing the GroupMetadataManager, so the
simple-classic-group creation on the load path is modelled faithfully.
The clean/dirty boundary is only split between batches, never inside one,
because log compaction cleans at segment granularity and a single write's
records are always compacted atomically. Splitting within a batch generates
logs real compaction can never produce (false positives).
Scenarios: rebalance, subscription change, member leave (tombstones), static
member rejoin, classic->consumer upgrade, and consumer/streams groups with an
offset commit. The last two reproduce a bug where, after compaction, the
offset commit sorts before the group records, creating a simple classic group
that the consumer/streams records must load on top of. Both fail if their
respective fixes in getOrMaybeCreatePersisted{Consumer,Streams}Group are
reverted.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… isolation Restore the fresh-coordinator-per-variant isolation in assertCompactedVariantsLoadCleanly (a Supplier, not a single reused context) so compacted variants can no longer leak state into one another. Add two scenarios that reproduce the simple-classic-group load path over a group that started classic and holds committed offsets: - testClassicToConsumerUpgradeWithOffsetCommit... (upgrade to consumer) - testClassicToStreamsMigrationWithOffsetCommit... (offline migration to streams) Both were verified to fail (IllegalStateException "not a consumer/streams group") when the respective isSimpleGroup() branch is disabled. Make the docstrings consistent: tag the streams paths to KAFKA-20254, describe the consumer paths as its pre-existing counterpart, and scope out the "still owned at epoch" path (covered by GroupMetadataManagerTest) since the clean-prefix/dirty-suffix model cannot reproduce it. Fix a typo and trailing whitespace. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the per-batch clean-prefix model with a single cleaning window over
the log: a record is compactable when it is superseded by a later record with
the same key or is a tombstone, and each variant removes every compactable
record in one contiguous stretch. This subsumes the previous prefix-only model
and, because the window can start in the middle of the log, it also produces
the missed-unassignment shape behind KAFKA-19862, which a compacted prefix
cannot express. The separate drop-tombstones flag is gone.
Two constraints keep the variants to logs a load can actually observe: the
window falls on batch boundaries, since the records of one write share a
segment and therefore share the cleaner's verdict; and a tombstone is only
dropped once no earlier record for its key survives, since dropping an
aged-out tombstone happens in a later pass than the one that collapsed the key.
Collapse the nine narrow scenarios into two lifecycle scenarios driven through
the real request paths, as reviewed: a group created on the classic protocol
that commits offsets, rebalances, is upgraded online to the consumer protocol,
rolls its classic members onto the new protocol and scales out; and a Kafka
Streams application migrated offline from the classic protocol to the streams
protocol across three processes. Both mix classic GroupMetadata records and
their tombstone, offset commits and modern group records, and move partitions
or tasks between owners repeatedly.
Verified by reverting each fix in turn and re-running:
- Streams simple-classic-group handling in
getOrMaybeCreatePersistedStreamsGroup: "Group ... is not a streams group"
- KAFKA-19862 ConsumerGroup partition epochs: "Cannot set the epoch of ...
because the partition is still owned at epoch ..."
- KAFKA-19862 StreamsGroup process ids: "Cannot remove the process ID ...
because it does not have any processId"
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reuse of the existing harness: - StreamsGroupTestUtil.staticHeartbeat/staticJoinHeartbeat now build the streams join, leave and reconcile requests. - Remove follow SyncGroup block, default to SyncGroupRequestBuilder Deduplication: - Add leaveStreamsMember, joinConsumerMember, joinStreamsMember and sleepCapturing, and fold the two reconciliation loops into a single driver. Compaction model: - compact() returns a CompactedVariant carrying the records, the cleaned window and the removed positions. - Use a single GroupCoordinatorMetrics instance. - Pass withLogContext to the OffsetMetadataManager.Builder, as production does. The captured logs are unchanged: both scenarios still produce exactly the same records, batches and compactable positions as before (69/30/52 for the consumer scenario, 51/26/42 for the streams one). Verified by reverting each fix and re-running: KAFKA-20254 fails the streams scenario, KAFKA-19862 fails both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
github-actions
Bot
added
triage
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds GroupMetadataManagerCompactionReplayTest, which tests partition
loading after compaction for scenarios involving offset commit combined
with classic->consumer/streams upgrades, as seen in
KAFKA-19862/KAFKA-20254.
Tests capture records, compact a contiguous stretch of the resulting log
(both prefix and mid-log), and replay the result(s) through a new
GroupCoordinatorShard.
testClassicGroupUpgradeToConsumerGroupWithOffsetCommit: A classicgroup has an offset commit, rebalance, consumer member join, online
upgrade (classic->consumer), new member join. Verified this test fails
when KAFKA-19862 is reverted.
testClassicGroupUpgradeToStreamsGroupWithOffsetCommit: Streams appruns on classic protocol, commits offsets, rebalances, then upgrades
offline to streams protocol. Verified this test fails when KAFKA-20254
is reverted.