[ISSUE #10987] Fix queryMinOffsetInAllGroup deleting consumer offsets from the live offset table - #10991
Conversation
…ffsets from the live offset table
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
This PR fixes a critical bug where queryMinOffsetInAllGroup was mutating the live offsetTable by deleting consumer offsets for filtered groups. The original code iterated over the live keySet and called removeConsumerOffset(), which destroyed offset data. The fix works on a snapshot of keys (new HashSet<>(this.offsetTable.keySet())) and uses removeIf on the snapshot instead. A comprehensive test verifies that filtered group offsets are preserved after the query.
LGTM — excellent fix for this data corruption bug!
Automated review by github-manager-bot
…sts so offset deletion and the malformed-key AIOOBE fail independently
Test evidence (before → after)The regression tests were verified in both directions on JDK 8 ( On the unfixed code (fix reverted, tests kept), the two regression tests now fail independently and each one demonstrates one defect:
Note the query returned while
With this PR: I split the originally single test into Side note on CI: the workflow runs for this PR are in |
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
This PR modifies 2 files (108 lines changed).
Automated scan completed. A maintainer should do a detailed review of the logic changes.
Files Changed
broker/src/main/java/org/apache/rocketmq/broker/offset/ConsumerOffsetManager.javabroker/src/test/java/org/apache/rocketmq/broker/offset/ConsumerOffsetManagerTest.java
Automated review by github-manager-bot
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Summary
This PR fixes a critical bug where queryMinOffsetInAllGroup was mutating the live offsetTable by removing filtered groups' offsets via Iterator.remove() and removeConsumerOffset(). The fix correctly works on a HashSet snapshot and uses the filtered set as a membership guard in the subsequent iteration — no more side-effects from a read-only query.
The added tests cover both the core regression (filtered group's offsets survive the query) and the edge case of malformed keys without the @ separator.
Looks good. 👍
Automated review by github-manager-bot
Which Issue(s) This PR Fixes
Brief Description
ConsumerOffsetManager#queryMinOffsetInAllGroup(topic, filterGroups)iterated the liveoffsetTable.keySet()and calledit.remove()on it to exclude the filter groups. SinceConcurrentHashMap.keySet()is a live view, running the read-only admin operationQUERY_CORRECTION_OFFSET(AdminBrokerProcessor#queryCorrectionOffset, exposed viaDefaultMQAdminExt#queryCorrectionOffset) permanently deleted everytopic@groupoffset entry of the filtered groups:persist()makes the deletion permanent (consumerOffset.json);RocksDBConsumerOffsetManager,removeConsumerOffsetdeletes the rows from RocksDB immediately;-1fromqueryOffsetand re-initialize perconsumeFromWhere→ mass duplicate consumption or skipping to max;topicAtGroup.split(TOPIC_GROUP_SEPARATOR)[1]also threwArrayIndexOutOfBoundsExceptionon a malformed key without@.This PR makes the exclusion work on a snapshot of the key set, so the query no longer mutates
offsetTableat all (and never callsremoveConsumerOffset), while preserving the original filter semantics: offsets of the filter groups are excluded from the min-offset computation. The malformed-key AIOOBE is fixed by checkingarrays.length == 2.How Did You Test This Change?
Added
ConsumerOffsetManagerTest#testQueryMinOffsetInAllGroupDoesNotDeleteOffsets:queryOffsetstill returns them) after the query;@to cover the AIOOBE.mvn -pl broker test -Dtest=ConsumerOffsetManagerTestpasses (6/6). On the unfixed code the new test fails withArrayIndexOutOfBoundsException/ observes the deleted entry.