Skip to content

feat(stream): support XACKDEL and XDELEX commands#3499

Closed
kirito632 wants to merge 2 commits into
apache:unstablefrom
kirito632:feat/stream-xackdel-xdelex
Closed

feat(stream): support XACKDEL and XDELEX commands#3499
kirito632 wants to merge 2 commits into
apache:unstablefrom
kirito632:feat/stream-xackdel-xdelex

Conversation

@kirito632
Copy link
Copy Markdown
Contributor

What

This PR introduces the XACKDEL and XDELEX stream commands with configurable
PEL cleanup behaviors.

It also fixes an existing issue in XINFO CONSUMERS where consumer metadata
from different groups could leak into the same scan result.

XINFO CONSUMERS Fix

Root cause:
GetConsumerInfo() sets iterate_upper_bound by incrementing only the version
while keeping the sub-key prefix (which includes the group name) unchanged.
Since the version field precedes the sub-key in the InternalKey encoding
([version][sub_key]), consumer keys from other groups can fall within the
[version, version+1) range when their group name sorts lexicographically
at or after the target group's name prefix.

Fix:
Extract and validate the group name from each internal key during iteration,
filtering out consumers belonging to other groups.

XACKDEL / XDELEX

Supported deletion strategies:

  • KEEPREF (default):
    Delete the stream entry only.

  • DELREF:
    Delete the stream entry and remove it from all groups' PELs.

  • ACKED:
    Delete the stream entry only if it has been acknowledged by all consumer groups.

Both commands return per-ID status codes:

  • 1: entry deleted
  • 2: entry retained (e.g. ACKED condition not satisfied)
  • -1: entry not found or duplicate ID within the same request

Implementation Notes

  • Pending-number decrements are aggregated in memory and flushed once
    through a single WriteBatch, avoiding repeated metadata updates during
    DELREF operations.

  • Duplicate IDs within the same request are deduplicated to preserve
    idempotency and prevent stream metadata corruption.

  • The originating command name (XACKDEL or XDELEX) is propagated into
    WriteBatchExtractor so replicas replay the exact same semantics.

  • Internal stream metadata / PEL delete events are filtered out in
    WriteBatchExtractor by skipping synthetic subkeys whose entry-id prefix
    starts with UINT64_MAX, preventing invalid replicated XDEL commands.

Testing

Added Go integration tests covering:

  • multi-group DELREF cleanup
  • ACKED blocking semantics
  • duplicate-ID idempotency
  • invalid syntax handling
  • XINFO CONSUMERS group isolation

AI-Assisted Contribution Disclosure

This contribution complies with the ASF AI-assisted contribution guidelines.

AI tools were used to assist with English phrasing and some test scaffolding.
The core C++ implementation, debugging, correctness analysis, and final verification
were independently completed by me.

@kirito632 kirito632 changed the title Feat/stream xackdel xdelex feat(stream): support XACKDEL and XDELEX commands May 24, 2026
@sonarqubecloud
Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
44.4% Coverage on New Code (required ≥ 50%)
22.8% Duplication on New Code (required ≤ 20%)

See analysis details on SonarQube Cloud

@kirito632
Copy link
Copy Markdown
Contributor Author

I noticed the SonarCloud Quality Gate failed. I am currently working on a quick update to resolve this.

@aleksraiden
Copy link
Copy Markdown
Contributor

@torwig please, review this pr, thanks

@jihuayu
Copy link
Copy Markdown
Member

jihuayu commented May 25, 2026

Hi @kirito632 Thanks for your contribution. However, I feel this PR is a bit too large. If possible, I’d suggest splitting it into two PRs, with each PR adding one command.

@kirito632
Copy link
Copy Markdown
Contributor Author

Got it, thanks. I’ll split it into two PRs as suggested.

@kirito632
Copy link
Copy Markdown
Contributor Author

Hi @jihuayu ,I have split this into #3502 (XDELEX) and #3504 (XACKDEL). Please review. Thanks!

@jihuayu
Copy link
Copy Markdown
Member

jihuayu commented May 27, 2026

@kirito632 Thanks. You can close this PR directly

@jihuayu jihuayu closed this May 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants