[improve][pip] PIP-486: Scalable Topic Key-Shared Consumption#26077
Open
merlimat wants to merge 2 commits into
Open
[improve][pip] PIP-486: Scalable Topic Key-Shared Consumption#26077merlimat wants to merge 2 commits into
merlimat wants to merge 2 commits into
Conversation
Adds the PIP-486 design document: key-shared (per-message-key) ordered consumption on scalable topics via producer-side entry-bucketing, decoupling the producer's batching mode from the consumer's subscription mode. Discussion to follow on dev@pulsar.apache.org.
lhotari
reviewed
Jun 22, 2026
…trary bucket boundaries - The entry now carries the effective hashB range of its messages (entry_hash_min/ entry_hash_max) — the actual min/max, a tighter bound than the nominal bucket. This lets a geo-replication relay forward a batch as-is when its real contents still fit a differently-bounded destination segment, re-batching only on a genuine straddle. - Add a forward-compatibility note: the range-based wire/dispatch do not require equal-width buckets, so the controller could later place arbitrary bucket boundaries to balance buckets by traffic (as PIP-468 does for segments).
lhotari
approved these changes
Jun 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
PIP-468 and
PIP-483 give scalable topics a DAG of
range segments whose steady-state ordered-consumption model is one consumer per segment. Several
situations need key-shared (per-message-key) ordered consumption instead: draining a sealed-segment
backlog after a scale-up, consolidating many low-throughput topics, and getting consumer parallelism
beyond the max-segments ceiling.
Today, key-shared dispatch couples the producer's batching mode to the consumer's subscription mode
(the producer must disable batching or use the key-based batcher). This PIP removes that coupling for
scalable topics.
Modifications
Adds the design document
pip/pip-486.md. In summary:hashB, the low half of the same 32-bit keyhash whose high half drives segment routing) divides each segment into a configurable number of
buckets; the producer keeps each batch within one bucket and stamps the bucket's
hashBrange in theouter
MessageMetadata.per-key hashing, no decompression, one entry to exactly one consumer.
count is immutable per segment, changed only by a controller-driven "rebucket rollover" (a no-op
split reusing the PIP-468 seal/redirect flow).
Key_Sharedblocked-hash handling, tracking pending state per bucket. No shared-entry ack machinery.
This is a documentation-only PR (the design proposal). Implementation will follow in separate PRs.
A discussion thread will be started on dev@pulsar.apache.org.