Skip to content

KAFKA-20768: Add transactional and IQ isolation to RocksDB window and session stores#22754

Open
nicktelford wants to merge 1 commit into
apache:trunkfrom
nicktelford:KIP-892/txn-rocksdb-window-session
Open

KAFKA-20768: Add transactional and IQ isolation to RocksDB window and session stores#22754
nicktelford wants to merge 1 commit into
apache:trunkfrom
nicktelford:KIP-892/txn-rocksdb-window-session

Conversation

@nicktelford

@nicktelford nicktelford commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

The persistent RocksDB window and session stores were the last KIP-892
state stores without transactional support or isolation-level
interactive queries: a READ_COMMITTED IQ against them returned writes
that were still staged in an uncommitted transaction.

Because each segment is itself a RocksDBStore, the write and commit
path already stages writes and flushes them per segment. This change
fills the two remaining gaps in AbstractRocksDBSegmentedBytesStore:

  • Staged positionput now stages the Position of an
    uncommitted write in a pendingPosition, which is merged into the
    committed position atomically with the per-segment buffer flush on
    commit. getPosition() reflects staged writes (for the owner's own
    view and READ_UNCOMMITTED queries), while the new
    getCommittedPosition() excludes them to bound READ_COMMITTED queries.
  • Isolation-aware reads — a readOnly(IsolationLevel) view over the
    segmented store, exposed to RocksDBWindowStore/RocksDBSessionStore
    so their readOnly(...) views (and StoreQueryUtils dispatch) hide
    uncommitted writes under READ_COMMITTED.

Reads follow the same convention as RocksDBStore and the in-memory
stores: the owner (stream thread) reads live through the vanilla store
methods, while interactive queries read through
readOnly(IsolationLevel). The transaction buffer's own owner-thread
check keeps the owner's reads lock-free and snapshots non-owner
READ_UNCOMMITTED reads under the read lock; READ_COMMITTED bypasses the
buffer to read committed data directly. Consequently no new isolation
surface is added to the SegmentedBytesStore interface — the isolation
methods are private utilities shared between the vanilla methods and the
read view — and RocksDBWindowStore/RocksDBSessionStore wrap the
concrete AbstractRocksDBSegmentedBytesStore so they reach that view
without casting.

This covers the standard and timestamped window stores, the session
store, and their WithHeaders variants by inheritance. The time-ordered
/ dual-schema segmented stores (used by stream-stream joins) are out of
scope and remain non-transactional; they will be addressed as a
follow-up.

Testing: AbstractRocksDBSegmentedBytesStoreTest gains transactional
cases parameterised over both the window and session key schemas (so
they run against RocksDBSegmentedBytesStore,
RocksDBTimestampedSegmentedBytesStore, and the with-headers variant),
asserting that READ_COMMITTED hides staged writes while READ_UNCOMMITTED
exposes them, that both levels converge after commit, that the committed
position excludes staged writes until commit, and that a
non-transactional store reads identically across levels. The end-to-end
IQv1/IQv2 isolation-level integration tests live on a separate branch
and pass with this change.

🤖 Generated with Claude Code

Reviewers: kkgounder95-code (github:kkgounder95-code)

… session stores

The persistent RocksDB window and session stores were the last KIP-892
state stores without transactional support or isolation-level
interactive queries; a READ_COMMITTED interactive query against them
returned uncommitted data.

Because each segment is a RocksDBStore, the write/commit path already
stages and flushes per segment. This change fills the two remaining gaps
in AbstractRocksDBSegmentedBytesStore: it stages the Position of
uncommitted writes, merging it into the committed position atomically
with the segment-buffer flush on commit, and adds an isolation-aware
read view. Reads follow the same convention as RocksDBStore and the
in-memory stores: the owner (stream thread) reads live via the vanilla
methods, while interactive queries read through readOnly(IsolationLevel).
The transaction buffer's own owner-thread check handles the non-owner
READ_UNCOMMITTED snapshot, and READ_COMMITTED bypasses the buffer to read
committed data. RocksDBWindowStore and RocksDBSessionStore now wrap the
concrete AbstractRocksDBSegmentedBytesStore so they can reach its read
view without casting.

The end-to-end interactive-query isolation integration tests live on a
separate branch; the persistent window and session cases there pass with
this change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added triage PRs from the community streams labels Jul 3, 2026
@nicktelford

Copy link
Copy Markdown
Contributor Author

@bbejeck

@github-actions github-actions Bot removed the triage PRs from the community label Jul 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants