Skip to content

[feat][bk] Add per-read no-read-ahead hint#4772

Open
Denovo1998 wants to merge 1 commit intoapache:masterfrom
Denovo1998:per_read_no_read_ahead
Open

[feat][bk] Add per-read no-read-ahead hint#4772
Denovo1998 wants to merge 1 commit intoapache:masterfrom
Denovo1998:per_read_no_read_ahead

Conversation

@Denovo1998
Copy link
Copy Markdown

Descriptions of the changes in this PR:

Fix #xyz

Main Issue: #xyz

BP: #xyz

Motivation

Pulsar delayed delivery uses BucketDelayedDeliveryTracker when there are too many delayed messages to keep all delayed indexes in broker memory. In that mode, delayed indexes are persisted as bucket snapshot segments in BookKeeper, and the broker lazily loads the next snapshot segment entry only after the current segment has been drained by time-driven delivery.

This access pattern is a sparse point-read workload, not a normal sequential scan. A single lazy-load read for snapshot entry N can currently trigger bookie-side dbStorage_readAheadCache prefill for N+1, N+2, and later entries, but those entries may not be read until much later, or may be evicted before use. This can waste bookie IO and direct memory, and can pollute the read-ahead cache needed by ordinary sequential reads such as backlog consumption, catch-up reads, and ledger scans.

This change adds a per-read no-read-ahead hint so callers such as Pulsar can opt out only for these point-read paths while preserving the default read-ahead behavior for normal sequential workloads.

Changes

  • Add ReadOptions with ReadOptions.DEFAULT and ReadOptions.builder().disableReadAhead(true).build() for request-scoped read options.
  • Add ReadOptions overloads to ReadHandle and LedgerHandle, including legacy callback APIs and unconfirmed read APIs that share the same internal read path.
  • Propagate the hint from PendingReadOp to bookies through a new BookieProtocol.FLAG_NO_READ_AHEAD bit.
  • Extend v3 protobuf read requests with ReadRequest.readFlags so bitmask-style read flags can coexist with the existing single-value ReadRequest.flag enum.
  • Pass the no-read-ahead hint through v2/v3 read processors into Bookie.readEntry(..., noReadAhead).
  • Add bookie, ledger descriptor, and ledger storage read overloads while keeping existing methods defaulting to normal read-ahead behavior.
  • Update DbLedgerStorage / SingleDirectoryDbLedgerStorage so noReadAhead=true still uses and populates the target entry cache, but skips the extra fillReadAheadCache(...) prefill for following entries.
  • Add focused tests for storage read-cache behavior, protocol flag propagation, request processor handling, and client-side PendingReadOp flag construction.

@Denovo1998
Copy link
Copy Markdown
Author

@zymap @lhotari @merlimat @nodece @hangc0276
PTAL. This is my first contribution to the BookKeeper code repository. Please point out any issues, and thank you all for your help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant