Add pulled Bloom summary peer lookup - #1087
Merged
Merged
Conversation
Test Impact PlanDeterministic summary of how this PR changes tests, CI runners, and coverage-risk signals. Summary
Signals
Coverage risk: neutral or increased No coverage-reduction warnings detected. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in pulled-summary peer lookup mode while keeping
CACHE_PEER_LOOKUP_MODE=probeas the default/current behavior.In
summarymode, each cache proxy maintains a fixed-layout counting Bloomfilter incrementally on cache insertion and eviction. It serves an immutable,
versioned snapshot through
GET /cache/summary; receivers pull a deterministicsubset of peers that fits their configured memory budget. Bloom filters are
used only to eliminate definite-negative peers. Every positive or uncovered
candidate is exact-confirmed through bounded parallel
/cache/hasrequestsbefore one
/cache/getis sent to a confirmed holder.This avoids fleet-wide request fanout while preserving safe origin fallback.
No cache bodies are proactively replicated, and raw URLs, ranges, object paths,
organization identifiers, and cache locators are never included in summaries.
Bounds and lifecycle
CACHE_MAX_ENTRIES=1000000: soft/convergent local index target, alongsidethe existing 80% disk target.
CACHE_SUMMARY_MEMORY_LIMIT_BYTES=536870912: reserves the local countingfilter, overlapping snapshots, four bounded pulls, headers/bodies, and the
retained remote filters. The default fits 415 peer filters.
CACHE_PEER_MAX_PROBES=5: request-wide confirmation cap, shared acrossmissing blocks.
CACHE_MAX_PEER_PROBES_IN_FLIGHT=64: pod-wide non-blocking semaphore; excessconfirmations skip peer work and use origin rather than queueing.
45-second TTL. Pulls use four workers, two-second request deadlines, a
15-second fair rotating cycle deadline, bounded headers, and cancellation on
shutdown.
peer block GETs for the entire client request.
When a new peer appears, membership selection is recomputed and a selected peer
is pulled immediately. Until a valid summary arrives, it remains uncovered and
is eligible only for bounded confirmation. A peer outside the memory-selected
subset also remains uncovered. When a peer disappears or is deselected, its
summary is removed and in-flight receipts are revalidated so stale state cannot
be reinserted. Pull failure retains the last valid record only until its
advertised TTL.
See the design and operator runbook
for sizing, false-positive math, failure behavior, and recovery.
Rollout
default
probevalue.CACHE_PEER_LOOKUP_MODE=summaryin non-production first.peers uncovered; confirmation remains capped and requests safely fall back
to origin.
expectation (
N * p, capped at five), confirmed peer GET usefulness, peerbytes, origin bytes/latency, summary age/coverage, pull outcomes, resident
summary memory, and process RSS.
CACHE_PEER_LOOKUP_MODE=probe; cache contents andlayout are unchanged and must not be deleted.
Known limitation: a fully covered Bloom-negative key goes directly to origin,
so a key inserted or entering an in-flight fill after the last snapshot can be
missed until the next pull. This is a locality loss, not a correctness failure.
Validation
just test-cache-proxygo test -race ./cmd/cache-proxy/... -count=1just test-unitjust lint