Skip to content

Detach ingester series from gRPC buffers to reduce heap#7670

Draft
justinjung04 wants to merge 3 commits into
cortexproject:masterfrom
justinjung04:ingester-heap-improvement
Draft

Detach ingester series from gRPC buffers to reduce heap#7670
justinjung04 wants to merge 3 commits into
cortexproject:masterfrom
justinjung04:ingester-heap-improvement

Conversation

@justinjung04

@justinjung04 justinjung04 commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

What this PR does:
Detach series label and chunk data from gRPC unmarshal buffers in the ingester streaming path (distributor_queryable.go), allowing the Go GC to reclaim receive buffers to reduce heap usage. This is the ingester-side counterpart to #7519, which applied the same fix for the store-gateway streaming path.

Specifically:

  • Use FromLabelAdaptersToLabelsWithCopy instead of FromLabelAdaptersToLabels (unsafe cast that pins the buffer)
  • Copy chunk Data byte slices into independent allocations via detachChunksFromBuffer

Without this change, long-running queries hold references into gRPC receive buffers through label strings and chunk data, preventing GC from reclaiming them and causing heap to grow under heavy ingester query load.

Benchmark (100 series, 1KB chunks each):

ns/op B/op allocs/op
current (no detach) 8,000 23,416 402
this PR 31,500 161,016 902

The per-call cost of copying is ~137KB additional short-lived allocations per 100 series. The tradeoff is that gRPC receive buffers (~32KB+ each) are freed immediately after processing instead of being pinned for the entire query lifetime. Under concurrent query load, this prevents heap accumulation from unreclaimable buffers.

Which issue(s) this PR fixes:
N/A

Checklist

  • Tests updated
  • [NA] Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • [NA] docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

Signed-off-by: Justin Jung <jungjust@amazon.com>
@justinjung04 justinjung04 force-pushed the ingester-heap-improvement branch from 998479c to f04cbc8 Compare July 3, 2026 22:48
Signed-off-by: Justin Jung <jungjust@amazon.com>
Signed-off-by: Justin Jung <jungjust@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant