Repository navigation
feat(data-plane): read query-time raw windows from a local raw buffer and Prometheus remote read - #831
Draft
zzylol wants to merge 2 commits into
Draft
feat(data-plane): read query-time raw windows from a local raw buffer and Prometheus remote read#831zzylol wants to merge 2 commits into
zzylol wants to merge 2 commits into
Conversation
… and Prometheus remote read - A local raw buffer keeps the recent samples Remote Write delivered, sharded like ingestion: each ingestion worker appends its shard. It keeps the plan's longest query-time raw read back from the newest sample: the largest `end_offset + width` of `query_time_raw_windows` and of the panes built at query time and kept (their retention, which covers range evaluations). `--raw-buffer-ms` overrides it; 0 keeps none. - A raw window is read from the buffer; the part before the buffer's cut (its first sample, or the trimmed horizon of any shard) is read with Prometheus remote read (protobuf, `/api/v1/read`). The JSON query API is no longer used for raw samples. Forwarding of unplanned queries is unchanged. - The data plane logs the buffer's series and samples every 30 s.
…est query-time read The buffer kept every metric for the plan's single longest raw read: in the benefit plan, 60 minutes of the counter whose raw reads span one minute, and of metrics no raw window reads. It now keeps only the metrics the plan reads at query time, each for its own longest read (query-time raw windows, and panes built at query time and kept).
zzylol
added
enhancement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 4/4. Base: #830.
Why
Every query-time raw window, and every missing kept pane, was an instant range-selector query to Prometheus decoded as JSON. With raw data retained that was 15.5% of the data plane's CPU (146 MB of JSON per 90 minutes at 2,000 series). User decision (Stage D): query-time raw windows come from a local in-memory ring buffer, fed and sharded like ingestion and sized by the plan. Anything older comes from Prometheus remote read (protobuf), not the JSON API.
What
end_offset + widthof itsquery_time_raw_windows, and of its panes built at query time and kept (their retention, which covers the declared range evaluations). Metrics no raw window reads are not buffered.--raw-buffer-msoverrides the horizon; 0 keeps none. (A first commit used one horizon, the plan's longest read, for every metric. The second sizes it per metric, which is the plan's longest read applied to each metric; the table shows both.)/api/v1/read, snappy protobuf, sampled response). The JSON raw fetch is removed. Forwarding of unplanned queries is unchanged.How
RawBufferholds, per shard, the series' samples by metric and label set in time order, under a read-write lock, and trims each metric to its horizon. A read takes each shard's read lock, collects the samples in(start, end]and takes the latest trim point over the shards as the cut. Samples at or before the cut come from remote read, so a shard trimmed during the read cannot leave a gap.Before this PR
Raw windows: one JSON instant query per window or missing kept pane, decoded through
serde_json::Value.After this PR
A data plane that has received the samples since it started answers raw windows from memory, with no Prometheus request. Remote read covers only what predates its first sample or its horizon.
Verification
cargo test --workspacepasses, includinge2e_examples(10),answers,range,kept.e2e_examples::example1_raw_windows_from_the_buffer_and_remote_read, failing on feat(precompute): update each open pane's summary state per sample #830. Prometheus holds samples from 0 and the data plane receives them from 4:45. At 5 min the 1-minute window takes 4:15–4:45 from remote read and 5:00 from the buffer. At 10 min the window is the buffer's alone, with 0 remote reads. Every answer equals the Planner executor's over all samples, and the JSON API is never used for raw samples.raw_buffertests: reads equal the samples of the metric in(start, end]; a read past the horizon splits at the cut; samples before startup come from Prometheus.--raw-buffer-ms 0, so they count remote reads; the counts are unchanged (5, 1, 1 for B3; 1 per range query).Benefit harness
Same setup as #828. Each run writes every sample to the data plane from the start, so the buffer covers every raw window and no remote read happens.
s10
s10-noraw
s50
s50-noraw
"pr4-single-horizon" is this PR's first commit (one horizon for every metric); "pr4" is its head (per-metric horizons). "Prometheus alone" is the baseline of the pr3 run.
CPU saved against the JSON raw fetch (raw data retained; head against pr3):
Raw buffer size, at its largest, from the data plane's log. The plan's horizons are
http_requests_total1 min andlatency_ms60 min.The bytes are the samples' 16 bytes each; per-series map entries come on top. Peak RSS does not change measurably between pr3 and pr4.