Repository navigation
fix(executor): DDSketch interpolated quantiles and top-k membership on sketchlib planner-pin/0.4.0-plus - #803
Merged
Merged
Conversation
zzylol
force-pushed
the
fix/sketchlib-0.4.0-plus-followups
branch
from
October 21, 2026 07:30
9754da7 to
bf2d390
Compare
…e_cont do The DDSketch kernel read sketchlib's nearest rank ceil(q n). PromQL quantile_over_time and SQL percentile_cont interpolate linearly between the order statistics at ranks floor(q (n - 1)) and ceil(q (n - 1)), so answers missed the exact ones by more than alpha: [20, 40] at q = 0.9 read 40 against 38. The kernel now reads DdSketch::quantile_interpolated (sketchlib #248). An empty sketch still gives no answer, and a q outside [0, 1] (NaN included) is still an error; the Planner never sends one, since Pass 1 prunes the sketch candidate (QuantileOutsideUnitInterval) and the exact plan answers +-Inf. The Planner's DDSketch rule (ddsketch_relative_error_alpha_v1) states alpha relative error on the value read and assumes no rank convention; the interpolated read is within alpha when both neighbours share a sign, which the sign rule already requires before the bound composes. The bound is unchanged. The two integration tests that rebuilt interpolation from nearest-rank reads now call quantile_interpolated.
sketchlib's heap rescores only the key an update touches, so a resident that shares a cell with it keeps a stale count, and a new key could be admitted against that stale minimum and evict a resident whose current estimate is higher. The old kernel rescored every candidate after every update. Each heap update now writes the counters, calls rescore_heap (sketchlib #245), and only then offers the updated key, so admission compares against current estimates; O(heap_size) per update, as before. rows() already reads every retained key at its current estimate, and a merge re-estimates both heaps' keys, so the readout needs no extra pass. A reference-model test (written first, failing at update 15 without the rescore) replays colliding Count-Min and Count Sketch streams and checks after every update that the retained keys match rescoring every candidate, up to exact ties. Weight checks now use sketchlib's check_weight and try_update_weighted: the counters stay finite per probed cell, as in the old kernel, instead of the Planner-side bound on the summed weight magnitude, so a merge checks the merged counters instead.
zzylol
force-pushed
the
fix/sketchlib-0.4.0-plus-followups
branch
from
October 21, 2026 07:43
bf2d390 to
8eec7ba
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#783 moved the executor to sketchlib main. Two kernel answers then drifted from what PromQL and the old kernels returned:
⌈q·n⌉. PromQLquantile_over_timeand SQLpercentile_continstead interpolate between adjacent order statistics. On the coverage run, 26 of the 90 quantile queries missed their exact answers by more than the 5% target (max 0.0517).The fixes need sketchlib PRs that are still open, so this PR pins a branch that carries them.
What
asap_sketchlibis pinned toplanner-pin/0.4.0-plus(63e5655): release 0.4.0 (eb04e7f) plus sketchlib QueryExpr: merge BinaryOp into Compare/Arithmetic (Instance 2 of #220) #245, refactor(types): collapse QueryExpr::PromqlScalar into Literal (#220) #246, feat(asap-aware-mapping): optimization-applicability rule framework #247 and feat(sketch): posterior (query-time) error estimation for CMS/CU-Sketch/Count-Sketch (Chen et al. IMC'21) #248. The pin moves to a release once those merge.Quantileestimate readsDdSketch::quantile_interpolated(feat(sketch): posterior (query-time) error estimation for CMS/CU-Sketch/Count-Sketch (Chen et al. IMC'21) #248). An empty sketch still gives no answer. Aqoutside [0, 1], NaN included, is still an error; the Planner never sends one, because Pass 1 prunes the sketch candidate (QuantileOutsideUnitInterval) and the exact plan answers ±Inf.rescore_heap(QueryExpr: merge BinaryOp into Compare/Arithmetic (Instance 2 of #220) #245), and only then offers the updated key. Admission therefore compares against current estimates, at O(heap_size) per update, the same as the old kernel.rows()already reads every retained key at its current estimate, and a merge re-estimates both heaps' keys, so the readout needs no extra pass.check_weight/try_update_weighted. Counters stay finite per probed cell, as in the old kernel, replacing refactor!: summary kernels on asap_sketchlib main (de15d66) #783's bound on the summed weight magnitude. A merge checks the merged counters.How
inner.quantile_interpolated(q)?. This turnsOk(None)into the empty-population error, and sketchlib's error for an out-of-rangeqpasses through.WeightedFrequency::update_keyusestry_update_weightedon the inner Count-Min or Count Sketch, thenrescore_heap(), thenheap_mut().update(key, estimate). Plain sketches (capacity 0) usetry_update_weighteddirectly.ddsketch_relative_error_alpha_v1) states alpha relative error on the value read and assumes no rank convention. The interpolated read is within alpha when both neighbours share a sign or one of them is zero. Since fix(executor): a DDSketch takes zero and negative samples #802 a DDSketch also takes negative samples, and an undeclared gauge's quantile keeps its relative bound (throughmax, for example). When the two neighbours have strictly opposite signs, the error is bounded only in absolute terms,alpha · ((1 − f) |lower| + f |upper|). For example, the median of -100, -100, 99, 99 reads -0.748 against -0.5. The nearest-rank read was further off there (-100), so this PR narrows the gap rather than opening it. A DDSketch quantile's relative bound fails when the interpolated neighbours straddle zero #807 tracks the guarantee. The bound is unchanged.Key code interfaces added / modified
Cargo.tomlcrates/executor/src/summary_kernels/weighted_frequency.rsThere are no other interface changes. The DDSketch kernel keeps its signatures; only the value it returns changes.
Usage. The coverage harness plans each random-workload quantile query (for example
seed-862 workload-01 q03-quantile) with--no-raw --failure-probability 0.3. Plan selection picks a per-series DDSketch. At each check step the executor'sSummaryEstimatecallsDDSketchAccumulator::estimate(Quantile { q }), which now readsquantile_interpolated. The harness compares the answer with the exact PromQL quantile: the max relative error is 0.0500 here, against 0.0517 before.Before this PR
f64::MAX, even when the weights cancelled.After this PR
f64::MAXis refused, and the state is unchanged.Verification
dd_sketch::quantile_interpolates_between_adjacent_order_statistics: [20, 40] at q = 0.9 is within alpha of 38. It failed on nearest rank and passes now.weighted_frequency::membership_matches_rescoring_every_candidate_after_every_update: a reference model replays colliding Count-Min and Count Sketch streams. Without the rescore it fails at update 15 ([15, 4, 6, 23]vs[15, 1, 4, 6]); with it, it passes.invalid_updates_leave_the_state_unchangednow expects the refusal from a counter overflow (the same key's cells atf64::MAXtwice), not from the summed magnitude.quantile_interpolated.factory::ddsketch_domain::quantiles_over_zero_and_signed_samplesexpected nearest-rank values. It now expects PromQL's interpolated ones (0, 0, 3, 5 at q = 0.5 is 1.5; −7, −3, 0, 4 at q = 0.25 is −4). It failed with the old expectations after the rebase.cargo test --workspace --lockedafter rebasing onto main6849b1f1(with test(planner): assert the undeclared-gauge p99 outcome, not a candidate id #801, fix(executor): a DDSketch takes zero and negative samples #802 and fix(planner): Stage 3 rejects plans the executor cannot compile; no PromQL top-k heap over its instant vector #805): 1,824 passed, 0 failed, 36 ignored.asapquery_random_workload, the coverage script's flags, all 90 PromQL quantile queries in seed-862, seed-862-x5 and seed-862-long, as is): max relative error 0.04996, none above 5%. The earlier run on825504e6had max 0.0517, with 26 above 5%.Follow-up: move the pin to a sketchlib release once #245–#248 merge.