Repository navigation
Conversation
Over Vector2D<f64> counters, check_weight and try_update_weighted on CountMinSketch and CountSketch reject a non-finite weight (and a negative one for Count-Min) and a counter overflow, leaving the sketch unchanged. CountMinSketchWithHeap and CountSketchWithHeap gain try_update_weighted, which updates the heap after the checked update, and rescore_heap, which re-reads every resident's count from the counters.
zzylol
force-pushed
the
feat/f64-heap-sketch-checks
branch
from
October 20, 2026 23:41
9085765 to
c9c7e7a
Compare
CountMinSketchWithHeap and CountSketchWithHeap over Vector2D<f64> forward check_weight to the inner sketch, the way check_shape is forwarded, so a caller checks a weight against the type it updates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
ASAPPlanner's executor runs Float64-weighted top-k on
main'sCountMinSketchWithHeap<Vector2D<f64>>andCountSketchWithHeap<Vector2D<f64>>, keyed byTupleKeybytes.mainalready hascheck_shape,clearand the memory estimates (#242). Two gaps remain:update_weightedoverf64counters accepts NaN and ±inf, and Count-Min also accepts negative weights. Any of these silently corrupts every later estimate. Count-Min's estimate is an upper bound only while every update adds. A counter can also overflow to inf partway through an update, after some rows have already changed.This is what remains of the closed
WeightedFrequencystack (#160, #225, #226, #227), redone onmain's sketches.What
CountMinSketch<Vector2D<f64>, RegularPath>andCountSketch<Vector2D<f64>, RegularPath>:check_weight(weight). Count-Min accepts finite, nonnegative weights. Count Sketch accepts any finite weight, including negative ones.try_update_weighted(key, weight). A weightcheck_weightrejects, or an update that would carry a probed counter pastf64::MAX, returnsErrand leaves the sketch unchanged.CountMinSketchWithHeapandCountSketchWithHeap:check_weight(overVector2D<f64>, RegularPath): forwards to the inner sketch's, the waycheck_shapeis forwarded.try_update_weighted(overVector2D<f64>, RegularPath): the checked update, then the heap.rescore_heap()(any RegularPath storage): re-reads every resident's count from the counters and keeps the same residents. It is opt-in: call it before reading the heap.Existing methods are unchanged.
Not changed:
main'scheck_shapeaccepts more thanMATRIX_MAX_ROWS(20) rows. Past that, a row reuses another row's seed, andto_asapv1refuses the sketch. A caller that needs independent rows and an encodable state must also checkrows <= MATRIX_MAX_ROWS. ASAPPlanner's executor will.How
try_update_weightedcomputes each row's column once, plus the signed delta for Count Sketch, in aSmallVecsizedMATRIX_MAX_ROWS. It checks that every probed counter stays finite, then applies the update. So it hashes once per row, asupdate_weighteddoes.rescore_heapestimates each resident and writes the count back throughTopKHeap::update_owned. That re-sifts the resident and never evicts it. It costs one estimate per resident.rescore_heapclones each resident's key and looks it up in the index. ATopKHeapmethod that rewrites each slot's count by position and re-heapifies once in O(k), rebuilding the index, would avoid that.WeightedFrequencyrescored every candidate on each update, which costO(k)estimates per update. Here the caller pays that once per read. One difference remains: admission still compares a new key against the root's possibly stale count, asmain's heap always has.Key code interfaces added / modified
src/sketches/countminsketch.rssrc/sketches/countsketch.rssrc/sketches/countminsketch_heap.rs,src/sketches/countsketch_heap.rs(the same onCountSketchWithHeap)Usage, end to end: an executor runs one window of
topk(2, sum by (region, port) (bytes_rate))on a Count-Min plan. This was run as a test at the head commit; the output is shown in the comments.Before this PR
A NaN or negative rate in one input row silently corrupted the window's Count-Min estimates. An overflowing update could leave some rows changed and others not. A top-k read could report a resident's count from before later updates moved its cells.
After this PR
The executor checks each weight on update and gets bad input back as an
Errwith the state intact. It callsrescore_heapbefore reading top-k when it needs current scores. The state ismain's f64 heap sketch, serialized as ASAPv1.Verification
f64_checked_updates_are_all_or_nothing(both heap sketches): checks fractional and signed sums and the heap count. Bad weights and anf64::MAXoverflow returnErr, with counters and heap identical before and after. The old uncheckedupdate_weightedfails these checks: a NaN weight changes the probed cells, and a secondf64::MAXdrives the estimate to inf (to -inf for Count Sketch). This was checked with a throwaway test at the head commit.rescore_heap_rereads_every_resident(both heap sketches): 60 colliding updates in 8 columns leave some residents stale, and afterrescore_heapevery resident's count equals its estimate.cargo fmt --all -- --check: clean.cargo clippy --workspace --all-targets --locked -- -D warnings, with and without--all-features: clean.cargo test --lockedandcargo test --all-features --locked: all pass.Rebase note: rebased onto
maineb04e7f. #242 already addedcheck_shape,clearand the memory estimates, so this PR's versions of those were dropped in favor ofmain's. Onlycheck_weight,try_update_weightedandrescore_heapremain.