Repository navigation
feat!: bring the sketch fixes ASAPPlanner pins onto main (#221, #222, #223, #228, #229, #233, #238) - #242
Merged
Conversation
`UnivMon::check_shape(heap_size, rows, cols, layers)` accepts exactly the shapes `UnivMon::new` builds: positive dimensions, at most UNIVMON_MAX_LAYERS layers, at most MATRIX_MAX_ROWS rows, rows within the 128-bit hash budget and a counter count that fits usize. `new` panics exactly when it fails, so it no longer builds a pyramid the ASAPv1 wire refuses. Ports #229. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d error bounds `estimate_cardinality` counts every heap key, including zero-weight keys and keys whose Count Sketch estimate is not positive. `estimate_l2` reads layer 0's Count Sketch F2 estimate in standard mode and sums the strata's F2 estimates in terminal mode; `estimate_g_sum` keeps the paper's recurrence, and its flag is now `count_every_key` (`true` counts every heap key). New `top_layer_complete()`. The L2 bound and the distinct-count bound, stated jointly with `top_layer_complete()`, are derived in docs/api/api_univmon.md, with Monte Carlo tests. Ports #221. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`UnivMonTwoTier` gets the estimators `UnivMon` has: `estimate_cardinality` counts every heap key whatever its estimate, zero-weight keys included; `estimate_l2` reads layer 0's Count Sketch F2 estimate in standard mode and sums the strata's in terminal mode; new `top_layer_complete()`. The API doc states the bounds in the two-tier layout's terms, with Monte Carlo tests. Ports #222. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`Hydra::query_group_total(key)` returns a subpopulation's estimated total mass: the minimum over rows of the subkey's cell `HydraCounter::total_mass()`, a Count-Min cell's inner row sum or a KLL cell's retained weight. Count Sketch, HyperLogLog and UnivMon counters return an error. The total is read from existing cell state, so the wire format is unchanged. `query_key` and the new query share one subkey hash helper. Ports #223. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Kll's item buffer starts empty, then holds 16 items and doubles up to max_capacity; a full buffer compacts only at max_capacity, so retained items, coin tosses, quantiles, merges and ASAPv1 bytes match the eagerly allocated buffer for the same input and seed. The merge scratch buffer starts empty; merge, from_portable_state and the ASAPv1 decoder size the buffer to what they hold. lazy_allocation_matches_eager compares every level and the coin with a test-only eager copy (k = 8 to 1024, up to 50,000 items, with a merge). Ports #238. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both report the bytes the sketch holds now: Kll its struct, item buffer, level array, merge scratch space and memoized CDF; KllDynamic its struct, item buffer and level bounds. Ports #238. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… reject past the k-th hash Kmv::estimate takes &self and Kmv::merge takes &Kmv. Kmv::memory_bytes is the struct plus the hash slots the heap has allocated (CommonHeap::allocated_slots, crate-private). A full sketch turns away a hash at or above its largest retained one before scanning the retained hashes for a duplicate; the retained set and every estimate are unchanged. Ports #228. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TopKHeap::estimated_memory_bytes(heap_size, key) is the bytes a heap of that capacity holds in steady state once keys shaped like key churn through it: the struct, entry and digest arrays, the digest index (std HashMap layout, grown once more when tombstones force a resize of an index over half full) and the owned key bytes. A heap holds at most this. UnivMonQ::estimated_memory_bytes(&self) becomes memory_bytes(&self) and counts what the instance holds: its counters, candidate tables (bucket count from the largest capacity() seen), candidate heaps and ordered-sample heap. tests/heap_memory_bytes.rs checks both against a counting global allocator: the heap estimate as an upper bound after a fill and exactly after churn, UnivMon-Q exactly. Ports #233 (the heap half). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…x sketches CountMinSketch, CountSketch and their WithHeap and WithSpaceSaving forms gain check_shape(rows, cols), which accepts the ASAPv1 geometry, 1 <= rows <= MATRIX_MAX_ROWS and 1 <= cols <= u32::MAX, on every hashing path. with_dimensions, new and from_storage panic on any other shape. Past 20 rows the regular path and the fast path's per-row (Rows) layout hash row r + 20 with row r's seed; the packed layouts reuse no seed. Tests that built such a shape to watch the encoder refuse it check check_shape instead, Hydra's included. clear() zeroes the counters (and empties the heap for the WithHeap types) and returns the Nitro sampler to its start (Nitro::reset, through the new MatrixStorage::reset_sampler), keeping shape, allocation and admission rate. TopKHeap::memory_bytes reports the bytes a heap holds, taking the digest index's bucket count from the largest capacity() it has reported, which evictions' tombstones cannot lower. The WithHeap types add memory_bytes and estimated_memory_bytes(rows, cols, heap_size, key). tests/heap_memory_bytes.rs checks all of them against a counting allocator, after a fill and after 50 times heap_size evicting keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…w count Seed index i hashes with seed_for_index(i) = SEED_LIST[i % 20] * (i / 20 + 1) + i / 20 in wrapping u64 arithmetic; indexes below 20 keep their seed. The matrix hash passes seed indexes through unreduced, so rows past 20 on the regular path and the fast path's per-row layout hash independently, and UnivMon-Q's occurrence priorities hash at their own seed. check_shape on Count-Min, Count Sketch and their WithHeap and WithSpaceSaving forms refuses only a zero dimension and a counter count past usize; Elastic, Hydra and UnivMon build any row count. to_asapv1 and from_asapv1 keep the MATRIX_MAX_ROWS bound. The UnivMon-Q residual-CDF accuracy test hashes each of its 32 sketches at its own seed index. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hout type parameters CountMinSketch and CountSketch gain memory_bytes(&self): the struct plus its rows * cols counters, exact for Vector2D storage, checked against a counting allocator in tests/heap_memory_bytes.rs. CountMinF64 is re-exported from sketches and the crate root. check_shape on CountMinSketch, CountSketch, their WithHeap and WithSpaceSaving forms is defined on the type with its default parameters, so CountMinSketch::check_shape(3, 64) resolves without naming storage, path or hasher. A call naming non-default type parameters no longer compiles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ASAPv1 seed_derivation string names seed_for_index. Bytes carrying "seed_list_index_wrap" no longer decode. The golden submodule moves to the regenerated fixtures (sketchlib-golden-bytes#3). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GordonYuanyc
marked this pull request as ready for review
October 20, 2026 20:16
3 tasks
This was referenced Oct 20, 2026
zzylol
pushed a commit
to ProjectASAP/ASAPPlanner
that referenced
this pull request
Oct 21, 2026
refactor: follow asap_sketchlib's UnivMon, Hydra and HLL renames UnivMon fields rows/cols/layers/total_weight/layer_heaps, new/clear/ update_weighted/estimate_*, OwnedDataInput, UNIVMON_MAX_LAYERS; Hydra rows/cols and HydraCounter::CountMinSketch; HLL update_with_hash. refactor!: HLL, KLL, DDSketch, Count-Min and KMV kernels on native asap_sketchlib types The portable HllSketch, KllSketch, DdSketch and CountMinSketch are gone from asap_sketchlib. HLL is a classic HyperLogLogImpl at each precision 4..=18 (register types from impl_hll_registers!); KLL is Kll::with_k; DDSketch the native DdSketch (get_value_at_quantile, get_count); plain Count-Min a regular-path CountMinF64 that adds every weight, zero and negative ones included; KMV the native Kmv. Memory charges come from Kll::memory_bytes, Kmv::memory_bytes and CountMinSketch::memory_bytes. The DDSketch ratio tests interpolate PromQL quantiles from the sketch's order statistics. refactor!: weighted frequency on asap_sketchlib's f64 Count-Min and Count Sketch asap_sketchlib drops WeightedFrequency. The Planner kernel keeps its API over CountMinSketchWithHeap / CountSketchWithHeap (capacity > 0) or a plain CountMinSketch / CountSketch (capacity 0), all Vector2D<f64> on the regular path. An identity tuple is one TupleKey, hashed and retained as bytes. Planner checks what sketchlib does not: finite weights (and nonnegative for Count-Min), finite Float64 identities, an odd Count Sketch depth, and a total weight magnitude within f64. Top-k rows are the heap's keys at their current estimate, descending, ties by key bytes. refactor!: HydraKLL on asap_sketchlib's Hydra with KLL cells (#644) HydraKllSketch is gone from asap_sketchlib. The kernel is a one-column Hydra over HydraCounter::Kll, read by Hydra's median of the cells' KLL quantiles, and charged the cells' Kll::memory_bytes. refactor: box the weighted frequency state held as a template Its heap makes it far larger than the other group-state kinds. test: fixtures that still exercise native KLL memory and DDSketch quantiles Kll allocates its item buffer as items arrive, so each stored-pane state takes 1,100 samples to hold its full buffer (about 11 kB) again; with one sample the merge fit the 4 MiB limit. DdSketch reads its minimum and maximum exactly, so the cross-zero ratio fixture interpolates interior order statistics: [-2, -1, 1.011, 2] instead of [-1, 1.011]. chore!: pin asap_sketchlib to main (de15d66) asap_sketchlib main, with ProjectASAP/asap_sketchlib#242 merged. UnivMon builds any row count its 128-bit key hash covers, so the UnivMon shape tests probe that budget instead of a 20-row limit. Merge remote-tracking branch 'origin/main' into chore/asap-sketchlib-main fix: cap UnivMon at the 20 rows its ASAPv1 state encodes sketchlib main's UnivMon::check_shape accepts any row count within the 128-bit key hash, but its ASAPv1 encoder still refuses more than MATRIX_MAX_ROWS rows. Keep the executor's builds and the Planner's hash_independent at 20 rows, as on the old pin, so every UnivMon state the executor builds can be stored. fix: count the KLL and weighted frequency structs once in memory charges Kll::memory_bytes and the heap sketches' memory_bytes already include their own struct; adding size_of::<Self>() on top counted it twice. chore: pin asap_sketchlib to release 0.4.0 (eb04e7f) eb04e7f is the 0.4.0 release commit on top of de15d66; the library code is the same, only the version, CHANGELOG and README change. Merge remote-tracking branch 'origin/main' into chore/asap-sketchlib-main # Conflicts: # crates/plan-selection/src/lib.rs fix: clippy on main's new code and the renumbered P99 pane candidate Planner main fails clippy (-D warnings) and one runtime_install test since #787, #789, #795 and the joint search commit: - summary_coverage.rs: a Window alias for the relative time window (type_complexity). - workload_candidates.rs: an array instead of vec! (useless_vec). - query_candidates.rs: OperatorNode::reachable instead of a closure. - stage1_to_stage3.rs: allow too_many_arguments on search_stage2_and_stage3, as lib.rs already does once. - joint_search_equivalence.rs: struct initializer for the settings, and slice::from_ref instead of a clone. - runtime_install.rs: the merged-pane DDSketch candidate of the p99 avg workload is now P3-m1 (was P5-m1); every other assertion of that test passes unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
ASAPPlanner pins
planner-pin/sketch-library-fixes(3f44cf1): an old main plus open PRs #160, #221, #225–#230. Those PRs conflict with main after #232 and #236–#241 (renames, legacy formats removed), so Planner cannot move to main. This PR brings their functionality onto main under the current names. The original PRs stay open until Planner moves off the pin.What
One commit per unit; ported commits keep their original author.
fe3f930UnivMon::check_shape(heap_size, rows, cols, layers);newpanics exactly when it failsb77175dtop_layer_complete(), error bounds indocs/api/api_univmon.md.estimate_g_sum's flag iscount_every_key93d9df1UnivMonTwoTier9d356c9Hydra::query_group_total,HydraCounter::total_mass; bound stated for Count-Min cellsf693c6834fb4c5Kll::memory_bytes,KllDynamic::memory_bytes23f7deeKmv::estimate(&self),merge(&mut self, &Kmv),memory_bytes, early reject past the k-th hashdc729feTopKHeap::estimated_memory_bytes,UnivMonQ::memory_bytes(renamed fromestimated_memory_bytes, now what the instance holds)06e1fcacheck_shape(rows, cols),clear, WithHeapmemory_bytes/estimated_memory_bytes,TopKHeap::memory_bytes,Nitro::reset4ec1de6ihashes withSEED_LIST[i % 20] * (i / 20 + 1) + i / 20(wrapping); indexes below 20 keep their seed. Matrix sketches take any row count in memory;to_asapv1/from_asapv1keep the 20-row limit. ASAPv1 goldens are unchangedc2450a4memory_byteson plain Count-Min / Count Sketch,CountMinF64at the crate root,CountMinSketch::check_shape(3, 64)without type parameters52006eeseed_derivationis"seed_list_index_scaled"; golden submodule moves to sketchlib-golden-bytes#3Not ported:
WeightedFrequency(feat: own Float64 weighted CMS and CountSketch kernels in sketchlib #160, feat(weighted_frequency): public point estimate and heap-less capacity 0 #225–perf(weighted_frequency): rescore retained candidates without copying them #227, the WF half of feat(heap): estimated_memory_bytes for HHHeap and WeightedFrequency #233). Main's f64CountMinSketchWithHeap/CountSketchWithHeapwithTupleKeykeys cover it: on Zipf streams (5×2048, k=100) top-100 recall and relative error match, and an update costs 0.22–0.38 µs vs 22–29 µs. ItsASAP-WFREQ-1bytes had no reader.06e1fcaadds what Planner used it for.Before / After
Before: ASAPPlanner main against sketchlib main fails with 68 errors (
executor,logical-asap-optimization).After: a local ASAPPlanner branch on this PR's tip builds, and its 1767 tests pass.
Key code interfaces added / modified
UnivMon::check_shape,UnivMon::top_layer_complete,UnivMonTwoTier::top_layer_complete,Hydra::query_group_total,HydraCounter::total_mass,Kll::memory_bytes,KllDynamic::memory_bytes,Kmv::memory_bytes,TopKHeap::{memory_bytes, estimated_memory_bytes},check_shape/clearonCountMinSketch,CountSketchand their WithHeap and SpaceSaving forms, WithHeapmemory_bytes/estimated_memory_bytes,Nitro::reset,MatrixStorage::reset_sampler.estimate_g_sum(heavy_only)→estimate_g_sum(count_every_key)with the meaning oftrueinverted;Kmv::estimate/mergetake&self/&Kmv;UnivMonQ::estimated_memory_bytes→memory_bytes; seed indexes of 20 and above hash with new seeds (UnivMon-Q's occurrence priorities among them);check_shapeis defined on each type's default parameters;seed_derivation"seed_list_index_wrap"→"seed_list_index_scaled"(old bytes no longer decode).Tests
cargo fmt --all --check,cargo clippy --workspace --all-targets --all-features --locked -- -D warnings(and without features),RUSTDOCFLAGS="-D warnings" cargo doc --all-features --no-deps --locked,cargo test --all-features --locked --tests: all pass (lib 1107, e2e 594, pbt 407, asapv1_golden 71, others).🤖 Generated with Claude Code