Repository navigation
feat(asap-physical-operators): execute HydraCms grouped counts and filtered aggregates - #695
Merged
Merged
Conversation
zzylol
force-pushed
the
stack/g7a-hydra-filtered-executor
branch
from
October 15, 2026 03:39
12328f1 to
79792ef
Compare
…ltered aggregates HydraCms kernel over asap_sketchlib's Hydra: one shared grid of Count-Min cells for all groups, built by a new SharedSummaryBuild operator from a unit-count SummaryAgg with one item column. Each group's row state reads its item frequencies (Hydra's median estimator) and merges with the same group's state. A group total is not answered: sketchlib's Hydra has no group-mass query. Filtered aggregates: per-measure filters on Aggregate and a row filter on every summary build. A row passes only when the predicate is true; a group with no passing row keeps its row (count 0, SUM/AVG/MIN/MAX and quantiles NULL). Empty KLL and DDSketch states report is_empty, so a nullable quantile reads NULL. A filtered SummaryAgg runs only at query time: precompute compilation and the binding of an ingestion-timed SummaryAgg reject it (validate_ingestion_time_summary), so ingestion-time eligibility can rely on it.
…oup-total rejection A HydraCms SummaryAgg now accepts a weight column proven non-negative (e.g. bytes per item per group) besides the unit count. Weights go through asap_sketchlib's Hydra update count, which adds i32 weights, so each row's weight must be a non-negative integer within i32: a negative, fractional or non-finite weight fails the build with a clear error, and a NULL weight adds nothing. A weight of unknown sign is rejected at binding. A HydraCms group total is rejected with a message naming the missing sketchlib group-mass query.
… nullable A KLL quantile over a nullable column errored for a group whose values were all NULL, and an ungrouped one over no row errored too, failing the whole query, because only a filtered SummaryAgg's estimate was declared nullable. Over a relation, a quantile Aggregate and the SummaryEstimate realizing it are now nullable when the summarized column is nullable or the reduction is global, so the empty state reads NULL as in SQL.
… reserve shared grids once Every group row of a HydraCms build points at one shared grid, but each row reserved the whole grid and SummaryMerge merged the grid once per group, so a few hundred groups exceeded the memory limit. A state shared by several rows now reports itself through AggregateCore::shared_state; batches and SummaryMerge reserve each shared grid once, and SummaryMerge merges each distinct pair of grids once, sharing the result across the pair's groups.
…es identity A filtered MIN/MAX over PromQL series identity read NULL into a non-nullable output. PromQL emits no per-measure filters, so they are now rejected there. Also document that planner-side timing validation does not apply the query-time-only rule for filtered SummaryAggs yet.
zzylol
force-pushed
the
stack/g7a-hydra-filtered-executor
branch
from
October 15, 2026 04:41
79792ef to
9169be1
Compare
This was referenced Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
G7a is the executor side of grouped and filtered counts (part of #509 and #580). The planner side, which offers HydraCms in Pass 1, is G7b and lands after #690. Before Pass 1 can emit a shared Hydra grid or a filtered
SummaryAgg, the executor has to run both. G8 also needs a guarantee to rely on: a filteredSummaryAggnever runs at ingestion time.Supersedes #600 (HydraCms kernel) and #607 (filtered aggregates). Both stay open for reference.
What
HydraCms grouped counts and weighted sums. A
SummaryAggwithSharedMultiSubpopulation { kind: HydraCms }grouping now executes. It builds one shared grid for all groups and emits one row per group. That group's row answers a valuedPointCount, which is an item's count, or its summed weight, within the group. Two of these states merge withSummaryMerge. The plan contract it binds:Sketch(Cms{width,depth}, …), wherewidthanddepthmatch theHydraParams;Constant(1.0)withUnitCount(a count), or a weight column (Float64 or Int64) with aNonNegativeweight domain, e.g. bytes per item per group. Each row's weight must be a non-negative integer withini32, a NULL weight adds nothing, and an unknown-sign weight is rejected at binding;Reduce(group keys).Error and cost. A weighted point count overestimates by at most the Count-Min bound over the stream's total weight (‖stream‖₁), not its row count, at both Hydra levels. The cost per row is unchanged: one
insert_manywith the row's weight.Filtered aggregates. Per-measure filters on
Aggregateand a row filter on aSummaryAggnow execute with SQLFILTER (WHERE …)semantics:trueupdate the measure;A filtered
SummaryAggruns only at query time. Both precompute compilation and the binding of anySummaryAggtimedIngestionTimereject it.A quantile over a relation is nullable when it can have no value. This covers a nullable summarized column and an ungrouped reduction, in addition to the filtered case. In both the
Aggregateand theSummaryEstimaterealizing it, an all-NULL group or a global quantile over no row now reads NULL instead of failing the query.Measure filters over PromQL series identity are rejected. PromQL emits none, and its outputs are non-nullable.
How
summary_kernels/hydra_cms.rswrapsasap_sketchlib::HydrawithHydraCounter::CMcells. It has no local copy of sketchlib logic.update(.., count: Option<i32>), which calls the cell's Count-Mininsert_many. Because those weights arei32, a negative, fractional or non-finite row weight fails the build with a clear error.i32counters.SharedSummaryBuildoperator builds the grid.HydraCmsGroupis the row state: anArcto the grid plus the group it reads.factory::create_hydra_cmsbuilds the grid. The capability checks cover the update, the native family, the evaluation and the state payload.Aggregategainsfilters(with_measure_filters). The per-group, keyed and shared summary builds gainfilter(with_row_filter). Both fields are omitted from serialization when absent, so existing plans are unchanged.AggregateCore::is_empty(KLL, DDSketch) lets a nullable quantile read NULL for an empty state. An empty KLL quantile is now an error, not 0.0.capability::validate_ingestion_time_summaryis the single ingestion-time check.AggregateCore::shared_statelets a row report a state it shares with other rows (the Hydra grid).Batch::bytesandSummaryMergereserve each shared grid once, not once per group.SummaryMergemerges each distinct pair of grids once, memoized on the pair ofArcs, and every group of the pair reads the one merged grid.ir/asap.rs,ir/non_asap.rs): quantile nullability follows the rule of SUM over a relation. It is nullable for a filtered measure, a nullable input column, or a global reduction, and the same rule is seen throughSummaryMerge.SummaryAgglowering moved, unchanged, intophysical_planner::summary_build, so the filter is applied in one place. The binding-time rejections of filtered aggregates, of filtered summary updates and of all shared grouping are deleted.Key code interfaces added / modified
crates/asap-physical-operators/src/summary_kernels/hydra_cms.rs,mod.rs,factory.rscrates/asap-physical-operators/src/summary_kernels/traits.rs,datasketches_kll.rs,dd_sketch.rscrates/asap-physical-operators/src/operators/summary/mod.rs,operators/aggregate/mod.rscrates/asap-physical-operators/src/operators/mod.rs,operators/unchecked.rs(serialized operator format; existing plans serialize unchanged)crates/asap-physical-operators/src/capability.rscrates/types/src/ir/asap.rs,non_asap.rsUsage, end to end (the plan a HydraCms
SummaryAggbinds, as intests/hydra_cms_execution.rs; numbers from a scratch run of that harness at1b2899a5). Pass 1 does not emit this node yet (G7b), so step 1 writes the plan node a planner candidate will carry:Before this PR
Using the table
events(g, x, y): group a has x values 1, -1, 2; b has -3, 0; c has 5.SELECT g, COUNT(*) FILTER (WHERE x > 0) … GROUP BY g(exact, through the planner pipeline)filtered summary update has no native implementationSELECT g, SUM(x) FILTER (WHERE x > 0) … GROUP BY gSELECT g, COUNT(y) … GROUP BY g(nullabley)SummaryAggwith HydraCms grouping (count or weighted)shared summary grouping has no registered kernelAfter this PR
COUNT(*)SUM(x)COUNT(y)count of service="checkout" by (job)SummaryMergeequals one build over the whole stream.PointCountwith no item)HydraCms does not answer a group total: asap_sketchlib's Hydra has no group-mass query …SELECT g, approx_percentile_cont(y, 0.5) … GROUP BY g, nullabley, KLL candidateSummaryMergewith 400 groupsSummaryAggtimedIngestionTimea filtered summary update runs only at query timeLimitations and follow-ups
count by (g)read from the Hydra grid) needs a group-mass query, andasap_sketchlib's Hydra has none. executor: HydraCms kernel for count and point queries #600 mirrored sketchlib's private subkey encoding to get it. That would be a local copy, so this PR rejects the query with an explicit error. Group totals will come from an upstream sketchlib API and a move of the pin, as a separate task.i32weights. Fractional weights, such as a PromQL rate, cannot be added and fail the build, and the total weight of one grid is capped ati32::MAX.COUNT(*) GROUP BY kwith one Count-Min per group is pruned in Pass 1, so this PR does not change the keyed build for Constant weights.SummaryMergebinding checks it, throughrequire_sound_merge, before the kernel merges. The kernel adds only runtime checks: the same shape, the same group, and thei32total.PruneReason, a plan placing a filteredSummaryAggat ingestion is accepted by the planner and rejected only when the executor compiles it.row_bytes, e.g. in joins) do not count a shared grid. Hydra states flow only intoSummaryMergeandSummaryEstimate, which do count it.SummaryMergeremains reserved until that merge finishes.Verification
integration-tests/tests/filtered_aggregates.rs: exact filtered COUNT, filtered SUM andCOUNT(nullable)through the planner pipeline and the executor, plus hand-filtered exact Count and KLLSummaryAggs.physical_semantics::filtered_summary_build_is_rejected_at_ingestion_time.tests/hydra_cms_execution.rs: point counts within the CMS bound, merge equals one build, a filtered Hydra build, and rejection of group totals;many_groups_merge_one_grid_pair: 400 groups in both builds, merged within the default memory limit. It failed first.filtered_aggregates::kll_quantile_of_all_null_group_is_nullandglobal_kll_quantile_over_no_row_is_null. Both failed first.physical_semantics::measure_filters_are_rejected_over_promql_series_identity. It failed first.weighted_hydra_cms_sums_item_weights_per_group(bounded sums, plus a split-stream merge),unknown_sign_weight_is_rejectedandinvalid_weight_values_fail_the_build(negative, fractional, NaN, infinite);hydra_cms.rsandfactory.rs;physical_semanticstests for measure filters, filtered summary builds and serde round trips;precompute_population.79792ef894f7682896ec8c2c9afa3801ec36e83e, rebased on main3a25450d(feat(types): family-capability table and executor tumbling pane merges #696):cargo fmt --all -- --check: pass;cargo clippy --workspace --all-targets --all-features --locked -- -D warnings: pass;cargo nextest run --workspace: 1690 passed, 46 skipped.