[zzylol]

Why

No PromQL top-k could ever use a heap sketch. The fused whole-expression heap (#692) is the only heap Pass 1 still offers for a PromQL top-k (#805 pruned the others). Its SummaryEstimate{TopK} declared one Utf8 topk value per partition. The executor's keyed evaluation returns k rows per partition: the partition columns, the ranked item, and a Float64 estimate. So the executor refused every such plan, and since #805 Stage 3's compile check rejected all of them (ExecutorCannotCompile). The ASAPQuery top-k-sum queries (q06-topk-sum, topk by (region, service) (4, sum_over_time(data[1m]))) could not pick the cheaper heap plan even with guarantees ignored.

What

A heap top-k plan now compiles, and it answers k rows per partition: the right group labels, the ranked series (or group), and its estimated value. This covers:

• topk by (G) (k, sum_over_time(m[W])) and topk by (G) (k, count_over_time(m[W])): one heap per partition over the raw samples, ranking series identities;
• topk(k, sum by (G) (increase(m[W]))) and the same with rate, over counters: one heap over the exact per-series values, ranking the G groups.

Each answer has the exact top-k's physical shape, [keys.., topk_k, item labels..]. This is the same shape #794/#805 gave exact top-k answers. A range function's result has no __name__, as in Prometheus.

Not covered: topk(k, sum by (G) (sum_over_time(m[W]))) and ... count_over_time .... The PromQL frontend lowers them to exact Sort + Limit, so Pass 1 has no heap candidate for them (unchanged).

How

• Estimate schema (asap-types). SummaryEstimate{TopK} over a keyed build derives [partition columns.., the build's item columns.., topk: Float64]. That is exactly what Operator::keyed_evaluation emits.
• Candidate root (Pass 1). The heap's estimate is projected to the top-k's [keys.., topk_k] and then, over PromQL rows, the item columns (ranked_as_topk). The fused heap and a top-k's own heap both use it. A SQL top-k keeps only its logical columns, as its exact plan does.
• Counter top-k heaps are offered again. Rows that carry the PromQL series identity have a closed schema, and whole_expression_input refused every closed child. It now refuses only when it needs an encoded label set. The G labels are plain item columns.
• The compile check sees the plan the executor runs. Stage 3 now compiles the candidate with its weight requirements confirmed (with_confirmed_weights), as Selection::plan holds it. Before, a Count-Min heap over counter rates failed with "CMS requires a nonnegative weight contract" even when the counter was declared.
• Metric name. The executor drops __name__ from a heap top-k's rows, as it already does for every other temporal summary evaluation.
• Heaps are asap_sketchlib's CountMinSketchWithHeap / CountSketchWithHeap (with rescore_heap, fix(executor): DDSketch interpolated quantiles and top-k membership on sketchlib planner-pin/0.4.0-plus #803), unchanged. Nothing was missing from the library.

Key code interfaces added / modified

No Rust signature changes. The DAG format changes. A heap top-k estimate node's output schema in the plan file (PhysicalASAPDAG, InstallablePlan) is now:

crates/types/src/ir/operator/asap.rs

/// New (private): the rows a heap sketch's top-k estimate returns. Per state row, its `k`
/// heaviest items: the state's partition columns, then the build's item columns (the inner
/// group labels, or the PromQL series identity), then `topk: Float64`. Was
/// `[partition columns.., topk: Utf8]`, one row per partition.
fn topk_estimate_schema(state: &OperatorNode) -> Option<Schema>;

crates/logical-asap-optimization/src/per_computation_candidates.rs

/// New (private): a heap's top-k estimate in the top-k's columns, `[keys.., topk_k]` (topk_k
/// the Float64 estimate), then, over PromQL rows, the item columns.
fn ranked_as_topk(
    target: &OperatorNode,
    estimate: Rc<OperatorNode>,
) -> Result<Rc<OperatorNode>, PerComputationCandidateError>;

Usage

From a_forced_heap_topk_sum_plan_compiles_and_answers_within_its_bound (crates/planner/tests/runtime_install.rs), run at the linked commit:

// The user's query: the top 3 series per region by their 1-minute sum, within 0.05 at
// failure probability 0.3, every 10 s, over 10,000 `data` gauge series sampled every 100 ms.
let workload = heap_topk_workload(
    "topk by (region) (3, sum_over_time(data[1m]))", "data", r#""metric_type": "gauge""#);
// Guarantees ignored (a top-k estimate has none), no raw samples kept, and Stage 3's executor
// check (HeapPlansOnly) refuses non-heap plans and otherwise runs PlannerExecutorCompileCheck:
// the cheapest CountSketchWithHeap plan that the executor compiles is selected.
let plan = plan_ignoring_guarantees(&workload, Some(SketchAlgorithm::CountSketchWithHeap))
    .await?;
// Pass 1's fused heap: one Count Sketch heap per region over the raw samples, item = the
// series identity, in panes built at ingestion and merged at query time; the estimate
// returns [region, $promql_series_identity, topk] and the root projects
// [region, topk_3, $promql_series_identity].
plan.check_compiles()?;
// A stand-in runtime builds the panes from 80 skewed series (rank r sums to 1000 / r^1.1),
// merges them and answers.
let run = run_data_plane(&serde_json::to_string(&plan)?, &samples, EVALUATION_MS, &mut store);
// Per region: 3 rows, the true top 3 series with their full labels and no `__name__`, each
// within sqrt(3/w)·L2 of its exact sum; the query-time reference answers the same.
assert_heap_topk_answers(&run.answers_per_query[0], &truth, (Some("region"), "instance", 3),
    (&SketchAlgorithm::CountSketchWithHeap, width));

Before this PR

With guarantees ignored, every heap top-k candidate was rejected as ExecutorCannotCompile, because its estimate's declared shape is not the keyed evaluation's. Counter top-k (topk(k, sum by (G) (increase|rate(...)))) got no heap candidate at all through plan_installable. With the audit's throwaway weight fix, q06 alone selected the heap at 2.243654e-8 and then could not run.

After this PR

Heap top-k plans compile and run. Stage 3 picks one when it is the cheapest valid plan. With guarantees ignored, ASAPQuery's q06 now selects the Count-Min heap and passes the converter's check (results below). Together with #815, the heap is a Count-Min heap over a gauge declared non-negative.

Verification

Tests written first:

• runtime_install::a_forced_heap_topk_sum_plan_compiles_and_answers_within_its_bound (Count Sketch heap, gauge), ..._topk_count_plan_... (both heaps), ..._counter_topk_plan_... (increase and rate, both heaps). They go through plan_installable and the stand-in runtime on skewed data. Each checks that the forced heap plan compiles (Stage 3's check and check_compiles), that the true top 3 is recalled in every partition, that each estimate is within the heap's bound of the exact value (Count-Min: 0 ≤ est − exact ≤ (e/w)·L1; Count Sketch: |est − exact| ≤ √(3/w)·L2 ≤ ε·L1), that labels are kept without __name__, and that the query-time reference passes the same checks. Before the fix, the sum test failed with "no valid candidate": the Count Sketch heap was rejected by the compile check and the Count-Min heap by its weights. The counter shape had no heap candidate (Stage 3 checked 1 candidate, the exact one). With only the shape fix, the sum test failed on __name__. The count test was added after those runs and was not run against main.
• weighted_topk_binding::planner_weighted_topk_binds_at_either_deployment_phase is un-ignored. topk by (job) (2, sum by (service, job) (rate(m[1m]))) with both heaps, at ingestion and at query time, answers the exact top 2 per job. Its plan now has the weights confirmed for a declared counter, as Stage 3 hands it over, and it reads the ranked value from topk_2.
• Updated: promql_candidates_compose accepts a heap top-k root's [keys.., topk_k: Float64, items..]. The Example 1 stage fixture is regenerated (stage_pipeline --exhaustive) and the dag-viewer tests pass. The acceptance doc's Q2 summary shape is updated, and its paragraph on the Example 1 every-candidate compile test is dropped (that ignored test was deleted on main).
• cargo test --workspace --locked --no-fail-fast on the head: 151 test binaries, 1,844 passed, 0 failed, 35 ignored (one fewer ignored than main).

Real workloads (seed-862 workload-01.json, the local asapquery_random_workload converter rebuilt on this branch with #815 merged in, --no-raw --failure-probability 0.3 --check, 200 series, 3 evaluations):

Queries	Flags	Selected	Cost / s	Check per evaluation (recall, max rel. error)
q06	as-is	exact P1-m1	2.383717e-8	1.0, 0 (×3)
q06	--ignore-accuracy-guarantees	CmsWithHeap P2-m1 (w 55, d 2, heap 20)	2.243654e-8	0.9875, 0.029 / 0.9875, 0.019 / 0.9875, 0.024; passes
q07	as-is	exact P1-m1	1.452217e-8	1.0, 0 (×3)
q07	--ignore-accuracy-guarantees	exact P1-m1 (heap runner-up 1.923338e-8)	1.452217e-8	1.0, 0 (×3)
q06 + q07	as-is	exact P1-m3	3.835934e-8	1.0, 0 for both
q06 + q07	--ignore-accuracy-guarantees	P7-m3: heap q06, exact q07	3.695871e-8	q06 0.975, 0.024 / 0.9875, 0.016 / 0.9875, 0.022; q07 1.0, 0

As-is, the heap still has no guarantee (no stream_norms declared, and a top-k membership has none), so the exact plan wins there, as expected.

Follow-ups, not in this PR:

• topk(k, sum by (G) (sum_over_time|count_over_time(...))) has no heap candidate (frontend lowering, above).
• The two other ignored weighted_topk_binding tests are about a heap over one row per series (pruned since fix(planner): Stage 3 rejects plans the executor cannot compile; no PromQL top-k heap over its instant vector #805). They stay ignored.