Repository navigation
Capabilities lost with the legacy replacement search: maintenance-time exact operations, extension intents, nested folds, rate-maintained summaries and guards #735
Description
Problem, by example
sum by (service) (count_over_time(metrics[5m])) at an exact target:
| Plan | |
|---|---|
Legacy replacement search (SemanticEquivalentRewriteStrategy) |
one Count aggregate by (service) over the 5m TimeRange: one exact state per service |
| Pass 1 today | Sum accumulator over the finalized per-series Count accumulators: two exact states, one per series and one per service |
The legacy search (ASAPStrategies and its strategies in replacement.rs, rewrite.rs and exact_composition.rs) was deleted in #736. The user approved this as an exception to the "delete only what the stages implement" rule on 2026-10-16. Before the deletion, nothing in the planner called that search, so planning did not change. But these capabilities now have no implementation anywhere. Each item below names one query, what the legacy engine produced (from its tests, which were deleted with it), and what Pass 1 / Stage 2 produce now (from a run on the PR head).
- Exact operation maintained at ingestion under a summary.
quantile(0.99, deriv(latency[5m])).- Legacy:
ExactCompositionStrategyplacedderivat ingestion time under a maintained KLL (OperationPlacement::Maintenance, testouter_summary_over_an_exact_function_composes_at_ingestion_time). TheCostModelvalue-operation hooks gated it. - Now: Pass 1 offers the KLL over the plain
derivaggregate. Stage 2'singestion_eligibilityrequires a build's child to be aTimeRange, so the KLL can only be built at query time (WindowNotFixed).
- Legacy:
- Extension aggregations.
AggIntent::Extension { .. }.- Legacy: realized through the deployment's
CostModel::realize_extension/evaluation_extension. - Now:
computation_candidates_for_intentgives only[Exact]. The extension contract is Design the extension contract for new ASAP primitives and sketch algorithms #548.
- Legacy: realized through the deployment's
- Nested aggregate fusion beyond sum∘sum.
sum(count_over_time(m[5m])),min(min_over_time(m[5m])),max(max_over_time(m[5m])).- Legacy: each folded into one aggregate (
Count,Min,Max). - Now: two stacked accumulators, e.g.
Finalize(Min) ← SummaryAgg(Min) ← Finalize(Min) ← SummaryAgg(Min).add_nested_sum_targetsmerges only a sum of sums.
- Legacy: each folded into one aggregate (
- Summary maintained over finalized per-window rates.
topk(2, sum by (job) (rate(m[1m]))).- Legacy:
retime_rate_finalize/fixed_window_rate_candidatesbuilt a CMS+heap or aSumover the finalized per-window rates, as a maintained candidate. - Now: Pass 1 offers
CmsWithHeap{272, 5, 100}overFinalize(Rate), but Stage 2 cannot maintain it at ingestion, for the sameTimeRangereason as item 1.
- Legacy:
- Reset-aware counter proof for a CMS top-k over a rate. Same query as item 4.
- Legacy: the heap update carried
WeightDomain::NonNegative { proof: ResetAwareCounterDerivative }(realize_counter_value_summary_input). - Now: the update is
UnknownOrSigned, so Stage 3 rejects the Count-Min heap as invalid for signed weights.
- Legacy: the heap update carried
- Top-k membership guarantee from a distinct-items bound.
topk(5, sum by (job) (m))with an evidence provider that bounds the distinct items.- Legacy:
finish_weighted_topkreadAccuracyEvidenceProvider::topk_max_distinct_itemsand theTopKSelectionmargin statistics, and gave the estimate a joint guarantee. - Now: Pass 1 gives a top-k estimate no guarantee (
estimate_has_boundis false for TopK). Stage 3 then accepts it only withoutput_guaranteeevidence.topk_max_distinct_itemsandpropagation_statswere removed fromAccuracyEvidenceProviderbecause nothing read them.PropagationStats'topk_*fields remain, read only by theTopKSelectionrule.
- Legacy:
- UnivMon for an exact count.
count by (job) (m)at an exact target.- Legacy: offered the exact
Countaccumulator and a UnivMon sized at ε = 0. - Now: only
ExactAccumulator([Count]).
- Legacy: offered the exact
- Sketch state cap. Any sketch whose state exceeds 512 MiB.
- Legacy:
sketch_realizationsdropped it (sketch_state_bytes,DEFAULT_MAX_SKETCH_STATE_BYTES). - Now: Pass 1 keeps it. Stage 3 prices its memory and only cost model 3 (
ClusterCapacity) enforces a limit.
- Legacy:
- Expected-case sizing.
posterior_aware_size_params/ExpectedCaseSizing(Add posterior (query-time) error estimation for CMS/CountSketch, per Chen et al. IMC'21 #239) sized a sketch below the worst case under a declared non-adversarial assumption. It was called only by its tests; the stages size withdefault_size_params. - No checked division inside a maintained state.
quantile(0.9, avg_over_time(m[5m])).- Legacy:
maintenance_exact_valuesrefused to feed a maintained summary from aBinaryOpwithchecked_finite_division, because an invalid value cannot be removed from a sketch once it is in. The division stayed at read time. - Now: Pass 1 offers
KLL ← BinaryOp(Div, checked) ← Finalize(Sum), Finalize(Count)with no such guard.
- Legacy:
- DDSketch operand-domain check. A DDSketch ratio
quantile_over_time(0.9, d[5m]) / quantile_over_time(0.5, d[5m])with a subnormal domain[f64::MIN_POSITIVE / 2, f64::MIN_POSITIVE / 2], ormax_samplesabove 2^53.- Legacy:
OperandDomain::supports_ddsketchrejected both. It also required the domain to lie withinddsketch_indexable_bounds(alpha)and the contract to be non-empty. - Now: Stage 3's division rule rejects zero-crossing, non-finite and inverted domains, but accepts these two.
- Legacy:
Hydra grouping (HydraGroupingStrategy) is tracked in #703 (G7b). Maintained-population top-k (MaintainedPopulationStrategy in Pass 1) is tracked in #681. Neither is repeated here.
Why it isn't done today
Pass 1 (per_computation_candidates.rs) was built as a new candidate generator, not a port of ASAPStrategies. The parity list in #580 covered the strategies (rollup, avg rewrite, top-k reuse, sharing, reconciliation, Hydra, populations), but not these narrower shapes and guards inside the legacy realization engine.
Proposal
Add each capability in the stage that owns it:
- items 3 and 7: Pass 1 choices;
- items 1 and 4: Stage 2 materialization (a summary over a finalized per-window value maintained at ingestion);
- items 5, 6, 10 and 11: Pass 1 update contracts or the Stage 3 accuracy check;
- item 8: a Pass 1 prune with a
PruneReason, or a deployment capability (Deployment inputs: one bundle for cost model, accuracy model and capabilities — after #511 #525); - items 2 and 9: the extension contract (Design the extension contract for new ASAP primitives and sketch algorithms #548) and sizing.
Several of these change candidate sets and selections, so each needs its own PR with the example tests re-checked.
Scope and acceptance
One test per item that fails on the PR head and passes with the capability, using the query above.
Status and related work
- Deleted in refactor(asap-aware-mapping)!: delete the legacy replacement search and CostModel #736, user-approved exception (2026-10-16).
- Related: Pass 2 sharing and Stage 1 coverage parity with the retired MajorPass #580 (Pass 2 and Stage 1 parity), Hydra follow-ups: group totals in the executor, Pass 1 HydraCms candidates, Hydra top-k #703 (Hydra), Offer maintained-population top-k candidates in Pass 1 #681 (maintained populations), Design the extension contract for new ASAP primitives and sketch algorithms #548 (extension contract), Deployment inputs: one bundle for cost model, accuracy model and capabilities — after #511 #525 (deployment inputs), Add posterior (query-time) error estimation for CMS/CountSketch, per Chen et al. IMC'21 #239 (expected-case sizing), Calibrate plan selection's costs from measured sketch costs #679 (measured costs).
Activity
refactor(asap-aware-mapping)!: delete the legacy replacement search a…