Skip to content

Capabilities lost with the legacy replacement search: maintenance-time exact operations, extension intents, nested folds, rate-maintained summaries and guards #735

Open

Description

@zzylol

Problem, by example

sum by (service) (count_over_time(metrics[5m])) at an exact target:

Plan
Legacy replacement search (SemanticEquivalentRewriteStrategy) one Count aggregate by (service) over the 5m TimeRange: one exact state per service
Pass 1 today Sum accumulator over the finalized per-series Count accumulators: two exact states, one per series and one per service

The legacy search (ASAPStrategies and its strategies in replacement.rs, rewrite.rs and exact_composition.rs) was deleted in #736. The user approved this as an exception to the "delete only what the stages implement" rule on 2026-10-16. Before the deletion, nothing in the planner called that search, so planning did not change. But these capabilities now have no implementation anywhere. Each item below names one query, what the legacy engine produced (from its tests, which were deleted with it), and what Pass 1 / Stage 2 produce now (from a run on the PR head).

  1. Exact operation maintained at ingestion under a summary. quantile(0.99, deriv(latency[5m])).
    • Legacy: ExactCompositionStrategy placed deriv at ingestion time under a maintained KLL (OperationPlacement::Maintenance, test outer_summary_over_an_exact_function_composes_at_ingestion_time). The CostModel value-operation hooks gated it.
    • Now: Pass 1 offers the KLL over the plain deriv aggregate. Stage 2's ingestion_eligibility requires a build's child to be a TimeRange, so the KLL can only be built at query time (WindowNotFixed).
  2. Extension aggregations. AggIntent::Extension { .. }.
  3. Nested aggregate fusion beyond sum∘sum. sum(count_over_time(m[5m])), min(min_over_time(m[5m])), max(max_over_time(m[5m])).
    • Legacy: each folded into one aggregate (Count, Min, Max).
    • Now: two stacked accumulators, e.g. Finalize(Min) ← SummaryAgg(Min) ← Finalize(Min) ← SummaryAgg(Min). add_nested_sum_targets merges only a sum of sums.
  4. Summary maintained over finalized per-window rates. topk(2, sum by (job) (rate(m[1m]))).
    • Legacy: retime_rate_finalize / fixed_window_rate_candidates built a CMS+heap or a Sum over the finalized per-window rates, as a maintained candidate.
    • Now: Pass 1 offers CmsWithHeap{272, 5, 100} over Finalize(Rate), but Stage 2 cannot maintain it at ingestion, for the same TimeRange reason as item 1.
  5. Reset-aware counter proof for a CMS top-k over a rate. Same query as item 4.
    • Legacy: the heap update carried WeightDomain::NonNegative { proof: ResetAwareCounterDerivative } (realize_counter_value_summary_input).
    • Now: the update is UnknownOrSigned, so Stage 3 rejects the Count-Min heap as invalid for signed weights.
  6. Top-k membership guarantee from a distinct-items bound. topk(5, sum by (job) (m)) with an evidence provider that bounds the distinct items.
    • Legacy: finish_weighted_topk read AccuracyEvidenceProvider::topk_max_distinct_items and the TopKSelection margin statistics, and gave the estimate a joint guarantee.
    • Now: Pass 1 gives a top-k estimate no guarantee (estimate_has_bound is false for TopK). Stage 3 then accepts it only with output_guarantee evidence. topk_max_distinct_items and propagation_stats were removed from AccuracyEvidenceProvider because nothing read them. PropagationStats' topk_* fields remain, read only by the TopKSelection rule.
  7. UnivMon for an exact count. count by (job) (m) at an exact target.
    • Legacy: offered the exact Count accumulator and a UnivMon sized at ε = 0.
    • Now: only ExactAccumulator([Count]).
  8. Sketch state cap. Any sketch whose state exceeds 512 MiB.
    • Legacy: sketch_realizations dropped it (sketch_state_bytes, DEFAULT_MAX_SKETCH_STATE_BYTES).
    • Now: Pass 1 keeps it. Stage 3 prices its memory and only cost model 3 (ClusterCapacity) enforces a limit.
  9. Expected-case sizing. posterior_aware_size_params / ExpectedCaseSizing (Add posterior (query-time) error estimation for CMS/CountSketch, per Chen et al. IMC'21 #239) sized a sketch below the worst case under a declared non-adversarial assumption. It was called only by its tests; the stages size with default_size_params.
  10. No checked division inside a maintained state. quantile(0.9, avg_over_time(m[5m])).
    • Legacy: maintenance_exact_values refused to feed a maintained summary from a BinaryOp with checked_finite_division, because an invalid value cannot be removed from a sketch once it is in. The division stayed at read time.
    • Now: Pass 1 offers KLL ← BinaryOp(Div, checked) ← Finalize(Sum), Finalize(Count) with no such guard.
  11. DDSketch operand-domain check. A DDSketch ratio quantile_over_time(0.9, d[5m]) / quantile_over_time(0.5, d[5m]) with a subnormal domain [f64::MIN_POSITIVE / 2, f64::MIN_POSITIVE / 2], or max_samples above 2^53.
    • Legacy: OperandDomain::supports_ddsketch rejected both. It also required the domain to lie within ddsketch_indexable_bounds(alpha) and the contract to be non-empty.
    • Now: Stage 3's division rule rejects zero-crossing, non-finite and inverted domains, but accepts these two.

Hydra grouping (HydraGroupingStrategy) is tracked in #703 (G7b). Maintained-population top-k (MaintainedPopulationStrategy in Pass 1) is tracked in #681. Neither is repeated here.

Why it isn't done today

Pass 1 (per_computation_candidates.rs) was built as a new candidate generator, not a port of ASAPStrategies. The parity list in #580 covered the strategies (rollup, avg rewrite, top-k reuse, sharing, reconciliation, Hydra, populations), but not these narrower shapes and guards inside the legacy realization engine.

Proposal

Add each capability in the stage that owns it:

Several of these change candidate sets and selections, so each needs its own PR with the example tests re-checked.

Scope and acceptance

One test per item that fails on the PR head and passes with the capability, using the query above.

Status and related work

Activity

  1. added
    accuracyAccuracy requirements, guarantees, and propagation
    enhancementNew feature or request
    optimizerWorkload optimization and plan selection
    replacement-strategyASAP replacement candidates and physical strategies
    subpopulationsGrouped, hierarchical, or multi-subpopulation summaries
    and removed
    subpopulationsGrouped, hierarchical, or multi-subpopulation summaries
    on Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    accuracyAccuracy requirements, guarantees, and propagationenhancementNew feature or requestextensibilityoptimizerWorkload optimization and plan selectionreplacement-strategyASAP replacement candidates and physical strategies

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions