Skip to content

fix(plan-selection): prove Count-Min weights over a gauge declared non-negative - #815

Merged
zzylol merged 1 commit into
mainfrom
fix/cms-weights-declared-non-negative
Oct 21, 2026
Merged

zzylol merged 1 commit into
mainfrom
fix/cms-weights-declared-non-negative

Conversation

@zzylol

@zzylol zzylol commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Why

The ASAPQuery workloads' top-k-sum queries (q06-topk-sum, q07-topk-sum, e.g. topk by (region, service) (4, sum_over_time(data[1m]))) read data, a gauge the converter declares non-negative (values_non_negative, #780). Every Count-Min and CmsWithHeap candidate over it was still rejected as count_min_weights_not_proven_non_negative, "not declared a counter". The sign gate in the accuracy model already reads the declaration, but the weight check that guards the executor did not.

What

  • An open WeightDomain::SamplesOfMetric requirement is now proven when the metric is declared a counter or declared non-negative.
  • The new proof NonNegativeWeightProof::DeclaredNonNegativeSamples records the second case in the selected plan the executor runs.
  • CounterDerivativeOfMetric (rate, increase, irate) stays counter-only, because a reset-corrected rate of a gauge can be negative.
  • The rejection reason now says "not declared a counter or non-negative".

How

confirmed_weights passes both declared_counter and the existing declared_non_negative (the same rule the DDSketch sign composition and the values_non_negative[m] sign gate use) to WeightDomain::confirmed. A counter keeps the CounterSamples proof. A metric that is only declared non-negative gets DeclaredNonNegativeSamples. The executor accepts any NonNegative proof for a keyed Count-Min build, so it needs no change.

Key code interfaces added / modified

crates/types/src/ir/schema/sketch.rs

/// Stays. Why a summary update's weights are non-negative.
pub enum NonNegativeWeightProof {
    /// Weight 1 per row (a count). // unchanged
    UnitCount,
    /// rate/increase/irate of a declared counter, or plain sums of them. // unchanged
    ResetAwareCounterDerivative,
    /// Raw samples of a declared counter, or plain sums of them. // unchanged
    CounterSamples,
    /// Raw samples of a metric declared non-negative (a gauge, say), or plain sums of them;
    /// given by plan selection when the metric is not declared a counter. // new
    DeclaredNonNegativeSamples,
    /// A positive constant weight. // unchanged
    PositiveConstant,
}

impl WeightDomain {
    /// The domain once the declared facts answer an open requirement: samples are proven by a
    /// counter or a non-negative declaration, a counter derivative only by a counter.
    pub fn confirmed(
        &self,
        is_counter: impl Fn(&str) -> bool,
        is_declared_non_negative: impl Fn(&str) -> bool, // new parameter
    ) -> WeightDomain;
}

crates/plan-selection/src/lib.rs

/// Signature unchanged. Now also proves samples of a metric declared non-negative.
pub fn confirmed_weights(domain: &WeightDomain, data: &DataWorkload) -> WeightDomain;

Usage

// The user's query: the top 10 series per job by their 1-minute sum, ε 0.01 at δ 0.001
// (stream-L1), every 10 s, over a gauge `m` declared non-negative.
let target = AccuracyTarget::max_error(0.01)
    .failure_probability(0.001)
    .accuracy_metric(AccuracyMetric::ABSOLUTE_STREAM_L1);
// The data facts: `m` is a gauge whose samples are all >= 0.
let mut gauge = with_metric_types(data(), &[("m", MetricType::Gauge)]);
gauge.sources.get_mut("m").unwrap().values_non_negative = true;
// Every Pass 1 combination as a physical candidate. One of them is the fused
// CmsWithHeap whose weights carry the open requirement SamplesOfMetric { metric: "m" }.
let candidates = all_candidates("topk by (job) (10, sum_over_time(m[1m]))", &target);
// Stage 3: accuracy_check calls confirmed_weights on each Count-Min build.
let selection = select_plan(&candidates, &[entry_with(target, repeating(10_000))], &gauge,
    PlanningModels::builtin());
// Before: P2 (the CmsWithHeap) was rejected as CountMinWeightsNotProvenNonNegative,
// "samples of `m`, which is not declared a counter". Now it passes the weight check and
// goes on to the accuracy check (no guarantee for a top-k, so the built-in model rejects it
// as NoAccuracyGuarantee unless evidence or ignore_accuracy_guarantees accepts it).
// The plan the executor runs carries the proof:
// with_confirmed_weights(&p2.roots, &gauge) -> NonNegative { proof: DeclaredNonNegativeSamples }

This is count_min_over_samples_of_a_gauge_declared_non_negative_is_admitted, run at the linked commit.

Before this PR

Count-Min and CmsWithHeap over a non-negative gauge were always invalid, whatever the accuracy settings, because the samples were proven non-negative only for a counter.

After this PR

They pass the weight check and are judged on accuracy and cost like any other candidate. A rate or increase of that gauge is still rejected.

Limitation: through plan_installable, a heap top-k plan is still refused by the executor compile check (ExecutorCannotCompile), because the executor can't run the fused heap's output yet. #831 fixes that. With both PRs, ASAPQuery's q06 (topk by (region, service) (4, sum_over_time(data[1m])) over the converter's non-negative gauge, --no-raw --failure-probability 0.3 --check --ignore-accuracy-guarantees) selects the CmsWithHeap plan at 2.243654e-8 per second instead of the exact one at 2.383717e-8. It passes the check: recall 0.9875 and max relative error at most 0.029 in each of 3 evaluations. The full table is in #831.

Verification

  • New test count_min_over_samples_of_a_gauge_declared_non_negative_is_admitted: topk by (job) (10, sum_over_time(m[1m])) over a gauge declared non-negative. Every CmsWithHeap candidate passes the weight check, and the confirmed plan carries DeclaredNonNegativeSamples. topk(10, sum by (job) (increase(m[5m]))) over the same gauge is still rejected for its weights.
  • Extended confirmed_weights_rewrite_only_the_summary_update with the declared-non-negative gauge.
  • Test first: with confirmed_weights not reading the declaration, both tests fail (P2 rejected as CountMinWeightsNotProvenNonNegative; domain UnknownOrSigned).
  • cargo test --workspace --locked --no-fail-fast on the head (rebased on main at 334d549): 1,849 passed, 0 failed, 36 ignored.

@zzylol zzylol added accuracy Accuracy requirements, guarantees, and propagation bug Something isn't working optimizer Workload optimization and plan selection promql PromQL front-end lowering (L1→L2) labels Oct 21, 2026
@zzylol
zzylol force-pushed the fix/cms-weights-declared-non-negative branch from 9f3cfe0 to f363e51 Compare October 21, 2026 17:59
@zzylol
zzylol force-pushed the fix/cms-weights-declared-non-negative branch from f363e51 to 9d929ef Compare October 21, 2026 19:49
…tive are proven

confirmed_weights answered an open SamplesOfMetric requirement from the
declared metric type alone, so raw samples of a gauge declared
values_non_negative stayed unproven and every Count-Min and CmsWithHeap
candidate over them was rejected as count_min_weights_not_proven_non_negative.

WeightDomain::confirmed now takes both facts: samples are proven by a
declared counter (CounterSamples) or else a declared non-negative metric
(the new DeclaredNonNegativeSamples). A counter derivative stays
counter-only, since a reset-corrected rate of a gauge can be negative.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accuracy Accuracy requirements, guarantees, and propagation bug Something isn't working optimizer Workload optimization and plan selection promql PromQL front-end lowering (L1→L2)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant