Skip to content

test(e2e): answer #509's PromQL examples like the Planner executor - #822

Draft
zzylol wants to merge 1 commit into
feat/query-enginefrom
test/e2e-examples
Draft

zzylol wants to merge 1 commit into
feat/query-enginefrom
test/e2e-examples

Conversation

@zzylol

@zzylol zzylol commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Stack 5/5. Base: #821.

Why

First milestone of executing the Planner's plan: the data plane's answers must equal what the Planner executor computes for the same selected plan.

What

data_plane/tests/e2e_examples.rs runs the real data_plane binary for #509's PromQL Examples 1, 3a, 3b, 4a and 4b:

  • data goes in through Remote Write;
  • a stand-in Prometheus serves the raw ranges (and answers forwarded queries with an empty vector, so a query sent before its panes are built is retried);
  • each answer is compared with the Planner executor running the selected plan with nothing materialized.

control_plane::plan_selected_at_query_time gives that reference: the Stage 2 candidate P<n> implementing the selected candidate's logical candidate L<n> with nothing materialized, sized for the failure probabilities and with confirmed weights as Stage 3 does for the selected one. Same summaries and parameters, everything at query time.

examples/planner-layering-example3a.json (Pattern A, ad hoc at T) and -example4a.json (Pattern A, monthly) are added in the current workload format.

Example Deployment Selected Stored definitions Result
1 default P1 (exact, query time) 0 exact
3a default P485 (KLL k=1246, query time) 0 within 10 (one query off by 1)
3b default P2 (KLL, query time) 0 exact
3b no raw data 1-min KLL panes at ingestion 1 exact
4a default P4125 (KLL k=1246, query time) 0 within 10 (exact this run)
4b no raw data P3-m1 (10-min KLL panes at ingestion) 1 exact

3a/4a match within a value tolerance of 10, about twice KLL's 0.005 rank error: their five-year windows hold more samples than k, and asap_sketchlib's KLL seeds compaction from the thread RNG, so two builds above k can differ.

How

Per example: plan with plan_workload (the plan installed) and plan_selected_at_query_time (the reference), start data_plane --allowed-lateness-ms 0, Remote Write every sample, poll each planned query until x-asap-execution: plan, and compare series by series with the reference run through QueryEngine over the same samples.

Key code interfaces added / modified

// control_plane/src/lib.rs
/// The selected plan with nothing materialized: the Stage 2 candidate that
/// implements the same logical candidate with every summary built at query
/// time, sized as Stage 3 sizes the selected one.
pub fn plan_selected_at_query_time(workload: &PlanningWorkload, inputs: &DeploymentInputs)
    -> Result<InstalledPlan, PlanError>;

Before this PR

No test checked that the data plane's answers match the Planner's.

After this PR

Every example's planned queries are answered by the plan and match the Planner executor (numbers below).

Usage

# Run the five examples against the real data_plane binary; --nocapture prints
# each series as "<query>: <labels> plan <data plane> executor <reference> difference <d>".
cargo test -p data_plane --test e2e_examples -- --nocapture
sum by (job) (rate(http_requests_total[1m])): {"job": "api"} plan 2.9999999999999996 executor 2.9999999999999996 difference 0
topk by (job) (10, sum_over_time(http_requests_total[1m])): {"job": "web"} plan 3330 executor 3330 difference 0
quantile_over_time(0.99, latency_ms[5y]): {"service": "a"} plan 987 executor 988 difference -1
quantile_over_time(0.99, latency_ms[3y] offset 2y): {"service": "b"} plan 10986 executor 10986 difference 0
quantile_over_time(0.99, latency_ms[5m]): {"service": "a"} plan 36 executor 36 difference 0
quantile_over_time(0.99, latency_ms[60m]): {"service": "b"} plan 1237 executor 1237 difference 0
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 378.70s

Verification

  • cargo test -p data_plane --test e2e_examples (5 tests, about 6 minutes in a debug build: Examples 3a/4a enumerate thousands of candidates).
  • cargo test -p control_plane -p data_plane (control plane adds reference_plan_materializes_nothing), cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings.

This was referenced Oct 7, 2026
For Examples 1, 3a, 3b, 4a and 4b: plan the workload, start the data_plane
binary against a Prometheus holding generated samples, Remote Write the
same samples, and require every planned query to be answered by the plan
with the Planner executor's result for the selected logical candidate with
nothing materialized (control_plane::plan_selected_at_query_time). 3b also
runs, and 4b only runs, on a deployment without raw data, so their panes are
built from Remote Write. Exact results must match exactly; Example 3a/4a's
five-year KLL windows exceed k and KLL compaction is randomized, so they
must agree within twice the 0.005 rank target.
@zzylol
zzylol force-pushed the feat/query-engine branch from 6f7b53c to b1b1e5a Compare October 18, 2026 15:40
@zzylol
zzylol force-pushed the test/e2e-examples branch from 4ab777a to f7faf1b Compare October 18, 2026 15:40
@zzylol zzylol added area: data plane Runtime ingestion, storage, and query-serving data paths integration: ASAPPlanner ASAPPlanner API, IR, candidate, and selected-plan integration labels Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: data plane Runtime ingestion, storage, and query-serving data paths integration: ASAPPlanner ASAPPlanner API, IR, candidate, and selected-plan integration quality: accuracy Exact and approximate accuracy contracts, error bounds, and routing test: end-to-end Process-boundary, differential, conformance, and full-stack tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant