Add differential e2e testing against real Prometheus using promql-compliance-tester #594

Problem

We have no differential/e2e test harness comparing ASAPQuery's PromQL output against real Prometheus. A large fraction of recent open bugs are exactly the class this would catch automatically: instant-vs-range query divergence (#589, #584, #583, #581, #587), missing counter-reset correction (#476), dropped outer aggregation (#496), sum/count semantics confusion (#505, #503). These keep getting found by hand, one at a time.

Proposal

Adopt prometheus/compliance/promql (the PromLabs promql-compliance-tester) as a differential test harness: it runs arbitrary PromQL query strings against a reference target and a test target and diffs the results, with per-test-case config for float tolerance (AdjustValueTolerance{Fraction, Margin}) and label-drop tweaks — which we need anyway since sketch-backed paths are approximate by design.

Two gaps to close before it's usable for us:

1. It only tests range queries. Comparer.Compare (in comparer.go) calls QueryRange exclusively; the PromAPI.Query (instant) method is declared in the interface but never invoked anywhere in the module. Since several of our worst bugs are specifically instant-vs-range divergence, a tool that never exercises the instant path can't catch them. We need to fork/patch Comparer.Compare to also run and diff an instant query per test case (~15-20 lines, mirrors the existing range logic).

2. Its documented data-loading story doesn't fit us. The upstream pattern is: run a real Prometheus scraping a demo exporter, mirror that scrape via remote_write into the target system, and wait ~1 hour for enough data to accumulate. That's non-deterministic and slow, and upstream itself lists "automate data loading" as an open TODO.

Instead: write our own data seeder that pushes hand-authored samples directly via remote write to both targets — build a prompb.WriteRequest with explicit timestamps/values/labels, snappy-compress it, POST to /api/v1/write on both a real Prometheus (started with --web.enable-remote-write-receiver) and ASAPQuery's existing remote-write endpoint. Same code path hits both targets, so there's no risk of the two ingestion mechanisms behaving differently and producing false diffs. Fully deterministic — we choose the exact data, so expected query results are computable by hand for regression cases.

Plan

• Write the remote-write seeder (Go, prompb + snappy) — pushes a fixed dataset to two query_urls
• Fork/vendor compliance/promql, patch Comparer.Compare to also run instant queries
• Write a config.yaml pointing reference_target_config at a local Prometheus and test_target_config at ASAPQuery's query endpoint
• Port known regressions (Range queries drop self-keyed accumulator expansion (top-k) for single-population metrics #584, Range query key expansion uses one snapshot instead of per-step keys #583, Catalog: instant-query vs range-query behavioral inconsistencies #589's catalog) into test cases as a starting seed set
• Wire into CI

References

• prometheus/compliance/promql — https://github.com/prometheus/compliance/tree/main/promql
• Related: Catalog: instant-query vs range-query behavioral inconsistencies #589, Range queries drop self-keyed accumulator expansion (top-k) for single-population metrics #584, Range query key expansion uses one snapshot instead of per-step keys #583, Unify PromQL instant and range query fetch/merge paths #581, fix(query-engine): expand self-keyed accumulators in range queries #587, Test suite ideas for instant/range PromQL query paths #590, rate/increase functions do no counter-reset correction #476, Nested queries: outer aggregation is silently dropped for planner-generated (non-collapsable) agg(fn(m[w])) queries #496, CountMinSketch (no heap) ignores aggregation_sub_type sum/count distinction, always uses SUM semantics #505, count precomputes accumulate sums, not counts #503