Skip to content

feat(frontend-sql): recognize the SQL frequency L2 and entropy idioms - #598

Closed
zzylol wants to merge 2 commits into
stack/509-w7b-1-frequency-runtimefrom
stack/509-w7b-2-sql-frequency-recognition
Closed

zzylol wants to merge 2 commits into
stack/509-w7b-1-frequency-runtimefrom
stack/509-w7b-2-sql-frequency-recognition

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Rebased on main d4869a7 (DF 54).

Wave 1 chain: #594 → #593 → #592 → #591 → #595 → #596 → #597 → #598 → #599 (on #589)

Why

#509 Example 2 writes two of its three statistics as SQL idioms:

-- Q2: Entropy(src_ip)
SELECT -SUM(p * LN(p)) FROM (SELECT COUNT(*) * 1.0 / SUM(COUNT(*)) OVER () AS p FROM flows WHERE ... GROUP BY src_ip);
-- Q3: L2(src_ip)
SELECT SQRT(SUM(c * c)) FROM (SELECT src_ip, COUNT(*) AS c FROM flows WHERE ... GROUP BY src_ip);

and says "The SQL frontend does not yet recognize the Q2 and Q3 forms as Entropy and L2. TODO: add this recognition to the per-language frontends." Without an Entropy/L2 intent, Pass 1 has nothing to attach the exact, summary and UnivMon candidates to, and Pass 2 cannot share a UnivMon with Q1.

The parked #562 (L2) and #564 (entropy) added these rules as a Pass 1 SemanticEquivalentRewriteStrategy in the old asap-aware-mapping. This PR ports them. Part of #509 / #580 (W8).

What

Before (Q2, lowered):

Project(-col0) ← Aggregate(SUM(p*LN(p))) ← Project(p = COUNT*1.0/total) ← SQLWindowFunc(SUM(COUNT) OVER ()) ← Aggregate(COUNT(*) GROUP BY src_ip) ← Scan(flows)

After:

Project(CASE WHEN population_count = 0 THEN NULL ELSE -(0.0 - frequency_entropy * LN 2) END)
└── Join(Cross)
    ├── Aggregate[FrequencyEntropy { col: src_ip, accuracy: <query target> }]
    └── Aggregate[Count { accuracy: Exact }]          (both read the filtered scan)

Q3 becomes the same shape with FrequencyL2 and no unit change. Stage 1 then offers PassThrough (the exact reducer from #597) and summary alternatives, including UnivMon, for each intent.

How

  • Rules (verbatim from feat(planner): recognize floating SQL frequency L2 idioms #562 + feat(planner): recognize normalized SQL entropy in nats #564). crates/frontend-sql/src/frequency.rs: follow projections only; require a single-key unfiltered COUNT(*) over a non-nullable Bool/Int64/Utf8 key, no HAVING; for L2, SQRT(SUM(c*c)) with a product already typed Float64; for entropy, LN, p = COUNT(*)[*1.0] / SUM(COUNT(*)) OVER () over the complete unordered frame. The result converts bits to nats, keeps SQL's -0.0, and uses an exact population count for SQL's empty-input NULL, so an approximate estimate never decides NULL. The schema must equal the original.
  • Adaptation: where the rule runs. The old rule was a Pass 1 replacement strategy used by the legacy search, which plan_stages does not run: Stage 1 only enumerates realizations of single-measure aggregates. Following the design doc's TODO ("per-language frontends"), lower_sql_dialect now applies the rule at the query root. So the original relational DAG is no longer kept as a separate candidate. The exact alternative is the native exact reducer, and the tests show that it agrees with the original SQL.
  • SQL sqrt (from feat(planner): recognize floating SQL frequency L2 idioms #562). The executor's expression compiler accepts sqrt on Int64/Float64/NULL.

Refused, as before: the uncast integer product (which is the design's literal Q3, because SQL Int64 overflow is observable), nullable keys, HAVING, scaled probabilities, LOG2, and partitioned or ordered windows.

Tests

Ported from #562/#564: frontend-sql/tests/frequency_{l2,entropy}.rs (recognition, refusals, accuracy propagation, exact population guard) and integration-tests/tests/sql_frequency_{l2,entropy}.rs (original and recognized L2 agree; entropy in nats, -0.0, NULL on empty input). The legacy-search inventory tests become stage1_offers_exact_and_summary_l2_alternatives over enumerate_local_logical_candidates. Without the lower_sql call, the recognition and entropy execution tests fail.

Gate: fmt and clippy (-D warnings) pass. cargo test --workspace: 1,538 passed, 10 ignored (#589: 1,517 / 10).

Ports #562, #564. Stacked on #597. Next: #599.

@zzylol
zzylol force-pushed the stack/509-w7b-1-frequency-runtime branch from 502e276 to bceb13e Compare October 4, 2026 18:25
@zzylol
zzylol force-pushed the stack/509-w7b-2-sql-frequency-recognition branch from f047e41 to 66bc1a6 Compare October 4, 2026 18:25
@zzylol
zzylol force-pushed the stack/509-w7b-2-sql-frequency-recognition branch 2 times, most recently from e211fba to fc32eae Compare October 8, 2026 16:47
@zzylol
zzylol force-pushed the stack/509-w7b-1-frequency-runtime branch from c0cf2fc to c77ad29 Compare October 10, 2026 14:32
@zzylol
zzylol force-pushed the stack/509-w7b-2-sql-frequency-recognition branch from fc32eae to 303b22e Compare October 10, 2026 14:32
zzylol and others added 2 commits October 11, 2026 16:22
Port of the parked #562 and #564 for #509 Example 2. The SQL frontend
names `SQRT(SUM(c*c))` over Float64 grouped unit counts as FrequencyL2,
and `-SUM(p*LN(p))` with `p = COUNT(*)*1.0 / SUM(COUNT(*)) OVER ()` as
FrequencyEntropy converted to nats. An exact population count guards
SQL's empty-input NULL. Integer products, nullable keys, filters, HAVING,
other log bases and partial windows are refused.

The rules are the old Pass 1 SemanticEquivalentRewriteStrategy rules, run
by lower_sql at the query root so the Stage 1 pipeline sees the intents;
the exact candidate is the native exact reducer. The executor gains SQL
sqrt (from #562).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pass 1 takes the declared metric types since #593.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/509-w7b-2-sql-frequency-recognition branch from 303b22e to ae6d2f6 Compare October 11, 2026 17:11
zzylol added a commit that referenced this pull request Oct 13, 2026
The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
…598

Running the ignored SQL test composes every candidate and fails only on
Q2's options (per-source counts until #598), so #603 is not a blocker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
…stages (#577)

* test: acceptance spec and pending tests for planner-layering Example 1

Define MVP acceptance for #509 Example 1 before the Phase C stage APIs
exist: 1 -> 6 -> 6 -> 1 candidates (Pass 1 x identical-expression
sharing; physical operator implementation only, no materialization).

- Spec with per-stage candidate tables, invariants and doc ambiguities.
- Integration tests against todo!() stage stubs, ignored until the
  stages land; workload and Stage 0 frontend-shape tests run now.
- Expected-only asap-stage-pipeline/v1 fixture for the DAG viewer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run the Example 1 acceptance tests against the real stages

Replace the todo!() stubs with adapters over Stage 0-3, un-ignore the 11
tests that pass, and give each still-ignored test the precise difference
between the implementation and the spec. Add runtime capability checks:
the selected plan compiles in the physical planner; compiling every
candidate stays ignored with the two runtime gaps it finds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read Example 1 logical stages as flat DAGs after the export removal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 acceptance on the composed stages (1 -> 3 -> 3 -> 1)

The tests run Pass 1 and composition, optimize_physical_asap and
select_plan through a shared tests/planner_layering_common pipeline, and
follow the decided behavior: three Pass 1 candidates (Q2 exact, CMS+heap,
CountSketch+heap), every sub-DAG NotMaterialized, P1 selected, CMS+heap
invalid (weights not proven non-negative) and CountSketch+heap invalid (a
Top-K output has no guarantee and no deployment evidence). Hydra (#605),
Pass 2 sharing (#588) and the two runtime gaps (#593, #579) stay ignored
with the PR that delivers them. Tests are named after the stages; the
compile check no longer uses the PromQL fallback. The spec lists where the
doc and the planner differ. The expected-only viewer fixture is removed:
the generated four-stage fixture replaces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: port #590's Example 2-4 acceptance specs and tests

The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* feat(devtools): stage_pipeline examples for #509 Example 3

Ported from #590: --example planner-layering-3a (Pattern A, ad hoc batch of
five p99 reports) and planner-layering-3b (Pattern B, repeating 5-min p99),
the Example 3 acceptance tests' workloads. Example 1's output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: follow #576's per-consumer charging and exact top-k columns

Plan selection charges a NotMaterialized node once per consuming query,
so the charging helper checks NodeCost.multiplier and the Example 4 A3
ambiguity is recorded as decided. Example 1 checks that exact Q2 sorts by
the ranked value and returns [job, topk_10]. The stage viewer guide's
tab 1 row now describes the actual all-exact result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(devtools): one planning pipeline for stage_pipeline and the acceptance tests

run_planning_stages and the #509 example workloads move into the
asap-devtools library; stage_pipeline serializes the run and the
planner-layering tests adapt it. The shared data workload drops
ingestion_rate for data at rest, which validate() requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(devtools): select over every candidate; --max-displayed limits only the document

stage_pipeline capped enumeration at 64, so Example 3a (243 candidates)
selected P41 instead of the cheapest P122 (KLL for all five queries).
The pipeline now enumerates, optimizes and prices every candidate; the
document shows the first --max-displayed (default 64) plus the selected
one, with `combinations` and `capped`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: Example 2's SQL candidates compose; the SQL test waits only on #598

Running the ignored SQL test composes every candidate and fails only on
Q2's options (per-source counts until #598), so #603 is not a blocker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: name stages by number and name; say query-language frontends

The frontends' Run field, id and locals become stage0_query_language_frontends,
and test names carry stage0..stage3 with the stage name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: charging helper follows plan selection's per-consumer rule

With a materialized node the helper accepted any multiplier >= 1. It now
computes each node's computations as plan selection does (walks from
query roots and materialized nodes' inputs, stopping at materialized
nodes), checked on a hand-built candidate with a materialized node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 picks Q2 candidates by (option, no shared input)

physical_with and the invalid-candidate check no longer assume one
candidate per Q2 option, so #588's shared-input variants need no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: Example 3 Pattern B charges each node once per consuming query

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs(stage-viewer): rows 2, 3a and 3b describe the planner's current output

Read from uncapped runs: Example 2's sketch plans are rejected for having
no (ε, δ) guarantee, 3a selects P122 (first of 108 tied cheapest plans,
nothing shared yet) and 3b selects query-time KLL; panes and the latency
check come with the window-summary PRs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs(stage-viewer): row 2 from the SQL run; 3a ties 32 plans, not 108

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 counts only unshared candidates; stage3_ charging-helper names

The 3-logical and 3-physical counts now hold when #588 adds shared-input
variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(devtools): PlanningStagesRun fields carry stage number and name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test(devtools): unique temp file per stage_pipeline run

cargo test runs the tests as threads of one process, so two runs of the same
example shared a file name and one deleted the other's output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 15, 2026
#509 Example 2 writes L2(src_ip) and Entropy(src_ip) as SQL idioms over
grouped unit counts. The SQL lowering (single query and batch) now names
them at the query root as FrequencyL2 / FrequencyEntropy aggregates
carrying the query's accuracy target, so Pass 1 offers exact and summary
candidates for each:

- L2: SQRT(SUM(c * c)) over COUNT(*) per key, only when the product is
  already Float64 (CAST(c AS DOUBLE) factors or CAST(COUNT(*) AS DOUBLE)).
  The Int64 product stays relational: SQL integer overflow is observable.
- Entropy: -SUM(p * LN(p)) with p = COUNT(*)[*1.0] / SUM(COUNT(*)) OVER ()
  over the complete unordered frame; the IR's bits are converted to nats
  and SQL's -0.0 is kept.

Both require a single non-nullable Bool/Int64/Utf8 key, no HAVING and no
measure filter, follow projections only, keep the original output schema,
and decide SQL's empty-input NULL by an exact population count joined
beside the statistic, never by an estimate. Scaled probabilities, LOG2,
partitioned or ordered windows and nullable keys stay relational.

Ports the frontend part of #598 (originally #562 and #564).
zzylol added a commit that referenced this pull request Oct 15, 2026
#509 Example 2 writes L2(src_ip) and Entropy(src_ip) as SQL idioms over
grouped unit counts. The SQL lowering (single query and batch) now names
them at the query root as FrequencyL2 / FrequencyEntropy aggregates
carrying the query's accuracy target, so Pass 1 offers exact and summary
candidates for each:

- L2: SQRT(SUM(c * c)) over COUNT(*) per key, only when the product is
  already Float64 (CAST(c AS DOUBLE) factors or CAST(COUNT(*) AS DOUBLE)).
  The Int64 product stays relational: SQL integer overflow is observable.
- Entropy: -SUM(p * LN(p)) with p = COUNT(*)[*1.0] / SUM(COUNT(*)) OVER ()
  over the complete unordered frame; the IR's bits are converted to nats
  and SQL's -0.0 is kept.

Both require a single non-nullable Bool/Int64/Utf8 key, no HAVING and no
measure filter, follow projections only, keep the original output schema,
and decide SQL's empty-input NULL by an exact population count joined
beside the statistic, never by an estimate. Scaled probabilities, LOG2,
partitioned or ordered windows and nullable keys stay relational.

Ports the frontend part of #598 (originally #562 and #564).
zzylol added a commit that referenced this pull request Oct 15, 2026
…and UnivMon frequency statistics (#707)

feat(types): summary-capability facts in the family-capability table

Each family lists the readouts one of its states answers (Readout:
total, item frequency, any quantile rank, distinct count, L2, entropy,
top-k), so a planner rule can check that one state serves every
consumer's statistic: UnivMon answers its total and every frequency
moment, KLL every rank, CountSketch item frequencies and L2, the heap
sketches top-k and item frequencies. Exact accumulators are read as
finalized values and answer no readout.
feat(logical-asap-optimization): Pass 2 summary-capability rule

Composition sizes each consumer's summary build alone, for its own
target. The summary-capability rule groups the builds of one candidate
that chose the same family over the same input sub-DAG (window, WHERE
and time range included), item and weight, grouping and filter, whose
every reader is an estimate the family answers (the family-capability
table's readouts), and that differ only in size. For every choice, per
group, of no sharing or one subset of two or more members whose sizes
differ (partial sharing included), the subset becomes one build sized
with each parameter's largest value among the members (a KLL's failure
probability and DDSketch's alpha: the smallest), and every guarantee
above it is composed again, so plan selection checks each consumer
against the shared state. Builds of equal size stay the
identical-expression rule's; unshared builds are untouched and no
consumer changes family.

Stage 1 lists each composed candidate, its identical-expression
candidates, then each summary-capability candidate followed by its own
identical-expression candidates, every distinct DAG once. The Pass 2
record (Pass2Candidate) names the shared summaries ("UnivMon for Q1,
Q3") and the shared identical groups.

Example 2 (PromQL stand-in): with #702's sizing the three UnivMons
differ (Q1 a distinct-count heap on 19 layers, Q2 the baseline shape for
entropy, Q3 a one-layer Count Sketch for L2), so the rule offers one
UnivMon for each pair and one for all three, each sized with the
members' largest heap, rows, columns and layers and the smallest
recorded distinct-count failure probability; the third query keeps its
own options. A shared UnivMon whose rows and columns together exceed
asap_sketchlib's hash budget (rows * (ceil(log2 cols) + 1) <= 128) is
not offered, since the library would not build it. The shared UnivMon
stays one build in Stage 2.
feat(frontend-sql): recognize the SQL frequency L2 and entropy idioms

#509 Example 2 writes L2(src_ip) and Entropy(src_ip) as SQL idioms over
grouped unit counts. The SQL lowering (single query and batch) now names
them at the query root as FrequencyL2 / FrequencyEntropy aggregates
carrying the query's accuracy target, so Pass 1 offers exact and summary
candidates for each:

- L2: SQRT(SUM(c * c)) over COUNT(*) per key, only when the product is
  already Float64 (CAST(c AS DOUBLE) factors or CAST(COUNT(*) AS DOUBLE)).
  The Int64 product stays relational: SQL integer overflow is observable.
- Entropy: -SUM(p * LN(p)) with p = COUNT(*)[*1.0] / SUM(COUNT(*)) OVER ()
  over the complete unordered frame; the IR's bits are converted to nats
  and SQL's -0.0 is kept.

Both require a single non-nullable Bool/Int64/Utf8 key, no HAVING and no
measure filter, follow projections only, keep the original output schema,
and decide SQL's empty-input NULL by an exact population count joined
beside the statistic, never by an estimate. Scaled probabilities, LOG2,
partitioned or ordered windows and nullable keys stay relational.

Ports the frontend part of #598 (originally #562 and #564).
feat(physical-operators): exact distinct, L2 and entropy reducers

Exact Cardinality, FrequencyL2 and FrequencyEntropy aggregates had no
native implementation, so the exact option of #509 Example 2's three
statistics could not execute.

- Reduction::Cardinality(cols) counts distinct tuples; a tuple with any
  NULL is skipped, and grouped key encoding keeps tuple boundaries and
  folds signed zero and NaN payloads.
- Reduction::FrequencyL2(col) and FrequencyEntropy(col) count the unit
  frequencies of one Bool/Int64/Float64/Utf8 identity column: NULL is
  skipped, a non-finite float is an error, an empty population gives 0.0,
  and entropy is in bits, as AggIntent::FrequencyEntropy and UnivMon's
  readout define it.
- The dictionaries are memory-accounted and cooperative.
- Logical aggregates bind them: grouped ones (SQL COUNT(DISTINCT)) and
  per-series range reductions (distinct_over_time, l2_over_time,
  entropy_over_time), which reduce each series' window samples, restore
  the run's timestamp and drop the metric name as range functions do.
- The analytical cost lowering prices them as hash aggregates.

Ports the exact-reducer part of #597 (from #559 and #563).
feat(physical-operators): native UnivMon state over typed identities

UnivMon had a kernel but no native DAG state: a build was rejected
("summary family has no native DAG state implementation"), a summary
build accepted only Float64 samples, and Pass 1's UnivMon update (each
value of the item column one unit occurrence) did not bind, so no
UnivMon candidate of #509 Example 2 could execute.

- UnivMon is a native family; its states validate against their
  parameters, and one state answers its total, distinct count, L2 norm
  and entropy.
- A UnivMon build takes Bool, Int64 and Utf8 identities as type-tagged
  key bytes (Int64 above 2^53 stays distinct) besides Float64 samples;
  NULL is skipped. Retained heap key bytes count toward state memory.
- A unit-frequency update over one item column binds, for UnivMon and
  HLL, as a value-frequency build over that column: grouped (SQL
  src_ip) and per series (PromQL sample values).

Ports the UnivMon part of #597 (from #557).
test(integration-tests): Example 2 in SQL; the recognized idioms execute

Example 2's SQL Q3 casts its counts to DOUBLE (the design's Int64
SUM(c * c) can overflow and stays relational), and the SQL Q2 and Q3
now lower to the entropy and L2 intents, so the SQL workload has the
PromQL stand-in's Pass 1 options. The recognized entropy executes in
nats with SQL's -0.0, L2 executes exactly, and both give NULL on empty
input, decided by the exact population count.
feat(devtools): stage_pipeline example planner-layering-2

stage_pipeline plans #509 Example 2: three SQL dashboard panels over
the last minute of flows (distinct sources at eps 0.02, entropy at
0.05, L2 at 0.01, delta 0.01), lowered over its flows(ts, src_ip)
table. SQL workloads lower every entry, batch and repeating, each with
its own accuracy target (lower_sql_batch read only batch entries).

The stage-viewer tab 2 row is regenerated from it: 82 plans, 16 from
Pass 1, the rest from the identical-expression and summary-capability
rules (one UnivMon for each pair of queries and for all three); 6
valid, P1 (all exact) selected at 6.08 per second. The row previously
said 8 plans were valid.
feat(logical-asap-optimization): guarantees through a join beside exact inputs and an exactly decided CASE

SQL Example 2's L2 decides the empty-input NULL from an exact population
count cross-joined beside the estimate and a CASE on that count. No
guarantee rule covered the join or the CASE, so every SQL plan with
UnivMon for Q3 was rejected ("the output has no guarantee"), although
the L2 estimate is certified.

- A join (inner, outer or cross) whose inputs are exact but one keeps
  that input's guarantee when its predicate reads only exact columns:
  which rows match, and how often, then does not depend on an estimate.
  A key on the approximate input's columns, a semi- or anti-join, or two
  approximate inputs have none.
- A CASE whose operand and conditions are exact keeps its weakest
  branch's guarantee; a literal or NULL branch, or a missing ELSE, is
  exact. A condition over an approximate value has none.
- A projection over such a join composes each output column this way.

Guarantees are per node, so every column of the approximate input counts
as approximate; column-level tracking (#689) stays the general solution.

A root's default metric (#690) looks through the same shape: a
projection whose CASE value branches read columns (or literals, cast or
not) takes those columns' metric, following them through joins and
row-preserving operators to an aggregate's measure, so SQL's L2 root
resolves to a relative target as the PromQL one does. Any other
expression stays absolute.
docs(stage-viewer): regenerate the Example 2 row with SQL L2 through UnivMon

With the join and CASE guarantee rules, the SQL Q3 UnivMon plans are
valid: 12 of the 82 plans are valid (Q3 exact or UnivMon), 70 invalid
(Q1 HLL or UnivMon without a distinct bound, Q2 UnivMon without an
entropy bound). P1 (all exact) is still selected at 6.08 per second;
UnivMon on Q3 costs 8.08.
fix(physical-operators): charge UnivMon key memory without walking its heaps

A build charges a state's memory after every input row, and UnivMon's
charge summed the bytes of every key on every layer's heap, so building
was quadratic in the heap: a 20,000-key heap on 19 layers over 40,000
keys took 3.3 s in release. The charge now bounds the retained keys by
every heap slot holding the longest key seen (tracked on insert, merge
and adoption of a decoded sketch), in constant time. A test checks that
the charge does not grow with heap occupancy; a timing test (ignored,
release) checks the large build.
fix(logical-asap-optimization): summary-capability review fixes

- A capability group of more than 63 builds is an error: one 64-bit mask
  enumerates the subsets of at most 63, and the shift was clamped, so a
  larger group was enumerated wrongly (and took 2^63 steps) instead of
  refused.
- A summary-capability variant whose guarantee composition fails carries
  its own error, so Stage 1 lists it as a candidate that cannot be
  composed and keeps the remaining variants; one failure used to drop
  them all.
- shared_size_params no longer claims every sizing bound is monotone (a
  Count Sketch's collision bound is tighter at a power-of-two width than
  one column wider); the shared state is checked by composing every
  guarantee above it again.
fix(logical-asap-optimization): read each column's guarantee in a projection over a join

A projection's guarantee assumed the error lies only in its child's
sample-value column (the column named "value", or the only numeric
one). Over a join that column can be an exact input's, so the
approximate input's columns were read as exact: projecting
sum(m) * 1000 beside the exact side's value composed to the
approximate side's absolute error 1, while the truth is 1000.

A projection over a join (directly, or through projections, filters,
sorts, limits or deduplication) now reads each column's own guarantee
(column_guarantees), and a projection over a computed projection over a
join, whose columns' guarantees are unknown, has none. Plan selection's
operand-domain lookup reads columns the same way. column_guarantees and
join_guarantee document their assumption that every approximate node of
Stage 1 has an exact row set. Per-column tracking in general is #689.
fix(logical-asap-optimization): refuse summary-capability groups of more than 20 builds

Every subset of two or more builds of a group is a candidate, 2^n of
them; past 20 builds (about a million) the rule returns an error naming
the limit instead of enumerating.
fix(physical-operators): bound UnivMon key memory by the key bytes inserted

Charging every heap slot at the longest key seen overcounts badly: one
1 KB key in a sketch of 19 layers of 20,000 slots was charged 380 MB.
Each layer's heap keeps at most one copy of a key, so the retained keys
are also bounded by the layers times the bytes of every key inserted
(or merged in); the charge takes the smaller bound, still in constant
time. asap_sketchlib exposes no eviction hook for an exact running
total.
@zzylol

zzylol commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #707 (merged): G5, Example 2 — Pass 2 summary-capability rule, UnivMon, SQL L2/entropy idioms.

@zzylol zzylol closed this Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant