Repository navigation
Conversation
This was referenced Oct 4, 2026
zzylol
force-pushed
the
stack/509-w7b-1-frequency-runtime
branch
from
October 4, 2026 18:25
502e276 to
bceb13e
Compare
zzylol
force-pushed
the
stack/509-w7b-2-sql-frequency-recognition
branch
from
October 4, 2026 18:25
f047e41 to
66bc1a6
Compare
This was referenced Oct 4, 2026
Closed
zzylol
force-pushed
the
stack/509-w7b-2-sql-frequency-recognition
branch
2 times, most recently
from
October 8, 2026 16:47
e211fba to
fc32eae
Compare
zzylol
force-pushed
the
stack/509-w7b-1-frequency-runtime
branch
from
October 10, 2026 14:32
c0cf2fc to
c77ad29
Compare
zzylol
force-pushed
the
stack/509-w7b-2-sql-frequency-recognition
branch
from
October 10, 2026 14:32
fc32eae to
303b22e
Compare
Port of the parked #562 and #564 for #509 Example 2. The SQL frontend names `SQRT(SUM(c*c))` over Float64 grouped unit counts as FrequencyL2, and `-SUM(p*LN(p))` with `p = COUNT(*)*1.0 / SUM(COUNT(*)) OVER ()` as FrequencyEntropy converted to nats. An exact population count guards SQL's empty-input NULL. Integer products, nullable keys, filters, HAVING, other log bases and partial windows are refused. The rules are the old Pass 1 SemanticEquivalentRewriteStrategy rules, run by lower_sql at the query root so the Stage 1 pipeline sees the intents; the exact candidate is the native exact reducer. The executor gains SQL sqrt (from #562). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pass 1 takes the declared metric types since #593. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
force-pushed
the
stack/509-w7b-2-sql-frequency-recognition
branch
from
October 11, 2026 17:11
303b22e to
ae6d2f6
Compare
zzylol
added a commit
that referenced
this pull request
Oct 13, 2026
The specs and tests from #590 run the current stages through the shared planner_layering_common pipeline (Pass 1 and composition, optimize_physical_asap, select_plan); tests are named after the stages. Materialization is read from the physical DAG (#667); window forms and retention stay pending adapters. Behavior that works passes (29 tests); every expectation that needs a feature not in this stack is ignored with the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596), window composition (#601, #628, #634), materialization enumeration and MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL recognition and COUNT(*) composition (#598, #603) and plan selection row estimates (#632). Each spec's "Today" column states the current output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
3 tasks
zzylol
added a commit
that referenced
this pull request
Oct 13, 2026
The specs and tests from #590 run the current stages through the shared planner_layering_common pipeline (Pass 1 and composition, optimize_physical_asap, select_plan); tests are named after the stages. Materialization is read from the physical DAG (#667); window forms and retention stay pending adapters. Behavior that works passes (29 tests); every expectation that needs a feature not in this stack is ignored with the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596), window composition (#601, #628, #634), materialization enumeration and MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL recognition and COUNT(*) composition (#598, #603) and plan selection row estimates (#632). Each spec's "Today" column states the current output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol
added a commit
that referenced
this pull request
Oct 13, 2026
…598 Running the ignored SQL test composes every candidate and fails only on Q2's options (per-source counts until #598), so #603 is not a blocker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol
added a commit
that referenced
this pull request
Oct 13, 2026
…stages (#577) * test: acceptance spec and pending tests for planner-layering Example 1 Define MVP acceptance for #509 Example 1 before the Phase C stage APIs exist: 1 -> 6 -> 6 -> 1 candidates (Pass 1 x identical-expression sharing; physical operator implementation only, no materialization). - Spec with per-stage candidate tables, invariants and doc ambiguities. - Integration tests against todo!() stage stubs, ignored until the stages land; workload and Stage 0 frontend-shape tests run now. - Expected-only asap-stage-pipeline/v1 fixture for the DAG viewer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: run the Example 1 acceptance tests against the real stages Replace the todo!() stubs with adapters over Stage 0-3, un-ignore the 11 tests that pass, and give each still-ignored test the precise difference between the implementation and the spec. Add runtime capability checks: the selected plan compiles in the physical planner; compiling every candidate stays ignored with the two runtime gaps it finds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: read Example 1 logical stages as flat DAGs after the export removal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test: Example 1 acceptance on the composed stages (1 -> 3 -> 3 -> 1) The tests run Pass 1 and composition, optimize_physical_asap and select_plan through a shared tests/planner_layering_common pipeline, and follow the decided behavior: three Pass 1 candidates (Q2 exact, CMS+heap, CountSketch+heap), every sub-DAG NotMaterialized, P1 selected, CMS+heap invalid (weights not proven non-negative) and CountSketch+heap invalid (a Top-K output has no guarantee and no deployment evidence). Hydra (#605), Pass 2 sharing (#588) and the two runtime gaps (#593, #579) stay ignored with the PR that delivers them. Tests are named after the stages; the compile check no longer uses the PromQL fallback. The spec lists where the doc and the planner differ. The expected-only viewer fixture is removed: the generated four-stage fixture replaces it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test: port #590's Example 2-4 acceptance specs and tests The specs and tests from #590 run the current stages through the shared planner_layering_common pipeline (Pass 1 and composition, optimize_physical_asap, select_plan); tests are named after the stages. Materialization is read from the physical DAG (#667); window forms and retention stay pending adapters. Behavior that works passes (29 tests); every expectation that needs a feature not in this stack is ignored with the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596), window composition (#601, #628, #634), materialization enumeration and MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL recognition and COUNT(*) composition (#598, #603) and plan selection row estimates (#632). Each spec's "Today" column states the current output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * feat(devtools): stage_pipeline examples for #509 Example 3 Ported from #590: --example planner-layering-3a (Pattern A, ad hoc batch of five p99 reports) and planner-layering-3b (Pattern B, repeating 5-min p99), the Example 3 acceptance tests' workloads. Example 1's output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test: follow #576's per-consumer charging and exact top-k columns Plan selection charges a NotMaterialized node once per consuming query, so the charging helper checks NodeCost.multiplier and the Example 4 A3 ambiguity is recorded as decided. Example 1 checks that exact Q2 sorts by the ranked value and returns [job, topk_10]. The stage viewer guide's tab 1 row now describes the actual all-exact result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * refactor(devtools): one planning pipeline for stage_pipeline and the acceptance tests run_planning_stages and the #509 example workloads move into the asap-devtools library; stage_pipeline serializes the run and the planner-layering tests adapt it. The shared data workload drops ingestion_rate for data at rest, which validate() requires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * fix(devtools): select over every candidate; --max-displayed limits only the document stage_pipeline capped enumeration at 64, so Example 3a (243 candidates) selected P41 instead of the cheapest P122 (KLL for all five queries). The pipeline now enumerates, optimizes and prices every candidate; the document shows the first --max-displayed (default 64) plus the selected one, with `combinations` and `capped`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * docs: Example 2's SQL candidates compose; the SQL test waits only on #598 Running the ignored SQL test composes every candidate and fails only on Q2's options (per-source counts until #598), so #603 is not a blocker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test: name stages by number and name; say query-language frontends The frontends' Run field, id and locals become stage0_query_language_frontends, and test names carry stage0..stage3 with the stage name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test: charging helper follows plan selection's per-consumer rule With a materialized node the helper accepted any multiplier >= 1. It now computes each node's computations as plan selection does (walks from query roots and materialized nodes' inputs, stopping at materialized nodes), checked on a hand-built candidate with a materialized node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test: Example 1 picks Q2 candidates by (option, no shared input) physical_with and the invalid-candidate check no longer assume one candidate per Q2 option, so #588's shared-input variants need no change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * docs: Example 3 Pattern B charges each node once per consuming query Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * docs(stage-viewer): rows 2, 3a and 3b describe the planner's current output Read from uncapped runs: Example 2's sketch plans are rejected for having no (ε, δ) guarantee, 3a selects P122 (first of 108 tied cheapest plans, nothing shared yet) and 3b selects query-time KLL; panes and the latency check come with the window-summary PRs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * docs(stage-viewer): row 2 from the SQL run; 3a ties 32 plans, not 108 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test: Example 1 counts only unshared candidates; stage3_ charging-helper names The 3-logical and 3-physical counts now hold when #588 adds shared-input variants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * refactor(devtools): PlanningStagesRun fields carry stage number and name Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test(devtools): unique temp file per stage_pipeline run cargo test runs the tests as threads of one process, so two runs of the same example shared a file name and one deleted the other's output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Oct 15, 2026
#509 Example 2 writes L2(src_ip) and Entropy(src_ip) as SQL idioms over grouped unit counts. The SQL lowering (single query and batch) now names them at the query root as FrequencyL2 / FrequencyEntropy aggregates carrying the query's accuracy target, so Pass 1 offers exact and summary candidates for each: - L2: SQRT(SUM(c * c)) over COUNT(*) per key, only when the product is already Float64 (CAST(c AS DOUBLE) factors or CAST(COUNT(*) AS DOUBLE)). The Int64 product stays relational: SQL integer overflow is observable. - Entropy: -SUM(p * LN(p)) with p = COUNT(*)[*1.0] / SUM(COUNT(*)) OVER () over the complete unordered frame; the IR's bits are converted to nats and SQL's -0.0 is kept. Both require a single non-nullable Bool/Int64/Utf8 key, no HAVING and no measure filter, follow projections only, keep the original output schema, and decide SQL's empty-input NULL by an exact population count joined beside the statistic, never by an estimate. Scaled probabilities, LOG2, partitioned or ordered windows and nullable keys stay relational. Ports the frontend part of #598 (originally #562 and #564).
zzylol
added a commit
that referenced
this pull request
Oct 15, 2026
#509 Example 2 writes L2(src_ip) and Entropy(src_ip) as SQL idioms over grouped unit counts. The SQL lowering (single query and batch) now names them at the query root as FrequencyL2 / FrequencyEntropy aggregates carrying the query's accuracy target, so Pass 1 offers exact and summary candidates for each: - L2: SQRT(SUM(c * c)) over COUNT(*) per key, only when the product is already Float64 (CAST(c AS DOUBLE) factors or CAST(COUNT(*) AS DOUBLE)). The Int64 product stays relational: SQL integer overflow is observable. - Entropy: -SUM(p * LN(p)) with p = COUNT(*)[*1.0] / SUM(COUNT(*)) OVER () over the complete unordered frame; the IR's bits are converted to nats and SQL's -0.0 is kept. Both require a single non-nullable Bool/Int64/Utf8 key, no HAVING and no measure filter, follow projections only, keep the original output schema, and decide SQL's empty-input NULL by an exact population count joined beside the statistic, never by an estimate. Scaled probabilities, LOG2, partitioned or ordered windows and nullable keys stay relational. Ports the frontend part of #598 (originally #562 and #564).
zzylol
added a commit
that referenced
this pull request
Oct 15, 2026
…and UnivMon frequency statistics (#707) feat(types): summary-capability facts in the family-capability table Each family lists the readouts one of its states answers (Readout: total, item frequency, any quantile rank, distinct count, L2, entropy, top-k), so a planner rule can check that one state serves every consumer's statistic: UnivMon answers its total and every frequency moment, KLL every rank, CountSketch item frequencies and L2, the heap sketches top-k and item frequencies. Exact accumulators are read as finalized values and answer no readout. feat(logical-asap-optimization): Pass 2 summary-capability rule Composition sizes each consumer's summary build alone, for its own target. The summary-capability rule groups the builds of one candidate that chose the same family over the same input sub-DAG (window, WHERE and time range included), item and weight, grouping and filter, whose every reader is an estimate the family answers (the family-capability table's readouts), and that differ only in size. For every choice, per group, of no sharing or one subset of two or more members whose sizes differ (partial sharing included), the subset becomes one build sized with each parameter's largest value among the members (a KLL's failure probability and DDSketch's alpha: the smallest), and every guarantee above it is composed again, so plan selection checks each consumer against the shared state. Builds of equal size stay the identical-expression rule's; unshared builds are untouched and no consumer changes family. Stage 1 lists each composed candidate, its identical-expression candidates, then each summary-capability candidate followed by its own identical-expression candidates, every distinct DAG once. The Pass 2 record (Pass2Candidate) names the shared summaries ("UnivMon for Q1, Q3") and the shared identical groups. Example 2 (PromQL stand-in): with #702's sizing the three UnivMons differ (Q1 a distinct-count heap on 19 layers, Q2 the baseline shape for entropy, Q3 a one-layer Count Sketch for L2), so the rule offers one UnivMon for each pair and one for all three, each sized with the members' largest heap, rows, columns and layers and the smallest recorded distinct-count failure probability; the third query keeps its own options. A shared UnivMon whose rows and columns together exceed asap_sketchlib's hash budget (rows * (ceil(log2 cols) + 1) <= 128) is not offered, since the library would not build it. The shared UnivMon stays one build in Stage 2. feat(frontend-sql): recognize the SQL frequency L2 and entropy idioms #509 Example 2 writes L2(src_ip) and Entropy(src_ip) as SQL idioms over grouped unit counts. The SQL lowering (single query and batch) now names them at the query root as FrequencyL2 / FrequencyEntropy aggregates carrying the query's accuracy target, so Pass 1 offers exact and summary candidates for each: - L2: SQRT(SUM(c * c)) over COUNT(*) per key, only when the product is already Float64 (CAST(c AS DOUBLE) factors or CAST(COUNT(*) AS DOUBLE)). The Int64 product stays relational: SQL integer overflow is observable. - Entropy: -SUM(p * LN(p)) with p = COUNT(*)[*1.0] / SUM(COUNT(*)) OVER () over the complete unordered frame; the IR's bits are converted to nats and SQL's -0.0 is kept. Both require a single non-nullable Bool/Int64/Utf8 key, no HAVING and no measure filter, follow projections only, keep the original output schema, and decide SQL's empty-input NULL by an exact population count joined beside the statistic, never by an estimate. Scaled probabilities, LOG2, partitioned or ordered windows and nullable keys stay relational. Ports the frontend part of #598 (originally #562 and #564). feat(physical-operators): exact distinct, L2 and entropy reducers Exact Cardinality, FrequencyL2 and FrequencyEntropy aggregates had no native implementation, so the exact option of #509 Example 2's three statistics could not execute. - Reduction::Cardinality(cols) counts distinct tuples; a tuple with any NULL is skipped, and grouped key encoding keeps tuple boundaries and folds signed zero and NaN payloads. - Reduction::FrequencyL2(col) and FrequencyEntropy(col) count the unit frequencies of one Bool/Int64/Float64/Utf8 identity column: NULL is skipped, a non-finite float is an error, an empty population gives 0.0, and entropy is in bits, as AggIntent::FrequencyEntropy and UnivMon's readout define it. - The dictionaries are memory-accounted and cooperative. - Logical aggregates bind them: grouped ones (SQL COUNT(DISTINCT)) and per-series range reductions (distinct_over_time, l2_over_time, entropy_over_time), which reduce each series' window samples, restore the run's timestamp and drop the metric name as range functions do. - The analytical cost lowering prices them as hash aggregates. Ports the exact-reducer part of #597 (from #559 and #563). feat(physical-operators): native UnivMon state over typed identities UnivMon had a kernel but no native DAG state: a build was rejected ("summary family has no native DAG state implementation"), a summary build accepted only Float64 samples, and Pass 1's UnivMon update (each value of the item column one unit occurrence) did not bind, so no UnivMon candidate of #509 Example 2 could execute. - UnivMon is a native family; its states validate against their parameters, and one state answers its total, distinct count, L2 norm and entropy. - A UnivMon build takes Bool, Int64 and Utf8 identities as type-tagged key bytes (Int64 above 2^53 stays distinct) besides Float64 samples; NULL is skipped. Retained heap key bytes count toward state memory. - A unit-frequency update over one item column binds, for UnivMon and HLL, as a value-frequency build over that column: grouped (SQL src_ip) and per series (PromQL sample values). Ports the UnivMon part of #597 (from #557). test(integration-tests): Example 2 in SQL; the recognized idioms execute Example 2's SQL Q3 casts its counts to DOUBLE (the design's Int64 SUM(c * c) can overflow and stays relational), and the SQL Q2 and Q3 now lower to the entropy and L2 intents, so the SQL workload has the PromQL stand-in's Pass 1 options. The recognized entropy executes in nats with SQL's -0.0, L2 executes exactly, and both give NULL on empty input, decided by the exact population count. feat(devtools): stage_pipeline example planner-layering-2 stage_pipeline plans #509 Example 2: three SQL dashboard panels over the last minute of flows (distinct sources at eps 0.02, entropy at 0.05, L2 at 0.01, delta 0.01), lowered over its flows(ts, src_ip) table. SQL workloads lower every entry, batch and repeating, each with its own accuracy target (lower_sql_batch read only batch entries). The stage-viewer tab 2 row is regenerated from it: 82 plans, 16 from Pass 1, the rest from the identical-expression and summary-capability rules (one UnivMon for each pair of queries and for all three); 6 valid, P1 (all exact) selected at 6.08 per second. The row previously said 8 plans were valid. feat(logical-asap-optimization): guarantees through a join beside exact inputs and an exactly decided CASE SQL Example 2's L2 decides the empty-input NULL from an exact population count cross-joined beside the estimate and a CASE on that count. No guarantee rule covered the join or the CASE, so every SQL plan with UnivMon for Q3 was rejected ("the output has no guarantee"), although the L2 estimate is certified. - A join (inner, outer or cross) whose inputs are exact but one keeps that input's guarantee when its predicate reads only exact columns: which rows match, and how often, then does not depend on an estimate. A key on the approximate input's columns, a semi- or anti-join, or two approximate inputs have none. - A CASE whose operand and conditions are exact keeps its weakest branch's guarantee; a literal or NULL branch, or a missing ELSE, is exact. A condition over an approximate value has none. - A projection over such a join composes each output column this way. Guarantees are per node, so every column of the approximate input counts as approximate; column-level tracking (#689) stays the general solution. A root's default metric (#690) looks through the same shape: a projection whose CASE value branches read columns (or literals, cast or not) takes those columns' metric, following them through joins and row-preserving operators to an aggregate's measure, so SQL's L2 root resolves to a relative target as the PromQL one does. Any other expression stays absolute. docs(stage-viewer): regenerate the Example 2 row with SQL L2 through UnivMon With the join and CASE guarantee rules, the SQL Q3 UnivMon plans are valid: 12 of the 82 plans are valid (Q3 exact or UnivMon), 70 invalid (Q1 HLL or UnivMon without a distinct bound, Q2 UnivMon without an entropy bound). P1 (all exact) is still selected at 6.08 per second; UnivMon on Q3 costs 8.08. fix(physical-operators): charge UnivMon key memory without walking its heaps A build charges a state's memory after every input row, and UnivMon's charge summed the bytes of every key on every layer's heap, so building was quadratic in the heap: a 20,000-key heap on 19 layers over 40,000 keys took 3.3 s in release. The charge now bounds the retained keys by every heap slot holding the longest key seen (tracked on insert, merge and adoption of a decoded sketch), in constant time. A test checks that the charge does not grow with heap occupancy; a timing test (ignored, release) checks the large build. fix(logical-asap-optimization): summary-capability review fixes - A capability group of more than 63 builds is an error: one 64-bit mask enumerates the subsets of at most 63, and the shift was clamped, so a larger group was enumerated wrongly (and took 2^63 steps) instead of refused. - A summary-capability variant whose guarantee composition fails carries its own error, so Stage 1 lists it as a candidate that cannot be composed and keeps the remaining variants; one failure used to drop them all. - shared_size_params no longer claims every sizing bound is monotone (a Count Sketch's collision bound is tighter at a power-of-two width than one column wider); the shared state is checked by composing every guarantee above it again. fix(logical-asap-optimization): read each column's guarantee in a projection over a join A projection's guarantee assumed the error lies only in its child's sample-value column (the column named "value", or the only numeric one). Over a join that column can be an exact input's, so the approximate input's columns were read as exact: projecting sum(m) * 1000 beside the exact side's value composed to the approximate side's absolute error 1, while the truth is 1000. A projection over a join (directly, or through projections, filters, sorts, limits or deduplication) now reads each column's own guarantee (column_guarantees), and a projection over a computed projection over a join, whose columns' guarantees are unknown, has none. Plan selection's operand-domain lookup reads columns the same way. column_guarantees and join_guarantee document their assumption that every approximate node of Stage 1 has an exact row set. Per-column tracking in general is #689. fix(logical-asap-optimization): refuse summary-capability groups of more than 20 builds Every subset of two or more builds of a group is a candidate, 2^n of them; past 20 builds (about a million) the rule returns an error naming the limit instead of enumerating. fix(physical-operators): bound UnivMon key memory by the key bytes inserted Charging every heap slot at the longest key seen overcounts badly: one 1 KB key in a sketch of 19 layers of 20,000 slots was charged 380 MB. Each layer's heap keeps at most one copy of a key, so the retained keys are also bounded by the layers times the bytes of every key inserted (or merged in); the charge takes the smaller bound, still in constant time. asap_sketchlib exposes no eviction hook for an exact running total.
Contributor
Author
|
Superseded by #707 (merged): G5, Example 2 — Pass 2 summary-capability rule, UnivMon, SQL L2/entropy idioms. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased on main d4869a7 (DF 54).
Wave 1 chain: #594 → #593 → #592 → #591 → #595 → #596 → #597 → #598 → #599 (on #589)
Why
#509 Example 2 writes two of its three statistics as SQL idioms:
and says "The SQL frontend does not yet recognize the Q2 and Q3 forms as
EntropyandL2. TODO: add this recognition to the per-language frontends." Without anEntropy/L2intent, Pass 1 has nothing to attach the exact, summary and UnivMon candidates to, and Pass 2 cannot share a UnivMon with Q1.The parked #562 (L2) and #564 (entropy) added these rules as a Pass 1
SemanticEquivalentRewriteStrategyin the oldasap-aware-mapping. This PR ports them. Part of #509 / #580 (W8).What
Before (Q2, lowered):
After:
Q3 becomes the same shape with
FrequencyL2and no unit change. Stage 1 then offers PassThrough (the exact reducer from #597) and summary alternatives, including UnivMon, for each intent.How
crates/frontend-sql/src/frequency.rs: follow projections only; require a single-key unfilteredCOUNT(*)over a non-nullable Bool/Int64/Utf8 key, no HAVING; for L2,SQRT(SUM(c*c))with a product already typed Float64; for entropy,LN,p = COUNT(*)[*1.0] / SUM(COUNT(*)) OVER ()over the complete unordered frame. The result converts bits to nats, keeps SQL's-0.0, and uses an exact population count for SQL's empty-input NULL, so an approximate estimate never decides NULL. The schema must equal the original.plan_stagesdoes not run: Stage 1 only enumerates realizations of single-measure aggregates. Following the design doc's TODO ("per-language frontends"),lower_sql_dialectnow applies the rule at the query root. So the original relational DAG is no longer kept as a separate candidate. The exact alternative is the native exact reducer, and the tests show that it agrees with the original SQL.sqrt(from feat(planner): recognize floating SQL frequency L2 idioms #562). The executor's expression compiler acceptssqrton Int64/Float64/NULL.Refused, as before: the uncast integer product (which is the design's literal Q3, because SQL Int64 overflow is observable), nullable keys, HAVING, scaled probabilities,
LOG2, and partitioned or ordered windows.Tests
Ported from #562/#564:
frontend-sql/tests/frequency_{l2,entropy}.rs(recognition, refusals, accuracy propagation, exact population guard) andintegration-tests/tests/sql_frequency_{l2,entropy}.rs(original and recognized L2 agree; entropy in nats,-0.0, NULL on empty input). The legacy-search inventory tests becomestage1_offers_exact_and_summary_l2_alternativesoverenumerate_local_logical_candidates. Without thelower_sqlcall, the recognition and entropy execution tests fail.Gate: fmt and clippy (
-D warnings) pass.cargo test --workspace: 1,538 passed, 10 ignored (#589: 1,517 / 10).Ports #562, #564. Stacked on #597. Next: #599.