Skip to content

Normalizer-based SQL pattern recognition (planner + query engine + ingest) - #561

Draft
hshafqat-art wants to merge 1 commit into
mainfrom
sql-normalizer-pattern-recognition
Draft

Normalizer-based SQL pattern recognition (planner + query engine + ingest)#561
hshafqat-art wants to merge 1 commit into
mainfrom
sql-normalizer-pattern-recognition

Conversation

@hshafqat-art

Copy link
Copy Markdown

Also adds ComputedLabelConfig/StatefulTransitionConfig as shared asap_types,
wires ingest-time computed-label and stateful-transition execution, and adds
the 200-query bgp_jan2024_rrc00 analyst workload.

detects ClickHouse-specific complex SQL shapes (CTE/window functions,
tokenized subqueries, MOAS, multi-aggregate) and rewrites them into
simplified queries. Both the planner and the query engine call
into this implementation.

Also adds ComputedLabelConfig/StatefulTransitionConfig as shared asap_types,
wires ingest-time computed-label and stateful-transition execution, and adds
the 200-query bgp_jan2024_rrc00 analyst workload.
@milindsrivastava1997

Copy link
Copy Markdown
Contributor

FYI @zzylol

@milindsrivastava1997

milindsrivastava1997 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

@hshafqat-art This is my understanding of what's in here. Can you confirm if this is correct and complete?

1. SQL pattern-rewrite feature — detecting ClickHouse-specific complex shapes (CTE/window funcs, tokenized subqueries, MOAS, multi-aggregate, bucketed countIf) and rewriting/planning/serving them. Spans sql_utilities/ast_matching/*, the new ComputedLabelConfig/StatefulTransitionConfig shared types, planner/sql.rs + sql/generator.rs, and the query-engine serving/ingest side (simple_engine/sql.rs, clickhouse_http.rs, servers/http.rs, engine_config.rs, new SetAggregator/DeltaSetAggregator accumulators, HLL string-hashing for non-numeric distinct targets, precompute_engine/computed_labels.rs + stateful_transition.rs).

2. CSV ingest: accept ClickHouse DateTime string timestamps, not just Unix-ms (parse_timestamp_ms/days_from_civil in csv_ingest.rs). No dependency on (1)

3. Finite CSV-ingest lifecycle fixes — explicit ctx.router.broadcast_flush() before shutdown (final windows were being dropped) + keeping the HTTP query server alive after CSV EOF instead of exiting (precompute_engine.rs bin,
file_ingest_mode/keep_query_server_alive). Also independent of (1)

4. Ingest-routing now enforces spatial_filter predicates (ingest_source.rs, sample_matches_spatial_filter + clause parser for =/!=/<>/IN (...)). Before this, an unrecognized filter clause silently let every sample through unfiltered

5. BGP query workload. Already taken care of by https://github.com/ProjectASAP/ASAPPlanner/pull/219

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants