Skip to content

docs: design of the backend on ASAPPlanner's physical DAG and the Planner/backend boundary - #826

Draft
zzylol wants to merge 4 commits into
feat/range-queriesfrom
docs/planner-backend-design
Draft

zzylol wants to merge 4 commits into
feat/range-queriesfrom
docs/planner-backend-design

Conversation

@zzylol

@zzylol zzylol commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Why

The backend's design and its boundary with ASAPPlanner lived only in a shared draft. This puts them in the repo, next to the code.

What

docs/design/backend-on-planner-physical-dag.md: overview, architecture, the ASAPPlanner/backend boundary (responsibilities, what crosses it, rules that keep it), the SDS, the precompute and query engines, evaluation (correctness results and the benefit experiment), and the implementation plan.

Two new sections before the implementation plan:

  • Overhead of the data plane, compared with ASAPQuery: measured totals, a profile of where the data plane's CPU goes, the same steps run in-process on both implementations, and a ranked table of causes with evidence, ASAPQuery's handling of each step and the fix. Summary kernels are 3–7% of the data plane's CPU; the executor's row-at-a-time data path is 68–75%.
  • Merging with ASAPQuery: benefits and features: a feature matrix of both implementations, which component comes from which, the boundary with ASAPPlanner, a staged plan (A–G) with measured targets, risks and open decisions. The implementation plan gains rows A–G.

Notes

Update: execution engine

  • New section "Execution engine: a columnar DAG executor". The data plane always executes the installed DAG node by node, for every query shape; there is no shape-matched fast path. ASAPPlanner provides the physical operator library (one columnar operator per plan-node kind, per-sample summary updaters, RawSampleSelector); this repo's execution engine runs the DAG with it.
  • It specifies the batch format (selection vectors, interned series and group ids, dictionary labels), how sketch state is columnar (exact accumulators as numeric columns, fixed-size sketches in flat buffers, variable-size sketches one object per group, stored panes as immutable Arc<StateBatch>), every operator's columnar implementation, the per-sample ingestion pipeline, precompute worker threads, and the query engine (one query per thread, RwLock + Arc reads, never deep copies).
  • Stage C of the merge plan is now the columnar operator library in ASAPPlanner.

@zzylol zzylol added documentation Improvements or additions to documentation integration: ASAPPlanner ASAPPlanner API, IR, candidate, and selected-plan integration labels Oct 19, 2026
…e two

Profiles and same-data measurements show the data plane's extra CPU comes
from the executor's row-at-a-time data path, query-time recomputation,
JSON raw fetches, the glibc allocator and one engine thread; summary
kernels are 3-7% of it. Adds a ranked cause table with ASAPQuery's handling
of each step, a feature matrix of both implementations, the component
choice, the staged plan with targets, risks and open decisions, and the
new implementation-plan rows.
… library; per-thread query engine; per-sample ingestion
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation integration: ASAPPlanner ASAPPlanner API, IR, candidate, and selected-plan integration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant