docs: propose workload-wide planning, summary sharing, and materialization by zzylol · Pull Request #509 · ProjectASAP/ASAPPlanner

zzylol · 2026-09-30T17:59:24Z

Summary

Propose ASAPPlanner's planning stages, from language-specific frontends through logical and physical optimization to selection of one physical plan for the whole workload. The deployment supplies empirical cost and accuracy models and execution capabilities, then executes the selected plan.

Add docs/design_docs/proposals/planner-layering.md and link it from the proposals index.

Before this PR

The planning boundary needs an explicit contract for workload-wide candidates, summary sharing, materialization, and plan selection.

After this PR

Frontends produce CandidateLogicalDAGs; logical optimization produces CandidateLogicalASAPDAGs; physical optimization produces CandidatePhysicalASAPDAGs; selection returns one PhysicalASAPDAG.
Logical optimization generates local exact and summary alternatives, then applies ASAP-aware common-subexpression elimination across computations and windows.
Physical optimization explores materialization, retention and operator implementations. Outputs may be maintained at ingestion time, retained at query time, or rebuilt without materialization.
Selection evaluates complete workload candidates using deployment cost and accuracy models, rejects candidates that fail requirements or capabilities, and chooses the cheapest remaining plan.

Four worked examples cover candidate growth (3 local, 54 logical and 156 physical alternatives), UnivMon sharing across three computations, KLL sharing across windows, and ingestion/query-time materialization choices.

Scope and validation

Documentation-only proposal; no runtime or API changes. Deployment-input types, summary subtract/delete design, and physical parallelism, partitioning and resource management remain TODO.

The branch's file contents have been restored to commit 8c29c2e376800ad0823d4917e121eed3b60ed3ab and verified identical with git diff. No runtime tests run.

Add the layering proposal, its proposals-index entry, and an Output layers section in the input/output/workflow design. Every Planner layer outputs all legal candidates; the deployment keeps every summary-family candidate and selects with its own costs. Design DAG names are marked as the target API with the current main type alongside. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Split physical planning into physical design (summary materialization, workload-level, like materialized-view selection) and physical implementation (per-node lowering and cutting). The annotated graph stays a Post-ASAP DAG; materialization is decided in design and realized by the cut. State node-by-node correspondence and the Fallback exception that the operator-flattening proposal removes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Describe current Planner behaviour (no family pruning) and the backend gap, and replace the Binary 'exception' with the general rule that a timing- sensitive node needs one compilation per distinct timing, with an example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ng proposal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Planner takes the query and data workloads plus the deployment's cost model, accuracy requirements and capabilities, and returns one optimal PhysicalDAG; candidate sets stay internal. Name the annotated stage MaterializedPostASAPDAG, tabulate what each DAG encodes, fold timing and selection into the layer descriptions, rename physical design to summary materialization, and make the example trace one query through every DAG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…g row A lifecycle fixes materialization, timing, maintenance, retention and window framework together, so the stage and its DAG are named after the lifecycle (LifecyclePostASAPDAG) rather than one of those aspects. The deployment row no longer mentions ranking or selection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Describe the two plain candidate collections, the view-based compile, the caller-typed physical candidate errors, lifecycle_guarantee, and PhysicalExecution as an execution handle. Remove APIs the docs said were removed but never existed (compile_timed_candidates, PhysicalDAGCandidate), the agent instructions in the alignment proposal's baseline, and the ingestion-time Binary exception from design docs, where it is an implementation detail (the developer migration guide keeps it). Restore #485's statement that candidates do not choose placement and #508's CandidatePostASAPDAGs<Id> names in input-output-workflow.md, rejoin the split test table in physical-planning-and-deployment.md, and take planner-backend-layering.md verbatim from #509. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copies docs/design_docs/proposals/planner-backend-layering.md from #509's head 3607270 so this file merges cleanly after #509. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The layering proposal (#509) names a single annotated DAG, LifecyclePostASAPDAG: a PostASAPDAG plus its lifecycle assignment. The code kept a SummaryMaintenanceLifecyclePlan beside the DAG and named the timed collection CandidatePostASAPDAGsWithTiming. - Rename SummaryMaintenanceLifecyclePlan to LifecyclePostASAPDAG and its error to LifecyclePostASAPDAGError. The type already held the root and each state's lifecycle, retention and window framework; per-node timing stays derived by execution_assignment as the existing PostASAPDAGAssignment overlay, so no timed graph is stored beside it and no new type is added. - Rename CandidatePostASAPDAGsWithTiming to CandidateLifecyclePostASAPDAGs; CandidatePostASAPDAGs stays the logical collection. - Call layer 2 "summary lifecycle planning" and reword comments that had the deployment choose or rank; selection is Planner's, over the deployment's cost model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Several docs still said the deployment or "downstream" selects, ranks or chooses the plan, and the glossary said physical plans are owned by downstream systems. Per the layering proposal (#509), Planner compiles and selects the physical plan using the deployment's cost model; the deployment supplies prices and executes the selected plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

#509 carries the proposal; #480's copy now matches its head a095ce5 exactly, so merging either PR first leaves no conflict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Every stage after the frontend is a Post-ASAP DAG; the prefix says how far planning has gone: Logical, Lifecycle, Physical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Summary candidates are unsized (LocalCandidate::Summary carries only the algorithm); sizing happens after Pass 2 sharing. Every pruned candidate is recorded on its target with a PruneReason (RequiresExact, NullableAvgInput). Each measure of an aggregate with a realization besides exact execution is its own target; input containing ASAP operators is rejected. avg is rewritten as exact sum/count accumulator states. The Example 1 test runs Q1 as exact, and the dev doc places composition in Pass 1. - One exact option per target: the exact accumulator (ExactAccumulator, or AvgAsSumCount for avg) when it can be built, otherwise plain Exact (NullableAvgInput prune, or no accumulator form). The plain form would duplicate "accumulator, not materialized"; Stage 2/3 decide materialization and sharing. - Per-measure filters and HAVING keep every candidate. The target records the measure's filter (applied to the state as SummaryAgg.filter) and a HavingPlacement for the aggregate's having or a Filter directly above it (the SQL frontend's HAVING form): BelowAggregate when it reads only group keys (pushed below as a row predicate), else AboveOutput (a Filter above the finalize/estimate output; approximate candidates carry guarantee = None). Composition (#575) applies them; PruneReason::FilteredAggregate is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

Stage 2 materialization (#509) will decide per sub-DAG whether and when to materialize, so the Planner no longer chooses a maintenance lifecycle. - MajorPass now runs search_workload_with_targets -> global_selection -> assemble_selected_dag per root -> share_common_sub_dags. - QueryLifecyclePlan becomes QueryPlan { entry_index, root }; PlanOutput's execution_timed_dag times the roots directly. LifecycleInput, the lifecycle errors and UserInput's `lifecycle` field are gone (public API break). - Delete summary_maintenance_lifecycle, summary_maintenance_cost, summary_maintenance_dag_export, post_asap::{summary_maintenance, summary_maintenance_lifecycle} and SummaryWindowFramework (the pane primitives stay), the lifecycle CostModel hooks, CandidateCostOverrides and EmpiricalEvidenceProvider::lifecycle_cost_inputs. - Delete the lifecycle e2e test and the viewer's lifecycle-plan UI; rewire e2e_plan, summary_sharing, operator_design_examples and weighted_topk_binding to the new pipeline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Rename LifecycleAssignment to MaterializationAssignment and apply_lifecycle_timings to apply_materialization_timings. The default assignment is now all_query_time(): nothing is materialized until Stage 2 materialization (#509) decides per sub-DAG. all_ingestion_time() and set() assign maintenance explicitly. - validate_default becomes validate_maintained: candidate legality is still checked with every summary maintained, so candidate generation is unchanged. planned_data_state and fixed_window_rate_candidates use the same maintained assumption, and the maintained precompute compilers in promql_rows assign ingestion time explicitly. - PlanOutput::execution_timed_dag, show_post_asap_ir and the default test helpers now emit query-time summaries. - forces_query_time (#541) treats a SummaryAgg as query time when the assignment, including its default, puts it there. - Tests that exercise maintenance assign it explicitly (new `maintained` helpers); a new unit test pins the query-time default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zzylol and others added 3 commits September 30, 2026 17:59

docs: treat source binding as a PhysicalDAG execution step

f1c13d4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zzylol mentioned this pull request Sep 30, 2026

docs: take node timing from lifecycle assignment in the flattening proposal #481

Closed

zzylol and others added 5 commits September 30, 2026 18:27

docs: point per-operator export at #481

6702d3a

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: drop Binary lowering detail from the layering proposal

1e84c15

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: drop mixed-placement and compatibility sections from the layeri…

cd6a18d

…ng proposal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: drop the implementation-status section from the layering proposal

aa0b959

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Selvomega self-requested a review September 30, 2026 18:36

docs: leave input-output-workflow unchanged in the layering PR

62ae3e7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Selvomega requested changes Sep 30, 2026

View reviewed changes

Comment thread docs/design_docs/architecture/input-output-workflow.md Outdated

Comment thread docs/design_docs/architecture/input-output-workflow.md Outdated

Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated

docs: move selection into ASAPPlanner; deployment supplies prices

3607270

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Selvomega requested changes Sep 30, 2026

View reviewed changes

Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated

Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated

Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated

zzylol mentioned this pull request Sep 30, 2026

refactor!: align DAG APIs and share candidate graphs #480

Closed

zzylol requested a review from Selvomega September 30, 2026 20:37

Selvomega previously approved these changes Sep 30, 2026

View reviewed changes

Comment thread docs/design_docs/proposals/planner-layering.md Outdated

zzylol requested a review from milindsrivastava1997 September 30, 2026 20:51

docs: rename layering proposal to planner-layering.md

a3c6dbc

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zzylol dismissed Selvomega’s stale review via a3c6dbc September 30, 2026 21:54

zzylol added a commit that referenced this pull request Sep 30, 2026

docs: follow #509's rename to planner-layering.md

67d01c9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: name the DAG stages LogicalPostASAPDAG and PhysicalPostASAPDAG

41076ca

Every stage after the frontend is a Post-ASAP DAG; the prefix says how far planning has gone: Logical, Lifecycle, Physical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zzylol mentioned this pull request Oct 12, 2026

refactor(planner): replace the summary lifecycle with #509 materialization options #667

Merged

3 tasks

Navigation Menu

Repository navigation

docs: propose workload-wide planning, summary sharing, and materialization - #509

docs: propose workload-wide planning, summary sharing, and materialization#509
zzylol merged 56 commits into
mainfrom
docs/planner-layering-contract

zzylol commented Sep 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

zzylol commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Before this PR

After this PR

Scope and validation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zzylol commented Sep 30, 2026 •

edited

Loading