From 7bc1ffb6284a148f8b6a4496bc3104fed62d9b09 Mon Sep 17 00:00:00 2001 From: zzylol Date: Tue, 20 Oct 2026 19:58:02 +0000 Subject: [PATCH 1/8] docs: design the backend on ASAPPlanner's physical DAG; restore the precompute and query DAG execution designs --- .../design/backend-on-planner-physical-dag.md | 321 ++++++++++ docs/design_docs/precompute-dag-execution.md | 563 ++++++++++++++++++ docs/design_docs/query-dag-execution.md | 320 ++++++++++ 3 files changed, 1204 insertions(+) create mode 100644 docs/design/backend-on-planner-physical-dag.md create mode 100644 docs/design_docs/precompute-dag-execution.md create mode 100644 docs/design_docs/query-dag-execution.md diff --git a/docs/design/backend-on-planner-physical-dag.md b/docs/design/backend-on-planner-physical-dag.md new file mode 100644 index 00000000..d7aff6e8 --- /dev/null +++ b/docs/design/backend-on-planner-physical-dag.md @@ -0,0 +1,321 @@ +# ASAPQuery-backend on ASAPPlanner's physical DAG + +ASAPPlanner turns a query workload and deployment inputs (such as runtime capabilities, prices, and measured costs) into one physical execution plan: a directed acyclic graph (DAG) of operators. ASAPQuery-backend installs that plan, ingests samples, maintains the summaries specified by the plan, and executes its operators to answer queries. + +This document designs the backend: what it must achieve, how its components work, where it meets ASAPPlanner, and how it merges with ASAPQuery. The current implementation status and the latest measurements are in the appendix. + +## Contents + +- [Problem](#problem) +- [Goals and non-goals](#goals-and-non-goals) +- [Architecture](#architecture) +- [Boundary between ASAPPlanner and ASAPQuery-backend](#boundary-between-asapplanner-and-asapquery-backend) +- [Component design](#component-design) + - [Control plane](#control-plane) + - [Stored summaries (SDS)](#stored-summaries-sds) + - [Precompute engine](#precompute-engine) + - [Query engine](#query-engine) + - [Raw samples](#raw-samples) + - [Concurrency and memory](#concurrency-and-memory) + - [Failure handling](#failure-handling) +- [Evaluation](#evaluation) +- [Alternatives considered](#alternatives-considered) +- [Merging with ASAPQuery](#merging-with-asapquery) +- [Open decisions](#open-decisions) +- [Related designs](#related-designs) +- [Appendix: status and latest measurements](#appendix-status-and-latest-measurements) + +## Problem + +A metrics workload repeats many queries over the same data: dashboards, alerts, recording rules. ASAPPlanner chooses, for the whole workload, what to precompute as summaries (exact accumulators or sketches), when to build them (during ingestion or at query time), and how long to keep them. That choice only pays off if a runtime executes it faithfully and cheaply. The runtime must: + +- build exactly the summaries the plan names, with the same semantics as the plan's reference execution; +- answer every planned query from them; +- cost less than answering the same queries with the database alone. + +## Goals and non-goals + +**Goals** + +- **One planner.** Every planned query executes the plan ASAPPlanner selected, unchanged. +- **Correct answers.** Each answer equals the Planner's reference execution of the same plan, with every node at query time, within the plan's accuracy target. +- **A measurable benefit.** Lower total CPU, memory and latency than Prometheus alone on the same workload, reported in absolute numbers. +- **Any query shape the Planner emits** runs through the same DAG execution, with no per-shape code. + +**Non-goals** + +- Re-planning when the workload or data changes. A new workload is planned and installed again. +- Durable state and recovery after restart. State is in memory and is rebuilt from new samples. +- SQL workloads. The control plane plans PromQL today, and the design keeps the language a Planner concern. + +## Architecture + +```mermaid +flowchart TB + W[Workload + deployment inputs] --> PL + subgraph CP[Control plane] + PL[Call ASAPPlanner once] --> P[(InstallablePlan)] + end + subgraph DP[Data plane] + subgraph ING[Ingestion] + RW[Remote Write] --> PE[Precompute engine
Update state per sample] + end + subgraph QUERY[Query execution] + Q[Planned query at T] --> QE[Query engine
Execute the query's DAG root] + QE <--> PROM[Prometheus
remote read, unplanned queries] + end + PE -->|Store summaries| SDS[(SDS
Immutable summaries)] + QE <-->|Read summaries / retain if planned| SDS + end + + P -.-> PE + P -.-> QE + P -.-> SDS + + classDef input fill:#f8fafc,stroke:#64748b,color:#0f172a + classDef engine fill:#dbeafe,stroke:#2563eb,color:#1e3a8a + classDef storage fill:#fef3c7,stroke:#d97706,color:#78350f + class W,RW,Q,PROM input + class PL,PE,QE engine + class P,SDS storage + style CP fill:#f0f9ff,stroke:#0284c7,color:#0c4a6e + style DP fill:#f8fafc,stroke:#64748b,color:#0f172a + style ING fill:#eff6ff,stroke:#93c5fd,color:#1e3a8a + style QUERY fill:#eff6ff,stroke:#93c5fd,color:#1e3a8a +``` + +Dashed arrows show which data-plane components receive the installed plan; solid arrows show planning inputs and runtime operations. +- The precompute engine writes summaries to SDS. +- The query engine reads summaries, and can store query-time summaries for reuse. +- A query's DAG root is the operator that produces its final answer. +- SDS stores the immutable summaries the plan specifies: time panes, rollups over time ranges, rollups across series, and sliding-window summaries. A pane is one kind of summary, covering a bounded time interval; it is not the only kind SDS can hold. + +**Repositories.** The backend's code lives in ASAPQuery-backend for now and later merges into ASAPQuery. ASAPPlanner stays a separate repository, used as a library pinned to a specific revision. + +## Boundary between ASAPPlanner and ASAPQuery-backend + +ASAPPlanner decides **what to compute**, **whether to compute it during ingestion or at query time**, and **how long to retain stored outputs** before eviction. ASAPQuery-backend decides **how to execute that plan on incoming samples and query requests**. The backend calls the Planner once per installed workload and executes the returned plan without modifying it. The Planner consumes descriptions of deployment capabilities and costs; it does not operate the deployment's storage or services. + +| Concern | ASAPPlanner | ASAPQuery-backend | +| --- | --- | --- | +| Plan search | Stage 1 generates candidate plans; Stage 2 chooses which intermediate summaries to store; Stage 3 selects a plan using cost and accuracy targets | None | +| Installed plan | One validated DAG, with the operators for constructing and combining time panes already included | Installs it | +| Stored outputs | Specifies each output's computation phase (ingestion or query time), time coverage, and retention duration | Stores them | +| Identifying shared summaries, avoiding duplicate storage, storage layout, eviction | None | SDS | +| Operators | The physical operator library (`asap-executor`): one columnar operator per plan-node kind, per-sample state updaters, `RawSampleSelector` | The execution engine: walks the DAG node by node with those operators | +| Ingestion, watermark, pane close, late data | None | Precompute engine | +| Threads, scheduling, binding a query to its evaluation timestamp | None | Query engine and precompute engine | +| Raw samples | Specifies the time ranges of raw samples a query needs | Local ring buffer, Prometheus remote read | +| Unplanned queries | None | Forwarded to Prometheus, or refused in no-forward mode | +| Deployment inputs (capabilities, AWS Fargate prices, measured costs, data facts) | Consumes them | Supplies them | + +**Planning interface: workload and deployment models → executable plan** + +Once per installed workload, the backend control plane supplies the query workload and deployment inputs (capabilities, costs, and data facts) to `asap_planner::plan_installable`. The Planner returns `InstallablePlan`, which the backend saves and installs directly. + +| Plan field | Role | +| --- | --- | +| `dag` | The selected operators and their dependencies. | +| `queries` | Identifies each planned query and the DAG root that answers it. | +| `stored_outputs` | Tells SDS and the engines which summaries to retain, whether ingestion or query execution builds them, their time coverage, and their retention duration. | +| `query_time_raw_windows` | Tells the runtime which raw-sample windows to fetch for query execution, and sizes its local raw buffer. | +| `selection` | Metadata identifying the selected candidate and its estimated price per second, used for logging and reporting rather than query computation. | + +The DAG describes the computation. The stored-output and raw-window metadata make its storage and input requirements explicit, so the backend does not reconstruct them by walking the DAG. Selection metadata describes the planning decision and is separate from the computation itself. + +**Runtime interface: samples and summaries → query answers** + +After installation, the backend uses the operator library to execute the plan. These are runtime helpers, not plan fields or calls to the Planner. + +| Runtime helper | Role | +| --- | --- | +| `RawSampleSelector` | Selects the raw samples that belong to a pane or query-time window: the window's metric, its `Scan` label matchers, the left-open time range, the latest sample per series for instant selectors, and no stale markers. | +| `StateColumnUpdater` | Updates aggregate or sketch states as samples arrive, merges states across shards, and seals them when a pane closes. | +| `Batch::try_from_columns` | Constructs a batch from state columns. The backend stores the batch as an immutable summary; a sealed pane is one example. | +| Compiled operators for DAG nodes | Execute the operations specified by the plan, using input batches to produce output batches. | + +The Planner specifies **which summaries to compute and retain**. The operator library supplies **the computation implementations**. The backend supplies **ingestion, scheduling, storage, and query serving**. + +**Rules** + +- The backend depends only on `asap_planner` and `asap_executor`. Planner internals (frontier search, pane-chain expansion) are crate-private. +- The Planner never sees samples, stores, clocks or services. The backend never changes a plan. +- The data plane always executes the installed DAG node by node, for every query shape. Query shapes do not have separate execution paths that bypass the DAG. +- Each operator has exactly one implementation, in the operator library. The Planner's tests, the correctness reference and the data plane all use it. +- If the backend cannot execute a plan, installation fails with a reason before any part of that plan runs. + +## Component design + +### Control plane + +- **Input:** a PromQL workload (queries, recurrence, accuracy targets, optional range evaluation) and deployment inputs: + - capabilities: supported summaries, whether raw data is retained, whether query-time results may be kept; + - plan selection settings: cost model, unit prices, measured operation costs; + - data facts: metric types, cardinalities, value domains. +- **Planning:** one `plan_installable` call. The control plane never selects among candidates. +- **Install:** the plan is validated and written as `plan.json`. The data plane loads it at startup. In no-forward mode, installation fails unless every query has an executable plan. + +### Stored summaries (SDS) + +SDS holds every summary the plan stores, and nothing else. + +- **Definitions.** Each stored output in `stored_outputs` maps to one summary definition. A definition's identity is a hash of the sub-DAG that builds it: payloads and schemas, with offset-only `TimeShift`s removed. Outputs that differ only in time offset therefore share one definition. For example, six 10-minute KLL panes read at offsets 0 to 50 minutes are one definition. +- **Keys.** Summaries are keyed by (definition, end of their time coverage). A time-bounded summary covers the left-open interval `(end − width, end]`; a pane is the case built directly from raw samples. +- **Kinds.** Any output the plan stores: time panes, rollups over time ranges, rollups across series, sliding-window summaries, and query-time results kept across evaluations. Each comes from `stored_outputs` with its inputs (raw samples or other stored outputs), its time coverage, its build phase and its retention. +- **Immutability.** A summary never changes after insertion. It is shared by reference (`Arc`) and never copied. +- **Retention.** Each definition is kept for the largest `retention_ms` among its outputs, which already includes range-evaluation spans. Older summaries are evicted per definition. +- **Not read:** summary coverage. Reuse decisions were made by the Planner; SDS stores what the plan names. + +### Precompute engine + +The precompute engine builds each ingestion-time summary exactly once, as soon as its inputs are complete: a summary over raw samples when the watermark closes its time window, and a summary over other stored summaries (a rollup over time ranges or across series, or a sliding-window summary) once all of those inputs are sealed. Summaries are built in the plan's dependency order. + +1. **Route.** Each Remote Write series is interned to a dense series id at decode. Then, once per series, it is matched against each definition's `RawSampleSelector` (metric and label matchers), and its group in that definition is computed. The result is cached by series id. +2. **Update.** Each sample updates its open pane's state in place through `StateColumnUpdater`, the same code the batch build uses. A per-sample pane therefore equals a pane built from the same samples in one batch. +3. **Close.** A pane `(end − width, end]` closes when the watermark passes `end`. + - The watermark is the newest routed sample's time minus the allowed lateness. + - Samples with timestamps beyond a bounded distance into the future are dropped and counted; they never advance the watermark. +4. **Seal.** Each worker seals its states, `Batch::try_from_columns` turns them into a batch, and the batch is inserted into SDS. A group split across workers is built from the combined samples with the plan's batch build. +5. **Evict.** Per definition, by retention. + +**Rules** + +- **Late samples:** a sample for an already-built pane is dropped and counted. There are no revisions. +- **Startup:** the first pane is skipped, because it may be partial. +- **Back-pressure:** Remote Write returns 503 when the queued samples exceed a bound. +- **Counters:** accepted, late-dropped, first-pane-skipped and future-dropped samples are exposed. + +### Query engine + +A planned query is answered by executing its DAG root at each evaluation time `T`. + +- **Compile once.** The plan is compiled once at install and shared by every query thread. +- **One query per thread.** Queries run on a fixed pool. One evaluation, and all steps of a range query, run on one thread. +- **Inputs of one evaluation at `T`:** + + | Input | How it is read | + | --- | --- | + | Summary built at ingestion | The definition's summary whose coverage ends at `T − offset` | + | Query-time summary kept across evaluations | Reused if an earlier evaluation built it after its coverage closed; otherwise computed now, and stored in SDS only once its coverage is closed | + | Query-time raw window | Raw samples for `(T − offset − width, T − offset]`, selected by the window's `RawSampleSelector` | + +- **Range queries.** One evaluation per step, `T = start + k · step`, on the query's thread. Raw samples are read once for the range and sliced per step, and summaries are shared across steps. +- **Forwarding.** An evaluation that cannot run as planned is forwarded whole to Prometheus, with the reason in `x-asap-forward-reason`. Reasons include: + - a missing summary, for example one older than its retention; + - a `T` off the pane grid; + - a root that cannot be compiled. + + In **no-forward mode** the same cases return an error with the reason, so every answer is computed by ASAP. + +### Raw samples + +- **Local buffer.** Query-time raw windows are served from a per-metric ring buffer filled by Remote Write. Each metric's buffer is sized by the longest query-time raw window of that metric in the plan. Metrics with no query-time raw window are not buffered. +- **Remote read.** Anything older than the buffer is read from Prometheus with remote read (protobuf). +- Stale markers are not samples. `RawSampleSelector` drops them, and an instant selector whose latest sample is stale yields no row. + +### Concurrency and memory + +- **Workers.** Ingestion is sharded by series id across N workers. Each worker owns its open-pane states, so updates take no lock. The global watermark is the minimum of the workers' watermarks. +- **Store.** SDS is a read-write lock over (definition → end → `Arc` summary). + - Queries take the read lock only to clone the `Arc`s they need. + - Inserts and evictions take the write lock briefly. +- **Memory accounting.** A query's execution memory limit charges what the query allocates: new states, merged results, copies. It does not charge the shared summaries it reads, which SDS already holds. +- **Allocator.** jemalloc is the global allocator. + +### Failure handling + +- **Installation** fails, with a reason, on a plan the backend cannot run. +- **Out-of-range timestamps** and arithmetic on pane boundaries are checked, so a bad sample is dropped and counted rather than stalling or crashing the engine. +- **Engine failure:** if an engine thread fails, the data plane reports an internal error and becomes unready. It does not report itself busy indefinitely. + +## Evaluation + +- **Correctness.** End-to-end tests run the data-plane binary on the planner-layering PromQL examples. Data enters through Remote Write, and every answer is compared with the Planner's reference execution of the same plan. Equality is exact, except within a stated tolerance for sketches with randomized compaction. +- **Benefit.** The same workload runs on one deployment twice: + - once with Prometheus alone answering every query exactly (the baseline); + - once with the backend executing the plan. + + Reported, in absolute numbers: + - CPU core-seconds of every service, including the backend deployment's own Prometheus; + - peak memory; + - p50/p99 latency per query; + - error against the exact answer, and whether it meets the target; + - the number of forwarded evaluations. + + Benefit runs use no-forward mode, so forwarded work cannot hide in Prometheus' CPU. +- **Workloads.** The planner-layering examples, the Kubernetes recording-rule set, and ASAPQuery's random mixed workloads, over seeded generated data. The deployment inputs are AWS Fargate us-east-1 unit prices, measured operation costs, and distinct-count bounds read from the generated data. +- **Cost feedback.** The runtime measures its own per-operation costs on the deployment machine and supplies them as Planner inputs. Stage 3's predicted resource use for the selected plan is reported next to the measured use. + +## Alternatives considered + +- **Shape-matched fast paths** (hand-written code for common query shapes beside the DAG). Rejected: they need a second implementation of every shape kept in step with the DAG, and they cannot cover arbitrary roots. The data plane always executes the DAG; speed comes from the columnar operator library. +- **SDS inside ASAPPlanner.** Rejected: identity, deduplication, layout and eviction are deployment concerns. The Planner states only each output's computation, coverage and retention. +- **Row-at-a-time execution.** Replaced: per-row values, clones, grouping keys and memory accounting dominated data-plane CPU, and the sketch kernels were a small fraction. All operators now work on Arrow column batches with state columns. +- **The runtime deriving storage needs from the DAG.** Replaced by `stored_outputs`. The Planner already knows each output's materialization and retention, including range-evaluation spans, so the runtime reads them instead of re-deriving them. +- **Storing only raw-sample panes.** Rejected: the plan also stores rollups, sliding-window summaries and kept query-time results. Storing only panes would recompute those at every evaluation, or refuse them at install. + +## Merging with ASAPQuery + +The merged runtime keeps ASAPQuery's low-overhead data path and the backend's single planner and DAG execution. + +**Key points** + +1. **One planner.** ASAPPlanner replaces `asap-planner`. Every planned query executes the plan selected by Stage 3. +2. **One executor.** The data plane executes the installed DAG with ASAPPlanner's columnar operator library. It replaces `SimpleEngine` and pattern matching. +3. **ASAPQuery's data path.** Sharded per-sample ingestion with a per-worker watermark (the global watermark is their minimum), immutable `Arc` state shared by reference, jemalloc. The per-sample updates use the operator library's updaters, so incremental state updates produce the same states as building them from a batch of samples. +4. **The backend's SDS.** Identify summaries by their definitions, reuse them across query time offsets, and evict summaries according to the plan's retention duration. +5. **Read raw samples locally when available.** A ring buffer filled by Remote Write serves query-time raw windows; Prometheus remote read covers anything older. +6. **ASAPQuery's features come along.** OTLP and VictoriaMetrics ingestion, idle-window close, the query tracker (now feeding ASAPPlanner), plan swap at runtime, the drop-in deployment and the PromQL compliance suite. +7. **Measured costs feed future planning.** Supply runtime measurements as deployment inputs for subsequent Planner calls; they do not modify an already installed plan. + +**Component choice** + +| Component | From | +| --- | --- | +| Planner, installed plan, stored outputs, retention | ASAPPlanner | +| Operators and per-sample updaters | ASAPPlanner's operator library | +| DAG execution engine, SDS | Backend | +| Ingestion drivers, sharded router, watermark | ASAPQuery | +| Store concurrency | ASAPQuery's model with the backend's immutable summaries | +| Local raw windows, remote read | Backend | +| HTTP adapters, forwarding, tracker, drop-in | ASAPQuery | + +**Remaining integration work** + +1. Replace ASAPQuery's planner and query-pattern execution paths with the backend's `InstallablePlan` installation and compiled DAG execution. +2. Connect ASAPQuery's additional ingestion drivers, workload discovery, query tracking, runtime plan replacement, and deployment support to that execution path. +3. Preserve summary identity, retention, and per-sample update semantics across the combined runtime, and validate answers against the Planner's reference execution. +4. Extend measured-cost coverage to every operator the runtime pays for, so Stage 3's predictions match measured use. + +## Open decisions + +- **Pane retention.** The runtime keeps ingestion-built panes for one extra pane width beyond the plan's retention, to cover evaluations just behind the newest closed pane. Whether the Planner should model the deployment's evaluation delay explicitly remains a design question. +- **Idle shards.** A shard that stops receiving samples no longer advances its watermark (the timestamp used to decide which panes can close). Its panes remain open unless the runtime closes them using elapsed wall-clock time; that mechanism is not implemented yet. +- **The local raw buffer.** Expose it as an explicit capability with a cost the Planner can compare, or keep it as a runtime choice whose effect is reflected in measured costs. +- **Planner dependency weight.** `plan_installable` lives in `asap-planner`, which also builds the SQL and MetricsQL frontends. A PromQL-only runtime would benefit if those frontends were optional features. + +## Related designs + +The precompute and query engines have their own detailed designs: +- [Precompute DAG execution](../design_docs/precompute-dag-execution.md): installing the plan, routing samples to workers, executing ready work, and publishing the selected DAG's outputs. +- [Query DAG execution](../design_docs/query-dag-execution.md): compiling, binding and executing a query's DAG root, and request consistency. + +Both were written for the earlier implementation, before the runtime moved onto ASAPPlanner's installable plan. Where they differ from this document, this document is current. Reconciling them with it is open work. + +## Appendix: status and latest measurements + +**Status.** The backend runtime is a stack of pull requests in ASAPQuery-backend: control plane, SDS, precompute engine, query engine, correctness tests, kept query-time results, range queries, the installable-plan handoff, concurrency, per-sample ingestion, local raw windows, the columnar engine, series interning and cost feedback. They merge in that order after review. ASAPPlanner's columnar operator library is complete. + +**Latest reported performance.** This is the 90-minute benefit harness with Prometheus 3.5.0, four queries, and four CPU cores allocated to each service. `s10` and `s50` are the 1,000- and 5,000-series configurations. Prometheus alone was measured in the same session, and the backend setup's Prometheus still receives the samples. +- **CPU** is core-seconds over the complete run; lower is better. +- **Total CPU** includes both services. +- **Peak RSS** is backend-only, not total system memory. + +| Configuration | Prometheus alone CPU | Backend CPU | Backend's Prometheus CPU | Total CPU | Backend peak RSS (GiB) | Forwarded evaluations | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| s10, raw retained | 18.9 | 11.2 | 8.6 | 19.8 | 0.04 | 90 | +| s10, no raw retained | 19.9 | 19.4 | 6.3 | 25.7 | 0.34 | 5 | +| s50, raw retained | 86.7 | 52.0 | 41.7 | 93.7 | 0.11 | 90 | +| s50, no raw retained | 85.3 | 84.3 | 29.9 | 114.2 | 0.81 | 5 | + +In the raw-retained runs, the 90 forwarded evaluations are quantile queries whose selected exact plan the executor could not run at the time. The Planner now rejects such candidates and the executor runs exact per-series aggregates. Because forwarded work was done by Prometheus, the lower backend CPU and RSS in those runs partly reflect work done upstream. Combined CPU is above Prometheus alone in all four configurations, so these results do not yet establish a whole-system saving. All evaluations met their accuracy targets. diff --git a/docs/design_docs/precompute-dag-execution.md b/docs/design_docs/precompute-dag-execution.md new file mode 100644 index 00000000..d7f0163a --- /dev/null +++ b/docs/design_docs/precompute-dag-execution.md @@ -0,0 +1,563 @@ +# Precompute DAG execution + +Audience: backend designers and developers. + +This document specifies the precompute engine design: how it receives a +Planner-provided physical candidate selected by Backend, runs its DAGs over +data partitions, and publishes the +stored outputs consumed by query plans. The [plan split](https://github.com/ProjectASAP/ASAPQuery-backend/blob/54067d4b44fe812fdd746209b522e77003e8d480/docs/design_docs/asapplanner-integration.md) +and [SDS contract](https://github.com/ProjectASAP/ASAPQuery-backend/blob/54067d4b44fe812fdd746209b522e77003e8d480/docs/design_docs/summary-catalog-sds-architecture.md) define the compiler and +storage contracts used here. + +## 1. Intuition: compute once, reuse across queries, amortize the cost + +Precompute performs shared work ahead of query execution and stores its selected +outputs. Later queries reuse those outputs repeatedly, amortizing the cost of +constructing and maintaining them across the queries they serve. The purpose is +to reduce both total resource cost and query latency: repeated queries avoid +repeating expensive input scans and computation, while the work left on the +query path is a stored-state read and the remaining query operators. + +For example, repeated p50 and p99 requests for request latency by service over +the same five-minute range can share one Planner-selected KLL producer. +Precompute builds the KLL state once per service and scheduled window. Every +compatible request then reads that stored state and applies its percentile +readout, rather than scanning the samples and rebuilding the summary for each +request. Reuse applies both to repeated executions of a query and to different +queries that share the selected output. + +Over a workload, the resource cost is the cost of construction, maintenance and +storage plus the reads and remaining query computation. Reuse must save enough +repeated work to offset those costs; this is what makes precomputation worthwhile. +Once the required output is ready, query latency excludes its construction work. +“Compute once” refers to a particular output, population and window: subsequent +windows still require construction or updates under the selected schedule. + +The physical compiler splits the selected DAG at its stored outputs: + +```mermaid +flowchart LR + subgraph P[PrecomputePlan] + I[Read latency samples] --> G[Group by service] + G --> K[Build KLL state] + K --> W[Write latency-kll] + end + W --> S[SummaryStore] + subgraph Q[QueryPlans] + R[Read latency-kll] --> P50[Estimate p50] + R --> P99[Estimate p99] + end + S --> R +``` + +These labels describe the computation, not a separate backend operator language. +The executable nodes preserve Planner operator semantics and their provenance; +the compiler adds concrete source and storage bindings. + +A worker runs the **whole precompute subgraph for its assigned data partition**. +For this example, one worker handles `service=api` and another can handle +`service=worker`. Both execute the same graph. Each worker runs its nodes +sequentially in dependency order; workers process independent partitions +concurrently. A worker is an execution task, not necessarily a dedicated OS +thread. + +Only outputs selected for persistence become stored summaries. Intermediate +values remain in worker memory unless the plan explicitly stores them. A single +precompute subgraph can have several stored outputs, and several query plans +can read the same output. + +## 2. Inputs and outputs + +The physical compiler receives the selected post-ASAP DAG and the selected +deployment guarantee and schedule/retention. It emits one coherent plan version +containing definition rows, a PrecomputePlan, and matching QueryPlans. + +| Input to the precompute engine | Purpose | +| --- | --- | +| `PrecomputePlan` | Defines the precompute subgraph, node IDs, operator parameters, dependency edges, input bindings and stored-output bindings. Carries the selected deployment guarantee and schedule/retention. | +| Definition rows | Describe the input, Planner family, parameters, grouping and time semantics of each selected stored output. Installed in `SummaryStore.summary_definitions`. | +| Source data | Raw samples or rows, or explicitly referenced committed summary state. The plan determines which input type each node accepts. | +| Execution triggers | Input arrival, scheduled evaluation endpoints, and input-completion signals, according to the selected maintenance mode. | +| `PrecomputeEngineConfig` | Supplies worker count, queue capacity and operational settings such as flush polling interval. These settings cannot change computation, logical windows or the selected schedule/retention. | + +The engine produces committed `StoredSummary` records for the selected outputs. +It also reports completion or failure for scheduled work and drain requests. +Installing a plan does not imply that its output records are ready. + +A QueryPlan consumes these records through a `StoredOutputReference`. That +reference names a stored output and definition within the installed plan +version; population and window selection identify the records to read. + +## 3. Install the plan once + +The backend derives an immutable `InstalledPrecomputePlan` from the supplied +PrecomputePlan. This runtime representation avoids validating the graph and +binding operators again for every input batch. + +Installation performs four steps: + +1. Validate the graph and operator input/output types, including ordered edge + roles for operators with multiple inputs. Validate source, definition and + stored-output bindings together with the matching QueryPlans. +2. Bind supported runtime operators and compile a dependency execution order. + Preserve Planner families and parameters: Rate and Increase remain distinct, + and grouping does not create a separate backend family. +3. Derive a partitioning rule that keeps all required dependencies and + reductions local to a worker, as described below. Check that execution can + satisfy the selected deployment guarantee and schedule/retention. +4. Build node and source-routing indexes, register the validated definition + rows, and activate the coherent plan version for routing and execution. + +Unsupported operators, incompatible bindings, cycles and unsatisfied deployment +requirements fail installation before the plan becomes active. + +| Object | Lifetime and contents | +| --- | --- | +| `PrecomputePlan` | Compiler-supplied computation and bindings for one plan version. | +| `InstalledPrecomputePlan` | Backend-derived execution order, immutable operator programs, bindings, routing indexes and validated partitioning rule. Shared by the router and workers. | +| Worker execution state | Mutable node state, input ordering buffers, windows and intermediate results for that worker's assigned partitions. | + +`InstalledPrecomputePlan` is internal runtime data, not another serialized +configuration or independently installable plan. All execution lookup tables +come from the validated DAG and its bindings. There is no separately maintained +aggregation list. Operator fusion is permitted only when it preserves the DAG's +semantics and physical-to-semantic provenance. + +## 4. Route data so each worker can execute the whole subgraph + +Partitioning is a property of the entire precompute subgraph. All inputs that a +node must combine must reach the same worker. The router applies the installed +partitioning rule to canonical input labels and assigns that partition to a +worker for the lifetime of the active execution state. + +| Computation | Required data placement | +| --- | --- | +| KLL or Sum grouped by service | Keep all inputs for one service together. | +| Per-series Rate | Keep each series together and preserve its sample-time semantics. | +| Per-series Rate followed by Sum by service | Partition by service; keep separate Rate state for each series within the worker, then reduce locally. | +| Global Sum | Place all inputs on one worker in the simple execution model. | + +For Rate followed by Sum, distributing individual series independently could +put two series of the same service on different workers. Neither worker could +then compute the complete service result. Partitioning by service satisfies +both the per-series state requirement and the downstream reduction. + +The simple design uses one validated partitioning rule for the installed +precompute subgraph. If independent partitions cannot be established, use one +worker when the deployment requirements permit it; otherwise reject the plan. +There is no implicit shuffle or cross-worker partial-result merge. A hot +partition therefore remains limited by its owning worker. + +Partition ownership is separate from stored-output identity. Hashing a stored +output ID does not establish that its input dependencies are local. Multiple +stored outputs can share one data partition and one worker. + +```mermaid +flowchart TD + I[Bound input data] --> R[Route by validated data partition] + P[InstalledPrecomputePlan] -. shared program .-> A + P -. shared program .-> B + R --> A[Worker 0: entire subgraph for service api] + R --> B[Worker 1: entire subgraph for service worker] + A --> S[SummaryStore] + B --> S +``` + +Worker queues are bounded. Admission applies backpressure when a destination +queue is full; an atomic input batch must reserve its required queue capacity +before being acknowledged. Sequential queue consumption does not itself sort +event timestamps. Input ordering, allowed lateness and counter-reset handling +must follow the bound operators' time semantics. + +## 5. Execute ready work inside the owning worker + +Each worker holds the same installed program and separate mutable state. An +input batch updates only the state for its assigned data partition. A scheduled +evaluation identifies the logical window whose outputs are due. Nodes run when +the inputs required for that evaluation are available. + +The following diagram expands the KLL example for one scheduled window. The +router assigns partition work; each worker reads only its bound partition and +executes every required node of the same installed precompute subgraph. Dashed +arrows denote shared immutable program access; solid arrows denote work or data. + +```mermaid +flowchart TB + D[Selected Planner DAG] --> C[Physical compiler: split at stored outputs] + C --> P[PrecomputePlan: Read -> Group -> KLL -> Write] + C --> QP[QueryPlans: Read stored KLL -> Estimate] + P --> IP[InstalledPrecomputePlan: shared immutable program] + T[Scheduled window and source partitions] --> R[Route by service] + + subgraph W0[Worker 0: service=api] + direction TB + A0[Read partition input] --> A1[Group by service] + A1 --> A2[Build KLL: private state] + A2 --> A3[Write latency-kll for api and window] + end + subgraph W1[Worker 1: service=worker] + direction TB + B0[Read partition input] --> B1[Group by service] + B1 --> B2[Build KLL: private state] + B2 --> B3[Write latency-kll for worker and window] + end + + IP -. same program .-> A0 + IP -. same program .-> B0 + R -->|api partition| A0 + R -->|worker partition| B0 + A3 --> S[SummaryStore: separate committed population/window records] + B3 --> S + S --> QR[Query engine: bound stored-state read] + QP -. query program .-> QR + QR --> E[Estimate p50 or p99 on each query] +``` + +Worker 0 and Worker 1 can run concurrently. Within each worker the arrows +establish dependency order: KLL construction follows its input computation, +and publication follows completion of the required state. Neither worker owns +an exclusive operator stage; both execute Read, Group, KLL and Write. They share +the program but not mutable KLL state. The two records use the same plan version, +stored-output ID and definition, with different population keys. Query execution +starts from the committed output rather than running this precompute path again. + +The selected maintenance mode determines how state is constructed. An +incremental operator updates its window state as input arrives. A batch operator +reads the bound range and builds its result at the scheduled endpoint. The +runtime cannot replace one mode with another merely because both produce the +same family of summary. + +Within one evaluation, the worker follows this workflow: + +### Shared physical operator execution + +The shared library lives in ASAPPlanner alongside post-ASAP IR. Backend pins +that library and IR to the same revision. +Installed precompute DAGs execute through its `PhysicalDag` runtime. The backend supplies +storage frontiers, declared edge order, window completeness and durable commit +keys. It does not own a second dependency walker. + +For completed-window DAGs, SummaryAgg uses the native summary builder, +SummaryMerge uses the native state merge, finalization uses native typed readout, +and Binary lowers aligned rows to native Project using the Planner binary contract, including checked division. Batch conversion +preserves the installed population and timestamp bindings. Native calls receive +the surrounding execution context, so they share its memory budget and +cancellation. The scheduler accepts the caller's run context rather than creating +an independent budget. Physical cancellation and memory errors retain their +error types through maintenance execution. A failed worker stops processing +input and publishing outputs, and closes admission; the failing drain reports +the original failure and shutdown +does not flush more state. Recovery requires restarting the failed execution. +Query execution uses the same library's operations. + +Raw ingestion retains per-window accumulator state through shared-library +updaters; worker routing and window completion remain backend responsibilities. +Storage publication occurs only after successful DAG execution. It is separate +from the library's request-local caching of intermediate results. + +```text +execute(partition, evaluation_window, bound_inputs): + check input identities and required completeness + create one intermediate-result map for this evaluation + for node in installed dependency order: + if this evaluation does not require node: continue + obtain inputs in their declared edge-role order + if a required input is incomplete: keep dependent work pending + otherwise: + execute node using this partition's node state + retain its result for every downstream consumer + publish completed selected outputs with their stored-output bindings +``` + +A shared upstream node is evaluated once for the same partition, window and +input revision, even when several downstream sinks consume it. Its result stays +available until those consumers finish. A later input revision or evaluation +window requires fresh evaluation; the cache is not keyed only by node ID across +unrelated work. + +Readiness belongs to dependencies, not merely to elapsed wall-clock time. For a +node consuming several sources, all required sources must satisfy the bound +window and population-completion rules. A flush timer cannot make missing input +complete. On a finite input run, admission closes and completion barriers follow +all accepted input; workers then finish eligible downstream work and acknowledge +drain only after the required output commits finish. + +A derived summary reads explicitly referenced committed source records and +checks their coverage before executing downstream operators. That read is a +frontier: the worker does not repeat the source records' upstream computation. +Derived work follows the same partition ownership and worker execution model. + +For the initial deployment, derived graphs use one owning worker and execute +serially. Cross-worker shuffle and parallel grouped maintenance are outside this +scope. Independent raw-input partitions may still execute on separate workers. + +### Continuous local input: revision snapshots + +The first continuous-input deployment receives Remote Write directly in Backend. +A window may be revised under the maintenance lifecycle of the physical candidate +selected by Backend from Planner-provided candidates. Each evaluation consumes a +fixed input revision; its dependent outputs must describe +compatible snapshots. Later input can produce a new revision of the same stored +output without installing a new plan version. + +```text +Window W, input revision r1: values [2, 3] -> Sum(W, r1) = 5 +Late input creates revision r2: values [2, 3, 4] -> Sum(W, r2) = 9 +``` + +Publishing r1 does not assert that no more events for W can arrive. A downstream +recomputation must replace or version the previous result according to the +selected contract; adding the complete r2 result to r1 would count old input +twice. Queries cannot combine dependent outputs from incompatible revisions. +An external producer/partition completion protocol is not required for this +initial local-input design. + +The query selects the latest compatible input snapshot that satisfies its +selected freshness requirement. Physical record versions need not have identical +counters: an unchanged output may remain valid for a newer snapshot. An output +whose dependencies changed cannot be carried forward until its replacement is +ready. For two affected outputs: + +| Available records | Query needs A | Query needs A and B | +| --- | --- | --- | +| A:r1, B:r1 | Read r1 | Read r1 | +| A:r1/r2, B:r1; B:r2 pending | Read r2 | Read r1 if fresh enough | +| A:r1/r2, B:r1/r2 | Read r2 | Read r2 | +| Only common r1 exceeds freshness | A may use r2 | No eligible snapshot | + +Independent publication is permitted. Query eligibility is checked for the +whole required read set before execution, and that read set stays pinned for +the query. Concurrent publication or retention cannot switch one branch to a +different snapshot. An unavailable snapshot follows the installed QueryPlan's +availability policy; cancellation and resource errors remain execution failures. + +### Bounded corrections and recovery + +Deployment requirements include a finite correction horizon. Planner enumerates +legal summary maintenance lifecycle choices and compiles them into physical +candidates. A candidate supporting corrections must retain enough input to +recompute affected windows. Backend selects a feasible physical candidate, binds +its inputs and stored outputs, and enforces its retention and correction +requirements. Here, maintenance describes the lifecycle; the executable graph is +the precompute Physical DAG, following Planner’s terminology. The deduplication cache lifetime, +flush timer, and maximum observed event timestamp are not substitutes for this +contract. + +Admission validates the complete Remote Write request against the retained +correction range before enqueuing any of it. An out-of-range write is rejected +explicitly, including an unseen series in an old window. A rejected batch must +not advance the input revision, update deduplication state, or partially modify +stored results. Keeping output bytes alone does not establish that the inputs +needed for a correction remain available. + +A successful input snapshot identifies a durable, fixed set of accepted input. +Source capture must exclude later admissions and retain dependency membership, +including all contributing populations. Derived execution may overlap later +admission, but must use the captured snapshot rather than rereading mutable live +state. Every affected sink is evaluated against that same snapshot; shared +ancestors execute once for the selected sinks. + +Recovery restores the installed generation, input revision, retained correction +boundary, and committed result versions before accepting input or serving reads. +A partially published newer revision does not invalidate an older compatible +snapshot that remains fresh enough. Retrying the same input snapshot must not +add a full recomputed result to its prior version or publish it twice. Starting +a new plan version does not implicitly inherit this revision history. + +| Acceptance case | Required behavior | +| --- | --- | +| Late value 4 joins values 2 and 3 | r2 reads 9, never 14; r1 remains 5 while eligible | +| A:r2 commits and B:r2 fails | A+B reads compatible r1 if fresh; never A:r2 with affected B:r1 | +| A changes but B's dependency set does not | Unchanged B may serve the newer compatible snapshot | +| Old common revision exceeds freshness | The query cannot serve it as an eligible summary hit | +| One sample in a batch exceeds the correction range | Reject the complete batch before any admission | +| Crash between sibling publications | Recover committed versions and resume without duplicate publication | +| Restart with the same plan version | Preserve the correction boundary and revision eligibility | +| Install a new plan version | Require its own input/history and warm-up | + +The local Remote Write implementation captures and checkpoints accepted input +before execution. Installation first verifies that each selected raw producer has +a supported native recovery codec; unsupported storage formats fail before input +admission. It publishes each stored output independently, using Planner's native +typed state codec. A query pins one compatible snapshot before preparing +its inputs; the same snapshot supplies every branch and every range-query step. +The store exposes committed records through an immutable read view, never through +the additive producer-write path. + +This initial realization serially replays bounded retained input through the +selected operators. It deliberately trades update cost for simple correction +semantics; it is not an incremental-update optimization. Input retention covers +the larger of the correction horizon, installed query lookback and configured +output retention, plus overlapping windows and query freshness. The correction +horizon bounds sample age against Backend's admission clock. Whole requests with +an older or future-dated sample are rejected before durable admission. + +A periodic capture closes newly elapsed windows even without a later sample. It +only describes locally accepted input as of that capture; it does not claim that +all upstream events have arrived. A late event inside the horizon creates another +revision. A failed capture stops execution; pending durable input can be retried +or recovered without replacing already committed randomized sketch bytes. + +Enable this path with `--remote-write-revision-dir`. Deployment configuration sets +`--remote-write-correction-horizon-ms`, `--remote-write-revision-freshness-ms` and +`--remote-write-revision-max-bytes`. The last separately bounds serialized +checkpoint size and operator workspace; exceeding it returns an explicit resource error. The checkpoint is +one atomically replaced, fsynced file per plan version, protected by a single-writer +lock. Both raw input history and eligible output history are bounded. The existing +finite-input path remains available when continuous revisions are not configured; +its drain barrier cannot seal a continuous input. + +Process E2E tests exercise one- and two-source Planner-generated DAGs through HTTP +Remote Write, derived summary construction, bound queries, late corrections, +retries, whole-batch rejection, same-version restart and new-version warm-up. +Storage regression tests cover partial publication, fresh common-snapshot +selection, pinned readers, recovery and explicit memory/cancellation failures. +Cross-worker shuffle and cross-producer completion protocols remain outside this +local-input realization. + +## 6. Publish the selected DAG outputs + +One `SummaryStore` owns two logical tables: + +| Table | Record contents | +| --- | --- | +| `summary_definitions` | `SummaryDefinition`: canonical input, Planner family, parameters, grouping and time semantics. Shared across compatible records. | +| `stored_summaries` | `StoredSummary`: record key, definition ID, actual format and coverage, and payload. | + +The compiler assigns a `stored_output_id` to each output selected for +persistence. The precompute writer and query readers carry the same +`StoredOutputReference`. The reference is a plan binding, not a third catalog +object. + +A concrete stored record has the key: + +```text +(plan_id, plan_version, stored_output_id, population_key, window) +``` + +`population_key` preserves canonical label names and values. `window` identifies +the intended time partition and its boundary convention; actual coverage is +validated separately. Worker IDs and transient runtime handles are not record +identities. A store may use a local numeric row handle for indexing or wire +compression; that handle cannot select another output, definition or plan +version. The writer supplies a `StoredSummaryKey` and the reader supplies the +same selected `StoredOutputReference` within its installed plan version. +Durable metadata retains this binding so recovery validates it before exposing +payloads. V1 does not implicitly reuse records from another plan version. +The durable binding format is version 5; earlier SID metadata is rejected rather +than inferred to refer to a selected output. + +Before publication, validate that the output matches its bound definition, +format, population and window. Payload and identifying metadata become visible +as one committed record. A reader must not observe metadata pointing to an +uncommitted payload. Repeating the publication of the same completed result must +not add that result twice; conflicting writes must not silently overwrite a +record under the same key. Any permitted replacement follows the selected +update policy and preserves coherent reads. + +Sibling outputs may commit independently. A query needing several outputs must +wait for the required set at compatible revisions; it cannot read a partially +published evaluation during a quiet interval between commits. Publication +receipts and the query-wide revision fence must cover all dependent branches. + +QueryPlan and derived precompute readers perform indexed lookup using the +installed reference and requested population/window. They check definition, +format, plan version and coverage before consuming a record. Missing or +incomplete state remains unavailable; query fallback is governed by QueryPlan. + +## 7. Worked example: one KLL producer, two percentile readers + +Use the latency example with this selected deployment: + +- Plan version `42`; group input by `service`. +- Build KLL with `k=200` over `(T - 5m, T]`. +- Rebuild from data at rest every minute, aligned to the Unix epoch. +- Retain completed summaries for ten minutes. +- Store output `latency-kll`, defined by `def-api-latency-kll`. + +The following YAML illustrates the bindings and stored record, not a Rust wire +schema: + +```yaml +plan_version: 42 +writer_reference: + stored_output_id: latency-kll + definition_id: def-api-latency-kll +reader_references: + p50: {stored_output_id: latency-kll, definition_id: def-api-latency-kll} + p99: {stored_output_id: latency-kll, definition_id: def-api-latency-kll} + +summary_definitions: + def-api-latency-kll: + input: {metric: request_latency_seconds, value: sample_value} + family: {kind: Sketch, algorithm: KLL, parameters: {k: 200}} + group_by: [service] + time_semantics: {range: 5m, bounds: "(start, end]"} + output_type: kll_state + +stored_summaries: + - key: + plan_version: 42 + stored_output_id: latency-kll + population_key: {service: api} + window: {start_exclusive: '12:00', end_inclusive: '12:05'} + definition_id: def-api-latency-kll + format: {schema: kll-v1, encoding: kll-binary-v1} + coverage: {start_exclusive: '12:00', end_inclusive: '12:05'} + payload: +``` + +At the `12:05` evaluation endpoint: + +1. The schedule triggers construction for `(12:00, 12:05]`. The bound source + supplies the required data and completion evidence for that range. +2. Routing sends `service=api` data to its owning worker. That worker executes + the read, grouping and KLL construction nodes for the partition. Another + worker can execute the same graph for `service=worker` concurrently. +3. The first worker commits the record illustrated above. The other worker + commits a separate record with `population_key: {service: worker}`. +4. Both percentile QueryPlans select the `service=api` record for endpoint + `12:05`. They read the same KLL state and apply their different estimates. + +At `12:06`, the same graph builds new records for `(12:01, 12:06]`. The definition, +stored-output ID and plan version stay the same; the window part of the key +changes. Ten-minute retention controls how long completed records remain +available, not how much input each KLL summarizes. Building a percentile result +at query time does not create another stored output. + +## 8. Plan changes, recovery and validation + +Routing, operator bindings and output references belong to one coherent plan +version. Accepted work retains that version until completion. Activation drains +or isolates incompatible old worker state before new work uses a different +partition assignment. Changing worker count requires coordinated reassignment; +changing a hash modulus while state is active is insufficient. + +A newly activated version is cold until its own stored outputs are available. +Queries must not substitute the previous version's payload to hide that gap. +Continuous warm service across activation requires an explicit readiness or +state-migration protocol; atomic plan publication alone does not provide it. + +Recovery validates persisted record identities, definitions, formats and +coverage before exposing them to readers. A matching definition alone does not +authorize reuse across plan versions. Runtime caches and routing indexes are +rebuilt from the installed plan. Input replay and durable operator-state +checkpointing require an explicit source/update contract; record publication +alone does not guarantee exactly-once ingestion. + +Acceptance checks follow the workflow: + +| Check | Required observation | +| --- | --- | +| Planner-to-runtime binding | Node semantics, edge roles, families and stored-output references agree; invalid plans fail installation. | +| One worker versus several | Identical results for grouped Sum, per-series Rate and Rate followed by service reduction, including counter resets and window boundaries. | +| Partition locality | Every dependency needed for a partition stays with its owner; a rule splitting a required reduction is rejected. | +| Shared computation | A common upstream node runs once per evaluation/input revision across its consumers, including multiple stored sinks. | +| Readiness and backpressure | Incomplete inputs do not produce readable complete records; admission and drain honor queued work and commits. | +| Store and query integration | Writer and reader use the same composite identity; format, coverage and version mismatches fail validation. | +| Restart and activation | Committed records recover coherently; retries do not duplicate completed publication; old and new plan state do not mix. | + +The simple execution model does not require node-level parallel scheduling, +cross-worker shuffle, or separate metadata and payload services. Its unit of +parallel work is a complete precompute subgraph over an independent data +partition. diff --git a/docs/design_docs/query-dag-execution.md b/docs/design_docs/query-dag-execution.md new file mode 100644 index 00000000..5c8b7bf9 --- /dev/null +++ b/docs/design_docs/query-dag-execution.md @@ -0,0 +1,320 @@ +# Query DAG Execution + +## 1. Overview + +ASAP uses one physical-operator library and one DAG runtime for both +precomputation and query execution. + +```mermaid +flowchart LR + L["Logical Post-ASAP DAG"] -->|Candidate generation| M["Summary Maintenance Lifecycle"] + M -->|Physical Plan Compiler| P["Physical DAG candidates"] + P -->|Backend selection and Deployment Plan Compiler| D["Deployment Plan / DAG"] + D --> Pre["PrecomputePlan"] + D --> Query["QueryPlan"] + Pre --> Runtime["Shared physical operators and DAG runtime"] + Query --> Runtime +``` + +This follows [Planner #462's architecture](https://github.com/ProjectASAP/ASAPPlanner/blob/feat/shared-physical-operators/docs/design_docs/physical-planning-and-deployment.md): + +| Layer | Responsibility | +| --- | --- | +| Logical Post-ASAP DAG | Computation semantics | +| Summary Maintenance Lifecycle | Build, retention, reuse, and window strategy | +| Physical DAG(s) | Supported physical candidates, executable operators, and typed boundaries | +| Deployment Plan / DAG | Selected candidate, concrete bindings, and operational lifecycle | + +ASAPPlanner owns the first three layers. Backend selects a feasible physical +candidate using deployment statistics, resource limits and costs, then binds it +into `PrecomputePlan` and `QueryPlan`. It preserves the candidate's operators, +dependencies, sharing and materialization frontier. + +The lifecycle is a planning contract, not another computation IR. “Maintenance” +here names Planner's lifecycle concept; input reception is ingestion and execution +before a request is precomputation. Executing a bound Physical DAG introduces +neither an **Execution DAG** abstraction nor another planning layer. + +## 2. Ownership + +### ASAPPlanner + +ASAPPlanner owns the post-ASAP IR, logical and physical operator implementations, +`asap-physical-operators`, `physical_planner`, `CompiledPhysicalDag`, and the +independent DAG runtime. A change to an IR operation and its execution +implementation can therefore be reviewed in one Planner PR. + +The runtime owns generic execution behavior: + +- dependency scheduling; +- shared-producer execution; +- bounded buffering; +- cancellation; +- per-run reuse of intermediate outputs; +- operator workspace accounting. + +A producer with multiple consumers executes once per DAG run, while each +consumer independently receives its output. + +### Backend + +Backend owns deployment-specific concerns: + +- source binding and window selection; +- catalog and storage access; +- revision and coverage admission; +- summary publication; +- protocol conversion; +- query and precomputation bindings. + +It does not duplicate Planner operators or maintain a second DAG runtime. +JSON, Arrow, SQL, and PromQL are boundary representations; they do not determine +the internal representation of every ASAP summary. + +See the [shared library design](https://github.com/ProjectASAP/ASAPPlanner/blob/feat/shared-physical-operators/docs/design_docs/physical-planning-and-deployment.md) +for the broader architecture and DataFusion comparison. + +## 3. Compilation and execution + +```text +Planner: Logical Post-ASAP DAG + lifecycle candidate + ↓ Physical Plan Compiler + supported Physical DAG candidates + ↓ Backend selection and Deployment Plan Compiler + Deployment Plan / DAG (PrecomputePlan + QueryPlan) + ↓ execute bound Physical DAGs through shared runtime + stored outputs / query results +``` + +### 3.1 Compile + +ASAPPlanner compiles semantically legal computation and lifecycle candidates +into supported Physical DAG candidates with typed input/output boundaries. +Compilation does not require live storage readers. `CompiledPhysicalDag` is a +Rust implementation type for physical compilation, not a separate architectural +layer or an intermediate Execution DAG. + +### 3.2 Bind + +Backend checks candidate feasibility and costs, selects a physical candidate, +and binds its inputs in the Deployment Plan / DAG. At request time, those binding +rules resolve concrete eligible records and readers. Conceptually: + +```text +Summary-state input → eligible SummaryStore records +Vector input → labeled values and a PromQL window +Relation input → typed SQL/storage relation +``` + +Admission and binding validate types, state formats, grouping, windows, +revisions, and coverage. They do not repeat operator lowering. + +### 3.3 Execute + +The shared runtime executes the bound Physical DAG. Execution does not lower +operators again or make new placement decisions. + +```mermaid +flowchart LR + A["Input"] --> B["Shared producer"] + B --> C["Consumer A"] --> E["Root 1"] + B --> D["Consumer B"] --> F["Root 2"] +``` + +The shared producer executes once. Multiple roots share upstream computation, +cancellation, and the enclosing memory budget. + +## 4. Precomputation and query execution + +Ingestion means accepting input data. Precomputation means executing a selected +DAG before a query request, including work triggered by ingestion or scheduled +window processing. + +Operator type does not determine execution phase. The selected plan determines +whether an operation executes during precomputation or at query time. + +For example, partial precomputation can use: + +```text +Precomputation: raw data → BuildKLL → SummaryStore +Query: SummaryStore → ReadKLL → Quantile(0.99) → result +``` + +The same operator library executes both DAGs. + +| Mode | Pre-request work | Request-time work | +| --- | --- | --- | +| No precomputation | None | All required computation | +| Partial precomputation | Store intermediate values or states | Complete the remaining DAG | +| Full precomputation | Complete all data-dependent computation | Retrieve the prepared result | + +These definitions are independent of the algorithm. Reading stored KLL state +and computing its quantile at query time is partial precomputation. +Build, merge, and readout are reusable operators rather than phase-specific operators. + +## 5. Query execution + +An installed query brings together these contracts: + +```text +Installed query + ├── QueryPlanEntry: nodes, dependencies and root + ├── selected Query Physical DAG + ├── deployment bindings and source references + ├── window contract + └── catalog generation +``` + +This is an architectural view, not the serialized field layout of +`QueryPlanEntry`; some contracts live in its enclosing plan or bound runtime graph. + +At request time: + +```mermaid +flowchart LR + Request["Query request"] --> Entry["Installed query"] + Entry --> Validate["Admit sources and pin eligible snapshot"] + Validate --> Bind["Bind inputs within request budget"] + Bind --> Execute["Execute DAG"] + Execute --> Convert["Protocol conversion"] --> Result["Query result"] +``` + +SQL relation nodes and PromQL vector operations use the same Planner physical +DAG representation. SQL adapters bind relations and convert results at the +protocol boundary. PromQL adapters bind labeled vectors and windows. Common +arithmetic, aggregation, sorting, limiting, joins, and temporal computation +belong in the shared physical-operator library. + +## 6. Request consistency + +One installed query request owns one cancellation signal and one resource +budget. A request over continuous local inputs also pins one eligible input +snapshot. Range steps have separate DAG execution state but share these controls. + +```text +range request + ├─ pin eligible snapshot + ├─ prepare inputs ┐ + ├─ execute t1 → retain result ├─ shared budget + ├─ execute t2 → retain result │ and cancellation + └─ return accumulated result ┘ +``` + +Tracked memory includes pinned SDS/current-series state, prepared inputs, +operator workspace, and retained results. JSON decoding and SQL encoding use +conservative workspace estimates; accounting is not a hard process RSS limit. + +Resource exhaustion or cancellation terminates the request without exact +fallback, even if a revision changes concurrently. + +### Snapshot compatibility + +All local inputs used by one query must admit a common eligible revision in the +selected catalog generation. A newer partially published revision does not +invalidate an older complete revision that still satisfies freshness and coverage. + +External exact adapters expose evaluation time but not a source snapshot token. +The current admission rule therefore rejects a candidate combining local +SDS/current-series inputs with external exact inputs, including candidate pruning. +The rule is checked at installation and before source access. + +```text +local SDS @ r1 external exact @ unknown revision + \ / + \------ operation ---/ + │ + ▼ + rejected +``` + +Equal timestamps are insufficient to establish equal revisions. Whole-query +external exact execution remains a separate deployment choice; a resource or +cancellation failure in a local DAG cannot trigger that choice. + +## 7. Operator and plan acceptance + +A local DAG is accepted only if every reachable operation has a compatible +implementation: + +```text +operation + ├── implementation exists + ├── input/output types match + ├── parameters are supported + ├── grouping semantics are supported + └── required state format is available +``` + +Deployment additionally validates source bindings, stored-state formats, +grouping, windows, revisions, coverage, and approximation evidence where required. +An operator implementation establishes execution support, not deployment feasibility. + +Unsupported operations are rejected explicitly. Routing them to an external +source does not make them locally supported. The [Planner library contract](https://github.com/ProjectASAP/ASAPQuery-backend/blob/54067d4b44fe812fdd746209b522e77003e8d480/docs/design_docs/physical-operators.md) +owns operator support; Backend does not maintain a second operator matrix. + +## 8. Candidate pruning + +Candidate pruning is ordinary DAG composition: + +```mermaid +flowchart LR + Summary["Candidate summary"] --> Keys["Candidate keys"] + Keys --> Join["SemiJoin(keys)"] + Values["Authoritative values"] --> Join + Join --> Sort["Sort(score)"] --> Limit["Limit per group"] +``` + +This path needs no dedicated `MembershipFilter` or grouped TopK operator. +Its correctness rules are: + +- grouped Limit is distinct from global Limit; +- candidate completeness comes from the pruning certificate; +- exact scoring cannot recover a candidate omitted by pruning; +- missing authoritative values invalidate certified pruning; +- local candidate and scoring inputs must use the same pinned snapshot. + +Best-effort pruning remains explicitly approximate. + +## 9. Runtime and cancellation + +Planner's native context is not `Send`, so each request executes through one +dedicated worker invocation. The worker owns thread-local execution context for +that request. Async callers signal cancellation, which remote waits and native +polling observe. + +When execution finishes, the worker clears the context before the thread is +reused. Cancellation is cooperative at polling boundaries; it does not preempt +an individual synchronous kernel. + +## 10. Testing and acceptance + +| Layer | Required coverage | +| --- | --- | +| Planner library | Shared-producer diamonds, multiple consumers, bounded buffering/backpressure, cancellation, memory accounting, both execution phases, typed expressions, joins, grouped Sort/Limit, and window computation | +| Backend | Installed source bindings, revision and coverage admission, stored-state reconstruction, end-to-end query results, and rejection of unsupported plans | +| Request contracts | Shared budgets across range steps and nested execution, terminal resource/cancellation errors, snapshot pinning, and rejection of mixed local/external inputs before reads | + +Results are compared against exact computation or the declared approximation +guarantee. These acceptance requirements do not imply that every deployment or +operator combination has been tested. + +## 11. Scope + +This design establishes the common execution path: + +```text +Logical Post-ASAP DAG + Summary Maintenance Lifecycle + → Physical DAG candidates + → Deployment Plan / DAG + → execute bound Physical DAGs + → query/precomputation output +``` + +Local raw Scan and universal support for every Planner aggregate or extension +remain outside this migration. Fallback routing does not establish local support; +tests supplied with already-decoded batches do not prove raw-source integration. + +ASAPPlanner #462 owns `physical_planner`, native physical operators, and DAG +execution. Backend's `DeploymentPlanCompiler` owns state identities, window +contracts, source bindings, and deployment of the resulting DAG. From af62b384b42e065e516b965a0092b9a668f555c5 Mon Sep 17 00:00:00 2001 From: zzylol Date: Tue, 20 Oct 2026 20:00:34 +0000 Subject: [PATCH 2/8] docs: SDS keys summaries by the Planner's summary coverage (definition + selection) --- .../design/backend-on-planner-physical-dag.md | 34 ++++++++++++++----- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/docs/design/backend-on-planner-physical-dag.md b/docs/design/backend-on-planner-physical-dag.md index d7aff6e8..2093e8b8 100644 --- a/docs/design/backend-on-planner-physical-dag.md +++ b/docs/design/backend-on-planner-physical-dag.md @@ -157,14 +157,32 @@ The Planner specifies **which summaries to compute and retain**. The operator li ### Stored summaries (SDS) -SDS holds every summary the plan stores, and nothing else. - -- **Definitions.** Each stored output in `stored_outputs` maps to one summary definition. A definition's identity is a hash of the sub-DAG that builds it: payloads and schemas, with offset-only `TimeShift`s removed. Outputs that differ only in time offset therefore share one definition. For example, six 10-minute KLL panes read at offsets 0 to 50 minutes are one definition. -- **Keys.** Summaries are keyed by (definition, end of their time coverage). A time-bounded summary covers the left-open interval `(end − width, end]`; a pane is the case built directly from raw samples. -- **Kinds.** Any output the plan stores: time panes, rollups over time ranges, rollups across series, sliding-window summaries, and query-time results kept across evaluations. Each comes from `stored_outputs` with its inputs (raw samples or other stored outputs), its time coverage, its build phase and its retention. -- **Immutability.** A summary never changes after insertion. It is shared by reference (`Arc`) and never copied. -- **Retention.** Each definition is kept for the largest `retention_ms` among its outputs, which already includes range-evaluation spans. Older summaries are evicted per definition. -- **Not read:** summary coverage. Reuse decisions were made by the Planner; SDS stores what the plan names. +SDS holds every summary the plan stores, and nothing else. It is the runtime half of the summary model in ASAPPlanner's [Schema and Physical Data for ASAP Primitives](https://github.com/ProjectASAP/ASAPPlanner/blob/main/docs/design_docs/proposals/asap-primitive-schema.md). That document defines what a summary state means. Its §4.3 leaves to the runtime how a summary's definition and selection are stored. This section specifies that. + +**Summary coverage.** In the Planner's IR, every summary node has a **coverage** with two parts (§4.2 of that document): + +| Part | Question | Example: `quantile_over_time(0.99, m{job="api"}[1m] offset 2m)` | +| --- | --- | --- | +| `definition` | *What* is computed: the sub-DAG below the `SummaryAgg` with the selected conditions taken out, the summary family and its parameters, the input column and the grouping | KLL(value) per series over `Scan m` | +| `selection` | *Which rows* went in: value sets, value ranges, and time relative to the evaluation time | time `(−3m, −2m]`, `job ∈ {api}`, once label matchers below the window are lifted | + +The Planner derives coverage from the DAG; it is a property of the IR, not a deployment concept. SDS uses it as follows. + +- **Definition identity.** A summary definition is identified by the hash of its coverage `definition`, ignoring planning details (`timing`, `guarantee`). Two stored outputs with equal definitions are one SDS definition, whatever their time offsets or selected label values. Six 10-minute KLL panes read at offsets 0 to 50 minutes, for example, are one definition. +- **Summary instances.** Each stored summary is one instance of a definition, keyed by its **concrete selection**: the selection with relative time bound to an absolute interval when the summary is built, `(end − width, end]` for a left-open window. + - A pane is an instance whose selection is one time interval. + - A rollup over a time range is an instance whose selection is the union of its inputs' intervals. + - A rollup across series keeps its definition's coarser grouping. + - Instances are immutable and shared by reference (`Arc`). +- **Lookups.** A query binds each stored input at evaluation time `T`. It turns the input's relative selection into a concrete one and fetches the instance with that key. +- **Reuse rules, from the coverage model.** These are the Planner's rules; SDS stores instances so that a plan which uses them can find them. + - Instances of one definition can be merged when their selections do not overlap (`SummaryMerge`). That is how panes form a window and how per-value summaries form a union. + - An instance can be sliced only on a grouping column, and rolled up to a coarser grouping only when its family allows it (§5.6). + - SDS never merges, slices or rolls up on its own. It stores and returns the instances the plan names. +- **Kinds.** Any output the plan stores: time panes, rollups over time ranges, rollups across series, sliding-window summaries, and query-time results kept across evaluations. Each arrives in `stored_outputs` with its coverage, its inputs (raw samples or other stored outputs), its build phase and its retention. +- **Retention.** Each definition is kept for the largest `retention_ms` among its outputs. That value already includes range-evaluation spans. Instances whose concrete time selection ends before that horizon are evicted. + +**Contract with the Planner.** `InstallablePlan.stored_outputs` should carry each output's coverage: definition identity and relative selection. Then SDS identity, keys and lookups follow the Planner's semantics, and the runtime does not re-derive them from the DAG. Today the plan describes only raw-sample panes, by their time window. Describing every stored summary with its coverage is tracked in ASAPPlanner #781 and backend #837. ### Precompute engine From b53de895caf9986152fd0e0a6c5b09099d783657 Mon Sep 17 00:00:00 2001 From: zzylol Date: Tue, 20 Oct 2026 20:02:06 +0000 Subject: [PATCH 3/8] docs: the event-time barrier closes panes of idle shards --- docs/design/backend-on-planner-physical-dag.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/design/backend-on-planner-physical-dag.md b/docs/design/backend-on-planner-physical-dag.md index 2093e8b8..8143dc41 100644 --- a/docs/design/backend-on-planner-physical-dag.md +++ b/docs/design/backend-on-planner-physical-dag.md @@ -308,7 +308,9 @@ The merged runtime keeps ASAPQuery's low-overhead data path and the backend's si ## Open decisions - **Pane retention.** The runtime keeps ingestion-built panes for one extra pane width beyond the plan's retention, to cover evaluations just behind the newest closed pane. Whether the Planner should model the deployment's evaluation delay explicitly remains a design question. -- **Idle shards.** A shard that stops receiving samples no longer advances its watermark (the timestamp used to decide which panes can close). Its panes remain open unless the runtime closes them using elapsed wall-clock time; that mechanism is not implemented yet. +- **Idle shards.** A shard that stops receiving samples no longer advances its watermark (the timestamp used to decide which panes can close). Its panes then stay open. + - A producer can close them with the event-time barrier `POST /api/v1/precompute/watermark?time_ms=T`. The barrier is queued behind acknowledged writes, closes every pane ending at or before `T`, is capped at the wall clock, and leaves input open. With several producers, it must be sent only after all of them have written through `T`. + - Whether the runtime should also close idle panes by elapsed wall-clock time, without a producer barrier, remains open. - **The local raw buffer.** Expose it as an explicit capability with a cost the Planner can compare, or keep it as a runtime choice whose effect is reflected in measured costs. - **Planner dependency weight.** `plan_installable` lives in `asap-planner`, which also builds the SQL and MetricsQL frontends. A PromQL-only runtime would benefit if those frontends were optional features. From 6a769b93d84de434c9f867072d005b41ebe8a3eb Mon Sep 17 00:00:00 2001 From: zzylol Date: Tue, 20 Oct 2026 23:35:47 +0000 Subject: [PATCH 4/8] docs: replace the removed Summary Maintenance Lifecycle with Stage 2 materialization and Stage 3 retention --- docs/design_docs/query-dag-execution.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/docs/design_docs/query-dag-execution.md b/docs/design_docs/query-dag-execution.md index 5c8b7bf9..fdd44de0 100644 --- a/docs/design_docs/query-dag-execution.md +++ b/docs/design_docs/query-dag-execution.md @@ -7,7 +7,7 @@ precomputation and query execution. ```mermaid flowchart LR - L["Logical Post-ASAP DAG"] -->|Candidate generation| M["Summary Maintenance Lifecycle"] + L["Logical Post-ASAP DAG"] -->|Candidate generation| M["Materialization (Stage 2) and retention (Stage 3)"] M -->|Physical Plan Compiler| P["Physical DAG candidates"] P -->|Backend selection and Deployment Plan Compiler| D["Deployment Plan / DAG"] D --> Pre["PrecomputePlan"] @@ -21,7 +21,7 @@ This follows [Planner #462's architecture](https://github.com/ProjectASAP/ASAPPl | Layer | Responsibility | | --- | --- | | Logical Post-ASAP DAG | Computation semantics | -| Summary Maintenance Lifecycle | Build, retention, reuse, and window strategy | +| Materialization (Stage 2) and retention (Stage 3) | When each stored summary is built (ingestion or query time), how long it is kept, reuse, and window strategy; delivered to the runtime as `InstallablePlan.stored_outputs` | | Physical DAG(s) | Supported physical candidates, executable operators, and typed boundaries | | Deployment Plan / DAG | Selected candidate, concrete bindings, and operational lifecycle | @@ -30,9 +30,8 @@ candidate using deployment statistics, resource limits and costs, then binds it into `PrecomputePlan` and `QueryPlan`. It preserves the candidate's operators, dependencies, sharing and materialization frontier. -The lifecycle is a planning contract, not another computation IR. “Maintenance” -here names Planner's lifecycle concept; input reception is ingestion and execution -before a request is precomputation. Executing a bound Physical DAG introduces +Materialization and retention are planning decisions, not another computation IR. +Input reception is ingestion, and execution before a request is precomputation. Executing a bound Physical DAG introduces neither an **Execution DAG** abstraction nor another planning layer. ## 2. Ownership @@ -77,7 +76,7 @@ for the broader architecture and DataFusion comparison. ## 3. Compilation and execution ```text -Planner: Logical Post-ASAP DAG + lifecycle candidate +Planner: Logical Post-ASAP DAG + materialization and retention ↓ Physical Plan Compiler supported Physical DAG candidates ↓ Backend selection and Deployment Plan Compiler @@ -88,7 +87,7 @@ Planner: Logical Post-ASAP DAG + lifecycle candidate ### 3.1 Compile -ASAPPlanner compiles semantically legal computation and lifecycle candidates +ASAPPlanner compiles semantically legal computation and materialization candidates into supported Physical DAG candidates with typed input/output boundaries. Compilation does not require live storage readers. `CompiledPhysicalDag` is a Rust implementation type for physical compilation, not a separate architectural @@ -304,7 +303,7 @@ operator combination has been tested. This design establishes the common execution path: ```text -Logical Post-ASAP DAG + Summary Maintenance Lifecycle +Logical Post-ASAP DAG + materialization and retention → Physical DAG candidates → Deployment Plan / DAG → execute bound Physical DAGs From c8b57d7e4a7f90bcb4aeff324968722e243bf4fb Mon Sep 17 00:00:00 2001 From: zzylol Date: Tue, 20 Oct 2026 23:49:39 +0000 Subject: [PATCH 5/8] docs: terminology shared with ASAPPlanner --- .../design/backend-on-planner-physical-dag.md | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/design/backend-on-planner-physical-dag.md b/docs/design/backend-on-planner-physical-dag.md index 8143dc41..746feb1a 100644 --- a/docs/design/backend-on-planner-physical-dag.md +++ b/docs/design/backend-on-planner-physical-dag.md @@ -8,6 +8,7 @@ This document designs the backend: what it must achieve, how its components work - [Problem](#problem) - [Goals and non-goals](#goals-and-non-goals) +- [Terminology](#terminology) - [Architecture](#architecture) - [Boundary between ASAPPlanner and ASAPQuery-backend](#boundary-between-asapplanner-and-asapquery-backend) - [Component design](#component-design) @@ -48,6 +49,30 @@ A metrics workload repeats many queries over the same data: dashboards, alerts, - Durable state and recovery after restart. State is in memory and is rebuilt from new samples. - SQL workloads. The control plane plans PromQL today, and the design keeps the language a Planner concern. +## Terminology + +This document, the backend's code and its PR descriptions use ASAPPlanner's terms. Planner terms come from its IR and from [Schema and Physical Data for ASAP Primitives](https://github.com/ProjectASAP/ASAPPlanner/blob/main/docs/design_docs/proposals/asap-primitive-schema.md) §2 and §4. Backend-only terms are listed with what they mean. + +| Term | Source | Meaning | +| --- | --- | --- | +| relation, row, column, schema | Planner (§2) | As in relational databases. For PromQL, a row is one sample of one series at one time. | +| summary state | Planner (§2) | A column whose type is a summary type: a sketch or an exact accumulator. | +| summary family | Planner (`SketchAlgorithm`) | The kind of summary, e.g. KLL, DDSketch, Count-Min, HLL. | +| `SummaryCoverage` (`definition`, `selection`) | Planner (§4.2) | What a summary state computes, and which rows went into it. | +| materialization | Planner (`Materialization`) | When a summary is built: `MatAtIngestion`, `MatAtQuery`, `MatAtQueryKeptAcrossEvaluations`, or `NotMaterialized`. | +| `InstallablePlan`, `StoredOutput`, `StoredOutputBuild` (`IngestionRun`, `QueryRun`), `RawSampleWindow`, `retention_ms` | Planner (`asap_planner`) | The plan a runtime installs, its stored outputs, which run builds each one, the raw-sample windows read at query time, and how long each output is kept. | +| evaluation time, `RangeEvaluation` (step, span) | Planner (workload) | The time `T` a query is answered for, and the steps of a range query. | +| pane, pane chain | Planner | A summary over one left-open time interval `(end − width, end]`; the panes that make up a window. | +| raw sample | Backend | One value of one series at one timestamp, as received over Remote Write. Not a sampling-based approximation. | +| summary instance | Backend | One stored summary state with its concrete `selection` (relative time bound to an absolute interval). | +| SDS | Backend | The backend's store of summary instances, keyed by (`definition` identity, concrete `selection`). | +| watermark | Backend | The newest routed raw sample's time, capped at the wall clock, minus the allowed lateness. Panes that end at or before it are closed. | +| forward | Backend | Answer a query with Prometheus instead of the plan, with the reason in `x-asap-forward-reason`. | + +Rules: +- Use the Planner's name for anything the Planner defines. Don't introduce a backend synonym: "kept query-time result", for example, is a stored output with `StoredOutputBuild::QueryRun`, built because of `MatAtQueryKeptAcrossEvaluations`. +- A backend-only term gets one entry in this table and is used with that meaning everywhere. + ## Architecture ```mermaid From 281cb3054a63c70c05e8ee915117854f64db8d47 Mon Sep 17 00:00:00 2001 From: zzylol Date: Tue, 20 Oct 2026 23:54:11 +0000 Subject: [PATCH 6/8] docs: add event-time barrier and bound raw-sample window to the terminology --- docs/design/backend-on-planner-physical-dag.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/design/backend-on-planner-physical-dag.md b/docs/design/backend-on-planner-physical-dag.md index 746feb1a..5924721f 100644 --- a/docs/design/backend-on-planner-physical-dag.md +++ b/docs/design/backend-on-planner-physical-dag.md @@ -68,6 +68,8 @@ This document, the backend's code and its PR descriptions use ASAPPlanner's term | SDS | Backend | The backend's store of summary instances, keyed by (`definition` identity, concrete `selection`). | | watermark | Backend | The newest routed raw sample's time, capped at the wall clock, minus the allowed lateness. Panes that end at or before it are closed. | | forward | Backend | Answer a query with Prometheus instead of the plan, with the reason in `x-asap-forward-reason`. | +| event-time barrier | Backend | A producer's statement that every raw sample up to time `T` has been written (`POST /api/v1/precompute/watermark?time_ms=T`). It closes every pane that ends at or before `T`. | +| bound raw-sample window | Backend | A `RawSampleWindow` bound to one evaluation time: the absolute interval `(T − offset − width, T − offset]`. | Rules: - Use the Planner's name for anything the Planner defines. Don't introduce a backend synonym: "kept query-time result", for example, is a stored output with `StoredOutputBuild::QueryRun`, built because of `MatAtQueryKeptAcrossEvaluations`. From ee8a94955e56edd11e187352be1ee3dd9496656a Mon Sep 17 00:00:00 2001 From: zzylol Date: Wed, 21 Oct 2026 00:01:17 +0000 Subject: [PATCH 7/8] docs: ingestion workers, shards and the global watermark in the terminology --- docs/design/backend-on-planner-physical-dag.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/design/backend-on-planner-physical-dag.md b/docs/design/backend-on-planner-physical-dag.md index 5924721f..930a9cd6 100644 --- a/docs/design/backend-on-planner-physical-dag.md +++ b/docs/design/backend-on-planner-physical-dag.md @@ -66,7 +66,8 @@ This document, the backend's code and its PR descriptions use ASAPPlanner's term | raw sample | Backend | One value of one series at one timestamp, as received over Remote Write. Not a sampling-based approximation. | | summary instance | Backend | One stored summary state with its concrete `selection` (relative time bound to an absolute interval). | | SDS | Backend | The backend's store of summary instances, keyed by (`definition` identity, concrete `selection`). | -| watermark | Backend | The newest routed raw sample's time, capped at the wall clock, minus the allowed lateness. Panes that end at or before it are closed. | +| ingestion worker, shard | Backend | Ingestion is split across N ingestion workers by series id; a worker's shard is the set of series routed to it. Each worker owns its open-pane states. | +| watermark | Backend | Per worker: the newest raw sample it received, capped at the wall clock, minus the allowed lateness. The global watermark is the minimum over the workers that hold it: every worker that has been sent samples and is not idle. Panes ending at or before the global watermark are closed. | | forward | Backend | Answer a query with Prometheus instead of the plan, with the reason in `x-asap-forward-reason`. | | event-time barrier | Backend | A producer's statement that every raw sample up to time `T` has been written (`POST /api/v1/precompute/watermark?time_ms=T`). It closes every pane that ends at or before `T`. | | bound raw-sample window | Backend | A `RawSampleWindow` bound to one evaluation time: the absolute interval `(T − offset − width, T − offset]`. | From 0ebbcd254f462a16f747fef155bf5a2ae7fbd4c9 Mon Sep 17 00:00:00 2001 From: zzylol Date: Wed, 21 Oct 2026 00:18:26 +0000 Subject: [PATCH 8/8] docs: per-sample ingestion in the terminology --- docs/design/backend-on-planner-physical-dag.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/design/backend-on-planner-physical-dag.md b/docs/design/backend-on-planner-physical-dag.md index 930a9cd6..05bbd176 100644 --- a/docs/design/backend-on-planner-physical-dag.md +++ b/docs/design/backend-on-planner-physical-dag.md @@ -63,6 +63,7 @@ This document, the backend's code and its PR descriptions use ASAPPlanner's term | `InstallablePlan`, `StoredOutput`, `StoredOutputBuild` (`IngestionRun`, `QueryRun`), `RawSampleWindow`, `retention_ms` | Planner (`asap_planner`) | The plan a runtime installs, its stored outputs, which run builds each one, the raw-sample windows read at query time, and how long each output is kept. | | evaluation time, `RangeEvaluation` (step, span) | Planner (workload) | The time `T` a query is answered for, and the steps of a range query. | | pane, pane chain | Planner | A summary over one left-open time interval `(end − width, end]`; the panes that make up a window. | +| per-sample ingestion | Backend | Updating each open pane's summary state in place as each raw sample arrives (`StateColumnUpdater`), then sealing it into a summary instance when the pane closes. It gives the same state as building the pane from all its raw samples at once. | | raw sample | Backend | One value of one series at one timestamp, as received over Remote Write. Not a sampling-based approximation. | | summary instance | Backend | One stored summary state with its concrete `selection` (relative time bound to an absolute interval). | | SDS | Backend | The backend's store of summary instances, keyed by (`definition` identity, concrete `selection`). |