Plan selection: price storage I/O, handoffs and cache, from deployment evidence with provenance #733

Example

A deployment keeps raw samples in object storage, where each GET request is billed, and moves sketch state between an ingest node and a query node over a billed network. It runs one query:

sum by (job) (rate(http_requests_total[1h]))   # repeated every 60 s

Stage 2 offers two candidates:

Today: Stage 3 (stage3_plan_selection::price) charges the raw candidate only for its scanned bytes (ResourceUnitCost::cost_per_scanned_byte). It charges the maintained candidate for CPU and memory. It does not count the scan's object-storage requests or the bytes the maintained state sends to the query node. A deployment whose per-request price dominates cannot express that, and Stage 3 can pick the raw plan even though its GET bill is the larger one.

Expected: given the deployment's storage and handoff prices, Stage 3 should add:

• the scan's request cost (cost_per_object_get × requests);
• the handoff's cost (cost_per_network_byte × encoded bytes × copies).

It should then choose with those costs included and report them per node.

Why it happens

These capabilities lived in the legacy core's physical_plan_cost_model::PhysicalPlanCostModel, a legacy CostModel adapter over replacement candidates. The measured-operation-costs PR removed it, together with empirical_cost and empirical_resources, without wiring these capabilities into Stage 3:

• (a) Deployment-supplied per-node physical evidence.
  • What it was: PlannerPhysicalPlanProvider captured a versioned PhysicalEvidenceSnapshot (comparison scope, evidence version). It returned per-node evidence for the query operators and the deployment's own physical DAG for a summary.
  • Stage 3 today: it prices from its own illustrative statistics (DEFAULT_GROUP_COUNT, the data workload's rows per second).
• (b) Storage I/O and handoff pricing.
  • What it was: the snapshot's StorageIoProfile (disk reads/writes, object GETs/PUTs) was priced by storage_io::estimate_storage_io. Its PhysicalHandoffProfile (network and materialization bytes) was priced by physical_handoff_cost::estimate_physical_handoffs.
  • Stage 3 today: it prices neither.
• (c) Cache profiles.
  • What it was: CacheProfile evidence (result and buffer hit ratios) reduced CPU and scanned bytes. Mixing it with storage or handoff pricing was refused.
  • Stage 3 today: it has no cache input.
• (d) Provenance and applicability of measurements.
  • What it was: empirical_cost::EvidenceArtifact carried per-record provenance (command, dataset, source revision, repetitions), distribution and environment descriptors, and a validity window. EmpiricalEvidenceProvider::lookup refused records that were stale, from another distribution or machine, or ambiguous.
  • Stage 3 today: its measured table (PlanSelectionSettings::measured_operation_costs) is a plain per-(sketch, operation) CPU table, and the deployment is trusted to have converted applicable measurements.
• (e) Measured memory and per-instance CPU.
  • What it was: empirical_resources::ResourceMeasurements carried measured retained, peak and serialized bytes (and disk and scan bytes) per sketch configuration. It also carried the CPU to build one instance (build_cpu_ns) and to prepare a read (prepare_cpu_ns), beside update, merge and read.
  • Stage 3 today: it sizes state from its formulas only. convert_rqe_atomic_costs drops sketch-bench's mem_bytes_per_instance. The measured table has update, merge and estimate only.
• (f) Measured exact-accumulator costs.
  • What it was: empirical_resources::ExactResourceMeasurements gave the same measured dimensions for exact accumulators.
  • Stage 3 today: the measured table prices sketches only. The converter skips exact-* rows.
• (g) Measured offline error.
  • What it was: empirical_cost::OfflineError recorded the error observed on offline ground truth (metric, mean, max, trials, ground-truth method, query) per measurement, without inferring a guarantee.
  • Stage 3 today: it reports only formal guarantees. Measured error is neither carried nor shown.

Three modules that did this pricing are kept, tested, and currently unwired:

• crates/asap-aware-mapping/src/storage_io.rs (StorageIoProfile, StorageCalibration, estimate_storage_io), with tests/storage_io.rs;
• crates/asap-aware-mapping/src/physical_handoff_cost.rs (PhysicalHandoffProfile, PhysicalHandoffCalibration, estimate_physical_handoffs), with tests/physical_handoff_cost.rs;
• crates/asap-aware-mapping/src/query_physical_lowering.rs (lower_query_physical_dag, PhysicalNodeEvidenceProvider).

Proposal

1. Storage and handoffs in price().

   • Settings: add optional storage I/O and handoff prices to PlanSelectionSettings, next to measured_operation_costs.
   • Stage 3: charges a query-time scan's storage requests and a materialized node's handoff bytes per computation, through storage_io and physical_handoff_cost.
   • Reporting: a new NodeUsage field per dimension. Each cost model prices it: cost model 1 per request or byte, cost model 2 as provisioned bandwidth, and cost model 3 as a limit or objective.

2. Deployment per-node evidence.

   • Settings: an optional provider (query_physical_lowering's PhysicalNodeEvidenceProvider, or its successor) that supplies a node's edge statistics.
   • Stage 3: uses those statistics instead of the illustrative ones.
   • Reporting: each node says whether its statistics came from the deployment or the defaults, as NodeCost::price_source does for CPU prices.

3. Cache profiles.

   • Settings: an optional cache profile.
   • Stage 3: applies the hit ratios to scanned bytes and CPU, and refuses to combine them with storage request pricing until there is per-extent cache evidence. This is the same rule the legacy adapter had.

4. Provenance and applicability of the measured table.

   • Table: optional per-entry provenance (benchmark version, environment, data shape and validity window).
   • Stage 3: rejects an entry measured for another environment, or one past its validity, as bad input.
   • Converter: convert_rqe_atomic_costs fills the fields from sketch-bench's measured_at.

5. Measured memory and exact accumulators.

   • Table: an optional measured state size per entry (retained, peak, serialized bytes), and entries for exact accumulators.
   • Converter: it fills them from mem_bytes_per_instance and the exact-* rows.
   • Stage 3: it prices a node's memory from the measured size when there is one, and reports it in price_source.

6. Measured offline error.

   • Carried beside the measured table and shown with a candidate's guarantee, as evidence only. It never replaces or weakens a formal guarantee.

Scope and acceptance

• In scope: (a)–(g) in Stage 3 and the deployment settings. Out of scope: runtime feedback, and executors reporting measurements back.
• Acceptance:
  • a Stage 3 test where adding an object-GET price makes the maintained candidate win over the raw one in the example above. With no storage profile, the raw candidate's price is unchanged.
  • a handoff test where the network-byte price changes exactly the materialized node's price, and that node reports it.
  • a test where a measured entry outside its validity window is rejected.
  • a test where a measured state size changes exactly that node's memory and reports it.
  • The existing example tests that pin selections and costs pass without new settings.

Status and related work

• Decided (2026-10-16): the measured-operation-costs PR deletes physical_plan_cost_model.rs, empirical_cost.rs and empirical_resources.rs as a deliberate exception to "delete only when replaced". It keeps the three modules above for this issue.
• Related: Calibrate plan selection's costs from measured sketch costs #679 and feat(asap-aware-mapping)!: price sketch nodes from measured operation costs in Stage 3 #734 (measured per-operation CPU costs in Stage 3), Stage 3: memory, disk and latency trade-offs per candidate, and cost models beyond cloud billing #660 (deployment cost models), Planning roadmap: move ASAPQuery planning into ASAPPlanner, MILP plan selection over shared DAGs, benchmark-based accuracy #671 (selection search).