Skip to content

feat(pass2): shared year-aligned window segments (Q60) - #628

Draft
zzylol wants to merge 2 commits into
stack/windows-1-b3from
stack/windows-2-segments
Draft

zzylol wants to merge 2 commits into
stack/windows-1-b3from
stack/windows-2-segments

Conversation

@zzylol

@zzylol zzylol commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Stack: #574 → #620 → #618 → #621 → #627 → #625 → #628 → #632 → #634 → #616 → #617 → #622 → #624 → #629 → #630 → #631 → #633 → #635 → #636 → #637

Problem

Example 3's Pattern A (five p99 reports over [5y], [1y], [1y offset 1y], [1y offset 2y] and [3y offset 2y]) and Example 4a (the same, repeated monthly) had no plan that shares window summaries across the reports. The spec's Exponential Histogram tests were ignored.

Changes

Stacked on #625. Decisions Q60–Q62.

  • Pass 2 shared-segment rule. share_window_segments in window_composition.rs groups approximate single-estimate targets over windows of one scan: same estimate up to accuracy, same grouping, no filters, offset ≥ 0. The segment width is the gcd of every window's lookback and offset, so every window boundary lies on the grid. For Pattern A that gives five 1-year segments.
    • Each query merges the segments its range covers, built by the existing tumbling-pane construction (WindowForm::Segments). Their coverage is derived and checked by tumbling_state, as for panes.
    • Composition builds identical segments, which the identical-expression rule merges, so each segment is one node read by every covering query's merge.
    • Merging disjoint KLLs is exact, so no ε is split. Each segment is sized for the strictest consumer, reusing the summary-capability rule's strictest and with_accuracy.
  • All-shared only (Q62). The rule's output is its own Stage 1 variant, Sharing::WindowSegments, labeled · shared segments. In it each grouped target has the segment form as its only alternative, so Pattern A gains exactly one logical candidate (486 → 487). There are no partial or pairwise groupings.
  • Skips. A group is skipped when its windows are all identical (the summary-capability rule's case), when the union has more than MAX_PANES (64) segments, or when its queries recur at different cadences.
  • Materialization (Q61). Unchanged. 3a, a one-off ad hoc batch, gets no ingestion-time option. 4a, monthly and predictable, gets one ingestion-time option for the segments through Stage 2's existing eligibility, so 4a has 488 plans. No kept option is offered, because the 1-year segment width is not the monthly cadence.
  • Display. duration_label gains days, so segments read 365d segments.
  • Viewer stories. The 3a and 4a stories are updated.

Tests

  • Rewritten and un-ignored:
    • stage1_a_window_composition_adds_one_eh_for_all_five becomes stage1_a_shared_segments_serve_all_five: five 1-year KLL segments together read by all five queries, each query covering as many segments as its range has years, and five p99 estimates each reading a merge.
    • stage1_a_keeps_independent_and_shared_window_summaries (Example 3).
    • stage2_a_materialized_eh_is_built_once_for_all_consumers becomes stage2_a_segments_are_built_once_for_all_consumers: five builds read by all five queries through five merges, charged once. Its options are {not materialized} for the one-off batch and {ingestion time, not materialized} for monthly.
  • Still ignored, with new reasons:
    • stage1_a_window_composition_groups_every_pair: no partial or pairwise groupings (Q62).
    • stage2_a_shared_eh_has_three_materialization_options, stage2_a_at_rest_drops_the_ingestion_time_option, stage3_a_rebuilding_per_query_costs_more_than_building_once, stage3_a_once_adhoc_prefers_the_query_time_eh and stage3_a_monthly_amortizes_ingestion_time_maintenance. These expect A1 (kept for one batch), A3 (Q44) or a one-off ingestion-time option (Q61), none of which is generated.
    • No remaining test is EH-only, so none uses the "true EH later" reason.
  • New:
    • pattern_a_shares_five_one_year_segments and segments_need_one_cadence_and_different_windows, unit tests in Pass 2.
    • shares_segments in the test adapter.
    • Example 3's raw-input-only and shared-scan tests skip the segment candidate.
    • The devtool test example4a_repeats_monthly_with_nothing_maintainable becomes ..._with_only_segments_maintainable: 488 plans, the only ingestion-time one being the segments.
  • Example 1 fixture: unchanged. Example 1 has no group.

Finding: segments do not win under today's cost model

The 3a segment plan costs 44 384/s against 39 128/s for the selected shared-input KLLs. In 4a it is 61.64/s at query time and 1 536.9/s at ingestion time, against 54.34/s.

The plan builds 5 years of rows into KLLs where the independent KLLs build 11 years. It loses on pass-through work instead: Stage 3 charges each query-time TimeShift and TimeRange per row of its input, and every segment's shift sees the whole 5-year scan. That is 10 such nodes against 8 in the selected plan, at 2 920 each.

Pricing a query-time shift as free, and a range on the rows it keeps (a time-ordered scan seeks), would make segments cheapest in 3a: roughly 18 100 against 25 100. That pricing change is not made here.

Test plan

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace: 1689 passed, 0 failed, 13 ignored (re-run after the rebase on main d4869a7)
  • python3 -m unittest test_viewer (tools/dag-viewer): 25 OK
  • Regenerated the six example documents. Every selection is unchanged: Example 1 P82, Example 2 P86, 3a P365, 3b P2, 4a P365, 4b P4-m1.

@zzylol
zzylol force-pushed the stack/windows-1-b3 branch from 5540b73 to 9a2a154 Compare October 5, 2026 03:07
@zzylol
zzylol force-pushed the stack/windows-2-segments branch from 7148bf3 to 14921d0 Compare October 5, 2026 03:07
@zzylol
zzylol force-pushed the stack/windows-1-b3 branch from 9a2a154 to aa2ddc0 Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/windows-2-segments branch from 14921d0 to 29b20b8 Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/windows-1-b3 branch from aa2ddc0 to cfafcf1 Compare October 8, 2026 16:47
@zzylol
zzylol force-pushed the stack/windows-2-segments branch from 29b20b8 to 1d70fb2 Compare October 8, 2026 16:47
@zzylol
zzylol force-pushed the stack/windows-1-b3 branch from cfafcf1 to cd7ba50 Compare October 10, 2026 14:32
@zzylol
zzylol force-pushed the stack/windows-2-segments branch from 1d70fb2 to 7d76639 Compare October 10, 2026 14:32
zzylol and others added 2 commits October 11, 2026 16:22
Pass 2's window-composition rule gains shared segments: approximate
single-estimate targets over windows of one scan (same estimate up to
accuracy, same grouping, no filters) share one summary per segment of a
grid whose width is the gcd of every window's lookback and offset. Each
query merges the segments its range covers; composition builds identical
segments, which the identical-expression rule merges. Merging disjoint
KLLs is exact, so each segment is sized for the strictest consumer and no
accuracy is split.

Only the all-shared form is offered (Q62), as its own Stage 1 variant
(Sharing::WindowSegments, "· shared segments") in which each grouped
target has the segment form as its only alternative. A group is skipped
when its windows are identical, the union has more than MAX_PANES
segments, or its queries recur at different cadences. Example 3a gets one
candidate with five 1-year KLL segments; 4a, repeated monthly, can also
maintain them at ingestion time (Q61).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ved by tumbling_state

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
@zzylol
zzylol force-pushed the stack/windows-1-b3 branch from cd7ba50 to bdde9b5 Compare October 11, 2026 17:11
@zzylol
zzylol force-pushed the stack/windows-2-segments branch from 7d76639 to 335d385 Compare October 11, 2026 17:11
zzylol added a commit that referenced this pull request Oct 13, 2026
The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
…stages (#577)

* test: acceptance spec and pending tests for planner-layering Example 1

Define MVP acceptance for #509 Example 1 before the Phase C stage APIs
exist: 1 -> 6 -> 6 -> 1 candidates (Pass 1 x identical-expression
sharing; physical operator implementation only, no materialization).

- Spec with per-stage candidate tables, invariants and doc ambiguities.
- Integration tests against todo!() stage stubs, ignored until the
  stages land; workload and Stage 0 frontend-shape tests run now.
- Expected-only asap-stage-pipeline/v1 fixture for the DAG viewer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run the Example 1 acceptance tests against the real stages

Replace the todo!() stubs with adapters over Stage 0-3, un-ignore the 11
tests that pass, and give each still-ignored test the precise difference
between the implementation and the spec. Add runtime capability checks:
the selected plan compiles in the physical planner; compiling every
candidate stays ignored with the two runtime gaps it finds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read Example 1 logical stages as flat DAGs after the export removal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 acceptance on the composed stages (1 -> 3 -> 3 -> 1)

The tests run Pass 1 and composition, optimize_physical_asap and
select_plan through a shared tests/planner_layering_common pipeline, and
follow the decided behavior: three Pass 1 candidates (Q2 exact, CMS+heap,
CountSketch+heap), every sub-DAG NotMaterialized, P1 selected, CMS+heap
invalid (weights not proven non-negative) and CountSketch+heap invalid (a
Top-K output has no guarantee and no deployment evidence). Hydra (#605),
Pass 2 sharing (#588) and the two runtime gaps (#593, #579) stay ignored
with the PR that delivers them. Tests are named after the stages; the
compile check no longer uses the PromQL fallback. The spec lists where the
doc and the planner differ. The expected-only viewer fixture is removed:
the generated four-stage fixture replaces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: port #590's Example 2-4 acceptance specs and tests

The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* feat(devtools): stage_pipeline examples for #509 Example 3

Ported from #590: --example planner-layering-3a (Pattern A, ad hoc batch of
five p99 reports) and planner-layering-3b (Pattern B, repeating 5-min p99),
the Example 3 acceptance tests' workloads. Example 1's output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: follow #576's per-consumer charging and exact top-k columns

Plan selection charges a NotMaterialized node once per consuming query,
so the charging helper checks NodeCost.multiplier and the Example 4 A3
ambiguity is recorded as decided. Example 1 checks that exact Q2 sorts by
the ranked value and returns [job, topk_10]. The stage viewer guide's
tab 1 row now describes the actual all-exact result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(devtools): one planning pipeline for stage_pipeline and the acceptance tests

run_planning_stages and the #509 example workloads move into the
asap-devtools library; stage_pipeline serializes the run and the
planner-layering tests adapt it. The shared data workload drops
ingestion_rate for data at rest, which validate() requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(devtools): select over every candidate; --max-displayed limits only the document

stage_pipeline capped enumeration at 64, so Example 3a (243 candidates)
selected P41 instead of the cheapest P122 (KLL for all five queries).
The pipeline now enumerates, optimizes and prices every candidate; the
document shows the first --max-displayed (default 64) plus the selected
one, with `combinations` and `capped`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: Example 2's SQL candidates compose; the SQL test waits only on #598

Running the ignored SQL test composes every candidate and fails only on
Q2's options (per-source counts until #598), so #603 is not a blocker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: name stages by number and name; say query-language frontends

The frontends' Run field, id and locals become stage0_query_language_frontends,
and test names carry stage0..stage3 with the stage name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: charging helper follows plan selection's per-consumer rule

With a materialized node the helper accepted any multiplier >= 1. It now
computes each node's computations as plan selection does (walks from
query roots and materialized nodes' inputs, stopping at materialized
nodes), checked on a hand-built candidate with a materialized node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 picks Q2 candidates by (option, no shared input)

physical_with and the invalid-candidate check no longer assume one
candidate per Q2 option, so #588's shared-input variants need no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: Example 3 Pattern B charges each node once per consuming query

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs(stage-viewer): rows 2, 3a and 3b describe the planner's current output

Read from uncapped runs: Example 2's sketch plans are rejected for having
no (ε, δ) guarantee, 3a selects P122 (first of 108 tied cheapest plans,
nothing shared yet) and 3b selects query-time KLL; panes and the latency
check come with the window-summary PRs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs(stage-viewer): row 2 from the SQL run; 3a ties 32 plans, not 108

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 counts only unshared candidates; stage3_ charging-helper names

The 3-logical and 3-physical counts now hold when #588 adds shared-input
variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(devtools): PlanningStagesRun fields carry stage number and name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test(devtools): unique temp file per stage_pipeline run

cargo test runs the tests as threads of one process, so two runs of the same
example shared a file name and one deleted the other's output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant