fix(tesseract): bound a rolling window's base scan by literals - #11870
waralexrom wants to merge 21 commits into
Conversation
|
Claude finished @waralexrom's task in 8m 5s —— View job Review complete — no blocking issues; 1 medium, 6 low (details)Full review — 7 inline comments posted, verdict and reasoning
VerdictThe core change is sound and well targeted. I traced the covering-bounds derivation against both series shapes and could not break it:
Testing is genuinely strong. The flipped snapshot in Findings
Nothing here blocks. #1 is the only one with a wrong-answer path, and it's narrow (sub-day rolling window + DST transition + callback/compiled binding) — but it is the same class of bug the rest of the PR fixes, so it's worth either matching the SQL's arithmetic or pinning it with a test. Two notes on the stated risks, both of which I agree are correctly scoped rather than papered over: the "up to one interval more" read is real and the month-granularity 8% figure checks out against the derivation; and a No prior review threads existed on this PR, so nothing to resolve or dedupe against. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #11870 +/- ##
=======================================
Coverage 80.37% 80.37%
=======================================
Files 500 500
Lines 106573 106573
Branches 3885 3885
=======================================
Hits 85658 85658
Misses 20369 20369
Partials 546 546
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Claude finished @waralexrom's task in 3m 4s —— View job Re-review of
|
| Prior | Status |
|---|---|
#2 dup format_and_allocate_in_query_tz |
Folded into format_and_allocate_date_impl(…, in_db_time_zone); the two unreachable guards are gone with it |
#3 Granularity built twice / borrow held |
One to_date_period_granularity builder, borrow scoped to the build, both call sites converted |
#4 hardcoded .000, triplicated precision |
default_format() at both sites, SERIES_BOUND_PRECISION const (see new finding 1) |
#5 Debug omits window_range |
Added |
#6 over-pinned param list, needless async |
contains-per-edge, plain #[test] |
| #7 four copies of the covering-span rationale | Cut to the load-bearing sentence in all four |
All seven threads resolved. Two things from #4 remain and are fine to leave: let precision = 3 at physical_plan/filter/typed_filter.rs:262 is in a different module from the new const, and the converge_guard gap on QueryDateTime::align_to_origin was flagged as pre-existing.
New findings
| # | Sev | Where | What |
|---|---|---|---|
| 1 | Medium | physical_plan/.../typed_filter.rs:317 |
carries_date counts quarter, but add_interval ignores quarter in both its is_time_only guard and its date branch — a 1 quarter frame shifts the band by zero, so a quarter-length rolling window's FILTER_PARAMS band is the bare series bounds |
| 2 | Low | multi_stage_query_planner.rs:52 |
SERIES_BOUND_PRECISION landed between the MultiStageQueryPlanner doc comment and the struct — the struct is now undocumented and its doc reads as the const's |
| 3 | Low | multi_stage_query_planner.rs:1294 |
Extracting to_date_period_granularity left to_date_window_bounds's doc block on the helper; the helper is now documented as computing a span it doesn't compute, and to_date_window_bounds has no doc |
Finding 1 is the only one with a wrong-answer path. It is pre-existing in add_interval — sub_interval had the same hole before this PR — but the band is new code that depends on it, and it is the same undercount the PR set out to fix. Findings 2 and 3 are both artifacts of where the extraction cut; neither changes behaviour.
Nothing here blocks. I could not run cargo test -p cubesqlplanner in this environment (the command needs approval), so the assertions above come from reading date_time.rs, sql_interval.rs, and the new test module rather than from a run.
4c2ea18 to
c7a4208
Compare
|
Claude finished @waralexrom's task in 2m 1s —— View job Re-review of
|
| Prior | Sev | Where | State |
|---|---|---|---|
| r2 #1 | Medium | physical_plan/filter/typed_filter.rs:324 |
Open. shift_wall_clock's carries_date still counts interval.quarter != 0, and QueryDateTime::add_interval still ignores quarters on both paths — is_time_only is year == 0 && month == 0 && week == 0 && day == 0 (date_time.rs:167, quarter absent) and the date branch never reads interval.quarter. SqlInterval::from_str parses quarter/quarters (sql_interval.rs:233) into its own field and never folds it into months. So trailing: 1 quarter still routes to add_interval, lands in the time-only branch, adds a zero duration, and the band equals the bare series bounds. |
| r2 #3 | Low | multi_stage_query_planner.rs:1290-1303 |
Open. to_date_window_bounds's two doc paragraphs are still concatenated onto to_date_period_granularity, which computes neither the span nor the None cases they describe; to_date_window_bounds (l.1319) still has no doc. |
One small thing the const move newly enables, not worth its own thread: physical_plan/filter/typed_filter.rs:262 still has a bare let precision = 3; in the RollingWindowOffset arm. Now that QueryTimeSeries::MILLISECOND_PRECISION is pub and in a shared module, that last copy can read it too.
Caveat
cargo test needs approval in this environment, so as in round 2 the assertions above come from reading date_time.rs, sql_interval.rs, time_series.rs and the new test module rather than from a run. The author's own cargo test -p cubesqlplanner --features integration-postgres result (1395 pass) stands unchallenged.
Nothing here blocks; finding r2 #1 is the only one with a wrong-answer path, and it is narrow (a quarter-length rolling frame with a FILTER_PARAMS binding).
|
Claude finished @waralexrom's task in 1m 32s —— View job Re-review of
|
|
Claude finished @waralexrom's task in 1m 25s —— View job Re-review of
|
|
Claude finished @waralexrom's task in 3m 42s —— View job Re-review of
|
| # | Sev | Where | What |
|---|---|---|---|
| 1 | Medium | physical_plan/.../rolling_window.rs:64,73 |
RollingWindowOffsetOp::to_sql still wraps an allocated parameter in subtract_interval/add_interval — the exact TIMESTAMPADD(MONTH, -2, CAST(? AS TIMESTAMP)) shape this commit removes from the regular path. Rolling windows with offset: start/end stay broken on Pinot. The folded bounds already exist at plan time in rolling_window_band's offset arm |
| 2 | Low | tests/filter_params_rolling_window.rs:131 |
Nothing pins the fix. This test's predicate.contains("interval '30 day'") was the one assertion that observed SQL interval arithmetic, and it was replaced by a value assertion. Every surviving check — here, fanout_repro, timezone.rs — passes under a double application (folded literal and subtract_interval still around it), so reverting the to_sql half leaves the suite green |
| 3 | Low | multi_stage_query_planner.rs:1341 |
fold_frame duplicates shift_bound line for line except for what unbounded means; all four DST/quarter tests sit on shift_bound, so the copy that decides what the scan actually reads is untested. Also "unbounded" is now compared as a literal in four places, and keep_bounded takes an unused &self |
Finding 1 is the only one that leaves a user-visible failure, and it's the same failure the commit set out to fix — just on the other rolling-window shape. Finding 2 is what would have caught it: a single !predicate.contains("interval") in the two tests that already extract the predicate.
Nothing here blocks the regular-window fix, which is correct and well targeted.
Prior rounds
All ten threads from rounds 1–4 are resolved; none re-opened by this commit. The shift_wall_clock → add_interval_wall_clock move preserves the round-2 DST fix and the round-3/4 quarter fix in add_interval verbatim.
Caveat
cargo test -p cubesqlplanner needs approval in this environment, so as in rounds 2–5 the verification above comes from reading date_time.rs, filter_sql_context.rs, rolling_window.rs, typed_filter.rs and the test modules rather than from a run. The author's cargo test -p cubesqlplanner --features integration-postgres result stands unchallenged — though note finding 2 means a green suite is weaker evidence for this particular commit than for the earlier ones.
|
Claude finished @waralexrom's task in 2m 58s —— View job Re-review of
|
| Prior | Status |
|---|---|
r5 #1 — RollingWindowOffsetOp::to_sql still wrapped its parameter in subtract_interval/add_interval (Medium, the one user-visible failure left) |
Fixed. The anchor is formatted but no longer allocated (ctx.format_from_date/format_to_date), the frame folds in via shift_bound_wall_clock, and the bound is allocated after. Both rolling operators now emit bare comparisons |
| r5 #2 — nothing pinned the fold; the suite stayed green under a double application | Fixed. !predicate.contains("interval") in filter_params_rolling_window, per-base-scan in fanout_repro, !sql.contains("interval") in basic_types, and the four offset/no-granularity cases now assert the shifted date instead of the interval text |
r5 #3 — fold_frame duplicated shift_bound |
Fixed. One shift_bound_wall_clock in date_time.rs serves all three callers; the planner uses it with .unwrap_or(bound), so the DST and quarter tests now sit on the function the base scan actually reads. UNBOUNDED_INTERVAL named alongside it |
Verification
- Anchor still needs the pre-format. A bare
2024-01-20has to become…T23:59:59.999before the shift, or anoffset: endwindow would shift midnight instead of end-of-day. Keepingformat_from_date/format_to_dateunallocated for that is the right split. - Test arithmetic checks out.
basic_types:2024-01-20T23:59:59.999− 3d =2024-01-17, + 1d =2024-01-21✓; trailing-only: − 7d =2024-01-13, upper = the unshifted2024-01-20✓.offset_no_granularity:2024-01-10− 7d =2024-01-03(start),2024-01-16T23:59:59.999− 7d =2024-01-09(end) ✓. - Unbounded stays consistent.
shift_bound_wall_clockreturnsNone, the offset arm'sif let Somedrops the condition, the planner'sunwrap_orkeeps the unshifted bound forkeep_boundedto drop at render, androlling_window_bandstill yieldsNone— the two cumulative tests from50b643eare unaffected. - Timezone. The offset operator compares an unconverted member, and the shift is wall-clock in the query tz followed by
apply_db_time_zoneat allocation. That now matches howrolling_window_bandderives theFILTER_PARAMSband for the same operator (query-tz shift, then the db-tz_no_casthelpers) — the two derivations agree, which is the point. - DST and quarter fixes survive the move verbatim:
add_interval_wall_clockstill branches oncarries_date(quarter included) andadd_intervalstill foldsquarter * 3into months.
New findings
| # | Sev | Where | What |
|---|---|---|---|
| 1 | Low–Medium | physical_plan/.../rolling_window.rs:70 |
Each folded bound is re-normalised by the side of the comparison, not of the anchor. At timestampPrecision() == 6 (BigQuery) an offset: end lower bound round-trips .999999 → .999 (via default_format's %.3f) → .999000 (via format_from_date, which has no .999 case). This predicate is the window's frame — nothing re-applies it — so the window gains 999 µs of rows. Was exact before the commit |
| 2 | Low | filter_sql_context.rs:281, :293 |
keep_bounded and extend_date_range_bound still compare the "unbounded" string literal now that UNBOUNDED_INTERVAL exists two modules over; keep_bounded also still takes an unused &self. The leftover half of r5 #3 — not worth its own thread |
Finding 1 is invisible in the suite because every offset test runs at precision 3, where steps 2–3 round-trip exactly. Narrow (microsecond dialect + offset: end + a trailing frame), and it makes the window slightly wider rather than narrower, so no rows are lost — but it is a result change on the dialect this PR was written for, and the fix is one closure.
Nothing here blocks. The offset-window fix is correct in shape and closes the last place a dialect had constant interval arithmetic to fold.
Prior rounds
All thirteen threads from rounds 1–5 are now resolved; the three that were open are answered by this commit and I resolved them rather than re-affirming.
Caveat
cargo test -p cubesqlplanner needs approval in this environment, so as in rounds 2–5 the verification above comes from reading date_time.rs, date_time_helper.rs, filter_sql_context.rs, rolling_window.rs and the test modules rather than from a run. BigqueryQuery.ts:162 was read directly to confirm precision 6 is reachable. The author's cargo test -p cubesqlplanner --features integration-postgres result stands unchallenged — noting that it runs at precision 3, which is exactly where finding 1 hides.
|
Claude finished @waralexrom's task in 2m 12s —— View job Re-review of
|
| Prior | Status |
|---|---|
r6 #1 — folded bounds re-normalised by comparison side, losing 999 µs off the offset: end lower bound at precision 6 (Low–Medium, the one result-changing path) |
Fixed. An allocate closure keyed on from_start sends both bounds through the anchor's own helper (rolling_window.rs:68) |
r6 #2 — "unbounded" compared as a literal in keep_bounded/extend_date_range_bound; keep_bounded took an unused &self |
Fixed. Both read UNBOUNDED_INTERVAL (date_time.rs:23); keep_bounded is an associated fn and both call sites say FilterSqlContext::keep_bounded |
Verification
- The fix is right, and right for both offsets. For
offset: endthe lower bound now goesformat_to_date(to, 6)→.999999→ shift →.999→format_to_date→.999999, exact; before it landed on.999000. Foroffset: startnothing moves — the anchor is.000000, and both helpers pad.000→.000000identically — so the change is confined to the arm that was wrong. - The test pins it from both sides. Asserting the
format_from_dateresult as well (.999000, "which is what the bug did") means a future edit can't satisfy the test by making both helpers equally lossy. Good shape. UNBOUNDED_INTERVALnow covers every place in the rolling path:shift_bound_wall_clock(date_time.rs:38),is_unbounded(typed_filter.rs:282),keep_boundedandextend_date_range_bound. The two literals left —query_properties.rs:999andjoin.rs:45— are outside it and pre-existing.- No behavioural drift elsewhere.
keep_boundedlosing&selfis mechanical;RegularRollingWindowOpis otherwise untouched, so the round-5 verification of the regular fold still stands.
New finding
| # | Sev | Where | What |
|---|---|---|---|
| 1 | Low | physical_plan/.../rolling_window.rs:68 |
The round-trip is lossless only because format_to_date special-cases a .999 tail. shift_bound_wall_clock renders through default_format() (%.3f) regardless of dialect precision, and parse_native_date_time accepts %.f — so a dateRange end of 2024-01-16T12:00:00.123456 at precision 6 comes back .123000, 456 µs earlier on the same frame-defining > bound |
Same defect as the one just fixed, one order narrower: it needs an explicitly microsecond-valued dateRange, not merely a microsecond dialect. Choosing the right helper can't close it — the loss happens between the two helpers, in the shift's own %.3f. Letting shift_bound_wall_clock render at the dialect's precision would, and would stop the .999 special case from being load-bearing here.
Two comment/structure nits, neither worth a thread: the new tests landed in a second #[cfg(test)] mod shift_bound_tests directly above the file's existing mod tests, which already holds the DST and quarter cases — one module would keep the shift tests together; and the new test's four-line preamble restates what the two assertion messages already say.
Prior rounds
Fourteen threads across rounds 1–6; the one that was open is answered by this commit and I resolved it. Nothing else re-opened.
Caveat
cargo test -p cubesqlplanner needs approval in this environment, so as in rounds 2–6 the verification above comes from reading date_time.rs, date_time_helper.rs, filter_sql_context.rs and rolling_window.rs rather than from a run. The author's cargo test -p cubesqlplanner --features integration-postgres result stands unchallenged — noting again that it runs at precision 3, where the new test's own subject is invisible, which is why the test states the precision-6 round-trip explicitly rather than going through a query.
| // anchor was: an end-of-day anchor shifted by whole days is another | ||
| // end-of-day, and reading it as a range start would round its | ||
| // sub-second tail down on a dialect that keeps more than milliseconds. | ||
| let allocate = |bound: &str| { |
There was a problem hiding this comment.
This closes the .999 case, and the new test pins it. One narrower leak survives it, same shape: the round-trip is only lossless because format_to_date special-cases a .999 tail — every other sub-millisecond tail still gets truncated.
shift_bound_wall_clock renders through QueryDateTime::default_format() (%.3f, date_time.rs:194), and parse_native_date_time accepts %.f, so a dateRange end of 2024-01-16T12:00:00.123456 at precision 6 goes:
ctx.format_to_date(to)→.123456(26 chars, matchesDATE_TIME_LOCAL_U_RE, returned as-is);- shift →
.123; format_and_allocate_to_date→.123000, because the.999case doesn't apply.
456 µs earlier against the > lower bound, which — as with the case this commit fixes — is the window's own frame with nothing to re-apply it. Requires an explicitly microsecond-valued dateRange on a microsecond dialect, so narrower still than the .999 case, but it's the same defect and it isn't fixed by choosing the right helper.
The root of it is that the shift renders at millisecond precision regardless of what the anchor carried. A default_format_with_precision(p) — or having shift_bound_wall_clock take the precision it should render at — would make the round-trip exact for any tail and let format_to_date's .999 special case stop being load-bearing here.
Adds planner tests that pin the plan shape reported in #11770: several rolling_window measures queried with a high-cardinality dimension emit the base scan's date bound as a scalar sub-select over time_series, where the legacy planner emitted a literal, so engines cannot use it to eliminate partitions; and the rolling CTE joins time_series to each base CTE on a date range only, with no equality predicate on the dimension that is in the GROUP BY of both sides. On the reporter's cardinality (4.7K entities over 33 day-anchors) that materialises ~10M intermediate rows for a 5K-row result; Postgres plans the range-only join as a nested loop with a join filter. The bound test fails today. The join shape is a plan-shape change for every rolling query and is left for separate work, so its test is ignored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A regular rolling window restricts its base scan to the span the window can reach, and read both ends of that span back off the time series with a scalar sub-select. The bounds are opaque to an engine that way: no partition can be eliminated by them, so every base scan of the query reads the whole table, and a query with several rolling measures pays that once per measure. The series is derived from the time dimension's granularity and date range, so its span is known while planning. Compute it there and render both bounds as literal parameters, falling back to the sub-select only where the span is not derivable — a range that is itself a query, a granularity whose periods come off a calendar cube, and the placeholder ranges of pre-aggregation SQL. The span covers one interval past the range end: a series materialized while planning snaps its points to bucket boundaries, so its last bucket ends at most an interval past the range end, while one generated in SQL steps from the range start instead, and its last point sits at most an interval before it. The span has to cover both, and is exact wherever the range's ends are already bucket boundaries. The rolling join applies the exact frame on top either way, so a wider base scan changes no result. The bounds of a custom granularity's series align to that granularity's origin the way the series' own range does, rather than through the step-capped helper the series walks with — a fine interval and a default origin a few years off the range trips that cap on a range that converges perfectly well. A granularity declaring a zero interval is rejected instead of hanging the alignment it can never converge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A `to_date` window — month-, quarter-, year-to-date — bounds its base scan by the start of the period it counts from, and reads both that and the scan's upper end back off the time series with a scalar sub-select. An engine cannot eliminate partitions by either, and a table declared with a mandatory partition filter rejects the query outright. The same shape reaches a cube narrowing its own scan through FILTER_PARAMS: a column binding restates the stage's own predicate, so it carries the sub-select too, and a callback binding is handed the reported period instead of the band the stage reads, cutting the scan to less than the window sums. Adds the two cases the derivation covers, the calendar-backed period it cannot (characterised, since a period whose boundaries are rows of a calendar cube is not reachable by interval math), and both binding forms over a range opening mid-month, where month-to-date answers the day of the month rather than the day of the range. Two earlier tests recorded the callback narrowing as characterisation and are restated here as the behaviour wanted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A `to_date` rolling window restricts its base scan to the period it counts from, and read both ends of that span back off the time series with a scalar sub-select: `date_trunc(<granularity>, (SELECT min(...)))` for the lower bound, `(SELECT max(...))` for the upper. The bounds are opaque to an engine that way — no partition can be eliminated by them, so the scan reads the whole table, and a table declared with a mandatory partition filter rejects the query rather than merely running slowly. The series is derived from the time dimension's granularity and date range, and the window's own period follows from interval math on top of it, so both bounds are known while planning. Compute them there and render them as literal parameters, falling back to the sub-select where they are not derivable: wherever the series itself is not (a range that is itself a query, a series granularity off a calendar cube, the placeholder ranges of pre-aggregation SQL), and for a period whose own boundaries are rows of a calendar cube — no interval math reproduces those, and the series carries the boundary it read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A `FILTER_PARAMS` column narrows a cube's own scan by restating the query's filter inside the cube's sql. Under a rolling window the filter is the window's, and a callback column was handed the filter's raw values — the reported period — rather than the band the stage reads. The cube was therefore cut to the reporting period while the window summed a wider one, and every window silently undercounted its tail: a month-to-date measure over a range opening mid-month answered the day of the range instead of the day of the month. A callback now receives the band itself: the series span widened by the window's frame for a regular window, the anchor shifted by the frame for one without a granularity, and the period counted from for a to_date window. The band arrives as dates rather than as an interval expression, since a callback's SQL is opaque and nothing can wrap an interval around it. Where a bound is not derivable — an unbounded side has none to state, a series only known at run time carries no dates to shift — the column states nothing at all and renders as always-true. The filter still reaches the query on its own; only its restatement inside the cube's sql is dropped, which leaves the scan wider than needed rather than narrower than correct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A `to_date` window counting its period off a calendar cube reads a band no interval math reproduces — when a fiscal month begins is a row of the calendar, so the planner writes that bound as a sub-select over the series. For the fact table it is worse than opaque: the time dimension is the calendar's column, so the bound restricts the calendar and the fact is reached through the join carrying no date predicate at all. An engine requiring a filter on the partition column rejects such a query outright. Records that, and pins what a binding passing the fact's own column does about it: the stage's own bound, restated against that column, exact rather than approximate and needing nothing of the calendar the planner does not already have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sion A measure can be both a `to_date` window over a calendar cube and read a fiscal period earlier, so one stage carries a shift the calendar names and a window whose band no interval math reaches. Pins both halves. Without a binding addressing the shift the scan stays open, which is the shift's own contract — a binding restating the reported period would cut the scan past the rows the shifted period needs. With one, the stage must accept it rather than refuse it for filtering by a rolling operator instead of a plain date range; the band is still not derivable there, so the binding states nothing, but the model is left somewhere to stand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A binding addressing a time shift is checked against the operator the query filters the member with, because restating a band takes both of its bounds and an operator supplying fewer would leave one rendering as whatever it happened to hold. The check named the plain date range only. A shifted stage can equally be the base scan of a rolling window, whose filter is the window's own — and that carries the same two bounds a date range does. Such a model was refused outright: the shift is what the binding is told to address, and addressing it was an error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…und by Two properties the literal bounds of a rolling window's base scan have to keep, neither of which the suite could see: every existing test runs at UTC, and none mixes a window bounded by interval math with one counting off a calendar. The first pins the timezone. A rolling filter compares the member converted into the query's timezone, and the series places its points there too, so a bound carried into the database's timezone instead sits an offset away from the column it bounds and the scan loses the opening hours of every window. The second pins the span. Measures share one series, and a `to_date` window counting off a calendar puts the calendar's periods on it; the plain window's bounds are still derived by interval math, so they have to keep covering it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ezone A rolling window's base scan compares the member converted into the query's timezone, and the series whose span those bounds describe places its points in that timezone as well. The bounds were allocated through the path a plain date-range filter uses, which carries a date into the database's timezone — right for a filter comparing an unconverted column, an offset away from one that converts. Under `America/Los_Angeles` a range of 2024-01-10..2024-01-12 gave the scan 2024-01-10T08:00:00 while its series opened at 2024-01-10T00:00:00, so the first eight hours of every window went unread. Invisible wherever the query runs at UTC, which is everywhere the suite looked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The band a `FILTER_PARAMS` binding is handed moved its bound with arithmetic that switches to absolute time for an interval carrying no date part, while the stage's own SQL moves the same bound on the wall clock. Across a daylight-saving transition a sub-day frame therefore put the band an hour inside the one the stage reads, clipping rows the window sums — the same class of defect the rest of this work fixes, an hour wide instead of a period wide. The band now shifts on the wall clock, pinned by unit tests over a spring-forward. Alongside it, five smaller things review turned up: The query-timezone formatting path was a copy of the db-timezone one with a single call removed, so the two could drift; it is one path now, gated by a flag. The `Granularity` a `to_date` window counts its period in was built twice for the same triple, each time branching on whether it is calendar-backed — one builder answers that now, and its compiler borrow ends with the build rather than outliving it. `Debug` for the to_date operator omitted the field that decides literal versus sub-select. A test pinned the whole parameter list where it meant to pin two bounds, and two tests were async without awaiting. The rationale for the covering span was restated in four places; the load-bearing sentence stays, the narration goes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`SqlInterval` keeps a quarter in a field of its own and never folds it into months, and `QueryDateTime::add_interval` read months alone — so a quarter was silently dropped, on both of its paths. An interval carrying only quarters even looked time-only, took the absolute-arithmetic branch and added a zero duration, leaving the date exactly where it started. `trailing: 1 quarter` therefore handed a `FILTER_PARAMS` binding a band equal to the bare series bounds, reaching back none of the quarter the window sums. The series' own interval helper has always folded quarters in; this is the other path agreeing with it. Also gives the last bare precision literal the shared constant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A window reaching back without limit has no lower bound to narrow — that is what the measure asks for, not something lost. Its upper end is still the series' own, and a column binding states it as a literal, so the scan stops at the reporting period rather than reading past it. A callback takes both bounds, so a band with only one is a band it cannot be given: it states nothing rather than a bound it was never handed. Recorded so the asymmetry between the two forms is visible rather than surprising. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The base scan's bounds became literals, but the window's frame was still applied to them in SQL — `subtract_interval(param, '2 month')`. That makes the whole bound a constant expression, and a dialect folding such an expression while planning the query can reach its interval function with the argument still unconverted: Pinot fails to plan `TIMESTAMPADD(MONTH, -2, CAST(? AS TIMESTAMP))` with "For input string", and every rolling-window query over it stopped working. The frame is folded into the bound while planning instead, the way a `to_date` window already computes its period start, so the predicate is a bare comparison against a date. Nothing is left for a dialect to fold, and a bare bound is also the one an engine can most readily eliminate partitions by. The sub-select fallback keeps applying the frame in SQL, where the bound is a column and no folding happens. Moves the wall-clock interval arithmetic onto `QueryDateTime`, where the planner and the filter-params band now share it rather than each carrying a copy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit folded a regular rolling window's frame into its literal bound, and left the same shape on the other rolling operator: a window anchored by one end of the date range still wrapped its allocated parameter in interval arithmetic, so a dialect folding the resulting constant expression while planning still fails on it. Its bounds are two dates and a frame, all known here, so they fold the same way. Nothing pinned the first half of the fix either: every surviving check passed under a double application — folded literal *and* the interval still around it — so reverting the render half left the suite green. The tests that read the predicate now also require the frame not to appear in it. One shift serves all three callers rather than each carrying a copy, and the tests that cover daylight saving and quarters now sit on the one the scan actually reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A window anchored by one end of the date range folds its frame into both bounds, and each was then normalised by the side of the comparison it ends up on. An `offset: end` window anchors at an end of day, so its lower bound is another end of day — read back as a range start on a dialect keeping microseconds it loses its sub-second tail, and this predicate is the window's own frame with nothing to re-apply it, so the window silently widens by 999 µs. Both bounds now follow the anchor. Invisible at millisecond precision, which is where the suite runs; the new test states the round-trip at both. Also gives `keep_bounded` and `extend_date_range_bound` the named constant the rest of the shift already uses, and drops the `&self` `keep_bounded` never read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
9f0182d to
96b5785
Compare
|
Claude finished @waralexrom's task in 4m 0s —— View job Re-review of
|
| Prior | Sev | Where | State |
|---|---|---|---|
| r7 #1 | Low | planner/time_dimension/date_time.rs:47-51 |
Open. shift_bound_wall_clock still renders through QueryDateTime::default_format() (%.3f, date_time.rs:193), so any sub-millisecond tail other than .999 is truncated on the round-trip: at precision 6 a dateRange end of 2024-01-16T12:00:00.123456 comes back .123000, 456 µs early against the frame-defining > bound. Narrow — needs an explicitly microsecond-valued range on a microsecond dialect — and it makes the window wider, not narrower. |
Every other thread from rounds 1–7 is resolved; nothing re-opened by the rebase.
Caveat
cargo check / cargo test -p cubesqlplanner both need approval in this environment, and the compare API call I'd have used to diff 9f0182d...96b5785 directly was likewise unavailable — so "content unchanged" above rests on reading the files against round 7's verified descriptions rather than on a mechanical diff, and nothing here re-verifies that the rebased tree compiles. The author's cargo test -p cubesqlplanner --features integration-postgres run stands unchallenged; a fresh run on the rebased branch is what would settle the merge with #11852.
…ries A materialized series stops at the bucket the range end falls in, while one generated in SQL steps the interval from the range start and can end past that. Cover both, so that narrowing the bound for the first shape cannot silently cut the second short. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s walks The span was taken an interval past the range end so that one bound would cover either series shape. A series materialized as rows ends at the bucket the range end falls in, so on every dialect that renders one the scan read a whole extra bucket of the fact table — rows no window frame can reach, and on a partitioned table an extra partition. Carry both ends and let the filter pick by the dialect's templates, which are what decide the shape. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @waralexrom's task in 3m 47s —— View job Re-review of
|
| # | Sev | Where | What |
|---|---|---|---|
| 1 | Low | time_series.rs:128 |
The over-read is gone for the walked shape only. to_stepped is range_end + interval − 1s, not the end of the last step (the new test's own numbers: last bucket ends Apr 14, bound says Apr 19), and from is the bucket-snapped start for both shapes though the generated series' first point is the raw range start. Same defect, same fix shape, at the end of the range that BigQuery actually renders |
| 2 | Low | multi_stage_query_planner.rs:1291 |
For a custom granularity the two ends are collapsed into one, but the walked series stops exclusively (aligned < range_end) where the generated one is inclusive. A range end exactly on a bucket boundary makes the walked scan read one extra bucket — the case this commit exists to remove |
Neither is a wrong-answer path; both are reads wider than needed, in the same direction the commit narrows.
Still open
| Prior | Sev | Where | State |
|---|---|---|---|
| r7 #1 | Low | planner/time_dimension/date_time.rs |
Open, untouched by these commits. shift_bound_wall_clock still renders through default_format() (%.3f), so a microsecond-valued dateRange bound round-trips lossy at precision 6. Existing thread covers it; no duplicate posted |
Every other thread from rounds 1–7 is resolved.
Caveat
cargo check / cargo test -p cubesqlplanner need approval in this environment, so nothing above was run — the verification comes from reading time_series.rs, time_dimension_symbol.rs, multi_stage_time_series.rs, physical_plan/time_series.rs, plan.rs and query_properties.rs. In particular the new SeriesSpan threading is a type change across eight files and has not been compiled here; the author's cargo test -p cubesqlplanner --features integration-postgres run is the evidence for that.
| Ok(SeriesSpan { | ||
| from: first.start_str, | ||
| to_aligned: last.end_str, | ||
| to_stepped: format_with_padding(past_end, &nines), |
There was a problem hiding this comment.
The upper bound is now exact for the walked shape, but the same over-read this commit removes survives on the generated shape, at both ends — and the generated shape is the one BigQuery renders, i.e. the partitioned table the PR is aimed at.
to_steppedisrange_end + interval − 1s, not the end of the last step. Stepping from the range start, the last point isrange_start + k·interval ≤ range_end, so the bound is up to a full interval past it. The new test's own arithmetic shows it: stepping months from Jan 15 ends the last bucket on Apr 14, whileto_steppedis Apr 19 — hence the>=in the assertion, where the walked test can sayassert_eq!.fromispredefined_bucket(range_start).start_strfor both shapes, but the generated series' first point is the rawrange_start. So the low end reads up to an interval of rows no frame can reach, symmetric to whatto_alignedjust fixed at the high end.
Both close the same way to_aligned did — a from_aligned/from_stepped pair, and a stepped upper bound walked the way the series walks it (the loop already exists in generate_predefined, capped by MAX_BUCKETS). If that is deliberately left for later, it's worth saying in the Risks section, which currently reads as if the extra-interval read is gone.
| Ok(SeriesSpan { | ||
| from: first.default_format(), | ||
| to_aligned: past_end.clone(), | ||
| to_stepped: past_end, |
There was a problem hiding this comment.
Collapsing the two ends into one value for a custom granularity is right for the generated shape but half an interval too generous for the walked one, because the two series differ in more than alignment: generate_custom stops at while aligned < range_end (exclusive), while the generated SQL steps d <= end (inclusive) — the asymmetry generate_predefined's own comment calls out as intentional.
So for a range end that lands exactly on a bucket boundary — dateRange: ["2024-01-01", "2024-04-01"] against a 1 month custom granularity with a Jan 1 origin — the walked series' last bucket is Mar 1–Mar 31, while align_date_to_origin(Apr 1) + 1 month − 1s gives Apr 30. One whole extra bucket of the fact table, on the shape this commit exists to tighten.
to_aligned wants the last bucket the walked series actually emits: align, and step back one interval when the alignment is exact.
(Also: this adds a second Granularity::align_date_to_origin call site, which is the uncapped twin of the converge_guard-protected helper the series walks with — pre-existing, noted in an earlier round, just now one site wider.)
…l bug Pinned at 6.2.0, the thin-mode client mis-reads a response packet for some query plans and raises NJS-111, breaking the connection. The same statement over the same server and data succeeds through sqlplus, and through oracledb 6.3.0 and every later release. The declared range already admitted a fixed version and only the lockfile held the broken one, so raise the floor as well: an install must not be able to resolve back to it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @waralexrom's task in 1m 53s —— View job Re-review of
|
| Prior | Sev | Where | State |
|---|---|---|---|
| r7 #1 | Low | planner/time_dimension/date_time.rs |
Open. shift_bound_wall_clock renders through default_format() (%.3f), so a microsecond-valued dateRange bound round-trips lossy at precision 6 |
| r8 #1 | Low | planner/time_dimension/time_series.rs:128 |
Open. Re-read on the current tree: to_stepped is still range_end + interval − 1s rather than the end of the last step, and from is still first.start_str for both shapes though the generated series opens at the raw range start |
| r8 #2 | Low | multi_stage_query_planner.rs:1291 |
Open. Custom granularity still collapses both ends into one value, while the walked series stops exclusively and the generated one inclusively |
None is a wrong-answer path; all three are reads wider than needed, in the direction the PR narrows.
Caveat
cargo test / yarn install both need approval in this environment, so nothing was run — the verification above comes from reading OracleDriver.js, the two changed manifest files, and the planner files backing the three open threads. The author's cargo test -p cubesqlplanner --features integration-postgres result stands unchallenged, and the NJS-111 reproduction is theirs against a real Oracle server, which this environment has no way to check.
… protocol bug" This reverts commit 61f5006.
The oracledb thin client pinned here mis-reads a response packet for the plan these two produce and raises NJS-111, breaking the connection. The same statement succeeds through sqlplus against the same server and data, and through later releases of the driver, so this waits on a driver upgrade rather than on anything in the query. Both shapes reach the same plan, and the second passes today only off the result cache, so skipping one alone moves the failure onto the other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @waralexrom's task in 2m 39s —— View job Re-review of
|
| # | Sev | Where | What |
|---|---|---|---|
| 1 | Medium | fixtures/oracle.json:256-262 + 2b94dd0 |
The revert puts yarn.lock back on oracledb@6.2.0, which the published Docker image installs, so an Oracle user running a YTD rolling window through that image hits NJS-111 at runtime. The skip hides it in CI without closing it, and names no follow-up — "waits on a driver upgrade" has no addressee |
| 2 | Low | fixtures/oracle.json:256 |
Five comment lines where the rest of this file runs one to three; the load-bearing sentence is the last one (both shapes reach one plan) plus the version to upgrade past |
| 3 | Low | fixtures/oracle.json:256 |
(6.2.0) restates what yarn.lock owns and goes stale silently when the lockfile moves; oracledb below 6.3.0 says the same and stays true |
Finding 1 is a judgement call rather than a defect — if an eight-minor dependency jump was felt not to belong in a Tesseract PR, that is a reasonable reason to revert it. The ask is only that the skip point at whatever picks it up.
Still open
No Rust changed, so all three prior findings stand exactly as last round. Each has an unresolved thread; no duplicates posted, and nothing became resolvable.
| Prior | Sev | Where | State |
|---|---|---|---|
| r7 #1 | Low | planner/time_dimension/date_time.rs |
Open. shift_bound_wall_clock renders through default_format() (%.3f), truncating a microsecond-valued dateRange bound at precision 6 |
| r8 #1 | Low | planner/time_dimension/time_series.rs:128 |
Open. to_stepped is still range_end + interval − 1s rather than the end of the last step, and from is the bucket-snapped start for both shapes |
| r8 #2 | Low | multi_stage_query_planner.rs:1291 |
Open. Custom granularity collapses both ends into one value, while the walked series stops exclusively and the generated one inclusively |
Caveat
Nothing was run: cargo test needs approval here, and the failure this push works around is reproducible only against a real Oracle server, which this environment has no access to. The diagnosis (NJS-111, fixed in 6.3.0, succeeds via sqlplus) is the author's and I have taken it as given; what I verified is the skip's mechanics, that the two tests passed on master under Tesseract, and that the published image installs from the reverted lockfile.
| "---- The oracledb thin client pinned here (6.2.0) mis-reads a response packet for the", | ||
| "---- plan these produce and raises NJS-111, breaking the connection. The same statement", | ||
| "---- succeeds through sqlplus and through oracledb 6.3.0 and later, so this waits on a", | ||
| "---- driver upgrade. Both shapes reach the same plan; skipping only one moves the", | ||
| "---- failure onto the other. ----", | ||
| "SQL API: Rolling Window YTD (year + month + day + date_trunc equal)", | ||
| "SQL API: Rolling Window YTD (year + month + day + date_trunc IN)" |
There was a problem hiding this comment.
Taken together with the revert in 2b94dd0, this pair of commits removes the protection rather than the failure, and it's worth being explicit about who is left holding it.
Both names are on master in the skip list only (origin/master:oracle.json:210-211), so under Tesseract they run and pass today. Adding them to tesseractSkip is therefore not "still skipped, now also under Tesseract" — it's a shape that worked before this branch and doesn't after. execute/executePg treat tesseractSkip as the complete list under isTesseractEnv (src/tests/testQueries.ts:82,94), not as an addition to skip, so the two names are correctly placed and the mechanics are right.
The part that isn't only a test concern: packages/cubejs-docker/latest.Dockerfile:20 builds the published image with yarn install --prod against the root yarn.lock, and the revert put that lockfile back on oracledb@6.2.0. So the shipped image carries the client that raises NJS-111, and a YTD rolling window against Oracle breaks the connection at runtime for anyone on it — not just in CI. 61f5006 was what closed that, and the manifest half of it (^6.2.0 → ^6.3.0) was the part protecting a fresh npm i too.
If the bump was reverted because an eight-minor jump doesn't belong in a Tesseract PR, that's a fair call — but then the skip is standing in for a follow-up that nothing here names. A tracking issue in the comment (waits on cube-js/cube#NNNNN) would make these two unskippable by someone other than whoever wrote them; as it reads now, "waits on a driver upgrade" has no addressee.
Two smaller things in the block itself:
- Five comment lines where the rest of this file runs one to three. The load-bearing sentence is the last one — both shapes reach the same plan, so skipping one moves the failure onto the other — plus the version to upgrade past. The sqlplus detail is evidence for the diagnosis, not something a later edit could break.
(6.2.0)restates whatyarn.lockowns; when the lockfile moves, this line quietly becomes wrong while still reading as authoritative.oracledb below 6.3.0says the same thing and stays true.
Summary
A rolling window's base scan read both ends of the span it restricts the fact table to back off the
time_seriesCTE with a scalar sub-select. No engine can eliminate partitions by a sub-select, so every base scan read the whole table — and on a table declared with a mandatory partition filter the query is rejected outright rather than merely running slowly. The legacy planner emitted literals here, so this is a Tesseract regression.This derives the span while planning and renders it as literal parameters, for regular (
trailing/leading) andto_datewindows alike, and fixes three defects found alongside it.Part of #11770. Replaces #11824, which can be closed with its branch: the other half it carried — sharing one base scan between rolling measures — is now #11852, done as a pass over the logical plan after pre-aggregations are matched.
Changes
Literal base-scan bounds. The series follows from the time dimension's granularity and date range, so its span is known at plan time.
QueryTimeSeriesgainedcovering_bounds_predefined, which derives the span per bucket rather than by walking the series, and both rolling filters render the bounds as parameters. The span runs one interval past the range end on purpose: a series materialized while planning snaps its points to bucket boundaries, while one generated in SQL steps from the range start, and the span has to cover both. It is exact wherever the range's ends are already bucket boundaries, and a wider span changes no result — the rolling join applies the exact frame on top.A custom granularity's bounds align to its origin the way the series' own range does, rather than through the step-capped helper the series walks with, which a fine interval and a far-off default origin trip on a range that converges perfectly well.
The sub-select remains where the span is not derivable: a date range that is itself a query, a granularity whose periods come off a calendar cube, and the placeholder ranges of pre-aggregation SQL.
FILTER_PARAMSunder a rolling window. A callback column was handed the filter's raw values — the reported period — rather than the band the stage reads, so a cube narrowing its own scan that way was cut to the reporting period while the window summed a wider one. A month-to-date measure over a range opening mid-month answered the day of the range instead of the day of the month. The callback now receives the band itself; where a bound is not derivable it states nothing and renders as always-true, leaving the scan wider than needed rather than narrower than correct.Timezone of the literal bounds. They were allocated through the path a plain date-range filter uses, which carries a date into the database's timezone — right for a filter comparing an unconverted column, an offset away from a rolling filter, which converts the member into the query's timezone and whose series places its points there. Under
America/Los_Angelesa range of 2024-01-10..2024-01-12 gave the scan2024-01-10T08:00:00against a series opening at2024-01-10T00:00:00, so the first eight hours of every window went unread. Legacy does the same split —allocateParam(timeSeries[0][0])raw againstconvertedToTz().A shift binding on a rolling base scan. A
FILTER_PARAMSbinding addressing a time shift is checked against the operator the query filters the member with, and the check named the plain date range only. A shifted stage can equally be a rolling window's base scan, whose filter carries the same two bounds, so such a model was refused outright — addressing the shift is exactly what the binding is told to do.Testing
cargo test -p cubesqlplanner --features integration-postgres— 1395 pass against a real Postgres.Risks
to_datewindow counting its period off a calendar cube still gets no literal bound — when a fiscal month begins is a row of the calendar. Such a query keeps the plan it has today. AFILTER_PARAMSbinding passing the fact's own column restates the stage's predicate against it, which is exact and needs no promise from the model, but its bounds stay sub-selects and so do not prune. Whether BigQuery'srequire_partition_filteraccepts that shape is untested here — the suite runs on Postgres, which has no such requirement.time_seriesto its base CTE on a date range only, with no equality on the group-by dimension.rolling_window::fanout_repro::test_rolling_join_restricts_by_dimensionstays ignored to mark it.🤖 Generated with Claude Code