perf: prune partitions in the remaining report queries - #2351
Open
tlangton3 wants to merge 2 commits into
Open
Conversation
`current_tests_run_results_query` filters both of its large CTEs with
`edr_datediff(edr_cast_as_timestamp(col), now, 'day') < days_back`. On BigQuery a
function applied to the column defeats partition pruning, so both scan all
history however small `days_back` is.
Adds a dispatched `days_back_filter` macro: the default implementation renders
exactly the predicate above, so nothing changes off BigQuery, while `bigquery__`
compares the column directly and adds a bound on the partition column.
Measured with a 7-day lookback, selecting a single column, on tables as they
already are — no repartitioning needed to get this:
elementary_test_results (12M rows, 13GB) 1,368 MB -> 13.68 MB
dbt_run_results (17M rows, 41GB) 1,627 MB -> 63.56 MB
For reference, an unfiltered scan of elementary_test_results reads 1,279 MB: the
current predicate reads *more* than no filter at all, since it also pays to read
the column it filters on.
The bound on `created_at` looks redundant — it is implied by the first predicate —
but BigQuery only prunes on the raw partition column, and `execute_completed_at`
is stored as a string, so a cast is unavoidable there and no predicate on it can
prune. dbt_run_results is not even partitioned on the warehouse measured above;
the 26x comes from block-level pruning on `created_at` alone, and partitioning it
would improve that further.
The bound is also safe: `created_at` is written at or after both columns, so any
row satisfying the first predicate satisfies the bound. Verified over 17.2M rows —
`created_at - generated_at` >= 0s, `created_at - execute_completed_at` >= 5s, and
zero rows violating either.
The guard is given a day of slack rather than the same bound: `created_at` is
stamped by the warehouse at insert while the columns it guards come from the dbt
client, so the two are not the same clock. Pruning is at day granularity, so the
slack costs at most one extra partition — measured as no difference at all on the
figures above, since the real predicate is the binding constraint.
…ilter elementary-data#1940 gave this macro a `bigquery__` override that duplicated the whole body just to change its `where` clause. Now that `days_back_filter` dispatches on the predicate itself, the two copies collapse back into one and the dispatch disappears. Behaviour is unchanged on every adapter: `default__days_back_filter` renders the datediff form the original had, and `bigquery__days_back_filter` renders the direct comparison the override had. It also gains the `created_at` bound. test_result_rows declares no partitioning today, so the direct comparison on `detected_at` was relying on block-level pruning alone; the bound takes a 7-day lookback from 1,060 MB to 101 MB on a 26M row / 41GB copy of that table, and improves further once it is partitioned.
Contributor
|
👋 @tlangton3 |
tlangton3
requested a deployment
to
elementary_test_env
September 10, 2026 17:05 — with 
GitHub Actions
Waiting
GitHub Actions
Waiting
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (3)
Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review. 📝 WalkthroughWalkthroughThe change adds an adapter-dispatched ChangesDays-back filtering
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to The change preserves non-BigQuery filtering while adding BigQuery partition pruning, with reported scan reductions and completed rendering and dry-run checks. No merge-blocking risk is currently identified. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The report queries filter their largest tables with
edr_datediff(edr_cast_as_timestamp(col), now, 'day') < days_back. On BigQuery, a function applied to the column defeats partition pruning, so these scan all history however smalldays_backis.This is the same defect #1940 fixed for
test_result_rows, applied to the remaining sites — and it generalises that fix so the pattern is in one place instead of copied per macro.Measured
A 7-day lookback selecting a single column, on tables exactly as they are — no repartitioning needed to get this:
elementary_test_results(12M rows, 13GB)dbt_run_results(17M rows, 41GB)test_result_rows(26M rows, 41GB)For scale, an unfiltered scan of
elementary_test_resultsreads 1,279 MB: the current predicate reads more than no filter at all, because it also pays to read the column it filters on.How
A dispatched
days_back_filtermacro.default__renders exactly the predicate these queries have always used, so nothing changes off BigQuery.bigquery__compares the column directly and adds a bound on the partition column.Applied to both CTEs of
current_tests_run_results_query, and toget_result_rows_agate— which lets that macro'sbigquery__override go away. #1940 had to duplicate the entire body just to change onewhereclause; with the dispatch in the predicate helper, the two copies collapse back into one.About the redundant-looking bound
The BigQuery variant emits e.g.
detected_at > X and created_at > X-1day. The second predicate is implied by the first — it exists only because BigQuery prunes on the raw partition column and nothing else. It is needed when the filtered column is not the partition column, and unavoidable fordbt_run_results, which storesgenerated_atandexecute_completed_atas strings: those need a cast, and no cast predicate can ever prune.It cannot exclude a row the real predicate wants, because
created_atis stamped byinsert_rowsfromedr_current_timestamp()at insert time and never read from the row payload — so it is always at or after the event timestamp it guards. Checked over 17.2M rows:created_at - generated_at>= 0s,created_at - execute_completed_at>= +5s,created_at - detected_at>= 0s across 12M and 26M row tables, zero violations of any.Since those two timestamps are not read from the same clock — one warehouse-side, one client-side — the bound is given a day of slack rather than the same bound, so client drift cannot cost a row. Pruning is at day granularity, so that is at most one extra partition; on the figures above it measured as no difference at all, the real predicate being the binding constraint.
Not included
get_models_runshas the same defect and is the largest single win, but it reads themodel_run_resultsview, which does not exposecreated_at— so it has no prunable column until elementary-data/dbt-data-reliability#1057 lands. Deliberately left for a follow-up rather than shipped broken.get_source_freshness_resultsandcan_upload_source_freshnesscarry the same shape. I have no source-freshness data to measure, so I have not touched them — happy to if you would like them included.Testing
Rendering verified for each branch of the macro: timestamp column, string column (cast applied), the column being the partition column itself (bound correctly suppressed), a string-typed
days_back, and the default path (byte-identical semantics to the predicate it replaces). Resulting predicates dry-run against real tables for the figures above.typosclean; the repo's Python hooks do not apply to.sql.Summary by CodeRabbit
Performance
Consistency