perf: partition elementary_test_results and test_result_rows on created_at - #1057
perf: partition elementary_test_results and test_result_rows on created_at#1057tlangton3 wants to merge 2 commits into
Conversation
…ed_at Both tables are append-only and grow without bound, and both are read with a `days_back` time filter, but neither declares `partition_by` — so on BigQuery every read scans all history. They are the two largest tables the package creates. On one warehouse `test_result_rows` is 26M rows / 41GB and `elementary_test_results` is 12M rows / 13GB, against which a 7-day report query reads the whole table. With `created_at` partitioning the same query prunes to the requested window. Uses `get_partition_by()` with no argument, i.e. the package's existing `created_at` default, matching how dbt_run_results and dbt_invocations already declare it. `created_at` rather than `detected_at` because it is each table's declared `meta.timestamp_column` and it is monotonic with insertion, so on an append-only table writes stay in the newest partition. No effect on non-BigQuery adapters: `default__get_partition_by` returns none. Existing tables keep their current layout until a full refresh, since dbt only applies partitioning at creation.
`model_run_results` selects an explicit column list from dbt_run_results and omits `created_at`, so consumers of the view cannot filter on it. That matters on BigQuery because `created_at` is dbt_run_results' partition column, and the view's other timestamps cannot prune: `generated_at` and `execute_completed_at` are stored as strings, so any predicate on them needs a cast, and a cast defeats partition pruning. Without `created_at` a query over this view has no prunable column at all and must scan all history. The view is rebuilt on every run and this only adds a column, so nothing downstream breaks.
|
👋 @tlangton3 |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (3)
Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review. 📝 WalkthroughWalkthroughThe run-results models now use shared partition configuration for two incremental models. The ChangesRun-results model updates
Estimated code review effort: 1 (Trivial) | ~5 minutes Suggested reviewers: Merge Risk: ⚪ Minimal · up to The PR adds BigQuery-aware partitioning to two run-results tables and exposes created_at for filtering. No concrete merge-blocking risk is identified. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
Companion CLI change: elementary-data/elementary#2351. The view column added here is also what unblocks the |
What
Two changes, both one-liners:
partition_by=elementary.get_partition_by()onelementary_test_resultsandtest_result_rows— the package's existingcreated_atdefault, matching howdbt_run_resultsanddbt_invocationsalready declare it.run_results.created_atadded to themodel_run_resultsview's select list.Why
These are the two largest tables the package creates, both append-only, both read with a
days_backlookback — and neither declarespartition_by, so on BigQuery every read scans all history. On one warehouse they are 26M rows / 41GB and 12M rows / 13GB.created_atrather thandetected_atbecause it is each table's declaredmeta.timestamp_column, and because it is monotonic with insertion:insert_rowsstamps it fromedr_current_timestamp()at insert and never reads it from the row payload, so on an append-only table writes always land in the newest partition. A business timestamp could be backdated and scatter writes across old partitions.The view change
model_run_resultsselects an explicit column list and omitscreated_at, so consumers of the view have no prunable column at all: its other timestamps (generated_at,execute_completed_at) are stored as strings, so any predicate on them needs a cast, and a cast defeats pruning. Adding the column is a prerequisite for fixing the CLI'sget_models_runs, which reads this view and currently scans the whole table on every report.The view is rebuilt on every run and this only adds a column, so nothing downstream breaks.
Effect on existing installations
None automatically, and nothing breaks. dbt applies partitioning only when a table is created, so existing tables keep their current layout and their queries remain valid — they simply do not gain pruning until the table is rebuilt.
Worth being explicit that
--full-refreshis not the way to get it: these models compile to an empty table and are populated byon-run-endhooks, so a full refresh would discard all history. The package already prevents this —elementary_full_refreshdefaults tofalse— but anyone who has enabled that var should not reach for it here. The safe route is a one-off rebuild that preserves rows:(BigQuery rejects
CREATE OR REPLACEacross a partitioning change, hence build-and-swap.)Non-BigQuery adapters
No-op.
default__get_partition_byreturnsnone, andbigquery__get_partition_byalso returnsnonewhenbigquery_disable_partitioningis set, so the existing opt-out is respected.Testing
sqlfmtreports no changes on all three files. The partition column exists on both models —empty_table.sqldeclarescreated_atas atimestampfor each.I have not run the full integration suite across all five platforms; these are config-only additions that are inert off BigQuery, and I am happy to run the BigQuery suite or make changes if you would like.
Summary by CodeRabbit
created_attimestamps to model run results, providing more complete run metadata.