Skip to content

perf: partition elementary_test_results and test_result_rows on created_at - #1057

Open
tlangton3 wants to merge 2 commits into
elementary-data:masterfrom
tlangton3:feat/partition-test-results-tables
Open

perf: partition elementary_test_results and test_result_rows on created_at#1057
tlangton3 wants to merge 2 commits into
elementary-data:masterfrom
tlangton3:feat/partition-test-results-tables

Conversation

@tlangton3

@tlangton3 tlangton3 commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

What

Two changes, both one-liners:

  1. partition_by=elementary.get_partition_by() on elementary_test_results and test_result_rows — the package's existing created_at default, matching how dbt_run_results and dbt_invocations already declare it.
  2. run_results.created_at added to the model_run_results view's select list.

Why

These are the two largest tables the package creates, both append-only, both read with a days_back lookback — and neither declares partition_by, so on BigQuery every read scans all history. On one warehouse they are 26M rows / 41GB and 12M rows / 13GB.

created_at rather than detected_at because it is each table's declared meta.timestamp_column, and because it is monotonic with insertion: insert_rows stamps it from edr_current_timestamp() at insert and never reads it from the row payload, so on an append-only table writes always land in the newest partition. A business timestamp could be backdated and scatter writes across old partitions.

The view change

model_run_results selects an explicit column list and omits created_at, so consumers of the view have no prunable column at all: its other timestamps (generated_at, execute_completed_at) are stored as strings, so any predicate on them needs a cast, and a cast defeats pruning. Adding the column is a prerequisite for fixing the CLI's get_models_runs, which reads this view and currently scans the whole table on every report.

The view is rebuilt on every run and this only adds a column, so nothing downstream breaks.

Effect on existing installations

None automatically, and nothing breaks. dbt applies partitioning only when a table is created, so existing tables keep their current layout and their queries remain valid — they simply do not gain pruning until the table is rebuilt.

Worth being explicit that --full-refresh is not the way to get it: these models compile to an empty table and are populated by on-run-end hooks, so a full refresh would discard all history. The package already prevents this — elementary_full_refresh defaults to false — but anyone who has enabled that var should not reach for it here. The safe route is a one-off rebuild that preserves rows:

CREATE TABLE `<project>.<elementary_schema>.elementary_test_results__partitioned`
PARTITION BY timestamp_trunc(created_at, DAY)
AS SELECT * FROM `<project>.<elementary_schema>.elementary_test_results`;
-- verify row counts match, then drop the original and rename this into place

(BigQuery rejects CREATE OR REPLACE across a partitioning change, hence build-and-swap.)

Non-BigQuery adapters

No-op. default__get_partition_by returns none, and bigquery__get_partition_by also returns none when bigquery_disable_partitioning is set, so the existing opt-out is respected.

Testing

sqlfmt reports no changes on all three files. The partition column exists on both models — empty_table.sql declares created_at as a timestamp for each.

I have not run the full integration suite across all five platforms; these are config-only additions that are inert off BigQuery, and I am happy to run the BigQuery suite or make changes if you would like.

Summary by CodeRabbit

  • New Features
    • Improved partitioning configuration for test result data processing.
    • Added created_at timestamps to model run results, providing more complete run metadata.

…ed_at

Both tables are append-only and grow without bound, and both are read with a
`days_back` time filter, but neither declares `partition_by` — so on BigQuery
every read scans all history.

They are the two largest tables the package creates. On one warehouse
`test_result_rows` is 26M rows / 41GB and `elementary_test_results` is 12M rows
/ 13GB, against which a 7-day report query reads the whole table. With
`created_at` partitioning the same query prunes to the requested window.

Uses `get_partition_by()` with no argument, i.e. the package's existing
`created_at` default, matching how dbt_run_results and dbt_invocations already
declare it. `created_at` rather than `detected_at` because it is each table's
declared `meta.timestamp_column` and it is monotonic with insertion, so on an
append-only table writes stay in the newest partition.

No effect on non-BigQuery adapters: `default__get_partition_by` returns none.
Existing tables keep their current layout until a full refresh, since dbt only
applies partitioning at creation.
`model_run_results` selects an explicit column list from dbt_run_results and omits
`created_at`, so consumers of the view cannot filter on it.

That matters on BigQuery because `created_at` is dbt_run_results' partition
column, and the view's other timestamps cannot prune: `generated_at` and
`execute_completed_at` are stored as strings, so any predicate on them needs a
cast, and a cast defeats partition pruning. Without `created_at` a query over this
view has no prunable column at all and must scan all history.

The view is rebuilt on every run and this only adds a column, so nothing
downstream breaks.
@github-actions

Copy link
Copy Markdown
Contributor

👋 @tlangton3
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in the elementary repository.

@coderabbitai

coderabbitai Bot commented Sep 10, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: d0d4b022-f7cd-49df-b632-685fdac371e5

📥 Commits

Reviewing files that changed from the base of the PR and between 9a7e2b5 and 4f7a1ee.

📒 Files selected for processing (3)
  • models/edr/run_results/elementary_test_results.sql
  • models/edr/run_results/model_run_results.sql
  • models/edr/run_results/test_result_rows.sql

Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review.


📝 Walkthrough

Walkthrough

The run-results models now use shared partition configuration for two incremental models. The model_run_results view also exposes created_at.

Changes

Run-results model updates

Layer / File(s) Summary
Model configuration and view output
models/edr/run_results/elementary_test_results.sql, models/edr/run_results/test_result_rows.sql, models/edr/run_results/model_run_results.sql
The two incremental models now use elementary.get_partition_by(). The view now selects run_results.created_at.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested reviewers: haritamar

Merge Risk: ⚪ Minimal · up to 4f7a1

The PR adds BigQuery-aware partitioning to two run-results tables and exposes created_at for filtering. No concrete merge-blocking risk is identified.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the primary performance change: partitioning elementary_test_results and test_result_rows by created_at. It does not mention the related view change, but the title d…
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@tlangton3

Copy link
Copy Markdown
Contributor Author

Companion CLI change: elementary-data/elementary#2351. The view column added here is also what unblocks the get_models_runs fix, which is left out of that PR because it would have no prunable column until this merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant