Skip to content

perf(solana): convert solana_utils_epochs to incremental (CUR2-2823)#9878

Draft
0xRobin wants to merge 1 commit into
mainfrom
robin/cur2-2823-perfsolana_utils_epochs-full-rebuild-scans-all-solanablocks
Draft

perf(solana): convert solana_utils_epochs to incremental (CUR2-2823)#9878
0xRobin wants to merge 1 commit into
mainfrom
robin/cur2-2823-perfsolana_utils_epochs-full-rebuild-scans-all-solanablocks

Conversation

@0xRobin

@0xRobin 0xRobin commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Convert solana_utils.epochs from materialized='table' to materialized='incremental' (merge on block_slot) with a bounded ~2-epoch rolling window. Prod runs full-rebuild ~36× / day (~1.02 CPU-hrs/day, ~0.6 GB IO/run scanning all of solana.blocks); the incremental path only re-emits the last two epochs so an in-progress epoch's epoch_next_start_slot still hydrates once epoch+1 starts. not_null on block_slot and a model-level dbt_utils.unique_combination_of_columns are added per repo policy for incremental spells.

Correctness

Two CTEs make the window safe:

  • base_raw filters solana.blocks to time >= now() - interval '7' day (generous cover for two Solana epochs at ~2 days each).
  • base_filtered trims to epoch >= max(epoch) - 1 in the incremental branch so partially-observed older epochs don't corrupt first_block_epoch / self-join outputs.

incremental_predicates scopes the merge target to block_time >= now() - interval '7' day, matching the source window.

Equivalence check

Ran the incremental SQL live against solana.blocks and compared to prod solana_utils.epochs for the last-two-epochs overlap window (epochs 998–999, slots 431,136,000–431,655,830):

metric value
new rows (last 2 epochs) 517,799
prod rows in same slot range 510,040
rows in prod not in new (row hash) 0
rows in new within [new_min_slot, prod_max_slot] 510,040
prod rows in same bounded range 510,040

Every prod row in the last-2-epochs window is present with an identical row hash in the new output. The 7,759-row surplus in new is entirely explained by prod-side refresh lag (prod max_slot 431,647,658 < new max_slot 431,655,830, gap ≈ 8k slots) — that data will land on the next scheduled prod refresh.

Linear: CUR2-2823

Full-rebuild table -> merge-incremental keyed on block_slot with a
~2-epoch (~5-day) rolling window. Source-side filter limits to the last
7 days and a second CTE trims to the two newest epochs so the base
window functions and self-joins stay correct. Merge predicate scopes
target rows to the same 7-day tail. Adds not_null on block_slot and a
model-level dbt_utils.unique_combination_of_columns as required for
incremental spells.

CUR2-2823
@cursor

cursor Bot commented Jul 8, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Incremental windowing and epoch self-join logic must stay correct or epoch boundary columns could drift for downstream consumers; scope is limited to one utility model with merge predicates and tests.

Overview
solana_utils_epochs switches from a full table rebuild to incremental merge on block_slot, with merge scoped to block_time >= now() - 7 days.

Incremental runs read only recent solana.blocks (7-day tail) and then keep the last two epochs so first_block_epoch and the epoch boundary self-joins stay correct and epoch_next_start_slot can update as the next epoch starts. Row numbering moves into a base CTE after that filter.

Schema tests add not_null on block_slot and a model-level unique_combination_of_columns on block_slot for incremental spell policy.

Reviewed by Cursor Bugbot for commit a10fdc7. Configure here.

@github-actions github-actions Bot added WIP work in progress dbt: solana covers the Solana dbt subproject labels Jul 8, 2026
@0xRobin

0xRobin commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit a10fdc7. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dbt: solana covers the Solana dbt subproject WIP work in progress

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant