perf(solana): convert solana_utils_epochs to incremental (CUR2-2823)#9878
perf(solana): convert solana_utils_epochs to incremental (CUR2-2823)#98780xRobin wants to merge 1 commit into
Conversation
Full-rebuild table -> merge-incremental keyed on block_slot with a ~2-epoch (~5-day) rolling window. Source-side filter limits to the last 7 days and a second CTE trims to the two newest epochs so the base window functions and self-joins stay correct. Merge predicate scopes target rows to the same 7-day tail. Adds not_null on block_slot and a model-level dbt_utils.unique_combination_of_columns as required for incremental spells. CUR2-2823
PR SummaryMedium Risk Overview Incremental runs read only recent Schema tests add Reviewed by Cursor Bugbot for commit a10fdc7. Configure here. |
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit a10fdc7. Configure here.
Convert
solana_utils.epochsfrommaterialized='table'tomaterialized='incremental'(merge onblock_slot) with a bounded ~2-epoch rolling window. Prod runs full-rebuild ~36× / day (~1.02 CPU-hrs/day, ~0.6 GB IO/run scanning all ofsolana.blocks); the incremental path only re-emits the last two epochs so an in-progress epoch'sepoch_next_start_slotstill hydrates onceepoch+1starts.not_nullonblock_slotand a model-leveldbt_utils.unique_combination_of_columnsare added per repo policy for incremental spells.Correctness
Two CTEs make the window safe:
base_rawfilterssolana.blockstotime >= now() - interval '7' day(generous cover for two Solana epochs at ~2 days each).base_filteredtrims toepoch >= max(epoch) - 1in the incremental branch so partially-observed older epochs don't corruptfirst_block_epoch/ self-join outputs.incremental_predicatesscopes the merge target toblock_time >= now() - interval '7' day, matching the source window.Equivalence check
Ran the incremental SQL live against
solana.blocksand compared to prodsolana_utils.epochsfor the last-two-epochs overlap window (epochs 998–999, slots 431,136,000–431,655,830):Every prod row in the last-2-epochs window is present with an identical row hash in the new output. The 7,759-row surplus in new is entirely explained by prod-side refresh lag (prod
max_slot431,647,658 < newmax_slot431,655,830, gap ≈ 8k slots) — that data will land on the next scheduled prod refresh.Linear: CUR2-2823