Skip to content

feat: Calculate non-distinct sum from column statistics when available#23863

Open
AdamGS wants to merge 1 commit into
apache:mainfrom
AdamGS:adamg/sum-from-stat
Open

feat: Calculate non-distinct sum from column statistics when available#23863
AdamGS wants to merge 1 commit into
apache:mainfrom
AdamGS:adamg/sum-from-stat

Conversation

@AdamGS

@AdamGS AdamGS commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Seems like a valuable optimization for sources that report the sum statistic.

What changes are included in this PR?

Implements Sum::value_from_stats, supporting direct column expressions and cast(col) expressions introduced by type coercion.

Casts direct-column statistics to the aggregate return type so decimal sums use the widened SUM type.

Are these changes tested?

Unit coverage for statistics-backed SUM values and decimal accumulator widening, plus core integration coverage for integer and decimal sums from statistics.

Are there any user-facing changes?

None

@AdamGS AdamGS changed the title feature: Calculate non-distinct sum from column statistics when available feat: Calculate non-distinct sum from column statistics when available Jul 24, 2026
@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates core Core DataFusion crate common Related to common crate functions Changes to functions implementation labels Jul 24, 2026
@AdamGS
AdamGS marked this pull request as draft July 24, 2026 13:25
@github-actions github-actions Bot removed physical-expr Changes to the physical-expr crates common Related to common crate labels Jul 24, 2026
Signed-off-by: Adam Gutglick <adamgsal@gmail.com>
@AdamGS
AdamGS force-pushed the adamg/sum-from-stat branch from 5ea45cf to 28411d6 Compare July 24, 2026 13:41
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.94118% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.71%. Comparing base (18b1e35) to head (28411d6).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/functions-aggregate/src/sum.rs 92.94% 2 Missing and 4 partials ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #23863    +/-   ##
========================================
  Coverage   80.70%   80.71%            
========================================
  Files        1090     1090            
  Lines      370150   370424   +274     
  Branches   370150   370424   +274     
========================================
+ Hits       298746   298998   +252     
- Misses      53598    53611    +13     
- Partials    17806    17815     +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@AdamGS
AdamGS marked this pull request as ready for review July 24, 2026 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate functions Changes to functions implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sum aggregate fn can't evaluate value from statistics

2 participants