Skip to content

fix: sliding window min() returns wrong value for all-NULL windows#23874

Open
neilconway wants to merge 3 commits into
apache:mainfrom
neilconway:neilc/fix-sliding-min-empty-window
Open

fix: sliding window min() returns wrong value for all-NULL windows#23874
neilconway wants to merge 3 commits into
apache:mainfrom
neilconway:neilc/fix-sliding-min-empty-window

Conversation

@neilconway

@neilconway neilconway commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

SlidingMinAccumulator::update_batch skipped NULL values. This meant that if all the non-NULL values in a window frame were retracted, the window frame would not be empty but the MovingMin data structure by the SlidingMinAccumulator would not contain any values. This resulted in incorrectly returning a stale non-NULL value for a sliding min() over a window frame consisting of only NULL values.

Along the way, optimize the min and max sliding window accumulators to make them both more efficient and more symmetric with one another. In the original coding, SlidingMaxAccumulator included NULL values but SlidingMinAccumulator omitted them, in part because omitting NULLs made the original retract_batch implementation more expensive. This PR optimizes retract_batch, so we can now use the same scheme for both the min and max sliding accumulators:

  • Omit NULLs on update_batch (this improves on the prior behavior of max)
  • Efficiently account for NULLs in retract_batch (this improves on the prior behavior of min)
  • Ensure correct results for all-NULL window frames (this fixes the prior bug in min).

What changes are included in this PR?

  • Fix bug in min() over all-NULL window frames
  • Optimize SlidingMinAccumulator::retract_batch (avoid materializing values just to count NULLs)
  • Optimize SlidingMinAccumulator::update_batch (omit NULLs), also improving symmetry with min
  • Optimize both accumulators to stop caching the current min / max; this saves a few clones, but perhaps more importantly it is simpler and avoids the risk of inconsistency between the cached value and the underlying MovingMin / MovingMax data structure

Are these changes tested?

Yes, new tests added.

Are there any user-facing changes?

No, aside from the bug fix.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Jul 24, 2026
@neilconway neilconway changed the title fix: sliding window MIN returns wrong value for all-NULL windows fix: sliding window min() returns wrong value for all-NULL windows Jul 24, 2026
The sliding accumulators cached the current min/max in a ScalarValue
field, refreshed after every update_batch and retract_batch. The cache
cost up to two extra ScalarValue clones per output row and was the root
cause of the stale-value bug fixed in the previous commit: a cached
value can go stale, a derived one cannot.

Replace the cache with an immutable precomputed typed NULL that fixes
the accumulator's data type and serves as the result for frames with no
non-null values; evaluate() and state() now read the current extremum
directly from the window structure.
@codecov-commenter

codecov-commenter commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 59.32203% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.69%. Comparing base (16471ee) to head (0dd135d).

Files with missing lines Patch % Lines
datafusion/functions-aggregate/src/min_max.rs 59.32% 4 Missing and 20 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23874      +/-   ##
==========================================
- Coverage   80.69%   80.69%   -0.01%     
==========================================
  Files        1090     1090              
  Lines      370357   370389      +32     
  Branches   370357   370389      +32     
==========================================
+ Hits       298859   298873      +14     
  Misses      53684    53684              
- Partials    17814    17832      +18     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sliding window min() returns stale value when all non-NULL values leave the frame

2 participants