Skip to content

[python] Avoid full-window conversion for first and last aggregates - #10016

Open
QuakeWang wants to merge 1 commit into
apache:masterfrom
QuakeWang:window-endpoints
Open

QuakeWang wants to merge 1 commit into
apache:masterfrom
QuakeWang:window-endpoints

Conversation

@QuakeWang

Copy link
Copy Markdown
Member

Purpose

Window first and last aggregates converted the entire window to Python objects before selecting one non-null value.

Scan Arrow chunks from the requested endpoint and convert only the selected value, preserving null and value semantics.

Tests

  • Temporal suite: 76 passed, including chunk, null, nested-value, dictionary, and memory regressions.
  • The memory regression fails on the previous implementation.
  • Local Python allocation peak for 8,192 binary values of 1 KiB each: 8.38 MiB to about 1.6 KiB.
  • Flake8, license headers, Python 3.6 syntax, and git diff --check passed.

Scan Arrow chunks from the requested endpoint and convert only the selected
value instead of materializing the full window as Python objects. Preserve
null skipping, chunk boundaries, and nested and dictionary value semantics.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
@JingsongLi JingsongLi closed this Sep 20, 2026
@JingsongLi JingsongLi reopened this Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants