Skip to content

refactor(line): replace the nth kernel and grouped fold with 1 gather expr - #46

Merged
jvdd merged 3 commits into
mainfrom
refactor/nth-gather-expr
Sep 10, 2026
Merged

refactor(line): replace the nth kernel and grouped fold with 1 gather expr#46
jvdd merged 3 commits into
mainfrom
refactor/nth-gather-expr

Conversation

@jvdd

@jvdd jvdd commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

nth downsampling is now a single Polars gather (_nth_agg_expr) whose positions come from the column's own len(), so the same expression serves the resident select, the grouped group_by on both source kinds, and the geo line. It returns exactly what the Rust kernel returned: stride max(1, len // n_points), at most n_points rows, every row when short.

Removed:

  • the every_nth Rust kernel, its Python binding, namespace method and tests
  • grouped_nth_plan, the collect_batches fold and the _NTH_* constants
  • _geo_line_nth_agg_expr and geo_line's plugin import

Kept nth_plan for the ungrouped scan: its two streaming passes stay flat in rows and its frame-level viewport filter prunes row groups, which the expression cannot do.

A grouped nth on a scan now gathers in memory, because Polars 1.44 streams no non-reduction aggregation. test_ooc.py marks line-grouped-nth a strict xfail. Zoomed, its memory follows the visible window and it is 2.7-3.4x faster than the fold; unzoomed it is O(rows), which was accepted.

The fold was also the site of two Polars hazards found on 16- and 32-thread machines: the streaming engine runs ahead of a slow batch consumer by about 25 MB per thread, and a sparse filter at low thread counts can return a zero-copy view that pins whole batches.

BREAKING CHANGE: pl.Expr.flexviz.every_nth no longer exists.

…e gather expression

`nth` downsampling is now a single Polars `gather` (`_nth_agg_expr`) whose
positions come from the column's own `len()`, so the same expression serves
the resident select, the grouped `group_by` on both source kinds, and the
geo line. It returns exactly what the Rust kernel returned: stride
`max(1, len // n_points)`, at most `n_points` rows, every row when short.
Removed:
- the `every_nth` Rust kernel, its Python binding, namespace method and tests
- `grouped_nth_plan`, the `collect_batches` fold and the `_NTH_*` constants
- `_geo_line_nth_agg_expr` and geo_line's plugin import
Kept `nth_plan` for the ungrouped scan: its two streaming passes stay flat in
rows and its frame-level viewport filter prunes row groups, which the
expression cannot do.
A grouped `nth` on a scan now gathers in memory, because Polars 1.44 streams
no non-reduction aggregation. `test_ooc.py` marks `line-grouped-nth` a strict
xfail. Zoomed, its memory follows the visible window and it is 2.7-3.4x
faster than the fold; unzoomed it is O(rows), which was accepted.
The fold was also the site of two Polars hazards found on 16- and 32-thread
machines: the streaming engine runs ahead of a slow batch consumer by about
25 MB per thread, and a sparse `filter` at low thread counts can return a
zero-copy view that pins whole batches.
BREAKING CHANGE: `pl.Expr.flexviz.every_nth` no longer exists.
@jvdd
jvdd merged commit 5c83bc1 into main Sep 10, 2026
6 checks passed
@jvdd
jvdd deleted the refactor/nth-gather-expr branch September 10, 2026 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant