Skip to content

Add PiecewiseSequence index array#8799

Draft
danking wants to merge 6 commits into
developfrom
codex/piecewise-sequential-array
Draft

Add PiecewiseSequence index array#8799
danking wants to merge 6 commits into
developfrom
codex/piecewise-sequential-array

Conversation

@danking

@danking danking commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds PiecewiseSequenceArray, a serializable index encoding for piecewise arithmetic sequences: starts[i] + j * multipliers[i] for j in 0..lengths[i].

This PR only introduces the index array and its execution/scalar behavior. It intentionally does not specialize take; dependent PRs consume this encoding for the constant-multiplier-1 contiguous case.

Validation

  • cargo fmt -p vortex-array
  • cargo test -p vortex-array piecewise_sequence

@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 2b6fa3e 1 Explore Profiling Data
Previous Runs (7)
Status Commit Job Attempt Link
🟢 Done 52affcd 1 Explore Profiling Data
🟢 Done 16ca5dc 2 Explore Profiling Data
🟢 Done 6eb0c92 1 Explore Profiling Data
🟢 Done ba7ef2e 1 Explore Profiling Data
🟢 Done 1748e31 1 Explore Profiling Data
🟢 Done 3edf6c8 1 Explore Profiling Data
🟢 Done aba7e4d 1 Explore Profiling Data

Powered by Polar Signals Cloud

@codspeed-hq

codspeed-hq Bot commented Jul 16, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 11.17%

⚡ 1 improved benchmark
✅ 1669 untouched benchmarks
⏩ 52 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation true_count_vortex_buffer[128] 580.6 ns 522.2 ns +11.17%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing codex/piecewise-sequential-array (2b6fa3e) with develop (3975895)

Open in CodSpeed

Footnotes

  1. 52 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +5.3%
Engines: DataFusion No clear signal (+6.1%, low confidence) · DuckDB No clear signal (+4.6%, low confidence)
Vortex (geomean): 1.047x ➖
Parquet (geomean): 1.020x ➖
Shifts: Parquet (control) +2.0% · Median polish +4.5%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (1.080x ➖, 0↑ 0↓)
name PR 2b6fa3e (ns) base 3975895 (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 10021365 9303558 1.08
vortex_q01/datafusion:vortex-file-compressed 6627376 6119813 1.08
datafusion / parquet (1.018x ➖, 0↑ 0↓)
name PR 2b6fa3e (ns) base 3975895 (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 20583727 20667041 1.00
vortex_q01/datafusion:parquet 4789370 4604281 1.04
duckdb / vortex-file-compressed (1.069x ➖, 0↑ 0↓)
name PR 2b6fa3e (ns) base 3975895 (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 10436535 9980727 1.05
vortex_q01/duckdb:vortex-file-compressed 6503703 5950906 1.09
duckdb / parquet (1.022x ➖, 0↑ 0↓)
name PR 2b6fa3e (ns) base 3975895 (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 23652549 23222387 1.02
vortex_q01/duckdb:parquet 9552799 9308681 1.03

No file size changes detected.

@danking
danking force-pushed the codex/piecewise-sequential-array branch 2 times, most recently from 3edf6c8 to 1748e31 Compare July 16, 2026 18:20
@danking danking changed the title Add PiecewiseSequential index array Add PiecewiseSequence index array Jul 16, 2026
ctx: &mut ExecutionCtx,
) -> VortexResult<(PrimitiveArray, PrimitiveArray, PrimitiveArray)> {
let starts = array.starts().clone().execute::<PrimitiveArray>(ctx)?;
let lengths = array.lengths().clone().execute::<PrimitiveArray>(ctx)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think while starts could be primitive we specifically want to handle cases where this is not primitive?

danking added 4 commits July 16, 2026 19:36
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
danking force-pushed the codex/piecewise-sequential-array branch from 6eb0c92 to 16ca5dc Compare July 16, 2026 23:36
@danking danking added the changelog/feature A new feature label Jul 16, 2026
danking added 2 commits July 16, 2026 20:13
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants