Push ScalarFn after Filter/Slice(x) if ScalarFn(x) has a registered kernel - #9766
Push ScalarFn after Filter/Slice(x) if ScalarFn(x) has a registered kernel#9766myrrc wants to merge 6 commits into
Conversation
e4023c5 to
5a0e7a5
Compare
Merging this PR will improve performance by 26.51%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | arrow_checked_add_u32_neon[16384] |
20.5 µs | 12.7 µs | +61.21% |
| ⚡ | Simulation | decompress[u8, (4000, 1024)] |
45.4 µs | 33.4 µs | +35.65% |
| ⚡ | WallTime | arrow_checked_add_u32_avx2[16384] |
21.4 µs | 17.7 µs | +21.2% |
| ⚡ | Simulation | decompress[u64, (4000, 1024)] |
85.5 µs | 70.6 µs | +21.04% |
| ⚡ | Simulation | decompress[u16, (4000, 256)] |
58.5 µs | 50.9 µs | +14.94% |
| ⚡ | WallTime | filtered_sink_i64_neon[NineNullsInTen] |
28.5 µs | 25.6 µs | +11.17% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing myrrc/push-binary-into-filter (50bc21e) with develop (a997c3f)
Footnotes
-
1990 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
|
this should be a reduce rule, we are just pushing things through |
|
Reduce rules don't have access to "ctx". |
|
Yes, so this is partially missing partially different. ExecutionCtx is for EXECUTION. Reduce rules need same reduce rule registry like we have for execute rules. This essentially hasn't been completely done yet but it's a last step in the whole optimiser migration. I think the reduce rules should get access to session, this then means you have to pass session EVERYWHERE because something like |
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.013x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: String Encoding 📖Commits: PR vortex / vortex-file-compressed / ms (0.996x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / % (1.000x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.983x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.940x ➖, 2↑ 1↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.980x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (1.032x ➖, 0↑ 4↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.896x ✅, 6↑ 0↓)
datafusion / vortex-compact / ns (0.990x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.949x ➖, 3↑ 1↓)
duckdb / vortex-compact / ns (1.031x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
File Size Changes (200 files changed, +0.1% overall, 109↑ 91↓)
Totals:
|
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.927x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (0.974x ➖, 1↑ 2↓)
datafusion / parquet / ns (0.980x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.937x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.028x ➖, 0↑ 0↓)
|
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.993x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.011x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.989x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 2↓)
datafusion / vortex-compact / ns (0.995x ➖, 1↑ 1↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 2↑ 1↓)
duckdb / vortex-compact / ns (1.026x ➖, 2↑ 5↓)
duckdb / parquet / ns (0.983x ➖, 7↑ 3↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 2↑ 1↓)
duckdb / vortex-compact / ns (1.007x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 2↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 2↓)
duckdb / vortex-compact / ns (0.991x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.037x ➖, 1↑ 4↓)
datafusion / vortex-compact / ns (1.026x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.055x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (1.060x ➖, 1↑ 2↓)
duckdb / vortex-compact / ns (1.083x ➖, 0↑ 3↓)
duckdb / parquet / ns (0.985x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.056x ➖, 2↑ 3↓)
datafusion / parquet / ns (0.966x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (1.020x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.942x ➖, 1↑ 0↓)
|
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (0.937x ➖, 2↑ 0↓)
random-access / vortex-file-compressed / ns (0.975x ➖, 0↑ 0↓)
random-access / parquet / ns (1.001x ➖, 0↑ 0↓)
random-access / lance / ns (0.985x ➖, 0↑ 0↓)
|
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.988x ➖, 1↑ 1↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.992x ➖, 1↑ 2↓)
vortex / parquet / ns (0.999x ➖, 1↑ 1↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.973x ➖, 2↑ 0↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
See clickbench q1:
SELECT * FROM hits WHERE ID <> 0.ID <> 0is a ScalarFn(Binary) over Sparse.
(Binary, Sparse)pair has a kernel so it should operate oncompressed data. However, FlatReader inserts a Filter after array so we get
ScalarFn(Filter(Sparse)).(Filter, Sparse)canonicalizes the data.This change implements an optimize rule that turns
Fn(Filter(X), consts)into
Filter(Fn(X, consts))if(Fn, X)has a registered kernel which agreesto apply. If it's not the case, reordering may produce more rows on which Fn
will operate so we can't always apply this optimization.
Same logic applies to
Slicebut with one exception: Slice benefits from early application because if you move it up, you'll have to operate on a full ArrayRef which is large. So the second optimizer rule for Slice executes it eagerly.