Skip to content

Own SQL benchmark suite#8719

Open
myrrc wants to merge 1 commit into
developfrom
myrrc/sql-bench
Open

Own SQL benchmark suite#8719
myrrc wants to merge 1 commit into
developfrom
myrrc/sql-bench

Conversation

@myrrc

@myrrc myrrc commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Testing SQL queries' correctness is done in vortex-sqllogictest, but we also
want to test relative performance of queries not covered in existing benchmarks.
One example is aggregation with filter which should be much more performant
once we use zone maps statistics, but this type of query isn't present is
existing benchmarks.

This PR introduces a new query-per-file dataset and support for running it in CI

@myrrc myrrc requested a review from AdamGS July 10, 2026 15:55
@myrrc myrrc force-pushed the myrrc/sql-bench branch from c16907b to 728668e Compare July 10, 2026 16:14
@codspeed-hq

codspeed-hq Bot commented Jul 10, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 14.72%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

❌ 1 regressed benchmark
✅ 1697 untouched benchmarks
⏩ 9 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime cuda/bitpacked_u8/unpack/3bw[100M] 300 µs 351.8 µs -14.72%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing myrrc/sql-bench (2c9252a) with develop (6b27c45)

Open in CodSpeed

Footnotes

  1. 9 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Comment thread vortex-bench/.gitignore Outdated
Comment thread vortex-bench/src/vortex_queries.rs
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
@myrrc myrrc force-pushed the myrrc/sql-bench branch from 728668e to 2c9252a Compare July 15, 2026 15:25
@myrrc myrrc requested a review from AdamGS July 15, 2026 15:25
@myrrc myrrc enabled auto-merge (squash) July 15, 2026 15:25
@myrrc

myrrc commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

As this is fast we should probably run this on every commit @AdamGS WDYT?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants