FileSystem for local reads - #9773
Conversation
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | mul_u32_nonnull_avx512 |
5.6 µs | 6.2 µs | -10.16% |
| ⚡ | WallTime | arrow_checked_add_u32_avx2[16384] |
21.4 µs | 17.6 µs | +21.31% |
| ⚡ | WallTime | arrow_checked_add_u32_avx512[16384] |
21.3 µs | 17.6 µs | +21.17% |
| ⚡ | WallTime | words_gather_scalar_avx2[65536] |
9.4 µs | 8.2 µs | +13.78% |
| ⚡ | Simulation | allocate_drop_arrow[0] |
456.9 ns | 402.7 ns | +13.45% |
| ⚡ | Simulation | allocate_drop_bytes[0] |
520.2 ns | 466 ns | +11.62% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing myrrc/local-fs-reads (ecbd0d3) with develop (a997c3f)
Footnotes
-
206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 1↑ 2↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.977x ➖, 2↑ 3↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.943x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
File Size Changes (100 files changed, +0.0% overall, 57↑ 43↓)
Totals:
|
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.035x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.936x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 1↑ 2↓)
duckdb / parquet / ns (1.013x ➖, 0↑ 3↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.994x ➖, 2↑ 3↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.991x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.970x ➖, 1↑ 3↓)
datafusion / parquet / ns (0.930x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.932x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.941x ➖, 0↑ 0↓)
|
| if url.scheme() == "file" { | ||
| return Ok(( | ||
| Arc::new(ObjectStoreFileSystem::local(RUNTIME.handle())), | ||
| Arc::new(StdFileSystem::new(RUNTIME.handle())), |
There was a problem hiding this comment.
If you want this can you make sure this is used everywhere instead of ObjectStoreFileSystem::local
We have vortex-cloud and ObjectStoreFileSystem which is a unified interface for
reads. However, it does many things which are needed for an object store but
excessive for local disk i.e. etag creation.
As another example, get_opts currently calls open() for every segment which we
also don't need because a local read may use the cached file descriptor.
This change implements a local FileSystem and adds it support to vortex-duckdb.