cache DTypes parsed from flatbuffers - #9765
Conversation
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
002b353 to
67e168c
Compare
|
Maybe FieldDType should have interior mutability and if we ever are forced to parse the view we mutate internal state and replace the view with the owned version? |
|
I've considered this option but it doesn't look neither less ugly nor more performant. |
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | random_i16[0.95] |
77.2 µs | 95.9 µs | -19.55% |
| ⚡ | WallTime | arrow_checked_add_u32_neon[16384] |
20.4 µs | 12.8 µs | +59.92% |
| ⚡ | Simulation | random_i16[0.8] |
92.7 µs | 75.3 µs | +22.98% |
| ⚡ | Simulation | allocate_drop_arrow[0] |
456.9 ns | 402.7 ns | +13.45% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing myrrc/cache-parsed-dtype-fields (67e168c) with develop (3170657)
Footnotes
-
206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
|
Even if we have interior mutability, the issue is value() + cloning the field. If they refer to a flatbuffer view, then we need to re-parse the copied view, so we need an Arc to prevent that. |
robert3005
left a comment
There was a problem hiding this comment.
I wish there was a nicer way but I can't think of one. You could come up with a way to apply expressions to dtypes but that still has to be progressively refined
In clickbench queries, duckdb_reader_open accounts for 13% CPU time of the scan.
This is because for every file's field for every file Vortex parses DType from
a flatbuffer.
This change adds a cache of DTypes parsed from flatbuffers so among multiple
query iterations parsing is done exactly once per field