Repository navigation
perf(spark): read Utf8View values in place on the FileFormat connector - #33
Merged
Merged
Conversation
… rest of the field
A nullable struct expands a field access `$.x` to `mask($.x, is_not_null($))`. For a
path below the field, such as `$.x.q`, the mask then sits between the path and the
field, so the partitioner hands the field's reader a bare `$` and the whole of `x`
is read. Mask the whole path instead.
On a file with `t: {x: {p, q}, r}`, both nullable, reading `$.t.x.q` or filtering on
`$.t.x.q > n` read 6.0 MB before and 160 KB after.
Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Timestamp scalars were validated against `jiff::Timestamp`'s range, which stops 26 hours short of the end of year 9999, so a value Arrow, Parquet and Spark allow, such as `9999-12-31T00:00:00Z`, could neither be used as a literal nor written: the file statistics built the min/max scalar with `Scalar::extension_ref` and panicked. Every `i64` count of the unit is a valid timestamp; jiff only bounds what `Display` can render, so drop the range check, fall back to the raw count when the calendar rendering is unavailable, and build the min/max scalars fallibly so a rejected extreme is an error rather than a panic. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Now that timestamps beyond jiff's range can be written, `split_temporal` could see second and millisecond counts more than i32::MAX days from the epoch. `split_with_divisor` truncated the days to i32 with `as_()`, so such values silently wrapped when compressed: `i64::MAX` seconds round-tripped as 50027779063807. The compress comment still claimed jiff validation made the narrowing safe, the btrblocks TemporalScheme applied the encoding with no range check, and `try_extract_days_constant` used the same truncating split, so `dtp <= lit` with such a literal was rewritten to a wrapped days constant. Microsecond and nanosecond counts cannot overflow the days part. `timestamp::split` now errors when the days do not fit, so the days pushdown declines the constant and the compare kernel falls back to the canonical comparison. `split_temporal` checks the valid min/max of second and millisecond arrays and errors instead of wrapping, and the new `can_split_temporal` lets the btrblocks TemporalScheme skip such arrays so the storage array is compressed and the data round-trips exactly. The narrowing compare test built its fixture from `i64::MAX` seconds and relied on the wrap; it now uses a value above i32::MAX whose days fit. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
…hold the literal Compression narrows the days part, for example to u16 for dates before 2149. A midnight literal past that range, such as 9999-12-31, made the comparison push-down cast its days to the narrowed type and fail with "Cannot cast i32 to u16". The rule now declines, and the comparison runs on the full timestamps. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
build() closed the exported ArrowSchema without calling its release callback. Native code only borrows the schema, so the child structs the export allocated were never freed, and every writer's allocator closed with outstanding memory. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Spark writes through a Java stream, so native workers run Java upcalls and attach themselves permanently. They never exit, and as non-daemon threads they kept the JVM alive after main returned. Attach them as daemons first, and detach them when the thread exits, since the jni crate only detaches attachments it made itself. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Vortex exports every string column to Arrow as Utf8View, so StringViewAccessor serves every string column Spark reads, and it copied each value into a new byte[]. Decode the 16-byte view and return UTF8String.fromAddress over vector memory, as StringAccessor already does for Utf8. Arrow 19's getDataPointer ignores the view's offset, so the view is decoded here. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
gitRasheed
force-pushed
the
rasheed/ff-string-views
branch
from
October 29, 2026 21:44
2d798c9 to
f0945f7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports #6 onto vortex-data#9658's connector. Part of #31. Stacked on #35, the vortex-data#9658 integration branch.
vortex-data#9658 moves
VortexArrowColumnVectorunchanged, soStringViewAccessorcopies everyUtf8Viewvalue into a newbyte[]again. The evaluation recorded 141byte[]allocation samples under it in a string-heavy scan.Changes
StringViewAccessor.getUTF8Stringdecodes the 16-byte view and returnsUTF8String.fromAddressover vector memory, asStringAccessoralready does forUtf8. Values of up to 12 bytes are read from the view, longer ones from the data buffer and offset the view names. A value is valid until its batch is released, like the values of Spark's own column vectors.Testing
VortexArrowColumnVectorTest.stringViewValuescovers inline, out-of-line, empty, multi-byte and null values across two data buffers, one at a nonzero offset. It fails if values are copies or the decoder ignores the offset.Checked locally:
:vortex-spark-4.0_2.13:test.I used AI tools to help write and review this change.