Repository navigation
perf(layout): read a path below a nullable struct's field without the rest of the field - #10952
Open
gitRasheed wants to merge 1 commit into
Open
gitRasheed wants to merge 1 commit into
gitRasheed wants to merge 1 commit into
Conversation
… rest of the field
A nullable struct expands a field access `$.x` to `mask($.x, is_not_null($))`. For a
path below the field, such as `$.x.q`, the mask then sits between the path and the
field, so the partitioner hands the field's reader a bare `$` and the whole of `x`
is read. Mask the whole path instead.
On a file with `t: {x: {p, q}, r}`, both nullable, reading `$.t.x.q` or filtering on
`$.t.x.q > n` read 6.0 MB before and 160 KB after.
Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reading a field two levels into nullable structs read the whole inner struct. For
t: {x: {p, q}, r}withtandxnullable, projecting$.t.x.qor filtering on$.t.x.q > nread 6.05 MB of a 23.7 MB file, as much as reading all ofx. With this change it reads 160 KB. One level,$.s.b, was already pruned.expand_struct_rootrewrites a field access$.xon a nullable struct tomask($.x, is_not_null($)), so the field keeps the struct's nulls. For$.x.qthat givesget_item(q, mask($.x, ...)), and the mask hides the path from the partitioner:x's reader gets$and reads every field. A Spark connector that pushes nested schema pruning down runs into this for any struct inside a nullable struct, since Spark structs are nearly always nullable.Changes
A
get_item/selectchain that starts at$.fieldis rewritten as a whole tomask(chain_on_field, is_not_null($)), so the field's reader gets the path and reads only what it needs.test_path_below_nullable_struct_fieldreads a leaf, aselect,is_not_nullof a leaf and a predicate through two nullable structs. It checks the results against in-memory evaluation and that no segment of the other fields is requested. Every case fails without the change.Checked locally:
cargo test -p vortex-layout --lib(696 passed), clippy with-D warnings, nightly fmt, and the byte counts above from a throwaway scan through a counting reader.I used AI tools to help write and review this change.