fix(writer): nullable primitives keep their validity in the encoding, as Rust (#604) by dfa1 · Pull Request #609 · dfa1/vortex-java

dfa1 · 2026-10-25T19:11:09Z

Fixes #604. A nullable primitive column no longer goes through vortex.masked: as in Rust, its validity is a plain vortex.bool child of the encoding that stores the values, and encodings above that pass it down to their children. The cascade measures the bitmap the way Rust does, so it picks the same scheme as Rust, and the dict layout runs Rust's probe on nullable columns too.

Comparing nullable columns against vortex-jni turned up eight more parity bugs. Each one has its own commit:

Keep an encoding only when it is smaller than the raw array (choose_and_compress acceptance).
Cost a child encoding by what it encodes to, not by its estimate.
FoR bit-packs its residuals directly instead of cascading them.
Narrow run-end ends and RLE indices/offsets as Rust's narrow does.
Compress zone stats tables with the full compressor (integer dict included).
ALP cascades the encoded ints of F32 columns.
ALP-RD packs its parts at the dictionary's widths.
Try encodings in Rust's registration order, so a tie between equal scores goes Rust's way.

Most of these also change non-null columns.

Tests

NullableIntegerInteropIntegrationTest: 17 shapes. Each asserts the same encoding tree as vortex-jni (data and zone table), then that vortex-jni reads Java's file back (default and compact; full, filtered, row-range).
NullableFloatInteropIntegrationTest: 12 F32/F64 shapes, checked the same way.
DictAdmissionInteropIntegrationTest gains nullable cases.
Unit tests for the acceptance rule, the nullable cascade, null-aware stats, FoR, ALP and ALP-RD.
./mvnw verify is green locally.

Known gaps, not in this PR

Rust stores chunk_offsets for bit-packing patches and Java does not. This predates the PR, so the cross-checks leave out patched columns.
One registration-order case is left: for integers Rust tries sparse before dict, and for floats dict before sparse, and one list can't do both. This only matters on an exact tie in the probe and the zone tables.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x

Rust's choose_and_compress (vortex-compressor compressor/cascade.rs) compresses the winning scheme on the full array and keeps the result only when its byte size is below the array's exact uncompressed size; otherwise the array stays canonical. Java ran the winner and kept whatever came out. The two disagree whenever the sample favors a scheme the full array does not. Bit-packing pads to 1024-value blocks, so a 1024-row sample of 1500 15-bit u16 values packs to 1920 of 2048 bytes and wins, while the full array pads to 2048 values: 3840 bytes over its 3000. Rust stores those as vortex.primitive. Found while comparing nullable columns against vortex-jni (#604), where it decides a zone table's four null counts (FoR wins their estimate, its bit-packed residuals pad past the 32 raw bytes) and a mostly-null column's sparse patch indices. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x

Rust measures a sampled scheme by compressing the sample with it, children included (compress_child runs the full choose_and_compress on each child), so a child's cost is its winner's real output, and the child stays canonical when that is not smaller. Java picked the child's winner the same way, but when the winner won on a stats estimate it took the estimate as the cost. The estimate can be far off: a FoR ratio says nothing of the 1024-value block its bit-packing pads a short child to. On a 1024-row sample of runs of 20 wide values with null runs, FoR estimated 81 bytes for run-end's 65 ends, which encode to 1283 and are kept as 130 raw bytes. Run-end came out 49 bytes too cheap and beat RLE, which Rust picks (found comparing nullable columns against vortex-jni, #604). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x

Rust's v1 FoRScheme declares no children: it subtracts the minimum and bit-packs the residuals at once with BITPACKING_V1 ("if any other scheme was preferable, it would be chosen instead of bitpacking", vortex-btrblocks schemes/integer/for_.rs), keeping them a primitive when the best width is the native one (compress_global returns None). Java opened the residuals as a cascade child, so any scheme could win them. On a zone table's null counts [4681, 4681, 4681, 4682] the child became a vortex.dict, giving for(dict) where vortex-jni writes a vortex.primitive (FoR wins the estimate, its bit-packing pads past the 32 raw bytes, and the array stays canonical). Found comparing nullable columns against vortex-jni (#604). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x

…does Rust narrows these position arrays before compressing them (PrimitiveArray::narrow: the smallest unsigned type that holds the largest value): runend_encode narrows its u64 ends, and rle_compress its u16 indices and u64 offsets. Java always wrote u32 ends, u16 indices and u64 offsets. The width decides what the cascade does with them. A 32768-row column's ends fit u16, and bit-packing 15 of 16 bits pads past their raw size, so Rust keeps them a vortex.primitive where Java's u32 ends went through FoR. RLE indices within a chunk rarely pass 255, so Rust's u8 indices cost half of Java's u16, which tipped the choice between RLE and run-end (found comparing nullable columns against vortex-jni, #604). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x

Rust's file strategy builds two compressors from one builder: the data compressor excludes IntDictScheme (DictStrategy already dictionary-encodes suitable columns as a layout), while the stats compressor, which also compresses dict values, keeps every scheme (vortex-file strategy.rs). Java compressed stats tables with its data cascade codecs, so an integer dict never competed there. A zone table column with a repeated value differs: the per-zone null counts [1, 0, 0, 0] of a column with one null are a vortex.dict in vortex-jni's file (16 bytes of values and 4 of codes against 32 raw). The candidates are now the probe's codecs, which are the data cascade with the integer dict back in (found comparing nullable columns, #604). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x

) Java wrote a nullable integer column as vortex.masked(values, validity) and ran the cascade over the values alone. Rust has no such wrapper: every array carries its own validity, so the leaf that stores the values holds it as a plain vortex.bool child (vortex.primitive, fastlanes.bitpacked after any patches), and the encodings above pass it down: FoR and ZigZag to their output, run-end to its run values (a run of nulls is one invalid run), RLE to its indices padded to a whole chunk, dict to its codes, Pco by compressing only the valid values. Rust never compresses that bitmap. The wrapper changed more than the tree. Rust's sampled schemes are measured with the bitmap, its estimated ones are not, so the two compressors ranked schemes differently on nullable data, and the dict layout could not take Rust's probe. Now: - NullableData flows through CascadingCompressor for integer dtypes. Stats skip the nulls as Rust's do (computed over the valid values: same min, max, distinct count and runs, since a null continues the run before it), the baseline adds the packed bitmap, and only encoders that keep validity compete (acceptsNullable). - The compressor's own checks come first, as in choose_and_compress: all null is a null constant, valid rows holding one value a constant masked by a plain bitmap (Rust's compress_constant). A column without nulls stores no bitmap, as Rust imports an all-valid Arrow array. - Sparse follows Sparse::encode: past 90% null the fill is null and the valid rows are the patches (with Rust's all-valid bitmap on their values); otherwise nulls are patches. - Bit-packing counts a null at width 0 and patches only valid values (bit_width_histogram, gather_patches); FoR takes its reference from the valid values and zeroes null residuals. - The dict layout probe now runs for nullable integers too, as Rust's DictStrategy does. Floats keep vortex.masked until ALP and ALP-RD carry a validity (keepsValidity). NullableIntegerInteropIntegrationTest writes 17 column shapes through vortex-jni and vortex-java and asserts the same encoding tree, data and zone table, then reads Java's file (default and compact) through vortex-jni, full, filtered and row-range. Not covered: patched bit-packing, whose chunk offsets Rust stores and Java does not yet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x

Rust's ALPScheme compresses the encoded integers through compress_child for either float width (vortex-btrblocks schemes/float/alp.rs). Java's encodeCascade only opened that child for F64 and fell back to the terminal encode for F32, which writes the i32s as a raw vortex.primitive, so an F32 column ALP won never got them bit-packed. vortex-jni writes an F32 column of halves as alp(bitpacked), and one with one decimal as alp(bitpacked) with patches; Java wrote alp(primitive) for both. The F32 path now mirrors the F64 one: computeF32 shared by both entry points, the patches owned by the node and the encoded ints left open (found comparing nullable floats against vortex-jni, #604). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x

Rust's encode_generic (encodings/alp/src/alp_rd/mod.rs) packs the dictionary-coded left parts at the encoder's left_bit_width (the width of the largest code, at least 1 bit: bit_width(dict_len - 1) in the alp crate) and the right parts at right_bit_width, both with bitpack_encode_unchecked. Java ran them through the bit-packing encoder, whose histogram finds narrower widths when the values allow: on quarters the right parts are all zero and packed in 0 bits. That made ALP-RD measure smaller in Java than in Rust, so it won columns Rust gives to a dictionary: a float column with five distinct values, or taxi's Airport_fee shape, which vortex-jni writes with a dict layout (found comparing nullable floats, #604). The parts now go through BitpackedEncodingEncoder.encodeAtWidth, Rust's unchecked pack. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x

The integer change extended to every primitive: a nullable float column no longer goes through vortex.masked. As in Rust: - ALP encodes every slot, keeps the validity on the encoded ints and drops the exceptions at null rows from its patches ("we remove null exceptions in order to keep the Patches small", alp_encode_components_typed). Its terminal encode, used without a cascade, keeps the encoded ints a primitive with the validity. - ALP-RD splits every slot and keeps the validity on its bit-packed left parts (encode_generic). - Sparse takes a nullable float only when it is more than 90% null (Rust's NullDominatedSparseScheme), and compresses only its indices: the patch values stay as they are, with the all-valid bitmap Rust's filter leaves on them. - The dict layout probe now runs for every primitive. NullableFloatInteropIntegrationTest writes 11 F32/F64 shapes through vortex-jni and vortex-java and asserts the same encoding tree, then reads Java's file (default and compact) through vortex-jni, full, filtered and row-range. EncodingTrees is the tree walk both nullable cross-checks share. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x

Rust breaks a tie between two equal scores by registration order within each pass (choose_best_scheme), and registers integers as FoR, ZigZag, BitPacking, Sparse, IntDict, RunEnd, Sequence, RLE, Delta and floats as ALP, ALP-RD, FloatDict, sparse, RLE (vortex-btrblocks session.rs). Java's list had RLE and run-end ahead of dict and bit-packing last, so a tie went the other way. Ties are rare but real: on a float column of runs with null runs, RLE and dict both measure 644 bytes on the dict layout probe's sample. Rust takes the dict and writes a dict layout; Java took RLE and wrote none (found comparing nullable floats, #604). One list serves both families, so Dict sits before Sparse as floats have them; for integers the two only compete in the probe and zone tables, where a tie of their two estimates is the one case still broken the other way. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x

dfa1 and others added 10 commits October 25, 2026 19:42

Navigation Menu

Repository navigation

fix(writer): nullable primitives keep their validity in the encoding, as Rust (#604) - #609

fix(writer): nullable primitives keep their validity in the encoding, as Rust (#604)#609
dfa1 wants to merge 10 commits into
mainfrom
fix/604-nullable-validity

dfa1 commented Oct 25, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

dfa1 commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tests

Known gaps, not in this PR

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dfa1 commented Oct 25, 2026 •

edited

Loading