fix(writer): nullable primitives keep their validity in the encoding, as Rust (#604)

CHANGELOG.md

@@ -24,6 +24,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Breaking:** CSV import stores an empty cell in a number or boolean column as null instead of `0` or `false`, so inferred number and boolean columns are now nullable (read through `MaskedArray`), and an empty cell or a missing column fails the import where the schema declares the column non-nullable. ([#592](https://github.com/dfa1/vortex-java/issues/592))
 
 ### Fixed
+- The cascade tries encodings in Rust's registration order, so a tie between two equally small encodings goes the way Rust breaks it. ([#604](https://github.com/dfa1/vortex-java/issues/604))
+- ALP-RD packs its left and right parts at the dictionary's widths, as Rust does, instead of narrower widths that made it win where Rust picks a dictionary. ([#604](https://github.com/dfa1/vortex-java/issues/604))
+- ALP on an `F32` column compresses its encoded integers through the cascade, as Rust does for both float widths, instead of storing them raw. ([#604](https://github.com/dfa1/vortex-java/issues/604))
+- A nullable integer or float column keeps its validity in the encoding that stores its values instead of a `vortex.masked` wrapper, as Rust writes it, so the cascade and the dict layout pick what Rust picks. ([#604](https://github.com/dfa1/vortex-java/issues/604))
+- Zone stats tables are compressed with integer dictionaries among the candidates, as Rust's stats compressor has them. ([#604](https://github.com/dfa1/vortex-java/issues/604))
+- Run-end ends and RLE indices and offsets are stored in the narrowest unsigned type that holds them, as Rust narrows them, instead of fixed `u32`, `u16` and `u64`. ([#604](https://github.com/dfa1/vortex-java/issues/604))
+- Frame-of-reference bit-packs its residuals directly, as Rust's FoR scheme does, instead of letting the cascade pick another encoding for them. ([#604](https://github.com/dfa1/vortex-java/issues/604))
+- The cascade costs a child encoding by what it encodes to, as Rust does, not by its estimate, which made it pick run-end where Rust picks RLE. ([#604](https://github.com/dfa1/vortex-java/issues/604))
+- The cascade keeps an encoding only when it comes out smaller than the raw array, as Rust does, so a short column no longer bit-packs into more bytes than it holds. ([#604](https://github.com/dfa1/vortex-java/issues/604))
 - An all-null chunk is written as one null `vortex.constant`, as Rust writes it, instead of a masked constant `0`. ([#594](https://github.com/dfa1/vortex-java/pull/594))
 - An all-null chunk stored as a null constant (as vortex-jni writes it) reads as a masked array with every row null, like any nullable chunk, instead of a `NullArray` that the CLI, SQL and Parquet export failed on. ([#594](https://github.com/dfa1/vortex-java/pull/594))
 - An all-null date, time, timestamp or uuid chunk is written as a bare null `vortex.constant`, as Rust writes it, without a `vortex.ext` around it. ([#595](https://github.com/dfa1/vortex-java/pull/595))

docs/compatibility.md

@@ -135,7 +135,7 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul
 | `vortex.map`                | `MapEncodingDecoder`             | `MapEncodingEncoder`             | ✅      | ✅      | Single `entries` child, a bare `vortex.listview` of non-nullable `{key, value}` structs (any other entries encoding is rejected, as in Rust); map validity is delegated to that child's validity slot. Single-flat columns only: neither `ChunkedArrayCombiner` nor `ScanIterator`'s row-window slicing has a `MapArray` arm yet, so a chunked map column or a split-window scan fails as `VortexException`. Pre-existing gap shared with `vortex.listview`; tracked separately |
 | `vortex.fixed_size_list`    | `FixedSizeListEncodingDecoder`   | `FixedSizeListEncodingEncoder`   | ✅      | ✅      |                                                                       |
 | `vortex.zstd`               | `ZstdEncodingDecoder`            | `ZstdEncodingEncoder`            | ✅      | ✅      | Primitive, Utf8, Binary                                               |
-| `vortex.masked`             | `MaskedEncodingDecoder`          | `MaskedEncodingEncoder`          | ✅      | ✅      | NullableData carrier; inner values cascade (Dict/FSST/ALP/…) when cascading is enabled; Utf8/Primitive still compete on measured size even at depth 0, else first-match FixedSizeList. A nullable Utf8 column whose per-chunk dict is smaller skips the mask entirely: null is one more dict entry, invalid in the pool's validity, with non-nullable codes, as Rust's dict builder writes it |
+| `vortex.masked`             | `MaskedEncodingDecoder`          | `MaskedEncodingEncoder`          | ✅      | ✅      | NullableData carrier. A nullable primitive column is not masked when cascading: as in Rust, its validity is a plain `vortex.bool` child of the encoding that stores the values (`vortex.primitive`, `fastlanes.bitpacked`, the run values of `vortex.runend`, the indices of `fastlanes.rle`, the codes of `vortex.dict`, the left parts of `vortex.alprd`, `vortex.pco` over the valid values only), passed down by `fastlanes.for`, `vortex.zigzag` and `vortex.alp` (which patches only valid exceptions), and one whose valid rows hold one value is a `vortex.constant` masked by a plain bitmap. Other nullable columns: inner values cascade (Dict/FSST/ALP/…) when cascading is enabled; Utf8/Primitive still compete on measured size even at depth 0, else first-match FixedSizeList. A nullable Utf8 column whose per-chunk dict is smaller skips the mask entirely: null is one more dict entry, invalid in the pool's validity, with non-nullable codes, as Rust's dict builder writes it |
 | `vortex.decimal`            | `DecimalEncodingDecoder`         | `DecimalEncodingEncoder`         | ✅      | ✅      | Precision 1–76 (i8 … i256 storage by precision), negative scales. Written from `BigDecimal[]` |
 | `vortex.decimal_byte_parts` | `DecimalBytePartsEncodingDecoder`| `DecimalBytePartsEncodingEncoder`| ✅      | ✅      | Write: precision ≤ 18 only (one i64 mantissa, no low parts); competes in the cascade |
 | `vortex.datetimeparts`      | `DateTimePartsEncodingDecoder`   | `DateTimePartsEncodingEncoder`   | ✅      | ✅      |                                                                       |

docs/explanation.md

@@ -354,8 +354,8 @@ single `entries` child.
   `vortex.masked` + `vortex.list` as before.
 - **A null *entry value*** (`DType.Map(key, value.asNullable(), ...)`, e.g. `{a: 1, b: null}`
   inside an otherwise-present map row) is a completely different bit: it rides the entry
-  struct's own `value` field validity — ordinary `vortex.masked` wrapping, exactly like a
-  nullable field anywhere else inside a `vortex.struct`. It has nothing to do with the map row
+  struct's own `value` field validity — kept by the encoding of a primitive value and wrapped in
+  `vortex.masked` otherwise, exactly like a nullable field anywhere else inside a `vortex.struct`. It has nothing to do with the map row
   being present or absent.
 
 See [how-to.md#write-and-read-a-map-column](how-to.md#write-and-read-a-map-column) for the

integration/src/test/java/io/github/dfa1/vortex/integration/DictAdmissionInteropIntegrationTest.java

@@ -72,11 +72,12 @@ static Stream<Arguments> columns() {
                 Arguments.of("f64, 5 distinct", Kind.F64, 5, 0),
                 Arguments.of("utf8, 5 distinct", Kind.UTF8, 5, 0),
                 Arguments.of("nullable i32, 10 distinct", Kind.I32, 10, 7),
-                // under half distinct, yet bit-packing 7 bits wins over a 100-value dictionary.
-                // A nullable column with 1000 distinct values still gets a dict in Java only: its
-                // admission keeps the cardinality check until the nullable cascade charges
-                // bit-packing the validity, as Rust's sampled sizes do.
+                // under half distinct, yet bit-packing 7 bits wins over a 100-value dictionary
                 Arguments.of("i64, 100 distinct", Kind.I64, 100, 0),
+                // the probe charges bit-packing the validity, as Rust's sampled sizes do (#604):
+                // 10 bits a row plus a null bit still beat a 1000-value dictionary
+                Arguments.of("nullable i32, 1000 distinct", Kind.I32, 1000, 7),
+                Arguments.of("nullable i64, 100 distinct", Kind.I64, 100, 7),
                 Arguments.of("i32, 1000 distinct", Kind.I32, 1000, 0),
                 Arguments.of("utf8, 1000 distinct", Kind.UTF8, 1000, 0),
                 Arguments.of("utf8, mostly distinct", Kind.UTF8, 1_000_000, 0));

integration/src/test/java/io/github/dfa1/vortex/integration/EncodingTrees.java

@@ -0,0 +1,60 @@
+package io.github.dfa1.vortex.integration;
+
+import io.github.dfa1.vortex.core.fbs.FbsArray;
+import io.github.dfa1.vortex.core.fbs.FbsArrayNode;
+import io.github.dfa1.vortex.core.model.EncodingId;
+import io.github.dfa1.vortex.reader.SegmentSpec;
+import io.github.dfa1.vortex.reader.VortexReader;
+import io.github.dfa1.vortex.reader.layout.Layout;
+
+import java.io.IOException;
+import java.lang.foreign.MemorySegment;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+
+import static io.github.dfa1.vortex.core.io.VortexFormat.LE_INT;
+
+/// A file's layouts and array encodings, for comparing what vortex-jni and vortex-java write.
+final class EncodingTrees {
+
+    private EncodingTrees() {
+    }
+
+    /// The file's layouts and array encodings, depth first, one indented id a line. A chunked
+    /// layout around a single chunk is left out: Java keeps one where Rust writes the flat layout
+    /// directly, and that layout shape is not what the encoding comparisons test.
+    static List<String> of(Path file) throws IOException {
+        List<String> out = new ArrayList<>();
+        try (VortexReader reader = VortexReader.open(file)) {
+            layoutTree(reader, reader.layout(), 0, out);
+        }
+        return out;
+    }
+
+    private static void layoutTree(VortexReader reader, Layout layout, int depth, List<String> out) {
+        if (layout.isChunked() && layout.children().size() == 1) {
+            layoutTree(reader, layout.children().getFirst(), depth, out);
+            return;
+        }
+        out.add("  ".repeat(depth) + layout.layoutId().id());
+        if (layout.isFlat() && !layout.segments().isEmpty()) {
+            SegmentSpec spec = reader.footer().segmentSpecs().get(layout.segments().getFirst());
+            MemorySegment seg = reader.rawSegment(spec);
+            int fbLen = seg.get(LE_INT, seg.byteSize() - 4L);
+            FbsArray array = FbsArray.getRootAsFbsArray(seg.asSlice(seg.byteSize() - 4L - fbLen, fbLen));
+            arrayTree(array.root(), reader.footer().arraySpecs(), depth + 1, out);
+            return;
+        }
+        for (Layout child : layout.children()) {
+            layoutTree(reader, child, depth + 1, out);
+        }
+    }
+
+    private static void arrayTree(FbsArrayNode node, List<EncodingId> specs, int depth, List<String> out) {
+        out.add("  ".repeat(depth) + specs.get(node.encoding()).id());
+        for (int i = 0; i < node.childrenLength(); i++) {
+            arrayTree(node.children(i), specs, depth + 1, out);
+        }
+    }
+}