Tracking Issue: Union Nullability Semantics

[connortsui20]

This is a tracking issue for figuring out the semantics we want for Union type, which we need to help us inform our implementation of Union.

This is a subissue of tracking issue #7882 and epic #7705.

Motivation

Union type has some weird semantics wrt nullability. If we look at the Arrow specification, Arrow simply says that the nullability of the value is determined by the child array. I would equate it to something like this (in Rust):

enum ArrowUnion {
    Foo(Option<i32>),
    Bar(Option<String>),
    Qux(Decimal),
}

assert_eq!(ArrowUnion::Foo(None), ArrowUnion::Bar(None));

So if we want to represent a null value with ArrowUnion, we must choose one of Foo and Bar to represent that null value. This also implies that in Arrow, ArrowUnion::Foo(None) is semantically equivalent to ArrowUnion::Bar(None), which means null union values may not necessarily roundtrip their physical representations.

The lack of a top-level null presents a problem in Vortex, as we have generally made the assumption that ALL dtypes store their nullability at the top-level (except for DType::Null, which is special). For example, DType::Struct can be null at the top level, but it can also have different nullabilities for each of its child fields.

Even if it were not the case that Vortex prefers a semantic difference between parent and child nulls, implementation of this null type will be somewhat confusing: needing to "choose" how to represent null / validity masks means that null-related compute (like mask and cast and take with null indices) is not immediately obvious how to implement.

Design

We propose designing Vortex's Union type to be a superset of the functionality provided by Arrow's union by distinguishing between top-level null Union and child nulls. Basically, instead of Foo, we will use Option<Foo> and say that None != Foo::Bar(None).

// Not real Rust, we cant define enums inline.

type VortexUnion = Option<enum Child {
    Foo(Option<i32>),
    Bar(Option<String>),
    Qux(Decimal),
}>;

let top_level_null = None;
let inner_null = Some(Child::Foo(None))
assert_ne!(top_level_null, inner_null);

// Still the same logic here as in the previous example.
let inner_null2 = Some(Child::Bar(None))
assert_eq!(inner_null, inner_null2);

So instead of Union(UnionVariants) that has to decide nullability at runtime by looking at the children, we have Union(UnionVariants, Nullability). This also allows us to store a validity mask at the top-level of the UnionArray struct (that we will implement soon 🤞).

This does mean that we will not directly roundtrip with Arrow, but it should be quite easy / cheap to convert a Vortex Union to an Arrow Union. The ONLY case we are worried about (that Arrow Union cannot express) is a nullable Vortex Union with non-nullable children.

There are multiple ways to do this, and they can be configuration options in our Arrow exporter:

• Add an additional validity child to the Arrow Union that is essentially just a validity mask
• Choose 1 of the children to be nullable and use that for storing nulls
• Force all of the children to be nullable (and maybe we can be clever here?)

Upstream to Arrow?

We could also propose that Arrow do a similar thing. Arrow could just add a validity mask to the top-level union to have the same logic as proposed above, and it would be backwards compatible because this is strictly more expressive.

Steps

TODO

Unresolved questions

• Is this a bad idea?

Implementation history

• Add UnionScalar #8739
• Propagate Union nullability to its variants #8768

[connortsui20]

(generated comment from Codex where I asked it to investigate the differences between arrow implementations wrt nullability)

The nullability question does not need to block the canonical sparse UnionArray.

The Arrow implementations agree on the behavior that matters for the array:

• arrow-rs has no parent null buffer and gathers logical validity from the selected child.
• Arrow C++ follows the selected child for IsNull(i).
• Arrow Java reports no physical nulls on the union itself, while value access still follows the selected child.

A union therefore has no validity mask of its own. A row is null when the selected child is null. Nulls in inactive sparse children do not matter.

The outer Arrow Field::nullable flag is less consistent. arrow-rs creates union fields as non-nullable, while the Java dense and sparse helpers use different defaults. That flag should not define the Vortex semantics.

The contract currently on develop is sufficient: a nullable union needs at least one nullable or Null variant, a non-nullable union cannot have either, and the array computes validity from the selected child.

The remaining question is what with_nullability, mask, or another null-producing operation should do when it needs to make an existing union nullable. The operation could make an existing variant nullable, require a Null variant, or require an explicit output type.

This can be deferred. The initial canonical array can require an already-consistent union DType and return unsupported from operations that would need to change it, including mask, nullable-index take, null-producing casts, and appending a null.

This does not affect the physical layout, so UnionArray can be implemented and used now. The concrete documentation for this boundary is in #8714.