Add `UnionScalar` by connortsui20 · Pull Request #8739 · vortex-data/vortex

connortsui20 · 2026-07-13T15:25:52Z

Rationale for this change

Depends on #8768.

Tracking Issue: #7882

UnionScalar is the row representation needed for scalar access and constant arrays before the canonical UnionArray. Non-null values store the selected type ID and child ScalarValue; null remains on the outer Scalar, avoiding a second null layer.

The DType/UnionVariants nullability propagation is split into #8768 so this PR contains only scalar-related changes.

What changes are included in this PR?

add UnionValue, UnionScalar, construction, downcasts, validation, display, equality, hashing, defaults, and sizing
add scalar protobuf round trips and keep non-identity Union casts unsupported
add arbitrary generation and ConstantArray scalar access coverage
integrate Scalar::into_nullable with the Union nullability behavior from Propagate Union nullability to its variants #8768

What APIs are changed? Are there any user-facing changes?

Adds ScalarValue::Union, UnionValue, UnionScalar, Scalar::union, and the Union scalar downcasts.

Validation

cargo nextest run -p vortex-array
cargo clippy -p vortex-array --all-targets --all-features
cargo +nightly fmt --all -- --check

Workspace-wide Clippy could not complete locally because the DuckDB build could not find duckdb.h.

codspeed-hq · 2026-07-13T15:31:30Z

Merging this PR will not alter performance

✅ 1660 untouched benchmarks
⏩ 47 skipped benchmarks¹

_{Comparing ct/union-scalar (bd74508) with codex/union-nullability-propagation (b5ec0a0)}

47 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

connortsui20 · 2026-07-13T18:57:15Z

+        // Union nullability is part of its variant dtypes, so the generic nullability-only cast
+        // below is not valid for unions. Keep all non-identity union casts unsupported until their
+        // semantics are defined.
+        if matches!(self.dtype(), DType::Union(_)) || matches!(target_dtype, DType::Union(_)) {
+            vortex_bail!(
+                "non-identity union scalar cast from {} to {target_dtype} is not supported",
+                self.dtype()
+            );
+        }


Interested in what people think about how we should deal with this

connortsui20 · 2026-07-13T19:43:23Z

+/// We implement `Hash` manually so top-level dtype nullability does not affect the hash.
 impl Hash for Scalar {
    fn hash<H: Hasher>(&self, state: &mut H) {
+        // TODO(connor): Recursively ignore dtype nullability to match Scalar equality (#8744).


robert3005 · 2026-07-14T14:00:50Z

+        // Union nullability is part of its variant dtypes, so the generic nullability-only cast
+        // below is not valid for unions. Keep all non-identity union casts unsupported until their
+        // semantics are defined.
+        if self.dtype().is_union() || target_dtype.is_union() {


I think you have to gate the other branch and make sure this falls back to union scalar

what is the other branch?

the one the comment refers to

I don't think I understand, are you saying that it can somehow get past this gate?

Basically @connortsui20 we should plumb this method fully such that we delegate to the match and fail there

robert3005 · 2026-07-14T15:37:44Z

            DType::List(..) | DType::FixedSizeList(..) => self.as_list().cast(target_dtype),
            DType::Struct(..) => self.as_struct().cast(target_dtype),
-            DType::Union(..) => todo!("TODO(connor)[Union]: unimplemented"),
+            DType::Union(..) => unreachable!("union casts are handled before scalar dispatch"),


@connortsui20 I am saying you should fallback here in your cast

I still don't really understand here, what exactly do you mean by "falls back to union scalar"? We already have a union scalar in the input

robert3005 · 2026-07-14T15:38:40Z

+    }
+
+    /// Returns the non-null zero value for `dtype`, or [`None`] if no such value exists.
+    fn try_zero_value(dtype: &DType) -> Option<Self> {


I don't get why this needs to return option, Null is a weird type

Null isn't the only type that returns None here though (see FSL, Struct, and now Union)? This just improves the error message in general

because null returns none, I think the develop behaviour is better

this is just an implementation detail though? this is a private helper so the behavior is the same (just different error message)?

robert3005 · 2026-07-14T15:39:42Z

                Self::Tuple(field_values)
            }
-            DType::Union(..) => todo!("TODO(connor)[Union]: unimplemented"),
+            DType::Union(_) => vortex_panic!("{dtype} has no default value"),


this could pick first variant

its not clear what to do if there are no variants (empty union / void type)

robert3005 · 2026-07-14T15:46:02Z

I think as_nullable for union has to traverse all the children and make them nullable. Since there's no top level nullability there's no other implementation that makes sense imho

connortsui20 · 2026-07-14T15:50:01Z

This does mean that as_nullable will have different behavior than struct for example which only does the top-level thing, but lets just do it and see how well that works

robert3005 · 2026-07-14T15:58:27Z

why is it different? Struct has top level nullability while the union doesn't

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

connortsui20 · 2026-07-15T14:47:31Z

See #8769 @robert3005 @joseph-isaacs

connortsui20 changed the title ~~Add Union scalar support~~ Add UnionScalar Jul 13, 2026

connortsui20 mentioned this pull request Jul 13, 2026

Tracking Issue: Add DType::Union #7882

Open

9 tasks

connortsui20 force-pushed the ct/union-scalar branch 4 times, most recently from 7e619d3 to 243969b Compare July 13, 2026 16:08

connortsui20 commented Jul 13, 2026

View reviewed changes

connortsui20 force-pushed the ct/union-scalar branch 3 times, most recently from 8bb8db6 to 7925944 Compare July 13, 2026 19:41

connortsui20 commented Jul 13, 2026

View reviewed changes

connortsui20 force-pushed the ct/union-scalar branch 3 times, most recently from 43d2fc9 to 9de7663 Compare July 13, 2026 20:01

connortsui20 added the changelog/feature A new feature label Jul 13, 2026

Base automatically changed from codex/reject-negative-union-type-ids to develop July 13, 2026 20:34

connortsui20 force-pushed the ct/union-scalar branch from 9de7663 to 4893177 Compare July 13, 2026 20:56

connortsui20 marked this pull request as ready for review July 13, 2026 22:05

joseph-isaacs reviewed Jul 14, 2026

View reviewed changes

Comment thread vortex-array/src/scalar/typed_view/union.rs Outdated

robert3005 reviewed Jul 14, 2026

View reviewed changes

Comment thread vortex-array/src/scalar/cast.rs Outdated

robert3005 reviewed Jul 14, 2026

View reviewed changes

connortsui20 force-pushed the ct/union-scalar branch from 4893177 to 8ee294c Compare July 14, 2026 15:47

connortsui20 requested a review from robert3005 July 14, 2026 16:09

connortsui20 mentioned this pull request Jul 15, 2026

Propagate Union nullability to its variants #8768

Draft

Add Union scalar support

bd74508

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

connortsui20 force-pushed the ct/union-scalar branch from 1e93238 to bd74508 Compare July 15, 2026 13:20

connortsui20 changed the base branch from develop to codex/union-nullability-propagation July 15, 2026 13:20

connortsui20 mentioned this pull request Jul 15, 2026

Tracking Issue: Union Nullability Semantics #8769

Open

1 task

Uh oh!

Conversation

connortsui20 commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

What APIs are changed? Are there any user-facing changes?

Validation

Uh oh!

codspeed-hq Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robert3005 commented Jul 14, 2026

Uh oh!

connortsui20 commented Jul 14, 2026

Uh oh!

robert3005 commented Jul 14, 2026

Uh oh!

connortsui20 commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

connortsui20 commented Jul 13, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 13, 2026 •

edited

Loading