[Variant] Consolidate Spark-compatible missing-value validation and errors

[peterxcli]

What is the problem the feature request solves?

#5868 checks missing values in rewrite_shredding_state, using allow_missing=false for the root and list elements and true for object fields. These checks run during residual rewriting because Arrow's unshredder does not enforce Spark's rules.

Move these checks into the recursive unshredder through an opt-in missing-value policy. Keep Arrow's default behavior: the Parquet specification requires Variant null when a required value is missing. Changing that default to match Spark would violate the specification.

This concerns full Variant reconstruction for native Parquet projection. Spark's pushed-down field extraction has different validation coverage and must not acquire whole-value validation through this change.

Required behavior

The relevant ShreddingUtils.rebuild source is identical in Spark 4.0.4, 4.1.3, and 4.2.0. Direct reconstruction probes against all three versions confirmed missing root/list values, missing and NULL object fields, and explicit Variant null.

Here, “missing” means both value and typed_value are absent or SQL NULL. It does not mean a present value = X'00', which encodes Variant null.

Input at an active row	Required Comet result
Root Variant struct is SQL NULL	SQL NULL; do not inspect its children
Valid root state, missing value; also a residual-only root with NULL value	MALFORMED_VARIANT
Valid list element state, missing value	MALFORMED_VARIANT
Valid object field state, missing value	Omit the field
NULL object field state struct under a present typed object	MALFORMED_VARIANT
Present residual X'00', no typed value	Explicit Variant null; retain the object field/list element
NULL typed_value with a present residual	Read the residual; do not inspect descendants of the inactive typed object/list
Valid root with NULL metadata	MALFORMED_VARIANT

A NULL list element struct needs a separate assertion. Spark's full reconstruction calls getStruct without checking element nullness and can throw NullPointerException; direct probes reproduced this on all three versions, and Parquet scans reproduced FAILED_READ_FILE.NO_HINT with an NPE cause on 4.0.4 with both vectorized and row readers. In contrast, Spark's field-extraction path explicitly checks null list elements. Preserve Comet's existing controlled MALFORMED_VARIANT rejection for this invalid state. This is an explicit exception to exact Spark error parity; do not reproduce the NPE.

Concrete implementation

1. Add an opt-in policy in parquet-variant-compute/src/unshred_variant.rs. Proposed API: unshred_variant_with_options(array, options), with MissingValuePolicy::{Parquet, Spark}. Existing unshred_variant(array) keeps the Parquet policy. The new option controls only missing values and state validity; scalar precedence and byte encoding remain separate work.

2. Check each original state before dispatching to its value builder. Retain the source StructArray and its context (root, object field, or list element) alongside the existing recursive row builder. In Spark mode, reject a NULL active nested state before reading its children. For a valid state with neither value present, omit an object field and return ArrowError::InvalidArgumentError for a root/list element. Otherwise delegate to the existing decoder. Check root metadata nullness before parsing it. Perform these checks during the existing traversal, without a separate recursive validation pass.

3. Preserve parent masking and list offsets. The root loop skips SQL NULL rows before checking metadata or values. Recurse into object fields/list elements only when their enclosing typed_value is valid. For lists, visit only the current row's element range, including sliced arrays and ListView offsets. Physical child contents beneath an inactive parent must not cause an error.

4. Cover the residual-only fast path. Before returning an unshredded array unchanged in Spark mode, check that every NULL value or metadata entry is masked by a NULL root. Return an error for an unmasked NULL while preserving valid input bytes. Checking only handle_unshredded_case! is insufficient: the fast path, ValueOnlyUnshredVariantBuilder, and NullUnshredVariantBuilder must obey the same policy. Arrow main's current implementation already materializes Variant null for missing root values; that still differs from Spark.

5. Switch Comet after the upstream API is available in its pinned dependency. Select the Spark policy in normalize_variant_array. Delete the missing-state loop and allow_missing argument from rewrite_shredding_state and its callers. Retain metadata_rows and the residual/metadata rewrites that still use it. Keep the existing Arrow error to SparkError::MalformedVariant mapping and JVM conversion in ShimSparkErrorConverter. Until the replacement is available, retain the current checks.

Verification and completion

• Add Arrow tests for both policies using the same root/object/list fixtures. Default mode must retain the specification's required-value-to-Variant-null behavior; Spark mode must enforce the table above. Cover residual-only states, explicit Variant null, parent masking with invalid physical children, and sliced lists.
• Extend normalize_rejects_missing_required_shredding_states and CometVariantProjectionSuite with the missing cases, including a separate NULL-list-struct regression. Assert CometNativeScanExec and the Spark error class for rejected inputs.
• Run the unchanged upstream VariantShreddingSuite assertions through Comet on 4.0.4, 4.1.3, and the experimental 4.2.0 profile. Compare full-value results/errors with vanilla Spark, with the documented NULL-list-struct exception.
• Completion means the Comet precheck is deleted, the replacement runs in the existing unshredding traversal, and these tests pass. Neither a fallback nor an Arrow upgrade alone completes this issue.

Additional context

Parent: #5477; current implementation: #5868. This upstream API is a proposal, not an existing feature. The current checks are embedded in a rewriting pass that remains necessary, so removing them alone does not eliminate that pass or establish a speedup.

#5978 tracks removing the extra Spark byte-reconstruction pass; its replacement must retain this validation. #5980 tracks typed-value precedence, and #5979 tracks missing metadata keys. apache/arrow-rs#10619 addresses a different malformed object-field layout.