[Variant] Preserve typed-value precedence without residual pre-rewriting

[peterxcli]

What is the problem the feature request solves?

#5868 masks residual values when a scalar or list typed_value is present. Spark ignores those residual bytes, but Arrow parses them and then rejects the two-present-values state. The mask also prevents Comet's own residual preparation from parsing bytes Spark ignores.

Move this decision before residual decoding in both readers of those bytes: Arrow's unshredder and Comet's preparation step. Then remove the temporary null bitmap and replacement residual column.

Verified behavior

ShreddingUtils.rebuild first branches on a present typed_value. Its scalar/list branches never read value; its object branch merges residual fields. Direct probes against Spark 4.0.4, 4.1.3, and 4.2.0 produced the same results:

State	Result
Typed integer 1, residual X'FF'	1; the malformed residual is ignored
Typed list containing 1, residual X'FF'	[1]
NULL typed scalar, residual X'FF'	MALFORMED_VARIANT
Present typed object, residual X'FF' or a non-object	MALFORMED_VARIANT
Residual object contains a field declared in the shredded object schema	MALFORMED_VARIANT, even when that typed field is missing in this row

An Arrow 59.3.0 probe rejected the first case while parsing the residual. Current Arrow main has the same parse-before-precedence structure.

The Parquet specification allows both representations only for partial objects. Keep canonical Arrow behavior as the default.

Concrete implementation

1. Add an opt-in NonObjectResidualPolicy::PreferTyped to the proposed UnshredOptions entry point from [Variant] Consolidate Spark-compatible missing-value validation and errors #5977. The default remains Reject. Thread it through the existing recursive row builders, including primitive, decimal, timestamp, and all supported list representations.

2. In handle_unshredded_case!, decide whether the residual is needed before calling Variant::try_new_with_metadata:

   ignore_residual = PreferTyped && typed_value.is_valid(row) && !partial_object
   residual = None if ignore_residual else decode_present_residual(row)?
   

   Continue through the existing typed decoder when the residual is ignored. A malformed typed value must still fail; do not retry the residual. Preserve the current object merge and schema-name conflict checks.

3. In Comet, change rewrite_residual_values to receive enough state to apply the same predicate. Skip validation, metadata remapping, and UTF-16 reordering for ignored rows. If another row causes an output buffer to be allocated, copy the ignored row's original bytes unchanged. Use the existing row loop and typed validity; do not build another masked array. Keep root metadata validation and parent-null masking.

4. Once the pinned Arrow dependency supports the option, enable it in normalize_variant_array and delete the scalar/list is_not_null/nullif masking branch and its field-nullability adjustment from rewrite_shredding_state. Changing Arrow alone is insufficient: deleting the mask before step 3 would make Comet reject X'FF' during preparation.

Verification and completion

• Test both Arrow policies with a valid but conflicting residual, explicit Variant null, and malformed bytes. With PreferTyped, a valid typed scalar/list must produce identical output for all three residuals; default mode must reject the two-present-values state.
• Test NULL typed values, invalid typed list elements, partial objects with nested residual objects/lists, and schema-name collisions including absent typed fields.
• Exercise the rule inside nested object fields and list elements, sliced List/LargeList/ListView arrays, mixed rows where only some need residual rewriting, and children masked by SQL NULL.
• Extend the existing CometVariantProjectionSuite cases and compare results/errors against supported Spark profiles while asserting CometNativeScanExec. Keep upstream Spark assertions unchanged.
• Completion requires removal of the masking branch with all checks passing. Other residual preparation remains necessary for its own tracked reasons; this change alone does not remove the entire preparation pass.

Additional context

Parent: #5477; implementation: #5868. #5977 owns missing-state validation, #5979 owns metadata rebuilding, and #5978 owns the final output encoding. The API above is a proposal for upstream review, not existing Arrow support. Research probes exercised reconstruction helpers; the full native scan matrix remains an implementation acceptance check.