feat: support direct Variant projection in native Parquet scans

[peterxcli]

Which issue does this PR close?

Closes #5551.
Closes #5546.

Rationale for this change

Complete the ordinary-Parquet admission step in #5546 so Spark can project whole Variant values through a native scan.

What changes are included in this PR?

Enable direct top-level Variant projection with allowReadingShredded=true and pushVariantIntoScan=false. Carry missing-column defaults as constant [value, metadata] structs while preserving their schema indexes, and retain Spark fallbacks for unsupported consumers and reader settings.

Reuse the Variant normalizer to handle Spark's empty object keys until Comet upgrades to an Arrow release containing apache/arrow-rs#10352. Document the supported surface and add removal TODOs for the compatibility paths, including #5477.

How are these changes tested?

Focused Spark 4.0/4.1 tests cover value parity, defaults, Unicode field matching, vector layout, and fallback behavior. Native Variant tests and clippy pass, and Spark 3.5 compiles. The new projection suite runs in Linux and macOS CI.

[sunchao]

Reviewed 6e556c94 against base 8b818b53. No verified P1/P2 findings.

Correctness

Previously, the native Parquet reader had Variant storage adaptation, but the scan and execution rules still rejected direct Variant output. This change admits a Variant at a required top-level Parquet field and preserves Spark's logical Variant identity over binary children ordered [value, metadata]. Nested Variant projections and pushed Variant extraction remain on Spark. The default-value serializer now keeps each value paired with its original required-schema index; if any default cannot be serialized, the scan falls back instead of shifting later defaults. Native planning accepts only literals or a constant, correctly typed Variant storage struct and checks index bounds.

The reconstruction changes match the maintained Spark 4.0 source for missing object fields, required shredding states, scalar/array typed-value precedence, and Java UTF-16 object ordering. Adding shredded field names to the metadata dictionary also remaps residual field IDs. The empty-key retry checks the original metadata encoding before rebuilding it. Parent SQL nulls remain distinct from a Variant null. Spark handles strict-reader layout validation, nondefault timestamp inference, encryption, Variant consumers, and columnar-to-row conversion.

At 2026-09-12 19:35:47 UTC, the head has 60 successful and 10 skipped checks. The Spark 4.0 scan job passed 510 tests and the Spark 4.1 scan job passed 518; both executed all 11 new projection tests. Native CI passed the Variant reconstruction tests, and Spark 4.1 exec passed the Variant Arrow-representation and Python-fallback cases. These jobs checked out merge 020dfcb2 (6751af02 + 6e556c94), whose base and tree differ from the assigned pair. The authored Variant implementation and tests match, but DataFusion and schema-adaptation context differ, so this is qualified merge-CI evidence. No local product build or tests ran. Maintained Spark 3.4/4.1 branches were unavailable for source comparison; CI does not close those source gaps.

Performance

The implementation retains scan pruning: unread Variant roots are omitted from the native data schema, and pruning a Variant child can leave supported siblings eligible for native scanning. It reuses the existing scan and normalization path without adding an extra execution operator. Output buffers are rebuilt lazily when values actually need rewriting, preserving the unchanged path.

Shredded values still require recursive reconstruction, metadata lookups and, when keys are missing, per-row dictionary rebuilding and residual remapping. The empty-key compatibility retry adds work only after normalization fails. I found no verified performance regression, but this review has no benchmark establishing a speedup. Could you add a matched scan microbenchmark separating canonical, partially shredded, fully shredded and empty-key data, including normalization allocations? Please include repeated metadata dictionaries so the results show whether reconstruction does avoidable per-row work.

Design

The scan-only exception is appropriately bounded. Exempting CometScanExec from the execution-level Variant guard permits direct projection while retaining the guard for operators consuming or producing Variant. Spark's columnar-to-row path uses the two binary children through its Variant getter; the explicit Python guard covers both input and output. This keeps scan eligibility independent from expression, shuffle, write and Python support.

The settings that change physical interpretation are checked before conversion to a native scan. Unsupported defaults also fail planning as a whole. The tests check both returned values and the expected execution/fallback nodes, including present nulls versus missing defaults, later default indexes, Unicode field matching, encrypted files and malformed layouts.

Abstraction & complexity

The additional complexity is concentrated in the existing Variant normalization module and one narrowly scoped default-value helper. It does not create a second general expression evaluator or broaden Variant support in unrelated type checkers. The Spark 3/4 shim keeps version-specific Variant objects out of shared code.

The metadata-extension and empty-key paths share residual rewriting instead of maintaining separate reconstruction engines. Their comments identify the Arrow follow-ups that can remove these compatibility paths. Retaining those removal conditions and the buffer-reuse/null-state tests will help keep this temporary machinery contained. No additional abstraction or blocking simplification is needed for this change.