[Variant] Track and remove native Variant compatibility workarounds after upstream fixes

[peterxcli]

What is the problem the feature request solves?

Native Variant projection from #5407, continued by #5868, includes compatibility paths for gaps in upstream Variant compute, Spark-compatible Parquet reader policy, and older Spark Variant encodings. As of 2026-09-13, Comet main locks DataFusion 55.1.0 and Arrow/Parquet 59.3.0; #5868 currently locks DataFusion 55.0.0 and Arrow/Parquet 59.3.0. The dependency upgrade is no longer a blocker. These paths are intentionally narrow, but some perform row-by-row decoding or metadata rebuilding and should not become permanent accidentally.

This issue records exactly which code is removable after an upstream fix ships, which upstream change owns it, and which nearby code is required Comet/Spark integration and must remain.

This is maintenance follow-up for #5407 under the Native Variant epic #5438. Spark UTF-16 output cleanup is tracked separately by #5474. Unicode case-insensitive Parquet field matching (#5495) is complete through #5602. The split series in #5546 reuses that shared matcher and does not carry the original PR's Unicode guards.

Arrow-rs compute workarounds

• Encoded Variant metadata

  • Comet recursively decodes Dictionary arrays before VariantArray::try_new in normalize_variant_storage.
  • Upstream [Variant] Accept Dictionary and RunEndEncoded metadata in VariantArray arrow-rs#10802 and fix(variant): accept Dictionary and RunEndEncoded metadata in VariantArray arrow-rs#10810 cover Dictionary and RunEndEncoded canonical metadata only.
  • After upgrading to a release containing that fix, stop decoding metadata locally and keep focused regressions exercising upstream directly. This does not remove compatibility for other encoded children.

• Dictionary-encoded value and typed_value

  • Dedicated compatibility feature request: [Variant] Support dictionary-encoded value and typed_value inputs arrow-rs#11068. Current Arrow-rs main rejects both representations in VariantArray::try_new; #10810 does not cover them.
  • The canonical Arrow Variant extension explicitly permits encodings for metadata, not these children. This is a compatibility-policy decision, not an unfixed canonical-input bug.
  • Embedded Arrow schemas can restore dictionary types. Retain the recursive decoder until upstream provides supported normalization, or Comet proves reader-side physical inference/fallback excludes such inputs at every supported entry point.

• Empty Variant object keys

  • Arrow-rs 59.2.0's full metadata validator still requires strictly increasing offsets for unsorted dictionaries, rejecting equal offsets used by empty dictionary entries. Comet retries unshredding in unshred_variant_for_spark, canonicalizes affected rows in canonicalize_spark_empty_key_metadata, and uses shallow metadata parsing while rebuilding values.
  • Upstream: Add proptest fuzzing to parquet-variant and implement fixes for findings arrow-rs#10352, merged but not present in Comet's current dependency.
  • After upgrading to a release containing the fix, remove the retry and canonicalizer, use fallible metadata construction followed by with_full_validation at source-consumption sites, and change tests that assert the old rejection to assert successful validation.
  • Here, “empty field name” means an empty Variant object key such as {"": 1}, not an empty Arrow schema field name.

• Unsigned shredded typed_value fields

  • Comet widens UInt8 -> Int16, UInt16 -> Int32, and UInt32 -> Int64 before unshredding. UInt64 -> Decimal(20,0) remains Spark-specific.
  • Upstream options are unresolved: Parquet Variant rejects canonical Arrow UInt8/UInt16/UInt32 typed values arrow-rs#10416 / variant: Support Arrow UInt8/UInt16/UInt32 typed values arrow-rs#10417 add widening, while [Docs][Format] Variant extension: align typed_value primitive type mappings with the Parquet shredding spec arrow#50622 / GH-50622: [Docs][Format] Align Variant typed_value primitive type mappings with the Parquet shredding spec arrow#50810 propose removing unsigned mappings from the canonical Variant shredding table.
  • Remove the first three unsigned arms if Arrow-rs provides the widening in a released dependency. If the canonical format rejects unsigned shredding instead, retain them only as an explicit legacy/noncanonical-file compatibility policy or fall back safely. Do not remove the shared recursive normalizer wholesale.

• Wide physical DECIMAL typed_value fields

  • Current Arrow-rs main still infers FIXED_LEN_BYTE_ARRAY wider than 16 bytes as Decimal256 even when the declared precision is at most 38. VariantArray rejects Decimal256, while Spark reads the declared DecimalType in its 128-bit representation.
  • Comet recursively narrows Decimal256(p, s) -> Decimal128(p, s) for p <= 38. Runtime regressions cover positive and negative DECIMAL(38,2) values stored in 17- and 32-byte physical fields.
  • Dedicated upstream bug: [Variant] Wide physical DECIMAL with precision <= 38 is inferred as Decimal256 and rejected arrow-rs#11067, reproduced on Arrow-rs 4cd8be954f6bc6b6dd265140207365b59a9900ec for both physical widths and signs. This is not fixed by stripping ARROW:schema or by variant: Support Arrow UInt8/UInt16/UInt32 typed values arrow-rs#10417. Remove the arm only if upstream Variant compute accepts the representation or the reader can supply Spark-compatible physical inference without it.

• Shredded object keys absent from metadata

  • feat: support direct Variant projection in native Parquet scans #5868 adds extend_shredded_metadata: add missing keys from the shredded schema and remap residual field IDs before unshredding.
  • Dedicated upstream panic bug: [Variant] unshred_variant panics when a shredded field name is absent from metadata arrow-rs#11069. Current Arrow-rs main panics when a present shredded field name is missing from the row's metadata.
  • The current Parquet specification requires all shredded field names in metadata. Comet's repair preserves permissive Spark compatibility for inputs outside that requirement.
  • An upstream error instead of a panic is insufficient to remove this repair. Remove it only if upstream supports equivalent permissive reconstruction, or Comet deliberately changes this compatibility policy to a safe fallback. Preserve residual ID remapping, nested objects/lists, absent fields, SQL NULL and Variant null coverage.

DataFusion/Arrow reader integration workaround

• Use Spark-compatible physical schema inference for projected Variant scans
  • Spark ignores the advisory ARROW:schema footer hint, while Arrow-rs honors it and can restore shredded leaves as Decimal256, Date64, Dictionary, or fixed-size types. Comet therefore replaces an embedded hint with physical Parquet inference for an unencrypted scan whose pruned required schema projects Variant.
  • Arrow-rs also maps physical Parquet ENUM to Binary, while Spark maps ENUM to String. Once Arrow inference has discarded that annotation, the Variant normalizer cannot distinguish ENUM from raw BINARY. Comet therefore adds back a physical-schema-derived hint that changes only ENUM leaves to Utf8 (ENUM mapping, footer rewrite). Runtime controls verify ENUM and STRING become Variant strings while unannotated BINARY remains binary.
  • All other key/value metadata, row groups, column orders, column indexes, and offset indexes are preserved. Using the pruned schema also preserves feat: Support native scans with unprojected Spark 4 VARIANT columns #5377: an unread Variant column does not activate this path. Encrypted Variant scans remain on Spark because reconstructing public FileMetaData cannot preserve arrow-rs's private decryptor state.
  • Reassessment: merged Allow specifying an arrow schema for PartitionedFile datafusion#22360 already supplies PartitionedFile.arrow_schema. Arrow's ArrowReaderOptions::with_schema suppresses advisory Arrow metadata and accepts nested Binary-to-Utf8 hints. A local check verified ENUM-to-Utf8, raw BINARY preservation, bypass of an invalid Arrow hint, and reuse of the original metadata Arc. No additional Arrow ENUM-conversion API is needed for a known full per-file schema.
  • Remaining upstream integration request: Allow deriving a Parquet file Arrow schema from metadata during lazy opening datafusion#25251. Comet learns each file's shredded layout during lazy footer loading; the current reader factory cannot return a newly derived schema/options to DataFusion's opener. Allow source-specific physical-schema policy after footer loading and before Arrow schema inference, preserving the original metadata, cache behavior and decryption state. #22360 is sufficient when that complete schema is already available before opening.
  • Parquet schema file type coercion functionality is not applied on all fields in nested schemas, only on the top-level fields datafusion#25192 covers nested table-schema coercions; it does not supply missing per-file shredded layouts or physical ENUM annotations.
  • Remove the footer reconstruction when the existing supplied-schema route can be used at Comet's lazy-open boundary. Reassess the encrypted fallback only after encryption-specific validation; preserving the metadata Arc alone is not an encrypted-scan test.

Describe the potential solution

For each checklist item:

1. Wait for the relevant upstream behavior to merge and appear in a released dependency or reachable DataFusion reader API.
2. Upgrade Arrow/Parquet/DataFusion through the normal dependency update.
3. Delete only the mapped compatibility branch; do not duplicate an upstream implementation locally.
4. Keep or adapt the existing regression tests to prove the upstream path works through Comet.
5. Run the focused Rust Variant tests, Spark 4 Variant SQL tests, Spark 3 compilation boundary, formatting/lint, and git diff --check.

Definition of done:

• Every temporary upstream workaround is either removed or explicitly retained with a current compatibility reason.
• Encoded metadata, the chosen encoded-child policy, empty object keys, missing shredded-key compatibility, wide decimals, the chosen unsigned-input policy, and physical-schema inference remain covered.
• Whole-value Variant projection still returns Spark's required [value, metadata] layout and preserves SQL NULL versus Variant JSON null.
• Unread Variant pruning remains native, and encrypted scans do not lose encryption state.
• No Spark/FFI integration code is removed merely because an Arrow-rs dependency was upgraded.

Additional context

The following nearby code is not covered by the upstream fixes above:

• Millisecond timestamp, unannotated fixed-length binary, UInt64 -> Decimal(20,0), and FixedSizeList -> List normalization are Spark-reader compatibility for physical types outside the canonical Arrow Variant mapping. variant: Support Arrow UInt8/UInt16/UInt32 typed values arrow-rs#10417 does not cover them.
• prepare_variant_for_unshredding, SparkMetadataBuilder, and the Spark-format reconstruction handle canonical UTF-8 versus legacy Spark UTF-16 ordering and Spark byte-format compatibility. Their output-side simplification belongs to [Variant] Remove Variant UTF-16 output rewriting #5474; an input-side path may remain for historical Spark-written files.
• ffi_schema_for_field substitutes embedded NUL because the Arrow C Data Interface uses NUL-terminated names. That is an ABI limitation, unrelated to Add proptest fuzzing to parquet-variant and implement fixes for findings arrow-rs#10352.
• Exporting the complete Arrow Field, preserving ARROW:extension:name=arrow.parquet.variant, unshredding to [value, metadata], materializing ordinary Binary children for Spark, preserving parent nulls, and retaining explicit fallback gates are required Comet integration.
• Spark-compatible case-insensitive Parquet column matching is implemented in the shared schema adapter by feat: support unicode case sensitive field names for reading parquet #5602 (closes Support Spark-compatible Unicode case-insensitive Parquet field matching #5495). The remaining Variant splits reuse it without the original PR's non-ASCII fallback or fail-closed guard. This does not remove [Variant] Remove Variant UTF-16 output rewriting #5474's encoded object-key ordering compatibility or the separate field-ID shadowing work in [Variant] Match Spark physical Parquet semantics for projected Variant columns #5550.

Dependency snapshot (2026-09-13): Comet main locks DataFusion 55.1.0 and Arrow/Parquet 59.3.0; #5868 locks DataFusion 55.0.0 and Arrow/Parquet 59.3.0. Links to the original #5407 implementation describe the compatibility branches to reassess, not the current main dependency. Checked upstream-fix entries do not by themselves mean Comet's released dependency contains the fix.