[Variant] Replace missing-key metadata repair with native permissive reconstruction

[peterxcli]

What is the problem the feature request solves?

#5868 uses extend_shredded_metadata to collect every shredded schema key, build an expanded dictionary, and remap residual field IDs before Arrow unshredding. Spark instead creates output metadata while reconstructing the row.

Use a fresh writable output dictionary during native reconstruction, while keeping the input dictionary for decoding residuals. This removes the need to repair the input before reconstruction.

Verified behavior and existing support

Spark's object reconstruction registers a present typed field by its schema name. It does not require that name in the input dictionary. Residual field names are resolved using the input dictionary and registered in the new output dictionary.

Direct probes on Spark 4.0.4, 4.1.3, and 4.2.0 confirmed:

• Empty metadata plus typed a=1 reconstructs {"a":1}; output metadata contains a.
• A valid but missing typed field contributes neither a field nor a dictionary entry.
• Input metadata containing only z, residual {"z":9}, and typed a=1 reconstructs {"a":1,"z":9}. Source ID z=0 becomes output ID z=1.
• A present empty-string key is reconstructed too.

Arrow's current unshredder uses VariantValueArrayBuilder::builder_ext, which binds a ReadOnlyMetadataBuilder to the input dictionary. A missing present key can therefore reach the panic tracked by apache/arrow-rs#11069. Reproduced on the pinned Arrow 59.3.0.

The required writable builder already exists: VariantArrayBuilder implements VariantBuilderExt and finalizes and resets its dictionary for each row. An isolated Arrow 59.3.0 probe using this builder successfully remapped z=0 to z=1 while retaining an explicitly wide residual Int64(9).

Concrete implementation

1. Add opt-in MetadataMode::Rebuild to the proposed unshred_variant_with_options API from [Variant] Consolidate Spark-compatible missing-value validation and errors #5977. Keep the existing input-metadata-preserving mode as the default. In rebuild mode, feed the existing UnshredVariantRowBuilder::append_row traversal into VariantArrayBuilder, producing both value and metadata columns. Keep the input row's VariantMetadata as a separate argument for decoding.

2. Register typed object field names only when the field is emitted. Preserve schema traversal order, visiting children before the next sibling, then residual fields in source order. Let ObjectFieldBuilder::append_null omit missing fields. Do not prepopulate the output dictionary from all schema keys or unused input entries.

3. Remap residuals during that same traversal. For each residual object field, resolve its name through the source metadata, insert it into the output dictionary, and recursively append its value. Traverse objects inside residual lists too. The existing logical append path (ValueBuilder::try_append_variant / ObjectBuilder::try_insert) provides this remapping. Replace StructUnshredVariantBuilder's insert_bytes shortcut in rebuild mode; the root residual path must also use logical append. Raw object/list copies retain old IDs and are incorrect when dictionaries differ. Scalar representation must remain intact.

4. Preserve validity checks: malformed source metadata or out-of-range residual IDs are errors, not missing keys to invent. Preserve the partial-object schema-name conflict check even when the corresponding typed field is absent in this row. Use fallible insertion/decoding and propagate errors. Skip SQL NULL roots and inactive typed children; validate required states according to [Variant] Consolidate Spark-compatible missing-value validation and errors #5977.

5. In Comet, opt into rebuilding and remove extend_shredded_metadata, its collect_keys traversal, and the extended.as_ref().or(metadata) selection. Remove only remapping made necessary by that extension. Keep target_metadata plumbing still needed by the separate empty-key repair, and retain legacy residual-order handling. Keep the existing Spark output rewrite until [Variant] Remove the extra Spark byte-reconstruction pass #5978 supplies the complete byte contract.

Verification and completion

• Cover missing keys at the root, nested objects and lists, empty keys, repeated names across nesting levels, absent fields, explicit Variant null, and parent SQL NULL.
• Include different dictionaries per row and sliced lists. Add residual objects nested inside lists so a raw-copy shortcut cannot silently leave stale IDs.
• Include malformed metadata/IDs and duplicate typed/residual names to prove permissive key registration does not accept unrelated corruption.
• Retain normalize_shredded_objects_extend_metadata_and_preserve_missing_fields and the native scan assertions in CometVariantProjectionSuite. Compare full output bytes with supported Spark versions through the existing final encoding path.
• Remove the input-extension pass only after these checks pass with a released, pinned upstream implementation. Measure allocation traffic and scan time with the existing benchmarks; add a missing-key fixture because the current ordinary shredded fixtures contain all keys. Keep results in the implementing PR description.

Additional context

Parent: #5477; current implementation: #5868. The new API is a proposal. The format requires shredded names in metadata, so this remains explicit permissive reading.

apache/arrow-rs#11069 can be fixed by returning an error; that does not complete this issue. Completion requires successful native reconstruction and deletion of the repair, with no Spark fallback. #5978 builds on this output-dictionary path for exact Spark encoding, including legacy traversal order and metadata flags.