Support Spark-compatible duplicate Parquet field resolution

[ErikBPF]

Description

Implement Spark-compatible duplicate Parquet sibling-name resolution in the native reader, replacing the explicit rejection proposed in PR #5786 for issue #5783.

As reported in the PR review, Spark's behavior depends on where the duplicate occurs:

• Root group: with message spark_schema { optional int64 a; optional int64 a; optional int64 b; } and one raw row a=1, a=2, b=3, an explicit a bigint read schema returns [1] (first child wins).
• Nested structs: issue Native Parquet scan multiplies rows for a struct with duplicate field names #5783 demonstrates last-wins resolution through caseSensitiveParquetFieldMap.

The native Arrow decoder can combine distinct leaves into one column before schema adaptation, multiplying rows or failing with a decoder synchronization error. Resolution must select the correct physical leaf before decoding, rather than dropping duplicate logical fields afterward.

Acceptance criteria

Match Spark results, row counts, and error behavior for root, nested struct, array-element, and map-value duplicates using explicit read schemas. Cover case sensitivity and field-ID reads; preserve unrelated-column projection, filter pruning, and metadata-cache behavior. Replace the scan compatibility limitation once supported.

Related: #5783, #5786.

Additional reproduction: reader-dependent nested behavior

Independent testing on Spark 4.1.3 with Comet disabled, spark.sql.caseSensitive=true, and Spark's vectorized Parquet reader enabled (FileScan parquet Batched:true) did not produce a simple last-wins result. The fixture:

spark.range(3).coalesce(1)
  .selectExpr("named_struct('dup', id, 'dup', id + 100) as s")
  .write.parquet(path)
spark.read.schema("s struct<dup: bigint>").parquet(path)

returned the multiset {0, 100, 1}, rather than {100, 101, 102}. Result order is not asserted. The root duplicate regression separately confirmed first-wins under case-sensitive resolution.

Treat the first/last-wins observations above as specific reader/source observations, not a universal resolution contract. Characterize Spark behavior across versions and vectorized/non-vectorized readers before choosing semantics; preserve the fail-fast guard for selected ambiguous groups until correct resolution is supported.