fix: support struct-typed scalar subquery results

[LinSimon-901101]

Which issue does this PR close?

Closes #5834.

Rationale for this change

Spark's merged subqueries produce struct-typed scalar results that currently force the consuming projection to fall back to Spark. Spark 4.2's MergeSubplans broadens this to queries without explicit subqueries, potentially taking Union and downstream aggregates off Comet as well.

What changes are included in this PR?

• Support non-empty struct scalar-subquery results with distinct field names and supported scalar or nested-struct fields.
• Transfer materialized results through Arrow IPC and cache owned native values within the expression's execution context.
• Preserve NULL-struct semantics, field order and planned metadata, while leaving existing non-struct paths unchanged.
• Add SQL, optimizer-plan and Rust regression tests, and document the supported scope.

Duplicate-name structs and unsupported field types retain fallback. This PR does not claim to resolve #4949 or the separate native shuffle/Parquet issues in #5605 and #5783. Existing malformed-UTF-8 limitations still apply.

How are these changes tested?

Local validation on macOS arm64:

• Before the upstream merge: focused regressions passed on Spark 3.4.3, 3.5.9, 4.0.4 and 4.1.3.
• After the upstream merge, at 918202206: native build succeeded; Spark 4.2.0 focused tests passed (3 tests), and Rust scalar-subquery tests passed (5 tests). Scalastyle and Spotless also passed.
• Red/green comparison on Spark 4.1.3: the same two regressions failed against the original base because of unsupported structs/projection fallback, then passed with the implementation.

The tests compare Spark results and assert native operator coverage, including the consuming projection and the Spark 4.2 Union/aggregate reproducer. The latter uses the existing test setup with Spark-to-columnar input conversion enabled.

Clippy and Apache RAT passed locally at 918202206. Scalafix also passed both the syntactic check and the semantic CHECK using Spark 4.0 / JDK 21.
These local checks do not cover the full CI matrix. No benchmarks were run, and no measured performance improvement is claimed.

[LinSimon-901101]

Local validation at 918202206 on macOS arm64:

• Spark 4.2.0 / JDK 21: 3 focused tests passed, 0 failed.
• Rust scalar-subquery tests: 5 passed, 0 failed.
• Native build, Scalastyle and Spotless passed.

The Spark 4.1.3 logs show the pre-merge red/green comparison: the same two regressions failed against the original base due to unsupported struct results/projection fallback, then passed with the fix. The baseline failures are expected.

Logs are attached below.
spark-4.2-focused.txt
rust-struct-subquery.txt
spark-4.1-baseline-expected failures.txt
spark-4.1-fixed-before upstream merge.txt

[sunchao]

Reviewed 91820220 against de1eb4f8. No verified P1/P2 findings.

Correctness

Previously, the struct returned by Spark's merged scalar subqueries caused the consuming projection to fall back. The recursive support check now admits non-empty structs with distinct field names and supported leaves. Array/map/interval fields, duplicate names, non-default collations and unsupported decimal scales retain fallback.

The one-row, one-column IPC envelope preserves the distinction between a null struct and a present struct containing null fields. Native decoding validates shape, field order, names, nullability and leaf types before restoring planned metadata. Timestamp fields retain UTC microseconds, while timestamp-without-time-zone fields remain unzoned. Spark still enforces scalar cardinality before native evaluation. Each task iterator builds a fresh native plan, so the owned cache stays within that execution; cache contents do not affect expression equality or hashing.

The SQL fixture covers supported leaves, nested nulls, empty results, separate subqueries and recursive fallback. The optimizer tests verify actual struct subqueries and a native consuming projection, including the Spark 4.2 union/aggregate case. I inspected the author's attached logs: they report three Spark 4.2 tests and five Rust tests passing at the stated head, plus the pre-merge Spark 4.1 red/green comparison. These are author-provided results, not independent CI. CI remains action_required with zero jobs; the sole successful check is labeling. I ran no local JVM/native tests. Canonical source checks covered the maintained Spark 3.5/4.0 branches; the required 3.4/4.1 branches were unavailable.

Performance

The first evaluation pays for row normalization, Arrow allocation, IPC serialization and decoding. Subsequent evaluations reuse the owned scalar, avoiding repeated IPC work per batch for that expression. Repeated references can still own separate caches, so this does not establish one transfer per logical subquery. Primitive transfer paths remain unchanged apart from the empty-cache lookup. No benchmark was run and no measured speedup is established. Could you add a focused comparison against the previous Spark fallback, varying struct width and input-batch count, to quantify first-transfer cost and how well the cache amortizes it?

Design

Reusing the existing Arrow writer keeps value transfer aligned with the project's supported primitive representations. The Scala support gate and native wire validation serve separate purposes: the former decides fallback before execution, while the latter rejects a malformed or mismatched result. Metadata restoration checks the value shape rather than introducing casts. The change addresses the struct-transfer seam and explicitly retains the existing malformed-UTF-8 limitation and separate scan/shuffle limitations.

Abstraction & complexity

The implementation adds a focused serializer, decoder and execution-local cache without a new plan protocol or global cache. The explicit equality/hash implementations preserve the original three identity fields while excluding mutable state. Recursive normalization and metadata alignment are limited to the supported struct shape; the added tests exercise those responsibilities directly. I found no additional abstraction or simplification that warrants a pre-merge change.

[sunchao]

Follow-up at unchanged 91820220 / base de1eb4f8:

[P2] Regenerate q9’s approved plan for the newly native struct-subquery projection. All five exec jobs fail only check simplified (tpcds-v1.4/q9): 3.4, 3.5, 4.0, 4.1, 4.2. The unchanged golden expects Project [COMET: Unsupported data type: StructType(...)]; each actual plan has CometColumnarToRow → CometProject and reports 38 accelerated operators instead of 37. Please regenerate and commit the affected approved plan using the documented golden-file workflow, then rerun the supported-profile plan-stability checks.

These are actual plan-comparison failures at the exact tested H/B merge. The native producer and JVM consumers share artifact 10307852860 and its verified digest. The merged-struct result/operator test and scalar_subquery_struct.sql pass on all five profiles; the five Rust struct tests also pass. This supports an expected-plan omission, without demonstrating a query-result regression. I replayed the captured plan comparison locally; I did not run Spark/Rust tests or benchmarks locally. Other CI jobs remain pending.

[LinSimon-901101]

Thanks for the review; I've regenerated q9's shared golden file to reflect the native CometProject, with no production code or assertion changes.
Both complete TPC-DS plan-stability suites passed locally on all five Spark profiles (3.4–4.2) with golden generation disabled: 129 tests per version, 0 failures.
Spark 3.4 used JDK 17 locally rather than CI's JDK 11, and CI validation of the update remains pending.