[SPARK-59624][PYTHON] Move element-wise UDF conversion behind Arrow batch interfaces

[Spenserrrr]

What changes were proposed in this pull request?

• Add concat_batches, flatten_elementwise_inputs, and renest_elementwise_outputs to ArrowBatchTransformer.
• Represent flattened UDF inputs and outputs as RecordBatches while retaining per-level shape metadata for null, empty, nested, ListArray, and LargeListArray values.
• Route scalar and iterator pandas/Arrow element-wise UDF conversion through batch-level APIs while retaining the row-at-a-time Python UDF invocation contract.

Why are the changes needed?

• The element-wise worker previously extracted individual Arrow arrays, called a private column converter, and reassembled converted arrays itself.
• Keeping conversion and structural transformations behind RecordBatch interfaces reduces worker coupling and lets lower-level conversion implementations change without rewriting worker orchestration.

Does this PR introduce any user-facing change?

No. This is an internal refactor that preserves existing UDF inputs, outputs, validation, and iterator behavior.

How was this patch tested?

• Added test_concat_batches and test_flatten_elementwise_inputs_and_renest_outputs, which require the new transformer APIs.
• Ran the conversion and higher-order-function UDF suites under pandas 2/PyArrow 24 and pandas 3/PyArrow 25; both profiles passed 109 tests and 117 subtests.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5)

[Spenserrrr]

Hi @gaogaotiantian @Yicong-Huang! When I worked on #58751, I noticed that the element-wise read path still handled per-array conversion directly in worker.py, so I opened this PR. It moves the flatten/re-nest operations behind ArrowBatchTransformer and uses the batch-level ArrowToPandasConversion.to_pandas API. Could you take a look when you have time and let me know whether this boundary matches what you had in mind? Thanks!