[SPARK-55674][PYTHON] Optimize 0-column table conversion in Spark Connect

[Yicong-Huang]

What changes were proposed in this pull request?

Replace pa.Table.from_struct_array(pa.array([{}] * len(data), type=pa.struct([]))) with pa.Table.from_batches([pa.RecordBatch.from_pandas(data)]) in connect/session.py when handling 0-column pandas DataFrames. This is O(1) operation, regardless how many rows are there.

Why are the changes needed?

The original approach constructs len(data) Python dict objects ([{}] * len(data)), which is O(n). pa.RecordBatch.from_pandas is an O(1) operation regardless of the number of rows, as it reads row
count directly from pandas index metadata without allocating per-row Python objects.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[ueshin]

Can't we apply this to

spark/python/pyspark/sql/conversion.py

Line 289 in 01bfd80

return pa.RecordBatch.from_struct_array(pa.array([{}] * len(data), arrow_type))

?

[Yicong-Huang]

Can't we apply this to ？

Not for this case. This case is "data" is empty but schema is non empty, so we could not use convert and preserve the information from data: the columns will mismatch.

[ueshin]

LGTM, pending tests.

[HyukjinKwon]

Merged to master.