[SPARK-55600][PYTHON] Fix pandas to arrow loses row count when schema has 0 columns on classic

[Yicong-Huang]

What changes were proposed in this pull request?

This PR fixes the row count loss issue when creating a Spark DataFrame from a pandas DataFrame with 0 columns in classic.

The issue occurs due to PyArrow limitations when creating RecordBatches or Tables with 0 columns - row count information is lost.

Why are the changes needed?

Before this fix:

import pandas as pd
from pyspark.sql.types import StructType

pdf = pd.DataFrame(index=range(5))  # 5 rows, 0 columns
df = spark.createDataFrame(pdf, schema=StructType([]))
df.count()  # Returns 0 (wrong!)

After this fix:

df.count()  # Returns 5 (correct!)

Does this PR introduce any user-facing change?

Yes. Creating a DataFrame from a pandas DataFrame with 0 columns now correctly preserves the row count in Classic Spark.

How was this patch tested?

Added unit test test_from_pandas_dataframe_with_zero_columns in test_creation.py that tests both Arrow-enabled and Arrow-disabled paths.

Was this patch authored or co-authored using generative AI tooling?

No

[holdenk]

CC @devin-petersohn if you've got the time for a quick review would appreciate it, I don't really know the semantics of a 0 column dataframe.

[devin-petersohn]

@Yicong-Huang Arrow batches keep pandas metadata around, would that be a better, more efficient way of keeping track of the original length?

In [3]: pa.RecordBatch.from_pandas(pd.DataFrame(index=range(5))).schema.pandas_metadata
Out[3]: 
{'index_columns': [{'kind': 'range',
   'name': None,
   'start': 0,
   'stop': 5,
   'step': 1}],
 'column_indexes': [{'name': None,
   'field_name': None,
   'pandas_type': 'int64',
   'numpy_type': 'int64',
   'metadata': None}],
 'columns': [],
 'creator': {'library': 'pyarrow', 'version': '18.1.0'},
 'pandas_version': '2.2.3'}

[Yicong-Huang]

@Yicong-Huang Arrow batches keep pandas metadata around, would that be a better, more efficient way of keeping track of the original length?

Thanks @devin-petersohn for the suggestion. I tested locally indeed for this edge case, using metadata would be much more efficient. changes pushed.

[devin-petersohn]

Thanks for the changes, LGTM

[Yicong-Huang]

cc @ueshin

[ueshin]

If this works more efficiently, shall we also use this in the similar parts, like:

• spark/python/pyspark/sql/connect/session.py

  Line 626 in 1e31b77

  _table = pa.Table.from_struct_array(pa.array([{}] * len(data), type=pa.struct([])))

• spark/python/pyspark/sql/conversion.py

  Line 289 in 1e31b77

  return pa.RecordBatch.from_struct_array(pa.array([{}] * len(data), arrow_type))

It can be in a separate PR, though.

[Yicong-Huang]

thanks. Let's address them with a separate PR, since this one is for a bug fix.

[ueshin]

Thanks! merging to master.