Skip to content

[python] Fix Arrow large_string compatibility in PyPaimon - #10002

Open
Zouxxyy wants to merge 2 commits into
apache:masterfrom
Zouxxyy:xinyu/arrow-string-layouts
Open

Zouxxyy wants to merge 2 commits into
apache:masterfrom
Zouxxyy:xinyu/arrow-string-layouts

Conversation

@Zouxxyy

@Zouxxyy Zouxxyy commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Purpose

PyPaimon rejects Arrow large_string inputs even when their values fit the table's existing STRING representation. This change accepts the logical type and normalizes string layouts at the core write boundary.

  • Map Arrow string and large_string to Paimon STRING, including nested schema fields.
  • Validate the existing schema contract, then safely convert large_string to string before routing, buffering or writing. Apply the same preparation to Table, RecordBatch, precomputed-bucket and postpone writes.
  • Remove the Daft catalog and Ray shuffle string workarounds. LeRobot schema guards use the shared compatibility check. Existing adapter value conversions and column policies remain in place.

Only string layouts are normalized; field names, nullability, metadata, numeric types and BYTES/BLOB semantics are retained. The buffer and read implementations are unchanged. Arrow's native nested-cast and offset limits still apply: unsupported input is rejected before reaching a data writer. This change does not add support for oversized string arrays or string_view, and does not change dependency versions.

Tests

  • PyArrow 6.0.1 / Python 3.6: 13 focused tests passed, including rejection of unsupported nested casts before routing.
  • PyArrow 12.0.1: 12 focused tests passed; 1 Arrow-6-only test skipped.
  • PyArrow 16.1.0: 157 core/write tests passed; 1 Arrow-6-only test skipped.
  • PyArrow 19.0.1: 505 core and adapter regression tests passed; 2 skipped (Arrow-6-only test and existing optional video-decoder test).
  • The focused schema tests are included in the legacy Python 3.6/3.7 CI test selection.
  • Original Parquet input containing large_string: all 4 rows roundtripped unchanged.
  • Changed-file Flake8, license headers, shell syntax and git diff --check passed.

@Zouxxyy
Zouxxyy marked this pull request as draft September 20, 2026 08:47
@Zouxxyy
Zouxxyy marked this pull request as ready for review September 20, 2026 09:09
@Zouxxyy Zouxxyy changed the title [python] Unify Arrow string layout handling [python] Normalize Arrow large_string inputs before writing Sep 20, 2026
@Zouxxyy Zouxxyy changed the title [python] Normalize Arrow large_string inputs before writing [python] Fix Arrow large_string compatibility in PyPaimon Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant