[python] Expand native read coverage for nested and streaming reads - #10009
Conversation
leaves12138
left a comment
There was a problem hiding this comment.
Reviewed with a freshly built apache/paimon-rust#887 native extension. The six targeted Python suites produced 171 passes and one failure in the new optional Mosaic subcase below. I also reported the separate empty-string MAP-key decoding issue on the paired Rust PR.
| 'top.with.dot': 'third'}, | ||
| ] | ||
| formats = ['parquet', 'orc', 'avro', 'row'] | ||
| if importlib.util.find_spec('mosaic') is not None: |
There was a problem hiding this comment.
[P2] Gate the Mosaic subcase on nested-ROW support, not just importability
paimon-mosaic==0.1.0 satisfies the existing optional dependency (paimon-mosaic>=0.1.0 in setup.py) and makes this condition true, but its writer cannot write the nested payload Struct used by this test. With that supported dependency version and the freshly built #887 extension, this test fails while creating the fixture, before it reaches either reader: RuntimeError: writer open failed: unsupported DataType: Struct([Field { name: "details", data_type: Struct(...) }, ...]). The other 171 tests in the targeted suites passed.
Please gate this subcase on a Mosaic version/capability that supports the schema, adapt its fixture to supported types, or explicitly raise the optional dependency minimum if requiring a newer writer is intentional. Otherwise installing an allowed Mosaic version makes the new native-read suite fail for reasons unrelated to the reader being tested.
leaves12138
left a comment
There was a problem hiding this comment.
Re-reviewed the update. The Mosaic test now probes nested-ROW write support instead of assuming that an importable package supports the fixture. With paimon-mosaic 0.1.0, the previously failing test now passes by excluding that unsupported subcase; I also checked that unrelated writer errors still propagate. Using a freshly rebuilt apache/paimon-rust#887 extension at 304703c6771c6389147823f51afed66dc7e8337d, all 172 targeted Python tests and four independent regression checks pass. The changed file passes the project flake8 configuration. No remaining blocking findings. LGTM.
Purpose
Use the Rust reader for more PyPaimon output paths instead of falling back to Python for nested projection, row kinds, and continuous streaming frames.
Changes
_row_kindschema through predicate-only output projectionDepends on apache/paimon-rust#887. pypaimon-rust has not been released, so this intentionally targets the paired API without a legacy compatibility shim.
Tests