Skip to content

[python] Expand native read coverage for nested and streaming reads - #10009

Merged
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-coverage-expansion
Sep 20, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-coverage-expansion

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Use the Rust reader for more PyPaimon output paths instead of falling back to Python for nested projection, row kinds, and continuous streaming frames.

Changes

  • forward nested name paths and row-kind requests to pypaimon-rust
  • flatten native nested ROW and literal MAP-key results with parent-null propagation
  • preserve the public _row_kind schema through predicate-only output projection
  • share batch ReadBuilder projection semantics with StreamReadBuilder
  • use native planning for eligible initial and delta streaming frames, with safe fallback for bucket filters, authorization, snapshot mismatch, and setup errors
  • cover Parquet shared-shredding MAP, ORC, Avro, ROW, optional Mosaic, primary-key merge, partial-update, schema rename, Data Evolution, predicate, limit, parallelism, and streaming reads

Depends on apache/paimon-rust#887. pypaimon-rust has not been released, so this intentionally targets the paired API without a legacy compatibility shim.

Tests

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with a freshly built apache/paimon-rust#887 native extension. The six targeted Python suites produced 171 passes and one failure in the new optional Mosaic subcase below. I also reported the separate empty-string MAP-key decoding issue on the paired Rust PR.

'top.with.dot': 'third'},
]
formats = ['parquet', 'orc', 'avro', 'row']
if importlib.util.find_spec('mosaic') is not None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Gate the Mosaic subcase on nested-ROW support, not just importability

paimon-mosaic==0.1.0 satisfies the existing optional dependency (paimon-mosaic>=0.1.0 in setup.py) and makes this condition true, but its writer cannot write the nested payload Struct used by this test. With that supported dependency version and the freshly built #887 extension, this test fails while creating the fixture, before it reaches either reader: RuntimeError: writer open failed: unsupported DataType: Struct([Field { name: "details", data_type: Struct(...) }, ...]). The other 171 tests in the targeted suites passed.

Please gate this subcase on a Mosaic version/capability that supports the schema, adapt its fixture to supported types, or explicitly raise the optional dependency minimum if requiring a newer writer is intentional. Otherwise installing an allowed Mosaic version makes the new native-read suite fail for reasons unrelated to the reader being tested.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the update. The Mosaic test now probes nested-ROW write support instead of assuming that an importable package supports the fixture. With paimon-mosaic 0.1.0, the previously failing test now passes by excluding that unsupported subcase; I also checked that unrelated writer errors still propagate. Using a freshly rebuilt apache/paimon-rust#887 extension at 304703c6771c6389147823f51afed66dc7e8337d, all 172 targeted Python tests and four independent regression checks pass. The changed file passes the project flake8 configuration. No remaining blocking findings. LGTM.

@JingsongLi
JingsongLi merged commit 1da1446 into apache:master Sep 20, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants