Skip to content

[python] Generate manifest sidecars on writes - #9956

Draft
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:codex/python-manifest-sidecar-write
Draft

leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:codex/python-manifest-sidecar-write

Conversation

@leaves12138

Copy link
Copy Markdown
Contributor

Purpose

Extract the remaining PyPaimon writer integration from #9743. This builds on the merged v1 codec/block-pruned reader in #9908 and the manifest-reference retry fix in #9921; it does not change the sidecar format or Java implementation.

  • Generate sidecars for ordinary and rolling manifest writes, including manifests produced by minor compaction.
  • Publish _EXTRA_FILES only after both immutable outputs close successfully. manifest.sidecar.enabled controls reads and writes and continues to inherit manifest-sort.enabled when unset.
  • Clean up newly created manifest/sidecar pairs on write or preparation failure, including previously completed rolling outputs. Preserve existing manifests and propagate cancellation. Commit outcomes that may be uncertain keep the existing no-cleanup behavior.
  • Keep coarse RowID bounds unknown for invalid ranges, consistent with sidecar coverage.
  • Document writer behavior and exercise the real writer in the existing pruning tests.

Tests

  • Python 3.11: 276 passed across manifest, commit/retry, explain, cache, file-type, and bucket-pruning suites.
  • Python 3.6: 116 passed across sidecar, block-index, and file-type suites.
  • Flake8 passed with paimon-python/dev/cfg.ini; git diff --check passed.
  • Added coverage for explicit disablement, write/close failures, interruption, rolling cleanup, commit cleanup ownership, and invalid RowID bounds.
  • Additional isolated Python-write/Java-read check: 12 manifests, 1,806 entries, null/deflate/Zstandard compression, ordinary and rolling outputs, an empty manifest, and 108 block-pruned read comparisons all matched the full Java reads.
PYTHONPATH=paimon-python python -m pytest \
  paimon-python/pypaimon/tests/manifest \
  paimon-python/pypaimon/tests/file_store_commit_test.py \
  paimon-python/pypaimon/tests/read_builder_explain_test.py \
  paimon-python/pypaimon/tests/caching_file_io_test.py \
  paimon-python/pypaimon/tests/file_type_test.py \
  paimon-python/pypaimon/tests/pushdown_bucket_test.py -q

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant