Skip to content

[python] Preserve retained snapshots in Ray row ID reads - #10042

Merged
JingsongLi merged 2 commits into
apache:masterfrom
TheR1sing3un:python-ray-retained-tag-read
Sep 21, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
TheR1sing3un:python-ray-retained-tag-read

Conversation

@TheR1sing3un

@TheR1sing3un TheR1sing3un commented Sep 21, 2026

Copy link
Copy Markdown
Member

Purpose

Ray read_by_row_id fails with Snapshot id '...' doesn't exist when a tag retains a snapshot whose main metadata file has expired. The reader converted the tag to a snapshot ID and resolved that ID again during planning.

Reuse the table's existing resolved-snapshot copy so planning and Ray workers carry the retained metadata and historical schema. A lazy Dataset keeps its captured snapshot even if the tag is replaced before execution. This also supports tags selected through scan.version. If a tag moves across schema versions during initial resolution, reject the inconsistent view with a retryable error.

Tests

  • pytest -q paimon-python/pypaimon/tests/ray_read_by_row_id_test.py: 27 passed.
  • Regression coverage includes expired snapshot metadata, both tag selectors, schema evolution, duplicate row IDs, tag replacement before lazy execution, and a tag moving across schema versions during resolution.
  • Flake8, changed-file license checks, Python 3.6 grammar checks, and git diff --check passed.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED (triage: GO)
Implementation: CLEAN

The retained-tag failure is concrete: resolving a tag to a snapshot ID and then reopening the expired main snapshot metadata loses a still-readable view. Carrying the resolved snapshot into planning and lazy Ray execution addresses that failure and keeps the historical schema. The tests cover expired metadata, both tag selectors, schema changes, and a tag moved before execution. I found no blocking issue in the changed path.

@JingsongLi
JingsongLi merged commit dd2bc70 into apache:master Sep 21, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants