Skip to content

[core] Persist row ID reassignment plans and mark snapshots - #10008

Draft
leaves12138 wants to merge 2 commits into
apache:masterfrom
leaves12138:codex/reassign-snapshot-plan
Draft

leaves12138 wants to merge 2 commits into
apache:masterfrom
leaves12138:codex/reassign-snapshot-plan

Conversation

@leaves12138

@leaves12138 leaves12138 commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Purpose

Row-ID reassignment currently publishes a metadata-only OVERWRITE snapshot without a durable record of the mapping it applied. Extract the existing Assignment into DataEvolutionRowIdAssignment and add file serialization directly to it. The same assignment rewrites the manifests and is persisted before each commit attempt; there is no separate persisted-plan model.

Record row-id-reassign.plan in snapshot properties, referencing a versioned, checksummed file under manifest/. Persist the assigned row-ID bounds and absolute mappings by partition. Snapshot IDs and the manifest rewrite list remain in the commit context and are not serialized.

Keep the marker local to the reassignment commit: ordinary manifest replacement, manifest compaction, and rollback must not inherit it. Include plan files in snapshot/tag retention and orphan cleanup, and remove plans from definitively rejected commit attempts. This PR adds persistence and readback only; compact conflict rebasing will follow separately.

Tests

Passed 78 tests on dev2 with Java 11, using the normal Maven profile (without fast-build):

  • Plan round trips, concurrent append retries, null partitions, shifted range serialization, malformed plans, marker inheritance, snapshot/tag expiration, and orphan cleanup.
  • Existing manifest commit, rollback, snapshot expiration, and orphan cleanup regression tests.
  • Maven Checkstyle, Spotless, Enforcer, and Apache RAT checks.
mvn -pl paimon-core -am -DfailIfNoTests=false -DwildcardSuites=none \
  '-Dtest=DataEvolutionRowIdReassignerTest,RowRangeMappingIndexTest,FileStoreCommitTest#testManifestCompact+testCommitManifestWithProperties+testRollbackToAsLatestFileLevelDeleteIsVisibleToStreaming,ExpireSnapshotsTest#testMixedSnapshotAndTagDeletion+testSidecarsFollowSnapshotAndTagRetention,LocalOrphanFilesCleanTest#testOrphanCleanupProtectsReferencedSidecars+testPreservesManifestExtraFiles' test
mvn -pl paimon-core validate
mvn -pl paimon-core apache-rat:check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant