[python] Add row kind support for TableRead#7394
Draft
tub wants to merge 6 commits intoapache:masterfrom
Draft
Conversation
- Add FollowUpScanner hierarchy (base, delta, changelog) - Add IncrementalDiffScanner for diff-based streaming reads - Add sharding support to FileScanner - Add row kind support to TableRead for changelog streams Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…olidate tests, fix parallelism - Collapse repetitive module/class/method docstrings to one-liners in all scanner files (follow_up_scanner, delta, changelog, incremental_diff) - Remove TDD process commentary from test docstrings - Consolidate DeltaFollowUpScanner false-case tests into one parameterized test - Remove misleading commit_kind from ChangelogFollowUpScanner test mocks - Extract duplicated mock helpers to module-level functions - Fix max(8, ...) parallelism bug: respect user-configured parallelism - Remove obvious/redundant inline comments - Standardize license headers to comment style, merge double docstrings - Add clarifying docstring to ManifestListManager.read_all Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the include_row_kind feature out of this PR into a separate branch (python-streaming-1b2-row-kind) to keep the scanners PR focused on scanners and sharding only. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add include_row_kind option to TableRead that prepends a _row_kind string column to Arrow output. For RecordBatchReader (append-only tables) all rows default to "+I"; for RowIterator (primary-key tables) row kind is read per-row via OffsetRow.get_row_kind(). The feature is opt-in (include_row_kind=False by default) so existing read paths are unaffected. StreamReadBuilder in the next PR will enable it for changelog/streaming reads. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
include_row_kindparameter toTableReadfor streaming change tracking_row_kindstring column (+I,-D,+U,-U) to Arrow batches when enabledRecordBatchReader(default+I) andOffsetRow-based readers (fromRowKind)Stacked PR series
This is PR 1b part 2 in the Python streaming read series:
AsyncStreamingTableScan, consumer management ([python] Add StreamReadBuilder and AsyncStreamingTableScan #7350)paimon tail([python] Add paimon tail CLI for streaming table reads #7351)Incremental diff (vs 1b): tub/paimon@python-streaming-1b-scanners...tub:paimon:python-streaming-1b2-row-kind
Test plan
flake8passespython -m pytestpasses🤖 Generated with Claude Code