feat(table): build a catalog-free Table from a portable TableDescriptor#595
Draft
JunRuiLee wants to merge 1 commit into
Draft
feat(table): build a catalog-free Table from a portable TableDescriptor#595JunRuiLee wants to merge 1 commit into
JunRuiLee wants to merge 1 commit into
Conversation
Add TableDescriptor (path + full TableSchema + options) — a slim, versioned, cross-language JSON wire form for rebuilding a read-only table without a catalog lookup — plus Table::from_descriptor_json and the C FFI paimon_table_from_descriptor. Includes a golden byte-identical to the Java TableDescriptorSerializer output for cross-language validation.
JunRuiLee
force-pushed
the
feat/table-descriptor
branch
from
July 23, 2026 04:19
a8cb300 to
83f2db7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
No dedicated issue yet — this is the Rust reader half of a cross-language contract; the Java producer half is apache/paimon#8819.
Add a portable, versioned
TableDescriptorand a catalog-free way to build a read-onlyTablefrom it, so a non-Java engine (e.g. Doris BE via the C FFI) can rebuild a table for scan/read without a catalog lookup. Today the only way to obtain aTableisCatalog::get_table, which re-resolves the table (risking schema/snapshot drift vs. what the planner used) and ties the reader to a filesystem-catalog layout.The descriptor carries only what a reader needs —
path+ the fullTableSchema+ optionaloptions/database/name/branch— as language-neutral JSON. It is byte-compatible with the JavaTableDescriptorSerializeroutput (apache/paimon#8819).Brief change log
crates/paimon/src/table/descriptor.rs:TableDescriptor(serde DTO) withfrom_json(rejects unknownversionand non-mainbranch; wraps serde errors asDataInvalid), andTable::from_descriptor_json(top-levelpathwins overschema.options["path"]; identifier requires bothdatabase+nameor neither; plainFileIO;rest_env = None; descriptoroptionsare not merged into table semantics in v1).bindings/c/src/table.rs: C FFIpaimon_table_from_descriptor(json, options, len), mirroring the existingpaimon_catalog_get_tablehandle/free contract, with an ABI signature guard; freed viapaimon_table_free.schema/schema-NbySchemaManager; the descriptor pins the planned/current schema only.Tests
table::descriptorunit tests (13): parse valid v1; reject unknown version / non-main / empty branch; missing-field & malformed JSON →DataInvalid; build table + path-option injection; identifier synthesis and partial-identifier rejection; descriptor options do not affect table semantics; andparses_java_produced_golden_v1— parses the byte-identical Java-produced golden and asserts the ARRAY field reconstructs toDataType::Array.paimon-cFFI tests (3): build a table from a descriptor; reject unknown version; reject null JSON.API and Format
paimon::table::TableDescriptor+Table::from_descriptor_json, and C FFIpaimon_table_from_descriptor(additive; existing symbols unchanged).version = 1). No change to any existing on-disk or wire format.Documentation
None yet (new, evolving cross-language contract); docs can follow once it stabilizes together with apache/paimon#8819.