Skip to content

feat(table): build a catalog-free Table from a portable TableDescriptor#595

Draft
JunRuiLee wants to merge 1 commit into
apache:mainfrom
JunRuiLee:feat/table-descriptor
Draft

feat(table): build a catalog-free Table from a portable TableDescriptor#595
JunRuiLee wants to merge 1 commit into
apache:mainfrom
JunRuiLee:feat/table-descriptor

Conversation

@JunRuiLee

Copy link
Copy Markdown
Contributor

Depends on apache/paimon#8819 (the Java producer side of this contract). Draft until that PR's TableDescriptor JSON is finalized — the golden fixture here is byte-identical to the one produced there, so it must be updated in lockstep if the Java wire form changes.

Purpose

No dedicated issue yet — this is the Rust reader half of a cross-language contract; the Java producer half is apache/paimon#8819.

Add a portable, versioned TableDescriptor and a catalog-free way to build a read-only Table from it, so a non-Java engine (e.g. Doris BE via the C FFI) can rebuild a table for scan/read without a catalog lookup. Today the only way to obtain a Table is Catalog::get_table, which re-resolves the table (risking schema/snapshot drift vs. what the planner used) and ties the reader to a filesystem-catalog layout.

The descriptor carries only what a reader needs — path + the full TableSchema + optional options/database/name/branch — as language-neutral JSON. It is byte-compatible with the Java TableDescriptorSerializer output (apache/paimon#8819).

Brief change log

  • crates/paimon/src/table/descriptor.rs: TableDescriptor (serde DTO) with from_json (rejects unknown version and non-main branch; wraps serde errors as DataInvalid), and Table::from_descriptor_json (top-level path wins over schema.options["path"]; identifier requires both database+name or neither; plain FileIO; rest_env = None; descriptor options are not merged into table semantics in v1).
  • bindings/c/src/table.rs: C FFI paimon_table_from_descriptor(json, options, len), mirroring the existing paimon_catalog_get_table handle/free contract, with an ABI signature guard; freed via paimon_table_free.
  • Historical per-file schemas are still read from the immutable on-disk schema/schema-N by SchemaManager; the descriptor pins the planned/current schema only.

Tests

  • table::descriptor unit tests (13): parse valid v1; reject unknown version / non-main / empty branch; missing-field & malformed JSON → DataInvalid; build table + path-option injection; identifier synthesis and partial-identifier rejection; descriptor options do not affect table semantics; and parses_java_produced_golden_v1 — parses the byte-identical Java-produced golden and asserts the ARRAY field reconstructs to DataType::Array.
  • paimon-c FFI tests (3): build a table from a descriptor; reject unknown version; reject null JSON.

API and Format

  • New public API: paimon::table::TableDescriptor + Table::from_descriptor_json, and C FFI paimon_table_from_descriptor (additive; existing symbols unchanged).
  • New JSON descriptor format, versioned (version = 1). No change to any existing on-disk or wire format.

Documentation

None yet (new, evolving cross-language contract); docs can follow once it stabilizes together with apache/paimon#8819.

Add TableDescriptor (path + full TableSchema + options) — a slim, versioned,
cross-language JSON wire form for rebuilding a read-only table without a
catalog lookup — plus Table::from_descriptor_json and the C FFI
paimon_table_from_descriptor. Includes a golden byte-identical to the Java
TableDescriptorSerializer output for cross-language validation.
@JunRuiLee
JunRuiLee force-pushed the feat/table-descriptor branch from a8cb300 to 83f2db7 Compare July 23, 2026 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant