[Feature] Atomic schema evolution for chain tables

[hbgstc123]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Background

Chain tables use three branches — main, snapshot, and delta — each maintaining its own independent schema log (schema/schema-N). The current documentation requires users to keep schemas consistent across all three branches manually (e.g. when setting up fallback branches or partition expiration options).

Today, a single logical ALTER TABLE on the base table only commits schema to one branch. There is no catalog-level coordination to update main, snapshot, and delta atomically.

Problem

Because each branch commits schema independently via SchemaManager.commit() → FileIO.tryToWriteAtomic, partial failures can leave the chain group in an inconsistent state:

1. Immediate read failure: FallbackReadFileStoreTable.validateSchema() / ChainGroupReadTable.newScan() checks row type and primary key consistency across branches. If e.g. ADD COLUMN succeeds on main but fails on snapshot/delta, the entire chain table becomes unreadable at scan planning time.

2. Operational burden: Users must remember to run the same DDL three times (main + snapshot + delta), which is error-prone and not enforced by the catalog.

Example failure after partial ADD COLUMN:

main latest:     schema-11, fields = [a, b, c]
snapshot latest: schema-10, fields = [a, b]
delta latest:    schema-10, fields = [a, b]

→ chain reader fails with "does not have the same row type" before any data file is read.

Current behavior (as of master)

• AbstractCatalog.alterTable delegates to alterTableImpl for a single identifier; no chain-table-aware multi-branch sync.
• FileStoreTableFactory.createChainTable loads snapshot/delta schemas from their respective branch-local SchemaManager.
• Docs explicitly state: "Chain table should ensure that the schema of each branch is consistent."
• No rollback if multi-branch DDL is attempted manually and fails midway.

Solution

1. Main branch schema log becomes the single canonical source for shared fields / field IDs / PK / partition keys.
2. One atomic write to main/schema/schema-N publishes the schema for the entire chain group.
3. Branch-specific options (e.g. different bucket per branch) are stored in the canonical main schema and materialized per branch at read/write time.
4. Legacy schemas before a cutover point continue to be resolved from branch-local schema files for backward compatibility.

Example: canonical main schema with per-branch options

Shared columns / PK / partition keys live in the top-level TableSchema. Per-branch effective options are stored as JSON strings in options (exact key names TBD):

{
  "id": 25,
  "fields": [
    {"id": 0, "name": "id", "type": "BIGINT NOT NULL"},
    {"id": 1, "name": "dt", "type": "STRING NOT NULL"}
  ],
  "partitionKeys": ["dt"],
  "primaryKeys": ["id", "dt"],
  "options": {
    "chain-table.enabled": "true",
    "scan.fallback-snapshot-branch": "snapshot",
    "scan.fallback-delta-branch": "delta",

    "bucket": "64",
    "bucket-key": "id",

    "chain-table.member-options.snapshot": "{\"bucket\":\"32\",\"bucket-key\":\"id\"}",
    "chain-table.member-options.delta": "{\"bucket\":\"128\",\"bucket-key\":\"id\"}"
  }
}

At resolve time, each branch gets a normal TableSchema with the same id, fields, PK, and partition keys, but different effective options:

Branch	Effective bucket	Source
main	64	top-level options
snapshot	32	member-options.snapshot
delta	128	member-options.delta

A base-table ALTER TABLE ... SET TBLPROPERTIES ('bucket' = '96') would update all three members in one canonical commit. An explicit ALTER TABLE t$branch_delta SET TBLPROPERTIES ('bucket' = '256') would only update member-options.delta, still via a single main schema write.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!