[SPARK-52729][SQL] Add MetadataOnlyTable and CREATE/ALTER VIEW support for DS v2 catalogs

[cloud-fan]

What changes were proposed in this pull request?

This PR exposes a DS v2 API for metadata-only tables (read side), CREATE VIEW, and ALTER VIEW ... AS (write side) so that third-party v2 catalogs can participate in Spark's resolution and creation flows without reimplementing read/write themselves.

The headline API choice is a new RelationCatalog interface that owns the cross-cutting rules for catalogs that expose both tables and views. TableCatalog and ViewCatalog remain strict single-kind APIs; mixing the two requires RelationCatalog. Details in section 4.

1. Read path — MetadataOnlyTable:

• New Table implementation that carries a TableInfo and delegates everything to it. Catalogs return it from TableCatalog.loadTable for data-source tables (Spark interprets the provider via V1 paths), and from RelationCatalog.loadRelation wrapping a ViewInfo for views (Spark expands the view text). Downstream consumers distinguish via getTableInfo() instanceof ViewInfo.
• Analyzer.lookupTableOrView and RelationResolution.tryResolvePersistent detect MetadataOnlyTable and route through a new V1Table.toCatalogTable adapter to the existing V1 data-source / view machinery.

2. Shared DTO — TableInfo:

• TableInfo.Builder gains convenience setters that write reserved keys into properties: withProvider, withLocation, withComment, withCollation, withOwner, withTableType, plus withSchema(StructType). The read side (MetadataOnlyTable) and the write side (createTable(ident, TableInfo)) use the same struct. View-specific fields live on a typed subclass (see section 3) so they are not encoded as string properties.
• withProperties takes a defensive copy so convenience setters don't mutate the caller's map.

3. Typed view DTO — ViewInfo:

• ViewInfo extends TableInfo carries the view-specific fields that cannot be represented as string table properties: queryText, currentCatalog, currentNamespace (multi-part, never null; empty when no namespace was captured), sqlConfigs (unprefixed SQL config keys), schemaMode (BINDING / COMPENSATION / TYPE EVOLUTION / EVOLUTION), and queryColumnNames (mapping query output to the view's declared columns; empty in EVOLUTION mode).
• ViewInfo.Builder extends TableInfo.BaseBuilder<Builder> adds typed setters: withQueryText, withCurrentCatalog, withCurrentNamespace, withSqlConfigs, withSchemaMode, withQueryColumnNames. The inherited TableInfo.BaseBuilder setters (schema, properties, owner, comment, collation, etc.) are available on the same builder so view and table writes share one fluent API.
• The ViewInfo constructor stamps PROP_TABLE_TYPE = TableSummary.VIEW_TABLE_TYPE into properties() so catalogs and generic viewers reading PROP_TABLE_TYPE from the properties bag (e.g. TableCatalog.listTableSummaries default impl, DESCRIBE) classify the entry as VIEW without requiring authors to remember withTableType(VIEW).
• ViewInfo is the typed payload returned by ViewCatalog.loadView and accepted by createView / replaceView. It still extends TableInfo so a RelationCatalog can opt into the single-RPC perf path by returning a MetadataOnlyTable(ViewInfo) from loadRelation (see section 4); pure view-only catalogs never see TableInfo directly because the typed builder covers everything they construct.

4. API design — RelationCatalog, ViewCatalog, the orthogonal split:

This PR's API surface is shaped by two principles:

• Orthogonal interfaces. Every TableCatalog method behaves as if views did not exist; every ViewCatalog method behaves as if tables did not exist. A pure TableCatalog impl and a pure ViewCatalog impl never need to know about the other kind.
• Single identifier namespace. When a catalog exposes both, tables and views share one keyspace within a namespace; the same identifier cannot resolve to both at the same time.

These two together require a third interface for the combined case — the orthogonality forbids cross-type clauses on TableCatalog / ViewCatalog methods, and the shared namespace forces cross-type rules to exist somewhere. That somewhere is RelationCatalog.

• ViewCatalog: plugin-facing API for views. listViews(namespace), loadView(ident) returning ViewInfo, createView(ident, ViewInfo), replaceView(ident, ViewInfo), dropView(ident), default viewExists(ident), default invalidateView(ident), default createOrReplaceView(ident, ViewInfo). The default createOrReplaceView impl tries replaceView and falls back to createView on NoSuchViewException (two RPCs, non-atomic across the pair); catalogs that can do an atomic upsert in a single transactional call should override to collapse to one RPC and make the swap atomic. No staging variant — replaceView is a single atomic-swap call. Class Javadoc reads as a strict view-only API and points mixed implementers at RelationCatalog.
• TableCatalog: unchanged surface, but the loadTable Javadoc reverts to strictly tables-only (the perf opt-in moved off this method, see below). Cross-type clauses scattered through method docs are removed; class doc points mixed implementers at RelationCatalog.
• RelationCatalog extends TableCatalog, ViewCatalog: the home for the combined case. Class Javadoc owns the two principles and two per-method contract tables — active rejection (5 write-side methods that throw on cross-type collision: createTable, renameTable, createView, createOrReplaceView, replaceView) and passive filtering (10 read / non-collision-mutation methods that behave as if the wrong kind doesn't exist). Catalogs that implement TableCatalog and ViewCatalog directly without RelationCatalog are rejected at Catalogs.load with a clear pointer at the right interface.
• Single-RPC perf entry points on RelationCatalog:
  • loadRelation(ident) — the resolver's per-identifier read path. Returns a regular Table for a table or a MetadataOnlyTable wrapping a ViewInfo for a view; saves the cold-cache loadTable → loadView two-step. Crucially, this lives only on RelationCatalog — TableCatalog.loadTable is strictly tables-only, which means tableExists's default impl (loadTable(ident) != null) is correct under any catalog with no override required. The earlier shape (perf opt-in via loadTable returning MetadataOnlyTable(ViewInfo)) silently broke tableExists's default impl and was a foot-gun on three independent framework call sites.
  • listRelationSummaries(namespace) — unified listing of tables and views as TableSummary[] with kind preserved. Default impl unions listTableSummaries + listViews (two round trips); override to fetch in one.
• Implementer ergonomics: RelationCatalog provides default impls of loadTable, loadView, tableExists, and viewExists that derive from loadRelation (a MetadataOnlyTable + ViewInfo discriminator routes the result to the right kind). A RelationCatalog author writes the read-side lookup once in loadRelation; the four kind-specific accessors come for free. They can still be overridden when a kind-specific path is materially cheaper than the unified one, but they're no longer required boilerplate.
• Resolver wiring: Analyzer.lookupTableOrView and RelationResolution.tryResolvePersistent check RelationCatalog first and call loadRelation. Otherwise they fall through to the existing two-step (loadTable if TableCatalog, then loadView if ViewCatalog). The MISSING_CATALOG_ABILITY.VIEWS gate in the resolver and CheckViewReferences for CREATE / ALTER VIEW continues to fire for catalogs that aren't ViewCatalog (and therefore aren't RelationCatalog).

5. Write path — DS v2 CREATE VIEW:

• DataSourceV2Strategy routes CreateView(ResolvedIdentifier(catalog, ident), …) to CreateV2ViewExec(catalog: ViewCatalog, …), which uses an act-then-decode pattern to keep the happy path at one RPC: plain CREATE calls createView; CREATE OR REPLACE calls createOrReplaceView (one RPC for catalogs that override the default; two for those that don't). CREATE VIEW IF NOT EXISTS short-circuits via an upfront viewExists probe so the view body isn't analyzed when the view already exists (matches v1 CreateViewCommand.run). On ViewAlreadyExistsException from the catalog (cross-type collision in a RelationCatalog, or a race for plain CREATE / OR REPLACE), the exec decodes with one tableExists probe: if a table is at the identifier, plain CREATE surfaces TABLE_OR_VIEW_ALREADY_EXISTS and CREATE OR REPLACE surfaces EXPECT_VIEW_NOT_TABLE.NO_ALTERNATIVE, while IF NOT EXISTS is a no-op (v1-parity); if a view is at the identifier, plain CREATE re-raises as viewAlreadyExists and IF NOT EXISTS is a no-op (lost the race for the same view). Because tableExists is now contract-clean (no perf-opt-in leak), the decoder gives the right error in every catalog shape.
• The exec builds the ViewInfo via a V2ViewPreparation trait reusing v1 ViewHelper helpers (aliasPlan, sqlConfigsToProps) to populate a ViewInfo.Builder with the current session's captured catalog/namespace and SQL configs. Cyclic-reference detection and auto-generated-alias rejection run once at analysis time in CheckViewReferences (see section 7).
• CreateView logical plan extends AnalysisOnlyCommand (same shape as V2CreateTableAsSelectPlan) so HandleSpecialCommand.markAsAnalyzed captures referredTempFunctions from AnalysisContext. The v1 rewriting path (ResolveSessionCatalog → CreateViewCommand) is unchanged.

6. Write path — DS v2 ALTER VIEW ... AS:

• AlterViewAs logical plan also extends AnalysisOnlyCommand so referredTempFunctions is captured for the non-session path.
• DataSourceV2Strategy routes AlterViewAs(ResolvedPersistentView(catalog, ident, _), …) to AlterV2ViewExec(catalog: ViewCatalog, …), which calls replaceView (the single atomic-swap entry point — no separate staging variant, since view REPLACE writes only metadata).
• A V2AlterViewPreparation trait (extends V2ViewPreparation) calls catalog.loadView(ident) once and uses the result to preserve user TBLPROPERTIES, comment, collation, owner, and schema-binding mode when constructing the replacement ViewInfo. Session-scoped fields (SQL configs, query column names) are re-emitted by buildViewInfo() from the active SparkSession, matching v1 AlterViewAsCommand.alterPermanentView. A racing DDL between analysis and exec (the view dropped, or replaced with a non-view table in a RelationCatalog) surfaces NoSuchViewException / EXPECT_VIEW_NOT_TABLE rather than a stale-resolution error.
• ResolvedViewIdentifier.unapply (in ResolveSessionCatalog) replaces its assert(isSessionCatalog) with an if isSessionCatalog guard so non-session ResolvedPersistentView plans fall through to the v2 strategy instead of tripping the assertion.

7. Post-analysis check — CheckViewReferences:

• New rule wired into BaseSessionStateBuilder.extendedCheckRules. Rejects permanent views that reference temporary objects and rejects view bodies with auto-generated aliases for both CreateView and AlterViewAs (v2 paths). v1 CreateViewCommand / AlterViewAsCommand keep their existing exec-time safety net — Dataset-built commands can be constructed with isAnalyzed=true directly and bypass the analyzer's re-capture path.

8. Listing — SHOW TABLES / SHOW VIEWS:

• TableCatalog.listTables returns table identifiers only — views (in a RelationCatalog) are listed separately via ViewCatalog.listViews. listTableSummaries's default impl enumerates via listTables + loadTable and returns one summary per table. This is an intentional v2 divergence from v1 SHOW TABLES, which includes both tables and views; RelationCatalog.listRelationSummaries is the unified entry point if a RelationCatalog wants to expose both in one call. Routing the SHOW TABLES SQL command through listRelationSummaries for v1-parity output is left as a follow-up so this PR's API surface stays narrowly scoped.
• SHOW VIEWS on a non-session ViewCatalog is routed through a new ShowViewsExec that enumerates via ViewCatalog.listViews(namespace). ResolveSessionCatalog.ShowViews skips (via guard) for ViewCatalog catalogs so they fall through to this strategy; non-session, non-ViewCatalog catalogs still hit the existing MISSING_CATALOG_ABILITY.VIEWS rejection. v2 catalogs have no temp views, so the isTemporary column is always false (mirroring v1, which only sets it true for local/global temp views).

Why are the changes needed?

A v2 Table is not always backed by a connector that implements read/write. Catalogs like HMS and Unity Catalog store only metadata and rely on Spark to interpret the table provider as a data source or to execute the view SQL. Previously the only way to achieve that was a hack around V1Table, which leaks private v1 types into v2 connectors (example: https://github.com/unitycatalog/unitycatalog/blob/main/connectors/spark/src/main/scala/io/unitycatalog/spark/UCSingleCatalog.scala).

Separately, v2 catalogs had no public way to handle CREATE VIEW or ALTER VIEW. ResolveSessionCatalog rejected CREATE VIEW on any non-session catalog with MISSING_CATALOG_ABILITY.VIEWS, so third-party catalogs could not own view lifecycle at all.

The new ViewCatalog interface gives catalogs a clean view-shaped API, and the new RelationCatalog interface (extending both TableCatalog and ViewCatalog) is the single home for the cross-cutting rules of the combined case — orthogonality, the shared identifier namespace, the per-method active-rejection / passive-filtering tables, and the single-RPC perf entry points (loadRelation, listRelationSummaries). This separation keeps TableCatalog and ViewCatalog Javadocs as strict single-kind APIs (a connector author writing only one of them never sees cross-type clauses), and gives RelationCatalog implementers a single document with the full contract.

Does this PR introduce any user-facing change?

Yes to connector developers:

• Third-party v2 TableCatalog implementations can now return a MetadataOnlyTable from loadTable to delegate reads to Spark.
• Third-party v2 connectors that expose only views implement just ViewCatalog (listViews / loadView / createView / replaceView / dropView, plus the default helpers). Connectors that expose both tables and views implement RelationCatalog (which extends both TableCatalog and ViewCatalog); implementing the two interfaces directly is rejected at catalog initialization. View text, schema, captured current catalog+namespace, SQL configs, and temp-object-reference rejection are handled the same way as for session-catalog views.

No SQL-level or user-visible behavior change for existing deployments.

Remaining work (follow-up PRs)

This PR covers the core read path, CREATE VIEW (all shapes), ALTER VIEW ... AS, DROP VIEW, and SHOW VIEWS. The following view-scoped plans for DS v2 catalogs are not yet supported and are tracked for follow-ups. Until the follow-ups land, each currently surfaces a clean UNSUPPORTED_FEATURE.TABLE_OPERATION error (wired up in DataSourceV2Strategy and pinned by tests in DataSourceV2MetadataOnlyViewSuite), so users get a meaningful message rather than a generic planner failure:

• ALTER VIEW ... SET/UNSET TBLPROPERTIES — separate logical plans (SetViewProperties, UnsetViewProperties); need their own DataSourceV2Strategy cases backed by new TableChange routing.
• ALTER VIEW ... RENAME TO — RenameTable at the logical level; needs v2 view awareness and the catalog-side rename semantics.
• ALTER VIEW ... WITH SCHEMA BINDING — AlterViewSchemaBinding logical plan; needs the same treatment as ALTER VIEW AS (AnalysisOnlyCommand shape + v2 exec).
• DESCRIBE / SHOW CREATE TABLE / SHOW TBLPROPERTIES / SHOW COLUMNS on v2 views — currently route through ResolvedViewIdentifier which only matches session-catalog views; the v2 equivalents need dedicated handling.
• SHOW TABLES v1-parity output (include views) on a v2 catalog — TableCatalog.listTables now intentionally returns tables only; routing the SQL SHOW TABLES through RelationCatalog.listRelationSummaries for v1-parity is a separate piece of work.

How was this patch tested?

Two new test suites: DataSourceV2MetadataOnlyTableSuite (4 tests, table-side) and DataSourceV2MetadataOnlyViewSuite (56 tests, view-side + mixed-catalog).

DataSourceV2MetadataOnlyTableSuite — table side

• File-source reads: non-partitioned and partitioned MetadataOnlyTable round-trip through SELECT / INSERT / INSERT OVERWRITE.
• DESCRIBE TABLE EXTENDED: Name row shows the real catalog-qualified identifier, not the wrapper class.
• Fully-qualified column references: 3-part catalog.schema.table.col references resolve correctly through a MetadataOnlyTable. The mechanism: V1Table.toCatalogTable sets CatalogTable.multipartIdentifier = [catalog, namespace…, table], and the SessionCatalog change in this PR makes getRelation prefer that over the hardcoded spark_catalog qualifier when wrapping in a SubqueryAlias. (1-part and 2-part references already worked via last-part suffix matching; the test pins both cases plus the 3-part case to make the alias-source change visible in the diff.)

DataSourceV2MetadataOnlyViewSuite — view side + mixed catalogs

Catalog fixtures: TestingRelationCatalog (RelationCatalog), TestingViewOnlyCatalog (pure ViewCatalog), TestingTableOnlyCatalog (pure TableCatalog).

• Read path — view-text expansion with captured SQL configs; unqualified references resolved via single-part and multi-part captured currentCatalog + currentNamespace; pure ViewCatalog read path (no TableCatalog mixin) confirms the resolver's loadView fallback fires when loadTable is skipped.
• V1Table.toCatalogTable(ViewInfo) round-trip — multi-part captured namespace flows through structurally (including parts containing dots, which would break a string-encoded path); absent-currentCatalog branch yields an empty viewCatalogAndNamespace.
• CREATE VIEW — end-to-end CREATE; CREATE VIEW IF NOT EXISTS no-op vs. failure; CREATE OR REPLACE VIEW replacement (including cyclic-reference detection); too-few / too-many user-specified columns; rejection of references to temp function / temp view / temp variable; DEFAULT COLLATION propagation into ViewInfo; cross-type collision over a non-view table entry surfaces TABLE_OR_VIEW_ALREADY_EXISTS (plain) / EXPECT_VIEW_NOT_TABLE.NO_ALTERNATIVE (OR REPLACE) / no-op (IF NOT EXISTS) — v1-parity; PROP_OWNER stamped on stored TableInfo; rejection on a TableCatalog-only catalog with MISSING_CATALOG_ABILITY.VIEWS.
• ALTER VIEW … AS — body replacement; missing-view fails at analysis; rejection of references to temp function / temp view / temp variable; preserves user-set TBLPROPERTIES, PROP_OWNER, and SCHEMA EVOLUTION binding mode (v1-parity); re-captures the current session's SQL configs; cyclic-reference detection; rejection on a TableCatalog-only catalog with MISSING_CATALOG_ABILITY.VIEWS; pure ViewCatalog flow; CREATE OR REPLACE VIEW whose new body references a nonexistent table fails at the right phase.
• Multi-level namespace handling — cyclic-reference detection distinguishes views across multi-level namespaces (CREATE and ALTER); error messages render the full multi-level namespace; temp-object reference errors render the full multi-level namespace.
• DROP VIEW — drops a view on ViewCatalog; DROP VIEW IF EXISTS is a no-op on missing; rejection on a non-view table entry (v1-parity, EXPECT_VIEW_NOT_TABLE); rejection on a non-ViewCatalog catalog.
• SHOW TABLES / SHOW VIEWS — SHOW TABLES on a v2 catalog returns tables only; SHOW VIEWS returns views only (isTemporary=false throughout for v2); SHOW VIEWS … LIKE filters by name; rejection on a non-ViewCatalog catalog.
• Unsupported v2 view DDL / inspection pinning — ALTER VIEW SET/UNSET TBLPROPERTIES, ALTER VIEW WITH SCHEMA, ALTER VIEW RENAME TO, SHOW CREATE TABLE, SHOW TBLPROPERTIES, SHOW COLUMNS, DESCRIBE TABLE, REFRESH TABLE, ANALYZE TABLE, ANALYZE TABLE … FOR COLUMNS against a v2 view all surface UNSUPPORTED_FEATURE.TABLE_OPERATION; DESCRIBE TABLE … COLUMN surfaces a clean AnalysisException. Pins the current failure mode so a future regression to a generic planner error is caught in the diff.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude (Anthropic)

[cloud-fan]

I think the current view implementation which stores the original SQL text and a bunch of context is too convoluted to put into the public DS v2 API. It's better if the view text is context-independent. We are going to improve it in #51410

Before the improvement is done, we only allow to read DS v2 views that has context-independent SQL text.

[aokolnychyi]

Don't we store the current catalog and namespace in the view metadata? Do we expect the connectors to modify the view SQL text?

[cloud-fan]

My proposal is to let Spark modify the view text before saving it into the catalog, so that the catalog does not need to store the current catalog/namespace.

[aokolnychyi]

So all identifiers in the view text will always include the catalog name as the first name part and the table name as the last name part? How hard will it be to modify the original SQL text? Will it cause any surprises to the users if the original and the persisted SQL text differ?

[aokolnychyi]

So all identifiers in the view text will always include the catalog name as the first name part and the table name as the last name part? How hard will it be to modify the original SQL text? Will it cause any surprises to the users if the original and the persisted SQL text differ?

[cloud-fan]

Hive already did so and its view has both "view_text" and "original_view_text" fields. All identifiers should be fully qualified (with catalog name and namespace) in the view text.

[cloud-fan]

cc @aokolnychyi @gengliangwang

[gengliangwang]

how about SPARK_TABLE_OR_VIEW

[gengliangwang]

shall we explicitly mention what operations will be affected?(read/write/DDL/...)

[cloud-fan]

The capability has since been reshaped: TableCapability is gone in favor of the concrete MetadataOnlyTable for the read side, and a TableCatalogCapability.SUPPORTS_VIEW gate for the write side. The SUPPORTS_VIEW javadoc now spells out the affected operations explicitly (CREATE VIEW / CREATE OR REPLACE VIEW / CREATE VIEW IF NOT EXISTS via createTable, ALTER VIEW ... AS via dropTable+createTable or stageReplace, and the read-path round-trip through MetadataOnlyTable). PTAL and let me know if anything is still unclear.

[aokolnychyi]

I'd love to take a look on Monday.

[aokolnychyi]

I understand we may want to use the new API to expose views, but what about the case with Spark table? When would this be helpful?

[cloud-fan]

For example, UC does not want to rely on the Spark file source, but just set the table provider to the file source name and leave read/write to Spark. Today this is done by a hack with V1Table: https://github.com/unitycatalog/unitycatalog/blob/main/connectors/spark/src/main/scala/io/unitycatalog/spark/UCSingleCatalog.scala#L303

[aokolnychyi]

Ah, got it. The use case is catalogs that govern tables but not necessarily implement read/write logic.

Have we considered offering a generic V2 table implementation for built-in formats that would be accessible to external connectors? Essentially, a public version of V1Table that doesn't need to expose CatalogTable. If we go with the table capability approach, then each connector will have to implement a custom V2 table for it to be simply replaced as a Parquet table or view. Each connector would have to be aware of how the translation will be done in Spark. For instance, it seems like we assume that serdeProps will be with prefixed with option..

Just curious to know your thinking, I don't have a strong opinion here.

[cloud-fan]

I see, this makes sense. A concrete class is easier to use than a table capability.

[gengliangwang]

Shall we add this test case later?

[cloud-fan]

This will be fixed shortly after the PR is merged.

[gengliangwang]

This affects the table read/write operations but not DDL operations.

It seems a bit unclear. Before the change, DDL operations of DSV2 tables relies on Spark too.

[cloud-fan]

maybe we should just remove this line? DDL operations do not call read/write APIs of Table anyway.

[szehon-ho]

Makes sense, just wondering if some of it can be less bound to HMS concepts

[szehon-ho]

Just an opinion, serde is quite bound to HMS (?) and the newer DSV2 Catalog dont have that, will it make sense to abstract this (maybe something like 'storageProperties')? This is just for HMS table support?

[cloud-fan]

good point. I was trying to save the work of splitting serde properties from table properties, but API simplicity is more important.

[szehon-ho]

curious, do we not need to set inputFormat/outputFormat?

[cloud-fan]

V1Table itself does not access the input/output format when translating v1 table to v2 table. I think Hive table is never supported by DS v2 and we will not handle it here.

[szehon-ho]

Not entirely familiar with the end use case, but the API looks good now, thanks

[dongjoon-hyun]

Hi, @cloud-fan . Do you have a plan to proceed this forward?

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[gengliangwang]

LGTM