diff --git a/docs/docs/concepts/rest/database-versioning.md b/docs/docs/concepts/rest/database-versioning.md new file mode 100644 index 000000000000..7057acf1bdd1 --- /dev/null +++ b/docs/docs/concepts/rest/database-versioning.md @@ -0,0 +1,544 @@ +--- +title: "Database Branches and Tags" +--- + + + +# Database Branches and Tags + +Database branches and tags extend Paimon's existing table branches and tags to a group of tables. +The catalog server coordinates the table operations and records database membership and table +versions. Table data stays in the existing Paimon storage layout. + +:::info Implementation status + +The Java client, database-name selector parser, and REST contracts are implemented. A catalog server +must implement database branch/tag storage, table orchestration, retention, and forward execution. +The stateful client fixture is not a production reference-management server. + +::: + +## Relationship to table branches and tags + +Let `D = /v1/{prefix}/databases/{database}` and `T = D/tables/{table}`. Management operations use +physical database names, without a `$branch_` or `$tag_` suffix. + +| Operation | Table REST | Database REST | Request / response | +| --- | --- | --- | --- | +| List branches | `GET T/branches` | `GET D/branches` | Shared `ListBranchesResponse`. | +| Create branch | `POST T/branches` | `POST D/branches` | Shared `CreateBranchRequest`; no response body. | +| Drop branch | `DELETE T/branches/{branch}` | `DELETE D/branches/{branch}` | No request or response body. | +| Forward | `POST T/branches/{branch}/forward` | `POST D/branches/{branch}/forward` | Shared empty `ForwardBranchRequest`; no response body. | +| List tags | `GET T/tags` | `GET D/tags` | Shared `ListTagsResponse` and pagination parameters. | +| Create tag | `POST T/tags` | `POST D/tags` | `CreateDatabaseTagRequest`; no response body. | +| Get tag | `GET T/tags/{tag}` | `GET D/tags/{tag}` | `GetDatabaseTagResponse`. | +| Delete tag | `DELETE T/tags/{tag}` | `DELETE D/tags/{tag}` | No request or response body. | + +Successful mutations return HTTP `200` with no body, following the table API. Branch listing returns +names; tag listing is paged and accepts `maxResults`, `pageToken`, and `tagNamePrefix`. There is no +combined reference list or `/trees` resource. Database merge is deferred; there is no merge endpoint, +merge mode, or three-way conflict-resolution contract in this version. + +The database tag request retains table tag field names `tagName` and `timeRetained`, and uses +`fromBranch` instead of `snapshotId`. Tables have independent snapshot IDs, so one numeric snapshot +ID cannot identify a database version. Getting a database tag returns its name, source branch and +optional creation/retention metadata. Read its table versions through the existing table APIs. + +Branch names and tag names belong to separate namespaces, as with table branches and tags. A branch +and a tag may have the same name. Both names match `[A-Za-z0-9][A-Za-z0-9._-]{0,127}`. Database tag +names are database-wide: the server stores each tag's source and backing table versions, so callers +can read `training$tag_train_v1` without remembering the source branch. Native table tags remain +branch-local; the server maps the database tag to the corresponding table pins. + +### Initial server MVP + +Start with managed native Paimon tables and a fixed set of logical table names. Create and populate +those tables on `main`. Use batch writers and pause writes during branch creation, tag creation and +forward. Invalidate cached tables and load them again after publication. + +Branch-local table creation, deletion and rename require versioned namespace storage and can be +deferred. Format Tables, Object Tables, external tables, views, functions and catalog permissions +are outside this initial versioned-table scope. Unsupported scoped operations return `501`. +These restrictions also apply to operations on main when they would affect retained references: +the absence of a database suffix does not permit deleting storage used by a branch or tag. + +## Branch management + +### Create a branch without data + +```http +POST /v1/catalog/databases/training/branches +Content-Type: application/json + +{"branch":"experiment"} +``` + +Like table `createBranch` without `fromTag`, this copies main's table membership, schemas and +properties, with no table snapshots. It does not copy main's current data. The server must preserve +empty tables explicitly, including their schema-only state. + +### Create a branch with data + +First capture a database tag, then create a branch from it: + +```http +POST /v1/catalog/databases/training/tags +Content-Type: application/json + +{"tagName":"baseline"} +``` + +```http +POST /v1/catalog/databases/training/branches +Content-Type: application/json + +{"branch":"experiment","fromTag":"baseline"} +``` + +`fromTag` names a database tag in the same database. The server restores its table membership, +schemas, properties and snapshots. A missing tag returns `404`; an existing branch returns `409`. +There is no generic `source: {type, name}` object. + +### List and drop branches + +```http +GET /v1/catalog/databases/training/branches +``` + +```json +{"branches":["main","experiment"]} +``` + +```http +DELETE /v1/catalog/databases/training/branches/experiment +``` + +The delete request has no body. The server protects `main` and rejects its deletion with `400`. +Deleting a branch does not authorize removing files or pins still required by a database tag or +another branch. + +### Forward a branch to main + +```http +POST /v1/catalog/databases/training/branches/experiment/forward +Content-Type: application/json + +{} +``` + +The path names the **source branch**, following Table REST. The database operation publishes it to +`main`. A tag is not a forward source. To publish a frozen tag, first create a temporary branch from +that tag, then forward that branch. + +Forward extends table fast-forward to the database's tables. It publishes source versions on main +and can replace target changes; it does not preserve independently changed target tables using +three-way conflict resolution. The first fixed-table server requires matching membership, including +both logical names and table identities, and a snapshot for each source table, as native table +fast-forward requires a populated source. An empty +source table is a `400`; namespace changes the server cannot handle are a `501`. The server validates +all tables before starting publication. Source `main` is invalid. + +Pause both source and main writers while forwarding. Preserve retained tags, keep the two branches +independently writable afterwards, and invalidate/reload main tables before resuming work. No public +multi-table transaction or atomic read view is required for this MVP. A successful response means +all planned table operations finished. Interrupted execution needs recoverable server bookkeeping. + +## Tag management + +### Freeze a branch + +```http +POST /v1/catalog/databases/training/tags +Content-Type: application/json + +{"tagName":"train_v1","fromBranch":"experiment","timeRetained":"7d"} +``` + +Omitting `fromBranch`, or setting it to null, selects `main`. `timeRetained` uses the table tag +retention-duration syntax and is optional. Tags freeze membership, schemas, properties, snapshots, +and the empty state of tables without snapshots. They cannot be moved or updated. Expiring a tag +must respect versions still used by other references; native table pins cannot expire independently +while the database tag is valid. + +### Inspect and list tags + +```http +GET /v1/catalog/databases/training/tags/train_v1 +``` + +```json +{ + "tagName":"train_v1", + "fromBranch":"experiment", + "tagCreateTime":1720000000000, + "tagTimeRetained":"7d" +} +``` + +`tagCreateTime` is milliseconds since the Unix epoch. `tagCreateTime` and `tagTimeRetained` are +optional. `fromBranch` records creation provenance; the source can later be deleted without making +the tag unreadable. Table snapshots are resolved using the tag-suffixed database name. + +```http +GET /v1/catalog/databases/training/tags?maxResults=100&tagNamePrefix=train_ +``` + +```json +{"tags":["train_v1"],"nextPageToken":"next-page"} +``` + +Pass the returned token unchanged to get the next page. A missing token ends iteration. An absent +or zero `maxResults` uses the server default. Pagination does not create a frozen cross-page view. + +### Delete a tag + +```http +DELETE /v1/catalog/databases/training/tags/train_v1 +``` + +The request has no body and does not need an expected reference type: the resource path identifies +a tag. A missing tag returns `404`. Logical deletion and physical cleanup can be separate operations. + +## Errors + +Use the existing `ErrorResponse` and table branch/tag resource types: + +| Situation | HTTP behavior | +| --- | --- | +| Missing database, branch or tag | `404`, with `DATABASE`, `BRANCH` or `TAG` resource details. | +| Creating an existing branch or tag | `409`, with the corresponding resource type and name. | +| Invalid name, protected main mutation, or invalid forward source | `400`. | +| Missing table or snapshot during table orchestration | `404`, identifying the affected resource. | +| Authorization failure | `403`. | +| Unsupported operation or scoped DDL | `501`. | + +There is no merge-specific exception translation. Creation conflicts use the same +`AlreadyExistsException` as table branch/tag creation. Errors never cause fallback to a different +branch or to a physical database without its selector. + +## Reference-scoped table API + +A database name can include exactly one reference selector: + +| Database name | Meaning | +| --- | --- | +| `training` | The main database branch, also addressed as `training$branch_main`. | +| `training$branch_experiment` | The writable database branch `experiment`. | +| `training$branch_main` | Explicit selection of the database branch `main`. | +| `training$tag_train_v1` | The immutable database tag `train_v1`. | + +The selector is carried in the existing database field, including inside `Identifier`. Encode the +complete database name once as one REST path segment. JSON names remain decoded. For example: + +```http +GET /v1/catalog/databases/training%24branch_experiment/tables/features +GET /v1/catalog/databases/training%24tag_train_v1/tables/features +POST /v1/catalog/databases/training%24branch_experiment/tables/features/commit +``` + +Branch and tag management use `/branches` and `/tags` on the physical database name. +Table access uses the existing table resource paths with the selected database name. + +Let `D = /v1/{prefix}/databases/{database}` below, where `database` may carry a reference suffix. +These are the existing operations and request/response structures: + +| Method and path | Existing request / response | Scope | +| --- | --- | --- | +| `GET D` | `GetDatabaseResponse`. | Validate the database and selected reference; return virtual database metadata. | +| `GET D/tables` | `ListTablesResponse`; existing paging/filter query parameters. | Table membership of the reference. | +| `GET D/table-details` | `ListTableDetailsResponse`; existing paging/filter query parameters. | Table definitions within the reference. | +| `GET D/tables/{table}` | `GetTableResponse`. | Selected schema, storage options and path. | +| `POST D/tables` | `CreateTableRequest`. | Create a table in a branch. | +| `POST D/tables/{table}` | `AlterTableRequest`. | Alter a table in a branch. | +| `DELETE D/tables/{table}` | Existing drop-table response. | Remove a table from a branch. | +| `GET D/tables/{table}/snapshot` | `GetTableSnapshotResponse`. | Current branch snapshot or pinned tag snapshot. | +| `GET D/tables/{table}/snapshots/{version}` | `GetVersionSnapshotResponse`. | Resolve a version within this reference. | +| `GET D/tables/{table}/snapshots` | `ListSnapshotsResponse`; existing pagination. | Snapshot history visible through this reference. | +| `GET D/tables/{table}/schemas/{version}` | `GetSchemaResponse`. | Resolve a schema ID or `LATEST` within this reference. | +| `GET D/tables/{table}/schemas` | `ListSchemasResponse`; existing pagination. | Schema history retained for this reference. | +| `POST D/tables/{table}/commit` | `CommitTableRequest` / `CommitTableResponse`. | Commit a snapshot to the selected branch. | +| `GET D/tables/{table}/token` | `GetTableTokenResponse`. | Credentials for the resolved table version. | +| `POST D/tables/{table}/auth` | `AuthTableQueryRequest` / `AuthTableQueryResponse`. | Authorize a read of the resolved table. | + +`GetTableResponse` retains the requested database name including its suffix and the logical table +name, such as `features`. It carries the resolved schema, path and storage options; the server may +supply a physical branch alias through existing schema options. Request identifiers retain the +same full database name. A commit keeps the existing `tableId`, `baseSnapshotUuid`, `snapshot`, and +`statistics` fields. The path selects the reference; request identifiers and table IDs must agree +with the resolved table. + +### Database lookup and naming rules + +`GET database` must resolve a suffixed name, because SQL engines can check namespace existence +before accessing a table. The response represents the virtual database and retains its full name. +Database listing returns physical database names only; use `/branches` and `/tags` to discover their names. + +CREATE, DROP and ALTER DATABASE do not accept reference suffixes. In particular, dropping a +virtual database must never drop its physical database. Create, delete and forward branches or manage tags through +`/databases/training/branches` and `/databases/training/tags` instead. This does not prevent ordinary create/alter/drop **table** +operations from modifying membership or metadata in a writable branch. + +The initial server rejects physical `DROP DATABASE` with `400` while any non-main database branch +or database tag exists, even if main has no tables. Remove those references before dropping the +database. The existing database deletion endpoint does not implicitly cascade through references. + +### Table creation, alteration and deletion + +Table operations keep their existing request and response structures. Namespace changes require +server-side versioned membership; accepting a suffix in the client does not imply that the server +implements them. A fixed-table server returns `501` for unsupported namespace changes. + +| Operation | Required server behavior | +| --- | --- | +| Create a table on a branch | Allocate a new table identity and storage, then add its logical name only to that branch after metadata is ready. Do not expose it on main as a side effect of physical creation. | +| Create a table on main after branching | Add it only to main. Existing branches and tags retain their own membership; a later forward can fail with `501` because the table sets differ. | +| Alter a table's schema or properties | Update the selected branch's backing table and recorded state. Other branches and existing tags retain their own definitions. | +| Drop a table on any branch, including main | Remove only that branch's membership entry. Keep metadata and data required by other branches or tags; do not recursively delete the shared table path. Return `501` if the server cannot preserve those references. | +| Recreate a dropped table with the same name | Allocate a new table identity. A same-name table retained on another branch is a different table and does not satisfy fixed-table forward validation. | +| Create, alter or drop through a tag | Return `403`; tag membership and table definitions are immutable. | + +For example, if main and experiment initially contain `features` and `labels`, creating +`training$branch_experiment.samples` adds `samples` only to experiment. Creating `training.metrics` +later adds `metrics` only to main. The fixed-table forward operation cannot publish these different +table sets; it rejects the operation before changing any target table. + +### Selector validation + +The markers `$branch_` and `$tag_` are case-sensitive reserved syntax. The base database must be +nonblank, and the reference follows the name rules above. Missing names, multiple selectors, or +invalid reference names are rejected rather than interpreted as literal database names. Other +uses of `$`, such as `training$archive`, remain literal. Catalogs adopting this contract must resolve +any pre-existing physical database names containing the reserved markers before enabling it; +lookup must not switch between literal and reference meanings based on which object exists. + +Caller-supplied table branch suffixes cannot be combined with a database selector. For example, +`training$branch_a.features$branch_b` is rejected. REST storage commits preserve the original logical +table identifier, including when bare main is mapped to a different physical backing branch after +forward. Explicit Table branch identifiers on an unsuffixed database retain their table selector. + +### Branch and tag behavior + +For a version-enabled database, both main aliases must resolve through the same mapping, including +after forward replaces its backing table branches. Writes through either alias update that mapping. + +A branch resolves to its current membership and table versions. A tag resolves to the membership, +schemas, options and snapshots captured when it was created, even after its source branch advances. +Tag snapshot listing exposes only the pinned snapshot. `LATEST` and `EARLIEST` select that snapshot; +other version selectors must resolve to it or return `404`. Schema reads may access the captured +schema and older schemas retained for reading the captured data, but never later source schemas. +Freezing REST responses alone is insufficient: native readers and system tables can read metadata +directly from storage. A server can return a dedicated frozen metadata branch, with read-only +credentials and no later source snapshots/schemas, through the existing path and branch options. +Another implementation must enforce the same boundary in native reads, including time travel and +schema/system-table access. A default scan option that callers can override does not enforce it. + +An empty captured table is still returned by `GET table`; snapshot lookup returns `404` with +`resourceType: SNAPSHOT`. + +The server rejects content changes through a tag with `403`. Read authorization remains allowed +through `POST .../auth`; HTTP method alone does not determine whether an operation is a write. +Tag credentials must permit reading without allowing mutation of retained metadata or data. + +Missing databases, references and tables return `404`. The selector chooses the branch or tag +namespace: `$branch_train_v1` returns `404` if only a tag with that name exists. Malformed selectors +return `400`. Reserve `409` for already-existing resources, following the table APIs. +Unsupported operations on references return `501`. None of these errors permits retrying the +request against the physical database without its suffix. + +### Java table usage + +Use the same catalog for ordinary databases and any number of database references: + +```java +import org.apache.paimon.catalog.Identifier; +import org.apache.paimon.table.Table; + +Identifier experiment = Identifier.create("training$branch_experiment", "features"); +Identifier trainingTag = Identifier.create("training$tag_train_v1", "features"); + +Table experimentFeatures = restCatalog.getTable(experiment); +Table trainingFeatures = restCatalog.getTable(trainingTag); +restCatalog.listTables("training$branch_experiment"); +restCatalog.getDatabase("training$tag_train_v1"); + +// Use experimentFeatures with the ordinary Paimon batch write API. +// Use trainingFeatures with the ordinary Paimon read API. +``` + +`RESTApi` uses these same identifiers with its existing table methods. `Identifier` already retains +the full database name through serialization and in table loaders; no extra reference fields are +stored in RESTCatalog or RESTCatalogLoader. Subsequent snapshot reads, schema changes, commits, +auth and token requests carry the same database name. Caches keyed by full table identifiers distinguish branches and tags. The two main aliases +(`training` and `training$branch_main`) refer to the same state. The REST catalog cache invalidates +both aliases when a table is altered, dropped or explicitly invalidated through either name. +After forward, invalidate each affected main table in every client cache before loading it again; +invalidating either main alias clears both. A repeated cached getTable call is not a reload. + +SQL clients can pass the selector as a quoted database name, using their ordinary identifier +quoting rules. For example: + +```sql +SELECT * FROM `training$branch_experiment`.features; +SELECT * FROM `training$tag_train_v1`.features; +``` + +A REST server implementing virtual database lookup and table resolution is required. There is no +new engine catalog option or reference-switch operation. + +Rename, register, replace, rollback, partition/consumer endpoints, nested table branch/tag +management, view writes, functions and table policies do not yet accept database reference suffixes +in the Java client. RESTCatalog validates the virtual database for read-only view probes, then +returns empty lists or a missing view so engine table discovery and DROP TABLE can proceed. Global table listing and lookup by table ID retain their physical-catalog meaning; +they have no database selector. Extending those operations to discover or address references is +additional work. Catalog-level permissions and reference management continue to use physical names. + +### Server routing and reuse + +Decode the database path segment and parse it with `DatabaseIdentifier.parse(name)`. The result +contains the physical database name and an optional typed `DatabaseReference`. Resolve that +reference and the logical table once into a request context with table identity and backing version, +then reuse the existing table handlers. Validate authentication against the actual request path +and authorize access to the resolved table. Preserve the full requested database name in returned +identifiers so follow-up calls stay on the same reference. + +Parsing the suffix does not replace reference management: listing still needs the selected +membership, tags need frozen metadata, and commits must update the selected branch's recorded +table state. The additional request cost is a reference/table mapping lookup, which can be cached; +this addressing scheme does not require proxying or copying table data. The storage and forward work +remains the server orchestration described below. + +## Java management usage + +`RESTCatalog.treeManagement()` shares the catalog's prefix, authentication and HTTP configuration. +Its operations follow the existing table branch/tag method names: + +```java +import org.apache.paimon.PagedList; +import org.apache.paimon.catalog.Identifier; +import org.apache.paimon.management.TreeManagement; +import org.apache.paimon.rest.responses.GetDatabaseTagResponse; + +TreeManagement trees = restCatalog.treeManagement(); +trees.createTag("training", "baseline", null, null); +trees.createBranch("training", "experiment", "baseline"); + +// Use ordinary batch writers on these tables. +restCatalog.getTable(Identifier.create("training$branch_experiment", "features")); + +// Stop experiment writes while capturing its training inputs. +trees.createTag("training", "train_v1", "experiment", "7d"); +GetDatabaseTagResponse tag = trees.getTag("training", "train_v1"); +PagedList page = trees.listTagsPaged("training", 100, null, "train_"); +restCatalog.getTable(Identifier.create("training$tag_train_v1", "features")); + +// Stop main and experiment writers before publishing the experiment's current state. +trees.fastForward("training", "experiment"); +// Invalidate cached main tables and reload them before resuming writers. + +trees.dropBranch("training", "experiment"); +trees.deleteTag("training", "train_v1"); +``` + +`RESTApi` exposes `listDatabaseBranches`, `createDatabaseBranch`, `dropDatabaseBranch`, +`fastForwardDatabase`, `createDatabaseTag`, `getDatabaseTag`, `listDatabaseTagsPaged`, and +`deleteDatabaseTag`. All management methods take the physical database name. Table access keeps +using the suffix in the ordinary `Identifier`. + +## Server implementation using table capabilities + +### Metadata and bootstrap + +Keep a database branch record with membership and logical-table-to-backing-branch mappings. Keep a +separate database tag record with its source branch, retention metadata, frozen membership and each +table's identity, schema, properties and snapshot (or explicit no-snapshot state). + +A version-enabled database starts with `main`. An existing database can be initialized while writers +are stopped. Normal database access and `$branch_main` must use the same record. Accepted table +commits and schema changes update its table state. Direct filesystem writes and unmanaged edits of +service-owned table references bypass this bookkeeping and are outside the MVP. + +Backing reference names are owned by the service. Reject collisions with unrelated table branches +or tags. Database and native table naming rules differ: a purely numeric database branch name needs +a valid physical alias because native table branches reject it. Name matching alone is not a safe +way to locate a backing version. + +### Create branches + +Without `fromTag`, call the schema-only table branch operation for each table on main. With +`fromTag`, resolve each frozen table entry and create its backing branch from the corresponding +native table tag. Paimon's `createBranch(name, tagName)` copies the selected snapshot and its schemas; +`createBranch(name)` copies schemas without data. + +Preserve captured schema-only changes newer than the snapshot schema. Empty captured tables need +schema-only branches. Publish the database branch only after all table entries are ready. No data +files need to be copied just to create a branch. Failed setup can leave private work to resume or +clean up. + +### Capture tags and retain data + +Pause source writers, capture table membership and versions, and pin each populated table snapshot. +Native table tags belong to physical table branches, so store that backing branch with each pin. +Empty tables need frozen schema and no-snapshot metadata because native createTag needs a snapshot. +A minimal server can explicitly reject empty-table tagging until that behavior is implemented. + +A database tag must remain readable after source writes, schema evolution, source deletion, and +forward. Provide frozen metadata to native table readers, including system tables and explicit +snapshot reads, as described above. Native automatic retention must not delete service-owned pins +before the database tag expires or is deleted. + +Physical cleanup must account for all database references. Native dropBranch deletes its metadata +directory, including its tags. Defer physical cleanup in the first MVP, or relocate retained metadata +before deleting a backing branch. Keep the data files referenced by every retained snapshot. + +### Execute forward + +1. Resolve the source branch and main. Validate the whole fixed-table membership, including logical + names and table identities, and all source snapshots before changing target state. +2. Resolve each source table version and prepare the corresponding main table state using native + table snapshot/schema mechanisms. +3. Preserve database tags before applying native fastForward: that operation can remove target + metadata and tags. An adapter can instead prepare fresh backing branches and publish their mapping. +4. Publish all target entries, keeping source and main independently writable. Do not point both at + one mutable table branch. Both main aliases resolve the published mapping. +5. Return success after the planned work completes. Keep an operation record for recovery from a + partial backend failure, and invalidate/reload client caches before writers resume. + +This uses table forward semantics, including replacement of target state. It needs no public hash, +reference ID, multi-table transaction endpoint, merge base or merge-mode API. More advanced merge +semantics will be designed separately. + +## Validate the fixed-table MVP + +Use two populated tables, `features` and `labels`: + +1. Capture main as `baseline`, then create `experiment` from that tag. Verify both tables contain + baseline data. Separately create a branch without `fromTag` and verify the schemas exist with no data. +2. Write both experiment tables through `$branch_experiment`; main must remain unchanged. +3. Capture `train_v1`, advance the source data and schema, and verify `$tag_train_v1` still reads the + captured versions, including native time travel and system-table boundaries. Tag writes must fail. +4. Stop writers and forward experiment to main. Invalidate/reload both main aliases and verify the + published tables. Resume independent writes on main and experiment and verify isolation. +5. Delete the experiment branch and verify the retained tag still reads correctly. Delete unused tags + and verify that cleanup preserves any versions retained by other branches or tags. + +The client tests cover shared Table REST payloads, paths, empty mutation responses, pagination, +authentication, error propagation, serialization and ordinary table reads/writes through suffixes. +Production branch/tag lifecycle, retention and forward still require integration tests against an +implementing catalog server. Branch-local namespace changes and additional table kinds are later work. diff --git a/docs/docs/concepts/rest/index.md b/docs/docs/concepts/rest/index.md index f2974ef44190..8e372fcb6cff 100644 --- a/docs/docs/concepts/rest/index.md +++ b/docs/docs/concepts/rest/index.md @@ -74,6 +74,8 @@ Choose the authentication guide for your service: ## API References - [REST Catalog API](./rest-api): the OpenAPI contract for catalog operations. +- [Database Branches and Tags](./database-versioning): experimental reference management, database-name selectors, and + the server MVP design using existing table branches and tags. - [REST Management API](./management-api): permissions, row filters, column masking, and the corresponding Spark SQL procedures. diff --git a/docs/docs/concepts/rest/rest-api.md b/docs/docs/concepts/rest/rest-api.md index 18e54faea4db..30a3e9afa1ee 100644 --- a/docs/docs/concepts/rest/rest-api.md +++ b/docs/docs/concepts/rest/rest-api.md @@ -52,7 +52,8 @@ payloads, and error responses are defined in the OpenAPI specification. | Commits and snapshots | Commit, roll back, and inspect table versions. | Table-scoped `commit`, `rollback`, `rollback-schema`, `snapshot`, and `snapshots`. | | Data access | Request storage credentials and authorize a query. | Table-scoped `token` and `auth`. | | Partitions | List, create, drop, and mark partitions done. | Table-scoped `partitions`. | -| Branches and tags | Manage named histories and retained snapshots. | Table-scoped `branches` and `tags`. | +| Table branches and tags | Manage named histories and retained snapshots. | Table-scoped `branches` and `tags`. | +| Database branches and tags | Create, list, inspect, delete, and forward. | Database-scoped `branches`, `tags`, and `branches/{branch}/forward`. | | Consumers | List and reset streaming consumer progress. | Table-scoped `consumers`. | | Views and functions | Manage reusable SQL and function definitions. | Database- and catalog-scoped `views` and `functions`. | @@ -60,6 +61,11 @@ In this table, **table-scoped** means `/v1/{prefix}/databases/{database}/tables/{table}`. Catalog-wide listing and detail-listing endpoints are described in the specification alongside their database-scoped counterparts. +See [Database Branches and Tags](./database-versioning) for table-aligned branch/tag examples, forward +semantics, and the server MVP design. Supported table operations select a reference with a database +name such as `training$branch_experiment` or `training$tag_train_v1`. The existing table paths, +request/response structures and Java methods carry the full database name. + ## Partition Compatibility Partition options use the existing `POST .../partitions` request. `partitionOptions` follows the diff --git a/docs/docs/program-api/rest-api.mdx b/docs/docs/program-api/rest-api.mdx index ed19d4e380c9..ec89c6dcb249 100644 --- a/docs/docs/program-api/rest-api.mdx +++ b/docs/docs/program-api/rest-api.mdx @@ -36,6 +36,8 @@ metadata requests without bringing in the full table read/write bundle. | Load a `Table` and read or write rows | [Java API](java-api) with a REST catalog | | Implement an HTTP client or catalog server | [REST API specification](../concepts/rest/rest-api) | | Administrative endpoints | [Management API](../concepts/rest/management-api) | +| Database branch/tag management | [Database Branches and Tags](../concepts/rest/database-versioning#java-management-usage) | +| Tables within a database branch/tag | [Database-name reference selectors](../concepts/rest/database-versioning#java-table-usage) | ## Dependency diff --git a/docs/scripts/validate-rest-openapi.js b/docs/scripts/validate-rest-openapi.js index e51eb2295da0..bf3d7d22b5b8 100644 --- a/docs/scripts/validate-rest-openapi.js +++ b/docs/scripts/validate-rest-openapi.js @@ -225,6 +225,46 @@ function requireExactEnum(contract, schemaName, expectedValues) { function validateCatalogOpenApi() { const contract = validateCommon('rest-catalog-open-api.yaml'); + contract.checkSpec( + !Object.keys(contract.spec.paths).some((path) => /\/trees(?:\/|$)/.test(path)), + 'Database branches and tags must use table-aligned paths; trees is not supported', + ); + [ + ['createDatabaseBranch', 'CreateBranchRequest'], + ['forwardDatabaseBranch', 'ForwardBranchRequest'], + ['createDatabaseTag', 'CreateDatabaseTagRequest'], + ].forEach(([operationId, schemaName]) => { + const operation = contract.requireOperation(operationId); + contract.checkSpec( + operation.requestBody.content['application/json'].schema.$ref === `#/components/schemas/${schemaName}`, + `${operationId} must use ${schemaName}`, + ); + contract.checkSpec(!operation.responses['200'].content, `${operationId} must return no body`); + }); + [['listDatabaseBranches', 'ListBranchesResponse'], ['listDatabaseTagsPaged', 'ListTagsResponse'], + ['getDatabaseTag', 'GetDatabaseTagResponse']].forEach(([operationId, schemaName]) => { + contract.checkSpec( + contract.requireOperation(operationId).responses['200'].content['application/json'].schema.$ref === `#/components/schemas/${schemaName}`, + `${operationId} must use ${schemaName}`, + ); + }); + ['dropDatabaseBranch', 'deleteDatabaseTag'].forEach((operationId) => { + const operation = contract.requireOperation(operationId); + contract.checkSpec(!operation.requestBody && !operation.responses['200'].content, + `${operationId} must have no request or response body`); + }); + contract.checkSpec(!Object.keys(contract.spec.paths).some((path) => + /\/databases\/\{database\}\/branches\/[^/]+\/merge$/.test(path)), + 'Database merge is deferred'); + const databaseParameter = contract.spec.components.parameters.Database; + contract.checkSpec( + databaseParameter.examples.branch.value === 'training$branch_experiment' && + databaseParameter.examples.tag.value === 'training$tag_train_v1', + 'Database reference examples must use the reserved branch and tag suffixes', + ); + ['getDatabase', 'listTables', 'getTable', 'commitTable', 'getSchema', 'listSchemas'].forEach( + (operationId) => contract.requireResponses(operationId, ['404', '409', '501']), + ); [ 'getConfig', 'createDatabase', diff --git a/docs/sidebars.js b/docs/sidebars.js index 9c9f11a2991c..cdc959aa0bb5 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -44,6 +44,7 @@ const sidebars = { "concepts/rest/tables", "concepts/rest/pvfs", "concepts/rest/rest-api", + "concepts/rest/database-versioning", "concepts/rest/management-api" ] }, diff --git a/docs/static/rest-catalog-open-api.yaml b/docs/static/rest-catalog-open-api.yaml index 02ee6df595be..223a07eb8459 100644 --- a/docs/static/rest-catalog-open-api.yaml +++ b/docs/static/rest-catalog-open-api.yaml @@ -84,11 +84,14 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/ListDatabasesResponse' + $ref: "#/components/schemas/ListDatabasesResponse" "401": - $ref: '#/components/responses/UnauthorizedErrorResponse' + $ref: "#/components/responses/UnauthorizedErrorResponse" "500": - $ref: '#/components/responses/ServerErrorResponse' + $ref: "#/components/responses/ServerErrorResponse" + description: >- + List physical databases. Branch and tag access names are not additional database entries; + discover database branches and tags through /databases/{database}/branches and /databases/{database}/tags. post: tags: - database @@ -104,16 +107,21 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/CreateDatabaseRequest' + $ref: "#/components/schemas/CreateDatabaseRequest" responses: "200": description: Success, no content + "400": + $ref: "#/components/responses/BadRequestErrorResponse" "401": - $ref: '#/components/responses/UnauthorizedErrorResponse' + $ref: "#/components/responses/UnauthorizedErrorResponse" "409": - $ref: '#/components/responses/DatabaseAlreadyExistErrorResponse' + $ref: "#/components/responses/DatabaseAlreadyExistErrorResponse" "500": - $ref: '#/components/responses/ServerErrorResponse' + $ref: "#/components/responses/ServerErrorResponse" + description: >- + Create a physical database. The reserved $branch_ and $tag_ suffix markers are not allowed; + create database branches and tags through /branches and /tags. /v1/{prefix}/databases/{database}: get: tags: @@ -126,24 +134,31 @@ paths: required: true schema: type: string - - name: database - in: path - required: true - schema: - type: string + - $ref: "#/components/parameters/Database" responses: "200": description: Get a database by database name. content: application/json: schema: - $ref: '#/components/schemas/GetDatabaseResponse' + $ref: "#/components/schemas/GetDatabaseResponse" + "400": + $ref: "#/components/responses/BadRequestErrorResponse" "401": - $ref: '#/components/responses/UnauthorizedErrorResponse' + $ref: "#/components/responses/UnauthorizedErrorResponse" "404": - $ref: '#/components/responses/DatabaseNotExistErrorResponse' + $ref: "#/components/responses/DatabaseNotExistErrorResponse" + "409": + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" "500": - $ref: '#/components/responses/ServerErrorResponse' + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + A database name with $branch_ or $tag_ selects an existing reference. Return + metadata for the virtual database, retaining the full requested name. This lookup supports + engine namespace existence checks. Missing databases or references return 404; a missing branch or tag of the selected + type also returns 404. Never resolve a missing reference to the base database. delete: tags: - database @@ -163,12 +178,19 @@ paths: responses: "200": description: Success, no content + "400": + $ref: "#/components/responses/BadRequestErrorResponse" "401": - $ref: '#/components/responses/UnauthorizedErrorResponse' + $ref: "#/components/responses/UnauthorizedErrorResponse" "404": - $ref: '#/components/responses/DatabaseNotExistErrorResponse' + $ref: "#/components/responses/DatabaseNotExistErrorResponse" "500": - $ref: '#/components/responses/ServerErrorResponse' + $ref: "#/components/responses/ServerErrorResponse" + description: >- + Database reference suffixes are not allowed for database mutation. Use the /branches and /tags management + endpoints with the physical database name to manage references. The initial server rejects physical + database deletion with 400 while any non-main database branch or database tag exists, even + when main has no tables. Deletion does not implicitly cascade through references. post: tags: - database @@ -189,26 +211,65 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/AlterDatabaseRequest' + $ref: "#/components/schemas/AlterDatabaseRequest" responses: "200": description: OK content: application/json: schema: - $ref: '#/components/schemas/AlterDatabaseResponse' + $ref: "#/components/schemas/AlterDatabaseResponse" + "400": + $ref: "#/components/responses/BadRequestErrorResponse" + "401": + $ref: "#/components/responses/UnauthorizedErrorResponse" + "404": + $ref: "#/components/responses/DatabaseNotExistErrorResponse" + "500": + $ref: "#/components/responses/ServerErrorResponse" + description: >- + Database reference suffixes are not allowed for database mutation. Use the /branches and /tags management + endpoints with the physical database name to manage references. + /v1/{prefix}/databases/{database}/branches: + get: + tags: + - branch + summary: List database branches + operationId: listDatabaseBranches + parameters: + - name: prefix + in: path + required: true + schema: + type: string + - name: database + in: path + required: true + schema: + type: string + responses: + "200": + description: OK + content: + application/json: + schema: + $ref: '#/components/schemas/ListBranchesResponse' "401": $ref: '#/components/responses/UnauthorizedErrorResponse' "404": $ref: '#/components/responses/DatabaseNotExistErrorResponse' "500": $ref: '#/components/responses/ServerErrorResponse' - /v1/{prefix}/databases/{database}/register: post: tags: - - table - summary: Register table - operationId: RegisterTable + - branch + summary: Create database branch + description: >- + Uses the table CreateBranchRequest. Without fromTag, create a branch with main's table + membership and schemas but no data. With fromTag, restore the membership and versions + captured by that database tag. Database must be a physical name without a selector. + Branch and tag names have separate namespaces. The server protects main. + operationId: createDatabaseBranch parameters: - name: prefix in: path @@ -224,7 +285,7 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/RegisterTableRequest' + $ref: '#/components/schemas/CreateBranchRequest' responses: "200": description: Success, no content @@ -233,17 +294,121 @@ paths: "401": $ref: '#/components/responses/UnauthorizedErrorResponse' "404": - $ref: '#/components/responses/DatabaseNotExistErrorResponse' + description: + Not Found + - DatabaseNotExistException, database does not exist + - TagNotExistException, the requested tag does not exist + content: + application/json: + schema: + $ref: '#/components/schemas/ErrorResponse' + examples: + DatabaseNotExist: + value: + code: 404 + resourceType: DATABASE + resourceName: training + message: Database does not exist + TagNotExist: + $ref: '#/components/examples/TagNotExistError' "409": - $ref: '#/components/responses/TableAlreadyExistErrorResponse' + $ref: '#/components/responses/BranchAlreadyExistErrorResponse' "500": $ref: '#/components/responses/ServerErrorResponse' - /v1/{prefix}/databases/{database}/tables: + /v1/{prefix}/databases/{database}/branches/{branch}: + delete: + tags: + - branch + summary: Drop database branch + description: >- + Delete a database branch by name, with no request or response body. Deleting main is a + bad request. Retain any backing metadata and data still needed by database tags or other + branches. Database must be a physical name without a selector. + operationId: dropDatabaseBranch + parameters: + - name: prefix + in: path + required: true + schema: + type: string + - name: database + in: path + required: true + schema: + type: string + - name: branch + in: path + required: true + schema: + type: string + responses: + "200": + description: Success, no content + "400": + $ref: '#/components/responses/BadRequestErrorResponse' + "401": + $ref: '#/components/responses/UnauthorizedErrorResponse' + "404": + $ref: '#/components/responses/BranchNotExistErrorResponse' + "500": + $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/databases/{database}/branches/{branch}/forward: + post: + tags: + - branch + summary: Forward main to a database branch + description: >- + Extends the table forward operation to the database's tables. The path names the source + branch; the physical database selects main as the target. Send an empty ForwardBranchRequest + and return success without a body. A tag is not a valid source. This publishes source table + versions on main; it does not combine target-only writes or perform three-way conflict + resolution. Source must be a non-main branch. Pause writers and reload tables after success. + For the fixed-table MVP, validate matching logical table names and table identities, and a + snapshot for every source table, before changing target state. Same-name tables recreated + with different identities do not match. Unsupported namespace changes return 501. Preserve retained tags + and keep source and main independently writable. No public multi-table transaction or atomic + read view is promised; a success means all planned table updates completed. + operationId: forwardDatabaseBranch + parameters: + - name: prefix + in: path + required: true + schema: + type: string + - name: database + in: path + required: true + schema: + type: string + - name: branch + in: path + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/ForwardBranchRequest' + responses: + "200": + description: Success, no content + "400": + $ref: '#/components/responses/BadRequestErrorResponse' + "401": + $ref: '#/components/responses/UnauthorizedErrorResponse' + "404": + $ref: '#/components/responses/BranchNotExistErrorResponse' + "500": + $ref: '#/components/responses/ServerErrorResponse' + "501": + $ref: '#/components/responses/ReferenceTableNotImplementedErrorResponse' + /v1/{prefix}/databases/{database}/tags: get: tags: - - table - summary: List tables - operationId: listTables + - tag + summary: List database tags + operationId: listDatabaseTagsPaged parameters: - name: prefix in: path @@ -264,8 +429,8 @@ paths: in: query schema: type: string - - name: tableNamePattern - description: A sql LIKE pattern (%) for table names. Currently, only prefix matching is supported. + - name: tagNamePrefix + description: A prefix for tag names. All tags will be returned if not set or empty. in: query schema: type: string @@ -275,18 +440,148 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/ListTablesResponse' + $ref: '#/components/schemas/ListTagsResponse' "401": $ref: '#/components/responses/UnauthorizedErrorResponse' "404": $ref: '#/components/responses/DatabaseNotExistErrorResponse' "500": $ref: '#/components/responses/ServerErrorResponse' + post: + tags: + - tag + summary: Create database tag + description: >- + Capture each table version from fromBranch (main when absent or null), using the table + tagName and timeRetained conventions. Each table has its own snapshot; there is no single + database snapshotId. Tags have database-wide names and freeze membership, schema, properties, + and data. Database must be a physical name without a selector. Pause writes while capturing. + Expiring a database tag must not remove versions still needed by a branch or another tag. + operationId: createDatabaseTag + parameters: + - name: prefix + in: path + required: true + schema: + type: string + - name: database + in: path + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/CreateDatabaseTagRequest' + responses: + "200": + description: Success, no content + "400": + $ref: '#/components/responses/BadRequestErrorResponse' + "401": + $ref: '#/components/responses/UnauthorizedErrorResponse' + "404": + description: + Not Found + - DatabaseNotExistException, database does not exist + - BranchNotExistException, the requested source branch does not exist + content: + application/json: + schema: + $ref: '#/components/schemas/ErrorResponse' + examples: + DatabaseNotExist: + value: + code: 404 + resourceType: DATABASE + resourceName: training + message: Database does not exist + BranchNotExist: + value: + code: 404 + resourceType: BRANCH + resourceName: experiment + message: Source branch does not exist + "409": + $ref: '#/components/responses/TagAlreadyExistErrorResponse' + "500": + $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/databases/{database}/tags/{tag}: + get: + tags: + - tag + summary: Get database tag + description: >- + Get database tag metadata. fromBranch records its source but is not needed to read it: + load tables through database$tag_tagName using the ordinary table endpoints. Database + must be a physical name without a selector. Tag lookup remains valid after source deletion. + operationId: getDatabaseTag + parameters: + - name: prefix + in: path + required: true + schema: + type: string + - name: database + in: path + required: true + schema: + type: string + - name: tag + in: path + required: true + schema: + type: string + responses: + "200": + description: OK + content: + application/json: + schema: + $ref: '#/components/schemas/GetDatabaseTagResponse' + "401": + $ref: '#/components/responses/UnauthorizedErrorResponse' + "404": + $ref: '#/components/responses/TagNotExistErrorResponse' + "500": + $ref: '#/components/responses/ServerErrorResponse' + delete: + tags: + - tag + summary: Delete database tag + operationId: deleteDatabaseTag + parameters: + - name: prefix + in: path + required: true + schema: + type: string + - name: database + in: path + required: true + schema: + type: string + - name: tag + in: path + required: true + schema: + type: string + responses: + "200": + description: Success, no content + "401": + $ref: '#/components/responses/UnauthorizedErrorResponse' + "404": + $ref: '#/components/responses/TagNotExistErrorResponse' + "500": + $ref: '#/components/responses/ServerErrorResponse' + /v1/{prefix}/databases/{database}/register: post: tags: - table - summary: Create table - operationId: createTable + summary: Register table + operationId: RegisterTable parameters: - name: prefix in: path @@ -302,7 +597,7 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/CreateTableRequest' + $ref: '#/components/schemas/RegisterTableRequest' responses: "200": description: Success, no content @@ -316,23 +611,113 @@ paths: $ref: '#/components/responses/TableAlreadyExistErrorResponse' "500": $ref: '#/components/responses/ServerErrorResponse' - /v1/{prefix}/databases/{database}/table-details: + /v1/{prefix}/databases/{database}/tables: get: tags: - table - summary: List table details - operationId: listTableDetails + summary: List tables + operationId: listTables parameters: - name: prefix in: path required: true schema: type: string - - name: database + - $ref: "#/components/parameters/Database" + - name: maxResults + in: query + schema: + type: integer + format: int32 + - name: pageToken + in: query + schema: + type: string + - name: tableNamePattern + description: A sql LIKE pattern (%) for table names. Currently, only prefix matching is supported. + in: query + schema: + type: string + responses: + "200": + description: OK + content: + application/json: + schema: + $ref: "#/components/schemas/ListTablesResponse" + "400": + $ref: "#/components/responses/BadRequestErrorResponse" + "401": + $ref: "#/components/responses/UnauthorizedErrorResponse" + "403": + $ref: "#/components/responses/ForbiddenErrorResponse" + "404": + $ref: "#/components/responses/ReferenceTableNotExistErrorResponse" + "409": + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" + "500": + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + The database path segment may select a database branch or immutable tag using its reserved + suffix. Resolve the table through that reference. Tags expose captured membership and + metadata, never newer source state. Missing references never fall back to the base database. A missing branch or tag of the selected type returns 404. + post: + tags: + - table + summary: Create table + operationId: createTable + parameters: + - name: prefix + in: path + required: true + schema: + type: string + - $ref: "#/components/parameters/Database" + requestBody: + content: + application/json: + schema: + $ref: "#/components/schemas/CreateTableRequest" + responses: + "200": + description: Success, no content + "400": + $ref: "#/components/responses/BadRequestErrorResponse" + "401": + $ref: "#/components/responses/UnauthorizedErrorResponse" + "403": + $ref: "#/components/responses/ForbiddenErrorResponse" + "404": + $ref: "#/components/responses/ReferenceTableNotExistErrorResponse" + "409": + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" + "500": + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + The database path segment may select a database branch or immutable tag using its reserved + suffix. Apply this operation to the selected branch; a tag returns 403. Identifiers in the + body must retain the full database name including its suffix and agree with the path. Table + IDs must match the resolved table. Missing references never fall back to the base database. A missing branch or tag of the selected type returns 404. + Create a new table identity and add its name only to the selected branch. New main tables + do not appear in existing branches or tags. Recreating a dropped name allocates a new + table identity. A fixed-table server returns 501 for unsupported namespace changes. + /v1/{prefix}/databases/{database}/table-details: + get: + tags: + - table + summary: List table details + operationId: listTableDetails + parameters: + - name: prefix in: path required: true schema: type: string + - $ref: "#/components/parameters/Database" - name: maxResults in: query schema: @@ -358,13 +743,25 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/ListTableDetailsResponse' + $ref: "#/components/schemas/ListTableDetailsResponse" + "400": + $ref: "#/components/responses/BadRequestErrorResponse" "401": - $ref: '#/components/responses/UnauthorizedErrorResponse' + $ref: "#/components/responses/UnauthorizedErrorResponse" + "403": + $ref: "#/components/responses/ForbiddenErrorResponse" "404": - $ref: '#/components/responses/DatabaseNotExistErrorResponse' + $ref: "#/components/responses/ReferenceTableNotExistErrorResponse" + "409": + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" "500": - $ref: '#/components/responses/ServerErrorResponse' + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + The database path segment may select a database branch or immutable tag using its reserved + suffix. Resolve the table through that reference. Tags expose captured membership and + metadata, never newer source state. Missing references never fall back to the base database. A missing branch or tag of the selected type returns 404. /v1/{prefix}/tables: get: tags: @@ -451,11 +848,7 @@ paths: required: true schema: type: string - - name: database - in: path - required: true - schema: - type: string + - $ref: "#/components/parameters/Database" - name: table in: path required: true @@ -467,13 +860,27 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/GetTableResponse' + $ref: "#/components/schemas/GetTableResponse" + "400": + $ref: "#/components/responses/BadRequestErrorResponse" "401": - $ref: '#/components/responses/UnauthorizedErrorResponse' + $ref: "#/components/responses/UnauthorizedErrorResponse" + "403": + $ref: "#/components/responses/ForbiddenErrorResponse" "404": - $ref: '#/components/responses/TableNotExistErrorResponse' + $ref: "#/components/responses/ReferenceTableNotExistErrorResponse" + "409": + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" "500": - $ref: '#/components/responses/ServerErrorResponse' + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + The database path segment may select a database branch or immutable tag using its reserved + suffix. Resolve the table through that reference. Tags expose captured membership and + metadata, never newer source state. Return the requested database name including its suffix, + the logical table name, and resolved schema, path and storage options. Internal branch aliases + may be carried in schema options. Missing references never fall back to the base database. A missing branch or tag of the selected type returns 404. post: tags: - table @@ -485,11 +892,7 @@ paths: required: true schema: type: string - - name: database - in: path - required: true - schema: - type: string + - $ref: "#/components/parameters/Database" - name: table in: path required: true @@ -499,20 +902,29 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/AlterTableRequest' + $ref: "#/components/schemas/AlterTableRequest" responses: "200": description: Success, no content "400": - $ref: '#/components/responses/BadRequestErrorResponse' + $ref: "#/components/responses/BadRequestErrorResponse" "401": - $ref: '#/components/responses/UnauthorizedErrorResponse' + $ref: "#/components/responses/UnauthorizedErrorResponse" + "403": + $ref: "#/components/responses/ForbiddenErrorResponse" "404": - $ref: '#/components/responses/TableNotExistErrorResponse' + $ref: "#/components/responses/ReferenceTableNotExistErrorResponse" "409": - $ref: '#/components/responses/TableAlreadyExistErrorResponse' + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" "500": - $ref: '#/components/responses/ServerErrorResponse' + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + The database path segment may select a database branch or immutable tag using its reserved + suffix. Apply this operation to the selected branch; a tag returns 403. Identifiers in the + body must retain the full database name including its suffix and agree with the path. Table + IDs must match the resolved table. Missing references never fall back to the base database. A missing branch or tag of the selected type returns 404. delete: tags: - table @@ -524,11 +936,7 @@ paths: required: true schema: type: string - - name: database - in: path - required: true - schema: - type: string + - $ref: "#/components/parameters/Database" - name: table in: path required: true @@ -537,12 +945,28 @@ paths: responses: "200": description: Success, no content + "400": + $ref: "#/components/responses/BadRequestErrorResponse" "401": - $ref: '#/components/responses/UnauthorizedErrorResponse' + $ref: "#/components/responses/UnauthorizedErrorResponse" + "403": + $ref: "#/components/responses/ForbiddenErrorResponse" "404": - $ref: '#/components/responses/TableNotExistErrorResponse' + $ref: "#/components/responses/ReferenceTableNotExistErrorResponse" + "409": + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" "500": - $ref: '#/components/responses/ServerErrorResponse' + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + The database path segment may select a database branch or immutable tag using its reserved + suffix. Apply this operation to the selected branch; a tag returns 403. Identifiers in the + body must retain the full database name including its suffix and agree with the path. Table + IDs must match the resolved table. Missing references never fall back to the base database. A missing branch or tag of the selected type returns 404. + Remove the table only from the selected branch membership, including when the database + has no suffix and selects main. Preserve storage and metadata referenced by any other + branch or tag. Return 501 if the server cannot preserve those references. /v1/{prefix}/tables/rename: post: tags: @@ -585,11 +1009,7 @@ paths: required: true schema: type: string - - name: database - in: path - required: true - schema: - type: string + - $ref: "#/components/parameters/Database" - name: table in: path required: true @@ -599,22 +1019,33 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/CommitTableRequest' + $ref: "#/components/schemas/CommitTableRequest" responses: "200": description: OK content: application/json: schema: - $ref: '#/components/schemas/CommitTableResponse' + $ref: "#/components/schemas/CommitTableResponse" "400": - $ref: '#/components/responses/BadRequestErrorResponse' + $ref: "#/components/responses/BadRequestErrorResponse" "401": - $ref: '#/components/responses/UnauthorizedErrorResponse' + $ref: "#/components/responses/UnauthorizedErrorResponse" + "403": + $ref: "#/components/responses/ForbiddenErrorResponse" "404": - $ref: '#/components/responses/TableNotExistErrorResponse' + $ref: "#/components/responses/ReferenceTableNotExistErrorResponse" + "409": + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" "500": - $ref: '#/components/responses/ServerErrorResponse' + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + The database path segment may select a database branch or immutable tag using its reserved + suffix. Apply this operation to the selected branch; a tag returns 403. Identifiers in the + body must retain the full database name including its suffix and agree with the path. Table + IDs must match the resolved table. Missing references never fall back to the base database. A missing branch or tag of the selected type returns 404. /v1/{prefix}/databases/{database}/tables/{table}/rollback: post: tags: @@ -714,11 +1145,7 @@ paths: required: true schema: type: string - - name: database - in: path - required: true - schema: - type: string + - $ref: "#/components/parameters/Database" - name: table in: path required: true @@ -730,13 +1157,27 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/GetTableDataTokenResponse' + $ref: "#/components/schemas/GetTableDataTokenResponse" + "400": + $ref: "#/components/responses/BadRequestErrorResponse" "401": - $ref: '#/components/responses/UnauthorizedErrorResponse' + $ref: "#/components/responses/UnauthorizedErrorResponse" + "403": + $ref: "#/components/responses/ForbiddenErrorResponse" "404": - $ref: '#/components/responses/TableNotExistErrorResponse' + $ref: "#/components/responses/ReferenceTableNotExistErrorResponse" + "409": + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" "500": - $ref: '#/components/responses/ServerErrorResponse' + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + The database path segment may select a database branch or immutable tag using its reserved + suffix. Resolve the table through that reference. Tags expose captured membership and + metadata, never newer source state. Tag credentials must allow reading without permitting + mutation of retained metadata or data. Missing references never fall back to the base + database. A missing branch or tag of the selected type returns 404. /v1/{prefix}/databases/{database}/tables/{table}/auth: post: tags: @@ -749,11 +1190,7 @@ paths: required: true schema: type: string - - name: database - in: path - required: true - schema: - type: string + - $ref: "#/components/parameters/Database" - name: table in: path required: true @@ -763,31 +1200,32 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/AuthTableQueryRequest' + $ref: "#/components/schemas/AuthTableQueryRequest" responses: "200": description: OK content: application/json: schema: - $ref: '#/components/schemas/AuthTableQueryResponse' + $ref: "#/components/schemas/AuthTableQueryResponse" + "400": + $ref: "#/components/responses/BadRequestErrorResponse" "401": - $ref: '#/components/responses/UnauthorizedErrorResponse' + $ref: "#/components/responses/UnauthorizedErrorResponse" "403": - $ref: '#/components/responses/ForbiddenErrorResponse' - 404: - description: - Not Found - - TableNotExistException, table does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorResponse' - examples: - TableNotExist: - $ref: '#/components/examples/TableNotExistError' + $ref: "#/components/responses/ForbiddenErrorResponse" + "404": + $ref: "#/components/responses/ReferenceTableNotExistErrorResponse" + "409": + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" "500": - $ref: '#/components/responses/ServerErrorResponse' + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + The database path segment may select a database branch or immutable tag using its reserved + suffix. Resolve the table through that reference. Tags expose captured membership and + metadata, never newer source state. Missing references never fall back to the base database. A missing branch or tag of the selected type returns 404. /v1/{prefix}/databases/{database}/tables/{table}/snapshot: get: tags: @@ -800,11 +1238,7 @@ paths: required: true schema: type: string - - name: database - in: path - required: true - schema: - type: string + - $ref: "#/components/parameters/Database" - name: table in: path required: true @@ -816,25 +1250,26 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/GetTableSnapshotResponse' + $ref: "#/components/schemas/GetTableSnapshotResponse" + "400": + $ref: "#/components/responses/BadRequestErrorResponse" "401": - $ref: '#/components/responses/UnauthorizedErrorResponse' - 404: - description: - Not Found - - TableNotExistException, table does not exist - - SnapshotNotExistException, the requested snapshot does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorResponse' - examples: - TableNotExist: - $ref: '#/components/examples/TableNotExistError' - SnapshotNotExist: - $ref: '#/components/examples/SnapshotNotExistError' + $ref: "#/components/responses/UnauthorizedErrorResponse" + "403": + $ref: "#/components/responses/ForbiddenErrorResponse" + "404": + $ref: "#/components/responses/ReferenceTableNotExistErrorResponse" + "409": + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" "500": - $ref: '#/components/responses/ServerErrorResponse' + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + The database path segment may select a database branch or immutable tag using its reserved + suffix. Resolve the table through that reference. Tags expose captured membership and + metadata, never newer source state. An existing empty table returns 404 with resourceType + SNAPSHOT. Missing references never fall back to the base database. A missing branch or tag of the selected type returns 404. /v1/{prefix}/databases/{database}/tables/{table}/snapshots/{version}: get: tags: @@ -847,11 +1282,7 @@ paths: required: true schema: type: string - - name: database - in: path - required: true - schema: - type: string + - $ref: "#/components/parameters/Database" - name: table in: path required: true @@ -868,25 +1299,27 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/GetVersionSnapshotResponse' + $ref: "#/components/schemas/GetVersionSnapshotResponse" + "400": + $ref: "#/components/responses/BadRequestErrorResponse" "401": - $ref: '#/components/responses/UnauthorizedErrorResponse' - 404: - description: - Not Found - - TableNotExistException, table does not exist - - SnapshotNotExistException, the requested snapshot does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorResponse' - examples: - TableNotExist: - $ref: '#/components/examples/TableNotExistError' - SnapshotNotExist: - $ref: '#/components/examples/SnapshotNotExistError' + $ref: "#/components/responses/UnauthorizedErrorResponse" + "403": + $ref: "#/components/responses/ForbiddenErrorResponse" + "404": + $ref: "#/components/responses/ReferenceTableNotExistErrorResponse" + "409": + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" "500": - $ref: '#/components/responses/ServerErrorResponse' + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + The database path segment may select a database branch or immutable tag using its reserved + suffix. Resolve the table through that reference. Tags expose captured membership and + metadata, never newer source state. A database tag exposes only its pinned snapshot: LATEST + and EARLIEST select it; other versions must resolve to it or return 404. Missing references + never fall back to the base database. A missing branch or tag of the selected type returns 404. /v1/{prefix}/databases/{database}/tables/{table}/snapshots: get: tags: @@ -899,11 +1332,7 @@ paths: required: true schema: type: string - - name: database - in: path - required: true - schema: - type: string + - $ref: "#/components/parameters/Database" - name: table in: path required: true @@ -924,13 +1353,112 @@ paths: content: application/json: schema: - $ref: '#/components/schemas/ListSnapshotsResponse' + $ref: "#/components/schemas/ListSnapshotsResponse" + "400": + $ref: "#/components/responses/BadRequestErrorResponse" "401": - $ref: '#/components/responses/UnauthorizedErrorResponse' + $ref: "#/components/responses/UnauthorizedErrorResponse" + "403": + $ref: "#/components/responses/ForbiddenErrorResponse" "404": - $ref: '#/components/responses/TableNotExistErrorResponse' + $ref: "#/components/responses/ReferenceTableNotExistErrorResponse" + "409": + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" "500": - $ref: '#/components/responses/ServerErrorResponse' + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + The database path segment may select a database branch or immutable tag using its reserved + suffix. Resolve the table through that reference. Tags expose captured membership and + metadata, never newer source state. A database tag exposes only its pinned snapshot: LATEST + and EARLIEST select it; other versions must resolve to it or return 404. Missing references + never fall back to the base database. A missing branch or tag of the selected type returns 404. + /v1/{prefix}/databases/{database}/tables/{table}/schemas: + parameters: + - $ref: "#/components/parameters/Prefix" + - $ref: "#/components/parameters/Database" + - $ref: "#/components/parameters/Table" + get: + tags: + - table + summary: List table schemas + operationId: listSchemas + parameters: + - name: maxResults + in: query + schema: + type: integer + minimum: 0 + - name: pageToken + in: query + schema: + type: string + responses: + "200": + description: OK + content: + application/json: + schema: + $ref: "#/components/schemas/ListSchemasResponse" + "400": + $ref: "#/components/responses/BadRequestErrorResponse" + "401": + $ref: "#/components/responses/UnauthorizedErrorResponse" + "403": + $ref: "#/components/responses/ForbiddenErrorResponse" + "404": + $ref: "#/components/responses/ReferenceTableNotExistErrorResponse" + "409": + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" + "500": + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + The database path segment may select a database branch or immutable tag using its reserved + suffix. Resolve the table through that reference. Tags expose captured membership and + metadata, never newer source state. For a database tag, LATEST selects the captured schema. + History is limited to schemas retained for its captured data, excluding newer source schemas. + Missing references never fall back to the base database. A missing branch or tag of the selected type returns 404. + /v1/{prefix}/databases/{database}/tables/{table}/schemas/{version}: + parameters: + - $ref: "#/components/parameters/Prefix" + - $ref: "#/components/parameters/Database" + - $ref: "#/components/parameters/Table" + - $ref: "#/components/parameters/Version" + get: + tags: + - table + summary: Get table schema + operationId: getSchema + responses: + "200": + description: OK + content: + application/json: + schema: + $ref: "#/components/schemas/GetSchemaResponse" + "400": + $ref: "#/components/responses/BadRequestErrorResponse" + "401": + $ref: "#/components/responses/UnauthorizedErrorResponse" + "403": + $ref: "#/components/responses/ForbiddenErrorResponse" + "404": + $ref: "#/components/responses/ReferenceTableNotExistErrorResponse" + "409": + $ref: "#/components/responses/ReferenceTableConflictErrorResponse" + "500": + $ref: "#/components/responses/ServerErrorResponse" + "501": + $ref: "#/components/responses/ReferenceTableNotImplementedErrorResponse" + description: >- + The database path segment may select a database branch or immutable tag using its reserved + suffix. Resolve the table through that reference. Tags expose captured membership and + metadata, never newer source state. For a database tag, LATEST selects the captured schema. + History is limited to schemas retained for its captured data, excluding newer source schemas. + Missing references never fall back to the base database. A missing branch or tag of the selected type returns 404. /v1/{prefix}/databases/{database}/tables/{table}/partitions: get: tags: @@ -2406,10 +2934,73 @@ paths: $ref: '#/components/responses/SemanticViewNotImplementedErrorResponse' components: + parameters: + Prefix: + name: prefix + in: path + required: true + schema: + type: string + Database: + name: database + in: path + required: true + schema: + type: string + description: >- + Decoded database name. On supported table operations and GET database, + $branch_ selects a writable branch and $tag_ selects + an immutable tag. The markers are case-sensitive and reserved; exactly one valid reference + suffix is allowed. Encode the complete name as one path segment. A suffix is not a physical + database name. + examples: + branch: + value: training$branch_experiment + tag: + value: training$tag_train_v1 + Table: + name: table + in: path + required: true + schema: + type: string + description: Logical table name. A database reference does not require a table branch suffix. + Version: + name: version + in: path + required: true + schema: + type: string + description: A schema ID or LATEST; resolved within the selected table version. + ############################# # Reusable Response Objects # ############################# responses: + ReferenceTableNotExistErrorResponse: + description: >- + Database, reference, table, snapshot or schema does not exist within the selected reference. + Return resourceType SNAPSHOT for an existing table with no snapshot. Never fall back to main. + content: + application/json: + schema: + $ref: "#/components/schemas/ErrorResponse" + ReferenceTableConflictErrorResponse: + description: >- + The table or column already exists within the selected branch. Tag writes return 403; + a missing branch or tag of the selected type returns 404. + content: + application/json: + schema: + $ref: "#/components/schemas/ErrorResponse" + ReferenceTableNotImplementedErrorResponse: + description: >- + The server does not implement this operation on a database reference. No fallback to the + physical database is allowed. + content: + application/json: + schema: + $ref: "#/components/schemas/ErrorResponse" SemanticViewNotExistErrorResponse: description: Database or semantic view does not exist. content: @@ -2665,6 +3256,37 @@ components: message: Internal Server Error code: 500 schemas: + TableSchema: + allOf: + - $ref: "#/components/schemas/Schema" + - type: object + properties: + version: + type: integer + id: + type: integer + format: int64 + highestFieldId: + type: integer + timeMillis: + type: integer + format: int64 + GetSchemaResponse: + type: object + properties: + schema: + $ref: "#/components/schemas/TableSchema" + ListSchemasResponse: + type: object + properties: + schemas: + type: array + items: + $ref: "#/components/schemas/TableSchema" + nextPageToken: + type: + - string + - "null" SemanticViewDefinition: type: object required: [ format, content ] @@ -3753,6 +4375,37 @@ components: $ref: '#/components/schemas/Identifier' nextPageToken: type: string + CreateDatabaseTagRequest: + type: object + required: + - tagName + properties: + tagName: + type: string + pattern: "^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$" + fromBranch: + type: [ string, "null" ] + description: Source database branch; absent or null selects main. + timeRetained: + type: [ string, "null" ] + description: Optional retention duration, using the table tag duration syntax. + GetDatabaseTagResponse: + type: object + required: + - tagName + - fromBranch + properties: + tagName: + type: string + fromBranch: + type: string + description: Source branch recorded at creation, even if subsequently deleted. + tagCreateTime: + type: [ integer, "null" ] + format: int64 + description: Creation time as milliseconds since the Unix epoch. + tagTimeRetained: + type: [ string, "null" ] ConfigResponse: type: object properties: @@ -3787,9 +4440,7 @@ components: type: string ForwardBranchRequest: type: object - properties: - branch: - type: string + description: Empty action request, shared by table and database forward operations. ListBranchesResponse: type: object properties: diff --git a/paimon-api/src/main/java/org/apache/paimon/management/TreeManagement.java b/paimon-api/src/main/java/org/apache/paimon/management/TreeManagement.java new file mode 100644 index 000000000000..87cf377d4554 --- /dev/null +++ b/paimon-api/src/main/java/org/apache/paimon/management/TreeManagement.java @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.management; + +import org.apache.paimon.PagedList; +import org.apache.paimon.annotation.Experimental; +import org.apache.paimon.rest.responses.GetDatabaseTagResponse; + +import javax.annotation.Nullable; + +import java.util.List; + +/** Database-level extensions of Paimon's table branch and tag operations. */ +@Experimental +public interface TreeManagement { + + /** Lists branch names, using the same response as table branch listing. */ + List listBranches(String databaseName); + + /** + * Creates a branch. Without fromTag, copies main's table schemas without data. With fromTag, + * copies the membership and table versions captured by that database tag. + */ + void createBranch(String databaseName, String branch, @Nullable String fromTag); + + /** Drops a database branch. The default main branch is protected. */ + void dropBranch(String databaseName, String branch); + + /** + * Forwards main to the named branch, extending the table fast-forward operation to the + * database's tables. The path names the source branch. This replaces target table state; it + * does not perform conflict resolution. Pause writers and reload tables after publication. + */ + void fastForward(String databaseName, String branch); + + /** + * Captures an immutable database tag from a branch. Null fromBranch selects main. There is no + * database-wide snapshot ID; each table contributes its own captured version. + */ + void createTag( + String databaseName, + String tagName, + @Nullable String fromBranch, + @Nullable String timeRetained); + + /** Gets database tag metadata. Table versions are read through the tag-suffixed database. */ + GetDatabaseTagResponse getTag(String databaseName, String tagName); + + /** Lists tag names with the same pagination and prefix filter as table tag listing. */ + PagedList listTagsPaged( + String databaseName, + @Nullable Integer maxResults, + @Nullable String pageToken, + @Nullable String tagNamePrefix); + + /** Deletes a database tag without deleting versions retained by another reference. */ + void deleteTag(String databaseName, String tagName); +} diff --git a/paimon-api/src/main/java/org/apache/paimon/rest/DatabaseIdentifier.java b/paimon-api/src/main/java/org/apache/paimon/rest/DatabaseIdentifier.java new file mode 100644 index 000000000000..6b21c26f7241 --- /dev/null +++ b/paimon-api/src/main/java/org/apache/paimon/rest/DatabaseIdentifier.java @@ -0,0 +1,94 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.rest; + +import org.apache.paimon.annotation.Experimental; +import org.apache.paimon.catalog.Identifier; + +import javax.annotation.Nullable; + +import static org.apache.paimon.utils.Preconditions.checkArgument; + +/** A REST database name and its optional database-level branch or immutable tag selector. */ +@Experimental +public final class DatabaseIdentifier { + + private static final String BRANCH_SUFFIX = "$branch_"; + private static final String TAG_SUFFIX = "$tag_"; + + private final String databaseName; + @Nullable private final DatabaseReference reference; + + private DatabaseIdentifier(String databaseName, @Nullable DatabaseReference reference) { + this.databaseName = databaseName; + this.reference = reference; + } + + /** + * Parses a decoded database name, such as {@code training$branch_experiment}. + * + *

The suffixes {@code $branch_} and {@code $tag_} are reserved. A name containing either + * marker must have exactly one valid reference suffix. Other dollar signs remain literal. + * Callers retain the original name in table identifiers and encode it as one REST path segment. + */ + public static DatabaseIdentifier parse(String name) { + checkArgument(name != null && !name.trim().isEmpty(), "Database name must not be blank"); + int branch = name.indexOf(BRANCH_SUFFIX); + int tag = name.indexOf(TAG_SUFFIX); + if (branch < 0 && tag < 0) { + return new DatabaseIdentifier(name, null); + } + boolean isBranch = branch >= 0 && (tag < 0 || branch < tag); + int separator = isBranch ? branch : tag; + String database = name.substring(0, separator); + checkArgument(!database.trim().isEmpty(), "Database name must not be blank"); + String reference = + name.substring( + separator + (isBranch ? BRANCH_SUFFIX.length() : TAG_SUFFIX.length())); + return new DatabaseIdentifier( + database, + new DatabaseReference( + isBranch ? DatabaseReferenceType.BRANCH : DatabaseReferenceType.TAG, + reference)); + } + + /** The physical database name, without the reference suffix. */ + public String getDatabaseName() { + return databaseName; + } + + @Nullable + public DatabaseReference getReference() { + return reference; + } + + static void checkNoReference(String database, String operation) { + if (parse(database).getReference() != null) { + throw new UnsupportedOperationException( + operation + " does not support database reference suffixes: " + database); + } + } + + static void checkTableName(String database, String table) { + checkArgument( + parse(database).getReference() == null + || Identifier.create(database, table).getBranchName() == null, + "Table branch suffixes cannot be combined with a database reference"); + } +} diff --git a/paimon-api/src/main/java/org/apache/paimon/rest/DatabaseReference.java b/paimon-api/src/main/java/org/apache/paimon/rest/DatabaseReference.java new file mode 100644 index 000000000000..845a68deb003 --- /dev/null +++ b/paimon-api/src/main/java/org/apache/paimon/rest/DatabaseReference.java @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.rest; + +import org.apache.paimon.annotation.Experimental; + +import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonCreator; +import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonGetter; +import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonProperty; + +import java.beans.ConstructorProperties; +import java.util.Objects; + +import static org.apache.paimon.utils.Preconditions.checkArgument; + +/** A named database-level branch or immutable tag. */ +@Experimental +public class DatabaseReference { + + private static final String FIELD_TYPE = "type"; + private static final String FIELD_NAME = "name"; + private static final String NAME_PATTERN = "[A-Za-z0-9][A-Za-z0-9._-]{0,127}"; + + @JsonProperty(FIELD_TYPE) + private final DatabaseReferenceType type; + + @JsonProperty(FIELD_NAME) + private final String name; + + @JsonCreator + @ConstructorProperties({FIELD_TYPE, FIELD_NAME}) + public DatabaseReference( + @JsonProperty(FIELD_TYPE) DatabaseReferenceType type, + @JsonProperty(FIELD_NAME) String name) { + checkArgument(type != null, "Reference type must not be null"); + checkArgument( + name != null && name.matches(NAME_PATTERN), "Invalid reference name: %s", name); + this.type = type; + this.name = name; + } + + @JsonGetter(FIELD_TYPE) + public DatabaseReferenceType getType() { + return type; + } + + @JsonGetter(FIELD_NAME) + public String getName() { + return name; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof DatabaseReference)) { + return false; + } + DatabaseReference that = (DatabaseReference) o; + return type == that.type && name.equals(that.name); + } + + @Override + public int hashCode() { + return Objects.hash(type, name); + } + + @Override + public String toString() { + return type + ":" + name; + } +} diff --git a/paimon-api/src/main/java/org/apache/paimon/rest/DatabaseReferenceType.java b/paimon-api/src/main/java/org/apache/paimon/rest/DatabaseReferenceType.java new file mode 100644 index 000000000000..6d13566bcb5d --- /dev/null +++ b/paimon-api/src/main/java/org/apache/paimon/rest/DatabaseReferenceType.java @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.rest; + +import org.apache.paimon.annotation.Experimental; + +/** Types of database-level references supported by the REST catalog. */ +@Experimental +public enum DatabaseReferenceType { + BRANCH, + TAG +} diff --git a/paimon-api/src/main/java/org/apache/paimon/rest/RESTApi.java b/paimon-api/src/main/java/org/apache/paimon/rest/RESTApi.java index 8246cb153335..14f4545b22a4 100644 --- a/paimon-api/src/main/java/org/apache/paimon/rest/RESTApi.java +++ b/paimon-api/src/main/java/org/apache/paimon/rest/RESTApi.java @@ -48,6 +48,7 @@ import org.apache.paimon.rest.requests.CommitTableRequest; import org.apache.paimon.rest.requests.CreateBranchRequest; import org.apache.paimon.rest.requests.CreateDatabaseRequest; +import org.apache.paimon.rest.requests.CreateDatabaseTagRequest; import org.apache.paimon.rest.requests.CreateFunctionRequest; import org.apache.paimon.rest.requests.CreatePartitionsRequest; import org.apache.paimon.rest.requests.CreateTableRequest; @@ -78,6 +79,7 @@ import org.apache.paimon.rest.responses.DropPartitionsResponse; import org.apache.paimon.rest.responses.ErrorResponse; import org.apache.paimon.rest.responses.GetDatabaseResponse; +import org.apache.paimon.rest.responses.GetDatabaseTagResponse; import org.apache.paimon.rest.responses.GetFunctionResponse; import org.apache.paimon.rest.responses.GetLabelResponse; import org.apache.paimon.rest.responses.GetSchemaResponse; @@ -312,6 +314,7 @@ public PagedList listDatabasesPaged( * this database */ public void createDatabase(String name, Map properties) { + DatabaseIdentifier.checkNoReference(name, "createDatabase"); CreateDatabaseRequest request = new CreateDatabaseRequest(name, properties); client.post(resourcePaths.databases(), request, restAuthFunction); } @@ -339,6 +342,7 @@ public GetDatabaseResponse getDatabase(String name) { * this database */ public void dropDatabase(String name) { + DatabaseIdentifier.checkNoReference(name, "dropDatabase"); client.delete(resourcePaths.database(name), restAuthFunction); } @@ -353,6 +357,7 @@ public void dropDatabase(String name) { * this database */ public void alterDatabase(String name, List removals, Map updates) { + DatabaseIdentifier.checkNoReference(name, "alterDatabase"); client.post( resourcePaths.database(name), new AlterDatabaseRequest(removals, updates), @@ -360,6 +365,88 @@ public void alterDatabase(String name, List removals, Map listDatabaseBranches(String databaseName) { + ListBranchesResponse response = + client.get( + resourcePaths.databaseBranches(databaseName), + ListBranchesResponse.class, + restAuthFunction); + return response.branches() == null ? emptyList() : response.branches(); + } + + /** Creates a schema-only branch, or restores the versions captured by fromTag. */ + @Experimental + public void createDatabaseBranch(String databaseName, String branch, @Nullable String fromTag) { + client.post( + resourcePaths.databaseBranches(databaseName), + new CreateBranchRequest(branch, fromTag), + restAuthFunction); + } + + /** Drops a database branch. */ + @Experimental + public void dropDatabaseBranch(String databaseName, String branch) { + client.delete(resourcePaths.databaseBranch(databaseName, branch), restAuthFunction); + } + + /** Forwards main to the source branch, using the table forward request. */ + @Experimental + public void fastForwardDatabase(String databaseName, String branch) { + client.post( + resourcePaths.forwardDatabaseBranch(databaseName, branch), + new ForwardBranchRequest(), + restAuthFunction); + } + + /** Captures the selected database branch; null fromBranch selects main. */ + @Experimental + public void createDatabaseTag( + String databaseName, + String tagName, + @Nullable String fromBranch, + @Nullable String timeRetained) { + client.post( + resourcePaths.databaseTags(databaseName), + new CreateDatabaseTagRequest(tagName, fromBranch, timeRetained), + restAuthFunction); + } + + /** Gets database tag metadata, without a fictitious database-wide snapshot ID. */ + @Experimental + public GetDatabaseTagResponse getDatabaseTag(String databaseName, String tagName) { + return client.get( + resourcePaths.databaseTag(databaseName, tagName), + GetDatabaseTagResponse.class, + restAuthFunction); + } + + /** Lists database tag names with table tag pagination and prefix filtering. */ + @Experimental + public PagedList listDatabaseTagsPaged( + String databaseName, + @Nullable Integer maxResults, + @Nullable String pageToken, + @Nullable String tagNamePrefix) { + ListTagsResponse response = + client.get( + resourcePaths.databaseTags(databaseName), + buildPagedQueryParams( + maxResults, pageToken, Pair.of(TAG_NAME_PREFIX, tagNamePrefix)), + ListTagsResponse.class, + restAuthFunction); + return new PagedList<>( + response.tags() == null ? emptyList() : response.tags(), + response.getNextPageToken()); + } + + /** Deletes a database tag. */ + @Experimental + public void deleteDatabaseTag(String databaseName, String tagName) { + client.delete(resourcePaths.databaseTag(databaseName, tagName), restAuthFunction); + } + /** * List tables for a database. * @@ -811,6 +898,7 @@ public PagedList listSchemasPaged( * creating table */ public void createTable(Identifier identifier, Schema schema) { + DatabaseIdentifier.checkTableName(identifier.getDatabaseName(), identifier.getObjectName()); CreateTableRequest request = new CreateTableRequest(identifier, schema); client.post(resourcePaths.tables(identifier.getDatabaseName()), request, restAuthFunction); } @@ -826,6 +914,8 @@ public void createTable(Identifier identifier, Schema schema) { * renaming table */ public void renameTable(Identifier fromTable, Identifier toTable) { + DatabaseIdentifier.checkNoReference(fromTable.getDatabaseName(), "renameTable"); + DatabaseIdentifier.checkNoReference(toTable.getDatabaseName(), "renameTable"); RenameTableRequest request = new RenameTableRequest(fromTable, toTable); client.post(resourcePaths.renameTable(), request, restAuthFunction); } @@ -1925,6 +2015,8 @@ public PagedList listViewsPagedGlobally( * views */ public void renameView(Identifier fromView, Identifier toView) { + DatabaseIdentifier.checkNoReference(fromView.getDatabaseName(), "renameView"); + DatabaseIdentifier.checkNoReference(toView.getDatabaseName(), "renameView"); RenameTableRequest request = new RenameTableRequest(fromView, toView); client.post(resourcePaths.renameView(), request, restAuthFunction); } diff --git a/paimon-api/src/main/java/org/apache/paimon/rest/RESTTreeManagement.java b/paimon-api/src/main/java/org/apache/paimon/rest/RESTTreeManagement.java new file mode 100644 index 000000000000..358c1e889f76 --- /dev/null +++ b/paimon-api/src/main/java/org/apache/paimon/rest/RESTTreeManagement.java @@ -0,0 +1,87 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.rest; + +import org.apache.paimon.PagedList; +import org.apache.paimon.annotation.Experimental; +import org.apache.paimon.management.TreeManagement; +import org.apache.paimon.rest.responses.GetDatabaseTagResponse; + +import javax.annotation.Nullable; + +import java.util.List; + +/** Database branch and tag management using the REST catalog's configuration. */ +@Experimental +public class RESTTreeManagement implements TreeManagement { + + private final RESTApi api; + + public RESTTreeManagement(RESTApi api) { + this.api = api; + } + + @Override + public List listBranches(String databaseName) { + return api.listDatabaseBranches(databaseName); + } + + @Override + public void createBranch(String databaseName, String branch, @Nullable String fromTag) { + api.createDatabaseBranch(databaseName, branch, fromTag); + } + + @Override + public void dropBranch(String databaseName, String branch) { + api.dropDatabaseBranch(databaseName, branch); + } + + @Override + public void fastForward(String databaseName, String branch) { + api.fastForwardDatabase(databaseName, branch); + } + + @Override + public void createTag( + String databaseName, + String tagName, + @Nullable String fromBranch, + @Nullable String timeRetained) { + api.createDatabaseTag(databaseName, tagName, fromBranch, timeRetained); + } + + @Override + public GetDatabaseTagResponse getTag(String databaseName, String tagName) { + return api.getDatabaseTag(databaseName, tagName); + } + + @Override + public PagedList listTagsPaged( + String databaseName, + @Nullable Integer maxResults, + @Nullable String pageToken, + @Nullable String tagNamePrefix) { + return api.listDatabaseTagsPaged(databaseName, maxResults, pageToken, tagNamePrefix); + } + + @Override + public void deleteTag(String databaseName, String tagName) { + api.deleteDatabaseTag(databaseName, tagName); + } +} diff --git a/paimon-api/src/main/java/org/apache/paimon/rest/ResourcePaths.java b/paimon-api/src/main/java/org/apache/paimon/rest/ResourcePaths.java index f3065708da9c..5477a27314d6 100644 --- a/paimon-api/src/main/java/org/apache/paimon/rest/ResourcePaths.java +++ b/paimon-api/src/main/java/org/apache/paimon/rest/ResourcePaths.java @@ -97,6 +97,7 @@ private static String encodePathSegment(String value) { @Experimental public String semanticViews(String database) { checkArgument(database != null && !database.trim().isEmpty(), "database must not be blank"); + DatabaseIdentifier.checkNoReference(database, "semanticViews"); return SLASH.join(V1, prefix, DATABASES, encodePathSegment(database), SEMANTIC_VIEWS); } @@ -127,6 +128,7 @@ public String revokePermission() { @Experimental public String policies(PermissionResource resource) { resource.validatePolicyAttachment(); + DatabaseIdentifier.checkNoReference(resource.getDatabase(), "policies"); return SLASH.join(table(resource.getDatabase(), resource.getTable()), POLICIES); } @@ -141,15 +143,46 @@ public String databases() { } public String database(String databaseName) { + DatabaseIdentifier.parse(databaseName); return SLASH.join(V1, prefix, DATABASES, encodeString(databaseName)); } + /** Database-level extension of table branch management. */ + @Experimental + public String databaseBranches(String databaseName) { + DatabaseIdentifier.checkNoReference(databaseName, "database branch management"); + return SLASH.join(database(databaseName), BRANCHES); + } + + @Experimental + public String databaseBranch(String databaseName, String branch) { + return SLASH.join(databaseBranches(databaseName), encodeString(branch)); + } + + /** The path names the source branch; main is the target. */ + @Experimental + public String forwardDatabaseBranch(String databaseName, String branch) { + return SLASH.join(databaseBranch(databaseName, branch), "forward"); + } + + /** Database-level extension of table tag management. */ + @Experimental + public String databaseTags(String databaseName) { + DatabaseIdentifier.checkNoReference(databaseName, "database tag management"); + return SLASH.join(database(databaseName), TAGS); + } + + @Experimental + public String databaseTag(String databaseName, String tagName) { + return SLASH.join(databaseTags(databaseName), encodeString(tagName)); + } + public String tables(String databaseName) { - return SLASH.join(V1, prefix, DATABASES, encodeString(databaseName), TABLES); + return SLASH.join(database(databaseName), TABLES); } public String tableDetails(String databaseName) { - return SLASH.join(V1, prefix, DATABASES, encodeString(databaseName), TABLE_DETAILS); + return SLASH.join(database(databaseName), TABLE_DETAILS); } public String tables() { @@ -161,13 +194,8 @@ public String table(String tableId) { } public String table(String databaseName, String objectName) { - return SLASH.join( - V1, - prefix, - DATABASES, - encodeString(databaseName), - TABLES, - encodeString(objectName)); + DatabaseIdentifier.checkTableName(databaseName, objectName); + return SLASH.join(tables(databaseName), encodeString(objectName)); } public String renameTable() { @@ -175,6 +203,7 @@ public String renameTable() { } public String replaceTable(String databaseName, String objectName) { + DatabaseIdentifier.checkNoReference(databaseName, "replaceTable"); return SLASH.join( V1, prefix, @@ -186,17 +215,11 @@ public String replaceTable(String databaseName, String objectName) { } public String commitTable(String databaseName, String objectName) { - return SLASH.join( - V1, - prefix, - DATABASES, - encodeString(databaseName), - TABLES, - encodeString(objectName), - "commit"); + return SLASH.join(table(databaseName, objectName), "commit"); } public String rollbackTable(String databaseName, String objectName) { + DatabaseIdentifier.checkNoReference(databaseName, "rollbackTable"); return SLASH.join( V1, prefix, @@ -208,6 +231,7 @@ public String rollbackTable(String databaseName, String objectName) { } public String rollbackSchemaTable(String databaseName, String objectName) { + DatabaseIdentifier.checkNoReference(databaseName, "rollbackSchemaTable"); return SLASH.join( V1, prefix, @@ -219,63 +243,28 @@ public String rollbackSchemaTable(String databaseName, String objectName) { } public String registerTable(String databaseName) { + DatabaseIdentifier.checkNoReference(databaseName, "registerTable"); return SLASH.join(V1, prefix, DATABASES, encodeString(databaseName), REGISTER); } public String tableToken(String databaseName, String objectName) { - return SLASH.join( - V1, - prefix, - DATABASES, - encodeString(databaseName), - TABLES, - encodeString(objectName), - "token"); + return SLASH.join(table(databaseName, objectName), "token"); } public String tableSnapshot(String databaseName, String objectName) { - return SLASH.join( - V1, - prefix, - DATABASES, - encodeString(databaseName), - TABLES, - encodeString(objectName), - "snapshot"); + return SLASH.join(table(databaseName, objectName), "snapshot"); } public String tableSnapshot(String databaseName, String objectName, String version) { - return SLASH.join( - V1, - prefix, - DATABASES, - encodeString(databaseName), - TABLES, - encodeString(objectName), - SNAPSHOTS, - version); + return SLASH.join(snapshots(databaseName, objectName), encodeString(version)); } public String snapshots(String databaseName, String objectName) { - return SLASH.join( - V1, - prefix, - DATABASES, - encodeString(databaseName), - TABLES, - encodeString(objectName), - SNAPSHOTS); + return SLASH.join(table(databaseName, objectName), SNAPSHOTS); } public String schemas(String databaseName, String objectName) { - return SLASH.join( - V1, - prefix, - DATABASES, - encodeString(databaseName), - TABLES, - encodeString(objectName), - SCHEMAS); + return SLASH.join(table(databaseName, objectName), SCHEMAS); } public String schemas(String databaseName, String objectName, String version) { @@ -283,17 +272,11 @@ public String schemas(String databaseName, String objectName, String version) { } public String authTable(String databaseName, String objectName) { - return SLASH.join( - V1, - prefix, - DATABASES, - encodeString(databaseName), - TABLES, - encodeString(objectName), - "auth"); + return SLASH.join(table(databaseName, objectName), "auth"); } public String partitions(String databaseName, String objectName) { + DatabaseIdentifier.checkNoReference(databaseName, "partitions"); return SLASH.join( V1, prefix, @@ -305,6 +288,7 @@ public String partitions(String databaseName, String objectName) { } public String dropPartitions(String databaseName, String objectName) { + DatabaseIdentifier.checkNoReference(databaseName, "dropPartitions"); return SLASH.join( V1, prefix, @@ -317,6 +301,7 @@ public String dropPartitions(String databaseName, String objectName) { } public String markDonePartitions(String databaseName, String objectName) { + DatabaseIdentifier.checkNoReference(databaseName, "markDonePartitions"); return SLASH.join( V1, prefix, @@ -329,6 +314,7 @@ public String markDonePartitions(String databaseName, String objectName) { } public String listPartitionsByNames(String databaseName, String objectName) { + DatabaseIdentifier.checkNoReference(databaseName, "listPartitionsByNames"); return SLASH.join( V1, prefix, @@ -341,6 +327,7 @@ public String listPartitionsByNames(String databaseName, String objectName) { } public String listPartitionsByFilter(String databaseName, String objectName) { + DatabaseIdentifier.checkNoReference(databaseName, "listPartitionsByFilter"); return SLASH.join( V1, prefix, @@ -353,6 +340,7 @@ public String listPartitionsByFilter(String databaseName, String objectName) { } public String branches(String databaseName, String objectName) { + DatabaseIdentifier.checkNoReference(databaseName, "branches"); return SLASH.join( V1, prefix, @@ -364,6 +352,7 @@ public String branches(String databaseName, String objectName) { } public String branch(String databaseName, String objectName, String branchName) { + DatabaseIdentifier.checkNoReference(databaseName, "branch"); return SLASH.join( V1, prefix, @@ -376,6 +365,7 @@ public String branch(String databaseName, String objectName, String branchName) } public String forwardBranch(String databaseName, String tableName, String branch) { + DatabaseIdentifier.checkNoReference(databaseName, "forwardBranch"); return SLASH.join( V1, prefix, @@ -389,6 +379,7 @@ public String forwardBranch(String databaseName, String tableName, String branch } public String tags(String databaseName, String objectName) { + DatabaseIdentifier.checkNoReference(databaseName, "tags"); return SLASH.join( V1, prefix, @@ -400,6 +391,7 @@ public String tags(String databaseName, String objectName) { } public String consumers(String databaseName, String objectName) { + DatabaseIdentifier.checkNoReference(databaseName, "consumers"); return SLASH.join( V1, prefix, @@ -411,6 +403,7 @@ public String consumers(String databaseName, String objectName) { } public String resetConsumer(String databaseName, String objectName) { + DatabaseIdentifier.checkNoReference(databaseName, "resetConsumer"); return SLASH.join( V1, prefix, @@ -423,6 +416,7 @@ public String resetConsumer(String databaseName, String objectName) { } public String tag(String databaseName, String objectName, String tagName) { + DatabaseIdentifier.checkNoReference(databaseName, "tag"); return SLASH.join( V1, prefix, @@ -435,10 +429,12 @@ public String tag(String databaseName, String objectName, String tagName) { } public String views(String databaseName) { + DatabaseIdentifier.checkNoReference(databaseName, "views"); return SLASH.join(V1, prefix, DATABASES, encodeString(databaseName), VIEWS); } public String viewDetails(String databaseName) { + DatabaseIdentifier.checkNoReference(databaseName, "viewDetails"); return SLASH.join(V1, prefix, DATABASES, encodeString(databaseName), VIEW_DETAILS); } @@ -447,6 +443,7 @@ public String views() { } public String view(String databaseName, String viewName) { + DatabaseIdentifier.checkNoReference(databaseName, "view"); return SLASH.join( V1, prefix, DATABASES, encodeString(databaseName), VIEWS, encodeString(viewName)); } @@ -456,6 +453,7 @@ public String renameView() { } public String functions(String databaseName) { + DatabaseIdentifier.checkNoReference(databaseName, "functions"); return SLASH.join(V1, prefix, DATABASES, encodeString(databaseName), FUNCTIONS); } @@ -464,10 +462,12 @@ public String functions() { } public String functionDetails(String databaseName) { + DatabaseIdentifier.checkNoReference(databaseName, "functionDetails"); return SLASH.join(V1, prefix, DATABASES, encodeString(databaseName), FUNCTION_DETAILS); } public String function(String databaseName, String functionName) { + DatabaseIdentifier.checkNoReference(databaseName, "function"); return SLASH.join( V1, prefix, diff --git a/paimon-api/src/main/java/org/apache/paimon/rest/requests/CreateDatabaseTagRequest.java b/paimon-api/src/main/java/org/apache/paimon/rest/requests/CreateDatabaseTagRequest.java new file mode 100644 index 000000000000..a454143cef7d --- /dev/null +++ b/paimon-api/src/main/java/org/apache/paimon/rest/requests/CreateDatabaseTagRequest.java @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.rest.requests; + +import org.apache.paimon.annotation.Experimental; +import org.apache.paimon.rest.RESTRequest; + +import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonCreator; +import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonGetter; +import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonProperty; + +import javax.annotation.Nullable; + +import java.beans.ConstructorProperties; + +/** Database extension of tag creation: capture a branch instead of one table snapshot ID. */ +@Experimental +@JsonIgnoreProperties(ignoreUnknown = true) +public class CreateDatabaseTagRequest implements RESTRequest { + + private static final String FIELD_TAG_NAME = "tagName"; + private static final String FIELD_FROM_BRANCH = "fromBranch"; + private static final String FIELD_TIME_RETAINED = "timeRetained"; + + private final String tagName; + @Nullable private final String fromBranch; + @Nullable private final String timeRetained; + + @JsonCreator + @ConstructorProperties({FIELD_TAG_NAME, FIELD_FROM_BRANCH, FIELD_TIME_RETAINED}) + public CreateDatabaseTagRequest( + @JsonProperty(FIELD_TAG_NAME) String tagName, + @Nullable @JsonProperty(FIELD_FROM_BRANCH) String fromBranch, + @Nullable @JsonProperty(FIELD_TIME_RETAINED) String timeRetained) { + this.tagName = tagName; + this.fromBranch = fromBranch; + this.timeRetained = timeRetained; + } + + @JsonGetter(FIELD_TAG_NAME) + public String tagName() { + return tagName; + } + + /** Null selects the database's main branch. */ + @Nullable + @JsonGetter(FIELD_FROM_BRANCH) + public String fromBranch() { + return fromBranch; + } + + @Nullable + @JsonGetter(FIELD_TIME_RETAINED) + public String timeRetained() { + return timeRetained; + } +} diff --git a/paimon-api/src/main/java/org/apache/paimon/rest/responses/GetDatabaseTagResponse.java b/paimon-api/src/main/java/org/apache/paimon/rest/responses/GetDatabaseTagResponse.java new file mode 100644 index 000000000000..9a5e88b53c59 --- /dev/null +++ b/paimon-api/src/main/java/org/apache/paimon/rest/responses/GetDatabaseTagResponse.java @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.rest.responses; + +import org.apache.paimon.annotation.Experimental; +import org.apache.paimon.rest.RESTResponse; + +import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonCreator; +import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonGetter; +import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonProperty; + +import javax.annotation.Nullable; + +/** + * Database tag metadata. Its captured table versions are accessed through the database selector. + */ +@Experimental +@JsonIgnoreProperties(ignoreUnknown = true) +public class GetDatabaseTagResponse implements RESTResponse { + + private static final String FIELD_TAG_NAME = "tagName"; + private static final String FIELD_FROM_BRANCH = "fromBranch"; + private static final String FIELD_TAG_CREATE_TIME = "tagCreateTime"; + private static final String FIELD_TAG_TIME_RETAINED = "tagTimeRetained"; + + private final String tagName; + private final String fromBranch; + @Nullable private final Long tagCreateTime; + @Nullable private final String tagTimeRetained; + + @JsonCreator + public GetDatabaseTagResponse( + @JsonProperty(FIELD_TAG_NAME) String tagName, + @JsonProperty(FIELD_FROM_BRANCH) String fromBranch, + @Nullable @JsonProperty(FIELD_TAG_CREATE_TIME) Long tagCreateTime, + @Nullable @JsonProperty(FIELD_TAG_TIME_RETAINED) String tagTimeRetained) { + this.tagName = tagName; + this.fromBranch = fromBranch; + this.tagCreateTime = tagCreateTime; + this.tagTimeRetained = tagTimeRetained; + } + + @JsonGetter(FIELD_TAG_NAME) + public String tagName() { + return tagName; + } + + @JsonGetter(FIELD_FROM_BRANCH) + public String fromBranch() { + return fromBranch; + } + + @Nullable + @JsonGetter(FIELD_TAG_CREATE_TIME) + public Long tagCreateTime() { + return tagCreateTime; + } + + @Nullable + @JsonGetter(FIELD_TAG_TIME_RETAINED) + public String tagTimeRetained() { + return tagTimeRetained; + } +} diff --git a/paimon-api/src/test/java/org/apache/paimon/rest/DatabaseIdentifierTest.java b/paimon-api/src/test/java/org/apache/paimon/rest/DatabaseIdentifierTest.java new file mode 100644 index 000000000000..edba6dcb8d54 --- /dev/null +++ b/paimon-api/src/test/java/org/apache/paimon/rest/DatabaseIdentifierTest.java @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.rest; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.NullAndEmptySource; +import org.junit.jupiter.params.provider.ValueSource; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** Tests the reserved database reference suffix grammar independently of HTTP encoding. */ +class DatabaseIdentifierTest { + + @Test + void testBranchAndTagSelectors() { + DatabaseIdentifier branch = DatabaseIdentifier.parse("training db$branch_experiment"); + assertThat(branch.getDatabaseName()).isEqualTo("training db"); + assertThat(branch.getReference()) + .isEqualTo(new DatabaseReference(DatabaseReferenceType.BRANCH, "experiment")); + + DatabaseIdentifier tag = DatabaseIdentifier.parse("training$literal$tag_train_v1"); + assertThat(tag.getDatabaseName()).isEqualTo("training$literal"); + assertThat(tag.getReference()) + .isEqualTo(new DatabaseReference(DatabaseReferenceType.TAG, "train_v1")); + + assertThat(DatabaseIdentifier.parse("training$branch_123").getReference()) + .isEqualTo(new DatabaseReference(DatabaseReferenceType.BRANCH, "123")); + } + + @ParameterizedTest + @ValueSource( + strings = { + "training", + "training$literal", + "training$Branch_main", + "training%24branch_main" + }) + void testOrdinaryNamesRemainLiteral(String name) { + DatabaseIdentifier identifier = DatabaseIdentifier.parse(name); + assertThat(identifier.getDatabaseName()).isEqualTo(name); + assertThat(identifier.getReference()).isNull(); + } + + @ParameterizedTest + @NullAndEmptySource + @ValueSource( + strings = { + " ", + "$branch_main", + "$tag_v1", + "training$branch_", + "training$tag_", + "training$branch_a/b", + "training$tag_..", + "training$branch_a$branch_b", + "training$branch_a$tag_b", + "training$tag_a$branch_b" + }) + void testMalformedSelectorsAreNotLiteralDatabaseNames(String name) { + assertThatThrownBy(() -> DatabaseIdentifier.parse(name)) + .isInstanceOf(IllegalArgumentException.class); + } +} diff --git a/paimon-api/src/test/java/org/apache/paimon/rest/RESTApiDatabaseBranchTagTest.java b/paimon-api/src/test/java/org/apache/paimon/rest/RESTApiDatabaseBranchTagTest.java new file mode 100644 index 000000000000..867ab09a147f --- /dev/null +++ b/paimon-api/src/test/java/org/apache/paimon/rest/RESTApiDatabaseBranchTagTest.java @@ -0,0 +1,306 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.rest; + +import org.apache.paimon.PagedList; +import org.apache.paimon.catalog.Identifier; +import org.apache.paimon.options.Options; +import org.apache.paimon.rest.responses.GetDatabaseTagResponse; + +import com.sun.net.httpserver.HttpExchange; +import com.sun.net.httpserver.HttpServer; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.InetSocketAddress; +import java.net.URLDecoder; +import java.nio.charset.StandardCharsets; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Queue; +import java.util.concurrent.ConcurrentLinkedQueue; +import java.util.concurrent.CopyOnWriteArrayList; + +import static org.apache.paimon.rest.RESTCatalogInternalOptions.PREFIX; +import static org.apache.paimon.rest.RESTCatalogOptions.TOKEN; +import static org.apache.paimon.rest.RESTCatalogOptions.TOKEN_PROVIDER; +import static org.apache.paimon.rest.RESTCatalogOptions.URI; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** HTTP contract tests for database-level branches and immutable tags. */ +class RESTApiDatabaseBranchTagTest { + + private static final String DATABASE_PATH = "/v1/catalog%2Fid/databases/training+db"; + + private final Queue replies = new ConcurrentLinkedQueue<>(); + private final List requests = new CopyOnWriteArrayList<>(); + + private HttpServer server; + private RESTApi api; + + @BeforeEach + void setUp() throws IOException { + server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0); + server.createContext( + "/v1/", + exchange -> { + requests.add(new Request(exchange)); + Reply reply = replies.poll(); + if (reply == null) { + reply = new Reply(500, "{\"code\":500,\"message\":\"unexpected request\"}"); + } + byte[] data = reply.body.getBytes(StandardCharsets.UTF_8); + exchange.getResponseHeaders().set("Content-Type", "application/json"); + exchange.sendResponseHeaders(reply.code, data.length == 0 ? -1 : data.length); + if (data.length > 0) { + try (OutputStream output = exchange.getResponseBody()) { + output.write(data); + } + } + exchange.close(); + }); + server.start(); + + Options options = new Options(); + options.set(URI, "http://127.0.0.1:" + server.getAddress().getPort()); + options.set(PREFIX, "catalog/id"); + options.set(TOKEN_PROVIDER, "bear"); + options.set(TOKEN, "test-token"); + api = new RESTApi(options, false); + } + + @AfterEach + void tearDown() { + if (server != null) { + server.stop(0); + } + } + + @Test + void testBranchRequestsMatchTableRequests() throws Exception { + Identifier table = Identifier.create("training db", "features"); + enqueue(200, ""); + api.createDatabaseBranch("training db", "experiment", "baseline"); + enqueue(200, ""); + api.createBranch(table, "experiment", "baseline"); + assertRequest(0, "POST", DATABASE_PATH + "/branches"); + assertRequest(1, "POST", DATABASE_PATH + "/tables/features/branches"); + assertBody(requests.get(0), "{\"branch\":\"experiment\",\"fromTag\":\"baseline\"}"); + assertThat(requests.get(0).body).isEqualTo(requests.get(1).body); + + enqueue(200, ""); + api.createDatabaseBranch("training db", "empty", null); + assertBody(requests.get(2), "{\"branch\":\"empty\",\"fromTag\":null}"); + + enqueue(200, ""); + api.fastForwardDatabase("training db", "experiment"); + enqueue(200, ""); + api.fastForward(table, "experiment"); + assertRequest(3, "POST", DATABASE_PATH + "/branches/experiment/forward"); + assertRequest(4, "POST", DATABASE_PATH + "/tables/features/branches/experiment/forward"); + assertBody(requests.get(3), "{}"); + assertThat(requests.get(3).body).isEqualTo(requests.get(4).body); + + enqueue(200, ""); + api.dropDatabaseBranch("training db", "experiment"); + assertRequest(5, "DELETE", DATABASE_PATH + "/branches/experiment"); + assertThat(requests.get(5).body).isEmpty(); + assertThat(requests.get(5).query).isNull(); + assertThat(requests).hasSize(6); + } + + @Test + void testListBranchesUsesTableResponse() { + enqueue(200, "{\"branches\":[\"main\",\"experiment\"]}"); + assertThat(api.listDatabaseBranches("training db")).containsExactly("main", "experiment"); + assertRequest(0, "GET", DATABASE_PATH + "/branches"); + assertThat(requests.get(0).query).isNull(); + enqueue(200, "{}"); + assertThat(api.listDatabaseBranches("training db")).isEmpty(); + } + + @Test + void testDatabaseTagMetadataAndDefaults() throws Exception { + enqueue(200, ""); + api.createDatabaseTag("training db", "train-v1", "experiment", "7d"); + assertRequest(0, "POST", DATABASE_PATH + "/tags"); + assertBody( + requests.get(0), + "{\"tagName\":\"train-v1\",\"fromBranch\":\"experiment\",\"timeRetained\":\"7d\"}"); + + enqueue(200, ""); + api.createDatabaseTag("training db", "baseline", null, null); + assertBody( + requests.get(1), + "{\"tagName\":\"baseline\",\"fromBranch\":null,\"timeRetained\":null}"); + + enqueue( + 200, + "{\"tagName\":\"train-v1\",\"fromBranch\":\"experiment\",\"tagCreateTime\":1720000000000,\"tagTimeRetained\":\"7d\",\"futureField\":true}"); + GetDatabaseTagResponse tag = api.getDatabaseTag("training db", "train-v1"); + assertThat(tag.tagName()).isEqualTo("train-v1"); + assertThat(tag.fromBranch()).isEqualTo("experiment"); + assertThat(tag.tagCreateTime()).isEqualTo(1720000000000L); + assertThat(tag.tagTimeRetained()).isEqualTo("7d"); + assertRequest(2, "GET", DATABASE_PATH + "/tags/train-v1"); + + enqueue(200, ""); + api.deleteDatabaseTag("training db", "train-v1"); + assertRequest(3, "DELETE", DATABASE_PATH + "/tags/train-v1"); + assertThat(requests.get(3).body).isEmpty(); + assertThat(requests.get(3).query).isNull(); + assertThat(requests).hasSize(4); + } + + @Test + void testTagPagesUseTableResponseAndPreserveTokens() { + enqueue(200, "{\"tags\":[\"train-v1\"],\"nextPageToken\":\"next +/%?&\"}"); + PagedList first = + api.listDatabaseTagsPaged("training db", 10, "start +/%", "train-"); + assertThat(first.getElements()).containsExactly("train-v1"); + assertThat(first.getNextPageToken()).isEqualTo("next +/%?&"); + assertRequest(0, "GET", DATABASE_PATH + "/tags"); + Map query = queryParameters(requests.get(0).query); + assertThat(query) + .hasSize(3) + .containsEntry("maxResults", "10") + .containsEntry("pageToken", "start +/%") + .containsEntry("tagNamePrefix", "train-"); + + enqueue(200, "{\"tags\":[\"train-v2\"]}"); + PagedList second = + api.listDatabaseTagsPaged("training db", null, first.getNextPageToken(), "train-"); + assertThat(second.getElements()).containsExactly("train-v2"); + assertThat(second.getNextPageToken()).isNull(); + assertThat(queryParameters(requests.get(1).query)) + .hasSize(2) + .containsEntry("pageToken", "next +/%?&") + .containsEntry("tagNamePrefix", "train-"); + + enqueue(200, "{\"tags\":null,\"nextPageToken\":\"continue\"}"); + PagedList empty = api.listDatabaseTagsPaged("training db", null, null, null); + assertThat(empty.getElements()).isEmpty(); + assertThat(empty.getNextPageToken()).isEqualTo("continue"); + assertThat(requests.get(2).query).isNull(); + } + + @Test + void testManagementRejectsVirtualDatabaseNamesBeforeSendingRequests() { + for (String database : + new String[] {"training db$branch_experiment", "training db$tag_train-v1"}) { + assertThatThrownBy(() -> api.listDatabaseBranches(database)) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> api.createDatabaseBranch(database, "new", null)) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> api.dropDatabaseBranch(database, "experiment")) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> api.fastForwardDatabase(database, "experiment")) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> api.createDatabaseTag(database, "new", null, null)) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> api.getDatabaseTag(database, "train-v1")) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> api.listDatabaseTagsPaged(database, null, null, null)) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> api.deleteDatabaseTag(database, "train-v1")) + .isInstanceOf(UnsupportedOperationException.class); + } + assertThat(requests).isEmpty(); + } + + private void enqueue(int code, String body) { + replies.add(new Reply(code, body)); + } + + private void assertRequest(int index, String method, String path) { + Request request = requests.get(index); + assertThat(request.method).isEqualTo(method); + assertThat(request.path).isEqualTo(path); + assertThat(request.authorization).isEqualTo("Bearer test-token"); + } + + private static void assertBody(Request request, String expectedJson) throws Exception { + assertThat(request.query).isNull(); + assertThat(RESTApi.fromJson(request.body, Map.class)) + .isEqualTo(RESTApi.fromJson(expectedJson, Map.class)); + } + + private static Map queryParameters(String query) { + Map values = new LinkedHashMap<>(); + if (query == null || query.isEmpty()) { + return values; + } + for (String parameter : query.split("&")) { + String[] pair = parameter.split("=", 2); + values.put(decode(pair[0]), decode(pair[1])); + } + return values; + } + + private static String decode(String value) { + try { + return URLDecoder.decode(value, "UTF-8"); + } catch (Exception e) { + throw new RuntimeException(e); + } + } + + private static class Reply { + private final int code; + private final String body; + + private Reply(int code, String body) { + this.code = code; + this.body = body; + } + } + + private static class Request { + private final String method; + private final String path; + private final String query; + private final String body; + private final String authorization; + + private Request(HttpExchange exchange) throws IOException { + method = exchange.getRequestMethod(); + path = exchange.getRequestURI().getRawPath(); + query = exchange.getRequestURI().getRawQuery(); + body = read(exchange.getRequestBody()); + authorization = exchange.getRequestHeaders().getFirst("Authorization"); + } + + private static String read(InputStream input) throws IOException { + ByteArrayOutputStream output = new ByteArrayOutputStream(); + byte[] buffer = new byte[1024]; + int length; + while ((length = input.read(buffer)) >= 0) { + output.write(buffer, 0, length); + } + return new String(output.toByteArray(), StandardCharsets.UTF_8); + } + } +} diff --git a/paimon-api/src/test/java/org/apache/paimon/rest/requests/RequestJacksonCompatibilityTest.java b/paimon-api/src/test/java/org/apache/paimon/rest/requests/RequestJacksonCompatibilityTest.java index e7e8a5517e94..83ad1da80192 100644 --- a/paimon-api/src/test/java/org/apache/paimon/rest/requests/RequestJacksonCompatibilityTest.java +++ b/paimon-api/src/test/java/org/apache/paimon/rest/requests/RequestJacksonCompatibilityTest.java @@ -158,6 +158,17 @@ public class RequestJacksonCompatibilityTest { "{\"schemaId\":44}", request -> assertThat(request.getSchemaId()).isEqualTo(44L), "schemaId"), + requestCase( + CreateDatabaseTagRequest.class, + "{\"tagName\":\"train-v1\",\"fromBranch\":\"experiment\",\"timeRetained\":\"7d\"}", + request -> { + assertThat(request.tagName()).isEqualTo("train-v1"); + assertThat(request.fromBranch()).isEqualTo("experiment"); + assertThat(request.timeRetained()).isEqualTo("7d"); + }, + "tagName", + "fromBranch", + "timeRetained"), requestCase( UpsertLabelRequest.class, "{\"value\":\"identifier\"}", @@ -211,6 +222,18 @@ void testConstructorPropertyNamesAndOrder(RequestCase requestCase) { .isEqualTo(requestCase.propertyNames); } + @Test + void testDatabaseTagRequestDefaultsRoundTrip() throws Exception { + CreateDatabaseTagRequest request = + EXTERNAL_MAPPER.readValue( + "{\"tagName\":\"train-v1\"}", CreateDatabaseTagRequest.class); + CreateDatabaseTagRequest roundTrip = + RESTApi.fromJson(RESTApi.toJson(request), CreateDatabaseTagRequest.class); + assertThat(roundTrip.tagName()).isEqualTo("train-v1"); + assertThat(roundTrip.fromBranch()).isNull(); + assertThat(roundTrip.timeRetained()).isNull(); + } + @Test void testRequestCreatorAllowlistsAreComplete() throws Exception { Set> simpleRequests = diff --git a/paimon-core/src/main/java/org/apache/paimon/catalog/CachingCatalog.java b/paimon-core/src/main/java/org/apache/paimon/catalog/CachingCatalog.java index 7fce5edf0ef4..b57c9532c894 100644 --- a/paimon-core/src/main/java/org/apache/paimon/catalog/CachingCatalog.java +++ b/paimon-core/src/main/java/org/apache/paimon/catalog/CachingCatalog.java @@ -25,6 +25,10 @@ import org.apache.paimon.options.Options; import org.apache.paimon.partition.Partition; import org.apache.paimon.partition.PartitionStatistics; +import org.apache.paimon.rest.DatabaseIdentifier; +import org.apache.paimon.rest.DatabaseReference; +import org.apache.paimon.rest.DatabaseReferenceType; +import org.apache.paimon.rest.RESTCatalog; import org.apache.paimon.schema.Schema; import org.apache.paimon.schema.SchemaChange; import org.apache.paimon.table.FileStoreTable; @@ -46,6 +50,7 @@ import java.util.Map; import java.util.Optional; +import static org.apache.paimon.catalog.Identifier.DEFAULT_MAIN_BRANCH; import static org.apache.paimon.options.CatalogOptions.CACHE_DV_MAX_NUM; import static org.apache.paimon.options.CatalogOptions.CACHE_ENABLED; import static org.apache.paimon.options.CatalogOptions.CACHE_EXPIRE_AFTER_ACCESS; @@ -385,6 +390,24 @@ public void alterPartitions(Identifier identifier, List par @Override public void invalidateTable(Identifier identifier) { + invalidateTableCache(identifier); + if (DelegateCatalog.rootCatalog(wrapped) instanceof RESTCatalog) { + DatabaseIdentifier database = DatabaseIdentifier.parse(identifier.getDatabaseName()); + DatabaseReference reference = database.getReference(); + String alias = null; + if (reference == null) { + alias = database.getDatabaseName() + "$branch_" + DEFAULT_MAIN_BRANCH; + } else if (reference.getType() == DatabaseReferenceType.BRANCH + && DEFAULT_MAIN_BRANCH.equals(reference.getName())) { + alias = database.getDatabaseName(); + } + if (alias != null) { + invalidateTableCache(Identifier.create(alias, identifier.getObjectName())); + } + } + } + + private void invalidateTableCache(Identifier identifier) { tableCache.invalidate(identifier); if (partitionCache != null) { partitionCache.invalidate(identifier); diff --git a/paimon-core/src/main/java/org/apache/paimon/catalog/CatalogSnapshotCommit.java b/paimon-core/src/main/java/org/apache/paimon/catalog/CatalogSnapshotCommit.java index fc50e33b24a0..f6ce3220003d 100644 --- a/paimon-core/src/main/java/org/apache/paimon/catalog/CatalogSnapshotCommit.java +++ b/paimon-core/src/main/java/org/apache/paimon/catalog/CatalogSnapshotCommit.java @@ -31,11 +31,21 @@ public class CatalogSnapshotCommit implements SnapshotCommit { private final Catalog catalog; private final Identifier identifier; @Nullable private final String uuid; + @Nullable private final String storageBranch; public CatalogSnapshotCommit(Catalog catalog, Identifier identifier, @Nullable String uuid) { + this(catalog, identifier, uuid, null); + } + + public CatalogSnapshotCommit( + Catalog catalog, + Identifier identifier, + @Nullable String uuid, + @Nullable String storageBranch) { this.catalog = catalog; this.identifier = identifier; this.uuid = uuid; + this.storageBranch = storageBranch; } @Override @@ -45,8 +55,14 @@ public boolean commit( String branch, List statistics) throws Exception { + // REST resolves the original logical identifier to its physical storage branch. Keep + // that identifier even when main is backed by a different branch after publication. + // An explicit switch away from the loaded storage branch still selects a table branch. Identifier newIdentifier = - new Identifier(identifier.getDatabaseName(), identifier.getTableName(), branch); + storageBranch != null && storageBranch.equals(branch) + ? identifier + : new Identifier( + identifier.getDatabaseName(), identifier.getTableName(), branch); return catalog.commitSnapshot(newIdentifier, uuid, baseSnapshotUuid, snapshot, statistics); } diff --git a/paimon-core/src/main/java/org/apache/paimon/catalog/CatalogUtils.java b/paimon-core/src/main/java/org/apache/paimon/catalog/CatalogUtils.java index 916f63350c7b..a3746e8dd938 100644 --- a/paimon-core/src/main/java/org/apache/paimon/catalog/CatalogUtils.java +++ b/paimon-core/src/main/java/org/apache/paimon/catalog/CatalogUtils.java @@ -394,7 +394,8 @@ public static Table loadTable( isRestCatalog ? null : lockContext, catalogContext, catalog.supportsVersionManagement(), - catalog.supportsPartitionModification()); + catalog.supportsPartitionModification(), + isRestCatalog ? options.branch() : null); Path path = new Path(schema.options().get(PATH.key())); FileStoreTable table = FileStoreTableFactory.create(dataFileIO.apply(path), path, schema, catalogEnv); diff --git a/paimon-core/src/main/java/org/apache/paimon/rest/RESTCatalog.java b/paimon-core/src/main/java/org/apache/paimon/rest/RESTCatalog.java index f691fc06f5b7..4f683864a38d 100644 --- a/paimon-core/src/main/java/org/apache/paimon/rest/RESTCatalog.java +++ b/paimon-core/src/main/java/org/apache/paimon/rest/RESTCatalog.java @@ -44,6 +44,7 @@ import org.apache.paimon.management.PermissionManagement; import org.apache.paimon.management.PolicyManagement; import org.apache.paimon.management.SemanticViewManagement; +import org.apache.paimon.management.TreeManagement; import org.apache.paimon.options.Options; import org.apache.paimon.partition.Partition; import org.apache.paimon.partition.PartitionStatistics; @@ -166,6 +167,11 @@ public SemanticViewManagement semanticViewManagement() { return new RESTSemanticViewManagement(api); } + @Experimental + public TreeManagement treeManagement() { + return new RESTTreeManagement(api); + } + @Override public List listDatabases() { return api.listDatabases(); @@ -218,6 +224,7 @@ public Database getDatabase(String name) throws DatabaseNotExistException { public void dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade) throws DatabaseNotExistException, DatabaseNotEmptyException { checkNotSystemDatabase(name); + DatabaseIdentifier.checkNoReference(name, "dropDatabase"); try { if (!cascade && !this.listTables(name).isEmpty()) { throw new DatabaseNotEmptyException(name); @@ -640,6 +647,10 @@ public void createTable(Identifier identifier, Schema schema, boolean ignoreIfEx Schema newSchema = inferSchemaIfExternalPaimonTable(schema); api.createTable(identifier, newSchema); } catch (AlreadyExistsException e) { + if (DatabaseIdentifier.parse(identifier.getDatabaseName()).getReference() != null + && !StringUtils.equals(e.resourceType(), ErrorResponse.RESOURCE_TYPE_TABLE)) { + throw e; + } if (!ignoreIfExists) { throw new TableAlreadyExistException(identifier); } @@ -686,6 +697,10 @@ public void alterTable( try { api.alterTable(identifier, changes); } catch (NoSuchResourceException e) { + if (!StringUtils.equals(e.resourceType(), ErrorResponse.RESOURCE_TYPE_TABLE) + && !StringUtils.equals(e.resourceType(), ErrorResponse.RESOURCE_TYPE_COLUMN)) { + throw e; + } if (!ignoreIfNotExists) { if (StringUtils.equals(e.resourceType(), ErrorResponse.RESOURCE_TYPE_TABLE)) { throw new TableNotExistException(identifier); @@ -695,6 +710,10 @@ public void alterTable( } } } catch (AlreadyExistsException e) { + if (DatabaseIdentifier.parse(identifier.getDatabaseName()).getReference() != null + && !StringUtils.equals(e.resourceType(), ErrorResponse.RESOURCE_TYPE_COLUMN)) { + throw e; + } throw new ColumnAlreadyExistException(identifier, e.resourceName()); } catch (ForbiddenException e) { throw new TableNoPermissionException(identifier, e); @@ -1213,6 +1232,10 @@ public PagedList listFunctionDetailsPaged( @Override public View getView(Identifier identifier) throws ViewNotExistException { try { + if (DatabaseIdentifier.parse(identifier.getDatabaseName()).getReference() != null) { + api.getDatabase(identifier.getDatabaseName()); + throw new ViewNotExistException(identifier); + } GetViewResponse response = api.getView(identifier); return toView(identifier.getDatabaseName(), response); } catch (NoSuchResourceException e) { @@ -1264,6 +1287,10 @@ public void createView(Identifier identifier, View view, boolean ignoreIfExists) @Override public List listViews(String databaseName) throws DatabaseNotExistException { try { + if (DatabaseIdentifier.parse(databaseName).getReference() != null) { + api.getDatabase(databaseName); + return Collections.emptyList(); + } return CatalogUtils.isSystemDatabase(databaseName) ? Collections.emptyList() : api.listViews(databaseName); @@ -1282,6 +1309,10 @@ public PagedList listViewsPaged( @Nullable String viewNamePattern) throws DatabaseNotExistException { try { + if (DatabaseIdentifier.parse(databaseName).getReference() != null) { + api.getDatabase(databaseName); + return new PagedList<>(Collections.emptyList(), null); + } return api.listViewsPaged(databaseName, maxResults, pageToken, viewNamePattern); } catch (NoSuchResourceException e) { throw new DatabaseNotExistException(databaseName); @@ -1298,6 +1329,10 @@ public PagedList listViewDetailsPaged( @Nullable String viewNamePattern) throws DatabaseNotExistException { try { + if (DatabaseIdentifier.parse(db).getReference() != null) { + api.getDatabase(db); + return new PagedList<>(Collections.emptyList(), null); + } PagedList views = api.listViewDetailsPaged(db, maxResults, pageToken, viewNamePattern); return new PagedList<>( diff --git a/paimon-core/src/main/java/org/apache/paimon/table/CatalogEnvironment.java b/paimon-core/src/main/java/org/apache/paimon/table/CatalogEnvironment.java index 66979ba811f2..fd9e062cc653 100644 --- a/paimon-core/src/main/java/org/apache/paimon/table/CatalogEnvironment.java +++ b/paimon-core/src/main/java/org/apache/paimon/table/CatalogEnvironment.java @@ -44,6 +44,7 @@ import javax.annotation.Nullable; import java.io.Serializable; +import java.util.Objects; import java.util.Optional; import java.util.function.LongConsumer; @@ -63,6 +64,9 @@ public class CatalogEnvironment implements Serializable { @Nullable private final CatalogContext catalogContext; private final boolean supportsVersionManagement; private final boolean supportsPartitionModification; + // Physical branch resolved when loading a REST table; retained across option + // copies/serialization. + @Nullable private final String storageBranch; public CatalogEnvironment( @Nullable Identifier identifier, @@ -73,6 +77,28 @@ public CatalogEnvironment( @Nullable CatalogContext catalogContext, boolean supportsVersionManagement, boolean supportsPartitionModification) { + this( + identifier, + uuid, + catalogLoader, + lockFactory, + lockContext, + catalogContext, + supportsVersionManagement, + supportsPartitionModification, + null); + } + + public CatalogEnvironment( + @Nullable Identifier identifier, + @Nullable String uuid, + @Nullable CatalogLoader catalogLoader, + @Nullable CatalogLockFactory lockFactory, + @Nullable CatalogLockContext lockContext, + @Nullable CatalogContext catalogContext, + boolean supportsVersionManagement, + boolean supportsPartitionModification, + @Nullable String storageBranch) { this.identifier = identifier; this.uuid = uuid; this.catalogLoader = catalogLoader; @@ -81,6 +107,7 @@ public CatalogEnvironment( this.catalogContext = catalogContext; this.supportsVersionManagement = supportsVersionManagement; this.supportsPartitionModification = supportsPartitionModification; + this.storageBranch = storageBranch; } public static CatalogEnvironment empty() { @@ -135,7 +162,9 @@ public SchemaModification schemaModification() { public SnapshotCommit snapshotCommit(SnapshotManager snapshotManager) { SnapshotCommit snapshotCommit; if (catalogLoader != null && supportsVersionManagement) { - snapshotCommit = new CatalogSnapshotCommit(catalogLoader.load(), identifier, uuid); + snapshotCommit = + new CatalogSnapshotCommit( + catalogLoader.load(), identifier, uuid, storageBranch); } else { Lock lock = Optional.ofNullable(lockFactory) @@ -253,7 +282,8 @@ public CatalogEnvironment copy(Identifier identifier) { lockContext, catalogContext, supportsVersionManagement, - supportsPartitionModification); + supportsPartitionModification, + Objects.equals(this.identifier, identifier) ? storageBranch : null); } public TableQueryAuth tableQueryAuth(CoreOptions options) { diff --git a/paimon-core/src/test/java/org/apache/paimon/catalog/CachingCatalogTest.java b/paimon-core/src/test/java/org/apache/paimon/catalog/CachingCatalogTest.java index 3de7bbf73f16..fbea25196b71 100644 --- a/paimon-core/src/test/java/org/apache/paimon/catalog/CachingCatalogTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/catalog/CachingCatalogTest.java @@ -103,6 +103,25 @@ public void testListDatabasesWhenNoDatabases() { assertThat(databases).contains("db"); } + @Test + public void testReferenceSuffixIsLiteralOutsideRestCatalog() throws Exception { + CachingCatalog cached = new CachingCatalog(catalog, new Options()); + String literalDatabase = "db$branch_main"; + cached.createDatabase(literalDatabase, false); + Identifier main = Identifier.create("db", "features"); + Identifier literal = Identifier.create(literalDatabase, "features"); + Schema schema = Schema.newBuilder().column("id", DataTypes.INT()).build(); + cached.createTable(main, schema, false); + cached.createTable(literal, schema, false); + Table literalTable = cached.getTable(literal); + + cached.alterTable(main, SchemaChange.addColumn("added", DataTypes.STRING()), false); + + assertThat(cached.getTable(main).rowType().getFieldNames()).containsExactly("id", "added"); + assertThat(cached.getTable(literal)).isSameAs(literalTable); + assertThat(literalTable.rowType().getFieldNames()).containsExactly("id"); + } + @Test public void testInvalidateWhenDatabaseIsAltered() throws Exception { Catalog mockcatalog = Mockito.mock(Catalog.class); diff --git a/paimon-core/src/test/java/org/apache/paimon/catalog/CatalogSnapshotCommitTest.java b/paimon-core/src/test/java/org/apache/paimon/catalog/CatalogSnapshotCommitTest.java index 1b5cfeb1cceb..b63b6a6f6a38 100644 --- a/paimon-core/src/test/java/org/apache/paimon/catalog/CatalogSnapshotCommitTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/catalog/CatalogSnapshotCommitTest.java @@ -19,6 +19,8 @@ package org.apache.paimon.catalog; import org.apache.paimon.Snapshot; +import org.apache.paimon.options.Options; +import org.apache.paimon.rest.RESTCatalog; import org.apache.paimon.utils.SnapshotManagerTest; import org.junit.jupiter.api.Test; @@ -59,4 +61,36 @@ public void testCommitForwardsBaseSnapshotUuid() throws Exception { snapshot, Collections.emptyList()); } + + @Test + public void testWrappedRestCatalogKeepsLogicalIdentifier() throws Exception { + RESTCatalog rest = mock(RESTCatalog.class); + Catalog catalog = new CachingCatalog(rest, new Options()); + Identifier identifier = Identifier.create("database", "table"); + Snapshot snapshot = SnapshotManagerTest.createSnapshotWithMillis(2L, 1000L); + when(rest.commitSnapshot( + identifier, + "table-uuid", + "base-snapshot-uuid", + snapshot, + Collections.emptyList())) + .thenReturn(true); + + CatalogSnapshotCommit commit = + new CatalogSnapshotCommit(catalog, identifier, "table-uuid", "physical-main"); + assertThat( + commit.commit( + "base-snapshot-uuid", + snapshot, + "physical-main", + Collections.emptyList())) + .isTrue(); + verify(rest) + .commitSnapshot( + identifier, + "table-uuid", + "base-snapshot-uuid", + snapshot, + Collections.emptyList()); + } } diff --git a/paimon-core/src/test/java/org/apache/paimon/rest/RESTCatalogReferenceTest.java b/paimon-core/src/test/java/org/apache/paimon/rest/RESTCatalogReferenceTest.java new file mode 100644 index 000000000000..3aec2ce1d7f7 --- /dev/null +++ b/paimon-core/src/test/java/org/apache/paimon/rest/RESTCatalogReferenceTest.java @@ -0,0 +1,726 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.rest; + +import org.apache.paimon.Snapshot; +import org.apache.paimon.catalog.CachingCatalog; +import org.apache.paimon.catalog.Catalog; +import org.apache.paimon.catalog.CatalogContext; +import org.apache.paimon.catalog.Identifier; +import org.apache.paimon.catalog.SnapshotCommit; +import org.apache.paimon.data.GenericRow; +import org.apache.paimon.data.InternalRow; +import org.apache.paimon.fs.local.LocalFileIO; +import org.apache.paimon.options.Options; +import org.apache.paimon.reader.RecordReader; +import org.apache.paimon.rest.exceptions.AlreadyExistsException; +import org.apache.paimon.rest.requests.CommitTableRequest; +import org.apache.paimon.rest.requests.CreateTableRequest; +import org.apache.paimon.rest.responses.GetSchemaResponse; +import org.apache.paimon.rest.responses.GetTableResponse; +import org.apache.paimon.schema.FileSystemSchemaManager; +import org.apache.paimon.schema.Schema; +import org.apache.paimon.schema.SchemaChange; +import org.apache.paimon.schema.TableSchema; +import org.apache.paimon.table.FileStoreTable; +import org.apache.paimon.table.sink.BatchTableCommit; +import org.apache.paimon.table.sink.BatchTableWrite; +import org.apache.paimon.table.sink.BatchWriteBuilder; +import org.apache.paimon.table.source.ReadBuilder; +import org.apache.paimon.types.DataTypes; +import org.apache.paimon.utils.InstantiationUtil; +import org.apache.paimon.utils.SnapshotManager; + +import okhttp3.mockwebserver.Dispatcher; +import okhttp3.mockwebserver.MockResponse; +import okhttp3.mockwebserver.MockWebServer; +import okhttp3.mockwebserver.RecordedRequest; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentLinkedQueue; +import java.util.concurrent.TimeUnit; + +import static java.util.Collections.emptyList; +import static java.util.Collections.singletonList; +import static org.apache.paimon.CoreOptions.BRANCH; +import static org.apache.paimon.rest.RESTCatalogOptions.TOKEN; +import static org.apache.paimon.rest.RESTCatalogOptions.TOKEN_PROVIDER; +import static org.apache.paimon.rest.RESTCatalogOptions.URI; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** Verifies reference scope through ordinary table APIs, serialization and storage commits. */ +class RESTCatalogReferenceTest { + + private static final String DATABASE = "training db"; + private static final Identifier TABLE = Identifier.create(DATABASE, "features"); + private static final String DATABASE_PATH = "/v1/catalog%2Fid/databases/training+db"; + private static final String SNAPSHOT_JSON = + "{\"version\":3,\"id\":7,\"schemaId\":2,\"uuid\":\"snapshot-7\"," + + "\"commitKind\":\"APPEND\",\"commitUser\":\"writer\",\"commitIdentifier\":1," + + "\"timeMillis\":1000,\"totalRecordCount\":3,\"deltaRecordCount\":3}"; + + @TempDir Path tempDir; + + private MockWebServer server; + private RESTCatalog catalog; + + @BeforeEach + void setUp() throws Exception { + server = new MockWebServer(); + server.start(); + enqueue( + 200, + "{\"defaults\":{},\"overrides\":{\"prefix\":\"catalog/id\"," + + "\"header.X-Catalog-Context\":\"configured\"}}"); + Options options = new Options(); + options.set(URI, server.url("/").toString()); + options.set(TOKEN_PROVIDER, "bear"); + options.set(TOKEN, "test-token"); + catalog = new RESTCatalog(CatalogContext.create(options)); + assertThat(server.takeRequest(10, TimeUnit.SECONDS).getPath()).isEqualTo("/v1/config"); + } + + @AfterEach + void tearDown() throws Exception { + catalog.close(); + server.shutdown(); + } + + @ParameterizedTest + @ValueSource(strings = {"$branch_experiment", "$tag_train_v1"}) + void testTableAndSerializedLoaderKeepReference(String reference) throws Exception { + String database = DATABASE + reference; + Identifier selected = Identifier.create(database, "features"); + String scope = DATABASE_PATH + reference.replace("$", "%24"); + enqueue(200, "{\"tables\":[\"features\",\"labels\"]}"); + assertThat(catalog.listTables(database)).containsExactly("features", "labels"); + takeRequest("GET", scope + "/tables"); + + enqueue(200, tableResponse(database, "physical-experiment", 2)); + FileStoreTable table = (FileStoreTable) catalog.getTable(selected); + assertThat(table.catalogEnvironment().identifier()).isEqualTo(selected); + assertThat(table.snapshotManager().branch()).isEqualTo("physical-experiment"); + assertThat(table.schema().id()).isEqualTo(2); + takeRequest("GET", scope + "/tables/features"); + + // A task receives a serialized table. Its identifier must retain the database suffix. + FileStoreTable restored = InstantiationUtil.clone(table); + enqueue(200, "{\"snapshot\":{\"snapshot\":" + SNAPSHOT_JSON + "}}"); + assertThat(restored.snapshotManager().latestSnapshot().id()).isEqualTo(7); + takeRequest("GET", scope + "/tables/features/snapshot"); + + RESTCatalog loaded = InstantiationUtil.clone(catalog.catalogLoader()).load(); + enqueue( + 200, + RESTApi.toJson( + new GetSchemaResponse( + TableSchema.create(2, schema("physical-experiment"))))); + assertThat(loaded.loadSchema(selected, "LATEST").get().id()).isEqualTo(2); + takeRequest("GET", scope + "/tables/features/schemas/LATEST"); + + // The same catalog also loads the ordinary database without reference state. + enqueue(200, tableResponse("main")); + FileStoreTable main = (FileStoreTable) catalog.getTable(TABLE); + assertThat(main.snapshotManager().branch()).isEqualTo("main"); + takeRequest("GET", DATABASE_PATH + "/tables/features"); + assertThat(server.getRequestCount()).isEqualTo(6); + } + + @ParameterizedTest + @ValueSource(strings = {"", "$branch_main", "$branch_experiment"}) + void testStorageCommitUsesLogicalTableAndExistingBody(String reference) throws Exception { + Identifier selected = Identifier.create(DATABASE + reference, "features"); + String scope = DATABASE_PATH + reference.replace("$", "%24"); + enqueue(200, tableResponse(selected.getDatabaseName(), "physical-experiment", 2)); + FileStoreTable table = InstantiationUtil.clone((FileStoreTable) catalog.getTable(selected)); + takeRequest("GET", scope + "/tables/features"); + + Snapshot snapshot = Snapshot.fromJson(SNAPSHOT_JSON); + enqueue(200, "{\"success\":true}"); + try (SnapshotCommit commit = + table.catalogEnvironment().snapshotCommit(table.snapshotManager())) { + assertThat( + commit.commit( + "snapshot-6", + snapshot, + table.snapshotManager().branch(), + emptyList())) + .isTrue(); + } + RecordedRequest request = takeRequest("POST", scope + "/tables/features/commit"); + CommitTableRequest body = + RESTApi.fromJson(request.getBody().readUtf8(), CommitTableRequest.class); + assertThat(body.getTableId()).isEqualTo("table-id"); + assertThat(body.getBaseSnapshotUuid()).isEqualTo("snapshot-6"); + assertThat(body.getSnapshot()).isEqualTo(snapshot); + assertThat(body.getStatistics()).isEmpty(); + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void testStorageCommitPreservesExplicitTableBranch(boolean dynamicBranch) throws Exception { + Identifier selected = + Identifier.create(DATABASE, dynamicBranch ? "features" : "features$branch_dev"); + enqueue(200, tableResponse(DATABASE, dynamicBranch ? "physical-main" : "dev", 2)); + FileStoreTable table = InstantiationUtil.clone((FileStoreTable) catalog.getTable(selected)); + takeRequest( + "GET", + DATABASE_PATH + "/tables/" + RESTUtil.encodeString(selected.getObjectName())); + if (dynamicBranch) { + table = + InstantiationUtil.clone( + table.copy(java.util.Collections.singletonMap(BRANCH.key(), "dev"))); + } + + enqueue(200, "{\"success\":true}"); + try (SnapshotCommit commit = + table.catalogEnvironment().snapshotCommit(table.snapshotManager())) { + assertThat( + commit.commit( + "snapshot-6", + Snapshot.fromJson(SNAPSHOT_JSON), + table.snapshotManager().branch(), + emptyList())) + .isTrue(); + } + takeRequest("POST", DATABASE_PATH + "/tables/features%24branch_dev/commit"); + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void testMainAliasesInvalidateTogether(boolean explicitMain) throws Exception { + CachingCatalog cached = new CachingCatalog(catalog, new Options()); + Identifier selected = + Identifier.create(DATABASE + (explicitMain ? "$branch_main" : ""), "features"); + Identifier other = + Identifier.create(DATABASE + (explicitMain ? "" : "$branch_main"), "features"); + String selectedPath = + DATABASE_PATH + (explicitMain ? "%24branch_main" : "") + "/tables/features"; + for (Identifier identifier : new Identifier[] {selected, other}) { + enqueue(200, tableResponse(identifier.getDatabaseName(), "physical-main", 2)); + cached.getTable(identifier); + takeRequest( + "GET", + new ResourcePaths("catalog/id") + .table(identifier.getDatabaseName(), identifier.getObjectName())); + } + + Identifier dev = Identifier.create(DATABASE + "$branch_dev", "features"); + Identifier tag = Identifier.create(DATABASE + "$tag_baseline", "features"); + enqueue(200, tableResponse(dev.getDatabaseName(), "physical-dev", 2)); + FileStoreTable devTable = (FileStoreTable) cached.getTable(dev); + takeRequest("GET", DATABASE_PATH + "%24branch_dev/tables/features"); + enqueue(200, tableResponse(tag.getDatabaseName(), "frozen-baseline", 2)); + FileStoreTable tagTable = (FileStoreTable) cached.getTable(tag); + takeRequest("GET", DATABASE_PATH + "%24tag_baseline/tables/features"); + + enqueue(200, ""); + cached.alterTable( + selected, + java.util.Collections.singletonList(SchemaChange.setOption("comment", "updated")), + false); + takeRequest("POST", selectedPath); + assertMainAliasesReload(cached, selected, other, 3); + + // Forward is followed by explicit invalidation, which must refresh both main aliases. + cached.invalidateTable(selected); + assertMainAliasesReload(cached, selected, other, 4); + + enqueue(200, ""); + cached.dropTable(selected, false); + takeRequest("DELETE", selectedPath); + for (Identifier identifier : new Identifier[] {selected, other}) { + enqueue(404, "{\"code\":404,\"resourceType\":\"TABLE\",\"message\":\"missing\"}"); + assertThatThrownBy(() -> cached.getTable(identifier)) + .isInstanceOf(Catalog.TableNotExistException.class); + takeRequest( + "GET", + new ResourcePaths("catalog/id") + .table(identifier.getDatabaseName(), identifier.getObjectName())); + } + assertThat(cached.getTable(dev)).isSameAs(devTable); + assertThat(cached.getTable(tag)).isSameAs(tagTable); + } + + @Test + void testStorageCommitAfterSwitchingBranchAndCopyingBack() throws Exception { + enqueue(200, tableResponse("main")); + FileStoreTable main = (FileStoreTable) catalog.getTable(TABLE); + takeRequest("GET", DATABASE_PATH + "/tables/features"); + main.schemaManager().copyWithBranch("dev").createTable(schema("dev")); + FileStoreTable copied = + InstantiationUtil.clone( + main.switchToBranch("dev") + .copy(java.util.Collections.singletonMap(BRANCH.key(), "main"))); + + enqueue(200, "{\"success\":true}"); + try (SnapshotCommit commit = + copied.catalogEnvironment().snapshotCommit(copied.snapshotManager())) { + assertThat( + commit.commit( + "snapshot-6", + Snapshot.fromJson(SNAPSHOT_JSON), + copied.snapshotManager().branch(), + emptyList())) + .isTrue(); + } + takeRequest("POST", DATABASE_PATH + "/tables/features/commit"); + } + + private void assertMainAliasesReload( + CachingCatalog cached, Identifier selected, Identifier other, long schemaId) + throws Exception { + for (Identifier identifier : new Identifier[] {selected, other}) { + enqueue(200, tableResponse(identifier.getDatabaseName(), "physical-main", schemaId)); + assertThat(((FileStoreTable) cached.getTable(identifier)).schema().id()) + .isEqualTo(schemaId); + takeRequest( + "GET", + new ResourcePaths("catalog/id") + .table(identifier.getDatabaseName(), identifier.getObjectName())); + } + } + + @Test + void testReadFollowUpsAndPaginationReuseProtocol() throws Exception { + RESTApi api = catalog.api(); + String database = DATABASE + "$tag_train_v1"; + Identifier selected = Identifier.create(database, "features"); + String scope = DATABASE_PATH + "%24tag_train_v1"; + String tablePath = scope + "/tables/features"; + enqueue(200, "{\"tables\":[\"features\"],\"nextPageToken\":\"next\"}"); + assertThat(api.listTablesPaged(database, 1, null, "feat%", null).getNextPageToken()) + .isEqualTo("next"); + RecordedRequest first = takeRequest("GET", scope + "/tables"); + assertThat(first.getRequestUrl().queryParameter("tableNamePattern")).isEqualTo("feat%"); + enqueue(200, "{\"tables\":[\"labels\"]}"); + assertThat(api.listTablesPaged(database, 1, "next", null, null).getElements()) + .containsExactly("labels"); + assertThat( + takeRequest("GET", scope + "/tables") + .getRequestUrl() + .queryParameter("pageToken")) + .isEqualTo("next"); + + enqueue( + 200, + "{\"tableDetails\":[" + tableResponse(database, "physical-experiment", 2) + "]}"); + GetTableResponse details = api.listTableDetails(database).get(0); + assertThat(details.getName()).isEqualTo("features"); + assertThat(details.getDatabase()).isEqualTo(database); + takeRequest("GET", scope + "/table-details"); + + enqueue(200, "{\"snapshot\":" + SNAPSHOT_JSON + "}"); + assertThat(api.loadSnapshot(selected, "LATEST").id()).isEqualTo(7); + takeRequest("GET", tablePath + "/snapshots/LATEST"); + enqueue(200, "{\"snapshots\":[" + SNAPSHOT_JSON + "]}"); + assertThat(api.listSnapshotsPaged(selected, 10, null).getElements().get(0).id()) + .isEqualTo(7); + takeRequest("GET", tablePath + "/snapshots"); + + TableSchema schema = TableSchema.create(2, schema("physical-experiment")); + enqueue(200, "{\"schemas\":[" + RESTApi.toJson(schema) + "]}"); + assertThat(api.listSchemasPaged(selected, 10, null).getElements()).containsExactly(schema); + takeRequest("GET", tablePath + "/schemas"); + + enqueue(200, "{\"token\":{\"key\":\"value\"},\"expiresAtMillis\":1234}"); + assertThat(api.loadTableToken(selected).getToken()).containsEntry("key", "value"); + takeRequest("GET", tablePath + "/token"); + enqueue(200, "{\"filter\":[],\"columnMasking\":{}}"); + api.authTableQuery(selected, singletonList("id")); + assertThat(takeRequest("POST", tablePath + "/auth").getBody().readUtf8()) + .isEqualTo("{\"select\":[\"id\"]}"); + } + + @Test + void testTableMutationsReuseRequestBodies() throws Exception { + Identifier selected = Identifier.create(DATABASE + "$branch_experiment", "features"); + RESTApi api = catalog.api(); + for (Identifier identifier : new Identifier[] {TABLE, selected}) { + enqueue(200, "{}"); + api.createTable(identifier, schema("main")); + enqueue(200, "{}"); + api.alterTable(identifier, singletonList(SchemaChange.setOption("key", "value"))); + enqueue(200, "{}"); + api.dropTable(identifier); + } + RecordedRequest[] original = { + takeRequest("POST", DATABASE_PATH + "/tables"), + takeRequest("POST", DATABASE_PATH + "/tables/features"), + takeRequest("DELETE", DATABASE_PATH + "/tables/features") + }; + String scope = DATABASE_PATH + "%24branch_experiment"; + RecordedRequest[] referenced = { + takeRequest("POST", scope + "/tables"), + takeRequest("POST", scope + "/tables/features"), + takeRequest("DELETE", scope + "/tables/features") + }; + CreateTableRequest plain = + RESTApi.fromJson(original[0].getBody().readUtf8(), CreateTableRequest.class); + CreateTableRequest branch = + RESTApi.fromJson(referenced[0].getBody().readUtf8(), CreateTableRequest.class); + assertThat(plain.getIdentifier()).isEqualTo(TABLE); + assertThat(branch.getIdentifier()).isEqualTo(selected); + assertThat(branch.getSchema()).isEqualTo(plain.getSchema()); + for (int i = 1; i < original.length; i++) { + assertThat(referenced[i].getBody().readUtf8()) + .isEqualTo(original[i].getBody().readUtf8()); + } + } + + @Test + void testErrorsDoNotFallBackToDefaultBranch() throws Exception { + Identifier selected = Identifier.create(DATABASE + "$tag_train_v1", "features"); + enqueue(404, "{\"message\":\"reference missing\",\"code\":404}"); + assertThatThrownBy(() -> catalog.getTable(selected)) + .isInstanceOf(Catalog.TableNotExistException.class); + takeRequest("GET", DATABASE_PATH + "%24tag_train_v1/tables/features"); + enqueue(403, "{\"message\":\"tag is immutable\",\"code\":403}"); + assertThatThrownBy( + () -> + catalog.commitSnapshot( + selected, + "table-id", + null, + Snapshot.fromJson(SNAPSHOT_JSON), + emptyList())) + .isInstanceOf(Catalog.TableNoPermissionException.class) + .hasMessageContaining("tag is immutable"); + takeRequest("POST", DATABASE_PATH + "%24tag_train_v1/tables/features/commit"); + assertThat(server.getRequestCount()).isEqualTo(3); + } + + @Test + void testReferenceErrorsAreNotIgnoredByTableDdl() throws Exception { + Identifier selected = Identifier.create(DATABASE + "$branch_experiment", "features"); + enqueue( + 409, + "{\"code\":409,\"resourceType\":\"BRANCH\",\"message\":\"branch is not writable\"}"); + assertThatThrownBy(() -> catalog.createTable(selected, schema("main"), true)) + .isInstanceOf(AlreadyExistsException.class) + .hasMessageContaining("not writable"); + takeRequest("POST", DATABASE_PATH + "%24branch_experiment/tables"); + + enqueue(409, "{\"code\":409,\"resourceType\":\"TABLE\",\"message\":\"table exists\"}"); + catalog.createTable(selected, schema("main"), true); + takeRequest("POST", DATABASE_PATH + "%24branch_experiment/tables"); + + for (boolean ignore : new boolean[] {false, true}) { + enqueue( + 404, + "{\"code\":404,\"resourceType\":\"BRANCH\",\"message\":\"branch missing\"}"); + assertThatThrownBy( + () -> + catalog.alterTable( + selected, + singletonList(SchemaChange.setOption("key", "value")), + ignore)) + .isInstanceOf(org.apache.paimon.rest.exceptions.NoSuchResourceException.class) + .hasMessageContaining("branch missing"); + takeRequest("POST", DATABASE_PATH + "%24branch_experiment/tables/features"); + } + } + + @Test + void testViewProbesAllowTableOnlyReferenceNamespaces() throws Exception { + String database = DATABASE + "$branch_experiment"; + String databasePath = DATABASE_PATH + "%24branch_experiment"; + for (int i = 0; i < 4; i++) { + enqueue(200, "{\"name\":\"training db$branch_experiment\",\"options\":{}}"); + } + assertThat(catalog.listViews(database)).isEmpty(); + assertThat(catalog.listViewsPaged(database, 10, null, null).getElements()).isEmpty(); + assertThat(catalog.listViewDetailsPaged(database, 10, null, null).getElements()).isEmpty(); + assertThatThrownBy(() -> catalog.getView(Identifier.create(database, "features"))) + .isInstanceOf(Catalog.ViewNotExistException.class); + for (int i = 0; i < 4; i++) { + takeRequest("GET", databasePath); + } + enqueue(404, "{\"code\":404,\"resourceType\":\"BRANCH\",\"message\":\"branch missing\"}"); + assertThatThrownBy(() -> catalog.listViews(database)) + .isInstanceOf(Catalog.DatabaseNotExistException.class); + takeRequest("GET", databasePath); + } + + @Test + void testUnsupportedDatabaseOperationsAndMixedSelectorsDoNotSendRequests() { + RESTApi api = catalog.api(); + String database = DATABASE + "$branch_experiment"; + Identifier selected = Identifier.create(database, "features"); + Identifier mixed = new Identifier(database, "features", "other"); + assertThatThrownBy(() -> api.getTable(mixed)).isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> api.createTable(mixed, schema("main"))) + .isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> api.listTables(DATABASE + "$tag_")) + .isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> api.renameTable(selected, TABLE)) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> api.renameTable(TABLE, selected)) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> api.createBranch(selected, "nested", null)) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> api.createDatabase(database, java.util.Collections.emptyMap())) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy( + () -> + api.alterDatabase( + database, emptyList(), java.util.Collections.emptyMap())) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> api.dropDatabase(database)) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> catalog.dropDatabase(database, true, false)) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> catalog.dropDatabase(database, true, true)) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> catalog.treeManagement().listBranches(database)) + .isInstanceOf(UnsupportedOperationException.class); + assertThat(server.getRequestCount()).isEqualTo(1); + } + + @ParameterizedTest + @ValueSource(strings = {"$branch_experiment", "$tag_train_v1"}) + void testDatabaseLookupPreservesVirtualName(String suffix) throws Exception { + String database = DATABASE + suffix; + enqueue( + 200, + "{\"name\":\"" + database + "\",\"location\":\"file:///training\",\"options\":{}}"); + assertThat(catalog.getDatabase(database).name()).isEqualTo(database); + takeRequest("GET", DATABASE_PATH + suffix.replace("$", "%24")); + enqueue(404, "{\"code\":404,\"message\":\"reference missing\"}"); + assertThatThrownBy(() -> catalog.getDatabase(database)) + .isInstanceOf(Catalog.DatabaseNotExistException.class); + takeRequest("GET", DATABASE_PATH + suffix.replace("$", "%24")); + assertThat(server.getRequestCount()).isEqualTo(3); + } + + @Test + void testBatchReadWriteAndPinnedTagWithRealDataFiles() throws Exception { + // A small stateful fixture resolves references; production reference lifecycle is separate. + org.apache.paimon.fs.Path location = + new org.apache.paimon.fs.Path(tempDir.resolve("features").toUri()); + LocalFileIO fileIO = LocalFileIO.create(); + for (String branch : new String[] {"main", "physical-experiment", "frozen-train-v1"}) { + new FileSystemSchemaManager(fileIO, location, branch).createTable(schema(branch)); + } + Map snapshots = new ConcurrentHashMap<>(); + ConcurrentLinkedQueue unexpected = new ConcurrentLinkedQueue<>(); + server.setDispatcher( + new Dispatcher() { + @Override + public MockResponse dispatch(RecordedRequest request) { + try { + String route = request.getRequestUrl().encodedPath(); + String prefix = "/v1/catalog%2Fid/databases/"; + if (!route.startsWith(prefix)) { + unexpected.add(route); + return response(500, "{}"); + } + String[] parts = route.substring(prefix.length()).split("/"); + DatabaseIdentifier database = + DatabaseIdentifier.parse(RESTUtil.decodeString(parts[0])); + if (!database.getDatabaseName().equals(DATABASE)) { + unexpected.add(route); + return response(500, "{}"); + } + String reference = + database.getReference() == null + ? "main" + : database.getReference().getName(); + if (parts.length < 3 + || !parts[1].equals("tables") + || !parts[2].equals("features")) { + unexpected.add(route); + return response(500, "{}"); + } + String branch = + reference.equals("main") + ? "main" + : reference.equals("train_v1") + ? "frozen-train-v1" + : "physical-experiment"; + if (request.getMethod().equals("GET") && parts.length == 3) { + return response( + 200, + tableResponse(RESTUtil.decodeString(parts[0]), branch, 0)); + } + if (request.getMethod().equals("GET") + && parts.length == 4 + && parts[3].equals("snapshot")) { + Snapshot snapshot = snapshots.get(reference); + return snapshot == null + ? response( + 404, + "{\"code\":404,\"resourceType\":\"SNAPSHOT\",\"message\":\"empty table\"}") + : response( + 200, + "{\"snapshot\":{\"snapshot\":" + + snapshot.toJson() + + "}}"); + } + if (request.getMethod().equals("POST") + && parts.length == 4 + && parts[3].equals("commit")) { + if (reference.equals("train_v1")) { + return response( + 403, "{\"code\":403,\"message\":\"tag is immutable\"}"); + } + CommitTableRequest commit = + RESTApi.fromJson( + request.getBody().readUtf8(), + CommitTableRequest.class); + Snapshot snapshot = commit.getSnapshot(); + fileIO.overwriteFileUtf8( + new SnapshotManager(fileIO, location, branch, null, null) + .snapshotPath(snapshot.id()), + snapshot.toJson()); + snapshots.put(reference, snapshot); + return response(200, "{\"success\":true}"); + } + unexpected.add(route); + return response(500, "{}"); + } catch (Exception e) { + unexpected.add(e.toString()); + return response(500, "{}"); + } + } + }); + + Identifier main = Identifier.create(DATABASE + "$branch_main", "features"); + Identifier experiment = Identifier.create(DATABASE + "$branch_experiment", "features"); + writeRows(main, 10); + writeRows(experiment, 20); + snapshots.put("train_v1", snapshots.get("experiment")); + // REST latest alone cannot constrain native metadata reads. Return frozen backing metadata. + fileIO.overwriteFileUtf8( + new SnapshotManager(fileIO, location, "frozen-train-v1", null, null) + .snapshotPath(snapshots.get("train_v1").id()), + snapshots.get("train_v1").toJson()); + Identifier tag = Identifier.create(DATABASE + "$tag_train_v1", "features"); + assertThat(readRows(tag)).containsExactly(20); + + writeRows(experiment, 30); + assertThat(readRows(main)).containsExactly(10); + assertThat(readRows(experiment)).containsExactlyInAnyOrder(20, 30); + // A newly loaded tag table must not follow the source branch's latest snapshot. + assertThat(readRows(tag)).containsExactly(20); + assertThatThrownBy(() -> writeRows(tag, 99)).hasStackTraceContaining("tag is immutable"); + assertThat(readRows(tag)).containsExactly(20); + assertThat(snapshots.get("train_v1").id()).isEqualTo(1); + assertThat(snapshots.get("experiment").id()).isEqualTo(2); + new FileSystemSchemaManager(fileIO, location, "physical-experiment") + .commitChanges(SchemaChange.addColumn("later", DataTypes.INT())); + FileStoreTable frozen = (FileStoreTable) catalog.getTable(tag); + assertThat(frozen.copyWithLatestSchema().schema().id()).isZero(); + assertThat(frozen.schemaManager().listAll()).hasSize(1); + assertThatThrownBy( + () -> + frozen.copy( + java.util.Collections.singletonMap( + "scan.snapshot-id", "2")) + .newReadBuilder() + .newScan() + .plan()) + .isInstanceOf(IllegalArgumentException.class); + assertThat(unexpected).isEmpty(); + } + + private void writeRows(Identifier selected, int value) throws Exception { + FileStoreTable table = InstantiationUtil.clone((FileStoreTable) catalog.getTable(selected)); + BatchWriteBuilder builder = table.newBatchWriteBuilder(); + try (BatchTableWrite write = builder.newWrite(); + BatchTableCommit commit = builder.newCommit()) { + write.write(GenericRow.of(value)); + commit.commit(write.prepareCommit()); + } + } + + private List readRows(Identifier selected) throws Exception { + FileStoreTable table = InstantiationUtil.clone((FileStoreTable) catalog.getTable(selected)); + ReadBuilder builder = table.newReadBuilder(); + List rows = new ArrayList<>(); + try (RecordReader reader = + builder.newRead().createReader(builder.newScan().plan().splits())) { + reader.forEachRemaining(row -> rows.add(row.getInt(0))); + } + return rows; + } + + private Schema schema(String branch) { + return Schema.newBuilder() + .column("id", DataTypes.INT()) + .option("bucket", "-1") + .option("commit.max-retries", "0") + .option(BRANCH.key(), branch) + .build(); + } + + private String tableResponse(String branch) throws Exception { + return tableResponse(DATABASE, branch, 2); + } + + private String tableResponse(String database, String branch, long schemaId) throws Exception { + return RESTApi.toJson( + new GetTableResponse( + "table-id", + database, + "features", + tempDir.resolve("features").toUri().toString(), + false, + schemaId, + schema(branch), + null, + 0, + null, + 0, + null)); + } + + private void enqueue(int status, String body) { + server.enqueue(response(status, body)); + } + + private MockResponse response(int status, String body) { + return new MockResponse() + .setResponseCode(status) + .setHeader("Content-Type", "application/json") + .setBody(body); + } + + private RecordedRequest takeRequest(String method, String path) throws Exception { + RecordedRequest request = server.takeRequest(10, TimeUnit.SECONDS); + assertThat(request).isNotNull(); + assertThat(request.getMethod()).isEqualTo(method); + assertThat(request.getRequestUrl().encodedPath()).isEqualTo(path); + assertThat(request.getHeader("Authorization")).isEqualTo("Bearer test-token"); + assertThat(request.getHeader("X-Catalog-Context")).isEqualTo("configured"); + assertThat(request.getHeader("Paimon-Reference")).isNull(); + return request; + } +} diff --git a/paimon-core/src/test/java/org/apache/paimon/rest/RESTCatalogTreeManagementTest.java b/paimon-core/src/test/java/org/apache/paimon/rest/RESTCatalogTreeManagementTest.java new file mode 100644 index 000000000000..9c4ab362392c --- /dev/null +++ b/paimon-core/src/test/java/org/apache/paimon/rest/RESTCatalogTreeManagementTest.java @@ -0,0 +1,226 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.rest; + +import org.apache.paimon.PagedList; +import org.apache.paimon.catalog.CatalogContext; +import org.apache.paimon.management.TreeManagement; +import org.apache.paimon.options.Options; +import org.apache.paimon.rest.exceptions.AlreadyExistsException; +import org.apache.paimon.rest.exceptions.BadRequestException; +import org.apache.paimon.rest.exceptions.NoSuchResourceException; +import org.apache.paimon.rest.exceptions.NotImplementedException; + +import okhttp3.mockwebserver.MockResponse; +import okhttp3.mockwebserver.MockWebServer; +import okhttp3.mockwebserver.RecordedRequest; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.util.Map; +import java.util.concurrent.TimeUnit; + +import static org.apache.paimon.options.CatalogOptions.WAREHOUSE; +import static org.apache.paimon.rest.RESTCatalogOptions.TOKEN; +import static org.apache.paimon.rest.RESTCatalogOptions.TOKEN_PROVIDER; +import static org.apache.paimon.rest.RESTCatalogOptions.URI; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** Exercises database tree management through a configured REST catalog and its HTTP client. */ +class RESTCatalogTreeManagementTest { + + private static final String DATABASE = "training db"; + private static final String DATABASE_PATH = "/v1/catalog%2Fid/databases/training+db"; + + private MockWebServer server; + private RESTCatalog catalog; + private TreeManagement trees; + + @BeforeEach + void setUp() throws Exception { + server = new MockWebServer(); + server.start(); + enqueue( + 200, + "{\"defaults\":{},\"overrides\":{\"prefix\":\"catalog/id\"," + + "\"header.X-Catalog-Context\":\"configured\"}}"); + + Options options = new Options(); + options.set(URI, server.url("/").toString()); + options.set(WAREHOUSE, "warehouse-id"); + options.set(TOKEN_PROVIDER, "bear"); + options.set(TOKEN, "test-token"); + catalog = new RESTCatalog(CatalogContext.create(options)); + trees = catalog.treeManagement(); + + RecordedRequest config = server.takeRequest(10, TimeUnit.SECONDS); + assertThat(config).isNotNull(); + assertThat(config.getRequestUrl().encodedPath()).isEqualTo("/v1/config"); + assertThat(config.getRequestUrl().queryParameter("warehouse")).isEqualTo("warehouse-id"); + assertThat(server.getRequestCount()).isEqualTo(1); + } + + @AfterEach + void tearDown() throws Exception { + if (catalog != null) { + catalog.close(); + } + if (server != null) { + server.shutdown(); + } + } + + @Test + void testBranchAndTagOperationsUseCatalogConfiguration() throws Exception { + enqueue(200, "{\"branches\":[\"main\",\"experiment\"]}"); + assertThat(trees.listBranches(DATABASE)).containsExactly("main", "experiment"); + takeRequest("GET", DATABASE_PATH + "/branches"); + + enqueue(200, ""); + trees.createTag(DATABASE, "baseline", null, null); + assertBody( + takeRequest("POST", DATABASE_PATH + "/tags"), + "{\"tagName\":\"baseline\",\"fromBranch\":null,\"timeRetained\":null}"); + + enqueue(200, ""); + trees.createBranch(DATABASE, "experiment", "baseline"); + assertBody( + takeRequest("POST", DATABASE_PATH + "/branches"), + "{\"branch\":\"experiment\",\"fromTag\":\"baseline\"}"); + + enqueue(200, ""); + trees.createTag(DATABASE, "train-v1", "experiment", "7d"); + assertBody( + takeRequest("POST", DATABASE_PATH + "/tags"), + "{\"tagName\":\"train-v1\",\"fromBranch\":\"experiment\",\"timeRetained\":\"7d\"}"); + + enqueue(200, "{\"tagName\":\"train-v1\",\"fromBranch\":\"experiment\"}"); + assertThat(trees.getTag(DATABASE, "train-v1").fromBranch()).isEqualTo("experiment"); + takeRequest("GET", DATABASE_PATH + "/tags/train-v1"); + + enqueue(200, ""); + trees.fastForward(DATABASE, "experiment"); + assertBody(takeRequest("POST", DATABASE_PATH + "/branches/experiment/forward"), "{}"); + + enqueue(200, ""); + trees.dropBranch(DATABASE, "experiment"); + assertThat(takeRequest("DELETE", DATABASE_PATH + "/branches/experiment").getBodySize()) + .isZero(); + + enqueue(200, ""); + trees.deleteTag(DATABASE, "train-v1"); + assertThat(takeRequest("DELETE", DATABASE_PATH + "/tags/train-v1").getBodySize()).isZero(); + assertThat(server.getRequestCount()).isEqualTo(9); + } + + @Test + void testTagPagesPreserveFilterAndTokens() throws Exception { + enqueue(200, "{\"tags\":[\"train-v1\"],\"nextPageToken\":\"next +/%?&\"}"); + PagedList page = trees.listTagsPaged(DATABASE, 10, "start +/%", "train-"); + assertThat(page.getElements()).containsExactly("train-v1"); + assertThat(page.getNextPageToken()).isEqualTo("next +/%?&"); + RecordedRequest first = takeRequest("GET", DATABASE_PATH + "/tags"); + assertThat(first.getRequestUrl().queryParameter("maxResults")).isEqualTo("10"); + assertThat(first.getRequestUrl().queryParameter("pageToken")).isEqualTo("start +/%"); + assertThat(first.getRequestUrl().queryParameter("tagNamePrefix")).isEqualTo("train-"); + + enqueue(200, "{\"tags\":[]}"); + PagedList last = + trees.listTagsPaged(DATABASE, null, page.getNextPageToken(), "train-"); + assertThat(last.getElements()).isEmpty(); + assertThat(last.getNextPageToken()).isNull(); + RecordedRequest second = takeRequest("GET", DATABASE_PATH + "/tags"); + assertThat(second.getRequestUrl().queryParameter("pageToken")).isEqualTo("next +/%?&"); + assertThat(second.getRequestUrl().queryParameter("tagNamePrefix")).isEqualTo("train-"); + assertThat(second.getRequestUrl().queryParameter("maxResults")).isNull(); + } + + @Test + void testErrorsKeepTableBranchAndTagConventions() throws Exception { + server.enqueue( + new MockResponse() + .setResponseCode(409) + .setHeader("Content-Type", "application/json") + .setHeader("x-request-id", "branch-request") + .setBody( + "{\"resourceType\":\"BRANCH\",\"resourceName\":\"experiment\",\"message\":\"branch exists\"}")); + assertThatThrownBy(() -> trees.createBranch(DATABASE, "experiment", null)) + .isExactlyInstanceOf(AlreadyExistsException.class) + .hasMessageContaining("branch exists") + .hasMessageContaining("branch-request"); + takeRequest("POST", DATABASE_PATH + "/branches"); + + enqueue( + 404, + "{\"code\":404,\"resourceType\":\"TAG\",\"resourceName\":\"baseline\",\"message\":\"tag missing\"}"); + assertThatThrownBy(() -> trees.createBranch(DATABASE, "experiment", "baseline")) + .isInstanceOfSatisfying( + NoSuchResourceException.class, + e -> { + assertThat(e.resourceType()).isEqualTo("TAG"); + assertThat(e.resourceName()).isEqualTo("baseline"); + }); + takeRequest("POST", DATABASE_PATH + "/branches"); + + enqueue(400, "{\"code\":400,\"message\":\"source table has no snapshot\"}"); + assertThatThrownBy(() -> trees.fastForward(DATABASE, "empty")) + .isInstanceOf(BadRequestException.class) + .hasMessageContaining("no snapshot"); + takeRequest("POST", DATABASE_PATH + "/branches/empty/forward"); + + enqueue(404, "{\"code\":404,\"message\":\"branch missing\"}"); + assertThatThrownBy(() -> trees.fastForward(DATABASE, "missing")) + .isInstanceOf(NoSuchResourceException.class) + .hasMessageContaining("branch missing"); + takeRequest("POST", DATABASE_PATH + "/branches/missing/forward"); + + enqueue(501, "{\"code\":501,\"message\":\"forward unsupported\"}"); + assertThatThrownBy(() -> trees.fastForward(DATABASE, "experiment")) + .isInstanceOf(NotImplementedException.class) + .hasMessageContaining("forward unsupported"); + takeRequest("POST", DATABASE_PATH + "/branches/experiment/forward"); + assertThat(server.getRequestCount()).isEqualTo(6); + } + + private void enqueue(int status, String body) { + server.enqueue( + new MockResponse() + .setResponseCode(status) + .setHeader("Content-Type", "application/json") + .setBody(body)); + } + + private RecordedRequest takeRequest(String method, String path) throws Exception { + RecordedRequest request = server.takeRequest(10, TimeUnit.SECONDS); + assertThat(request).isNotNull(); + assertThat(request.getMethod()).isEqualTo(method); + assertThat(request.getRequestUrl().encodedPath()).isEqualTo(path); + assertThat(request.getHeader("Authorization")).isEqualTo("Bearer test-token"); + assertThat(request.getHeader("X-Catalog-Context")).isEqualTo("configured"); + return request; + } + + private static void assertBody(RecordedRequest request, String expectedJson) throws Exception { + assertThat(request.getRequestUrl().query()).isNull(); + assertThat(RESTApi.fromJson(request.getBody().readUtf8(), Map.class)) + .isEqualTo(RESTApi.fromJson(expectedJson, Map.class)); + } +}