Skip to content

Deploing synced tables from Lakeflow Pipelines #5765

Description

@MeinAccount

When creating a synced table using DABs, the source table needs to exist. This causes ordering problems: One will often deploy a job that creates the source table per DAB, like using a Job and/or Lakeflow Declarative Pipeline.

Then the synced table cannot be declared in the same DAB, as the initial deployment results in TABLE_DOES_NOT_EXIST. This is similar to #4437 . I suggest to allow the creation of synced tables for non-existent tables. Only validate the metadata on (first) refresh.

In my particular use-case a materialized view from a Lakeflow Declarative Pipeline needs to be synced. This would also be solved by extending pipelines by adding synced tables as a new asset. Note that this doesn't solve all use-cases.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions