You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 195a384
Browse filesBrowse the repository at this point in the historyBrowse files
feat(duckdb): native MERGE improvements for catalog gating, type coercion, and composite keys
- Validate catalog types against native merge support (duckdb, ducklake, motherduck), properly checking catalog type for attached catalogs and removing unconditional MotherDuck bypass.
- Maintain standard SQL = key matching semantics on composite unique keys (Option A), document null matching behavior and coalesce usage in DuckDB integration and model kinds documentation, and add test coverage.
- Coerce source columns to target data types before DISTINCT ON deduplication while preserving extra columns referenced by WHEN MATCHED, merge_filter, or unique_key.
- Strip table qualifiers on LHS of UPDATE SET and INSERT tuples for both __MERGE_TARGET__ and target_table.name.
Signed-off-by: Vamsi-klu <nrvamsi13@gmail.com>
Copy file name to clipboardExpand all lines: docs/concepts/models/model_kinds.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -624,7 +624,7 @@ MODEL (
624
624
* Snowflake
625
625
* Spark
626
626
627
-
In DuckDB's case, the native `MERGE` statement is used on version 1.4 or later when the target table is in a DuckDB, DuckLake or MotherDuck catalog. Older DuckDB versions, and target tables in other attached catalogs (such as PostgreSQL, SQLite or MySQL), use `DELETE` + `INSERT` instead, and `when_matched` and `merge_filter` raise an error there. Source rows are de-duplicated by `unique_key` before the merge so there is one row per key. Which of several duplicate rows is kept is unspecified.
627
+
In DuckDB's case, the native `MERGE` statement is used on version 1.4 or later when the target table is in a DuckDB, DuckLake or MotherDuck catalog. Older DuckDB versions, and target tables in other attached catalogs (such as PostgreSQL, SQLite or MySQL), use `DELETE` + `INSERT` instead, and `when_matched` and `merge_filter` raise an error there. Source rows are de-duplicated by `unique_key` before the merge so there is one row per key. Which of several duplicate rows is kept is unspecified. Key matching follows standard SQL `=` equality semantics (`NULL` key values do not match existing rows), consistent with other native `MERGE` engines in SQLMesh. To achieve null-safe matching on nullable composite keys, specify an expression such as `COALESCE(key_col, '<sentinel>')` in `unique_key`.
628
628
629
629
In Redshift's case, to enable the use of the native `MERGE` statement, you need to pass the `enable_merge` flag in the connection and set it to `true`. It is disabled by default.
Copy file name to clipboardExpand all lines: docs/integrations/engines/duckdb.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -200,6 +200,8 @@ Example: mounting a SQLite database with the name `sqlite` that has a table `exa
200
200
201
201
On DuckDB version 1.4 or later, [`INCREMENTAL_BY_UNIQUE_KEY`](../../concepts/models/model_kinds.md#incremental_by_unique_key) models targeting a DuckDB, DuckLake or MotherDuck catalog use a single, atomic `MERGE` statement and support `when_matched` and `merge_filter`. Older DuckDB versions and tables in other attached catalogs (such as PostgreSQL or SQLite) use `DELETE` + `INSERT`, which runs as two statements because DuckDB adapter transactions are disabled, and do not support `when_matched` or `merge_filter`.
202
202
203
+
Native `MERGE` matches `unique_key` columns using standard SQL `=` equality. As a result, `NULL` key values do not match existing rows (`NULL = NULL` is unknown / evaluates to false in SQL), consistent with all other SQLMesh native `MERGE` engines and preserving dynamic join filter and zone-map pushdown in the DuckDB query optimizer. For models with nullable composite keys where nulls should be treated as matching, pass expressions such as `COALESCE(key_col, '<sentinel>')` in `unique_key`.
204
+
203
205
##### Sensitive fields in paths
204
206
205
207
If a connector, like Postgres, requires sensitive information in the path, it might support defining environment variables instead.
0 commit comments