You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 42b4b78
Browse filesBrowse the repository at this point in the historyBrowse files
feat(duckdb): use native MERGE for incremental by unique key models
On DuckDB >= 1.4, merge into DuckDB, DuckLake and MotherDuck catalogs
with a single native MERGE statement instead of the DELETE + INSERT
logical merge. This makes the upsert atomic and enables `when_matched`
and `merge_filter` on DuckDB.
- Drop LogicalMergeMixin from DuckDBEngineAdapter and fall back to
logical_merge for older DuckDB versions and for targets in attached
non-DuckDB catalogs (postgres, sqlite, mysql, ...).
- Strip the target qualifier from UPDATE SET and INSERT column lists,
which DuckDB rejects. sqlglot's DuckDB generator lacks the
merge_without_target_sql transform that Postgres and Trino use.
- De-duplicate the source by unique key with DISTINCT ON, keeping the
existing one-row-per-key result. A no-op base hook,
_prepare_merge_source_query, lets DuckDB wrap the source.
Fixes#4985Closes#4691
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Vamsi-klu <nrvamsi13@gmail.com>
Copy file name to clipboardExpand all lines: docs/concepts/models/model_kinds.md
+4-1Lines changed: 4 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -618,11 +618,14 @@ MODEL (
618
618
619
619
* BigQuery
620
620
* Databricks
621
+
* DuckDB (version 1.4 or later, see below)
621
622
* Postgres
622
623
* Redshift
623
624
* Snowflake
624
625
* Spark
625
626
627
+
In DuckDB's case, the native `MERGE` statement is used on version 1.4 or later when the target table is in a DuckDB, DuckLake or MotherDuck catalog. Older DuckDB versions, and target tables in other attached catalogs (such as PostgreSQL, SQLite or MySQL), use `DELETE` + `INSERT` instead, and `when_matched` and `merge_filter` raise an error there. Source rows are de-duplicated by `unique_key` before the merge so there is one row per key. Which of several duplicate rows is kept is unspecified.
628
+
626
629
In Redshift's case, to enable the use of the native `MERGE` statement, you need to pass the `enable_merge` flag in the connection and set it to `true`. It is disabled by default.
627
630
628
631
```yaml linenums="1"
@@ -668,7 +671,7 @@ Depending on the target engine, models of the `INCREMENTAL_BY_UNIQUE_KEY` kind a
668
671
| BigQuery | MERGE ON unique key |
669
672
| Redshift | MERGE ON unique key |
670
673
| Postgres | MERGE ON unique key |
671
-
| DuckDB | DELETE ON matched + INSERT new rows |
674
+
| DuckDB | MERGE ON unique key (version 1.4 or later, DuckDB, DuckLake and MotherDuck catalogs); otherwise DELETE ON matched + INSERT new rows |
672
675
673
676
## FULL
674
677
Models of the `FULL` kind cause the dataset associated with a model to be fully refreshed (rewritten) upon each model evaluation.
Copy file name to clipboardExpand all lines: docs/integrations/engines/duckdb.md
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -196,6 +196,10 @@ Some connections, like SQLite, do not support schema names and therefore objects
196
196
197
197
Example: mounting a SQLite database with the name `sqlite` that has a table `example_table` will be accessible as `sqlite.main.example_table`.
198
198
199
+
##### Incremental by unique key models
200
+
201
+
On DuckDB version 1.4 or later, [`INCREMENTAL_BY_UNIQUE_KEY`](../../concepts/models/model_kinds.md#incremental_by_unique_key) models targeting a DuckDB, DuckLake or MotherDuck catalog use a single, atomic `MERGE` statement and support `when_matched` and `merge_filter`. Older DuckDB versions and tables in other attached catalogs (such as PostgreSQL or SQLite) use `DELETE` + `INSERT`, which runs as two statements because DuckDB adapter transactions are disabled, and do not support `when_matched` or `merge_filter`.
202
+
199
203
##### Sensitive fields in paths
200
204
201
205
If a connector, like Postgres, requires sensitive information in the path, it might support defining environment variables instead.
0 commit comments