Skip to content

Commit 42b4b78

Browse files
Vamsi-kluclaude
andcommitted
feat(duckdb): use native MERGE for incremental by unique key models
On DuckDB >= 1.4, merge into DuckDB, DuckLake and MotherDuck catalogs with a single native MERGE statement instead of the DELETE + INSERT logical merge. This makes the upsert atomic and enables `when_matched` and `merge_filter` on DuckDB. - Drop LogicalMergeMixin from DuckDBEngineAdapter and fall back to logical_merge for older DuckDB versions and for targets in attached non-DuckDB catalogs (postgres, sqlite, mysql, ...). - Strip the target qualifier from UPDATE SET and INSERT column lists, which DuckDB rejects. sqlglot's DuckDB generator lacks the merge_without_target_sql transform that Postgres and Trino use. - De-duplicate the source by unique key with DISTINCT ON, keeping the existing one-row-per-key result. A no-op base hook, _prepare_merge_source_query, lets DuckDB wrap the source. Fixes #4985 Closes #4691 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Vamsi-klu <nrvamsi13@gmail.com>
1 parent 6bf4fb0 commit 42b4b78

6 files changed

Lines changed: 487 additions & 6 deletions

File tree

‎docs/concepts/models/model_kinds.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -618,11 +618,14 @@ MODEL (
618618

619619
* BigQuery
620620
* Databricks
621+
* DuckDB (version 1.4 or later, see below)
621622
* Postgres
622623
* Redshift
623624
* Snowflake
624625
* Spark
625626

627+
In DuckDB's case, the native `MERGE` statement is used on version 1.4 or later when the target table is in a DuckDB, DuckLake or MotherDuck catalog. Older DuckDB versions, and target tables in other attached catalogs (such as PostgreSQL, SQLite or MySQL), use `DELETE` + `INSERT` instead, and `when_matched` and `merge_filter` raise an error there. Source rows are de-duplicated by `unique_key` before the merge so there is one row per key. Which of several duplicate rows is kept is unspecified.
628+
626629
In Redshift's case, to enable the use of the native `MERGE` statement, you need to pass the `enable_merge` flag in the connection and set it to `true`. It is disabled by default.
627630

628631
```yaml linenums="1"
@@ -668,7 +671,7 @@ Depending on the target engine, models of the `INCREMENTAL_BY_UNIQUE_KEY` kind a
668671
| BigQuery | MERGE ON unique key |
669672
| Redshift | MERGE ON unique key |
670673
| Postgres | MERGE ON unique key |
671-
| DuckDB | DELETE ON matched + INSERT new rows |
674+
| DuckDB | MERGE ON unique key (version 1.4 or later, DuckDB, DuckLake and MotherDuck catalogs); otherwise DELETE ON matched + INSERT new rows |
672675

673676
## FULL
674677
Models of the `FULL` kind cause the dataset associated with a model to be fully refreshed (rewritten) upon each model evaluation.

‎docs/integrations/engines/duckdb.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,10 @@ Some connections, like SQLite, do not support schema names and therefore objects
196196

197197
Example: mounting a SQLite database with the name `sqlite` that has a table `example_table` will be accessible as `sqlite.main.example_table`.
198198

199+
##### Incremental by unique key models
200+
201+
On DuckDB version 1.4 or later, [`INCREMENTAL_BY_UNIQUE_KEY`](../../concepts/models/model_kinds.md#incremental_by_unique_key) models targeting a DuckDB, DuckLake or MotherDuck catalog use a single, atomic `MERGE` statement and support `when_matched` and `merge_filter`. Older DuckDB versions and tables in other attached catalogs (such as PostgreSQL or SQLite) use `DELETE` + `INSERT`, which runs as two statements because DuckDB adapter transactions are disabled, and do not support `when_matched` or `merge_filter`.
202+
199203
##### Sensitive fields in paths
200204

201205
If a connector, like Postgres, requires sensitive information in the path, it might support defining environment variables instead.

‎sqlmesh/core/engine_adapter/base.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1771,6 +1771,10 @@ def update_table(
17711771
) -> None:
17721772
self.execute(exp.update(table_name, properties, where=where))
17731773

1774+
def _prepare_merge_source_query(self, query: Query, unique_key: t.Sequence[exp.Expr]) -> Query:
1775+
"""Hook that lets an engine adjust the MERGE source query. By default it is used as is."""
1776+
return query
1777+
17741778
def _merge(
17751779
self,
17761780
target_table: TableName,
@@ -2316,7 +2320,7 @@ def merge(
23162320
with source_query as query:
23172321
self._merge(
23182322
target_table=target_table,
2319-
query=query,
2323+
query=self._prepare_merge_source_query(query, unique_key),
23202324
on=on,
23212325
whens=exp.Whens(expressions=match_expressions),
23222326
)

‎sqlmesh/core/engine_adapter/duckdb.py‎

Lines changed: 81 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,16 @@
11
from __future__ import annotations
22

33
import typing as t
4+
from functools import cached_property, partial
45
from sqlglot import exp
56
from pathlib import Path
7+
from packaging import version
68

9+
from sqlmesh.core.engine_adapter.base import MERGE_TARGET_ALIAS
710
from sqlmesh.core.engine_adapter.mixins import (
811
GetCurrentCatalogFromFunctionMixin,
9-
LogicalMergeMixin,
1012
RowDiffMixin,
13+
logical_merge,
1114
)
1215
from sqlmesh.core.engine_adapter.shared import (
1316
CatalogSupport,
@@ -21,11 +24,11 @@
2124

2225
if t.TYPE_CHECKING:
2326
from sqlmesh.core._typing import SchemaName, TableName
24-
from sqlmesh.core.engine_adapter._typing import DF
27+
from sqlmesh.core.engine_adapter._typing import DF, Query, QueryOrDF
2528

2629

2730
@set_catalog(override_mapping={"_get_data_objects": CatalogSupport.REQUIRES_SET_CATALOG})
28-
class DuckDBEngineAdapter(LogicalMergeMixin, GetCurrentCatalogFromFunctionMixin, RowDiffMixin):
31+
class DuckDBEngineAdapter(GetCurrentCatalogFromFunctionMixin, RowDiffMixin):
2932
DIALECT = "duckdb"
3033
SUPPORTS_TRANSACTIONS = False
3134
SCHEMA_DIFFER_KWARGS = {
@@ -37,6 +40,11 @@ class DuckDBEngineAdapter(LogicalMergeMixin, GetCurrentCatalogFromFunctionMixin,
3740
COMMENT_CREATION_VIEW = CommentCreationView.COMMENT_COMMAND_ONLY
3841
SUPPORTS_CREATE_DROP_CATALOG = True
3942
SUPPORTED_DROP_CASCADE_OBJECT_KINDS = ["SCHEMA", "TABLE", "VIEW"]
43+
# MERGE INTO was added in DuckDB 1.4.0
44+
MIN_NATIVE_MERGE_VERSION = version.parse("1.4.0")
45+
# Catalog types whose storage extension supports MERGE INTO. Other attached catalogs
46+
# (postgres, sqlite, mysql, ...) fall back to DELETE + INSERT.
47+
NATIVE_MERGE_CATALOG_TYPES = {"duckdb", "ducklake"}
4048

4149
@property
4250
def catalog_support(self) -> CatalogSupport:
@@ -72,6 +80,76 @@ def _drop_catalog(self, catalog_name: exp.Identifier) -> None:
7280
)
7381
)
7482

83+
@cached_property
84+
def _duckdb_supports_merge(self) -> bool:
85+
import duckdb
86+
87+
return version.parse(duckdb.__version__) >= self.MIN_NATIVE_MERGE_VERSION
88+
89+
def _supports_native_merge(self, target_table: TableName) -> bool:
90+
if not self._duckdb_supports_merge:
91+
return False
92+
if self._is_motherduck:
93+
return True
94+
catalog = exp.to_table(target_table).catalog or self.get_current_catalog()
95+
# Errors from the catalog lookup propagate on purpose; only a None/unknown type falls back.
96+
return self._get_catalog_type(catalog) in self.NATIVE_MERGE_CATALOG_TYPES
97+
98+
def merge(
99+
self,
100+
target_table: TableName,
101+
source_table: QueryOrDF,
102+
target_columns_to_types: t.Optional[t.Dict[str, exp.DataType]],
103+
unique_key: t.Sequence[exp.Expr],
104+
when_matched: t.Optional[exp.Whens] = None,
105+
merge_filter: t.Optional[exp.Expr] = None,
106+
source_columns: t.Optional[t.List[str]] = None,
107+
**kwargs: t.Any,
108+
) -> None:
109+
merge_impl = (
110+
super().merge
111+
if self._supports_native_merge(target_table)
112+
else partial(logical_merge, self)
113+
)
114+
merge_impl( # type: ignore
115+
target_table,
116+
source_table,
117+
target_columns_to_types,
118+
unique_key,
119+
when_matched=when_matched,
120+
merge_filter=merge_filter,
121+
source_columns=source_columns,
122+
)
123+
124+
def _prepare_merge_source_query(self, query: Query, unique_key: t.Sequence[exp.Expr]) -> Query:
125+
# A native MERGE inserts every unmatched source row, so duplicate keys in the source would
126+
# produce duplicate keys in the target. Keep one row per key, like the DELETE + INSERT path.
127+
return exp.select("*").distinct(*unique_key).from_(query.subquery("_src")) # type: ignore
128+
129+
def _merge(
130+
self,
131+
target_table: TableName,
132+
query: Query,
133+
on: exp.Expr,
134+
whens: exp.Whens,
135+
) -> None:
136+
# DuckDB rejects a qualified column on the left-hand side of UPDATE SET and in the INSERT
137+
# column list. Conditions and right-hand sides may keep the qualifier.
138+
# TODO: sqlglot's DuckDB generator doesn't apply `merge_without_target_sql` yet. Remove this
139+
# once the pinned sqlglot version does.
140+
whens = whens.copy()
141+
for when in whens.expressions:
142+
then = when.args.get("then")
143+
targets: t.List[exp.Expr] = []
144+
if isinstance(then, exp.Update):
145+
targets = [e.this for e in then.expressions if isinstance(e, exp.EQ)]
146+
elif isinstance(then, exp.Insert) and isinstance(then.this, exp.Tuple):
147+
targets = then.this.expressions
148+
for column in targets:
149+
if isinstance(column, exp.Column) and column.table.upper() == MERGE_TARGET_ALIAS:
150+
column.set("table", None)
151+
super()._merge(target_table, query, on, whens)
152+
75153
def _df_to_source_queries(
76154
self,
77155
df: DF,

‎tests/core/engine_adapter/integration/test_integration.py‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
from sqlmesh.core.environment import EnvironmentSuffixTarget
3232
from sqlmesh.core.dialect import select_from_values
3333
from sqlmesh.core.model import Model, load_sql_based_model
34+
from sqlmesh.core.engine_adapter.duckdb import DuckDBEngineAdapter
3435
from sqlmesh.core.engine_adapter.shared import DataObject, DataObjectType
3536
from sqlmesh.core.engine_adapter.mixins import RowDiffMixin, LogicalMergeMixin
3637
from sqlmesh.core.model.definition import create_sql_model
@@ -2857,11 +2858,16 @@ def test_incremental_by_unique_key_model_when_matched(ctx: TestContext):
28572858
if not ctx.supports_merge:
28582859
pytest.skip(f"{ctx.dialect} on {ctx.gateway} doesnt support merge")
28592860

2860-
# DuckDB and some other engines use logical_merge which doesn't support when_matched
2861+
# Engines that use logical_merge (e.g. MySQL) don't support when_matched
28612862
if isinstance(ctx.engine_adapter, LogicalMergeMixin):
28622863
pytest.skip(
28632864
f"{ctx.dialect} on {ctx.gateway} uses logical merge which doesn't support when_matched"
28642865
)
2866+
# DuckDB only merges natively from 1.4.0; older versions fall back to logical_merge
2867+
if isinstance(ctx.engine_adapter, DuckDBEngineAdapter) and not (
2868+
ctx.engine_adapter._duckdb_supports_merge
2869+
):
2870+
pytest.skip("The installed duckdb doesn't support MERGE (requires >= 1.4.0)")
28652871

28662872
def _mutate_config(current_gateway_name: str, config: Config):
28672873
connection = config.gateways[current_gateway_name].connection

0 commit comments

Comments
 (0)