Skip to content

Commit 195a384

Browse files
committed
feat(duckdb): native MERGE improvements for catalog gating, type coercion, and composite keys
- Validate catalog types against native merge support (duckdb, ducklake, motherduck), properly checking catalog type for attached catalogs and removing unconditional MotherDuck bypass. - Maintain standard SQL = key matching semantics on composite unique keys (Option A), document null matching behavior and coalesce usage in DuckDB integration and model kinds documentation, and add test coverage. - Coerce source columns to target data types before DISTINCT ON deduplication while preserving extra columns referenced by WHEN MATCHED, merge_filter, or unique_key. - Strip table qualifiers on LHS of UPDATE SET and INSERT tuples for both __MERGE_TARGET__ and target_table.name. Signed-off-by: Vamsi-klu <nrvamsi13@gmail.com>
1 parent 84af4c2 commit 195a384

5 files changed

Lines changed: 242 additions & 16 deletions

File tree

‎docs/concepts/models/model_kinds.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -624,7 +624,7 @@ MODEL (
624624
* Snowflake
625625
* Spark
626626

627-
In DuckDB's case, the native `MERGE` statement is used on version 1.4 or later when the target table is in a DuckDB, DuckLake or MotherDuck catalog. Older DuckDB versions, and target tables in other attached catalogs (such as PostgreSQL, SQLite or MySQL), use `DELETE` + `INSERT` instead, and `when_matched` and `merge_filter` raise an error there. Source rows are de-duplicated by `unique_key` before the merge so there is one row per key. Which of several duplicate rows is kept is unspecified.
627+
In DuckDB's case, the native `MERGE` statement is used on version 1.4 or later when the target table is in a DuckDB, DuckLake or MotherDuck catalog. Older DuckDB versions, and target tables in other attached catalogs (such as PostgreSQL, SQLite or MySQL), use `DELETE` + `INSERT` instead, and `when_matched` and `merge_filter` raise an error there. Source rows are de-duplicated by `unique_key` before the merge so there is one row per key. Which of several duplicate rows is kept is unspecified. Key matching follows standard SQL `=` equality semantics (`NULL` key values do not match existing rows), consistent with other native `MERGE` engines in SQLMesh. To achieve null-safe matching on nullable composite keys, specify an expression such as `COALESCE(key_col, '<sentinel>')` in `unique_key`.
628628

629629
In Redshift's case, to enable the use of the native `MERGE` statement, you need to pass the `enable_merge` flag in the connection and set it to `true`. It is disabled by default.
630630

‎docs/integrations/engines/duckdb.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,8 @@ Example: mounting a SQLite database with the name `sqlite` that has a table `exa
200200

201201
On DuckDB version 1.4 or later, [`INCREMENTAL_BY_UNIQUE_KEY`](../../concepts/models/model_kinds.md#incremental_by_unique_key) models targeting a DuckDB, DuckLake or MotherDuck catalog use a single, atomic `MERGE` statement and support `when_matched` and `merge_filter`. Older DuckDB versions and tables in other attached catalogs (such as PostgreSQL or SQLite) use `DELETE` + `INSERT`, which runs as two statements because DuckDB adapter transactions are disabled, and do not support `when_matched` or `merge_filter`.
202202

203+
Native `MERGE` matches `unique_key` columns using standard SQL `=` equality. As a result, `NULL` key values do not match existing rows (`NULL = NULL` is unknown / evaluates to false in SQL), consistent with all other SQLMesh native `MERGE` engines and preserving dynamic join filter and zone-map pushdown in the DuckDB query optimizer. For models with nullable composite keys where nulls should be treated as matching, pass expressions such as `COALESCE(key_col, '<sentinel>')` in `unique_key`.
204+
203205
##### Sensitive fields in paths
204206

205207
If a connector, like Postgres, requires sensitive information in the path, it might support defining environment variables instead.

‎sqlmesh/core/engine_adapter/base.py‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1771,7 +1771,16 @@ def update_table(
17711771
) -> None:
17721772
self.execute(exp.update(table_name, properties, where=where))
17731773

1774-
def _prepare_merge_source_query(self, query: Query, unique_key: t.Sequence[exp.Expr]) -> Query:
1774+
def _prepare_merge_source_query(
1775+
self,
1776+
query: Query,
1777+
target_table: TableName,
1778+
target_columns_to_types: t.Optional[t.Dict[str, exp.DataType]] = None,
1779+
unique_key: t.Optional[t.Sequence[exp.Expr]] = None,
1780+
when_matched: t.Optional[exp.Whens] = None,
1781+
merge_filter: t.Optional[exp.Expr] = None,
1782+
source_columns: t.Optional[t.List[str]] = None,
1783+
) -> Query:
17751784
"""Hook that lets an engine adjust the MERGE source query. By default it is used as is."""
17761785
return query
17771786

@@ -2320,7 +2329,15 @@ def merge(
23202329
with source_query as query:
23212330
self._merge(
23222331
target_table=target_table,
2323-
query=self._prepare_merge_source_query(query, unique_key),
2332+
query=self._prepare_merge_source_query(
2333+
query,
2334+
target_table=target_table,
2335+
target_columns_to_types=target_columns_to_types,
2336+
unique_key=unique_key,
2337+
when_matched=when_matched,
2338+
merge_filter=merge_filter,
2339+
source_columns=source_columns,
2340+
),
23242341
on=on,
23252342
whens=exp.Whens(expressions=match_expressions),
23262343
)

‎sqlmesh/core/engine_adapter/duckdb.py‎

Lines changed: 92 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ class DuckDBEngineAdapter(GetCurrentCatalogFromFunctionMixin, RowDiffMixin):
4444
MIN_NATIVE_MERGE_VERSION = version.parse("1.4.0")
4545
# Catalog types whose storage extension supports MERGE INTO. Other attached catalogs
4646
# (postgres, sqlite, mysql, ...) fall back to DELETE + INSERT.
47-
NATIVE_MERGE_CATALOG_TYPES = {"duckdb", "ducklake"}
47+
NATIVE_MERGE_CATALOG_TYPES = {"duckdb", "ducklake", "motherduck"}
4848

4949
@property
5050
def catalog_support(self) -> CatalogSupport:
@@ -89,11 +89,12 @@ def _duckdb_supports_merge(self) -> bool:
8989
def _supports_native_merge(self, target_table: TableName) -> bool:
9090
if not self._duckdb_supports_merge:
9191
return False
92-
if self._is_motherduck:
93-
return True
94-
catalog = exp.to_table(target_table).catalog or self.get_current_catalog()
92+
catalog = (
93+
exp.to_table(target_table, dialect=self.dialect).catalog or self.get_current_catalog()
94+
)
9595
# Errors from the catalog lookup propagate on purpose; only a None/unknown type falls back.
96-
return self._get_catalog_type(catalog) in self.NATIVE_MERGE_CATALOG_TYPES
96+
catalog_type = (self._get_catalog_type(catalog) or "").lower()
97+
return catalog_type in self.NATIVE_MERGE_CATALOG_TYPES
9798

9899
def merge(
99100
self,
@@ -121,10 +122,83 @@ def merge(
121122
source_columns=source_columns,
122123
)
123124

124-
def _prepare_merge_source_query(self, query: Query, unique_key: t.Sequence[exp.Expr]) -> Query:
125-
# A native MERGE inserts every unmatched source row, so duplicate keys in the source would
126-
# produce duplicate keys in the target. Keep one row per key, like the DELETE + INSERT path.
127-
return exp.select("*").distinct(*unique_key).from_(query.subquery("_src")) # type: ignore
125+
def _prepare_merge_source_query(
126+
self,
127+
query: Query,
128+
target_table: TableName,
129+
target_columns_to_types: t.Optional[t.Dict[str, exp.DataType]] = None,
130+
unique_key: t.Optional[t.Sequence[exp.Expr]] = None,
131+
when_matched: t.Optional[exp.Whens] = None,
132+
merge_filter: t.Optional[exp.Expr] = None,
133+
source_columns: t.Optional[t.List[str]] = None,
134+
) -> Query:
135+
if target_columns_to_types is None:
136+
try:
137+
target_columns_to_types = self.columns(target_table)
138+
except Exception:
139+
target_columns_to_types = None
140+
141+
unique_key = unique_key or []
142+
143+
if not target_columns_to_types or all(
144+
not dtype or dtype.is_type(exp.DataType.Type.UNKNOWN)
145+
for dtype in target_columns_to_types.values()
146+
):
147+
return exp.select("*").distinct(*unique_key).from_(query.subquery("_src", copy=False)) # type: ignore
148+
149+
source_columns_set = (
150+
{s.lower() for s in source_columns} if source_columns is not None else None
151+
)
152+
target_cols_to_project = (
153+
[col for col in target_columns_to_types if col.lower() in source_columns_set]
154+
if source_columns_set is not None
155+
else list(target_columns_to_types)
156+
)
157+
projected_target_cols = {col.lower(): col for col in target_cols_to_project}
158+
159+
projections: t.List[exp.Expr] = []
160+
for col in target_cols_to_project:
161+
dtype = target_columns_to_types[col]
162+
if dtype and not dtype.is_type(exp.DataType.Type.UNKNOWN):
163+
projections.append(exp.alias_(exp.cast(exp.column(col), to=dtype), col, copy=False))
164+
else:
165+
projections.append(exp.column(col))
166+
167+
seen_extra: t.Set[str] = set()
168+
extra_cols: t.List[str] = []
169+
target_names = {
170+
MERGE_TARGET_ALIAS,
171+
exp.to_table(target_table, dialect=self.dialect).name.upper(),
172+
}
173+
for expr in [when_matched, merge_filter, *unique_key]:
174+
if expr:
175+
for column_expr in expr.find_all(exp.Column):
176+
if column_expr.table and column_expr.table.upper() in target_names:
177+
continue
178+
col_name = column_expr.name
179+
if (
180+
col_name.lower() not in projected_target_cols
181+
and col_name.lower() not in seen_extra
182+
):
183+
seen_extra.add(col_name.lower())
184+
extra_cols.append(col_name)
185+
186+
inner_query = query.this if isinstance(query, exp.Subquery) else query
187+
if isinstance(inner_query, exp.Select):
188+
for name in inner_query.named_selects:
189+
if (
190+
name != "*"
191+
and name.lower() not in projected_target_cols
192+
and name.lower() not in seen_extra
193+
):
194+
seen_extra.add(name.lower())
195+
extra_cols.append(name)
196+
197+
for extra in extra_cols:
198+
projections.append(exp.column(extra))
199+
200+
coerced = exp.select(*projections).from_(query.subquery("_raw_src", copy=False), copy=False)
201+
return exp.select("*").distinct(*unique_key).from_(coerced.subquery("_src", copy=False)) # type: ignore
128202

129203
def _merge(
130204
self,
@@ -138,6 +212,10 @@ def _merge(
138212
# TODO: sqlglot's DuckDB generator doesn't apply `merge_without_target_sql` yet. Remove this
139213
# once the pinned sqlglot version does.
140214
whens = whens.copy()
215+
target_names = {
216+
MERGE_TARGET_ALIAS,
217+
exp.to_table(target_table, dialect=self.dialect).name.upper(),
218+
}
141219
for when in whens.expressions:
142220
then = when.args.get("then")
143221
targets: t.List[exp.Expr] = []
@@ -146,7 +224,11 @@ def _merge(
146224
elif isinstance(then, exp.Insert) and isinstance(then.this, exp.Tuple):
147225
targets = then.this.expressions
148226
for column in targets:
149-
if isinstance(column, exp.Column) and column.table.upper() == MERGE_TARGET_ALIAS:
227+
if (
228+
isinstance(column, exp.Column)
229+
and column.table
230+
and column.table.upper() in target_names
231+
):
150232
column.set("table", None)
151233
super()._merge(target_table, query, on, whens)
152234

‎tests/core/engine_adapter/test_duckdb.py‎

Lines changed: 128 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -320,7 +320,9 @@ def test_merge_native_sql(make_mocked_engine_adapter: t.Callable, mocker: Mocker
320320

321321
assert to_sql_calls(adapter) == [
322322
'MERGE INTO "tgt" AS "__MERGE_TARGET__" USING '
323-
'(SELECT DISTINCT ON ("id") * FROM (SELECT "id", "name", "value" FROM "src") AS "_src") AS "__MERGE_SOURCE__" '
323+
'(SELECT DISTINCT ON ("id") * FROM '
324+
'(SELECT CAST("id" AS INT) AS "id", CAST("name" AS TEXT) AS "name", CAST("value" AS INT) AS "value" FROM '
325+
'(SELECT "id", "name", "value" FROM "src") AS "_raw_src") AS "_src") AS "__MERGE_SOURCE__" '
324326
'ON "__MERGE_TARGET__"."id" = "__MERGE_SOURCE__"."id" '
325327
'WHEN MATCHED THEN UPDATE SET "id" = "__MERGE_SOURCE__"."id", "name" = "__MERGE_SOURCE__"."name", "value" = "__MERGE_SOURCE__"."value" '
326328
'WHEN NOT MATCHED THEN INSERT ("id", "name", "value") '
@@ -542,7 +544,7 @@ def test_merge_catalog_gate_uses_target_catalog_when_qualified(
542544
assert to_sql_calls(adapter)[0].startswith('MERGE INTO "lake"."s"."tgt"')
543545

544546

545-
def test_merge_motherduck_uses_native_without_catalog_lookup(
547+
def test_merge_motherduck_catalog_uses_native(
546548
make_mocked_engine_adapter: t.Callable, mocker: MockerFixture
547549
):
548550
adapter = _mocked_adapter(
@@ -556,7 +558,45 @@ def test_merge_motherduck_uses_native_without_catalog_lookup(
556558
target_columns_to_types=MERGE_COLUMNS,
557559
unique_key=[exp.to_column("id")],
558560
)
559-
get_type.assert_not_called()
561+
get_type.assert_called_once_with("md")
562+
assert to_sql_calls(adapter)[0].startswith('MERGE INTO "md"."s"."tgt"')
563+
564+
565+
def test_merge_motherduck_foreign_catalog_falls_back(
566+
make_mocked_engine_adapter: t.Callable, mocker: MockerFixture
567+
):
568+
adapter = _mocked_adapter(
569+
make_mocked_engine_adapter, mocker, "1.4.1", "sqlite", is_motherduck=True
570+
)
571+
get_type = t.cast(MagicMock, adapter._get_catalog_type)
572+
573+
adapter.merge(
574+
target_table="sqlite_db.s.tgt",
575+
source_table=parse_one("SELECT id, name, value FROM src"),
576+
target_columns_to_types=MERGE_COLUMNS,
577+
unique_key=[exp.to_column("id")],
578+
)
579+
get_type.assert_any_call("sqlite_db")
580+
sql_calls = to_sql_calls(adapter)
581+
assert not any(call.startswith("MERGE") for call in sql_calls)
582+
assert any(call.startswith('DELETE FROM "sqlite_db"."s"."tgt"') for call in sql_calls)
583+
584+
585+
def test_merge_local_duckdb_attached_motherduck_uses_native(
586+
make_mocked_engine_adapter: t.Callable, mocker: MockerFixture
587+
):
588+
adapter = _mocked_adapter(
589+
make_mocked_engine_adapter, mocker, "1.4.1", "motherduck", is_motherduck=False
590+
)
591+
get_type = t.cast(MagicMock, adapter._get_catalog_type)
592+
593+
adapter.merge(
594+
target_table="md.s.tgt",
595+
source_table=parse_one("SELECT id, name, value FROM src"),
596+
target_columns_to_types=MERGE_COLUMNS,
597+
unique_key=[exp.to_column("id")],
598+
)
599+
get_type.assert_called_once_with("md")
560600
assert to_sql_calls(adapter)[0].startswith('MERGE INTO "md"."s"."tgt"')
561601

562602

@@ -618,3 +658,88 @@ def test_merge_native_ducklake(ducklake_conn, tmp_path, mocker: MockerFixture):
618658
rows = ducklake_conn.execute("SELECT * FROM lake.s.tgt ORDER BY id").fetchall()
619659
assert [r[0] for r in rows] == [1, 2, 3]
620660
assert rows[1] == (2, "b", 21)
661+
662+
663+
def test_merge_native_pre_distinct_coercion_decimal_to_int(native_merge_adapter, duck_conn):
664+
_set_up_merge_tables(duck_conn, "(1, 'a', 10)")
665+
# DuckDB will create src with DOUBLE for 1.1 and 1.2
666+
duck_conn.execute("DROP TABLE src")
667+
duck_conn.execute("CREATE TABLE src (id DOUBLE, name TEXT, value INT)")
668+
duck_conn.execute("INSERT INTO src VALUES (1.1, 'a2', 20), (1.2, 'a3', 30)")
669+
670+
_merge(native_merge_adapter)
671+
rows = _rows(duck_conn)
672+
# The two decimal keys coerce to 1::INT before DISTINCT ON and join, updating in place without duplicates
673+
assert len(rows) == 1
674+
assert rows[0][0] == 1
675+
assert rows[0] in {(1, "a2", 20), (1, "a3", 30)}
676+
677+
678+
def test_merge_native_extra_source_columns_preserved(native_merge_adapter, duck_conn):
679+
_set_up_merge_tables(duck_conn, "(1, 'a', 10)")
680+
duck_conn.execute("DROP TABLE src")
681+
duck_conn.execute("CREATE TABLE src (id DOUBLE, name TEXT, value INT, op VARCHAR)")
682+
duck_conn.execute("INSERT INTO src VALUES (1.0, 'a', 10, 'D'), (2.0, 'b', 20, 'U')")
683+
684+
_merge(
685+
native_merge_adapter,
686+
source_table=parse_one("SELECT id, name, value, op FROM src"),
687+
when_matched=_when_matched(
688+
"WHEN MATCHED AND __MERGE_SOURCE__.op = 'D' THEN DELETE "
689+
"WHEN MATCHED THEN UPDATE SET value = __MERGE_SOURCE__.value"
690+
),
691+
)
692+
# op is preserved in _raw_src; id=1 is deleted and id=2 is inserted
693+
assert _rows(duck_conn) == [(2, "b", 20)]
694+
695+
696+
def test_merge_native_target_table_name_stripped_from_update_set(native_merge_adapter, duck_conn):
697+
_set_up_merge_tables(duck_conn, "(1, 'a', 10)", "(1, 'a2', 20)")
698+
_merge(
699+
native_merge_adapter,
700+
when_matched=_when_matched(
701+
"WHEN MATCHED THEN UPDATE SET tgt.value = __MERGE_SOURCE__.value"
702+
),
703+
)
704+
assert _rows(duck_conn) == [(1, "a", 20)]
705+
706+
707+
def test_merge_native_composite_unique_key_option_a(native_merge_adapter, duck_conn):
708+
duck_conn.execute("DROP TABLE IF EXISTS tgt")
709+
duck_conn.execute("DROP TABLE IF EXISTS src")
710+
duck_conn.execute("CREATE TABLE tgt (id INT, name TEXT, value INT)")
711+
duck_conn.execute("CREATE TABLE src (id INT, name TEXT, value INT)")
712+
713+
duck_conn.execute("INSERT INTO tgt VALUES (1, 'a', 10), (1, NULL, 20), (NULL, 'b', 30)")
714+
duck_conn.execute("INSERT INTO src VALUES (1, NULL, 25), (NULL, 'b', 35), (2, 'c', 40)")
715+
716+
# Standard SQL = equality on composite key: NULL != NULL
717+
_merge(
718+
native_merge_adapter,
719+
unique_key=[exp.to_column("id"), exp.to_column("name")],
720+
)
721+
rows = duck_conn.execute("SELECT * FROM tgt ORDER BY id NULLS LAST, name NULLS LAST").fetchall()
722+
assert rows == [
723+
(1, "a", 10),
724+
(1, None, 20),
725+
(1, None, 25),
726+
(2, "c", 40),
727+
(None, "b", 30),
728+
(None, "b", 35),
729+
]
730+
731+
# Null-safe matching with COALESCE expression keys
732+
duck_conn.execute("DELETE FROM tgt")
733+
duck_conn.execute("DELETE FROM src")
734+
duck_conn.execute("INSERT INTO tgt VALUES (1, 'a', 10), (1, NULL, 20)")
735+
duck_conn.execute("INSERT INTO src VALUES (1, NULL, 25)")
736+
737+
_merge(
738+
native_merge_adapter,
739+
unique_key=[
740+
parse_one("COALESCE(id, -1)", dialect="duckdb"),
741+
parse_one("COALESCE(name, '')", dialect="duckdb"),
742+
],
743+
)
744+
rows = duck_conn.execute("SELECT * FROM tgt ORDER BY id NULLS LAST, name NULLS LAST").fetchall()
745+
assert rows == [(1, "a", 10), (1, None, 25)]

0 commit comments

Comments
 (0)