Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/snippets/tables.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,8 @@ export const PyUpdateConnectLocal = "import lancedb\n\ndb = lancedb.connect(\"./

export const PyUpdateExampleTableSetup = "import pyarrow as pa\n\ntable = db.create_table(\n \"users_example\",\n data=pa.table(\n {\n \"id\": [1, 2],\n \"name\": [\"Alice\", \"Bob\"],\n \"login_count\": [10, 20],\n }\n ),\n mode=\"overwrite\",\n)\n";

export const PyUpdateExprFilter = "import pyarrow as pa\n\nfrom lancedb.expr import col\n\ntable = db.create_table(\n \"users_example\",\n data=pa.table(\n {\n \"id\": [1, 2],\n \"name\": [\"Alice\", \"O'Brien\"],\n \"login_count\": [10, 20],\n }\n ),\n mode=\"overwrite\",\n)\n# Plain Python values are encoded as SQL literals automatically,\n# so names with apostrophes or numeric-looking text are safe.\ntable.update(where=col(\"name\") == \"O'Brien\", values={\"login_count\": 30})\n";

export const PyUpdateOperation = "import pyarrow as pa\n\ntable = db.create_table(\n \"users_example\",\n data=pa.table(\n {\n \"id\": [1, 2],\n \"name\": [\"Alice\", \"Bob\"],\n \"login_count\": [10, 20],\n }\n ),\n mode=\"overwrite\",\n)\ntable.update(where=\"id = 2\", values={\"name\": \"Bobby\"})\n";

export const PyUpdateOptimizeCleanup = "from datetime import timedelta\n\ntable.optimize(cleanup_older_than=timedelta(days=1))\n";
Expand Down
24 changes: 22 additions & 2 deletions docs/tables/update.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
PyUpdateExampleTableSetup as UpdateExampleTableSetup,
PyUpdateOperation as UpdateOperation,
PyUpdateUsingSql as UpdateUsingSql,
PyUpdateExprFilter as UpdateExprFilter,
PyMergeMatchedUpdateOnly as MergeMatchedUpdateOnly,
PyInsertIfNotExists as InsertIfNotExists,
PyMergeUpdateInsert as MergeUpdateInsert,
Expand Down Expand Up @@ -47,7 +48,7 @@ import {
Updating or modifying data involves changing rows in an existing table.
LanceDB provides two families of write operations that can modify data in a table:

- `update(...)`: mutate existing rows that match a SQL filter.
- `update(...)`: mutate existing rows that match a filter. The filter is a SQL string, or a type-safe expression in Python.
- `merge_insert(...)`: compare incoming rows to existing rows by key, then choose what to do for each case.

The `update` method is simpler to use when you already know which rows you want to modify and you do not need to compare against an incoming dataset. The `merge_insert` method is more powerful when you have a new dataset that you want to merge into an existing table, and you want LanceDB to handle the logic of comparing against existing rows by key.
Expand Down Expand Up @@ -127,7 +128,7 @@ table creation patterns (Pandas, Polars, Pydantic, iterators, etc.) -- see the [

| Family | Method | Use this when... |
| --------------- | --------------- | ---------------- |
| `update` | `update(where=..., values=...)` | You want to edit rows that already exist, using a SQL filter. |
| `update` | `update(where=..., values=...)` | You want to edit rows that already exist, using a filter. |
| `merge_insert` | `.when_matched_update_all()` | You have incoming rows and want to update keys that already exist in the table. |
| `merge_insert` | `.when_not_matched_insert_all()` | You have incoming rows and want to insert keys that do not exist yet. |
| `merge_insert` | `.when_matched_update_all()` + `.when_not_matched_insert_all()` | You want both behaviors together (often called **upsert**: update existing keys **and** insert missing keys in the same operation). |
Expand Down Expand Up @@ -199,6 +200,25 @@ Expected table contents:
See the [SQL queries](/search/sql/) page for more information on the supported SQL syntax.
</Note>

## Update rows with a type-safe filter

In Python, the `where` argument of `update` also accepts a type-safe expression instead of a raw SQL string. Build the filter with `col` and `lit` from `lancedb.expr`, then combine expressions with Python operators. LanceDB encodes plain Python values as SQL literals for you. Text containing apostrophes or numeric-looking strings is always treated as text, so you do not need to escape interpolated values yourself.

<CodeGroup>
<CodeBlock filename="Python" language="Python" icon="python">
{UpdateExprFilter}
</CodeBlock>
</CodeGroup>

Expected table contents:

| id | name | login_count |
| --- | --- | --- |
| 1 | Alice | 10 |
| 2 | O'Brien | 30 |

Expression filters work on local, async, and remote tables. The `delete` method and query `where` clauses accept the same expressions.

When rows are updated, they are moved out of any existing index. The row will still show up in search queries, but the query will not be as fast as it would be if the row was in the index. If you update a large proportion of rows, consider triggering an index rebuild afterwards.

## Merge incoming rows by key
Expand Down
30 changes: 30 additions & 0 deletions tests/py/test_tables.py
Original file line number Diff line number Diff line change
Expand Up @@ -615,6 +615,36 @@ def test_update_using_sql(tmp_db):
]


def test_update_expr_filter(tmp_db):
db = tmp_db

# --8<-- [start:update_expr_filter]
import pyarrow as pa

from lancedb.expr import col

table = db.create_table(
"users_example",
data=pa.table(
{
"id": [1, 2],
"name": ["Alice", "O'Brien"],
"login_count": [10, 20],
}
),
mode="overwrite",
)
# Plain Python values are encoded as SQL literals automatically,
# so names with apostrophes or numeric-looking text are safe.
table.update(where=col("name") == "O'Brien", values={"login_count": 30})
# --8<-- [end:update_expr_filter]
rows = table.to_arrow().sort_by("id").to_pylist()
assert rows == [
{"id": 1, "name": "Alice", "login_count": 10},
{"id": 2, "name": "O'Brien", "login_count": 30},
]


def test_merge_matched_update_only(tmp_db):
db = tmp_db

Expand Down
Loading