Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions docs/embedding/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,11 @@ to `OpenAIEmbeddingFunction::new_with_model` in Rust.
| API key | `api_key="..."`, environment variables, or `$var:` | `apiKey: "..."`, environment variables, or `$var:` | Constructor argument or environment variable |
| Device | Provider-specific, for example `device="cuda"` | Provider-specific | Provider-specific |

When ingesting data with an embedding definition, LanceDB only computes the vector column if that
column is missing from the incoming batch or present but entirely null. If you provide any non-null
values in the vector column, LanceDB treats the column as user-supplied and does not backfill the
remaining rows in that batch.

For reusable runtime configuration, the registry also supports `$var:` placeholders in embedding-function config.
This is useful for provider secrets and environment-specific settings in Python and TypeScript.

Expand Down Expand Up @@ -127,6 +132,10 @@ maps one source column to one vector column, and the table schema can contain mu
The exact setup differs by SDK, but the underlying pattern is the same: define a distinct source/vector pair
for each embedding function you want applied during ingest.

In TypeScript, automatic query embedding currently uses the first embedding function stored in the
table metadata. If a table has multiple embedding definitions and you need to query a specific vector
column, compute the query embedding explicitly and pass the vector to the search builder.

## Embedding model providers

LanceDB supports most popular embedding providers.
Expand Down Expand Up @@ -187,6 +196,11 @@ For the Python remote client, `table.search("hello")` can take two different pat
If you want explicit vector or hybrid behavior and the client cannot resolve an embedding function from the
table metadata, generate the query embedding yourself and pass the vector directly.

TypeScript has a similar string-query distinction in `auto` mode: if no embedding providers have been
registered in the process, `search("text")` falls back to FTS. Once an embedding provider has been
imported and registered, the client expects table embedding metadata for automatic vector search and
raises an error if the table has none.

The manual query-embedding flow below works across Enterprise SDKs and is an explicit path you can use when you
want full control over query-time behavior.

Expand Down
9 changes: 9 additions & 0 deletions docs/embedding/quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ from lancedb.embeddings import get_registry
- `Vector`: Field type for storing vector embeddings
- `get_registry()`: Access to the embedding function registry. It has all the supported as well as custom embedding functions registered by the user
- TypeScript uses `lancedb.embedding.getRegistry()` and `lancedb.embedding.LanceSchema()` for the same registry/schema workflow
- In TypeScript, import the provider module before calling `getRegistry().get(...)`; the provider import is what registers names such as `"huggingface"` or `"openai"`

## Step 2: Connect to LanceDB

Expand Down Expand Up @@ -119,6 +120,10 @@ The `table.add()` call automatically:
- Generates embeddings using your chosen model
- Stores both the original text and the vector embeddings

If your input already includes the vector column, automatic embedding only runs when that column is
absent or entirely null for the batch. Partially supplied vectors are treated as manual data, so
LanceDB preserves them instead of filling only the missing rows.

## Step 6: Query with Automatic Embedding

Note: On LanceDB Enterprise, the server does not generate embeddings from query text. In the Python remote
Expand All @@ -144,3 +149,7 @@ The search process:
1. Automatically converts your query text to embeddings
2. Finds the most similar vectors in your table
3. Returns the matching documents

Automatic text search depends on the table's embedding metadata. If the client cannot reconstruct an
embedding function from that metadata, compute the query embedding yourself and search with the
vector directly.
3 changes: 3 additions & 0 deletions docs/indexing/fts-index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,9 @@ When using async connections (`connect_async`), use `create_index` with the `FTS
The `create_fts_index` method is not available on `AsyncTable`. Use `create_index` with `FTS` config instead.
</Note>

The current FTS implementation is Lance-native. Legacy Tantivy-only options, including
`use_tantivy`, are no longer accepted by the index creation APIs.

## Nested field paths

FTS indexes can target text leaves inside struct columns by passing a dotted path (for example, `payload.text`). The same path works for [`MatchQuery`](/search/full-text-search) and [`PhraseQuery`](/search/full-text-search), and for the `columns` argument on async `nearest_to_text` queries.
Expand Down
4 changes: 4 additions & 0 deletions docs/indexing/gpu-indexing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ into a synchronous process by waiting until the index is built.

`wait_for_index(...)` waits for the index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`. It can time out if the table is receiving continuous writes while the build is trying to catch up.

GPU acceleration changes how the vector index is built, not the lifecycle after the build. Rows
appended later still need `optimize()` before they are part of the index, and `fast_search()` only
searches indexed rows.

## Manual GPU indexing in LanceDB OSS

You can use the Python SDK to manually create the `IVF_PQ` index on a GPU. You'll need
Expand Down
9 changes: 7 additions & 2 deletions docs/indexing/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,10 @@ LanceDB provides a comprehensive suite of indexing strategies for different data
| `IVF` (Vector) | Large-scale vector search with configurable accuracy/speed trade-offs. Supports binary vectors with hamming distance. | Inverted File Index—a partition-based approximate nearest neighbor algorithm that groups similar vectors into partitions for efficient search.<br />Distance metrics: `l2` `cosine` `dot` `hamming`<br />Quantizations: `None/Flat` `PQ` `SQ` `RQ`|
| `IVF_HNSW` (Vector) | Large-scale vector search requiring both high recall and efficient partitioning. Combines the scalability of IVF with the search quality of HNSW. | Hybrid index combining IVF partitioning with HNSW graphs built within each partition. Provides improved search quality over pure IVF while maintaining scalability.<br />Distance metrics: `l2` `cosine` `dot`<br />Quantizations: `None/Flat` `SQ` `PQ`|
| `FTS` (Full-text search) | String columns (e.g., title, description, content) requiring keyword-based search with BM25 ranking. | Full-text search index using BM25 ranking algorithm. Tokenizes text with configurable tokenization, stemming, stop word removal, and language-specific processing. |
| `BTree` (Scalar) | Numeric, temporal, and string columns with mostly distinct values. Best for highly selective queries on columns with many unique values. | Sorted index storing sorted copies of scalar columns with block headers in a btree cache. Header entries map to blocks of rows (4096 rows per block) for efficient disk reads. |
| `BTree` (Scalar) | Numeric, temporal, and string columns with mostly distinct values. Best for selective equality, inequality, and range predicates. | Sorted index storing sorted copies of scalar columns with block headers in a btree cache. Header entries map to blocks of rows (4096 rows per block) for efficient disk reads. |
| `Bitmap` (Scalar) | Low-cardinality columns with few thousand or fewer distinct values. Accelerates equality and range filters. | Stores a bitmap for each distinct value in the column, with one bit per row indicating presence. Memory-efficient for low-cardinality data. |
| `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring array containment queries. | Scalar index for `List<T>` and `LargeList<T>` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. |
| `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring `array_contains_all` or `array_contains_any` filters. | Scalar index for `List<T>` and `LargeList<T>` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. |
| `FM` (Scalar) | String or binary columns that need raw substring search. | FM-Index over `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary` data for filters such as `contains(path, 'needle')`. Use FTS instead for tokenized word search and BM25 ranking. |

<Note>
TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization).
Expand All @@ -43,6 +44,10 @@ TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization).
**Operational checks**

For vector indexes, use the same distance metric when creating the index and searching it. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` if you need to confirm the index has caught up. `wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`; it can time out if writes keep adding unindexed rows.

By default, automatic vector indexing creates `IVF_PQ`, and scalar index creation defaults to
`BTree` unless you pass another scalar index config. `BTree` and `Bitmap` indexes target scalar
columns, not list columns; use `LabelList` for list containment filters.
</Info>

### Quantization Types
Expand Down
4 changes: 2 additions & 2 deletions docs/indexing/quantization.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -68,12 +68,12 @@ The full list of parameters to the algorithm are listed below.
- `distance_type`: Literal["l2", "cosine", "dot"], defaults to "l2"
The distance metric to use for similarity comparison. Choose "l2" for Euclidean, "cosine" for cosine similarity, or "dot" for dot product.
- `num_partitions`: Optional[int], defaults to None
Number of IVF partitions (affects index build time and query accuracy). More partitions can improve recall but may increase build time.
Number of IVF partitions (affects index build time and query accuracy). More partitions can improve recall but may increase build time. When unset, LanceDB chooses roughly the square root of the row count.
- `num_bits`: int, defaults to 1
Bits per dimension for quantization (1 is standard RaBitQ). Higher values improve fidelity, mainly at the cost of additional storage.
- `max_iterations`: int, defaults to 50
Maximum number of iterations for training the quantizer. Increase for larger datasets or to improve quantization quality.
- `sample_rate`: int, defaults to 256
Number of samples per partition during training. Higher values may improve accuracy but increase training time.
- `target_partition_size`: Optional[int], defaults to None
Target number of vectors per partition. Adjust to control partition granularity and memory usage.
Target number of vectors per partition. Adjust to control partition granularity and memory usage. If `num_partitions` is also set, `num_partitions` takes precedence.
7 changes: 7 additions & 0 deletions docs/indexing/vector-index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,10 @@ Although the `create_index` API returns immediately, the building of the vector

Use the same distance metric for index creation and search. Once a vector index exists, queries use the metric stored with that index. If you need to confirm an async build or refresh is finished, `wait_for_index(...)` waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`; it can time out if new writes keep arriving.

Rows appended after an index build remain outside that index until optimization refreshes it. Normal
search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that
fallback and searches only indexed rows.

## Choose the Right Index

Use this table as a quick starting point for choosing the right index type and quantization method for your use case:
Expand Down Expand Up @@ -109,6 +113,9 @@ Sometimes you need to configure the index beyond default parameters:
- `num_partitions`: use index-specific starting points from the section above:
- HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`): `num_rows // 1,048,576`
- `IVF_RQ` and `IVF_PQ`: `num_rows // 4096`
- `target_partition_size`: alternative IVF sizing knob that asks LanceDB to derive the partition
count from a target number of rows per partition. If you set both `num_partitions` and
`target_partition_size`, `num_partitions` takes precedence.
- `num_sub_vectors`: applies to `IVF_PQ`; start with `dimension // 8`. Larger values often improve recall but can slow search.

Let's take a look at a sample request for an IVF index:
Expand Down
4 changes: 4 additions & 0 deletions docs/namespaces/usage.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,10 @@ doesn't exist, or contains data:
Namespace path components can't be empty. Each component can contain only letters, numbers,
underscores, hyphens, and periods.

Listing APIs return the immediate children of the requested namespace path. Use `limit` with the
returned `page_token` to page through large catalogs; pass an empty namespace path (`[]`) when you
want to list from the root namespace.

## Namespaces in LanceDB Enterprise

In LanceDB Enterprise deployments, configure namespace-backed federated databases in a TOML file under your deployment's `config` directory.
Expand Down
6 changes: 5 additions & 1 deletion docs/reranking/cross_encoder.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,11 @@ Accepted Arguments
| `model_name` | `str` | `"cross-encoder/ms-marco-TinyBERT-L-6"` | The name of the reranker model to use.|
| `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. |
| `device` | `str` | `None` | The device to use for the cross encoder model. If None, will use "cuda" if available, otherwise "cpu". |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. |
| `trust_remote_code` | `bool` | `True` | Passed to Sentence Transformers model loading. Set this to `False` when you only want to load models that do not require custom repository code. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", returns only `_relevance_score`. If "all" is supported, returns relevance score along with the vector and/or FTS scores depending on query type. |

The reranker loads the model locally through `sentence-transformers`, so install the local model
runtime dependencies you need, such as PyTorch and any device-specific acceleration packages.

## Supported Scores for each query type
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
Expand Down
4 changes: 4 additions & 0 deletions docs/reranking/eval.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ Because the vector search operates in a dense embedding space and keyword-based
in a sparse embedding space, their relevance scores cannot be directly compared.
Combining results from multiple searches thus requires a reranking step.

Before evaluating hybrid search, build an FTS index on the text column and use a table with embedding
metadata or explicit query vectors for the vector side. Otherwise the evaluation is measuring setup
fallbacks or errors rather than the reranker.

## Reranking strategies

There are two common approaches for reranking search results from multiple sources.
Expand Down
8 changes: 6 additions & 2 deletions docs/reranking/linear_combination.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,12 @@ Accepted Arguments
----------------
| Argument | Type | Default | Description |
| --- | --- | --- | --- |
| `weight` | `float` | `0.7` | The weight to use for the semantic search score. The weight for the full-text search score is `1 - weights`. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all", will return all scores from the vector and FTS search along with the relevance score. |
| `weight` | `float` | `0.7` | The weight to use for the semantic search score. The weight for the full-text search score is `1 - weight`. |
| `fill` | `float` | `1.0` | Score used when a result is missing from one side of the hybrid query. The default strongly penalizes missing vector or FTS matches. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", returns only `_relevance_score`. If "all", returns all scores from the vector and FTS search along with the relevance score. |

`weight` must be between `0` and `1`. If either the vector or FTS side returns no rows, the reranker
returns the non-empty side with `_relevance_score` attached rather than failing.


## Supported Scores for each query type
Expand Down
3 changes: 3 additions & 0 deletions docs/reranking/mrr.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@ Accepted Arguments

**Note:** `weight_vector` + `weight_fts` must equal 1.0.

For multivector reranking, input result sets need `_rowid` so LanceDB can identify the same row
across the ranked lists. Add `.with_row_id(True)` to each vector search before passing the results
to the reranker.

## Supported Scores for each query type
You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type:
Expand Down
2 changes: 2 additions & 0 deletions docs/reranking/rrf.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ Accepted Arguments
| `K` | `int` | `60` | A constant used in the RRF formula (default is 60). Experiments indicate that k = 60 was near-optimal, but that the choice is not critical. |
| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score`. If "all", will return all scores from the vector and FTS search along with the relevance score. |

`K` must be greater than `0`. In TypeScript, construct the built-in reranker with
`await RRFReranker.create(k)` before passing it to `.rerank(...)`.

## Multi-vector reranking

Expand Down
3 changes: 2 additions & 1 deletion docs/search/hybrid-search.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -245,7 +245,8 @@ text_query = "flower moon"
Hybrid queries inherit the same builder API as vector and FTS queries, so the same knobs for filtering, distance bounds, and row identity apply. These compose with `.rerank(...)` and the explicit `.vector()` / `.text()` form shown above.

<Info>
Always set `.limit(...)` on production hybrid queries. Without an explicit cap, the query builder does not give you a useful top-k contract to tune, and it may materialize more rows than you intended before reranking.
Always set `.limit(...)` on production hybrid queries. LanceDB's default search limit is 10, but an
explicit cap gives you a clear top-k contract to tune before reranking.
</Info>

### Returning row IDs
Expand Down
6 changes: 5 additions & 1 deletion docs/search/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,9 @@ icon: "list"

- Vector search can run without an ANN index as an exhaustive scan. That's useful while prototyping, but build a vector index before relying on low-latency searches over larger tables.
- Full-text and hybrid text search require an FTS index on the text column you query. If a table has multiple FTS indexes, specify the target column. FTS also supports phrase, boolean, boosted, multi-match, and fuzzy query forms when you need more than plain terms.
- Phrase FTS queries require an index created with token positions enabled.
- Multivector search currently uses cosine similarity and accepts either one query vector or a matrix of query vectors; every query vector must match the inner dimension of the multivector column.
- Set an explicit `.limit(...)` for production queries. Query builders also support controls such as prefilter/postfilter, distance ranges, row-id inclusion, offset pagination, and Arrow/Pandas/list result materialization.
- Set an explicit `.limit(...)` for production queries. The default top-k is 10 for search builders,
but spelling it out makes latency and result-count assumptions visible. Query builders also
support controls such as prefilter/postfilter, distance ranges, row-id inclusion, offset
pagination, and Arrow/Pandas/list result materialization.
Loading
Loading