From 83b2072d794347a853cf18ae0cb9562cebff826c Mon Sep 17 00:00:00 2001 From: prrao87 <35005448+prrao87@users.noreply.github.com> Date: Mon, 13 Jul 2026 16:09:57 -0400 Subject: [PATCH] docs: clarify public API gap notes --- docs/embedding/index.mdx | 14 ++++++++++++++ docs/embedding/quickstart.mdx | 9 +++++++++ docs/indexing/fts-index.mdx | 3 +++ docs/indexing/gpu-indexing.mdx | 4 ++++ docs/indexing/index.mdx | 9 +++++++-- docs/indexing/quantization.mdx | 4 ++-- docs/indexing/vector-index.mdx | 7 +++++++ docs/namespaces/usage.mdx | 4 ++++ docs/reranking/cross_encoder.mdx | 6 +++++- docs/reranking/eval.mdx | 4 ++++ docs/reranking/linear_combination.mdx | 8 ++++++-- docs/reranking/mrr.mdx | 3 +++ docs/reranking/rrf.mdx | 2 ++ docs/search/hybrid-search.mdx | 3 ++- docs/search/index.mdx | 6 +++++- docs/search/vector-search.mdx | 4 ++++ docs/tables/create.mdx | 4 ++++ docs/tables/update.mdx | 4 ++++ 18 files changed, 89 insertions(+), 9 deletions(-) diff --git a/docs/embedding/index.mdx b/docs/embedding/index.mdx index ff6399a0..1a238c13 100644 --- a/docs/embedding/index.mdx +++ b/docs/embedding/index.mdx @@ -86,6 +86,11 @@ to `OpenAIEmbeddingFunction::new_with_model` in Rust. | API key | `api_key="..."`, environment variables, or `$var:` | `apiKey: "..."`, environment variables, or `$var:` | Constructor argument or environment variable | | Device | Provider-specific, for example `device="cuda"` | Provider-specific | Provider-specific | +When ingesting data with an embedding definition, LanceDB only computes the vector column if that +column is missing from the incoming batch or present but entirely null. If you provide any non-null +values in the vector column, LanceDB treats the column as user-supplied and does not backfill the +remaining rows in that batch. + For reusable runtime configuration, the registry also supports `$var:` placeholders in embedding-function config. This is useful for provider secrets and environment-specific settings in Python and TypeScript. @@ -127,6 +132,10 @@ maps one source column to one vector column, and the table schema can contain mu The exact setup differs by SDK, but the underlying pattern is the same: define a distinct source/vector pair for each embedding function you want applied during ingest. +In TypeScript, automatic query embedding currently uses the first embedding function stored in the +table metadata. If a table has multiple embedding definitions and you need to query a specific vector +column, compute the query embedding explicitly and pass the vector to the search builder. + ## Embedding model providers LanceDB supports most popular embedding providers. @@ -187,6 +196,11 @@ For the Python remote client, `table.search("hello")` can take two different pat If you want explicit vector or hybrid behavior and the client cannot resolve an embedding function from the table metadata, generate the query embedding yourself and pass the vector directly. +TypeScript has a similar string-query distinction in `auto` mode: if no embedding providers have been +registered in the process, `search("text")` falls back to FTS. Once an embedding provider has been +imported and registered, the client expects table embedding metadata for automatic vector search and +raises an error if the table has none. + The manual query-embedding flow below works across Enterprise SDKs and is an explicit path you can use when you want full control over query-time behavior. diff --git a/docs/embedding/quickstart.mdx b/docs/embedding/quickstart.mdx index 42d27422..08ff9ce8 100644 --- a/docs/embedding/quickstart.mdx +++ b/docs/embedding/quickstart.mdx @@ -38,6 +38,7 @@ from lancedb.embeddings import get_registry - `Vector`: Field type for storing vector embeddings - `get_registry()`: Access to the embedding function registry. It has all the supported as well as custom embedding functions registered by the user - TypeScript uses `lancedb.embedding.getRegistry()` and `lancedb.embedding.LanceSchema()` for the same registry/schema workflow +- In TypeScript, import the provider module before calling `getRegistry().get(...)`; the provider import is what registers names such as `"huggingface"` or `"openai"` ## Step 2: Connect to LanceDB @@ -119,6 +120,10 @@ The `table.add()` call automatically: - Generates embeddings using your chosen model - Stores both the original text and the vector embeddings +If your input already includes the vector column, automatic embedding only runs when that column is +absent or entirely null for the batch. Partially supplied vectors are treated as manual data, so +LanceDB preserves them instead of filling only the missing rows. + ## Step 6: Query with Automatic Embedding Note: On LanceDB Enterprise, the server does not generate embeddings from query text. In the Python remote @@ -144,3 +149,7 @@ The search process: 1. Automatically converts your query text to embeddings 2. Finds the most similar vectors in your table 3. Returns the matching documents + +Automatic text search depends on the table's embedding metadata. If the client cannot reconstruct an +embedding function from that metadata, compute the query embedding yourself and search with the +vector directly. diff --git a/docs/indexing/fts-index.mdx b/docs/indexing/fts-index.mdx index 874c6862..043d1410 100644 --- a/docs/indexing/fts-index.mdx +++ b/docs/indexing/fts-index.mdx @@ -49,6 +49,9 @@ When using async connections (`connect_async`), use `create_index` with the `FTS The `create_fts_index` method is not available on `AsyncTable`. Use `create_index` with `FTS` config instead. +The current FTS implementation is Lance-native. Legacy Tantivy-only options, including +`use_tantivy`, are no longer accepted by the index creation APIs. + ## Nested field paths FTS indexes can target text leaves inside struct columns by passing a dotted path (for example, `payload.text`). The same path works for [`MatchQuery`](/search/full-text-search) and [`PhraseQuery`](/search/full-text-search), and for the `columns` argument on async `nearest_to_text` queries. diff --git a/docs/indexing/gpu-indexing.mdx b/docs/indexing/gpu-indexing.mdx index 70240a1d..67025102 100644 --- a/docs/indexing/gpu-indexing.mdx +++ b/docs/indexing/gpu-indexing.mdx @@ -28,6 +28,10 @@ into a synchronous process by waiting until the index is built. `wait_for_index(...)` waits for the index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`. It can time out if the table is receiving continuous writes while the build is trying to catch up. +GPU acceleration changes how the vector index is built, not the lifecycle after the build. Rows +appended later still need `optimize()` before they are part of the index, and `fast_search()` only +searches indexed rows. + ## Manual GPU indexing in LanceDB OSS You can use the Python SDK to manually create the `IVF_PQ` index on a GPU. You'll need diff --git a/docs/indexing/index.mdx b/docs/indexing/index.mdx index 815cf5f2..fd15d051 100644 --- a/docs/indexing/index.mdx +++ b/docs/indexing/index.mdx @@ -31,9 +31,10 @@ LanceDB provides a comprehensive suite of indexing strategies for different data | `IVF` (Vector) | Large-scale vector search with configurable accuracy/speed trade-offs. Supports binary vectors with hamming distance. | Inverted File Index—a partition-based approximate nearest neighbor algorithm that groups similar vectors into partitions for efficient search.
Distance metrics: `l2` `cosine` `dot` `hamming`
Quantizations: `None/Flat` `PQ` `SQ` `RQ`| | `IVF_HNSW` (Vector) | Large-scale vector search requiring both high recall and efficient partitioning. Combines the scalability of IVF with the search quality of HNSW. | Hybrid index combining IVF partitioning with HNSW graphs built within each partition. Provides improved search quality over pure IVF while maintaining scalability.
Distance metrics: `l2` `cosine` `dot`
Quantizations: `None/Flat` `SQ` `PQ`| | `FTS` (Full-text search) | String columns (e.g., title, description, content) requiring keyword-based search with BM25 ranking. | Full-text search index using BM25 ranking algorithm. Tokenizes text with configurable tokenization, stemming, stop word removal, and language-specific processing. | -| `BTree` (Scalar) | Numeric, temporal, and string columns with mostly distinct values. Best for highly selective queries on columns with many unique values. | Sorted index storing sorted copies of scalar columns with block headers in a btree cache. Header entries map to blocks of rows (4096 rows per block) for efficient disk reads. | +| `BTree` (Scalar) | Numeric, temporal, and string columns with mostly distinct values. Best for selective equality, inequality, and range predicates. | Sorted index storing sorted copies of scalar columns with block headers in a btree cache. Header entries map to blocks of rows (4096 rows per block) for efficient disk reads. | | `Bitmap` (Scalar) | Low-cardinality columns with few thousand or fewer distinct values. Accelerates equality and range filters. | Stores a bitmap for each distinct value in the column, with one bit per row indicating presence. Memory-efficient for low-cardinality data. | -| `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring array containment queries. | Scalar index for `List` and `LargeList` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. | +| `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring `array_contains_all` or `array_contains_any` filters. | Scalar index for `List` and `LargeList` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. | +| `FM` (Scalar) | String or binary columns that need raw substring search. | FM-Index over `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary` data for filters such as `contains(path, 'needle')`. Use FTS instead for tokenized word search and BM25 ranking. | TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). @@ -43,6 +44,10 @@ TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). **Operational checks** For vector indexes, use the same distance metric when creating the index and searching it. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` if you need to confirm the index has caught up. `wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`; it can time out if writes keep adding unindexed rows. + +By default, automatic vector indexing creates `IVF_PQ`, and scalar index creation defaults to +`BTree` unless you pass another scalar index config. `BTree` and `Bitmap` indexes target scalar +columns, not list columns; use `LabelList` for list containment filters. ### Quantization Types diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index 85d277cb..c0d5156f 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -68,7 +68,7 @@ The full list of parameters to the algorithm are listed below. - `distance_type`: Literal["l2", "cosine", "dot"], defaults to "l2" The distance metric to use for similarity comparison. Choose "l2" for Euclidean, "cosine" for cosine similarity, or "dot" for dot product. - `num_partitions`: Optional[int], defaults to None - Number of IVF partitions (affects index build time and query accuracy). More partitions can improve recall but may increase build time. + Number of IVF partitions (affects index build time and query accuracy). More partitions can improve recall but may increase build time. When unset, LanceDB chooses roughly the square root of the row count. - `num_bits`: int, defaults to 1 Bits per dimension for quantization (1 is standard RaBitQ). Higher values improve fidelity, mainly at the cost of additional storage. - `max_iterations`: int, defaults to 50 @@ -76,4 +76,4 @@ The full list of parameters to the algorithm are listed below. - `sample_rate`: int, defaults to 256 Number of samples per partition during training. Higher values may improve accuracy but increase training time. - `target_partition_size`: Optional[int], defaults to None - Target number of vectors per partition. Adjust to control partition granularity and memory usage. + Target number of vectors per partition. Adjust to control partition granularity and memory usage. If `num_partitions` is also set, `num_partitions` takes precedence. diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 3eaacdcc..359a6d22 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -56,6 +56,10 @@ Although the `create_index` API returns immediately, the building of the vector Use the same distance metric for index creation and search. Once a vector index exists, queries use the metric stored with that index. If you need to confirm an async build or refresh is finished, `wait_for_index(...)` waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`; it can time out if new writes keep arriving. +Rows appended after an index build remain outside that index until optimization refreshes it. Normal +search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that +fallback and searches only indexed rows. + ## Choose the Right Index Use this table as a quick starting point for choosing the right index type and quantization method for your use case: @@ -109,6 +113,9 @@ Sometimes you need to configure the index beyond default parameters: - `num_partitions`: use index-specific starting points from the section above: - HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`): `num_rows // 1,048,576` - `IVF_RQ` and `IVF_PQ`: `num_rows // 4096` +- `target_partition_size`: alternative IVF sizing knob that asks LanceDB to derive the partition + count from a target number of rows per partition. If you set both `num_partitions` and + `target_partition_size`, `num_partitions` takes precedence. - `num_sub_vectors`: applies to `IVF_PQ`; start with `dimension // 8`. Larger values often improve recall but can slow search. Let's take a look at a sample request for an IVF index: diff --git a/docs/namespaces/usage.mdx b/docs/namespaces/usage.mdx index d71d0507..009ce9ff 100644 --- a/docs/namespaces/usage.mdx +++ b/docs/namespaces/usage.mdx @@ -98,6 +98,10 @@ doesn't exist, or contains data: Namespace path components can't be empty. Each component can contain only letters, numbers, underscores, hyphens, and periods. +Listing APIs return the immediate children of the requested namespace path. Use `limit` with the +returned `page_token` to page through large catalogs; pass an empty namespace path (`[]`) when you +want to list from the root namespace. + ## Namespaces in LanceDB Enterprise In LanceDB Enterprise deployments, configure namespace-backed federated databases in a TOML file under your deployment's `config` directory. diff --git a/docs/reranking/cross_encoder.mdx b/docs/reranking/cross_encoder.mdx index b332aa5b..de90435f 100644 --- a/docs/reranking/cross_encoder.mdx +++ b/docs/reranking/cross_encoder.mdx @@ -26,7 +26,11 @@ Accepted Arguments | `model_name` | `str` | `"cross-encoder/ms-marco-TinyBERT-L-6"` | The name of the reranker model to use.| | `column` | `str` | `"text"` | The name of the column to use as input to the cross encoder model. | | `device` | `str` | `None` | The device to use for the cross encoder model. If None, will use "cuda" if available, otherwise "cpu". | -| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all" is supported, will return relevance score along with the vector and/or fts scores depending on query type. | +| `trust_remote_code` | `bool` | `True` | Passed to Sentence Transformers model loading. Set this to `False` when you only want to load models that do not require custom repository code. | +| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", returns only `_relevance_score`. If "all" is supported, returns relevance score along with the vector and/or FTS scores depending on query type. | + +The reranker loads the model locally through `sentence-transformers`, so install the local model +runtime dependencies you need, such as PyTorch and any device-specific acceleration packages. ## Supported Scores for each query type You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type: diff --git a/docs/reranking/eval.mdx b/docs/reranking/eval.mdx index 6a2d1cfd..facf0d7c 100644 --- a/docs/reranking/eval.mdx +++ b/docs/reranking/eval.mdx @@ -11,6 +11,10 @@ Because the vector search operates in a dense embedding space and keyword-based in a sparse embedding space, their relevance scores cannot be directly compared. Combining results from multiple searches thus requires a reranking step. +Before evaluating hybrid search, build an FTS index on the text column and use a table with embedding +metadata or explicit query vectors for the vector side. Otherwise the evaluation is measuring setup +fallbacks or errors rather than the reranker. + ## Reranking strategies There are two common approaches for reranking search results from multiple sources. diff --git a/docs/reranking/linear_combination.mdx b/docs/reranking/linear_combination.mdx index c5216820..77975022 100644 --- a/docs/reranking/linear_combination.mdx +++ b/docs/reranking/linear_combination.mdx @@ -26,8 +26,12 @@ Accepted Arguments ---------------- | Argument | Type | Default | Description | | --- | --- | --- | --- | -| `weight` | `float` | `0.7` | The weight to use for the semantic search score. The weight for the full-text search score is `1 - weights`. | -| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score. If "all", will return all scores from the vector and FTS search along with the relevance score. | +| `weight` | `float` | `0.7` | The weight to use for the semantic search score. The weight for the full-text search score is `1 - weight`. | +| `fill` | `float` | `1.0` | Score used when a result is missing from one side of the hybrid query. The default strongly penalizes missing vector or FTS matches. | +| `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", returns only `_relevance_score`. If "all", returns all scores from the vector and FTS search along with the relevance score. | + +`weight` must be between `0` and `1`. If either the vector or FTS side returns no rows, the reranker +returns the non-empty side with `_relevance_score` attached rather than failing. ## Supported Scores for each query type diff --git a/docs/reranking/mrr.mdx b/docs/reranking/mrr.mdx index 2747ef92..cb6cbbe0 100644 --- a/docs/reranking/mrr.mdx +++ b/docs/reranking/mrr.mdx @@ -29,6 +29,9 @@ Accepted Arguments **Note:** `weight_vector` + `weight_fts` must equal 1.0. +For multivector reranking, input result sets need `_rowid` so LanceDB can identify the same row +across the ranked lists. Add `.with_row_id(True)` to each vector search before passing the results +to the reranker. ## Supported Scores for each query type You can specify the type of scores you want the reranker to return. The following are the supported scores for each query type: diff --git a/docs/reranking/rrf.mdx b/docs/reranking/rrf.mdx index c89396ca..7df34778 100644 --- a/docs/reranking/rrf.mdx +++ b/docs/reranking/rrf.mdx @@ -36,6 +36,8 @@ Accepted Arguments | `K` | `int` | `60` | A constant used in the RRF formula (default is 60). Experiments indicate that k = 60 was near-optimal, but that the choice is not critical. | | `return_score` | `str` | `"relevance"` | Options are "relevance" or "all". The type of score to return. If "relevance", will return only the `_relevance_score`. If "all", will return all scores from the vector and FTS search along with the relevance score. | +`K` must be greater than `0`. In TypeScript, construct the built-in reranker with +`await RRFReranker.create(k)` before passing it to `.rerank(...)`. ## Multi-vector reranking diff --git a/docs/search/hybrid-search.mdx b/docs/search/hybrid-search.mdx index d5785f36..a45f0df2 100644 --- a/docs/search/hybrid-search.mdx +++ b/docs/search/hybrid-search.mdx @@ -245,7 +245,8 @@ text_query = "flower moon" Hybrid queries inherit the same builder API as vector and FTS queries, so the same knobs for filtering, distance bounds, and row identity apply. These compose with `.rerank(...)` and the explicit `.vector()` / `.text()` form shown above. -Always set `.limit(...)` on production hybrid queries. Without an explicit cap, the query builder does not give you a useful top-k contract to tune, and it may materialize more rows than you intended before reranking. +Always set `.limit(...)` on production hybrid queries. LanceDB's default search limit is 10, but an +explicit cap gives you a clear top-k contract to tune before reranking. ### Returning row IDs diff --git a/docs/search/index.mdx b/docs/search/index.mdx index 739ce0aa..cec4613c 100644 --- a/docs/search/index.mdx +++ b/docs/search/index.mdx @@ -18,5 +18,9 @@ icon: "list" - Vector search can run without an ANN index as an exhaustive scan. That's useful while prototyping, but build a vector index before relying on low-latency searches over larger tables. - Full-text and hybrid text search require an FTS index on the text column you query. If a table has multiple FTS indexes, specify the target column. FTS also supports phrase, boolean, boosted, multi-match, and fuzzy query forms when you need more than plain terms. +- Phrase FTS queries require an index created with token positions enabled. - Multivector search currently uses cosine similarity and accepts either one query vector or a matrix of query vectors; every query vector must match the inner dimension of the multivector column. -- Set an explicit `.limit(...)` for production queries. Query builders also support controls such as prefilter/postfilter, distance ranges, row-id inclusion, offset pagination, and Arrow/Pandas/list result materialization. +- Set an explicit `.limit(...)` for production queries. The default top-k is 10 for search builders, + but spelling it out makes latency and result-count assumptions visible. Query builders also + support controls such as prefilter/postfilter, distance ranges, row-id inclusion, offset + pagination, and Arrow/Pandas/list result materialization. diff --git a/docs/search/vector-search.mdx b/docs/search/vector-search.mdx index 2d82095d..06381dfe 100644 --- a/docs/search/vector-search.mdx +++ b/docs/search/vector-search.mdx @@ -204,6 +204,10 @@ You can change `approx_mode` per query without rebuilding the index. For RQ inde - If recall is too low, increase `nprobes` gradually, but after a certain threshold, increasing `nprobes` yields only marginal accuracy gains. - If you need higher performance and have recall headroom, decrease `nprobes` gradually. +For filtered ANN searches, you can also set `minimum_nprobes` and `maximum_nprobes`. LanceDB starts +with the minimum and can scan more partitions up to the maximum if the filter leaves too few +candidates. Calling `nprobes(n)` fixes both values to `n`, which disables that adaptive behavior. + ### Vector Search with Prefiltering This is the default vector search setting. You can use prefiltering to boost query performance by reducing the search space before vector calculations begin. The system first applies your filter criteria to the dataset, then conducts vector search operations only on the remaining relevant subset. diff --git a/docs/tables/create.mdx b/docs/tables/create.mdx index 6a46b0ef..0e422df3 100644 --- a/docs/tables/create.mdx +++ b/docs/tables/create.mdx @@ -144,6 +144,10 @@ non-nullable column, or provide actual nulls for it, ingestion fails; nullable c omitted or written with null values. Without an explicit schema, Python infers list-like vector values as fixed-size `float32` vector fields from the observed dimension. +For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length, +null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes +`fill_value`, and `"null"` writes nulls. + ### From an Arrow Table You can also create LanceDB tables directly from Arrow tables. Rust uses an Arrow `RecordBatchReader` for the same Arrow-native ingest flow. diff --git a/docs/tables/update.mdx b/docs/tables/update.mdx index 11f4c431..ad4aaa8e 100644 --- a/docs/tables/update.mdx +++ b/docs/tables/update.mdx @@ -244,6 +244,10 @@ Like the create table and add APIs, the merge insert API will automatically comp During `merge_insert`, if the input data doesn't contain the source column (i.e., the original field used to generate embeddings, such as text for a text embedding model or `image_uri` for an image model), or if a vector value is already provided, LanceDB skips embedding generation for that row. Embeddings are only auto-generated when that source field is present in the incoming data, **and** the vector field is empty. +Primary keys in LanceDB are metadata used by operations such as `merge_insert`; they are not +enforced as uniqueness constraints on ordinary writes. Keep using `merge_insert` or your own +deduplication logic when you need key-based upsert semantics. + ### Update matched rows only This updates keys that already exist in the target table. Source rows with new keys are ignored.