From d8fc622c06565632796d23933cd6a99ed45c1cf2 Mon Sep 17 00:00:00 2001 From: "mintlify[bot]" <109931778+mintlify[bot]@users.noreply.github.com> Date: Wed, 26 Aug 2026 15:32:47 +0000 Subject: [PATCH] docs: document list-element FTS document granularity --- docs/indexing/fts-index.mdx | 29 ++++++++++++++++++++++++ docs/search/full-text-search.mdx | 39 ++++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+) diff --git a/docs/indexing/fts-index.mdx b/docs/indexing/fts-index.mdx index 400e07f..5cf85a2 100644 --- a/docs/indexing/fts-index.mdx +++ b/docs/indexing/fts-index.mdx @@ -97,6 +97,7 @@ await async_table.create_index("payload.text", config=FTS(with_position=True)) | `ngram_max_length` | int | `3` | Maximum n-gram length. Applies only when `base_tokenizer="ngram"`. | | `prefix_only` | bool | `False` | Index only prefix n-grams rather than all substrings. Applies only when `base_tokenizer="ngram"`. | | `block_size` | int | `128` | Number of documents per compressed posting block. Supported values are `128` and `256`. Setting this to `256` opts in to the experimental FTS V3 layout. | +| `document_granularity` | DocumentGranularity | `ROW` | Unit treated as one FTS document. `ROW` indexes each table row as one document. `LIST_ELEMENT` indexes each element of the deepest list on the field path as its own document. | - `max_token_length` can filter out base64 blobs or long URLs. @@ -136,6 +137,34 @@ await table.createIndex("text", { }); ``` +### Document granularity + +By default, an FTS index treats all of the text in one table row as a single document (`ROW` granularity). When the indexed field lives inside a list column, for example a `docs` column of type `list>`, you can instead index each element of the deepest list as its own document by setting `document_granularity` on the `FTS` config. + +Use list-element granularity when each list entry is a meaningful unit on its own, such as a list of passages, chunks, or reviews stored in one row. BM25 statistics are computed per element, so short matching elements are not diluted by unrelated text elsewhere in the row. + +```python Python icon="python" +from lancedb.index import FTS +from lancedb.query import DocumentGranularity + +# `docs` is a list> column. +table.create_index( + "docs.content", + config=FTS( + with_position=True, + document_granularity=DocumentGranularity.LIST_ELEMENT, + ), +) +``` + +Match and phrase queries against a list-element index return a `_doc_index` column. It contains a list of unsigned integers per row that identifies which list elements matched. See [querying with document granularity](/search/full-text-search#list-element-document-granularity) for query examples. + +Omitting `document_granularity` keeps the default row behavior, so existing index creation code is unaffected. + + +In LanceDB Enterprise, creating a list-element FTS index requires server API version 0.6.0 or later. Older servers reject the request instead of silently indexing with row semantics. Row-granularity requests work with all server versions. + + ### Phrase Query Configuration Enable phrase queries by setting: diff --git a/docs/search/full-text-search.mdx b/docs/search/full-text-search.mdx index 9764d46..31fd5fc 100644 --- a/docs/search/full-text-search.mdx +++ b/docs/search/full-text-search.mdx @@ -563,3 +563,42 @@ const phraseResults = await table.query() console.log(phraseResults); ``` + +### List-Element Document Granularity + +By default, all of the text in one row counts as a single document for BM25 scoring, even when the field is a list. You can instead treat each element of the deepest list as its own document by creating the index with `document_granularity=DocumentGranularity.LIST_ELEMENT`. See [document granularity](/indexing/fts-index#document-granularity) for index creation details. + +`MatchQuery` and `PhraseQuery` accept an optional `document_granularity` parameter in Python that selects which granularity to search: + +```python Python icon="python" +from lancedb.query import DocumentGranularity, MatchQuery, PhraseQuery + +# `docs` is a list> column indexed with +# document_granularity=DocumentGranularity.LIST_ELEMENT. +results = ( + table.search( + MatchQuery( + "alpha", + "docs.content", + document_granularity=DocumentGranularity.LIST_ELEMENT, + ) + ) + .limit(10) + .to_arrow() +) + +# _doc_index lists the indices of the matching list elements in each row. +print(results["_doc_index"]) +``` + +Results from a list-element search include a `_doc_index` column. Each value is a list of unsigned integers giving the positions of the matching elements within the row's list. Row-granularity searches do not return this column. + +The parameter resolves as follows: + +- If omitted, LanceDB infers the granularity from the index on the field. +- If the field has both a row and a list-element index, you must set `document_granularity` to pick one. +- If the field has no FTS index, LanceDB searches with row granularity. + + +In LanceDB Enterprise, list-element queries require server API version 0.6.0 or later. Explicit row-granularity queries work with all server versions. +