Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions docs/indexing/fts-index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ await async_table.create_index("payload.text", config=FTS(with_position=True))
| `ngram_max_length` | int | `3` | Maximum n-gram length. Applies only when `base_tokenizer="ngram"`. |
| `prefix_only` | bool | `False` | Index only prefix n-grams rather than all substrings. Applies only when `base_tokenizer="ngram"`. |
| `block_size` | int | `128` | Number of documents per compressed posting block. Supported values are `128` and `256`. Setting this to `256` opts in to the experimental FTS V3 layout. |
| `document_granularity` | DocumentGranularity | `ROW` | Unit treated as one FTS document. `ROW` indexes each table row as one document. `LIST_ELEMENT` indexes each element of the deepest list on the field path as its own document. |

<Note title="Key parameters">
- `max_token_length` can filter out base64 blobs or long URLs.
Expand Down Expand Up @@ -136,6 +137,34 @@ await table.createIndex("text", {
});
```

### Document granularity

By default, an FTS index treats all of the text in one table row as a single document (`ROW` granularity). When the indexed field lives inside a list column, for example a `docs` column of type `list<struct<content: string>>`, you can instead index each element of the deepest list as its own document by setting `document_granularity` on the `FTS` config.

Use list-element granularity when each list entry is a meaningful unit on its own, such as a list of passages, chunks, or reviews stored in one row. BM25 statistics are computed per element, so short matching elements are not diluted by unrelated text elsewhere in the row.

```python Python icon="python"
from lancedb.index import FTS
from lancedb.query import DocumentGranularity

# `docs` is a list<struct<content: string>> column.
table.create_index(
"docs.content",
config=FTS(
with_position=True,
document_granularity=DocumentGranularity.LIST_ELEMENT,
),
)
```

Match and phrase queries against a list-element index return a `_doc_index` column. It contains a list of unsigned integers per row that identifies which list elements matched. See [querying with document granularity](/search/full-text-search#list-element-document-granularity) for query examples.

Omitting `document_granularity` keeps the default row behavior, so existing index creation code is unaffected.

<Note>
In LanceDB Enterprise, creating a list-element FTS index requires server API version 0.6.0 or later. Older servers reject the request instead of silently indexing with row semantics. Row-granularity requests work with all server versions.
</Note>

### Phrase Query Configuration

Enable phrase queries by setting:
Expand Down
39 changes: 39 additions & 0 deletions docs/search/full-text-search.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -563,3 +563,42 @@ const phraseResults = await table.query()
console.log(phraseResults);
```
</CodeGroup>

### List-Element Document Granularity

By default, all of the text in one row counts as a single document for BM25 scoring, even when the field is a list. You can instead treat each element of the deepest list as its own document by creating the index with `document_granularity=DocumentGranularity.LIST_ELEMENT`. See [document granularity](/indexing/fts-index#document-granularity) for index creation details.

`MatchQuery` and `PhraseQuery` accept an optional `document_granularity` parameter in Python that selects which granularity to search:

```python Python icon="python"
from lancedb.query import DocumentGranularity, MatchQuery, PhraseQuery

# `docs` is a list<struct<content: string>> column indexed with
# document_granularity=DocumentGranularity.LIST_ELEMENT.
results = (
table.search(
MatchQuery(
"alpha",
"docs.content",
document_granularity=DocumentGranularity.LIST_ELEMENT,
)
)
.limit(10)
.to_arrow()
)

# _doc_index lists the indices of the matching list elements in each row.
print(results["_doc_index"])
```

Results from a list-element search include a `_doc_index` column. Each value is a list of unsigned integers giving the positions of the matching elements within the row's list. Row-granularity searches do not return this column.

The parameter resolves as follows:

- If omitted, LanceDB infers the granularity from the index on the field.
- If the field has both a row and a list-element index, you must set `document_granularity` to pick one.
- If the field has no FTS index, LanceDB searches with row granularity.

<Note>
In LanceDB Enterprise, list-element queries require server API version 0.6.0 or later. Explicit row-granularity queries work with all server versions.
</Note>