Skip to content

[python] Support data prefilters for full-text and hybrid search - #10006

Open
TheR1sing3un wants to merge 1 commit into
apache:masterfrom
TheR1sing3un:codex/full-text-scalar-filter
Open

TheR1sing3un wants to merge 1 commit into
apache:masterfrom
TheR1sing3un:codex/full-text-scalar-filter

Conversation

@TheR1sing3un

Copy link
Copy Markdown
Member

Purpose

Full-text search currently restricts pre_filter to partition columns, and hybrid queries reject data predicates when a full-text route is present. Support ordinary-column prefilters on data-evolution tables, applying the predicate before full-text Top-K selection and forwarding it to both hybrid routes.

Reuse the vector filter's snapshot-pinned data-read helper to collect exact matching row IDs from the filter columns, then pass that bitmap to native full-text search. This read uses full scalar-index coverage so partial scalar indexes cannot exclude rows covered by the full-text plan. Partition-only queries retain their pruning path, and vector candidate-refinement policy is unchanged. Raw fallback preserves the unfiltered BM25 corpus; full mode also covers selected partitions with no full-text index files. Document the additional filter-column read cost.

Tests

  • Native full-text indexes with no scalar index, BTree and Bitmap; partial scalar coverage, mixed indexed/raw data and raw-only partitions.
  • Non-partitioned tables, filtered Top-K, stable BM25 scores, AND/OR partition semantics, hybrid routing, empty results, historical snapshots/deletions and concurrent commits.
  • Relevant multimodal, vector filtering, exact-filter resource cleanup, Ray filter exactness, primary-key golden data and global-index coverage regressions passed.
  • Repository-configured Flake8, Apache license checks and Python 3.6 grammar checks passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant