Enhancement: native reranked retrieval + configurable BM25 text_config

[walcz-de]

Summary

LocalRecall (v0.6.3) does a solid RRF hybrid search — it fuses a BM25 rank list and a vector rank
list by reciprocal rank, and the vector arm uses the ANN index. Two things are missing that we've had
to work around downstream, and we'd like to discuss contributing both upstream:

1. No reranking step. LocalAI serves a cross-encoder reranker at /v1/rerank, but the retrieval
   path never calls it. Adding a rerank stage on top of the RRF result measurably improves relevance.
2. BM25 tokenization is hardcoded to English. text_config='english' is fixed in the index
   creation and the search_vector, which mis-tokenizes non-English corpora. We already have a small
   patch that makes it configurable.

Both are self-contained. We maintain a downstream fork and are happy to open the PRs.

1. What the code does today (v0.6.3)

• Retrieval (PostgresDB.Search → buildHybridSearchQuery) builds two rank lists — one from BM25
  (ORDER BY full_text <@> to_bm25query(...)) and one from vectors (ORDER BY embedding <=> q, which
  uses the ANN index) — and fuses them with weighted Reciprocal Rank Fusion (weight/(rrfK+rank),
  rrfK=60). This part is good; we're not proposing to change it.
• No rerank. /v1/rerank is served (the rerankers backend) but nothing in the retrieval path
  calls it. Knowledge-base recall and the collection-search endpoint return the RRF result directly.
  So a reranker is loaded and available, but retrieval never uses it.
• BM25 tokenization is English-only. The BM25 index is created WITH (text_config='english')
  and search_vector is built with to_tsvector('english', ...). For a German (or any non-English)
  corpus this applies the wrong stemming/stop-words, degrading the BM25 arm.

2. What we run in production, and what it shows

Because retrieval doesn't rerank, we wrote a thin tool that composes the primitives LocalAI already
provides:

search(query, k):
    candidates = collection_search(query, max_results = 200)      # 1. overfetch a candidate pool
    order      = POST /v1/rerank { query, documents = candidates } # 2. cross-encoder rerank
    return top_k(query, order)                                     # 3. keep the best k
    # if the reranker is unavailable, fall back to the original order — retrieval must never break

Measured on 98 German-language queries against a real document collection, adding the cross-encoder
rerank on top of the first-stage RRF result lifts strict chunk-hit@5 from 0.28 to 0.85 (a +0.57
absolute lift; MRR@5 0.73). In short, the rerank step is where most of the final ranking quality comes
from — the first-stage RRF alone leaves most relevant chunks outside the top-5.

The limitation is where this lives. The pipeline sits in a client-side tool, so only code that
explicitly calls that tool benefits. Two common retrieval paths do not go through it and
therefore get first-stage-only results:

1. the agent's automatic knowledge-base recall, which runs on every message when a knowledge base
   is attached — it queries LocalRecall directly, not through our tool; and
2. any application calling the collection-search endpoint (or /stores/*) directly.

So retrieval quality is currently opt-in: you only get it if you route through the wrapper. Anything
that talks to the platform directly gets the un-reranked result.

3. Why this is worth doing in the platform

The primitives are all here already — embeddings, reranking, chat, and the vector store, behind one
endpoint. The only missing piece is composing the reranker into retrieval natively, so it isn't
re-implemented outside the platform by every user who needs good retrieval.

For wider context (optional reading, not part of the proposal): Mozilla's State of Open Source AI
2026 argues that open-weight models have reached near-parity, and that the differentiation has moved
one layer up — to the "agentic harness" (orchestration, tools, memory/retrieval, permissions). Their
assessment scores the open stack lowest precisely on operational maturity and on the agent/retrieval
layer — summarized as "open ships easy, open deploys hard." Native reranked retrieval and correct
multilingual tokenization are squarely part of closing that gap.

4. Proposal — two small, independent contributions

1. Optional rerank stage in LocalRecall. After the RRF hybrid produces a candidate pool, rerank it
   via the configured reranker using the existing OpenAI-compatible /v1/rerank contract, then return
   the top k. Off by default. If the reranker fails for any reason, fall back to the RRF order —
   reranking must never break search. Pool size and reranker model configurable. This keeps the
   Search(query, k) contract stable, so callers (including the automatic knowledge-base recall) are
   unchanged.
2. Configurable text_config. Make the BM25 index/search_vector language configurable instead of
   hardcoded english, with a sensible default. (We have this as a small patch already and can send it
   as a standalone PR.)

Optionally, once (1) lands: expose the reranked pipeline through SearchCollectionEndpoint so
non-tool consumers get the same quality via the same URL.

Larger follow-ups we'd like to discuss but keep out of the first PRs: per-collection retrieval profiles
(embedding model, reranker, pool size as declarative config) and retrieval-level tracing/evaluation.

A licensing note for any bundled reranker example: jina-reranker-v2 is CC-BY-NC, while
bge-reranker-v2-m3 is Apache-2.0 and benchmarks within noise of it — so any shipped default should be
license-clean, and the stage itself should be model-agnostic.

5. What we're offering

We maintain a downstream fork and would be glad to open the PRs for both contributions, using our
production pipeline and benchmark harness as the reference implementation. The text_config change is
already forward-ported onto v0.6.3 and builds cleanly (a 74-line diff to rag/engine/postgres.go); the
rerank stage is what we run in production today. We are opening this issue first to agree on
the shape — an option vs. a separate Search variant, the config surface, and where the endpoint
boundary should sit — before sending code.

Would these be welcome contributions, and do you have a preference on the API shape?