Retrieval pipeline improvements: reranking, length normalization, noise filtering, adaptive recall

[bm-clawd]

Context

Analyzed memory-lancedb-pro (LanceDB-based OpenClaw memory plugin) and their demo. While architecturally different from BM (opaque vectors vs human-readable plain text), their retrieval pipeline has several techniques worth adopting.

Our LoCoMo benchmark baseline: Recall@5 76.4%, Recall@10 85.5%, MRR 0.658. Weakest areas: single_hop 57.7%, temporal 59.1%. Root cause identified in #577: RRF scoring flattens results, FTS outperforms vector for observations.

Proposed Improvements

1. Cross-encoder reranking (highest impact)

After initial hybrid retrieval (vector + FTS), run a second pass through a cross-encoder reranker (e.g., Jina reranker-v3, Voyage rerank-2.5). This would re-score candidates based on query-document semantic relevance rather than just embedding similarity.

memory-lancedb-pro uses: 60% cross-encoder score + 40% original fused score, with graceful fallback to cosine similarity on API failure.

Implementation options:

• Cloud API: Jina, Voyage, Pinecone (cheap per-query cost)
• Local: BAAI/bge-reranker via Ollama-compatible endpoint
• Config-driven: optional, off by default, provider-agnostic

This directly addresses #577 (RRF flattening) and should significantly improve Recall@5 and MRR.

2. Length normalization

Long notes/observations currently have an outsized influence on search results. A length normalization step (anchor around 500 chars) would prevent verbose entries from dominating over precise, short observations.

3. Noise filtering / adaptive retrieval

Skip memory retrieval entirely for queries that don't need it: greetings, slash commands, simple confirmations, emoji-only messages. Also filter low-quality content from capture: agent refusals, meta-questions, boilerplate.

This reduces wasted retrieval cycles and keeps the index cleaner.

4. MMR diversity filtering

When multiple results are very similar (cosine > 0.85), demote duplicates to increase result diversity. Prevents near-duplicate observations from consuming all top-K slots.

What we already do better

• Human-readable plain text (their memories are opaque vector rows)
• Knowledge graph with relational structure (observations + relations)
• Bidirectional editing (humans can correct memories by editing files)
• Schema system for structured note types
• Git history as provenance trail

Priority

1. Cross-encoder reranking (biggest recall improvement per effort)
2. Length normalization (quick win)
3. MMR diversity (moderate effort)
4. Noise filtering / adaptive retrieval (nice to have)

References

• memory-lancedb-pro: https://github.com/CortexReach/memory-lancedb-pro
• LanceDB demo: https://github.com/lancedb/openclaw-lancedb-demo
• LoCoMo benchmark results: #608
• RRF scoring issue: BUG: Hybrid RRF fusion dilutes vector scores with weak FTS scores #577

[claude]

Claude finished @bm-clawd's task in 1m 34s —— View job

---

Triage Analysis

• Analyze issue and classify
• Check for duplicate issues
• Apply labels
• Provide triage summary

---

Classification

Field	Value
Type	Enhancement — improvements to the existing search/retrieval pipeline
Priority	Medium — improves recall metrics but no breakage
Component	Search/Database (retrieval pipeline)
Complexity	Complex — 4 distinct features, some with architectural implications

Labels applied: enhancement

---

Duplicate / Related Issues

Partial overlap with #618 ("Add reranking step to search pipeline (local cross-encoder)"), which covers cross-encoder reranking in detail with a tiered approach (Cloud / local OpenAI-compatible API / no-key fallback). Issue #666 is broader in scope, covering:

1. Cross-encoder reranking ← overlaps with Add reranking step to search pipeline (local cross-encoder) #618
2. Length normalization ← new
3. Noise filtering / adaptive retrieval ← new
4. MMR diversity filtering ← new

Recommend either closing #618 as a sub-item of this issue, or keeping #618 open and cross-linking them.

---

Triage Notes

Item 1 — Cross-encoder reranking is well-specified and aligns with the existing #618 proposal. The config-driven, provider-agnostic design is the right approach. Key decision: whether to ship cloud-only, local-only, or both in a first pass.

Item 2 — Length normalization is a quick win. BM25/FTS already has some length normalization built in (b parameter), but applying it post-fusion to the combined score is straightforward. Anchoring around 500 chars is a reasonable heuristic to validate against the LoCoMo benchmark.

Item 3 — Noise filtering / adaptive retrieval (skip retrieval for greetings, slash commands, etc.) is useful but should be implemented carefully to avoid false negatives — short queries like "what did we discuss?" should still trigger retrieval.

Item 4 — MMR diversity filtering (demote near-duplicates with cosine > 0.85) is a well-understood technique and fits naturally as a post-reranking step.

Suggested implementation order (matching the issue's priority list):

1. Length normalization — lowest risk, no new dependencies, measurable benchmark impact
2. MMR diversity filtering — post-processing step, no external dependencies
3. Cross-encoder reranking — coordinate with Add reranking step to search pipeline (local cross-encoder) #618, config-gated, provider-agnostic
4. Noise filtering — requires query classification logic, most care needed