BM25 index rebuilds its average document length on every chunk (O(n²) ingest) and double-counts on re-ingest

[rajarshidattapy]

Description

Two defects in src/providers/rag/search.ts.

1. Quadratic index construction

// search.ts:183-193
function addToBM25Index(index: BM25Index, chunkId: string, text: string): void {
  const tokens = tokenize(text)
  index.docLengths.set(chunkId, tokens.length)
  index.docCount++

  let totalLength = 0
  for (const len of index.docLengths.values()) {   // full scan, every single insert
    totalLength += len
  }
  index.avgDocLength = totalLength / index.docCount
  ...
}

addChunks calls this once per chunk (search.ts:284-287), so inserting n chunks walks the docLengths map n times — O(n²) total. A LongMemEval question with 500 sessions produces several thousand chunks per container, and the harness runs containers for every question in the run.

The fix is a one-line running total:

-  let totalLength = 0
-  for (const len of index.docLengths.values()) totalLength += len
-  index.avgDocLength = totalLength / index.docCount
+  index.totalLength = (index.totalLength ?? 0) + tokens.length
+  index.avgDocLength = index.totalLength / index.docCount

2. Re-ingesting the same chunk corrupts the index

addChunks overwrites the chunk in the content map but appends to the BM25 index:

// search.ts:284-287
container.chunks.set(chunk.id, chunk)          // idempotent
addToBM25Index(container.bm25Index, chunk.id, chunk.content)   // NOT idempotent

addToBM25Index unconditionally does index.docCount++. Re-adding an existing chunk ID — which happens whenever a question is re-ingested with --force, or resumed such that a session is ingested twice — inflates docCount while docLengths.size stays constant. That corrupts:

• avgDocLength = totalLength / docCount → too small, which skews the BM25 length-normalisation term for every document,
• idf = log((docCount - df + 0.5) / (df + 0.5) + 1) (search.ts:219) → an inflated docCount inflates IDF for every term.

Chunk IDs are deterministic (${containerTag}_${sessionId}_${chunkIndex}, index.ts:167), so duplicates collide exactly rather than accumulating as separate entries — the corruption is invisible in getChunkCount().

Impact

Slow ingest that gets slower as the corpus grows, plus silently mis-scored keyword retrieval after any re-ingest. Since BM25 contributes 30% of the hybrid score (VECTOR_WEIGHT 0.7 / BM25_WEIGHT 0.3), this shifts the rag provider's measured retrieval quality in a way that depends on run history rather than on the algorithm — results are not reproducible between a fresh run and a resumed one.

Suggested fix

Maintain a running totalLength, and make addToBM25Index idempotent by removing an existing entry's postings and length before re-adding (or simply skip chunks whose ID is already present, since the ID is content-derived).