Reported Recall@K, F1@K and NDCG are mathematically degenerate — they measure nothing beyond Hit@K

[rajarshidattapy]

Description

calculateRetrievalMetrics has no notion of how many relevant documents actually exist for a question. It substitutes the number it happened to retrieve:

// src/orchestrator/phases/retrieval-eval.ts:119-134
const relevantRetrieved = relevanceScores.filter((r) => r === 1).length
const totalRelevant = Math.max(1, relevantRetrieved)     // <-- ground truth invented from the result

const hitAtK   = relevantRetrieved > 0 ? 1 : 0
const recallAtK = relevantRetrieved > 0 ? 1 : 0          // <-- identical to hitAtK, by construction
const f1AtK = ...                                        // <-- derived from that fake recall
const ndcg = calculateNDCG(relevanceScores, totalRelevant)

Consequences, each provable from the code alone:

1. recallAtK is literally hitAtK. Same expression, same value, always. Retrieve 1 of 20 relevant memories → recall reported as 100%. The report prints them as two separate rows (report.ts:342-344), implying independent signal that does not exist.
2. f1AtK is not an F1 score. With recallAtK pinned to 1, it collapses to 2p/(p+1) — a monotone re-encoding of precision that adds no information and is not comparable to F1 as published in any retrieval paper.
3. NDCG's ideal ranking is built from the retrieved set. calculateNDCG(relevanceScores, totalRelevant) sets IDCG to "the relevantRetrieved relevant items, ranked at the top" (retrieval-eval.ts:77-83). A provider that retrieves exactly one relevant item at rank 1 scores NDCG = 1.0, identical to one that retrieves ten relevant items at ranks 1–10. It measures ordering within what you found, never what you missed — the thing NDCG exists to measure.
4. A provider that returns nothing relevant and a provider that returns everything relevant can both score NDCG 0.0 / 1.0 respectively regardless of the corpus. The metric cannot distinguish coverage at all.

Additional defect: judge failures are scored as "not relevant"

Both error paths in evaluateAllChunks return all-zero relevance:

// retrieval-eval.ts:60-69
if (!jsonMatch) return searchResults.map((_, i) => ({ id: `result_${i+1}`, relevant: 0 as const }))
...
} catch { return searchResults.map((_, i) => ({ id: `result_${i+1}`, relevant: 0 as const })) }

A judge timeout, a rate-limit error, or a malformed JSON response is therefore indistinguishable from "the provider retrieved nothing useful", and it silently drags that provider's retrieval numbers down. These should be recorded as null/absent and excluded from the aggregate, not counted as zeros.

Impact

Four of the six numbers under "RETRIEVAL QUALITY" in every report are either duplicates or non-standard quantities presented under standard names. Readers will compare them against published Recall@K / NDCG figures from the LongMemEval and LoCoMo literature, where they mean something quite different.

Suggested fix

• LongMemEval ships has_answer per haystack message — which splitQuestions currently deletes (src/benchmarks/longmemeval/index.ts:170-178). Preserving it gives a real totalRelevant per question and makes Recall@K, F1@K and NDCG well-defined.
• Until real ground truth is wired in, stop emitting recallAtK, f1AtK and ndcg. Publishing Hit@K, Precision@K and MRR alone is honest and still useful.
• Propagate judge errors instead of coercing them to relevant: 0.