Description
aggregateRetrievalMetrics sums six fields but assigns the seventh:
// src/orchestrator/phases/report.ts:21-32
const sum = metrics.reduce(
(acc, m) => ({
hitAtK: acc.hitAtK + m.hitAtK,
...
k: m.k, // <-- assignment, not accumulation
}),
{ hitAtK: 0, ..., k: 10 }
)
...
return { ..., k: sum.k } // == the last element's kk is per-question and equals the number of results actually evaluated:
// src/orchestrator/phases/retrieval-eval.ts:97-108, 143
if (resultsToEval.length === 0) return { ..., k: 0, ... }
...
k: resultsToEval.lengthSo a question whose provider returned nothing contributes k: 0, and a question that returned 4 results contributes k: 4. Whichever question happens to be last in iteration order sets the k printed for the entire run.
Symptoms
RETRIEVAL QUALITY (K=0):
Hit@K: 62.0%
Precision: 41.3%
Hit@K and Precision@K are real aggregates over all questions, but the K label above them is one arbitrary question's value. When that question retrieved nothing, the header reads K=0, which is self-contradictory. The same value flows into byQuestionType[type].retrieval.k and into the per-type line at report.ts:364 (Hit@${stats.retrieval.k}=...), and into the UI wherever the report JSON is rendered.
Iteration order comes from benchmark.getQuestions() (report.ts:96), which per [[issue_13]] is filesystem-dependent — so the same run's report can print a different K on a different machine.
Impact
Cosmetic in the sense that no computed metric is wrong, but it is the label a reader uses to interpret every number beneath it. "Precision@0 = 41%" is not something a benchmark report should be able to emit.
Suggested fix
Report the configured k (the k: number = 10 default in calculateRetrievalMetrics) rather than a sampled per-question value, and if the effective k varies across questions, surface that explicitly — e.g. k as the max plus a count of questions that returned fewer than k results. The latter is genuinely useful information: it says how often the provider under-filled the result set.
Description
aggregateRetrievalMetricssums six fields but assigns the seventh:kis per-question and equals the number of results actually evaluated:So a question whose provider returned nothing contributes
k: 0, and a question that returned 4 results contributesk: 4. Whichever question happens to be last in iteration order sets thekprinted for the entire run.Symptoms
Hit@KandPrecision@Kare real aggregates over all questions, but theKlabel above them is one arbitrary question's value. When that question retrieved nothing, the header readsK=0, which is self-contradictory. The same value flows intobyQuestionType[type].retrieval.kand into the per-type line atreport.ts:364(Hit@${stats.retrieval.k}=...), and into the UI wherever the report JSON is rendered.Iteration order comes from
benchmark.getQuestions()(report.ts:96), which per [[issue_13]] is filesystem-dependent — so the same run's report can print a differentKon a different machine.Impact
Cosmetic in the sense that no computed metric is wrong, but it is the label a reader uses to interpret every number beneath it. "Precision@0 = 41%" is not something a benchmark report should be able to emit.
Suggested fix
Report the configured
k(thek: number = 10default incalculateRetrievalMetrics) rather than a sampled per-question value, and if the effectivekvaries across questions, surface that explicitly — e.g.kas the max plus a count of questions that returned fewer thankresults. The latter is genuinely useful information: it says how often the provider under-filled the result set.