Question order comes from `readdirSync`, so `--limit N` selects a filesystem-dependent subset and runs are not reproducible across machines

[rajarshidattapy]

Description

LongMemEvalBenchmark.loadQuestions takes whatever order the filesystem hands back:

// src/benchmarks/longmemeval/index.ts:187-208
const files = readdirSync(questionsDir).filter((f) => f.endsWith(".json"))
for (const file of files) {
  ...
  this.questions.push({ ... })
}

There is no .sort(). readdirSync order is not specified — it reflects directory entry order, which on ext4 with dir_index is hash-ordered, on APFS/NTFS is roughly but not reliably lexicographic, and can change when files are rewritten.

That order then decides which questions a limited run covers:

// src/orchestrator/index.ts:227
targetQuestionIds = allQuestions.slice(0, effectiveLimit).map((q) => q.questionId)

and likewise for sampling.mode === "limit" (line 43) and sampleType !== "random" (line 59).

Impact

• bun run src/index.ts run -p supermemory -b locomo --limit 50 on a maintainer's Mac and on CI's Linux box benchmark different sets of 50 questions, and the accuracy difference between them is indistinguishable from a real provider difference.
• Comparing two providers on separate machines — or after the questions directory is regenerated — is not apples-to-apples.
• The published numbers cannot be independently reproduced, which for a benchmark is the whole point.

The same pattern applies to whichever other benchmarks load from a directory listing; src/benchmarks/convomem and locomo should be checked.

Related: load() is not idempotent

loadQuestions pushes into this.data and this.questions without clearing them first, and sessionsMap.set overwrites. Calling load() twice on the same instance duplicates every question, and getQuestions() then returns each one twice — which would double-count in the report and silently halve effective --limit coverage. Not currently triggered by the CLI path, but it is a footgun for the server, which caches benchmark instances (routes/runs.ts:16-24).

Suggested fix

-const files = readdirSync(questionsDir).filter((f) => f.endsWith(".json"))
+const files = readdirSync(questionsDir).filter((f) => f.endsWith(".json")).sort()

and reset this.data / this.questions / this.sessionsMap at the top of loadQuestions. Recording the resolved targetQuestionIds in the report (not just the checkpoint) would also make any past run auditable.