Random sampling uses `sort(() => Math.random() - 0.5)`, which is a biased shuffle — sampled runs are not representative

[rajarshidattapy]

Description

// src/orchestrator/index.ts:55-58
if (sampling.sampleType === "random") {
  const shuffled = [...questions].sort(() => Math.random() - 0.5)
  selected.push(...shuffled.slice(0, sampling.perCategory).map((q) => q.questionId))
}

Array.prototype.sort with a comparator that ignores its arguments does not produce a uniform permutation. The result depends on the engine's sort algorithm (V8/JSC use insertion sort below ~10 elements and TimSort above), and the well-known outcome is that elements stay near their original positions far more often than chance. Additionally, an inconsistent comparator violates the contract sort requires, so the behaviour is formally unspecified.

Concretely, for the arrays this code shuffles (tens to hundreds of questions per category), items near the front of the input are substantially over-represented in the first perCategory slots — which is exactly the slice that gets selected.

Impact

sampling.mode === "sample" with sampleType: "random" is the intended way to get a quick, cheap, representative read on a provider before committing to a full run. Because the questions arrive in dataset order (and, per [[issue_13]], in filesystem order), a front-biased shuffle means "random sample of 10 per category" systematically favours the same questions every time. Two providers sampled this way are compared on a non-uniform subset, and the sample's accuracy is not an unbiased estimate of full-run accuracy.

There is no seed either, so the selection is neither uniform nor reproducible — the worst of both. A benchmark harness should be able to answer "which questions did this run actually cover, and can I reproduce it?"

Suggested fix

Use a Fisher–Yates shuffle, and take an optional seed so sampled runs are reproducible and comparable across providers:

function shuffle<T>(arr: T[], rand: () => number = Math.random): T[] {
  const a = [...arr]
  for (let i = a.length - 1; i > 0; i--) {
    const j = Math.floor(rand() * (i + 1))
    ;[a[i], a[j]] = [a[j], a[i]]
  }
  return a
}

With a seeded PRNG, record the seed in the checkpoint alongside sampling so a comparison run can select the identical subset.