Discovered while investigating #2436 (Pre-publish benchmark gate non-determinism).
scripts/incremental-benchmark.ts (feeding generated/benchmarks/INCREMENTAL-BENCHMARKS.md) computes each metric as a median of RUNS=5 in-process samples, with 2 warmup runs for noopRebuildMs/oneFileRebuildMs to absorb NAPI/rusqlite/OS-page-cache cold-start cost (see that file's own WARMUP_RUNS doc comment).
scripts/benchmark.ts (feeding generated/benchmarks/BUILD-BENCHMARKS.md) has its own WARMUP_RUNS/INCREMENTAL_RUNS constants for ITS incremental metrics, but its top-level build-time metric (the BUILD-BENCHMARKS.md equivalent of "Full build") is a single timed call with no median and no warmup at all — a worse instance of the exact same class of bug that made incremental-benchmark.ts's own fullBuildMs unreliable before #2436's fix.
Not fixed as part of #2436's own PR since it's a separate benchmark surface (BUILD-BENCHMARKS.md, not INCREMENTAL-BENCHMARKS.md) with its own regression-guard checks, and deserves its own scoped verification against that file's specific history/thresholds rather than being bundled into an unrelated fix.
Suggested fix: apply the same warmup-then-median pattern (scripts/lib/bench-timing.ts's timeMedian/timeMedianWithValue, WARMUP_RUNS) to scripts/benchmark.ts's build-time measurement.
Discovered while investigating #2436 (Pre-publish benchmark gate non-determinism).
scripts/incremental-benchmark.ts(feedinggenerated/benchmarks/INCREMENTAL-BENCHMARKS.md) computes each metric as a median ofRUNS=5in-process samples, with 2 warmup runs fornoopRebuildMs/oneFileRebuildMsto absorb NAPI/rusqlite/OS-page-cache cold-start cost (see that file's ownWARMUP_RUNSdoc comment).scripts/benchmark.ts(feedinggenerated/benchmarks/BUILD-BENCHMARKS.md) has its ownWARMUP_RUNS/INCREMENTAL_RUNSconstants for ITS incremental metrics, but its top-level build-time metric (theBUILD-BENCHMARKS.mdequivalent of "Full build") is a single timed call with no median and no warmup at all — a worse instance of the exact same class of bug that madeincremental-benchmark.ts's ownfullBuildMsunreliable before #2436's fix.Not fixed as part of #2436's own PR since it's a separate benchmark surface (
BUILD-BENCHMARKS.md, notINCREMENTAL-BENCHMARKS.md) with its own regression-guard checks, and deserves its own scoped verification against that file's specific history/thresholds rather than being bundled into an unrelated fix.Suggested fix: apply the same warmup-then-median pattern (
scripts/lib/bench-timing.ts'stimeMedian/timeMedianWithValue,WARMUP_RUNS) toscripts/benchmark.ts's build-time measurement.