Split off from #2436 during investigation, to keep that issue's confirmed, mechanical fix (Full build's missing warmup) separate from this genuinely open, infra-level design question.
Confirmed by investigation: each metric in scripts/incremental-benchmark.ts already computes a median of 5 in-process samples (with warmup for two of the three metrics — the third, fullBuildMs, was missing warmup, fixed separately). The non-determinism #2436 reports — the SAME commit failing on two DIFFERENT metrics across two CI re-runs, with local benchmarking on three separate trees showing no measurable difference between them — is cross-invocation (runner-to-runner / job-to-job) variance. An in-process median-of-N, however large, cannot smooth this out if the noise source affects the whole job's execution environment (thermal throttling, noisy-neighbor CI runners, etc.), not individual samples within one job.
tests/benchmarks/regression-guard.test.ts's KNOWN_REGRESSIONS allowlist already carries a 3.16.0:Full build / 3.16.0:1-file rebuild entry as a stopgap (added to unblock #2434/#2412), with a docstring explaining the exact flakiness — this is a workaround, not a fix.
Not attempted here because it requires either infrastructure changes I can't provision (a dedicated/consistent benchmark runner) or empirical validation against real CI behavior across many runs (does repeating the whole benchmark job 2-3x and taking a median across those separate job invocations actually reduce variance, versus just tripling CI cost for the same noisy result?) that isn't verifiable from a single investigative pass.
Separately, also noted in #2436: CODEGRAPH_FAST_SKIP_DIAG=1 (set in ci.yml/publish.yml/perf-canary.yml) only gates diagnostic info() logging (src/infrastructure/config.ts → pipeline.ts/detect-changes.ts's makeFastSkipLogger) — it does not alter control flow or add real work, so it's very unlikely to itself explain the documented ~3x CI-vs-local gap on the 1-file-rebuild metric. It's leftover instrumentation from the unrelated, already-closed #1066, per ci.yml's own comment ("remove once root cause is fixed" — never removed). Whether to clean it up is a separate, low-priority housekeeping question from the actual noise-source investigation.
Suggested next steps (from #2436's own text, still unaddressed):
- Record benchmarks on a dedicated/consistent runner, or normalize against a same-run calibration workload rather than a number recorded weeks ago on different hardware.
- Investigate replicating the whole benchmark job N times (not just samples within one job) and taking a median across invocations, if that's confirmed to actually reduce variance.
- Investigate the specific CI-vs-local ~3x gap on 1-file rebuild.
Split off from #2436 during investigation, to keep that issue's confirmed, mechanical fix (Full build's missing warmup) separate from this genuinely open, infra-level design question.
Confirmed by investigation: each metric in
scripts/incremental-benchmark.tsalready computes a median of 5 in-process samples (with warmup for two of the three metrics — the third,fullBuildMs, was missing warmup, fixed separately). The non-determinism #2436 reports — the SAME commit failing on two DIFFERENT metrics across two CI re-runs, with local benchmarking on three separate trees showing no measurable difference between them — is cross-invocation (runner-to-runner / job-to-job) variance. An in-process median-of-N, however large, cannot smooth this out if the noise source affects the whole job's execution environment (thermal throttling, noisy-neighbor CI runners, etc.), not individual samples within one job.tests/benchmarks/regression-guard.test.ts'sKNOWN_REGRESSIONSallowlist already carries a3.16.0:Full build/3.16.0:1-file rebuildentry as a stopgap (added to unblock #2434/#2412), with a docstring explaining the exact flakiness — this is a workaround, not a fix.Not attempted here because it requires either infrastructure changes I can't provision (a dedicated/consistent benchmark runner) or empirical validation against real CI behavior across many runs (does repeating the whole benchmark job 2-3x and taking a median across those separate job invocations actually reduce variance, versus just tripling CI cost for the same noisy result?) that isn't verifiable from a single investigative pass.
Separately, also noted in #2436:
CODEGRAPH_FAST_SKIP_DIAG=1(set inci.yml/publish.yml/perf-canary.yml) only gates diagnosticinfo()logging (src/infrastructure/config.ts→pipeline.ts/detect-changes.ts'smakeFastSkipLogger) — it does not alter control flow or add real work, so it's very unlikely to itself explain the documented ~3x CI-vs-local gap on the 1-file-rebuild metric. It's leftover instrumentation from the unrelated, already-closed #1066, perci.yml's own comment ("remove once root cause is fixed" — never removed). Whether to clean it up is a separate, low-priority housekeeping question from the actual noise-source investigation.Suggested next steps (from #2436's own text, still unaddressed):