From 82fd7a8811b3b37aff152dd6a9027558955c90bf Mon Sep 17 00:00:00 2001 From: Alireza Date: Thu, 9 Jul 2026 10:52:52 -0400 Subject: [PATCH 1/2] ci(codspeed): run the simulation bench gate on a dedicated self-hosted runner Move the codspeed-bench job off GitHub's shared pool (ubuntu-24.04) onto a fixed self-hosted runner labeled 'codspeed-bench'. Simulation instruction counts derive from the runner CPU's cache model, and the shared pool randomly assigns Intel vs AMD hardware, producing spurious cross-run 'Different runtime environments' deltas. A dedicated fixed box keeps every run on identical hardware so the gate is stable. Adds docs/ci/self-hosted-runner.md with the box requirements and cutover order. --- .github/workflows/pr-checks.yml | 19 ++++++---- docs/ci/self-hosted-runner.md | 65 +++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+), 7 deletions(-) create mode 100644 docs/ci/self-hosted-runner.md diff --git a/.github/workflows/pr-checks.yml b/.github/workflows/pr-checks.yml index 60a133a..516f637 100644 --- a/.github/workflows/pr-checks.yml +++ b/.github/workflows/pr-checks.yml @@ -332,13 +332,18 @@ jobs: codspeed-bench: needs: [detect-changes, build] if: needs.detect-changes.outputs.any == 'true' - # Pin the OS (not ubuntu-latest) so PR runs are measured in the same - # runtime environment as the baseline seeded on main. A drifting - # ubuntu-latest image changes the valgrind/toolchain fingerprint between - # baseline and PR, which makes CodSpeed report "Different runtime - # environments detected" and refuse to trust the comparison. The baseline - # on main must be re-seeded after changing this so both sides match. - runs-on: ubuntu-24.04 + # Runs on a DEDICATED self-hosted runner with fixed hardware (label + # `codspeed-bench`). GitHub's shared runners randomly assign different + # physical CPUs (Intel Xeon 8370C vs AMD EPYC 7763); simulation-mode + # instruction counts are derived from the runner CPU's cache model, so a + # baseline (main) and a PR landing on different CPUs produce spurious + # "Different runtime environments detected" deltas. One fixed box keeps + # every run on identical hardware, so Simulation is stable run-to-run. + # See docs/ci/self-hosted-runner.md for what the box must provide + # (valgrind, node 22, a pinned/isolated CPU) and the cutover steps. + # IMPORTANT: after this lands, the baseline on main must be re-seeded by + # one main run on THIS runner before PR comparisons are meaningful. + runs-on: [self-hosted, codspeed-bench] steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 diff --git a/docs/ci/self-hosted-runner.md b/docs/ci/self-hosted-runner.md new file mode 100644 index 0000000..0359d82 --- /dev/null +++ b/docs/ci/self-hosted-runner.md @@ -0,0 +1,65 @@ +# Self-hosted CodSpeed benchmark runner + +The `codspeed-bench` job in `.github/workflows/pr-checks.yml` runs on a +**dedicated self-hosted runner** (`runs-on: [self-hosted, codspeed-bench]`) +instead of GitHub's shared pool. + +## Why + +CodSpeed simulation mode (Cachegrind) derives its modeled CPU cache from the +**physical** runner CPU. GitHub's shared runners randomly assign different +hardware (Intel Xeon 8370C vs AMD EPYC 7763), so identical source produces +different instruction counts run-to-run and CodSpeed flags "Different runtime +environments detected". A single fixed box removes that variable: every run — +baseline and PR — is measured on the same hardware, so Simulation is stable +and the regression gate is trustworthy. + +## What the box must provide + +- **OS**: Linux x64. +- **valgrind** installed and on `PATH` (`valgrind --version`). CodSpeed's + simulation runs the benches under valgrind; the box must have it (the job + does not install it). +- **git** and outbound network (to reach github.com and codspeed.io). +- Node is provisioned per-job by `actions/setup-node@v4` (node 22), so it does + not need to be pre-installed — but the runner user must be able to write to + the actions tool cache. +- A C toolchain is **not** needed here: this job only downloads prebuilt + `dist-*` artifacts and runs `vitest bench`; the wasm is compiled in the + `build` job on GitHub-hosted runners. + +## Hardware hygiene (for stable numbers) + +- **Dedicate the box to this label.** Register it so it only picks up the + `codspeed-bench` label and runs **one job at a time** (the default). Do not + co-locate other CI or workloads on it — contention is noise. +- Pin the CPU: disable turbo boost and, ideally, SMT/hyper-threading; set the + CPU governor to `performance`. A VM is fine if it has a pinned vCPU and is + otherwise idle. + +## Registering the runner + +1. Repo → **Settings → Actions → Runners → New self-hosted runner** → Linux/x64. +2. Follow the download/configure steps GitHub shows. When configuring, add the + custom label so the workflow can target it: + `./config.sh --url https://github.com/cornerstonejs/codecs --token --labels codspeed-bench` + (the `self-hosted`, `Linux`, `X64` labels are added automatically). +3. Install it as a service so it survives reboots (`./svc.sh install && ./svc.sh start`). +4. Confirm it shows **Idle** in Settings → Actions → Runners. + +## Cutover order (important) + +1. **Register and start the runner first** (above). If the workflow merges + before a runner with the `codspeed-bench` label is online, the + `codspeed-bench` job queues indefinitely. +2. Merge this change to `main`. +3. The push to `main` runs `codspeed-bench` on the new runner and **re-seeds + the CodSpeed baseline on the fixed hardware**. Until that main run + completes, the first PR comparisons will show a one-time environment shift + (old shared-runner baseline vs new fixed-hardware head) — that is expected + and self-resolves after the baseline is re-seeded. + +## Rollback + +Revert `runs-on` back to `ubuntu-24.04` on the `codspeed-bench` job. The next +`main` run re-seeds the baseline on GitHub-hosted hardware again. From 22d2ec1a853b008df67229440d377ae51542b446 Mon Sep 17 00:00:00 2001 From: Alireza Date: Thu, 9 Jul 2026 10:54:21 -0400 Subject: [PATCH 2/2] docs(ci): clarify simulation runner needs no clock isolation; multi-core is a throughput win --- docs/ci/self-hosted-runner.md | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/docs/ci/self-hosted-runner.md b/docs/ci/self-hosted-runner.md index 0359d82..4b4bfd1 100644 --- a/docs/ci/self-hosted-runner.md +++ b/docs/ci/self-hosted-runner.md @@ -30,12 +30,29 @@ and the regression gate is trustworthy. ## Hardware hygiene (for stable numbers) -- **Dedicate the box to this label.** Register it so it only picks up the - `codspeed-bench` label and runs **one job at a time** (the default). Do not - co-locate other CI or workloads on it — contention is noise. -- Pin the CPU: disable turbo boost and, ideally, SMT/hyper-threading; set the - CPU governor to `performance`. A VM is fine if it has a pinned vCPU and is - otherwise idle. +This runner hosts the **simulation** gate only, which changes what matters: + +- **Simulation is clock- and concurrency-independent.** Cachegrind counts + instructions on a *modeled* CPU cache, per process — so turbo boost, SMT, + and the CPU governor do **not** affect the numbers, and running benches in + parallel does not corrupt them (each valgrind process models its own cache). + You do **not** need the strict single-core isolation that wall-clock + benchmarking requires. +- **Multiple cores are a throughput win.** valgrind runs ~60× slower than + native, so the job is CPU-bound; the workflow already fans out with + `lerna run bench --parallel`, which uses all available cores. More threads → + the job finishes faster, with identical instruction counts. +- **Dedicate the box to this label** and let it run **one GitHub job at a + time** (the default). This is about not co-scheduling *other* jobs on the + box mid-run, not about intra-job parallelism — that stays on. +- The only hard requirement for cross-run stability is a **fixed CPU model** + (don't migrate the box between different physical CPUs), since the modeled + cache is derived from it. + +> If the **walltime** job (`codspeed-macro`, `mode: walltime`) is ever moved +> onto a self-hosted box, this advice inverts: wall-clock needs an isolated, +> pinned core with turbo/SMT off and nothing else running. Keep walltime and +> simulation on separate runners. ## Registering the runner