test(bench): open the bench:ir window after V8's tier-up (LAB-12188)

CONTRIBUTING.md

@@ -147,14 +147,15 @@ Six harnesses measure the client.
 
 **Call shape (runs in `pnpm test`).** `packages/cachekit/test/transport/call-shape.test.ts` runs the CachekitIO backend against a local TLS fake of the SaaS. It asserts the exact requests each op sends (a wrap miss is `GET` then `PUT`, and with `writeBehind` the caller waits for the `GET` only; a locked miss is `GET`, lock, `PUT`, unlock) and the connections 100 ops use. Each extra request is a round trip, and each extra connection is a TCP and TLS handshake. So if you change one of these numbers, change the expectation in the same PR and say why. The fake mints a throwaway certificate with the `openssl` CLI, and it needs Node 22.19+ or 24.5+ for `tls.setDefaultCACertificates`. Where either is missing, the tests fail with a message that names the requirement.
 
-**Instructions per op (`pnpm --filter @cachekit-io/cachekit bench:ir`).** This counts the main-thread instructions of each hot path under callgrind: key generation through the NAPI hash (`keygen`, `keygen-1kb`) and the wasm hash (`keygen-wasm`, `keygen-wasm-1kb`), at 45 B and 1 KB of arguments, the serializer, the NAPI and wasm envelopes (1 KB, plus 40 B under the core's 64 B literals-only cutover and 64 KiB), both encrypted paths (1 KB, plus 64 KiB of incompressible bytes), and the L1 hit through `get()` (`get-l1-hit`) and through `wrap()`, once with an object among the arguments and once with the single integer argument the README benchmark passes, and the encrypted `wrap()` L1 hit at 100 B, 10 KB and 1 MB. Unlike wall-clock timing, instruction counts repeat on a busy machine. The suite pins V8's flags and counts only the measured ops. On Node 22 and 24, every path repeats within 0.01%. On Node 26, paths through the native cores repeat within about 0.3%, but `napi-encrypted` and the encrypted `wrap()` hits at 100 B and 1 MB sometimes have a run up to 1.7% off the rest, which makes them read inconclusive. Gate on Node 22 or 24. It needs Linux and valgrind. It imports `dist/` and the wasm build, so run `pnpm build` and `pnpm --filter @cachekit-io/cachekit-core-wasm build:wasm` first; the second needs the `wasm32-unknown-unknown` target, `wasm-bindgen` and `wasm-opt`, and without it pass `--only` to skip the seven wasm workloads (`wasm-envelope`, `wasm-envelope-40b`, `wasm-envelope-64kb`, `wasm-encrypted`, `wasm-encrypted-64kb`, `keygen-wasm`, `keygen-wasm-1kb`). To A/B a change, run it on both builds on the same machine:
+**Instructions per op (`pnpm --filter @cachekit-io/cachekit bench:ir`).** This counts the main-thread instructions of each hot path under callgrind: key generation through the NAPI hash (`keygen`, `keygen-1kb`) and the wasm hash (`keygen-wasm`, `keygen-wasm-1kb`), at 45 B and 1 KB of arguments, the serializer, the NAPI and wasm envelopes (1 KB, plus 40 B under the core's 64 B literals-only cutover and 64 KiB), both encrypted paths (1 KB, plus 64 KiB of incompressible bytes), and the L1 hit through `get()` (`get-l1-hit`) and through `wrap()`, once with an object among the arguments and once with the single integer argument the README benchmark passes, and the encrypted `wrap()` L1 hit at 100 B, 10 KB and 1 MB. Unlike wall-clock timing, instruction counts repeat on a busy machine. The suite pins V8's flags and counts only the measured ops. It also lowers V8's TurboFan threshold, so no optimizing compile lands in any workload's window. For all but the two 1 MB workloads, the window counts the code V8 runs in steady state. Those run so few ops that the `set()` and `wrap()` code around each 1 MB encode or decode stays in V8's interpreter for the whole run, which costs too little to show beside the encode or decode. A workload that tiers up later sets its own `warmup`; `bench/ir-workload.mjs` says how to check one with `--trace-opt`. On Node 22 and 24, every path repeats within 0.01%. On Node 26, paths through the native cores repeat within about 0.3%, but `napi-encrypted` and the encrypted `wrap()` hits at 100 B and 1 MB sometimes have a run up to 1.7% off the rest, which makes them read inconclusive. Gate on Node 22 or 24. It needs Linux and valgrind. It imports `dist/` and the wasm build, so run `pnpm build` and `pnpm --filter @cachekit-io/cachekit-core-wasm build:wasm` first; the second needs the `wasm32-unknown-unknown` target, `wasm-bindgen` and `wasm-opt`, and without it pass `--only` to skip the seven wasm workloads (`wasm-envelope`, `wasm-envelope-40b`, `wasm-envelope-64kb`, `wasm-encrypted`, `wasm-encrypted-64kb`, `keygen-wasm`, `keygen-wasm-1kb`). To A/B a change, run it on both builds on the same machine:
 
 ```bash
+git checkout <your-branch> -- packages/cachekit/bench                 # in the base checkout: your bench/, as CI does
 pnpm --filter @cachekit-io/cachekit bench:ir --save /tmp/base.json      # on the base build
 pnpm --filter @cachekit-io/cachekit bench:ir --compare /tmp/base.json   # on your build
 ```
 
-A workload more than 2% worse fails, and more than 1% worse warns, unless it sets its own `fail` threshold and warns at half of it: `plaintext-set-1mb` (10%) and `napi-encrypted` (4%) do, for the CI measurements noted beside them in `bench/ir-workload.mjs`. If the two runs come from a different node, V8 or valgrind, the compare refuses them. A run whose own repeats spread more than 0.4% (the A/A) reports inconclusive rather than passing. A compare also refuses (exit 2) two runs that measured different workloads, or two runs of the same build, and refuses to start when `--save` and `--compare` name the same file. A workload whose run fails (a valgrind crash, a missing wasm build) reads INCOMPLETE while the others still run, and any other failure (bad arguments, a missing `dist/`) stops the run; either way the exit code is 4 unless a regression (1) or a refusal (2) outranks it. Add `--wall 9` to print indicative ns per op; it is not saved, and it is never a measured saving.
+A workload more than 2% worse fails, and more than 1% worse warns, unless it sets its own `fail` threshold and warns at half of it: `plaintext-set-1mb` (10%) and `napi-encrypted` (4%) do, for the CI measurements noted beside them in `bench/ir-workload.mjs`. If the two runs come from a different node, V8 or valgrind, or from an `ir.mjs` with different V8 flags, the compare refuses them. Run one copy of `bench/` against both builds, as the CI gate does: a workload's warm-up and n are not checked, so two copies that set them differently measure different things. A run whose own repeats spread more than 0.4% (the A/A) reports inconclusive rather than passing. A compare also refuses (exit 2) two runs that measured different workloads, or two runs of the same build, and refuses to start when `--save` and `--compare` name the same file. A workload whose run fails (a valgrind crash, a missing wasm build) reads INCOMPLETE while the others still run, and any other failure (bad arguments, a missing `dist/`) stops the run; either way the exit code is 4 unless a regression (1) or a refusal (2) outranks it. Add `--wall 9` to print indicative ns per op; it is not saved, and it is never a measured saving.
 
 **The instruction-count gate (`.github/workflows/instruction-counts.yml`).** Every PR to `main` runs this compare in CI. Each job builds the PR's merge commit and its base on one GitHub-hosted runner, with the same Node and valgrind, at paths of equal length. Each tree builds with the wasm-bindgen CLI its own `Cargo.lock` locks, so a wasm-bindgen bump is measured too. Both builds run the PR's `bench/`, so a workload the PR adds is also measured on the base wherever the base can run it. A workload for an API the base lacks reads `new` and is not gated (`--new`). A workload that fails on either build reads INCOMPLETE, and so does one with no row at all, so the gate never passes on a workload it did not measure. A release-please PR is measured against its package's last release tag instead of `main` (`.github/scripts/perf-ir-base.sh`), so its comment shows each workload's drift since that release. When both builds hash alike, the job passes without measuring. `--build-hash` covers, for this package and both cores, `package.json` and every file its `files` list ships (`dist/` with its types and source maps, the NAPI loader and binary, the wasm glue JavaScript and module, and this package's README and LICENSE), plus `pnpm-lock.yaml`. So a PR that edits only one of those is measured too. A file a workload loads must therefore ship through a `files` entry, or a change to it passes unmeasured; `.github/scripts/perf-ir-hash-selftest.sh`, run in CI, checks the hash moves for each kind. Four shards split the workloads, each building both trees, balanced on the seconds one run of each workload took in CI (`bench/ir-seconds.json`). A workload missing from that file counts as the median, so add its figure from the gate's `base.json` artifact (`seconds` / `repeats`) when you add a workload. CI runs three repeats, the fewest `ir.mjs` takes, to stay well inside the job's 30 minutes. The verdicts go to one PR comment, edited in place on every push; a fork PR gets the job summary instead, because its token cannot comment.
 

packages/cachekit/bench/ir-seconds.json

@@ -1,32 +1,38 @@
 {
-  "l1-get-hit": 28,
-  "l1-set-at-capacity-1k": 37,
-  "l1-set-at-capacity-10k": 16,
-  "l1-set-at-capacity-100k": 24,
-  "plaintext-set-1kb": 42,
-  "plaintext-set-10kb": 82,
-  "plaintext-set-100kb": 203,
-  "plaintext-set-64b": 33,
-  "plaintext-set-1mb": 61,
-  "plaintext-set-containers": 92,
-  "keygen": 22,
-  "keygen-1kb": 43,
-  "keygen-wasm": 24,
-  "keygen-wasm-1kb": 40,
-  "serializer-roundtrip": 61,
-  "napi-envelope": 22,
-  "napi-envelope-40b": 12,
-  "napi-envelope-64kb": 39,
-  "wasm-envelope": 32,
-  "wasm-envelope-40b": 25,
-  "wasm-envelope-64kb": 54,
-  "napi-encrypted": 21,
-  "napi-encrypted-64kb": 66,
-  "wasm-encrypted": 21,
-  "wasm-encrypted-64kb": 92,
-  "secure-set-get": 73,
-  "wrap-l1-hit": 22,
-  "wrap-l1-hit-int": 20,
-  "wrap-l1-hit-cachekitio": 22,
-  "wrap-l1-hit-int-cachekitio": 20
+  "l1-get-hit": 36,
+  "l1-set-at-capacity-1k": 15,
+  "l1-set-at-capacity-10k": 12,
+  "l1-set-at-capacity-100k": 10,
+  "plaintext-set-1kb": 29,
+  "plaintext-set-10kb": 69,
+  "plaintext-set-100kb": 191,
+  "plaintext-set-64b": 27,
+  "plaintext-set-1mb": 86,
+  "plaintext-set-containers": 72,
+  "keygen": 17,
+  "keygen-1kb": 33,
+  "keygen-wasm": 18,
+  "keygen-wasm-1kb": 23,
+  "serializer-roundtrip": 59,
+  "napi-envelope": 23,
+  "napi-envelope-40b": 18,
+  "napi-envelope-64kb": 29,
+  "wasm-envelope": 24,
+  "wasm-envelope-40b": 17,
+  "wasm-envelope-64kb": 39,
+  "napi-encrypted": 18,
+  "napi-encrypted-64kb": 61,
+  "wasm-encrypted": 11,
+  "wasm-encrypted-64kb": 122,
+  "secure-set-get": 57,
+  "get-l1-hit": 11,
+  "wrap-l1-hit": 17,
+  "wrap-l1-hit-metrics": 12,
+  "get-l2-hit-metrics": 63,
+  "secure-wrap-l1-hit-100b": 47,
+  "secure-wrap-l1-hit-10kb": 83,
+  "secure-wrap-l1-hit-1mb": 166,
+  "wrap-l1-hit-int": 15,
+  "wrap-l1-hit-cachekitio": 34,
+  "wrap-l1-hit-int-cachekitio": 23
 }