From 2ac72a0b10c27edf89f79ec2ee5f0d3da5f2598f Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 1 Nov 2026 21:02:01 +1100 Subject: [PATCH 1/8] test(bench): open the bench:ir window after V8's tier-up V8 compiles a function with TurboFan after about 3,000 calls, and later for one called once per op, so a 2,000-op warm-up left most cheap workloads compiling inside the measured window: up to 48% of their Ir/op, moving with function size and inlining rather than per-op cost. ir.mjs now lowers the TurboFan threshold (--invocation-count-for-turbofan=150); after a 60,000-op warm-up it moves Ir/op by at most 0.32%. One loop function runs warm-up and window, so the module body no longer compiles and deoptimises in the window. Six workloads that tier up past 2,000 ops warm up 4,000; plaintext-set-100kb 1,300. l1-set-at-capacity-100k runs 20,000 ops so V8's marking of its old generation repeats. ir-seconds.json is refreshed for all workloads. --- CONTRIBUTING.md | 2 +- packages/cachekit/bench/ir-seconds.json | 61 +++++++++++--------- packages/cachekit/bench/ir-workload.mjs | 76 +++++++++++++++++-------- packages/cachekit/bench/ir.mjs | 14 ++++- 4 files changed, 97 insertions(+), 56 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4d5879f3..f1a2a00c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -147,7 +147,7 @@ Six harnesses measure the client. **Call shape (runs in `pnpm test`).** `packages/cachekit/test/transport/call-shape.test.ts` runs the CachekitIO backend against a local TLS fake of the SaaS. It asserts the exact requests each op sends (a wrap miss is `GET` then `PUT`, and with `writeBehind` the caller waits for the `GET` only; a locked miss is `GET`, lock, `PUT`, unlock) and the connections 100 ops use. Each extra request is a round trip, and each extra connection is a TCP and TLS handshake. So if you change one of these numbers, change the expectation in the same PR and say why. The fake mints a throwaway certificate with the `openssl` CLI, and it needs Node 22.19+ or 24.5+ for `tls.setDefaultCACertificates`. Where either is missing, the tests fail with a message that names the requirement. -**Instructions per op (`pnpm --filter @cachekit-io/cachekit bench:ir`).** This counts the main-thread instructions of each hot path under callgrind: key generation through the NAPI hash (`keygen`, `keygen-1kb`) and the wasm hash (`keygen-wasm`, `keygen-wasm-1kb`), at 45 B and 1 KB of arguments, the serializer, the NAPI and wasm envelopes (1 KB, plus 40 B under the core's 64 B literals-only cutover and 64 KiB), both encrypted paths (1 KB, plus 64 KiB of incompressible bytes), and the L1 hit through `get()` (`get-l1-hit`) and through `wrap()`, once with an object among the arguments and once with the single integer argument the README benchmark passes, and the encrypted `wrap()` L1 hit at 100 B, 10 KB and 1 MB. Unlike wall-clock timing, instruction counts repeat on a busy machine. The suite pins V8's flags and counts only the measured ops. On Node 22 and 24, every path repeats within 0.01%. On Node 26, paths through the native cores repeat within about 0.3%, but `napi-encrypted` and the encrypted `wrap()` hits at 100 B and 1 MB sometimes have a run up to 1.7% off the rest, which makes them read inconclusive. Gate on Node 22 or 24. It needs Linux and valgrind. It imports `dist/` and the wasm build, so run `pnpm build` and `pnpm --filter @cachekit-io/cachekit-core-wasm build:wasm` first; the second needs the `wasm32-unknown-unknown` target, `wasm-bindgen` and `wasm-opt`, and without it pass `--only` to skip the seven wasm workloads (`wasm-envelope`, `wasm-envelope-40b`, `wasm-envelope-64kb`, `wasm-encrypted`, `wasm-encrypted-64kb`, `keygen-wasm`, `keygen-wasm-1kb`). To A/B a change, run it on both builds on the same machine: +**Instructions per op (`pnpm --filter @cachekit-io/cachekit bench:ir`).** This counts the main-thread instructions of each hot path under callgrind: key generation through the NAPI hash (`keygen`, `keygen-1kb`) and the wasm hash (`keygen-wasm`, `keygen-wasm-1kb`), at 45 B and 1 KB of arguments, the serializer, the NAPI and wasm envelopes (1 KB, plus 40 B under the core's 64 B literals-only cutover and 64 KiB), both encrypted paths (1 KB, plus 64 KiB of incompressible bytes), and the L1 hit through `get()` (`get-l1-hit`) and through `wrap()`, once with an object among the arguments and once with the single integer argument the README benchmark passes, and the encrypted `wrap()` L1 hit at 100 B, 10 KB and 1 MB. Unlike wall-clock timing, instruction counts repeat on a busy machine. The suite pins V8's flags and counts only the measured ops. It also lowers V8's TurboFan threshold, so each workload's warm-up runs past V8's tier-up: the window counts the code V8 runs in steady state, not V8 compiling it. A workload that tiers up later sets its own `warmup`; `bench/ir-workload.mjs` says how to check one with `--trace-opt`. On Node 22 and 24, every path repeats within 0.01%. On Node 26, paths through the native cores repeat within about 0.3%, but `napi-encrypted` and the encrypted `wrap()` hits at 100 B and 1 MB sometimes have a run up to 1.7% off the rest, which makes them read inconclusive. Gate on Node 22 or 24. It needs Linux and valgrind. It imports `dist/` and the wasm build, so run `pnpm build` and `pnpm --filter @cachekit-io/cachekit-core-wasm build:wasm` first; the second needs the `wasm32-unknown-unknown` target, `wasm-bindgen` and `wasm-opt`, and without it pass `--only` to skip the seven wasm workloads (`wasm-envelope`, `wasm-envelope-40b`, `wasm-envelope-64kb`, `wasm-encrypted`, `wasm-encrypted-64kb`, `keygen-wasm`, `keygen-wasm-1kb`). To A/B a change, run it on both builds on the same machine: ```bash pnpm --filter @cachekit-io/cachekit bench:ir --save /tmp/base.json # on the base build diff --git a/packages/cachekit/bench/ir-seconds.json b/packages/cachekit/bench/ir-seconds.json index 51e929fe..6daad270 100644 --- a/packages/cachekit/bench/ir-seconds.json +++ b/packages/cachekit/bench/ir-seconds.json @@ -1,30 +1,35 @@ { - "l1-get-hit": 28, - "l1-set-at-capacity-1k": 37, - "l1-set-at-capacity-10k": 16, - "l1-set-at-capacity-100k": 24, - "plaintext-set-1kb": 42, - "plaintext-set-10kb": 82, - "plaintext-set-100kb": 203, - "plaintext-set-64b": 33, - "plaintext-set-1mb": 61, - "plaintext-set-containers": 92, - "keygen": 22, - "keygen-1kb": 43, - "keygen-wasm": 24, - "keygen-wasm-1kb": 40, - "serializer-roundtrip": 61, - "napi-envelope": 22, - "napi-envelope-40b": 12, - "napi-envelope-64kb": 39, - "wasm-envelope": 32, - "wasm-envelope-40b": 25, - "wasm-envelope-64kb": 54, - "napi-encrypted": 21, - "napi-encrypted-64kb": 66, - "wasm-encrypted": 21, - "wasm-encrypted-64kb": 92, - "secure-set-get": 73, - "wrap-l1-hit": 22, - "wrap-l1-hit-int": 20 + "l1-get-hit": 36, + "l1-set-at-capacity-1k": 15, + "l1-set-at-capacity-10k": 12, + "l1-set-at-capacity-100k": 10, + "plaintext-set-1kb": 29, + "plaintext-set-10kb": 69, + "plaintext-set-100kb": 191, + "plaintext-set-64b": 27, + "plaintext-set-1mb": 86, + "plaintext-set-containers": 72, + "keygen": 17, + "keygen-1kb": 33, + "keygen-wasm": 18, + "keygen-wasm-1kb": 23, + "serializer-roundtrip": 59, + "napi-envelope": 23, + "napi-envelope-40b": 18, + "napi-envelope-64kb": 29, + "wasm-envelope": 24, + "wasm-envelope-40b": 17, + "wasm-envelope-64kb": 39, + "napi-encrypted": 18, + "napi-encrypted-64kb": 61, + "wasm-encrypted": 11, + "wasm-encrypted-64kb": 122, + "secure-set-get": 57, + "get-l1-hit": 11, + "wrap-l1-hit": 17, + "wrap-l1-hit-metrics": 12, + "secure-wrap-l1-hit-100b": 47, + "secure-wrap-l1-hit-10kb": 83, + "secure-wrap-l1-hit-1mb": 166, + "wrap-l1-hit-int": 15 } diff --git a/packages/cachekit/bench/ir-workload.mjs b/packages/cachekit/bench/ir-workload.mjs index 2aa77416..2198cb01 100644 --- a/packages/cachekit/bench/ir-workload.mjs +++ b/packages/cachekit/bench/ir-workload.mjs @@ -2,10 +2,11 @@ // // node [v8 flags] bench/ir-workload.mjs [--wall] // -// A run loads only what its workload needs, warms the op past tier-up, then -// runs it n times between two process.cpuUsage() calls. ir.mjs has callgrind -// dump its counters on entry to node::CPUUsage, so the counted window holds -// the n ops and nothing else: module load, setup and warm-up stay out. +// A run loads only what its workload needs, warms the op until V8 has +// finished compiling it, then runs it n times between two process.cpuUsage() +// calls. ir.mjs has callgrind dump its counters on entry to node::CPUUsage, so +// the counted window holds the n ops and nothing else: module load, setup, +// warm-up and V8's optimizing compiles stay out. // --wall prints the steady-state ns per op of the n iterations instead // (indicative only). Imports the built package: run `pnpm build` first. import { existsSync, readFileSync } from 'node:fs'; @@ -73,8 +74,12 @@ async function incompressible() { } let sink = 0; -// Ops run before the window opens, past V8's tier-up. A workload whose op -// is long enough to tier up sooner sets its own `warmup`. +// Ops run before the window opens. Under ir.mjs's V8_FLAGS every function a +// cheap op runs has finished its Maglev and TurboFan compiles within 2,000 +// ops. A workload whose op tiers up later, or runs too long for 2,000, sets +// its own `warmup`. To check one, run it under V8_FLAGS with --trace-opt +// --trace-deopt and print a line at each process.cpuUsage() call: no compile +// or deopt may land between the two. const WARMUP = 2000; async function encryptedOp(manager, storage, bytes) { @@ -148,8 +153,9 @@ const plaintextSet = (rows, n, warmup) => plaintextSetOf(rowsOf(rows), n, warmup * build from before the key hash moved into the bindings has no * serialization/blake2b.js and hashes keys in JS: the base side of a compare. */ -const keygen = (args, n, wasmHash = false) => ({ +const keygen = (args, n, wasmHash = false, warmup) => ({ n, + warmup, setup: async () => { const hashing = existsSync(new URL('serialization/blake2b.js', dist)) ? await load('serialization/blake2b.js') @@ -251,17 +257,22 @@ export const WORKLOADS = { }, 'l1-set-at-capacity-1k': l1AtCapacity(1_000, 200_000), 'l1-set-at-capacity-10k': l1AtCapacity(10_000, 4_000), - 'l1-set-at-capacity-100k': l1AtCapacity(100_000, 2_000), + // 20,000 ops: 100,000 live entries keep V8 marking the old generation + // through the window, and over 2,000 ops the share of that marking in the + // window differed between runs by up to 1.9%; over 20,000 it repeats within + // 0.01%. + 'l1-set-at-capacity-100k': l1AtCapacity(100_000, 20_000), 'plaintext-set-1kb': plaintextSet(32, 5_000), 'plaintext-set-10kb': plaintextSet(320, 1_000), - // 200 warm-up ops, not 2,000: each op encodes 3,200 rows, so V8 has tiered - // up long before, and the default made one process run ~9 minutes in CI. - // The shorter warm-up moved its Ir/op by +0.4% in CI. - 'plaintext-set-100kb': plaintextSet(3_200, 1_000, 200), + // 1,300 warm-up ops, not 2,000: V8 has finished compiling the set() path + // around it by ~960 ops, and 2,000 made one process run ~9 minutes in CI. + 'plaintext-set-100kb': plaintextSet(3_200, 1_000, 1_300), // the ends of the object-heavy write range: one 64 B object, and 1,005,035 B // in 96 groups of 320 rows, so no collection nears the 10,000-element bound - // and the whole stays under the 1 MiB encode limit. A 1 MB op runs long - // enough under callgrind that 20 warm-up ops are past tier-up. + // and the whole stays under the 1 MiB encode limit. 20 warm-up ops for 1 MB: + // the encoders it calls per row reach TurboFan in the first op, and the + // set() path around them runs 40 times, too few for V8 to compile it, so + // the window holds no compile. 'plaintext-set-64b': plaintextSetOf( { id: 7, name: 'item-7', score: 10.5, ok: true, tags: ['a', 'b'], owner: 'user-424242' }, 20_000 @@ -293,9 +304,10 @@ export const WORKLOADS = { // 1 KB of arguments, where the hash's share of the cost grows with the bytes 'keygen-1kb': keygen([VALUE], 5_000), // the same on Workers, where the wasm binding hashes keys once its module is - // instantiated - 'keygen-wasm': keygen(ARGS, 20_000, true), - 'keygen-wasm-1kb': keygen([VALUE], 5_000, true), + // instantiated. 4,000 warm-up ops: the glue reads each hash back through + // Node's TextDecoder decode, which V8 compiles with TurboFan at ~2,400 ops. + 'keygen-wasm': keygen(ARGS, 20_000, true, 4_000), + 'keygen-wasm-1kb': keygen([VALUE], 5_000, true, 4_000), // set encodes (normalize + msgpack, in one walk), an L2 hit decodes // (depth bounds + msgpack) 'serializer-roundtrip': { @@ -327,9 +339,12 @@ export const WORKLOADS = { 'wasm-encrypted': wasmEncrypted(encoded, 5_000), 'wasm-encrypted-64kb': wasmEncrypted(incompressible, 50), // a secure cache's L2 round trip on Node, L1 off: set + get, each running the - // AAD-size pre-flight (validateKey) and then encrypt or decrypt + // AAD-size pre-flight (validateKey) and then encrypt or decrypt. 4,000 + // warm-up ops: the get reads the envelope header through Node's TextDecoder, + // whose decode V8 compiles with TurboFan at ~2,400 ops. 'secure-set-get': { n: 5_000, + warmup: 4_000, setup: async () => { const { createCache } = await load('index.js'); const cache = createCache({ @@ -345,9 +360,11 @@ export const WORKLOADS = { }, }, // get() served from a plaintext L1, which holds the decoded value: the - // lookup and the promises around it (l1-get-hit times L1Cache.get alone) + // lookup and the promises around it (l1-get-hit times L1Cache.get alone). + // 4,000 warm-up ops: V8 compiles getEntry with TurboFan at ~2,200. 'get-l1-hit': { n: 20_000, + warmup: 4_000, setup: async () => { const { createCache } = await load('index.js'); const cache = createCache({ backend: new MemoryBackend(), metrics: false }); @@ -402,9 +419,11 @@ export const WORKLOADS = { // an encrypted wrap() L1 hit at 100 B, 10 KB and 1 MB of msgpack (3, 320 // and 96 x 320 rows). On Node 22 and 24 each repeats within 0.01%. On Node // 26 the 100 B and 1 MB hits sometimes have a run up to 1.7% off the rest, - // and a 4x larger n did not remove it: gate on Node 22 or 24. - 'secure-wrap-l1-hit-100b': secureWrapHit(rowsOf(3), 40_000), - 'secure-wrap-l1-hit-10kb': secureWrapHit(rowsOf(320), 500), + // and a 4x larger n did not remove it: gate on Node 22 or 24. 4,000 warm-up + // ops at 100 B and 10 KB: V8 compiles normalize with TurboFan at ~2,200 ops, + // and the encoder deoptimises and is compiled again up to ~2,500. + 'secure-wrap-l1-hit-100b': secureWrapHit(rowsOf(3), 40_000, 4_000), + 'secure-wrap-l1-hit-10kb': secureWrapHit(rowsOf(320), 500, 4_000), 'secure-wrap-l1-hit-1mb': secureWrapHit( { groups: Array.from({ length: 96 }, () => rowsOf(320)) }, 50, @@ -427,6 +446,14 @@ export const WORKLOADS = { }, }; +// The warm-up and the window run their ops through this one function, so it +// is as warm as the op when the window opens. A loop in the module body +// instead had V8 compile the module body inside the window and deoptimise it +// at the window's end. +async function repeat(op, times) { + for (let i = 0; i < times; i++) await op(); +} + if (process.argv[1] === fileURLToPath(import.meta.url)) { const [name, nArg] = process.argv.slice(2); const n = Number(nArg); @@ -437,11 +464,10 @@ if (process.argv[1] === fileURLToPath(import.meta.url)) { throw new Error(`n must be a non-negative integer, got "${nArg}"`); const op = await WORKLOADS[name].setup(n); - const warmup = WORKLOADS[name].warmup ?? WARMUP; - for (let i = 0; i < warmup; i++) await op(); + await repeat(op, WORKLOADS[name].warmup ?? WARMUP); process.cpuUsage(); // window start: callgrind dump 1 const t0 = process.hrtime.bigint(); - for (let i = 0; i < n; i++) await op(); + await repeat(op, n); const ns = Number(process.hrtime.bigint() - t0); process.cpuUsage(); // window end: callgrind dump 2 if (process.argv.includes('--wall')) console.log(JSON.stringify({ nsPerOp: n ? ns / n : 0 })); diff --git a/packages/cachekit/bench/ir.mjs b/packages/cachekit/bench/ir.mjs index 8c4186ad..bca172db 100644 --- a/packages/cachekit/bench/ir.mjs +++ b/packages/cachekit/bench/ir.mjs @@ -6,14 +6,23 @@ // node bench/ir.mjs --build-hash // // Wall-clock microbenchmarks on a shared or hosted machine cannot resolve the -// 1-2% changes that matter here; instruction counts can, once three sources of -// run-to-run noise are removed: +// 1-2% changes that matter here; instruction counts can, once these sources of +// noise are removed: // - V8: default flags vary main-thread Ir 2-4x between identical runs // (background compilation and GC). V8_FLAGS pins them; --no-liftoff and // --predictable-gc-schedule are what make the wasm paths repeat. // - The fixed part: module load and warm-up vary by 1-3% of a whole process. // So Ir/op = Ir(window) / n, where the window is the n ops of one workload, // bracketed by ir-workload.mjs; nothing else is counted. +// - Tier-up, which moves Ir between builds rather than runs: V8 compiles a +// function with TurboFan after about 3,000 calls, and some functions called +// once per op got there only after 45,000 ops. A compile inside the window +// is a one-time cost divided by n that moves with a function's size and +// V8's inlining, not with the op's cost: it was up to 48% of a workload's +// Ir/op. --invocation-count-for-turbofan lowers the threshold so each +// workload's warm-up runs past it. It barely moves the steady state: after +// a 60,000-op warm-up, six workloads read within 0.32% of the default +// threshold's count. // On Node 22 and 24 every workload then repeats within 0.01%. On Node 26 the // workloads that cross into the NAPI or wasm core keep a residual of // 0.1-0.3%, and napi-encrypted an occasional run about 1% low; neither ASLR @@ -77,6 +86,7 @@ const V8_FLAGS = [ '--random-seed=1', '--no-liftoff', '--predictable-gc-schedule', + '--invocation-count-for-turbofan=150', ]; const N = Object.fromEntries(Object.entries(WORKLOADS).map(([name, w]) => [name, w.n])); const AA_LIMIT = 0.004; From d823842cfdb83fdf163108222597ec480f61783b Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 1 Nov 2026 21:12:54 +1100 Subject: [PATCH 2/8] docs(bench): qualify the steady-state claim for the 1 MB workloads and --wall --- CONTRIBUTING.md | 2 +- packages/cachekit/bench/ir-workload.mjs | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f1a2a00c..bfb28b4f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -147,7 +147,7 @@ Six harnesses measure the client. **Call shape (runs in `pnpm test`).** `packages/cachekit/test/transport/call-shape.test.ts` runs the CachekitIO backend against a local TLS fake of the SaaS. It asserts the exact requests each op sends (a wrap miss is `GET` then `PUT`, and with `writeBehind` the caller waits for the `GET` only; a locked miss is `GET`, lock, `PUT`, unlock) and the connections 100 ops use. Each extra request is a round trip, and each extra connection is a TCP and TLS handshake. So if you change one of these numbers, change the expectation in the same PR and say why. The fake mints a throwaway certificate with the `openssl` CLI, and it needs Node 22.19+ or 24.5+ for `tls.setDefaultCACertificates`. Where either is missing, the tests fail with a message that names the requirement. -**Instructions per op (`pnpm --filter @cachekit-io/cachekit bench:ir`).** This counts the main-thread instructions of each hot path under callgrind: key generation through the NAPI hash (`keygen`, `keygen-1kb`) and the wasm hash (`keygen-wasm`, `keygen-wasm-1kb`), at 45 B and 1 KB of arguments, the serializer, the NAPI and wasm envelopes (1 KB, plus 40 B under the core's 64 B literals-only cutover and 64 KiB), both encrypted paths (1 KB, plus 64 KiB of incompressible bytes), and the L1 hit through `get()` (`get-l1-hit`) and through `wrap()`, once with an object among the arguments and once with the single integer argument the README benchmark passes, and the encrypted `wrap()` L1 hit at 100 B, 10 KB and 1 MB. Unlike wall-clock timing, instruction counts repeat on a busy machine. The suite pins V8's flags and counts only the measured ops. It also lowers V8's TurboFan threshold, so each workload's warm-up runs past V8's tier-up: the window counts the code V8 runs in steady state, not V8 compiling it. A workload that tiers up later sets its own `warmup`; `bench/ir-workload.mjs` says how to check one with `--trace-opt`. On Node 22 and 24, every path repeats within 0.01%. On Node 26, paths through the native cores repeat within about 0.3%, but `napi-encrypted` and the encrypted `wrap()` hits at 100 B and 1 MB sometimes have a run up to 1.7% off the rest, which makes them read inconclusive. Gate on Node 22 or 24. It needs Linux and valgrind. It imports `dist/` and the wasm build, so run `pnpm build` and `pnpm --filter @cachekit-io/cachekit-core-wasm build:wasm` first; the second needs the `wasm32-unknown-unknown` target, `wasm-bindgen` and `wasm-opt`, and without it pass `--only` to skip the seven wasm workloads (`wasm-envelope`, `wasm-envelope-40b`, `wasm-envelope-64kb`, `wasm-encrypted`, `wasm-encrypted-64kb`, `keygen-wasm`, `keygen-wasm-1kb`). To A/B a change, run it on both builds on the same machine: +**Instructions per op (`pnpm --filter @cachekit-io/cachekit bench:ir`).** This counts the main-thread instructions of each hot path under callgrind: key generation through the NAPI hash (`keygen`, `keygen-1kb`) and the wasm hash (`keygen-wasm`, `keygen-wasm-1kb`), at 45 B and 1 KB of arguments, the serializer, the NAPI and wasm envelopes (1 KB, plus 40 B under the core's 64 B literals-only cutover and 64 KiB), both encrypted paths (1 KB, plus 64 KiB of incompressible bytes), and the L1 hit through `get()` (`get-l1-hit`) and through `wrap()`, once with an object among the arguments and once with the single integer argument the README benchmark passes, and the encrypted `wrap()` L1 hit at 100 B, 10 KB and 1 MB. Unlike wall-clock timing, instruction counts repeat on a busy machine. The suite pins V8's flags and counts only the measured ops. It also lowers V8's TurboFan threshold, so no optimizing compile lands in any workload's window. For all but the two 1 MB workloads, the window counts the code V8 runs in steady state. Those run so few ops that the `set()` and `wrap()` code around each 1 MB encode or decode stays in V8's interpreter for the whole run, which costs too little to show beside the encode or decode. A workload that tiers up later sets its own `warmup`; `bench/ir-workload.mjs` says how to check one with `--trace-opt`. On Node 22 and 24, every path repeats within 0.01%. On Node 26, paths through the native cores repeat within about 0.3%, but `napi-encrypted` and the encrypted `wrap()` hits at 100 B and 1 MB sometimes have a run up to 1.7% off the rest, which makes them read inconclusive. Gate on Node 22 or 24. It needs Linux and valgrind. It imports `dist/` and the wasm build, so run `pnpm build` and `pnpm --filter @cachekit-io/cachekit-core-wasm build:wasm` first; the second needs the `wasm32-unknown-unknown` target, `wasm-bindgen` and `wasm-opt`, and without it pass `--only` to skip the seven wasm workloads (`wasm-envelope`, `wasm-envelope-40b`, `wasm-envelope-64kb`, `wasm-encrypted`, `wasm-encrypted-64kb`, `keygen-wasm`, `keygen-wasm-1kb`). To A/B a change, run it on both builds on the same machine: ```bash pnpm --filter @cachekit-io/cachekit bench:ir --save /tmp/base.json # on the base build diff --git a/packages/cachekit/bench/ir-workload.mjs b/packages/cachekit/bench/ir-workload.mjs index 2198cb01..2ad966d0 100644 --- a/packages/cachekit/bench/ir-workload.mjs +++ b/packages/cachekit/bench/ir-workload.mjs @@ -7,8 +7,9 @@ // calls. ir.mjs has callgrind dump its counters on entry to node::CPUUsage, so // the counted window holds the n ops and nothing else: module load, setup, // warm-up and V8's optimizing compiles stay out. -// --wall prints the steady-state ns per op of the n iterations instead -// (indicative only). Imports the built package: run `pnpm build` first. +// --wall prints the ns per op of the n iterations instead (indicative only: +// ir.mjs runs it without V8_FLAGS, so V8 may still be compiling the op in +// the background). Imports the built package: run `pnpm build` first. import { existsSync, readFileSync } from 'node:fs'; import { createRequire } from 'node:module'; import { fileURLToPath } from 'node:url'; From cec88d8d693b0a1379303b88a2f883715c0a624e Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 1 Nov 2026 21:17:42 +1100 Subject: [PATCH 3/8] docs(contributing): run one copy of bench/ against both builds --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bfb28b4f..27e9a293 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -154,7 +154,7 @@ pnpm --filter @cachekit-io/cachekit bench:ir --save /tmp/base.json # on the pnpm --filter @cachekit-io/cachekit bench:ir --compare /tmp/base.json # on your build ``` -A workload more than 2% worse fails, and more than 1% worse warns, unless it sets its own `fail` threshold and warns at half of it: `plaintext-set-1mb` (10%) and `napi-encrypted` (4%) do, for the CI measurements noted beside them in `bench/ir-workload.mjs`. If the two runs come from a different node, V8 or valgrind, the compare refuses them. A run whose own repeats spread more than 0.4% (the A/A) reports inconclusive rather than passing. A compare also refuses (exit 2) two runs that measured different workloads, or two runs of the same build, and refuses to start when `--save` and `--compare` name the same file. A workload whose run fails (a valgrind crash, a missing wasm build) reads INCOMPLETE while the others still run, and any other failure (bad arguments, a missing `dist/`) stops the run; either way the exit code is 4 unless a regression (1) or a refusal (2) outranks it. Add `--wall 9` to print indicative ns per op; it is not saved, and it is never a measured saving. +A workload more than 2% worse fails, and more than 1% worse warns, unless it sets its own `fail` threshold and warns at half of it: `plaintext-set-1mb` (10%) and `napi-encrypted` (4%) do, for the CI measurements noted beside them in `bench/ir-workload.mjs`. If the two runs come from a different node, V8 or valgrind, or from an `ir.mjs` with different V8 flags, the compare refuses them. Run one copy of `bench/` against both builds, as the CI gate does: a workload's warm-up and n are not checked, so two copies that set them differently measure different things. A run whose own repeats spread more than 0.4% (the A/A) reports inconclusive rather than passing. A compare also refuses (exit 2) two runs that measured different workloads, or two runs of the same build, and refuses to start when `--save` and `--compare` name the same file. A workload whose run fails (a valgrind crash, a missing wasm build) reads INCOMPLETE while the others still run, and any other failure (bad arguments, a missing `dist/`) stops the run; either way the exit code is 4 unless a regression (1) or a refusal (2) outranks it. Add `--wall 9` to print indicative ns per op; it is not saved, and it is never a measured saving. **The instruction-count gate (`.github/workflows/instruction-counts.yml`).** Every PR to `main` runs this compare in CI. Each job builds the PR's merge commit and its base on one GitHub-hosted runner, with the same Node and valgrind, at paths of equal length. Each tree builds with the wasm-bindgen CLI its own `Cargo.lock` locks, so a wasm-bindgen bump is measured too. Both builds run the PR's `bench/`, so a workload the PR adds is also measured on the base wherever the base can run it. A workload for an API the base lacks reads `new` and is not gated (`--new`). A workload that fails on either build reads INCOMPLETE, and so does one with no row at all, so the gate never passes on a workload it did not measure. A release-please PR is measured against its package's last release tag instead of `main` (`.github/scripts/perf-ir-base.sh`), so its comment shows each workload's drift since that release. When both builds hash alike, the job passes without measuring. `--build-hash` covers, for this package and both cores, `package.json` and every file its `files` list ships (`dist/` with its types and source maps, the NAPI loader and binary, the wasm glue JavaScript and module, and this package's README and LICENSE), plus `pnpm-lock.yaml`. So a PR that edits only one of those is measured too. A file a workload loads must therefore ship through a `files` entry, or a change to it passes unmeasured; `.github/scripts/perf-ir-hash-selftest.sh`, run in CI, checks the hash moves for each kind. Four shards split the workloads, each building both trees, balanced on the seconds one run of each workload took in CI (`bench/ir-seconds.json`). A workload missing from that file counts as the median, so add its figure from the gate's `base.json` artifact (`seconds` / `repeats`) when you add a workload. CI runs three repeats, the fewest `ir.mjs` takes, to stay well inside the job's 30 minutes. The verdicts go to one PR comment, edited in place on every push; a fork PR gets the job summary instead, because its token cannot comment. From 120fa568a744d7fb6d610d9c66936c211b983df0 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 1 Nov 2026 23:38:33 +1100 Subject: [PATCH 4/8] docs(contributing): show copying bench/ into the base checkout for an A/B --- CONTRIBUTING.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 27e9a293..64e8af90 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -150,6 +150,7 @@ Six harnesses measure the client. **Instructions per op (`pnpm --filter @cachekit-io/cachekit bench:ir`).** This counts the main-thread instructions of each hot path under callgrind: key generation through the NAPI hash (`keygen`, `keygen-1kb`) and the wasm hash (`keygen-wasm`, `keygen-wasm-1kb`), at 45 B and 1 KB of arguments, the serializer, the NAPI and wasm envelopes (1 KB, plus 40 B under the core's 64 B literals-only cutover and 64 KiB), both encrypted paths (1 KB, plus 64 KiB of incompressible bytes), and the L1 hit through `get()` (`get-l1-hit`) and through `wrap()`, once with an object among the arguments and once with the single integer argument the README benchmark passes, and the encrypted `wrap()` L1 hit at 100 B, 10 KB and 1 MB. Unlike wall-clock timing, instruction counts repeat on a busy machine. The suite pins V8's flags and counts only the measured ops. It also lowers V8's TurboFan threshold, so no optimizing compile lands in any workload's window. For all but the two 1 MB workloads, the window counts the code V8 runs in steady state. Those run so few ops that the `set()` and `wrap()` code around each 1 MB encode or decode stays in V8's interpreter for the whole run, which costs too little to show beside the encode or decode. A workload that tiers up later sets its own `warmup`; `bench/ir-workload.mjs` says how to check one with `--trace-opt`. On Node 22 and 24, every path repeats within 0.01%. On Node 26, paths through the native cores repeat within about 0.3%, but `napi-encrypted` and the encrypted `wrap()` hits at 100 B and 1 MB sometimes have a run up to 1.7% off the rest, which makes them read inconclusive. Gate on Node 22 or 24. It needs Linux and valgrind. It imports `dist/` and the wasm build, so run `pnpm build` and `pnpm --filter @cachekit-io/cachekit-core-wasm build:wasm` first; the second needs the `wasm32-unknown-unknown` target, `wasm-bindgen` and `wasm-opt`, and without it pass `--only` to skip the seven wasm workloads (`wasm-envelope`, `wasm-envelope-40b`, `wasm-envelope-64kb`, `wasm-encrypted`, `wasm-encrypted-64kb`, `keygen-wasm`, `keygen-wasm-1kb`). To A/B a change, run it on both builds on the same machine: ```bash +git checkout -- packages/cachekit/bench # in the base checkout: your bench/, as CI does pnpm --filter @cachekit-io/cachekit bench:ir --save /tmp/base.json # on the base build pnpm --filter @cachekit-io/cachekit bench:ir --compare /tmp/base.json # on your build ``` From d5bd5040ffe978f2cb50cdba73818bac18c9f966 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 1 Nov 2026 14:27:24 +0000 Subject: [PATCH 5/8] test(bench): warm two workloads from main past their tier-up Two workloads that main added after this branch forked did not yet meet its rule that no optimizing compile lands in a window: - wrap-l1-hit-int-cachekitio: on Node 24, V8 compiles the wrap() closure with TurboFan at ~2,050 ops, just after the default 2,000-op warm-up. That compile was 43% of the workload's reading (callgrind, Node 24.22.1: 4,408 Ir/op at 2,000 warm-up ops, 2,533 at 4,000). It now warms 4,000 ops. - get-l2-hit-metrics: its 30,000-op warm-up and the "~40 compiles at 2,000" note predate the lowered TurboFan threshold. Under ir.mjs's flags the only late compile is Node's TextDecoder decode, at ~2,400 ops on Node 24, so it now warms 4,000 ops like secure-set-get. Ir/op moved +0.08% (263,208 to 263,406). Traced with --trace-opt --trace-deopt and a marker at each process.cpuUsage() call: no compile or deopt lands in any of the 36 windows on Node 22 or 24. (l1-get-hit's window still has a GC mark L1Cache.set's code for a lazy deopt, as before this commit; set never runs in that window.) Both workloads repeat within 0.02% (A/A). --- packages/cachekit/bench/ir-workload.mjs | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/packages/cachekit/bench/ir-workload.mjs b/packages/cachekit/bench/ir-workload.mjs index f3b1f9a9..012d9ffb 100644 --- a/packages/cachekit/bench/ir-workload.mjs +++ b/packages/cachekit/bench/ir-workload.mjs @@ -430,9 +430,9 @@ export const WORKLOADS = { // duration histogram, breaker gauge) around an unpack and a decode 'get-l2-hit-metrics': { n: 10_000, - // 2,000 ops leave ~40 TurboFan compiles inside the window (node --trace-opt - // with markers at the window); 30,000 leave none on Node 22. - warmup: 30_000, + // 4,000 warm-up ops: the hit reads through Node's TextDecoder, whose + // decode V8 compiles with TurboFan at ~2,400 ops on Node 24. + warmup: 4_000, setup: async () => { const { createCache } = await load('index.js'); const { Registry } = await import('prom-client'); @@ -491,9 +491,11 @@ export const WORKLOADS = { }, }, // wrap-l1-hit-int on the same CachekitIO backend: the memoised hit, where - // the timing draw is the largest share of a hit + // the timing draw is the largest share of a hit. 4,000 warm-up ops: on Node + // 24 V8 compiles the wrap() closure with TurboFan at ~2,050 ops. 'wrap-l1-hit-int-cachekitio': { n: 20_000, + warmup: 4_000, setup: async () => { stubCachekitIO(); const { createCache } = await load('index.js'); From ae9839ff90633b6ee679da04a0c9f847f76ee8d3 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 1 Nov 2026 15:34:02 +0000 Subject: [PATCH 6/8] test(bench): warm the CachekitIO workloads past a timed call at any RNG phase The cache times 1 call in 1,024 into its L1 latency report, drawn by a sampler seeded from Math.random. One extra draw before setup (a single Math.random() at a build's module load) moved the first timed call past V8's compile of the op: the op deoptimised and compiled again inside the window, and wrap-l1-hit-int-cachekitio read +85% with no hot-path change. warmTimed now runs each CachekitIO op on its window path, past V8's lazy feedback allocation, for 8 x oneIn calls and on until one is timed, before the warm-up opens the window. The count is fixed, so the scavenges in the window do not move with the sampler. Each warm-up is twice the longest recompile measured after a late timed call. ir-seconds.json gains the re-timed rows for the two workloads and get-l2-hit-metrics. --- packages/cachekit/bench/ir-seconds.json | 5 +- packages/cachekit/bench/ir-workload.mjs | 65 ++++++++++++++++++++++--- 2 files changed, 60 insertions(+), 10 deletions(-) diff --git a/packages/cachekit/bench/ir-seconds.json b/packages/cachekit/bench/ir-seconds.json index 4dad8488..c78a5d1d 100644 --- a/packages/cachekit/bench/ir-seconds.json +++ b/packages/cachekit/bench/ir-seconds.json @@ -28,10 +28,11 @@ "get-l1-hit": 11, "wrap-l1-hit": 17, "wrap-l1-hit-metrics": 12, + "get-l2-hit-metrics": 68, "secure-wrap-l1-hit-100b": 47, "secure-wrap-l1-hit-10kb": 83, "secure-wrap-l1-hit-1mb": 166, "wrap-l1-hit-int": 15, - "wrap-l1-hit-cachekitio": 22, - "wrap-l1-hit-int-cachekitio": 20 + "wrap-l1-hit-cachekitio": 30, + "wrap-l1-hit-int-cachekitio": 24 } diff --git a/packages/cachekit/bench/ir-workload.mjs b/packages/cachekit/bench/ir-workload.mjs index 012d9ffb..9ba22699 100644 --- a/packages/cachekit/bench/ir-workload.mjs +++ b/packages/cachekit/bench/ir-workload.mjs @@ -42,6 +42,42 @@ function stubCachekitIO() { globalThis.fetch = async (_url, init) => new globalThis.Response(null, { status: init?.method === 'GET' ? 404 : 200 }); } + +/** + * Warms op until the cache's sampler has timed one of its calls into the L1 + * latency report, so the warm-up that follows tiers the op up with the timed + * path in its type feedback. Which calls are timed follows the sampler, which + * is seeded from Math.random: one more draw anywhere before the cache is made + * (a build's module load included) moves them. Left to that, the first timed + * call can land after V8 has compiled the op, which then deoptimises and + * compiles again late in the warm-up or inside the window. + * + * Call it once op takes the path it takes in the window: a call timed on + * another path (the first hit of a memoised key) gives the window's path no + * feedback. Nor does a call before V8 has given each function its feedback + * vector, at about its 8th call (--invocation-count-for-feedback-allocation): + * so 64 calls first, then the report is emptied. Then 8 x oneIn calls, and + * more only while none has been timed (probability e^-8). The count is fixed + * because what the warm-up allocates decides how many scavenges land in the + * window: one more is +1.4% on wrap-l1-hit-int-cachekitio on Node 24. Reads + * the cache's private l1Latency; a build from before the report has no timed + * path. + */ +async function warmTimed(cache, op) { + const report = cache.l1Latency; + if (!report) return; + for (let i = 0; i < 64; i++) await op(); + report.take(); + let calls = 8 * report.oneIn; + for (let i = 0; i < calls; i++) { + await op(); + if (!report.empty) continue; + // Every call untimed for 64 x oneIn calls has probability e^-64. + if (i + 1 === 64 * report.oneIn) + throw new Error(`no call was timed in ${i + 1} calls: the L1 latency sampler never fired`); + calls = Math.max(calls, i + 2); + } +} const KEY = 'bench:getUser:' + 'a'.repeat(64); const ARGS = [42, 'user-profile', { region: 'au', tier: 3 }]; // 1,055 B of msgpack. Keep fixtures under the serializer's 10,000-element @@ -88,7 +124,9 @@ let sink = 0; // ops. A workload whose op tiers up later, or runs too long for 2,000, sets // its own `warmup`. To check one, run it under V8_FLAGS with --trace-opt // --trace-deopt and print a line at each process.cpuUsage() call: no compile -// or deopt may land between the two. +// or deopt may land between the two. An op with a path that only some calls +// take at random (a timed call, see warmTimed) must also pass with extra +// Math.random() draws made before its setup, which move those calls. const WARMUP = 2000; async function encryptedOp(manager, storage, bytes) { @@ -474,9 +512,12 @@ export const WORKLOADS = { // backend's latency report. It sets no option a build before the report // lacks, so the gate measures it on the base too. fetch is stubbed: the // warm-up's miss and write never leave the process, and the measured hits - // make no request. + // make no request. 4,000 warm-up ops after warmTimed: when its timed call + // comes after V8 has compiled the op, the op deoptimises there and takes up + // to ~1,600 ops to compile again on Node 24. 'wrap-l1-hit-cachekitio': { n: 10_000, + warmup: 4_000, setup: async () => { stubCachekitIO(); const { createCache } = await load('index.js'); @@ -485,27 +526,35 @@ export const WORKLOADS = { namespace: 'bench', ttl: 3600, }); - return async () => { + const op = async () => { sink += (await getUser(...ARGS)).id; }; + await warmTimed(cache, op); + return op; }, }, // wrap-l1-hit-int on the same CachekitIO backend: the memoised hit, where - // the timing draw is the largest share of a hit. 4,000 warm-up ops: on Node - // 24 V8 compiles the wrap() closure with TurboFan at ~2,050 ops. + // the timing draw is the largest share of a hit. 8,000 warm-up ops after + // warmTimed: on Node 24 V8 compiles the wrap() closure with TurboFan at + // ~2,050 ops, and when warmTimed's timed call comes after that, the closure + // deoptimises there and takes up to ~3,400 ops to compile again. 'wrap-l1-hit-int-cachekitio': { n: 20_000, - warmup: 4_000, + warmup: 8_000, setup: async () => { stubCachekitIO(); const { createCache } = await load('index.js'); const cache = createCache({ backend: { apiKey: FAKE_KEY }, metrics: false }); const getThing = cache.wrap(async (id) => ({ id }), { namespace: 'bench', ttl: 3600 }); - for (let i = 0; i < 64; i++) await getThing(i); + // The misses, then a hit each, which leaves the memo holding every + // key's L1 entry, as each call in the window finds it. + for (let i = 0; i < 128; i++) await getThing(i & 63); let i = 0; - return async () => { + const op = async () => { sink += (await getThing(i++ & 63)).id; }; + await warmTimed(cache, op); + return op; }, }, // the README benchmark's hit (bench/runtime/probe.mjs): one integer argument, From c405afdf369fa80c2d9ef002e6fd39a99acf654e Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 1 Nov 2026 15:58:12 +0000 Subject: [PATCH 7/8] test(bench): warm the CachekitIO workloads past 8 timed calls The functions only a timed call runs (recording its duration) get their feedback vectors at about their 8th call. With fewer timed calls before the window, their first calls landed in it: at the old 4,000-op warm-up that was a one-time 3.3M instructions, 132 Ir/op of wrap-l1-hit-int-cachekitio's 2,529, and how many timed calls came first followed the sampler's RNG phase. warmTimed now runs 24 x oneIn calls, more only while fewer than 8 have been timed, so the count stays fixed and the scavenges in the window with it. Its calls also cover the op's tier-up, so the two workloads keep their earlier warm-ups. ir-seconds.json takes the run times this costs. --- packages/cachekit/bench/ir-seconds.json | 6 +-- packages/cachekit/bench/ir-workload.mjs | 59 +++++++++++++------------ 2 files changed, 33 insertions(+), 32 deletions(-) diff --git a/packages/cachekit/bench/ir-seconds.json b/packages/cachekit/bench/ir-seconds.json index c78a5d1d..659acab8 100644 --- a/packages/cachekit/bench/ir-seconds.json +++ b/packages/cachekit/bench/ir-seconds.json @@ -28,11 +28,11 @@ "get-l1-hit": 11, "wrap-l1-hit": 17, "wrap-l1-hit-metrics": 12, - "get-l2-hit-metrics": 68, + "get-l2-hit-metrics": 63, "secure-wrap-l1-hit-100b": 47, "secure-wrap-l1-hit-10kb": 83, "secure-wrap-l1-hit-1mb": 166, "wrap-l1-hit-int": 15, - "wrap-l1-hit-cachekitio": 30, - "wrap-l1-hit-int-cachekitio": 24 + "wrap-l1-hit-cachekitio": 34, + "wrap-l1-hit-int-cachekitio": 23 } diff --git a/packages/cachekit/bench/ir-workload.mjs b/packages/cachekit/bench/ir-workload.mjs index 9ba22699..de546b41 100644 --- a/packages/cachekit/bench/ir-workload.mjs +++ b/packages/cachekit/bench/ir-workload.mjs @@ -44,37 +44,42 @@ function stubCachekitIO() { } /** - * Warms op until the cache's sampler has timed one of its calls into the L1 - * latency report, so the warm-up that follows tiers the op up with the timed - * path in its type feedback. Which calls are timed follows the sampler, which - * is seeded from Math.random: one more draw anywhere before the cache is made - * (a build's module load included) moves them. Left to that, the first timed - * call can land after V8 has compiled the op, which then deoptimises and - * compiles again late in the warm-up or inside the window. + * Warms op until the cache's sampler has timed 8 of its calls into the L1 + * latency report, so the window opens with the timed path compiled like the + * rest. Which calls are timed follows the sampler, which is seeded from + * Math.random: one more draw anywhere before the cache is made (a build's + * module load included) moves them. Left to that, the first timed call can + * land after V8 has compiled the op, which then deoptimises and compiles + * again late in the warm-up or inside the window. And the functions only a + * timed call runs (recording the duration) get their feedback vectors at + * about their 8th call (--invocation-count-for-feedback-allocation), so with + * fewer timed calls before the window, their first calls cost it up to 3.3M + * instructions. * * Call it once op takes the path it takes in the window: a call timed on * another path (the first hit of a memoised key) gives the window's path no - * feedback. Nor does a call before V8 has given each function its feedback - * vector, at about its 8th call (--invocation-count-for-feedback-allocation): - * so 64 calls first, then the report is emptied. Then 8 x oneIn calls, and - * more only while none has been timed (probability e^-8). The count is fixed - * because what the warm-up allocates decides how many scavenges land in the - * window: one more is +1.4% on wrap-l1-hit-int-cachekitio on Node 24. Reads - * the cache's private l1Latency; a build from before the report has no timed - * path. + * feedback, nor does a call before op's own functions have their feedback + * vectors: so 64 calls first, then the report is emptied. Then 24 x oneIn + * calls, and more only while fewer than 8 have been timed (probability + * below 0.01%), each timed call taken from the report as it comes. The count + * is fixed because what the warm-up allocates decides how many scavenges land + * in the window: one more is +1.4% on wrap-l1-hit-int-cachekitio on Node 24. + * Reads the cache's private l1Latency; a build from before the report has no + * timed path. */ async function warmTimed(cache, op) { const report = cache.l1Latency; if (!report) return; for (let i = 0; i < 64; i++) await op(); report.take(); - let calls = 8 * report.oneIn; - for (let i = 0; i < calls; i++) { + let calls = 24 * report.oneIn; + for (let i = 0, timed = 0; i < calls; i++) { await op(); - if (!report.empty) continue; - // Every call untimed for 64 x oneIn calls has probability e^-64. + if (report.take()) timed++; + if (timed >= 8) continue; + // Fewer than 8 timed in 64 x oneIn calls has probability below 10^-17. if (i + 1 === 64 * report.oneIn) - throw new Error(`no call was timed in ${i + 1} calls: the L1 latency sampler never fired`); + throw new Error(`${timed} of ${i + 1} calls were timed: the L1 latency sampler is broken`); calls = Math.max(calls, i + 2); } } @@ -512,12 +517,9 @@ export const WORKLOADS = { // backend's latency report. It sets no option a build before the report // lacks, so the gate measures it on the base too. fetch is stubbed: the // warm-up's miss and write never leave the process, and the measured hits - // make no request. 4,000 warm-up ops after warmTimed: when its timed call - // comes after V8 has compiled the op, the op deoptimises there and takes up - // to ~1,600 ops to compile again on Node 24. + // make no request. Setup ends with warmTimed. 'wrap-l1-hit-cachekitio': { n: 10_000, - warmup: 4_000, setup: async () => { stubCachekitIO(); const { createCache } = await load('index.js'); @@ -534,13 +536,12 @@ export const WORKLOADS = { }, }, // wrap-l1-hit-int on the same CachekitIO backend: the memoised hit, where - // the timing draw is the largest share of a hit. 8,000 warm-up ops after - // warmTimed: on Node 24 V8 compiles the wrap() closure with TurboFan at - // ~2,050 ops, and when warmTimed's timed call comes after that, the closure - // deoptimises there and takes up to ~3,400 ops to compile again. + // the timing draw is the largest share of a hit. Setup ends with warmTimed. + // 4,000 warm-up ops for a build without the report, where warmTimed runs + // none: on Node 24 V8 compiles the wrap() closure with TurboFan at ~2,050. 'wrap-l1-hit-int-cachekitio': { n: 20_000, - warmup: 8_000, + warmup: 4_000, setup: async () => { stubCachekitIO(); const { createCache } = await load('index.js'); From 927975a4e7f15bc03b1e5296fea02f601176aa96 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Sun, 1 Nov 2026 16:21:50 +0000 Subject: [PATCH 8/8] test(bench): find the latency report by its class, and refuse a cache that hides it warmTimed read the cache's private l1Latency and skipped its warm-up when the field was missing, as it is in a build from before the report. A build that only renamed the field then measured the timed path cold: wrap-l1-hit-int-cachekitio read +7% with no change behind it. warmTimed now skips only when the build has no L1LatencyReport class. Otherwise it finds the report among the cache's fields by its class, which holds across a rename on either side of a compare, and throws when the cache holds none it can read: telemetry switched off, or the report moved. --- packages/cachekit/bench/ir-workload.mjs | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/packages/cachekit/bench/ir-workload.mjs b/packages/cachekit/bench/ir-workload.mjs index de546b41..644176aa 100644 --- a/packages/cachekit/bench/ir-workload.mjs +++ b/packages/cachekit/bench/ir-workload.mjs @@ -64,12 +64,26 @@ function stubCachekitIO() { * below 0.01%), each timed call taken from the report as it comes. The count * is fixed because what the warm-up allocates decides how many scavenges land * in the window: one more is +1.4% on wrap-l1-hit-int-cachekitio on Node 24. - * Reads the cache's private l1Latency; a build from before the report has no - * timed path. + * + * The report is a private field of the cache, so this finds it among the + * cache's fields by its class, not by its name: the gate runs one copy of this + * file on a base and a head that may name it differently. A build without the + * class has no timed path, and the warm-up is skipped. A build with the class + * whose cache holds no report it can read throws: skipping there would leave + * the timed path cold in the window, a false regression of about 5%. */ async function warmTimed(cache, op) { - const report = cache.l1Latency; - if (!report) return; + const telemetry = existsSync(new URL('backends/telemetry.js', dist)) + ? await load('backends/telemetry.js') + : {}; + if (!telemetry.L1LatencyReport) return; + const report = Object.values(cache).find((v) => v instanceof telemetry.L1LatencyReport); + if (!(report?.oneIn >= 1)) { + throw new Error( + 'the build has an L1 latency report but the cache holds none this can read: ' + + 'unset CACHEKIT_TELEMETRY=off, or teach warmTimed where the report went' + ); + } for (let i = 0; i < 64; i++) await op(); report.take(); let calls = 24 * report.oneIn;