diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4d5879f..64e8af9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -147,14 +147,15 @@ Six harnesses measure the client. **Call shape (runs in `pnpm test`).** `packages/cachekit/test/transport/call-shape.test.ts` runs the CachekitIO backend against a local TLS fake of the SaaS. It asserts the exact requests each op sends (a wrap miss is `GET` then `PUT`, and with `writeBehind` the caller waits for the `GET` only; a locked miss is `GET`, lock, `PUT`, unlock) and the connections 100 ops use. Each extra request is a round trip, and each extra connection is a TCP and TLS handshake. So if you change one of these numbers, change the expectation in the same PR and say why. The fake mints a throwaway certificate with the `openssl` CLI, and it needs Node 22.19+ or 24.5+ for `tls.setDefaultCACertificates`. Where either is missing, the tests fail with a message that names the requirement. -**Instructions per op (`pnpm --filter @cachekit-io/cachekit bench:ir`).** This counts the main-thread instructions of each hot path under callgrind: key generation through the NAPI hash (`keygen`, `keygen-1kb`) and the wasm hash (`keygen-wasm`, `keygen-wasm-1kb`), at 45 B and 1 KB of arguments, the serializer, the NAPI and wasm envelopes (1 KB, plus 40 B under the core's 64 B literals-only cutover and 64 KiB), both encrypted paths (1 KB, plus 64 KiB of incompressible bytes), and the L1 hit through `get()` (`get-l1-hit`) and through `wrap()`, once with an object among the arguments and once with the single integer argument the README benchmark passes, and the encrypted `wrap()` L1 hit at 100 B, 10 KB and 1 MB. Unlike wall-clock timing, instruction counts repeat on a busy machine. The suite pins V8's flags and counts only the measured ops. On Node 22 and 24, every path repeats within 0.01%. On Node 26, paths through the native cores repeat within about 0.3%, but `napi-encrypted` and the encrypted `wrap()` hits at 100 B and 1 MB sometimes have a run up to 1.7% off the rest, which makes them read inconclusive. Gate on Node 22 or 24. It needs Linux and valgrind. It imports `dist/` and the wasm build, so run `pnpm build` and `pnpm --filter @cachekit-io/cachekit-core-wasm build:wasm` first; the second needs the `wasm32-unknown-unknown` target, `wasm-bindgen` and `wasm-opt`, and without it pass `--only` to skip the seven wasm workloads (`wasm-envelope`, `wasm-envelope-40b`, `wasm-envelope-64kb`, `wasm-encrypted`, `wasm-encrypted-64kb`, `keygen-wasm`, `keygen-wasm-1kb`). To A/B a change, run it on both builds on the same machine: +**Instructions per op (`pnpm --filter @cachekit-io/cachekit bench:ir`).** This counts the main-thread instructions of each hot path under callgrind: key generation through the NAPI hash (`keygen`, `keygen-1kb`) and the wasm hash (`keygen-wasm`, `keygen-wasm-1kb`), at 45 B and 1 KB of arguments, the serializer, the NAPI and wasm envelopes (1 KB, plus 40 B under the core's 64 B literals-only cutover and 64 KiB), both encrypted paths (1 KB, plus 64 KiB of incompressible bytes), and the L1 hit through `get()` (`get-l1-hit`) and through `wrap()`, once with an object among the arguments and once with the single integer argument the README benchmark passes, and the encrypted `wrap()` L1 hit at 100 B, 10 KB and 1 MB. Unlike wall-clock timing, instruction counts repeat on a busy machine. The suite pins V8's flags and counts only the measured ops. It also lowers V8's TurboFan threshold, so no optimizing compile lands in any workload's window. For all but the two 1 MB workloads, the window counts the code V8 runs in steady state. Those run so few ops that the `set()` and `wrap()` code around each 1 MB encode or decode stays in V8's interpreter for the whole run, which costs too little to show beside the encode or decode. A workload that tiers up later sets its own `warmup`; `bench/ir-workload.mjs` says how to check one with `--trace-opt`. On Node 22 and 24, every path repeats within 0.01%. On Node 26, paths through the native cores repeat within about 0.3%, but `napi-encrypted` and the encrypted `wrap()` hits at 100 B and 1 MB sometimes have a run up to 1.7% off the rest, which makes them read inconclusive. Gate on Node 22 or 24. It needs Linux and valgrind. It imports `dist/` and the wasm build, so run `pnpm build` and `pnpm --filter @cachekit-io/cachekit-core-wasm build:wasm` first; the second needs the `wasm32-unknown-unknown` target, `wasm-bindgen` and `wasm-opt`, and without it pass `--only` to skip the seven wasm workloads (`wasm-envelope`, `wasm-envelope-40b`, `wasm-envelope-64kb`, `wasm-encrypted`, `wasm-encrypted-64kb`, `keygen-wasm`, `keygen-wasm-1kb`). To A/B a change, run it on both builds on the same machine: ```bash +git checkout -- packages/cachekit/bench # in the base checkout: your bench/, as CI does pnpm --filter @cachekit-io/cachekit bench:ir --save /tmp/base.json # on the base build pnpm --filter @cachekit-io/cachekit bench:ir --compare /tmp/base.json # on your build ``` -A workload more than 2% worse fails, and more than 1% worse warns, unless it sets its own `fail` threshold and warns at half of it: `plaintext-set-1mb` (10%) and `napi-encrypted` (4%) do, for the CI measurements noted beside them in `bench/ir-workload.mjs`. If the two runs come from a different node, V8 or valgrind, the compare refuses them. A run whose own repeats spread more than 0.4% (the A/A) reports inconclusive rather than passing. A compare also refuses (exit 2) two runs that measured different workloads, or two runs of the same build, and refuses to start when `--save` and `--compare` name the same file. A workload whose run fails (a valgrind crash, a missing wasm build) reads INCOMPLETE while the others still run, and any other failure (bad arguments, a missing `dist/`) stops the run; either way the exit code is 4 unless a regression (1) or a refusal (2) outranks it. Add `--wall 9` to print indicative ns per op; it is not saved, and it is never a measured saving. +A workload more than 2% worse fails, and more than 1% worse warns, unless it sets its own `fail` threshold and warns at half of it: `plaintext-set-1mb` (10%) and `napi-encrypted` (4%) do, for the CI measurements noted beside them in `bench/ir-workload.mjs`. If the two runs come from a different node, V8 or valgrind, or from an `ir.mjs` with different V8 flags, the compare refuses them. Run one copy of `bench/` against both builds, as the CI gate does: a workload's warm-up and n are not checked, so two copies that set them differently measure different things. A run whose own repeats spread more than 0.4% (the A/A) reports inconclusive rather than passing. A compare also refuses (exit 2) two runs that measured different workloads, or two runs of the same build, and refuses to start when `--save` and `--compare` name the same file. A workload whose run fails (a valgrind crash, a missing wasm build) reads INCOMPLETE while the others still run, and any other failure (bad arguments, a missing `dist/`) stops the run; either way the exit code is 4 unless a regression (1) or a refusal (2) outranks it. Add `--wall 9` to print indicative ns per op; it is not saved, and it is never a measured saving. **The instruction-count gate (`.github/workflows/instruction-counts.yml`).** Every PR to `main` runs this compare in CI. Each job builds the PR's merge commit and its base on one GitHub-hosted runner, with the same Node and valgrind, at paths of equal length. Each tree builds with the wasm-bindgen CLI its own `Cargo.lock` locks, so a wasm-bindgen bump is measured too. Both builds run the PR's `bench/`, so a workload the PR adds is also measured on the base wherever the base can run it. A workload for an API the base lacks reads `new` and is not gated (`--new`). A workload that fails on either build reads INCOMPLETE, and so does one with no row at all, so the gate never passes on a workload it did not measure. A release-please PR is measured against its package's last release tag instead of `main` (`.github/scripts/perf-ir-base.sh`), so its comment shows each workload's drift since that release. When both builds hash alike, the job passes without measuring. `--build-hash` covers, for this package and both cores, `package.json` and every file its `files` list ships (`dist/` with its types and source maps, the NAPI loader and binary, the wasm glue JavaScript and module, and this package's README and LICENSE), plus `pnpm-lock.yaml`. So a PR that edits only one of those is measured too. A file a workload loads must therefore ship through a `files` entry, or a change to it passes unmeasured; `.github/scripts/perf-ir-hash-selftest.sh`, run in CI, checks the hash moves for each kind. Four shards split the workloads, each building both trees, balanced on the seconds one run of each workload took in CI (`bench/ir-seconds.json`). A workload missing from that file counts as the median, so add its figure from the gate's `base.json` artifact (`seconds` / `repeats`) when you add a workload. CI runs three repeats, the fewest `ir.mjs` takes, to stay well inside the job's 30 minutes. The verdicts go to one PR comment, edited in place on every push; a fork PR gets the job summary instead, because its token cannot comment. diff --git a/packages/cachekit/bench/ir-seconds.json b/packages/cachekit/bench/ir-seconds.json index c09028e..659acab 100644 --- a/packages/cachekit/bench/ir-seconds.json +++ b/packages/cachekit/bench/ir-seconds.json @@ -1,32 +1,38 @@ { - "l1-get-hit": 28, - "l1-set-at-capacity-1k": 37, - "l1-set-at-capacity-10k": 16, - "l1-set-at-capacity-100k": 24, - "plaintext-set-1kb": 42, - "plaintext-set-10kb": 82, - "plaintext-set-100kb": 203, - "plaintext-set-64b": 33, - "plaintext-set-1mb": 61, - "plaintext-set-containers": 92, - "keygen": 22, - "keygen-1kb": 43, - "keygen-wasm": 24, - "keygen-wasm-1kb": 40, - "serializer-roundtrip": 61, - "napi-envelope": 22, - "napi-envelope-40b": 12, - "napi-envelope-64kb": 39, - "wasm-envelope": 32, - "wasm-envelope-40b": 25, - "wasm-envelope-64kb": 54, - "napi-encrypted": 21, - "napi-encrypted-64kb": 66, - "wasm-encrypted": 21, - "wasm-encrypted-64kb": 92, - "secure-set-get": 73, - "wrap-l1-hit": 22, - "wrap-l1-hit-int": 20, - "wrap-l1-hit-cachekitio": 22, - "wrap-l1-hit-int-cachekitio": 20 + "l1-get-hit": 36, + "l1-set-at-capacity-1k": 15, + "l1-set-at-capacity-10k": 12, + "l1-set-at-capacity-100k": 10, + "plaintext-set-1kb": 29, + "plaintext-set-10kb": 69, + "plaintext-set-100kb": 191, + "plaintext-set-64b": 27, + "plaintext-set-1mb": 86, + "plaintext-set-containers": 72, + "keygen": 17, + "keygen-1kb": 33, + "keygen-wasm": 18, + "keygen-wasm-1kb": 23, + "serializer-roundtrip": 59, + "napi-envelope": 23, + "napi-envelope-40b": 18, + "napi-envelope-64kb": 29, + "wasm-envelope": 24, + "wasm-envelope-40b": 17, + "wasm-envelope-64kb": 39, + "napi-encrypted": 18, + "napi-encrypted-64kb": 61, + "wasm-encrypted": 11, + "wasm-encrypted-64kb": 122, + "secure-set-get": 57, + "get-l1-hit": 11, + "wrap-l1-hit": 17, + "wrap-l1-hit-metrics": 12, + "get-l2-hit-metrics": 63, + "secure-wrap-l1-hit-100b": 47, + "secure-wrap-l1-hit-10kb": 83, + "secure-wrap-l1-hit-1mb": 166, + "wrap-l1-hit-int": 15, + "wrap-l1-hit-cachekitio": 34, + "wrap-l1-hit-int-cachekitio": 23 } diff --git a/packages/cachekit/bench/ir-workload.mjs b/packages/cachekit/bench/ir-workload.mjs index fc7159c..644176a 100644 --- a/packages/cachekit/bench/ir-workload.mjs +++ b/packages/cachekit/bench/ir-workload.mjs @@ -2,12 +2,14 @@ // // node [v8 flags] bench/ir-workload.mjs [--wall] // -// A run loads only what its workload needs, warms the op past tier-up, then -// runs it n times between two process.cpuUsage() calls. ir.mjs has callgrind -// dump its counters on entry to node::CPUUsage, so the counted window holds -// the n ops and nothing else: module load, setup and warm-up stay out. -// --wall prints the steady-state ns per op of the n iterations instead -// (indicative only). Imports the built package: run `pnpm build` first. +// A run loads only what its workload needs, warms the op until V8 has +// finished compiling it, then runs it n times between two process.cpuUsage() +// calls. ir.mjs has callgrind dump its counters on entry to node::CPUUsage, so +// the counted window holds the n ops and nothing else: module load, setup, +// warm-up and V8's optimizing compiles stay out. +// --wall prints the ns per op of the n iterations instead (indicative only: +// ir.mjs runs it without V8_FLAGS, so V8 may still be compiling the op in +// the background). Imports the built package: run `pnpm build` first. import { existsSync, readFileSync } from 'node:fs'; import { createRequire } from 'node:module'; import { fileURLToPath } from 'node:url'; @@ -40,6 +42,61 @@ function stubCachekitIO() { globalThis.fetch = async (_url, init) => new globalThis.Response(null, { status: init?.method === 'GET' ? 404 : 200 }); } + +/** + * Warms op until the cache's sampler has timed 8 of its calls into the L1 + * latency report, so the window opens with the timed path compiled like the + * rest. Which calls are timed follows the sampler, which is seeded from + * Math.random: one more draw anywhere before the cache is made (a build's + * module load included) moves them. Left to that, the first timed call can + * land after V8 has compiled the op, which then deoptimises and compiles + * again late in the warm-up or inside the window. And the functions only a + * timed call runs (recording the duration) get their feedback vectors at + * about their 8th call (--invocation-count-for-feedback-allocation), so with + * fewer timed calls before the window, their first calls cost it up to 3.3M + * instructions. + * + * Call it once op takes the path it takes in the window: a call timed on + * another path (the first hit of a memoised key) gives the window's path no + * feedback, nor does a call before op's own functions have their feedback + * vectors: so 64 calls first, then the report is emptied. Then 24 x oneIn + * calls, and more only while fewer than 8 have been timed (probability + * below 0.01%), each timed call taken from the report as it comes. The count + * is fixed because what the warm-up allocates decides how many scavenges land + * in the window: one more is +1.4% on wrap-l1-hit-int-cachekitio on Node 24. + * + * The report is a private field of the cache, so this finds it among the + * cache's fields by its class, not by its name: the gate runs one copy of this + * file on a base and a head that may name it differently. A build without the + * class has no timed path, and the warm-up is skipped. A build with the class + * whose cache holds no report it can read throws: skipping there would leave + * the timed path cold in the window, a false regression of about 5%. + */ +async function warmTimed(cache, op) { + const telemetry = existsSync(new URL('backends/telemetry.js', dist)) + ? await load('backends/telemetry.js') + : {}; + if (!telemetry.L1LatencyReport) return; + const report = Object.values(cache).find((v) => v instanceof telemetry.L1LatencyReport); + if (!(report?.oneIn >= 1)) { + throw new Error( + 'the build has an L1 latency report but the cache holds none this can read: ' + + 'unset CACHEKIT_TELEMETRY=off, or teach warmTimed where the report went' + ); + } + for (let i = 0; i < 64; i++) await op(); + report.take(); + let calls = 24 * report.oneIn; + for (let i = 0, timed = 0; i < calls; i++) { + await op(); + if (report.take()) timed++; + if (timed >= 8) continue; + // Fewer than 8 timed in 64 x oneIn calls has probability below 10^-17. + if (i + 1 === 64 * report.oneIn) + throw new Error(`${timed} of ${i + 1} calls were timed: the L1 latency sampler is broken`); + calls = Math.max(calls, i + 2); + } +} const KEY = 'bench:getUser:' + 'a'.repeat(64); const ARGS = [42, 'user-profile', { region: 'au', tier: 3 }]; // 1,055 B of msgpack. Keep fixtures under the serializer's 10,000-element @@ -81,8 +138,14 @@ async function incompressible() { } let sink = 0; -// Ops run before the window opens, past V8's tier-up. A workload whose op -// is long enough to tier up sooner sets its own `warmup`. +// Ops run before the window opens. Under ir.mjs's V8_FLAGS every function a +// cheap op runs has finished its Maglev and TurboFan compiles within 2,000 +// ops. A workload whose op tiers up later, or runs too long for 2,000, sets +// its own `warmup`. To check one, run it under V8_FLAGS with --trace-opt +// --trace-deopt and print a line at each process.cpuUsage() call: no compile +// or deopt may land between the two. An op with a path that only some calls +// take at random (a timed call, see warmTimed) must also pass with extra +// Math.random() draws made before its setup, which move those calls. const WARMUP = 2000; async function encryptedOp(manager, storage, bytes) { @@ -156,8 +219,9 @@ const plaintextSet = (rows, n, warmup) => plaintextSetOf(rowsOf(rows), n, warmup * build from before the key hash moved into the bindings has no * serialization/blake2b.js and hashes keys in JS: the base side of a compare. */ -const keygen = (args, n, wasmHash = false) => ({ +const keygen = (args, n, wasmHash = false, warmup) => ({ n, + warmup, setup: async () => { const hashing = existsSync(new URL('serialization/blake2b.js', dist)) ? await load('serialization/blake2b.js') @@ -259,17 +323,22 @@ export const WORKLOADS = { }, 'l1-set-at-capacity-1k': l1AtCapacity(1_000, 200_000), 'l1-set-at-capacity-10k': l1AtCapacity(10_000, 4_000), - 'l1-set-at-capacity-100k': l1AtCapacity(100_000, 2_000), + // 20,000 ops: 100,000 live entries keep V8 marking the old generation + // through the window, and over 2,000 ops the share of that marking in the + // window differed between runs by up to 1.9%; over 20,000 it repeats within + // 0.01%. + 'l1-set-at-capacity-100k': l1AtCapacity(100_000, 20_000), 'plaintext-set-1kb': plaintextSet(32, 5_000), 'plaintext-set-10kb': plaintextSet(320, 1_000), - // 200 warm-up ops, not 2,000: each op encodes 3,200 rows, so V8 has tiered - // up long before, and the default made one process run ~9 minutes in CI. - // The shorter warm-up moved its Ir/op by +0.4% in CI. - 'plaintext-set-100kb': plaintextSet(3_200, 1_000, 200), + // 1,300 warm-up ops, not 2,000: V8 has finished compiling the set() path + // around it by ~960 ops, and 2,000 made one process run ~9 minutes in CI. + 'plaintext-set-100kb': plaintextSet(3_200, 1_000, 1_300), // the ends of the object-heavy write range: one 64 B object, and 1,005,035 B // in 96 groups of 320 rows, so no collection nears the 10,000-element bound - // and the whole stays under the 1 MiB encode limit. A 1 MB op runs long - // enough under callgrind that 20 warm-up ops are past tier-up. + // and the whole stays under the 1 MiB encode limit. 20 warm-up ops for 1 MB: + // the encoders it calls per row reach TurboFan in the first op, and the + // set() path around them runs 40 times, too few for V8 to compile it, so + // the window holds no compile. 'plaintext-set-64b': plaintextSetOf( { id: 7, name: 'item-7', score: 10.5, ok: true, tags: ['a', 'b'], owner: 'user-424242' }, 20_000 @@ -301,9 +370,10 @@ export const WORKLOADS = { // 1 KB of arguments, where the hash's share of the cost grows with the bytes 'keygen-1kb': keygen([VALUE], 5_000), // the same on Workers, where the wasm binding hashes keys once its module is - // instantiated - 'keygen-wasm': keygen(ARGS, 20_000, true), - 'keygen-wasm-1kb': keygen([VALUE], 5_000, true), + // instantiated. 4,000 warm-up ops: the glue reads each hash back through + // Node's TextDecoder decode, which V8 compiles with TurboFan at ~2,400 ops. + 'keygen-wasm': keygen(ARGS, 20_000, true, 4_000), + 'keygen-wasm-1kb': keygen([VALUE], 5_000, true, 4_000), // set encodes (normalize + msgpack, in one walk), an L2 hit decodes // (depth bounds + msgpack) 'serializer-roundtrip': { @@ -335,9 +405,12 @@ export const WORKLOADS = { 'wasm-encrypted': wasmEncrypted(encoded, 5_000), 'wasm-encrypted-64kb': wasmEncrypted(incompressible, 50), // a secure cache's L2 round trip on Node, L1 off: set + get, each running the - // AAD-size pre-flight (validateKey) and then encrypt or decrypt + // AAD-size pre-flight (validateKey) and then encrypt or decrypt. 4,000 + // warm-up ops: the get reads the envelope header through Node's TextDecoder, + // whose decode V8 compiles with TurboFan at ~2,400 ops. 'secure-set-get': { n: 5_000, + warmup: 4_000, setup: async () => { const { createCache } = await load('index.js'); const cache = createCache({ @@ -353,9 +426,11 @@ export const WORKLOADS = { }, }, // get() served from a plaintext L1, which holds the decoded value: the - // lookup and the promises around it (l1-get-hit times L1Cache.get alone) + // lookup and the promises around it (l1-get-hit times L1Cache.get alone). + // 4,000 warm-up ops: V8 compiles getEntry with TurboFan at ~2,200. 'get-l1-hit': { n: 20_000, + warmup: 4_000, setup: async () => { const { createCache } = await load('index.js'); const cache = createCache({ backend: new MemoryBackend(), metrics: false }); @@ -412,9 +487,9 @@ export const WORKLOADS = { // duration histogram, breaker gauge) around an unpack and a decode 'get-l2-hit-metrics': { n: 10_000, - // 2,000 ops leave ~40 TurboFan compiles inside the window (node --trace-opt - // with markers at the window); 30,000 leave none on Node 22. - warmup: 30_000, + // 4,000 warm-up ops: the hit reads through Node's TextDecoder, whose + // decode V8 compiles with TurboFan at ~2,400 ops on Node 24. + warmup: 4_000, setup: async () => { const { createCache } = await load('index.js'); const { Registry } = await import('prom-client'); @@ -441,9 +516,11 @@ export const WORKLOADS = { // an encrypted wrap() L1 hit at 100 B, 10 KB and 1 MB of msgpack (3, 320 // and 96 x 320 rows). On Node 22 and 24 each repeats within 0.01%. On Node // 26 the 100 B and 1 MB hits sometimes have a run up to 1.7% off the rest, - // and a 4x larger n did not remove it: gate on Node 22 or 24. - 'secure-wrap-l1-hit-100b': secureWrapHit(rowsOf(3), 40_000), - 'secure-wrap-l1-hit-10kb': secureWrapHit(rowsOf(320), 500), + // and a 4x larger n did not remove it: gate on Node 22 or 24. 4,000 warm-up + // ops at 100 B and 10 KB: V8 compiles normalize with TurboFan at ~2,200 ops, + // and the encoder deoptimises and is compiled again up to ~2,500. + 'secure-wrap-l1-hit-100b': secureWrapHit(rowsOf(3), 40_000, 4_000), + 'secure-wrap-l1-hit-10kb': secureWrapHit(rowsOf(320), 500, 4_000), 'secure-wrap-l1-hit-1mb': secureWrapHit( { groups: Array.from({ length: 96 }, () => rowsOf(320)) }, 50, @@ -454,7 +531,7 @@ export const WORKLOADS = { // backend's latency report. It sets no option a build before the report // lacks, so the gate measures it on the base too. fetch is stubbed: the // warm-up's miss and write never leave the process, and the measured hits - // make no request. + // make no request. Setup ends with warmTimed. 'wrap-l1-hit-cachekitio': { n: 10_000, setup: async () => { @@ -465,25 +542,34 @@ export const WORKLOADS = { namespace: 'bench', ttl: 3600, }); - return async () => { + const op = async () => { sink += (await getUser(...ARGS)).id; }; + await warmTimed(cache, op); + return op; }, }, // wrap-l1-hit-int on the same CachekitIO backend: the memoised hit, where - // the timing draw is the largest share of a hit + // the timing draw is the largest share of a hit. Setup ends with warmTimed. + // 4,000 warm-up ops for a build without the report, where warmTimed runs + // none: on Node 24 V8 compiles the wrap() closure with TurboFan at ~2,050. 'wrap-l1-hit-int-cachekitio': { n: 20_000, + warmup: 4_000, setup: async () => { stubCachekitIO(); const { createCache } = await load('index.js'); const cache = createCache({ backend: { apiKey: FAKE_KEY }, metrics: false }); const getThing = cache.wrap(async (id) => ({ id }), { namespace: 'bench', ttl: 3600 }); - for (let i = 0; i < 64; i++) await getThing(i); + // The misses, then a hit each, which leaves the memo holding every + // key's L1 entry, as each call in the window finds it. + for (let i = 0; i < 128; i++) await getThing(i & 63); let i = 0; - return async () => { + const op = async () => { sink += (await getThing(i++ & 63)).id; }; + await warmTimed(cache, op); + return op; }, }, // the README benchmark's hit (bench/runtime/probe.mjs): one integer argument, @@ -503,6 +589,14 @@ export const WORKLOADS = { }, }; +// The warm-up and the window run their ops through this one function, so it +// is as warm as the op when the window opens. A loop in the module body +// instead had V8 compile the module body inside the window and deoptimise it +// at the window's end. +async function repeat(op, times) { + for (let i = 0; i < times; i++) await op(); +} + if (process.argv[1] === fileURLToPath(import.meta.url)) { const [name, nArg] = process.argv.slice(2); const n = Number(nArg); @@ -513,11 +607,10 @@ if (process.argv[1] === fileURLToPath(import.meta.url)) { throw new Error(`n must be a non-negative integer, got "${nArg}"`); const op = await WORKLOADS[name].setup(n); - const warmup = WORKLOADS[name].warmup ?? WARMUP; - for (let i = 0; i < warmup; i++) await op(); + await repeat(op, WORKLOADS[name].warmup ?? WARMUP); process.cpuUsage(); // window start: callgrind dump 1 const t0 = process.hrtime.bigint(); - for (let i = 0; i < n; i++) await op(); + await repeat(op, n); const ns = Number(process.hrtime.bigint() - t0); process.cpuUsage(); // window end: callgrind dump 2 if (process.argv.includes('--wall')) console.log(JSON.stringify({ nsPerOp: n ? ns / n : 0 })); diff --git a/packages/cachekit/bench/ir.mjs b/packages/cachekit/bench/ir.mjs index 8c4186a..bca172d 100644 --- a/packages/cachekit/bench/ir.mjs +++ b/packages/cachekit/bench/ir.mjs @@ -6,14 +6,23 @@ // node bench/ir.mjs --build-hash // // Wall-clock microbenchmarks on a shared or hosted machine cannot resolve the -// 1-2% changes that matter here; instruction counts can, once three sources of -// run-to-run noise are removed: +// 1-2% changes that matter here; instruction counts can, once these sources of +// noise are removed: // - V8: default flags vary main-thread Ir 2-4x between identical runs // (background compilation and GC). V8_FLAGS pins them; --no-liftoff and // --predictable-gc-schedule are what make the wasm paths repeat. // - The fixed part: module load and warm-up vary by 1-3% of a whole process. // So Ir/op = Ir(window) / n, where the window is the n ops of one workload, // bracketed by ir-workload.mjs; nothing else is counted. +// - Tier-up, which moves Ir between builds rather than runs: V8 compiles a +// function with TurboFan after about 3,000 calls, and some functions called +// once per op got there only after 45,000 ops. A compile inside the window +// is a one-time cost divided by n that moves with a function's size and +// V8's inlining, not with the op's cost: it was up to 48% of a workload's +// Ir/op. --invocation-count-for-turbofan lowers the threshold so each +// workload's warm-up runs past it. It barely moves the steady state: after +// a 60,000-op warm-up, six workloads read within 0.32% of the default +// threshold's count. // On Node 22 and 24 every workload then repeats within 0.01%. On Node 26 the // workloads that cross into the NAPI or wasm core keep a residual of // 0.1-0.3%, and napi-encrypted an occasional run about 1% low; neither ASLR @@ -77,6 +86,7 @@ const V8_FLAGS = [ '--random-seed=1', '--no-liftoff', '--predictable-gc-schedule', + '--invocation-count-for-turbofan=150', ]; const N = Object.fromEntries(Object.entries(WORKLOADS).map(([name, w]) => [name, w.n])); const AA_LIMIT = 0.004;