Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ pnpm --filter @cachekit-io/cachekit bench:ir --compare /tmp/base.json # on you

A workload more than 2% worse fails, and more than 1% worse warns, unless it sets its own `fail` threshold and warns at half of it: `plaintext-set-1mb` (10%) and `napi-encrypted` (4%) do, for the CI measurements noted beside them in `bench/ir-workload.mjs`. If the two runs come from a different node, V8 or valgrind, or from an `ir.mjs` with different V8 flags, the compare refuses them. Run one copy of `bench/` against both builds, as the CI gate does: a workload's warm-up and n are not checked, so two copies that set them differently measure different things. A run whose own repeats spread more than 0.4% (the A/A) reports inconclusive rather than passing. A compare also refuses (exit 2) two runs that measured different workloads, or two runs of the same build, and refuses to start when `--save` and `--compare` name the same file. A workload whose run fails (a valgrind crash, a missing wasm build) reads INCOMPLETE while the others still run, and any other failure (bad arguments, a missing `dist/`) stops the run; either way the exit code is 4 unless a regression (1) or a refusal (2) outranks it. Add `--wall 9` to print indicative ns per op; it is not saved, and it is never a measured saving.

**The instruction-count gate (`.github/workflows/instruction-counts.yml`).** Every PR to `main` runs this compare in CI. Each job builds the PR's merge commit and its base on one GitHub-hosted runner, with the same Node and valgrind, at paths of equal length. Each tree builds with the wasm-bindgen CLI its own `Cargo.lock` locks, so a wasm-bindgen bump is measured too. Both builds run the PR's `bench/`, so a workload the PR adds is also measured on the base wherever the base can run it. A workload for an API the base lacks reads `new` and is not gated (`--new`). A workload that fails on either build reads INCOMPLETE, and so does one with no row at all, so the gate never passes on a workload it did not measure. A release-please PR is measured against its package's last release tag instead of `main` (`.github/scripts/perf-ir-base.sh`), so its comment shows each workload's drift since that release. When both builds hash alike, the job passes without measuring. `--build-hash` covers, for this package and both cores, `package.json` and every file its `files` list ships (`dist/` with its types and source maps, the NAPI loader and binary, the wasm glue JavaScript and module, and this package's README and LICENSE), plus `pnpm-lock.yaml`. So a PR that edits only one of those is measured too. A file a workload loads must therefore ship through a `files` entry, or a change to it passes unmeasured; `.github/scripts/perf-ir-hash-selftest.sh`, run in CI, checks the hash moves for each kind. Four shards split the workloads, each building both trees, balanced on the seconds one run of each workload took in CI (`bench/ir-seconds.json`). A workload missing from that file counts as the median, so add its figure from the gate's `base.json` artifact (`seconds` / `repeats`) when you add a workload. CI runs three repeats, the fewest `ir.mjs` takes, to stay well inside the job's 30 minutes. The verdicts go to one PR comment, edited in place on every push; a fork PR gets the job summary instead, because its token cannot comment.
**The instruction-count gate (`.github/workflows/instruction-counts.yml`).** Every PR to `main` runs this compare in CI. Each job builds the PR's merge commit and its base on one GitHub-hosted runner, with the same Node and valgrind, at paths of equal length. Each tree builds with the wasm-bindgen CLI its own `Cargo.lock` locks, so a wasm-bindgen bump is measured too. Both builds run the PR's `bench/`, so a workload the PR adds is also measured on the base wherever the base can run it. A workload for an API the base lacks reads `new` and is not gated (`--new`). A workload that fails on either build reads INCOMPLETE, and so does one with no row at all, so the gate never passes on a workload it did not measure. A release-please PR is measured against its package's last release tag instead of `main` (`.github/scripts/perf-ir-base.sh`), so its comment shows each workload's drift since that release. When both builds hash alike, the job passes without measuring. `--build-hash` covers, for this package and both cores, `package.json` and every file its `files` list ships (`dist/` with its types and source maps, the NAPI loader and binary, the wasm glue JavaScript and module, and this package's README and LICENSE), plus `pnpm-lock.yaml`. So a PR that edits only one of those is measured too. A file a workload loads must therefore ship through a `files` entry, or a change to it passes unmeasured; `.github/scripts/perf-ir-hash-selftest.sh`, run in CI, checks the hash moves for each kind. Four shards split the workloads, each building both trees, balanced on the seconds one run of each workload took in CI (`bench/ir-seconds.json`). A workload missing from that file counts as the median, so add its figure from the gate's `base.json` artifact (`seconds` / `repeats`) when you add a workload, and refresh its row the same way when you change its n or warm-up. Take the figures from CI: a local run's seconds do not scale to the runner's. Runners differ in speed by up to about 2x, and every workload on one runner moves together, so scale a new figure by how that shard's other workloads compare with their rows. A shard's base and head suites each wait for their slowest single run, so keep every workload's run under about 7 minutes in CI. CI runs three repeats, the fewest `ir.mjs` takes, to stay well inside the job's 30 minutes. The verdicts go to one PR comment, edited in place on every push; a fork PR gets the job summary instead, because its token cannot comment.

**Cold start (`pnpm --filter @cachekit-io/cachekit bench:cold-start`).** This times the import of the Node entry and the first three `wrap()` calls in fresh processes, interleaving the arms (default, metrics on, encryption on, and an A/A twin of default). Ops 2 and 3 are the warm baseline for op 1, and they show a cost that merely moved off op 1. `--entry cjs` requires `dist/cjs/index.js` instead of importing `dist/index.js`, and `--latency <ms>` makes every backend call wait that long on a timer, standing in for a network round trip. To A/B a change, copy the base build's `dist` inside `packages/cachekit` (so its dependencies resolve) and pass it as `--base <dir>`: every arm then runs on both builds in the same rotation, and the bench prints base minus head per arm as the median of the per-round differences. Claim only deltas larger than the A/A floor and the min-max band it prints.

Expand Down
64 changes: 32 additions & 32 deletions packages/cachekit/bench/ir-seconds.json
Original file line number Diff line number Diff line change
@@ -1,38 +1,38 @@
{
"l1-get-hit": 36,
"l1-set-at-capacity-1k": 15,
"l1-set-at-capacity-10k": 12,
"l1-set-at-capacity-100k": 10,
"plaintext-set-1kb": 29,
"plaintext-set-10kb": 69,
"plaintext-set-100kb": 191,
"plaintext-set-64b": 27,
"plaintext-set-1mb": 86,
"plaintext-set-containers": 72,
"keygen": 17,
"l1-get-hit": 37,
"l1-set-at-capacity-1k": 27,
"l1-set-at-capacity-10k": 13,
"l1-set-at-capacity-100k": 20,
"plaintext-set-1kb": 35,
"plaintext-set-10kb": 76,
"plaintext-set-100kb": 280,
"plaintext-set-64b": 31,
"plaintext-set-1mb": 88,
"plaintext-set-containers": 75,
"keygen": 18,
"keygen-1kb": 33,
"keygen-wasm": 18,
"keygen-wasm-1kb": 23,
"serializer-roundtrip": 59,
"keygen-wasm": 20,
"keygen-wasm-1kb": 44,
"serializer-roundtrip": 64,
"napi-envelope": 23,
"napi-envelope-40b": 18,
"napi-envelope-64kb": 29,
"wasm-envelope": 24,
"wasm-envelope-40b": 17,
"wasm-envelope-64kb": 39,
"napi-envelope-40b": 17,
"napi-envelope-64kb": 30,
"wasm-envelope": 25,
"wasm-envelope-40b": 18,
"wasm-envelope-64kb": 50,
"napi-encrypted": 18,
"napi-encrypted-64kb": 61,
"wasm-encrypted": 11,
"wasm-encrypted-64kb": 122,
"secure-set-get": 57,
"get-l1-hit": 11,
"wrap-l1-hit": 17,
"wrap-l1-hit-metrics": 12,
"get-l2-hit-metrics": 63,
"secure-wrap-l1-hit-100b": 47,
"secure-wrap-l1-hit-10kb": 83,
"secure-wrap-l1-hit-1mb": 166,
"napi-encrypted-64kb": 77,
"wasm-encrypted": 24,
"wasm-encrypted-64kb": 124,
"secure-set-get": 71,
"get-l1-hit": 14,
"wrap-l1-hit": 18,
"wrap-l1-hit-metrics": 22,
"get-l2-hit-metrics": 62,
"secure-wrap-l1-hit-100b": 49,
"secure-wrap-l1-hit-10kb": 87,
"secure-wrap-l1-hit-1mb": 153,
"wrap-l1-hit-int": 15,
"wrap-l1-hit-cachekitio": 34,
"wrap-l1-hit-int-cachekitio": 23
"wrap-l1-hit-cachekitio": 26,
"wrap-l1-hit-int-cachekitio": 20
}
6 changes: 5 additions & 1 deletion packages/cachekit/bench/ir-workload.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -332,7 +332,11 @@ export const WORKLOADS = {
'plaintext-set-10kb': plaintextSet(320, 1_000),
// 1,300 warm-up ops, not 2,000: V8 has finished compiling the set() path
// around it by ~960 ops, and 2,000 made one process run ~9 minutes in CI.
'plaintext-set-100kb': plaintextSet(3_200, 1_000, 1_300),
// 400 ops, not 1,000: one run of this workload is the longest in the suite,
// and a shard's base and head runs each wait for it, so it sets the gate's
// floor. At 1,000 one run took up to 566 s in CI; 400 cuts it by about a
// third and reads the same Ir/op within 0.02%, repeating within 0.01%.
'plaintext-set-100kb': plaintextSet(3_200, 400, 1_300),
// the ends of the object-heavy write range: one 64 B object, and 1,005,035 B
// in 96 groups of 320 rows, so no collection nears the 10,000-element bound
// and the whole stays under the 1 MiB encode limit. 20 warm-up ops for 1 MB:
Expand Down
Loading