Split callees and the rate among load generators; sample every one; count K8s restarts from pods

AGENTS.md

@@ -84,16 +84,17 @@ Breaking (a major release):
   60,000 source ports, about `cps × (hold + TIME_WAIT)`.
 - SIPp names its result files after its pid (`<scenario>_<pid>_<what>`); the files of another
   load generator get their worker number in front of the pid so two machines never collide.
-- Routr routes a call to the address in the top Via of the callee's REGISTER, so the callees
-  always run on the first load generator.
+- Routr routes a call to the address in the top Via of the callee's REGISTER, so each callee
+  process registers its own share of the users from its own address and port, and calls reach
+  the process that holds the user.
 
 ### performance-engineer
 
 - SIPp is single-threaded: about 350 calls/s per process over TCP, so a bigger load generator
   needs more `load.instances`, not more cores per process.
 - A capacity step is invalid when a SIPp process uses more than 80% of a core or the placed
-  rate is outside 95% to 105% of the offered one. The sampler watches the first load
-  generator's SIPp processes only, not the workers'.
+  rate is outside 95% to 105% of the offered one. Every load generator runs a sampler, and
+  the callees count too (one per instance).
 
 ### release-engineer
 
@@ -118,5 +119,6 @@ Breaking (a major release):
 - 2026-11-09: Every benchmark runs the same load generator layout so results compare:
   `load.instances` is the total SIPp uac processes, spread round robin over the load
   generators (two smaller droplets rather than one big one); benchmarks use 2, the Kubernetes
-  ones 8. The agent and the callees stay on the first; the others only run uac and take
-  commands over SSH.
+  ones 8. The agent stays on the first; the others only run SIPp (uac and callees) and take
+  commands over SSH. There are as many callee processes as instances, each with its share of
+  the users.

README.md

@@ -8,10 +8,10 @@ report page.
 perfkit benchmarks Fonoster projects (Routr first, then Roomr and Fonoster) the same way and
 publishes the results in one format.
 
-- **Two droplets per run.** Each run creates two dedicated-CPU DigitalOcean droplets in the same
-  region and private VPC: the unit under test, and a load generator that drives it with SIPp.
-  The images are pinned by digest when the run starts, and both droplets are deleted on every
-  exit path.
+- **A unit under test and its load generators.** Each run creates dedicated-CPU DigitalOcean
+  droplets in the same region and private VPC: the unit under test, and one to eight load
+  generators (`infra.loadgen.count`) that drive it with SIPp. The images are pinned by digest
+  when the run starts, and every droplet is deleted on every exit path.
 - **A Kubernetes cluster when a benchmark asks for one.** A benchmark with `uut.helm` installs a
   Helm chart on a DOKS cluster created for the run in the same VPC (kind with MetalLB on
   `--local`), and the page labels each benchmark with its target. The cluster is deleted with its
@@ -27,6 +27,54 @@ perfkit/v1`). See [docs/perfkit-yaml.md](docs/perfkit-yaml.md) for every field a
 [docs/report-v1.md](docs/report-v1.md) for `report.json` and `runs.json`. The JSON Schema files
 are in [schemas/](schemas/).
 
+## How it works
+
+`perfkit run` creates the machines and starts an agent on the first load generator. The agent
+runs every benchmark on its own, publishes `live.json` while it runs, and keeps the raw files.
+`perfkit collect` fetches them, writes `report.json` and deletes everything.
+
+```mermaid
+flowchart LR
+  subgraph CI["GitHub Actions or a laptop"]
+    RUN["perfkit run<br/>reads perfkit.yaml"]
+    COL["perfkit collect<br/>analyze, report.json"]
+  end
+
+  subgraph DO["DigitalOcean, one private VPC per run"]
+    direction TB
+    LG1["Load generator 1<br/>agent, sampler,<br/>SIPp callers and callees"]
+    LGN["Load generators 2 to 8<br/>sampler,<br/>SIPp callers and callees"]
+    UUT["Unit under test<br/>one server (Docker Compose)<br/>or a DOKS cluster (Helm)"]
+    LG1 -- "SSH, run-only key" --> LGN
+    LG1 -- SIP --> UUT
+    LGN -- SIP --> UUT
+  end
+
+  RUN -- "1 create droplets, firewall,<br/>cluster; start the agent" --> DO
+  LG1 -- "2 live.json" --> LIVE["Live page"]
+  DO -- "3 raw files" --> COL
+  COL -- "4 delete everything" --> DO
+  COL --> SITE["Report page<br/>perfkit render"]
+```
+
+Every benchmark spreads `load.instances` SIPp callers and as many callees round robin over the
+load generators. The callees register their share of the users, then answer the calls Routr
+routes to them; no load generator talks to another during the load.
+
+```mermaid
+flowchart TD
+  A["Next benchmark"] --> B["Deploy the unit under test<br/>compose up, or helm install and seed"]
+  B --> C{"Every callee<br/>registered its share?"}
+  C -- no --> X["Abort the benchmark"]
+  C -- yes --> K{kind}
+  K -- capacity --> D["Offer a rate for one step<br/>(30 s), then add a step"]
+  D --> F{"p99, failed calls,<br/>any load generator saturated?"}
+  F -- ok --> D
+  F -- "a limit broke twice" --> G["Limit: the last good step"]
+  F -- saturated --> H["Lower bound: the load generator<br/>ran out before the unit under test"]
+  K -- soak --> S["Hold a share of the measured limit<br/>for hours; memory, failed calls, restarts"]
+```
+
 ## Prerequisites
 
 - Node 22 or later, and npm 11 to run perfkit from a git tag (see below).
@@ -401,8 +449,8 @@ failed call is one SIPp counts as failed, and CPU is in % of one core, so 100 is
 ### Capacity
 
 SIPp adds load in steps, such as 50 more calls/s every minute. With callees (`load.register`),
-the ramp starts only once every callee has registered, with the soak's timeout (`users / 20` +
-60 s) and abort reason. The uac gives up on a message it expects after 10 s, and a call that
+the ramp starts only once every callee process has registered its whole share, with the soak's
+timeout (`users / 20` + 60 s) and abort reason. The uac gives up on a message it expects after 10 s, and a call that
 timed out counts as failed; with a hold, a call's teardown failure is counted when it happens, up
 to `hold` + 10 s after the call starts, so `every` should be at least 4 × (hold + 10 s). For each
 step:
@@ -420,7 +468,7 @@ step:
 perfkit checks the steps in order:
 
 1. **Is the measurement valid?** A step is invalid when the load generator, not the server, was
-   the bottleneck: any SIPp role (the uac, or the callees `sipp-register` and `sipp-uas`) used
+   the bottleneck: any SIPp process (a uac instance, or a callee, `sipp-register` and `sipp-uas`) used
    more than 80% of one core, placed less than 95% of the offered rate,
    placed more than 105% of it (a burst of calls it owed from earlier steps), or the load
    generator dropped UDP packets. Then the verdict is **inconclusive**, the reason
@@ -444,7 +492,7 @@ A soak holds a fixed rate for hours to find leaks. With `rate.fromCapacity`, the
 (when it was inconclusive because the load generator gave out, the highest step before that, if
 every one passed, as a lower bound), or `fallbackCps` when it found none; the progress log and
 `load.rateSource` in report.json say which.
-On the cluster, a restart is a container restart of a sampled pod (its restart count went up).
+On the cluster, a restart is a container restart of a sampled pod (its restart count went up) or a pod replaced by another; a sample the Metrics API left out is not a restart.
 
 - **Warm-up excluded.** Nothing measured during `warmup` counts toward any rule.
 - **Too little data.** When the samples after the warm-up cover less than 90% of `duration`, the
@@ -456,7 +504,8 @@ On the cluster, a restart is a container restart of a sampled pod (its restart c
 - **Callees not registered.** With a register role, the uac starts only once every callee has
   registered. When they have not within `users / 20` + 60 seconds, no call is placed and the
   verdict is **aborted** (`only 150 of 200 callees registered within 70 s, so no call was
-  placed`).
+  placed`). With several callee processes the reason names the one that fell short (`callee 2
+  registered 10 of 67 within 64 s, so no call was placed`).
 - **Memory growth.** For each process, the least-squares slope of its memory (RSS) in MB per
   hour. It fails at the threshold or above: 5.0 MB/h fails a 5 MB/h limit. The headline is the
   steepest process.
@@ -544,7 +593,7 @@ it to publish (or clear) a project's site before its first published run:
 
 ```console
 $ npx perfkit render --config perf/perfkit.yaml --out out/benchmarks
-wrote out/benchmarks/app/0.1.12/app.js (perfkit 0.1.12)
+wrote out/benchmarks/app/0.1.13/app.js (perfkit 0.1.13)
 wrote out/benchmarks/data/runs.json (0 runs)
 wrote out/benchmarks/data/site.json (live: https://perfkit.ngrok.app/live.json)
 wrote out/benchmarks/index.html (no runs yet)

docs/design-loadgen-count.md

@@ -20,16 +20,24 @@ infra:
 ```
 
 - `count` droplets, tagged like the load generator today; one is the primary (it runs the
-  agent, the samplers' collection and the callees), the others are workers.
+  agent and the samplers' collection), the others are workers.
 - Each worker runs its share of the uac: `instances` and the rate are per worker, so a
   benchmark offers `count × rate`. The users file is split across all `count × instances`
   processes.
 - Raw results: every worker's trace files are copied into the benchmark's raw directory
   (distinct names, as SIPp's pid already makes them), and `readSippResults` already merges
-  every pid. The sampler runs on each worker and writes `sampler-loadgen-<n>.jsonl`; the
-  analysis names processes `sipp-uac@<n>` so saturation is judged per process.
-- The callees (register and uas) stay on the primary: Routr routes a call to the address of the
-  REGISTER's top Via.
+  every pid. The sampler runs on each worker and writes `sampler-worker-<n>.jsonl` and
+  `udp-worker-<n>.jsonl`, copied back with the rest; process names are one per instance
+  (`sipp-uac-3`, `sipp-register-2`), so saturation is judged per process on any machine,
+  and UDP drops on any machine invalidate the step. The report's `run.infra.loadgen.count`
+  says how many droplets there were.
+- The callees (register and uas) are split like the uac: `instances` callee processes, each
+  registering its share of the users (round robin) from its own port and address, spread over
+  the load generators. Routr routes a call to the address of the REGISTER's top Via, so it goes
+  to the callee that registered the user. Before the ramp every callee must have registered its
+  share, or the benchmark aborts (`callee 2 registered 10 of 67 ...`). One callee process tops
+  out near 2,900 calls/s over UDP (about 17,000 messages/s), which capped RoutrOne's first
+  full run.
 
 ## What it touches
 

docs/perfkit-yaml.md

@@ -45,7 +45,7 @@ Rules for the whole file:
 | `infra.region` | string | yes | | A DigitalOcean region slug, like `nyc3`. Both droplets run there. |
 | `infra.uut.size` | string | yes | | Droplet size slug of the unit under test, a dedicated-CPU size like `c-8`. |
 | `infra.loadgen.size` | string | yes | | Droplet size slug of the load generator, like `c-16`. |
-| `infra.loadgen.count` | integer | no | `1` | Load generator droplets, 1 to 8, all of `size`. The first runs the agent and the callees; the others only run uac SIPp processes. |
+| `infra.loadgen.count` | integer | no | `1` | Load generator droplets, 1 to 8, all of `size`. The first runs the agent; the uac and callee SIPp processes are spread over all of them. |
 | `infra.cluster` | object | yes, when a benchmark has `uut.helm` | none | The Kubernetes cluster of the helm benchmarks. A run creates it only when a selected benchmark has `uut.helm`. |
 | `infra.cluster.size` | string | yes, if `cluster` is set | | Droplet size slug of the DOKS node pool, like `c-8`. |
 | `infra.cluster.nodes` | integer, 1 to 20 | yes, if `cluster` is set | | Nodes in the pool. With `--local`, the kind cluster's nodes: a control plane, then workers. |
@@ -152,7 +152,7 @@ the other kind rejects them.
 | `load.register.users` | path | yes, if `register` is set | | The agents to register (CSV). |
 | `load.register.expires` | integer (seconds) | no | `300` | The registration lifetime. |
 | `load.uas.scenario` | path | yes, if `uas` is set | | `load.uas` is optional, for either kind: the SIPp scenario that answers calls. |
-| `load.instances` | integer, 1 to 64 | no | `1` | How many SIPp processes place the calls. SIPp runs on one core, so a single process tops out at about 350 calls/s over TCP and a few thousand over UDP, however big the load generator is. With `instances: N`, perfkit starts N uac processes on the load generator, each with 1/N of the rate (`-r`, `-rate_increase`, `-rate_max`) and of the open calls (`-l`), its own local port (5080, 5081, ...) and Call-ID prefix (`uac-`, `uac2-`, ...), and its own share of `uac.users` (every N-th row, so a user belongs to one process). The callees (`register`, `uas`) stay one process. The report sums all of them. A capacity step is invalid when any one instance is over 80% of its core; set `instances` to about the cores you want to use, leaving a few for the callees and the sampler. A local cluster (`--local`, fixed SIPp addresses) accepts only `1`. |
+| `load.instances` | integer, 1 to 64 | no | `1` | How many SIPp processes place the calls. SIPp runs on one core, so a single process tops out at about 350 calls/s over TCP and a few thousand over UDP, however big the load generator is. With `instances: N`, perfkit starts N uac processes on the load generator, each with its share of the rate (`-r`, `-rate_increase`, `-rate_max`) in whole numbers that add up to the rate (50 over 8 is six 6s and two 7s, as SIPp's `-rate_increase` takes only integers; `rate.start` must be at least `instances`) and of the open calls (`-l`), its own local port (5080, 5081, ...) and Call-ID prefix (`uac-`, `uac2-`, ...), and its own share of `uac.users` (every N-th row, so a user belongs to one process). With a `register`, the callees are split the same way: `instances` callee processes (`sipp-register`, `sipp-register-2`, ...), each with its own share of `register.users` (every N-th row, so a callee belongs to one process), its own port (5070, then 5200 and up), Call-ID prefix and address; Routr sends a call to the process the callee registered from. A `uas` alone, with no `register`, stays one process. The report sums all of them. A capacity step is invalid when any one instance is over 80% of its core; set `instances` to about the cores you want to use, leaving a few for the callees and the sampler. A local cluster (`--local`, fixed SIPp addresses) accepts only `1`. |
 | `load.hold` | duration | no | `0s` | How long each call stays up before BYE. |
 | `load.unit` | string | no | `calls` | What one SIPp call is, as a lowercase plural noun shown on the page (`calls`, `messages`). Copied into each report.json benchmark as `unit`. |
 | `load.rate.start` | number (calls/s) | *capacity*: yes | | The first step's rate. |
@@ -322,8 +322,9 @@ connections per load generator address, which is `cps × (hold + TIME_WAIT)`.
 `infra.loadgen.count` adds load generator droplets, each with its own address and its own
 pool of source ports. `load.instances` stays the total number of uac SIPp processes and the
 rate stays the benchmark's: the instances are spread round robin over the droplets (the
-first on the first), each with its share of the callers. Two c-8 droplets cost about what one
-c-16 does and have twice the ports. The callees stay on the first droplet, and the others'
-SIPp files are copied back to it every 30 s and when the benchmark ends. Limits: the sampler
-watches the first droplet's SIPp processes only, so a saturated worker is not flagged.
+first on the first), each with its share of the callers, and so are the callees, each with its
+share of the users. Two c-8 droplets cost about what one c-16 does and have twice the ports. The
+others' SIPp files and sampler files are copied back to the first every 30 s and when the
+benchmark ends. Every droplet runs its own sampler, so a SIPp process over 80% of its core, or UDP
+drops, on any of them makes the step invalid.
 The design is in `docs/design-loadgen-count.md`.

docs/report-v1.md

@@ -54,7 +54,7 @@ Rules for both files:
 | `run.startedAt`, `run.endedAt` | timestamp | |
 | `run.infra.provider` | `digitalocean` or `local` | `local` for `perfkit run --local`, where both roles run on one machine's Docker; sizes then read `local`. |
 | `run.infra.region` | string | |
-| `run.infra.uut`, `run.infra.loadgen` | object | `size` (slug), `vcpus` and `memoryMB` of each droplet. |
+| `run.infra.uut`, `run.infra.loadgen` | object | `size` (slug), `vcpus` and `memoryMB` of each droplet. `loadgen` also has `count`, the number of load generator droplets, when there are 2 or more. |
 | `run.infra.os` | string | The droplets' OS, like `Ubuntu 24.04`. |
 | `run.infra.docker` | string | The droplets' Docker version. |
 | `run.infra.cluster` | object | Optional: only when the run had a Kubernetes cluster. `provider` (`digitalocean` or `kind`), `nodes`, `size` (the node slug, `local` for kind) and `version` (like `1.33.1-do.0`). |

package.json

@@ -1,6 +1,6 @@
 {
   "name": "perfkit",
-  "version": "0.1.12",
+  "version": "0.1.13",
   "description": "Benchmark harness for Fonoster projects: DigitalOcean runs, one report.json schema, one branded report page",
   "license": "MIT",
   "author": "Pedro Sanders <psanders@fonoster.com>",

schemas/report-v1.schema.json

@@ -145,6 +145,11 @@
                   "type": "integer",
                   "exclusiveMinimum": 0,
                   "maximum": 9007199254740991
+                },
+                "count": {
+                  "type": "integer",
+                  "minimum": 2,
+                  "maximum": 9007199254740991
                 }
               },
               "required": [

src/analyze/countRestarts.ts

@@ -12,12 +12,36 @@ import type { ProcessSample } from "./types.js";
  * that is a pid change. A tick where the process is missing counts 1; a run
  * of missing ticks counts once, and the pids it comes back with are not
  * counted again.
+ *
+ * On the cluster (`fromCounters`) a name's `pid` is 1 plus the restarts its
+ * pods' counters show (see `toKubeSamplerLines`), so the restarts are the
+ * rises of the pid, and a tick with no sample (the metrics API had no usage
+ * yet) is not a restart: only a counter says a pod restarted.
  */
 export function countRestarts(
   samples: readonly ProcessSample[],
-  processNames: readonly string[]
+  processNames: readonly string[],
+  options: { fromCounters?: boolean } = {}
 ): number {
   const uut = samples.filter((sample) => sample.host === "uut");
+  if (options.fromCounters) {
+    return processNames.reduce((total, name) => {
+      const pids = uut
+        .filter((sample) => sample.name === name)
+        .sort((a, b) => a.tMs - b.tMs)
+        .map((sample) => sample.pid);
+      return (
+        total +
+        pids.reduce(
+          (sum, pid, index) =>
+            index === 0
+              ? sum
+              : sum + Math.max(0, pid - (pids[index - 1] ?? pid)),
+          0
+        )
+      );
+    }, 0);
+  }
   const ticks = [...new Set(uut.map((sample) => sample.tMs))].sort(
     (a, b) => a - b
   );

src/analyze/createAnalyzeCapacity.ts

@@ -3,6 +3,7 @@
  */
 import { withErrorHandlingAndValidation } from "../errors/index.js";
 import {
+  calleeInstancesOf,
   type CapacityBenchmarkResult,
   type CapacityRule,
   type StoppedBy,
@@ -17,6 +18,7 @@ import { listRawDir } from "./listRawDir.js";
 import {
   type MeasuredStep,
   measureCapacityStep,
+  calleeProcessNames,
   uacProcessNames
 } from "./measureCapacityStep.js";
 import { parseJsonDocument } from "./parseJsonDocument.js";
@@ -89,6 +91,7 @@ export function createAnalyzeCapacity(deps: AnalyzeDeps) {
         udp,
         uutProcesses,
         uacProcesses: uacProcessNames(benchmark.load.instances),
+        calleeProcesses: calleeProcessNames(calleeInstancesOf(benchmark.load)),
         unit
       })
     );