feat: load.instances, spike tolerance and infra.loadgen.count

README.md

@@ -362,7 +362,7 @@ perfkit sweep [--config <perfkit.yaml>] [--run <run-id>] [--older-than <duration
 ```
 
 The backstop for a run that could not tear down, for example a killed CI job. Run it on a
-schedule. It deletes droplets tagged `perfkit` and named `perfkit-<run id>-uut` or `-loadgen`,
+schedule. It deletes droplets tagged `perfkit` and named `perfkit-<run id>-uut`, `-loadgen` or `-worker-<n>`,
 and Kubernetes clusters tagged `perfkit` and named `perfkit-<run id in lowercase>` (with their
 load balancers and volumes), that are older than `--older-than`, and then the firewall, VPC and
 SSH key named `perfkit-<run id>` of those runs. It touches nothing else: not an untagged droplet, and never the

docs/design-loadgen-count.md

@@ -0,0 +1,45 @@
+# Design: several load generator droplets (`infra.loadgen.count`)
+
+Status: implemented in a simpler shape than below: `load.instances` stays the total and is
+spread round robin over the droplets; workers run uac only and their SIPp files are pulled
+back to the first droplet over SSH (their pids are renamed so two machines never collide).
+Not done: sampling the workers, and `sipp-uac@<n>` per-process names.
+
+## Why
+
+One load generator address has about 60,000 TCP source ports per destination, so over TCP it
+caps at `cps × (hold + TIME_WAIT) ≈ 60,000` (for example 500 calls/s with 120 s holds). A very
+large run (a 100M-call challenge: 1,000 to 3,000 calls/s for 10 hours or more) needs more
+addresses, and several load generators also give the load balancer more than one source IP.
+
+## Shape
+
+```yaml
+infra:
+  loadgen: { size: c-16, count: 4 }   # default 1
+```
+
+- `count` droplets, tagged like the load generator today; one is the primary (it runs the
+  agent, the samplers' collection and the callees), the others are workers.
+- Each worker runs its share of the uac: `instances` and the rate are per worker, so a
+  benchmark offers `count × rate`. The users file is split across all `count × instances`
+  processes.
+- Raw results: every worker's trace files are copied into the benchmark's raw directory
+  (distinct names, as SIPp's pid already makes them), and `readSippResults` already merges
+  every pid. The sampler runs on each worker and writes `sampler-loadgen-<n>.jsonl`; the
+  analysis names processes `sipp-uac@<n>` so saturation is judged per process.
+- The callees (register and uas) stay on the primary: Routr routes a call to the address of the
+  REGISTER's top Via.
+
+## What it touches
+
+- `src/infra`: create, tag, wait for and sweep N droplets (the sweep already works by tag and
+  needs no change), the VPC and firewall rules for all of them, `RunState.hosts.loadgenIp`
+  becoming a list.
+- `src/orchestrate`: the agent's `Host` type has `"loadgen"` hard-wired (`run("loadgen", ...)`
+  in `createRunBenchmark`, the sampler start, `collect`); it needs a worker address and a way to
+  fetch each worker's raw directory (the `uut` file fetch in `collectUutFiles` is the model).
+- Live page and the report need nothing: they already sum the merged files.
+
+Cost: N × the load generator's hourly price; a c-16 is about $0.36/h, so `count: 4` adds about
+$1.1/h over one.

docs/perfkit-yaml.md

@@ -45,6 +45,7 @@ Rules for the whole file:
 | `infra.region` | string | yes | | A DigitalOcean region slug, like `nyc3`. Both droplets run there. |
 | `infra.uut.size` | string | yes | | Droplet size slug of the unit under test, a dedicated-CPU size like `c-8`. |
 | `infra.loadgen.size` | string | yes | | Droplet size slug of the load generator, like `c-16`. |
+| `infra.loadgen.count` | integer | no | `1` | Load generator droplets, 1 to 8, all of `size`. The first runs the agent and the callees; the others only run uac SIPp processes. |
 | `infra.cluster` | object | yes, when a benchmark has `uut.helm` | none | The Kubernetes cluster of the helm benchmarks. A run creates it only when a selected benchmark has `uut.helm`. |
 | `infra.cluster.size` | string | yes, if `cluster` is set | | Droplet size slug of the DOKS node pool, like `c-8`. |
 | `infra.cluster.nodes` | integer, 1 to 20 | yes, if `cluster` is set | | Nodes in the pool. With `--local`, the kind cluster's nodes: a control plane, then workers. |
@@ -151,6 +152,7 @@ the other kind rejects them.
 | `load.register.users` | path | yes, if `register` is set | | The agents to register (CSV). |
 | `load.register.expires` | integer (seconds) | no | `300` | The registration lifetime. |
 | `load.uas.scenario` | path | yes, if `uas` is set | | `load.uas` is optional, for either kind: the SIPp scenario that answers calls. |
+| `load.instances` | integer, 1 to 64 | no | `1` | How many SIPp processes place the calls. SIPp runs on one core, so a single process tops out at about 350 calls/s over TCP and a few thousand over UDP, however big the load generator is. With `instances: N`, perfkit starts N uac processes on the load generator, each with 1/N of the rate (`-r`, `-rate_increase`, `-rate_max`) and of the open calls (`-l`), its own local port (5080, 5081, ...) and Call-ID prefix (`uac-`, `uac2-`, ...), and its own share of `uac.users` (every N-th row, so a user belongs to one process). The callees (`register`, `uas`) stay one process. The report sums all of them. A capacity step is invalid when any one instance is over 80% of its core; set `instances` to about the cores you want to use, leaving a few for the callees and the sampler. A local cluster (`--local`, fixed SIPp addresses) accepts only `1`. |
 | `load.hold` | duration | no | `0s` | How long each call stays up before BYE. |
 | `load.unit` | string | no | `calls` | What one SIPp call is, as a lowercase plural noun shown on the page (`calls`, `messages`). Copied into each report.json benchmark as `unit`. |
 | `load.rate.start` | number (calls/s) | *capacity*: yes | | The first step's rate. |
@@ -172,10 +174,11 @@ Thresholds:
 | :--- | :--- | :--- | :--- |
 | `thresholds.p99LatencyMs` | capacity | ms | p99 time from INVITE to 200 OK. A step at or above it breaks the rule. |
 | `thresholds.failedCallsPct` | capacity, soak | % (0 to 100) | Failed calls. |
+| `thresholds.confirmSteps` | capacity | integer 1 to 5, default 2 | How many steps in a row must break a rule to end the ramp. A step that breaks a rule while the next one passes is a spike: the ramp goes on, the step stays in the report flagged `spike`, and it is not the limit. `1` ends the ramp at the first break. A break on the last step counts as confirmed; a step the load generator could not measure ends the ramp at once. |
 | `thresholds.memoryGrowthMBPerHour` | soak | MB/h | Memory growth of each process after the warm-up. |
 | `thresholds.restarts` | soak | count | Process restarts allowed. |
 
-`capacity` requires exactly `p99LatencyMs` and `failedCallsPct`. `soak` requires exactly
+`capacity` requires `p99LatencyMs` and `failedCallsPct`, and takes `confirmSteps`. `soak` requires exactly
 `memoryGrowthMBPerHour`, `failedCallsPct` and `restarts`.
 
 A capacity uac gives up on any message it expects after 10 s (`-recv_timeout 10000`), and a call
@@ -308,3 +311,19 @@ benchmarks:
     telemetry: { otlp: true, sampleRatio: 0.01 }
     thresholds: { memoryGrowthMBPerHour: 5, failedCallsPct: 0.1, restarts: 0 }
 ```
+
+### Scaling the load generator
+
+One c-16 load generator with `load.instances: 12` places about 12 times what one SIPp does.
+Beyond one machine the limits are network and source ports: over TCP each call holds a
+connection (`ip_local_port_range` is 1024-65535 with `tcp_tw_reuse`), so about 60,000
+connections per load generator address, which is `cps × (hold + TIME_WAIT)`.
+
+`infra.loadgen.count` adds load generator droplets, each with its own address and its own
+pool of source ports. `load.instances` stays the total number of uac SIPp processes and the
+rate stays the benchmark's: the instances are spread round robin over the droplets (the
+first on the first), each with its share of the callers. Two c-8 droplets cost about what one
+c-16 does and have twice the ports. The callees stay on the first droplet, and the others'
+SIPp files are copied back to it every 30 s and when the benchmark ends. Limits: the sampler
+watches the first droplet's SIPp processes only, so a saturated worker is not flagged.
+The design is in `docs/design-loadgen-count.md`.

docs/report-v1.md

@@ -115,7 +115,7 @@ Capacity `headline`:
 | `failedAtLimitPct`, `retransmissionsAtLimitPct` | number or null | Percentages at the limit. `null` when `limitCps` is. |
 
 Capacity `stoppedBy`: the step that ended the ramp, `{ offeredCps, rule, measured, limit }`, where
-`rule` is `p99LatencyMs` or `failedCallsPct`. `null` when `limitIsLowerBound` is `true`.
+`rule` is `p99LatencyMs` or `failedCallsPct`. A step may carry `spike: true` (optional, so older reports stay valid): it broke a rule but the next step passed, so it did not end the ramp. `null` when `limitIsLowerBound` is `true`.
 
 Capacity `series.steps`, one entry per step:
 

schemas/live-v1.schema.json

@@ -231,6 +231,11 @@
                   "failedCallsPct": {
                     "type": "number",
                     "minimum": 0
+                  },
+                  "confirmSteps": {
+                    "type": "integer",
+                    "exclusiveMinimum": 0,
+                    "maximum": 9007199254740991
                   }
                 },
                 "required": [

schemas/perfkit-v1.schema.json

@@ -96,6 +96,12 @@
             "size": {
               "type": "string",
               "minLength": 1
+            },
+            "count": {
+              "default": 1,
+              "type": "integer",
+              "minimum": 1,
+              "maximum": 8
             }
           },
           "required": [
@@ -396,6 +402,12 @@
                     ],
                     "additionalProperties": false
                   },
+                  "instances": {
+                    "default": 1,
+                    "type": "integer",
+                    "minimum": 1,
+                    "maximum": 64
+                  },
                   "hold": {
                     "default": "0s",
                     "type": "string",
@@ -512,6 +524,12 @@
                     "type": "number",
                     "minimum": 0,
                     "maximum": 100
+                  },
+                  "confirmSteps": {
+                    "default": 2,
+                    "type": "integer",
+                    "minimum": 1,
+                    "maximum": 5
                   }
                 },
                 "required": [
@@ -734,6 +752,12 @@
                     ],
                     "additionalProperties": false
                   },
+                  "instances": {
+                    "default": 1,
+                    "type": "integer",
+                    "minimum": 1,
+                    "maximum": 64
+                  },
                   "hold": {
                     "default": "0s",
                     "type": "string",

schemas/report-v1.schema.json

@@ -566,6 +566,9 @@
                             },
                             "valid": {
                               "type": "boolean"
+                            },
+                            "spike": {
+                              "type": "boolean"
                             }
                           },
                           "required": [
@@ -866,6 +869,9 @@
                             },
                             "valid": {
                               "type": "boolean"
+                            },
+                            "spike": {
+                              "type": "boolean"
                             }
                           },
                           "required": [

src/analyze/createAnalyzeCapacity.ts

@@ -16,7 +16,8 @@ import { computeHops } from "./computeHops.js";
 import { listRawDir } from "./listRawDir.js";
 import {
   type MeasuredStep,
-  measureCapacityStep
+  measureCapacityStep,
+  uacProcessNames
 } from "./measureCapacityStep.js";
 import { parseJsonDocument } from "./parseJsonDocument.js";
 import { parseOtlpSpanLines } from "./parseOtlpSpans.js";
@@ -26,6 +27,7 @@ import { parseUdpJsonl } from "./parseUdpJsonl.js";
 import { capacityMetaSchema } from "./rawResults.schema.js";
 import { readRawFile } from "./readRawFile.js";
 import { readSippResults } from "./readSippResults.js";
+import { findRampEnd } from "./findRampEnd.js";
 import { roundTo } from "./roundTo.js";
 import { toResources } from "./toResources.js";
 import type { AnalyzeDeps } from "./types.js";
@@ -37,7 +39,8 @@ import type { AnalyzeDeps } from "./types.js";
  * Steps are checked in order. A step whose measurement does not count (the
  * load generator saturated) makes the verdict inconclusive. Otherwise a step
  * passes when p99 < p99LatencyMs and failed % < failedCallsPct, both strict;
- * the first step that breaks a rule ends the ramp. The limit is the offered
+ * the ramp ends where `thresholds.confirmSteps` steps in a row break a rule
+ * (a shorter run followed by a pass is a spike, flagged in the series). The limit is the offered
  * rate of the last step that passed, and the rules are reported at it, or at
  * the first step when none passed. When no step broke a rule the limit is a
  * lower bound.
@@ -85,6 +88,7 @@ export function createAnalyzeCapacity(deps: AnalyzeDeps) {
         samples,
         udp,
         uutProcesses,
+        uacProcesses: uacProcessNames(benchmark.load.instances),
         unit
       })
     );
@@ -112,15 +116,22 @@ export function createAnalyzeCapacity(deps: AnalyzeDeps) {
       durationS: meta.durationS,
       tool: meta.tool
     };
-    const series = { steps: measured.map(({ step }) => step) };
     const resources = toResources(samples, uutProcesses);
 
-    // The ramp ends at the first step that is invalid or breaks a rule.
-    const endIndex = measured.findIndex(
-      (step) =>
-        step.invalidReason !== null ||
-        rulesOf(step).some((rule) => !rule.passed)
+    // The ramp ends at the first invalid step, or where confirmSteps steps in
+    // a row break a rule; a shorter run followed by a pass is a spike
+    const { endIndex, spikes } = findRampEnd(
+      measured.map((step) => ({
+        valid: step.invalidReason === null,
+        broken: rulesOf(step).some((rule) => !rule.passed)
+      })),
+      benchmark.thresholds.confirmSteps
     );
+    const series = {
+      steps: measured.map(({ step }, index) =>
+        spikes.includes(index) ? { ...step, spike: true } : step
+      )
+    };
     const end = endIndex === -1 ? undefined : measured[endIndex];
     if (end?.invalidReason) {
       return {

src/analyze/findRampEnd.ts

@@ -0,0 +1,56 @@
+/**
+ * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License.
+ */
+
+export interface RampStepState {
+  /** False when the step's measurement does not count (load generator gave out). */
+  valid: boolean;
+  /** True when the step broke a rule (p99 or failed calls). */
+  broken: boolean;
+}
+
+export interface RampEnd {
+  /** Index of the step that ends the ramp, or -1 when none does. */
+  endIndex: number;
+  /** Indexes of the steps that broke a rule but were not confirmed. */
+  spikes: number[];
+}
+
+/**
+ * Where a capacity ramp ends. An invalid step ends it at once. A step that
+ * breaks a rule ends it only when `confirmSteps` steps in a row break one
+ * (the run starts at the first of them); a shorter run followed by a step
+ * that passes is a spike: the ramp goes on and the steps are listed in
+ * `spikes`. A run that reaches the last step counts as confirmed, and so does
+ * a run when the ramp stopped for another reason than a passing step.
+ *
+ * @param steps - The steps in order
+ * @param confirmSteps - How many consecutive steps must break a rule
+ */
+export function findRampEnd(
+  steps: readonly RampStepState[],
+  confirmSteps: number
+): RampEnd {
+  const spikes: number[] = [];
+  const scan = (i: number): number => {
+    const step = steps[i];
+    if (!step) return -1;
+    if (!step.valid) return i;
+    if (!step.broken) return scan(i + 1);
+    const runEnd = steps.findIndex(
+      (next, index) => index > i && !(next.valid && next.broken)
+    );
+    const end = runEnd === -1 ? steps.length : runEnd;
+    const runLength = end - i;
+    const ranOut = end === steps.length;
+    const passedAfter = !ranOut && steps[end]?.valid === true;
+    if (runLength >= confirmSteps || ranOut) return i;
+    if (passedAfter) {
+      spikes.push(...Array.from({ length: runLength }, (_, k) => i + k));
+      return scan(end);
+    }
+    // The run is cut by an invalid step, which ends the ramp
+    return end;
+  };
+  return { endIndex: scan(0), spikes };
+}

src/analyze/index.ts

@@ -33,7 +33,9 @@ export {
   MAX_SIPP_CPU_PCT,
   measureCapacityStep,
   MIN_ACHIEVED_RATIO,
-  SIPP_UAC_PROCESS
+  SIPP_UAC_PROCESS,
+  uacProcessName,
+  uacProcessNames
 } from "./measureCapacityStep.js";
 export { nearestRank } from "./nearestRank.js";
 export { parseOtlpSpanLines, parseOtlpSpans } from "./parseOtlpSpans.js";