fix(analyze): judge soak memory growth on a Theil–Sen slope after a settle period

README.md

@@ -506,9 +506,15 @@ On the cluster, a restart is a container restart of a sampled pod (its restart c
   verdict is **aborted** (`only 150 of 200 callees registered within 70 s, so no call was
   placed`). With several callee processes the reason names the one that fell short (`callee 2
   registered 10 of 67 within 64 s, so no call was placed`).
-- **Memory growth.** For each process, the least-squares slope of its memory (RSS) in MB per
-  hour. It fails at the threshold or above: 5.0 MB/h fails a 5 MB/h limit. The headline is the
-  steepest process.
+- **Memory growth.** For each process (or Kubernetes pod), the Theil–Sen slope of its memory
+  (RSS) in MB per hour: the median of the slopes between every pair of samples (an evenly
+  spaced subset of 600 on long runs). The window starts after the warm-up plus a settle period
+  (`settle`, by default 30 minutes, or half of `duration` when that is shorter), so a heap that
+  climbs while it settles after start-up does not count, and ends when the load ends, so the
+  samples taken while it winds down don't either. A dip or a spike moves the median far less
+  than a least-squares fit; a steady leak still shows. It fails at the threshold or above:
+  5.0 MB/h fails a 5 MB/h limit. The headline is the steepest process, and the page says which
+  window the slope covers.
 - **Failed calls.** Failed % over every call after the warm-up. It fails at the threshold or
   above.
 - **Restarts.** A process whose PID changed, or that was missing from a sample, restarted once.

docs/perfkit-yaml.md

@@ -129,6 +129,7 @@ the other kind rejects them.
 | `narrative.how` | string | yes | | How the load is driven and what the verdict checks. |
 | `duration` | duration | *soak*: yes | | How long the load runs. |
 | `warmup` | duration | *soak*: yes; *capacity*: no (`0s`) | | *soak*: excluded from every verdict. *capacity*: the first step still runs its full `every`, but its measurement starts `warmup` after the step starts, so a cold server's first seconds don't count. Shorter than `load.rate.every`. |
+| `settle` | duration | *soak* only: no | `30m`, or half of `duration` when that is shorter | How long after the warm-up memory growth leaves out while the heap settles after start-up. The other rules count from the end of the warm-up. Shorter than `duration`. |
 | `uut` | object | yes | | Either `uut.compose` (with `uut.processes`) on the single server, or `uut.helm` on the cluster. |
 | `uut.compose` | path | yes, without `uut.helm` | | The Compose file the unit under test runs. |
 | `uut.setup` | path | no | none | A Node script that runs once the unit under test is up, for example to seed data. |
@@ -175,7 +176,7 @@ Thresholds:
 | `thresholds.p99LatencyMs` | capacity | ms | p99 time from INVITE to 200 OK. A step at or above it breaks the rule. |
 | `thresholds.failedCallsPct` | capacity, soak | % (0 to 100) | Failed calls. |
 | `thresholds.confirmSteps` | capacity | integer 1 to 5, default 2 | How many steps in a row must break a rule to end the ramp. A step that breaks a rule while the next one passes is a spike: the ramp goes on, the step stays in the report flagged `spike`, and it is not the limit. `1` ends the ramp at the first break. A break on the last step counts as confirmed; a step the load generator could not measure ends the ramp at once. |
-| `thresholds.memoryGrowthMBPerHour` | soak | MB/h | Memory growth of each process after the warm-up. |
+| `thresholds.memoryGrowthMBPerHour` | soak | MB/h | Memory growth of each process: the Theil–Sen slope from the end of the settle period to the end of the load. |
 | `thresholds.restarts` | soak | count | Process restarts allowed. |
 
 `capacity` requires `p99LatencyMs` and `failedCallsPct`, and takes `confirmSteps`. `soak` requires exactly

docs/report-v1.md

@@ -98,10 +98,11 @@ What else a benchmark carries depends on its verdict:
 | :--- | :--- | :--- |
 | `durationS` | number (s) | How long the benchmark ran. |
 | `warmupS` | number (s) | Soak: the warm-up, excluded from every verdict. |
+| `settleS` | number (s) | Soak, optional (older reports have none): the settle period after the warm-up that memory growth leaves out. |
 | `tool` | string | The load tool as shown on the page, like `SIPp 3.7.3, UDP`. |
 | `load.cps`, `load.holdS` | number | Soak: the rate it ran at (calls/s) and call hold time (s). |
 | `load.rateSource` | object | Soak, optional: only with `rate.fromCapacity`. `fromCapacity` (the capacity benchmark), `limit` (its limit in calls/s, or null when it found none or did not run, so `fallbackCps` ran), `fraction`, and `lowerBound: true` when the limit is a lower bound (the ramp reached `max`, or the load generator gave out first). |
-| `rules` | list | One entry per threshold checked: `id` (the threshold's name), `limit`, `measured` and `passed`. A soak's `memoryGrowthMBPerHour` rule has one entry per process, with `process`. |
+| `rules` | list | One entry per threshold checked: `id` (the threshold's name), `limit`, `measured` and `passed`. A soak's `memoryGrowthMBPerHour` rule has one entry per process, with `process`, and `window` (`from`, `to`: the samples its Theil–Sen slope covers, from the end of the warm-up plus the settle period to the end of the load; optional, older reports have none). |
 | `resources.processes` | list | Per process in perfkit.yaml: `name` and `samples`, each `{ t, rssMB, cpuPct, fds, pid }`. |
 | `hops` | object | `sampleRatio`, `spans` (count) and `services`, each `{ service, p50Ms, p99Ms }`. |
 
@@ -184,4 +185,5 @@ start), `estimatedStartAt` (queued only) and `plannedS`. Then by kind:
 - **soak:** `cps`, `warmupS`, `durationS`, `elapsedS`, `thresholds`, `calls` (`placed`, `failed`
   so far, SIPp's counters since the load started) and `memory`: per process, its raw RSS
   (`samples` of `t`, `rssMB`, at most 300, evenly spaced) and `growthMBPerHour`, the verdict's
-  least-squares slope, `null` until there is a full hour of samples after the warm-up.
+  Theil–Sen slope from the end of the warm-up plus the settle period to the latest sample,
+  `null` until there is a full hour of samples after the settle period.

schemas/perfkit-v1.schema.json

@@ -596,6 +596,10 @@
                 "type": "string",
                 "pattern": "^\\d+[smh]$"
               },
+              "settle": {
+                "type": "string",
+                "pattern": "^\\d+[smh]$"
+              },
               "uut": {
                 "anyOf": [
                   {

schemas/report-v1.schema.json

@@ -1123,6 +1123,10 @@
                     "type": "number",
                     "minimum": 0
                   },
+                  "settleS": {
+                    "type": "number",
+                    "minimum": 0
+                  },
                   "tool": {
                     "type": "string",
                     "minLength": 1
@@ -1208,6 +1212,26 @@
                             },
                             "passed": {
                               "type": "boolean"
+                            },
+                            "window": {
+                              "type": "object",
+                              "properties": {
+                                "from": {
+                                  "type": "string",
+                                  "format": "date-time",
+                                  "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z))$"
+                                },
+                                "to": {
+                                  "type": "string",
+                                  "format": "date-time",
+                                  "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z))$"
+                                }
+                              },
+                              "required": [
+                                "from",
+                                "to"
+                              ],
+                              "additionalProperties": false
                             }
                           },
                           "required": [

src/analyze/createAnalyzeSoak.ts

@@ -18,6 +18,7 @@ import { countRestarts } from "./countRestarts.js";
 import { durationToSeconds } from "./durationToSeconds.js";
 import { listRawDir } from "./listRawDir.js";
 import { memoryGrowthMBPerHour } from "./memoryGrowthMBPerHour.js";
+import { memoryGrowthWindow } from "./memoryGrowthWindow.js";
 import { nearestRank } from "./nearestRank.js";
 import { parseJsonDocument } from "./parseJsonDocument.js";
 import { parseOtlpSpanLines } from "./parseOtlpSpans.js";
@@ -27,6 +28,7 @@ import { soakMetaSchema } from "./rawResults.schema.js";
 import { readRawFile } from "./readRawFile.js";
 import { readSippResults } from "./readSippResults.js";
 import { roundTo } from "./roundTo.js";
+import { settleSeconds } from "./settleSeconds.js";
 import { sumStatRows } from "./sumStatRows.js";
 import { toResources } from "./toResources.js";
 import type { AnalyzeDeps, Call } from "./types.js";
@@ -55,7 +57,8 @@ const HOUR_MS = 3_600_000;
  * evaluated. It is aborted too when the uac started no call after the
  * warm-up (nothing was measured, so no rule can pass) or left no response
  * times file (no p99 latency). Otherwise the rules are, per process, memory growth (the
- * least-squares slope of RSS in MB per hour, failing at >= the threshold),
+ * Theil–Sen slope of RSS in MB per hour from the end of the warm-up plus
+ * the settle period to the end of the load, failing at >= the threshold),
  * failed calls (failing at >= the threshold) and restarts (failing above the
  * threshold).
  *
@@ -153,17 +156,31 @@ export function createAnalyzeSoak(deps: AnalyzeDeps) {
     });
     const { thresholds } = benchmark;
 
+    const settleS = settleSeconds(benchmark);
+    const window = memoryGrowthWindow({
+      warmupEndMs,
+      settleS,
+      loadEndMs: Math.min(
+        Date.parse(meta.endedAt),
+        after.at(-1)?.tMs ?? warmupEndMs
+      )
+    });
+    const growthWindow = {
+      from: new Date(window.fromMs).toISOString(),
+      to: new Date(window.toMs).toISOString()
+    };
     const growth = uutProcesses.map((name) => ({
       name,
-      slope: memoryGrowthMBPerHour(after, name, warmupEndMs)
+      slope: memoryGrowthMBPerHour(after, name, window)
     }));
     const rules: SoakRule[] = [
       ...growth.map<SoakRule>(({ name, slope }) => ({
         id: "memoryGrowthMBPerHour",
         process: name,
         limit: thresholds.memoryGrowthMBPerHour,
         measured: slope,
-        passed: slope < thresholds.memoryGrowthMBPerHour
+        passed: slope < thresholds.memoryGrowthMBPerHour,
+        window: growthWindow
       })),
       {
         id: "failedCallsPct",
@@ -200,6 +217,7 @@ export function createAnalyzeSoak(deps: AnalyzeDeps) {
       verdict: rules.every((rule) => rule.passed) ? "passed" : "failed",
       reason: null,
       warmupS,
+      settleS,
       load: meta.load,
       rules,
       headline: {

src/analyze/index.ts

@@ -29,6 +29,9 @@ export { createNodeAnalyzeDeps } from "./createNodeAnalyzeDeps.js";
 export { durationToSeconds } from "./durationToSeconds.js";
 export { leastSquaresSlope } from "./leastSquaresSlope.js";
 export { memoryGrowthMBPerHour } from "./memoryGrowthMBPerHour.js";
+export { memoryGrowthWindow } from "./memoryGrowthWindow.js";
+export { DEFAULT_SETTLE_S, settleSeconds } from "./settleSeconds.js";
+export { theilSenSlope, THEIL_SEN_MAX_POINTS } from "./theilSenSlope.js";
 export {
   calleeProcessName,
   calleeProcessNames,

src/analyze/memoryGrowthMBPerHour.ts

@@ -1,28 +1,33 @@
 /**
  * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License.
  */
-import { leastSquaresSlope } from "./leastSquaresSlope.js";
 import { roundTo } from "./roundTo.js";
+import { theilSenSlope } from "./theilSenSlope.js";
 import type { ProcessSample } from "./types.js";
 
 const HOUR_MS = 3_600_000;
 
 /**
- * One process's memory growth after the warm-up: the least-squares slope of
- * its RSS in MB per hour over the samples from `warmupEndMs` on, rounded as
- * the report rounds it. The soak verdict and the live page both use it.
+ * One process's memory growth: the Theil–Sen slope of its RSS in MB per
+ * hour over the samples in `[fromMs, toMs]`, rounded as the report rounds
+ * it. The soak verdict and the live page both pass `memoryGrowthWindow`.
  */
 export function memoryGrowthMBPerHour(
   samples: ReadonlyArray<Pick<ProcessSample, "name" | "tMs" | "rssKB">>,
   name: string,
-  warmupEndMs: number
+  window: { fromMs: number; toMs: number }
 ): number {
   return roundTo(
-    leastSquaresSlope(
+    theilSenSlope(
       samples
-        .filter((sample) => sample.name === name && sample.tMs >= warmupEndMs)
+        .filter(
+          (sample) =>
+            sample.name === name &&
+            sample.tMs >= window.fromMs &&
+            sample.tMs <= window.toMs
+        )
         .map((sample) => ({
-          x: (sample.tMs - warmupEndMs) / HOUR_MS,
+          x: (sample.tMs - window.fromMs) / HOUR_MS,
           y: sample.rssKB / 1024
         }))
     )

src/analyze/memoryGrowthWindow.ts

@@ -0,0 +1,18 @@
+/**
+ * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License.
+ */
+
+/**
+ * The window memory growth is judged on: from the end of the warm-up plus
+ * the settle period, while the heap settles after start-up, to the end of
+ * the load (samples after it, while the load winds down, don't count). An
+ * empty window at `toMs` when the load ended before the settle period did.
+ */
+export function memoryGrowthWindow(input: {
+  warmupEndMs: number;
+  settleS: number;
+  loadEndMs: number;
+}): { fromMs: number; toMs: number } {
+  const fromMs = input.warmupEndMs + input.settleS * 1000;
+  return { fromMs: Math.min(fromMs, input.loadEndMs), toMs: input.loadEndMs };
+}

src/analyze/settleSeconds.ts

@@ -0,0 +1,21 @@
+/**
+ * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License.
+ */
+import { durationToSeconds } from "./durationToSeconds.js";
+
+/** The settle period when a soak sets none, s. */
+export const DEFAULT_SETTLE_S = 1800;
+
+/**
+ * How long a soak's memory growth ignores after the warm-up, s: its
+ * `settle`, or by default 30 minutes, or half of `duration` when that is
+ * shorter, so a short smoke soak still has samples to fit.
+ */
+export function settleSeconds(benchmark: {
+  settle?: string | undefined;
+  duration: string;
+}): number {
+  return benchmark.settle === undefined
+    ? Math.min(DEFAULT_SETTLE_S, durationToSeconds(benchmark.duration) / 2)
+    : durationToSeconds(benchmark.settle);
+}

src/analyze/theilSenSlope.ts

@@ -0,0 +1,38 @@
+/**
+ * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License.
+ */
+
+/** Above this many points, an evenly spaced subset is used. */
+export const THEIL_SEN_MAX_POINTS = 600;
+
+/**
+ * The Theil–Sen slope of y against x: the median of the slopes of every pair
+ * of points with different x. A dip or a spike moves it far less than a
+ * least-squares fit. Above `THEIL_SEN_MAX_POINTS` points it uses that many,
+ * evenly spaced, so it stays fast (at most about 180,000 pairs). 0 with
+ * fewer than two distinct x.
+ */
+export function theilSenSlope(
+  points: ReadonlyArray<{ x: number; y: number }>
+): number {
+  const step = Math.max(1, points.length / THEIL_SEN_MAX_POINTS);
+  const used =
+    step === 1
+      ? points
+      : Array.from(
+          { length: THEIL_SEN_MAX_POINTS },
+          (_, i) => points[Math.floor(i * step)] as { x: number; y: number }
+        );
+  const slopes = used.flatMap((a, i) =>
+    used
+      .slice(i + 1)
+      .filter((b) => b.x !== a.x)
+      .map((b) => (b.y - a.y) / (b.x - a.x))
+  );
+  if (slopes.length === 0) return 0;
+  const sorted = Float64Array.from(slopes).sort();
+  const mid = Math.floor(sorted.length / 2);
+  return sorted.length % 2 === 1
+    ? (sorted[mid] as number)
+    : ((sorted[mid - 1] as number) + (sorted[mid] as number)) / 2;
+}

src/live/buildLiveSoak.ts

@@ -3,6 +3,8 @@
  */
 import { durationToSeconds } from "../analyze/durationToSeconds.js";
 import { memoryGrowthMBPerHour } from "../analyze/memoryGrowthMBPerHour.js";
+import { memoryGrowthWindow } from "../analyze/memoryGrowthWindow.js";
+import { settleSeconds } from "../analyze/settleSeconds.js";
 import {
   LIVE_MAX_SAMPLES,
   type LiveSoak,
@@ -20,9 +22,10 @@ const iso = (ms: number | null | undefined) =>
 /**
  * A soak's entry in live.json: how far it is, the calls so far, and each
  * process's raw memory, at most 300 samples. Growth in MB/h, the same
- * least-squares slope as the verdict, appears only once there is a full
- * hour of samples after the warm-up: over minutes it overstates growth
- * while the heap settles.
+ * Theil–Sen slope as the verdict (from the end of the warm-up plus the
+ * settle period to the last sample, or the end of the load), appears only
+ * once there is a full hour of samples after the settle period: over
+ * minutes it overstates growth.
  */
 export function buildLiveSoak(input: {
   benchmark: SoakBenchmarkConfig;
@@ -49,22 +52,29 @@ export function buildLiveSoak(input: {
   const warmupEndMs =
     loadStartedMs === null ? null : loadStartedMs + warmupS * 1000;
   const samples = progress?.samples ?? [];
+  const settleS = settleSeconds(benchmark);
   const memory = uutProcessNames(benchmark).map((name) => {
     const own = samples.filter((sample) => sample.name === name);
     const last = own.at(-1);
-    const hourAfterWarmup =
-      warmupEndMs !== null &&
-      last !== undefined &&
-      last.tMs - warmupEndMs >= HOUR_MS;
+    const window =
+      warmupEndMs === null || loadStartedMs === null || last === undefined
+        ? null
+        : memoryGrowthWindow({
+            warmupEndMs,
+            settleS,
+            loadEndMs: Math.min(last.tMs, loadStartedMs + plannedS * 1000)
+          });
+    const hourAfterSettle =
+      window !== null && window.toMs - window.fromMs >= HOUR_MS;
     return {
       name,
       samples: downsample(own, LIVE_MAX_SAMPLES).map((sample) => ({
         t: new Date(sample.tMs).toISOString(),
         rssMB: Math.round((sample.rssKB / 1024) * 10) / 10
       })),
       growthMBPerHour:
-        hourAfterWarmup && warmupEndMs !== null
-          ? memoryGrowthMBPerHour(own, name, warmupEndMs)
+        hourAfterSettle && window !== null
+          ? memoryGrowthMBPerHour(own, name, window)
           : null
     };
   });

src/render/live/renderLiveSoak.ts

@@ -9,12 +9,12 @@ import { liveSectionHead } from "./liveSectionHead.js";
 import type { LiveViewContext } from "./types.js";
 
 export const GROWTH_NOTE =
-  "Memory growth per hour appears once there is a full hour of data after the warm-up. Until then the page shows the raw curve only, because a slope over a few minutes overstates growth while the heap settles.";
+  "Memory growth per hour appears once there is a full hour of data after the warm-up and the settle period. Until then the page shows the raw curve only, because a slope over a few minutes overstates growth while the heap settles. Like the verdict, it is the Theil–Sen slope from the end of the settle period.";
 
 /**
  * A soak on the live page: how far it is and its calls so far, and the
  * raw memory per process with the warm-up shaded and a "now" line. MB/h
- * shows only once a full hour after the warm-up is in.
+ * shows only once a full hour after the warm-up and the settle period is in.
  */
 export function renderLiveSoak(
   benchmark: LiveSoak,