diff --git a/README.md b/README.md index ddcd22e..5261e1e 100644 --- a/README.md +++ b/README.md @@ -506,9 +506,15 @@ On the cluster, a restart is a container restart of a sampled pod (its restart c verdict is **aborted** (`only 150 of 200 callees registered within 70 s, so no call was placed`). With several callee processes the reason names the one that fell short (`callee 2 registered 10 of 67 within 64 s, so no call was placed`). -- **Memory growth.** For each process, the least-squares slope of its memory (RSS) in MB per - hour. It fails at the threshold or above: 5.0 MB/h fails a 5 MB/h limit. The headline is the - steepest process. +- **Memory growth.** For each process (or Kubernetes pod), the Theil–Sen slope of its memory + (RSS) in MB per hour: the median of the slopes between every pair of samples (an evenly + spaced subset of 600 on long runs). The window starts after the warm-up plus a settle period + (`settle`, by default 30 minutes, or half of `duration` when that is shorter), so a heap that + climbs while it settles after start-up does not count, and ends when the load ends, so the + samples taken while it winds down don't either. A dip or a spike moves the median far less + than a least-squares fit; a steady leak still shows. It fails at the threshold or above: + 5.0 MB/h fails a 5 MB/h limit. The headline is the steepest process, and the page says which + window the slope covers. - **Failed calls.** Failed % over every call after the warm-up. It fails at the threshold or above. - **Restarts.** A process whose PID changed, or that was missing from a sample, restarted once. diff --git a/docs/perfkit-yaml.md b/docs/perfkit-yaml.md index 3e5f6dd..0cabb3c 100644 --- a/docs/perfkit-yaml.md +++ b/docs/perfkit-yaml.md @@ -129,6 +129,7 @@ the other kind rejects them. | `narrative.how` | string | yes | | How the load is driven and what the verdict checks. | | `duration` | duration | *soak*: yes | | How long the load runs. | | `warmup` | duration | *soak*: yes; *capacity*: no (`0s`) | | *soak*: excluded from every verdict. *capacity*: the first step still runs its full `every`, but its measurement starts `warmup` after the step starts, so a cold server's first seconds don't count. Shorter than `load.rate.every`. | +| `settle` | duration | *soak* only: no | `30m`, or half of `duration` when that is shorter | How long after the warm-up memory growth leaves out while the heap settles after start-up. The other rules count from the end of the warm-up. Shorter than `duration`. | | `uut` | object | yes | | Either `uut.compose` (with `uut.processes`) on the single server, or `uut.helm` on the cluster. | | `uut.compose` | path | yes, without `uut.helm` | | The Compose file the unit under test runs. | | `uut.setup` | path | no | none | A Node script that runs once the unit under test is up, for example to seed data. | @@ -175,7 +176,7 @@ Thresholds: | `thresholds.p99LatencyMs` | capacity | ms | p99 time from INVITE to 200 OK. A step at or above it breaks the rule. | | `thresholds.failedCallsPct` | capacity, soak | % (0 to 100) | Failed calls. | | `thresholds.confirmSteps` | capacity | integer 1 to 5, default 2 | How many steps in a row must break a rule to end the ramp. A step that breaks a rule while the next one passes is a spike: the ramp goes on, the step stays in the report flagged `spike`, and it is not the limit. `1` ends the ramp at the first break. A break on the last step counts as confirmed; a step the load generator could not measure ends the ramp at once. | -| `thresholds.memoryGrowthMBPerHour` | soak | MB/h | Memory growth of each process after the warm-up. | +| `thresholds.memoryGrowthMBPerHour` | soak | MB/h | Memory growth of each process: the Theil–Sen slope from the end of the settle period to the end of the load. | | `thresholds.restarts` | soak | count | Process restarts allowed. | `capacity` requires `p99LatencyMs` and `failedCallsPct`, and takes `confirmSteps`. `soak` requires exactly diff --git a/docs/report-v1.md b/docs/report-v1.md index 993ac6f..94710f7 100644 --- a/docs/report-v1.md +++ b/docs/report-v1.md @@ -98,10 +98,11 @@ What else a benchmark carries depends on its verdict: | :--- | :--- | :--- | | `durationS` | number (s) | How long the benchmark ran. | | `warmupS` | number (s) | Soak: the warm-up, excluded from every verdict. | +| `settleS` | number (s) | Soak, optional (older reports have none): the settle period after the warm-up that memory growth leaves out. | | `tool` | string | The load tool as shown on the page, like `SIPp 3.7.3, UDP`. | | `load.cps`, `load.holdS` | number | Soak: the rate it ran at (calls/s) and call hold time (s). | | `load.rateSource` | object | Soak, optional: only with `rate.fromCapacity`. `fromCapacity` (the capacity benchmark), `limit` (its limit in calls/s, or null when it found none or did not run, so `fallbackCps` ran), `fraction`, and `lowerBound: true` when the limit is a lower bound (the ramp reached `max`, or the load generator gave out first). | -| `rules` | list | One entry per threshold checked: `id` (the threshold's name), `limit`, `measured` and `passed`. A soak's `memoryGrowthMBPerHour` rule has one entry per process, with `process`. | +| `rules` | list | One entry per threshold checked: `id` (the threshold's name), `limit`, `measured` and `passed`. A soak's `memoryGrowthMBPerHour` rule has one entry per process, with `process`, and `window` (`from`, `to`: the samples its Theil–Sen slope covers, from the end of the warm-up plus the settle period to the end of the load; optional, older reports have none). | | `resources.processes` | list | Per process in perfkit.yaml: `name` and `samples`, each `{ t, rssMB, cpuPct, fds, pid }`. | | `hops` | object | `sampleRatio`, `spans` (count) and `services`, each `{ service, p50Ms, p99Ms }`. | @@ -184,4 +185,5 @@ start), `estimatedStartAt` (queued only) and `plannedS`. Then by kind: - **soak:** `cps`, `warmupS`, `durationS`, `elapsedS`, `thresholds`, `calls` (`placed`, `failed` so far, SIPp's counters since the load started) and `memory`: per process, its raw RSS (`samples` of `t`, `rssMB`, at most 300, evenly spaced) and `growthMBPerHour`, the verdict's - least-squares slope, `null` until there is a full hour of samples after the warm-up. + Theil–Sen slope from the end of the warm-up plus the settle period to the latest sample, + `null` until there is a full hour of samples after the settle period. diff --git a/schemas/perfkit-v1.schema.json b/schemas/perfkit-v1.schema.json index 170f745..2c1ee15 100644 --- a/schemas/perfkit-v1.schema.json +++ b/schemas/perfkit-v1.schema.json @@ -596,6 +596,10 @@ "type": "string", "pattern": "^\\d+[smh]$" }, + "settle": { + "type": "string", + "pattern": "^\\d+[smh]$" + }, "uut": { "anyOf": [ { diff --git a/schemas/report-v1.schema.json b/schemas/report-v1.schema.json index 954f4ac..62fe8e2 100644 --- a/schemas/report-v1.schema.json +++ b/schemas/report-v1.schema.json @@ -1123,6 +1123,10 @@ "type": "number", "minimum": 0 }, + "settleS": { + "type": "number", + "minimum": 0 + }, "tool": { "type": "string", "minLength": 1 @@ -1208,6 +1212,26 @@ }, "passed": { "type": "boolean" + }, + "window": { + "type": "object", + "properties": { + "from": { + "type": "string", + "format": "date-time", + "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z))$" + }, + "to": { + "type": "string", + "format": "date-time", + "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z))$" + } + }, + "required": [ + "from", + "to" + ], + "additionalProperties": false } }, "required": [ diff --git a/src/analyze/createAnalyzeSoak.ts b/src/analyze/createAnalyzeSoak.ts index a4bd7f5..b3c56bf 100644 --- a/src/analyze/createAnalyzeSoak.ts +++ b/src/analyze/createAnalyzeSoak.ts @@ -18,6 +18,7 @@ import { countRestarts } from "./countRestarts.js"; import { durationToSeconds } from "./durationToSeconds.js"; import { listRawDir } from "./listRawDir.js"; import { memoryGrowthMBPerHour } from "./memoryGrowthMBPerHour.js"; +import { memoryGrowthWindow } from "./memoryGrowthWindow.js"; import { nearestRank } from "./nearestRank.js"; import { parseJsonDocument } from "./parseJsonDocument.js"; import { parseOtlpSpanLines } from "./parseOtlpSpans.js"; @@ -27,6 +28,7 @@ import { soakMetaSchema } from "./rawResults.schema.js"; import { readRawFile } from "./readRawFile.js"; import { readSippResults } from "./readSippResults.js"; import { roundTo } from "./roundTo.js"; +import { settleSeconds } from "./settleSeconds.js"; import { sumStatRows } from "./sumStatRows.js"; import { toResources } from "./toResources.js"; import type { AnalyzeDeps, Call } from "./types.js"; @@ -55,7 +57,8 @@ const HOUR_MS = 3_600_000; * evaluated. It is aborted too when the uac started no call after the * warm-up (nothing was measured, so no rule can pass) or left no response * times file (no p99 latency). Otherwise the rules are, per process, memory growth (the - * least-squares slope of RSS in MB per hour, failing at >= the threshold), + * Theil–Sen slope of RSS in MB per hour from the end of the warm-up plus + * the settle period to the end of the load, failing at >= the threshold), * failed calls (failing at >= the threshold) and restarts (failing above the * threshold). * @@ -153,9 +156,22 @@ export function createAnalyzeSoak(deps: AnalyzeDeps) { }); const { thresholds } = benchmark; + const settleS = settleSeconds(benchmark); + const window = memoryGrowthWindow({ + warmupEndMs, + settleS, + loadEndMs: Math.min( + Date.parse(meta.endedAt), + after.at(-1)?.tMs ?? warmupEndMs + ) + }); + const growthWindow = { + from: new Date(window.fromMs).toISOString(), + to: new Date(window.toMs).toISOString() + }; const growth = uutProcesses.map((name) => ({ name, - slope: memoryGrowthMBPerHour(after, name, warmupEndMs) + slope: memoryGrowthMBPerHour(after, name, window) })); const rules: SoakRule[] = [ ...growth.map(({ name, slope }) => ({ @@ -163,7 +179,8 @@ export function createAnalyzeSoak(deps: AnalyzeDeps) { process: name, limit: thresholds.memoryGrowthMBPerHour, measured: slope, - passed: slope < thresholds.memoryGrowthMBPerHour + passed: slope < thresholds.memoryGrowthMBPerHour, + window: growthWindow })), { id: "failedCallsPct", @@ -200,6 +217,7 @@ export function createAnalyzeSoak(deps: AnalyzeDeps) { verdict: rules.every((rule) => rule.passed) ? "passed" : "failed", reason: null, warmupS, + settleS, load: meta.load, rules, headline: { diff --git a/src/analyze/index.ts b/src/analyze/index.ts index 5f746c2..227b164 100644 --- a/src/analyze/index.ts +++ b/src/analyze/index.ts @@ -29,6 +29,9 @@ export { createNodeAnalyzeDeps } from "./createNodeAnalyzeDeps.js"; export { durationToSeconds } from "./durationToSeconds.js"; export { leastSquaresSlope } from "./leastSquaresSlope.js"; export { memoryGrowthMBPerHour } from "./memoryGrowthMBPerHour.js"; +export { memoryGrowthWindow } from "./memoryGrowthWindow.js"; +export { DEFAULT_SETTLE_S, settleSeconds } from "./settleSeconds.js"; +export { theilSenSlope, THEIL_SEN_MAX_POINTS } from "./theilSenSlope.js"; export { calleeProcessName, calleeProcessNames, diff --git a/src/analyze/memoryGrowthMBPerHour.ts b/src/analyze/memoryGrowthMBPerHour.ts index fc516c4..1d7f5e5 100644 --- a/src/analyze/memoryGrowthMBPerHour.ts +++ b/src/analyze/memoryGrowthMBPerHour.ts @@ -1,28 +1,33 @@ /** * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. */ -import { leastSquaresSlope } from "./leastSquaresSlope.js"; import { roundTo } from "./roundTo.js"; +import { theilSenSlope } from "./theilSenSlope.js"; import type { ProcessSample } from "./types.js"; const HOUR_MS = 3_600_000; /** - * One process's memory growth after the warm-up: the least-squares slope of - * its RSS in MB per hour over the samples from `warmupEndMs` on, rounded as - * the report rounds it. The soak verdict and the live page both use it. + * One process's memory growth: the Theil–Sen slope of its RSS in MB per + * hour over the samples in `[fromMs, toMs]`, rounded as the report rounds + * it. The soak verdict and the live page both pass `memoryGrowthWindow`. */ export function memoryGrowthMBPerHour( samples: ReadonlyArray>, name: string, - warmupEndMs: number + window: { fromMs: number; toMs: number } ): number { return roundTo( - leastSquaresSlope( + theilSenSlope( samples - .filter((sample) => sample.name === name && sample.tMs >= warmupEndMs) + .filter( + (sample) => + sample.name === name && + sample.tMs >= window.fromMs && + sample.tMs <= window.toMs + ) .map((sample) => ({ - x: (sample.tMs - warmupEndMs) / HOUR_MS, + x: (sample.tMs - window.fromMs) / HOUR_MS, y: sample.rssKB / 1024 })) ) diff --git a/src/analyze/memoryGrowthWindow.ts b/src/analyze/memoryGrowthWindow.ts new file mode 100644 index 0000000..af4bde4 --- /dev/null +++ b/src/analyze/memoryGrowthWindow.ts @@ -0,0 +1,18 @@ +/** + * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. + */ + +/** + * The window memory growth is judged on: from the end of the warm-up plus + * the settle period, while the heap settles after start-up, to the end of + * the load (samples after it, while the load winds down, don't count). An + * empty window at `toMs` when the load ended before the settle period did. + */ +export function memoryGrowthWindow(input: { + warmupEndMs: number; + settleS: number; + loadEndMs: number; +}): { fromMs: number; toMs: number } { + const fromMs = input.warmupEndMs + input.settleS * 1000; + return { fromMs: Math.min(fromMs, input.loadEndMs), toMs: input.loadEndMs }; +} diff --git a/src/analyze/settleSeconds.ts b/src/analyze/settleSeconds.ts new file mode 100644 index 0000000..c8bd5eb --- /dev/null +++ b/src/analyze/settleSeconds.ts @@ -0,0 +1,21 @@ +/** + * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. + */ +import { durationToSeconds } from "./durationToSeconds.js"; + +/** The settle period when a soak sets none, s. */ +export const DEFAULT_SETTLE_S = 1800; + +/** + * How long a soak's memory growth ignores after the warm-up, s: its + * `settle`, or by default 30 minutes, or half of `duration` when that is + * shorter, so a short smoke soak still has samples to fit. + */ +export function settleSeconds(benchmark: { + settle?: string | undefined; + duration: string; +}): number { + return benchmark.settle === undefined + ? Math.min(DEFAULT_SETTLE_S, durationToSeconds(benchmark.duration) / 2) + : durationToSeconds(benchmark.settle); +} diff --git a/src/analyze/theilSenSlope.ts b/src/analyze/theilSenSlope.ts new file mode 100644 index 0000000..99dd9ea --- /dev/null +++ b/src/analyze/theilSenSlope.ts @@ -0,0 +1,38 @@ +/** + * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. + */ + +/** Above this many points, an evenly spaced subset is used. */ +export const THEIL_SEN_MAX_POINTS = 600; + +/** + * The Theil–Sen slope of y against x: the median of the slopes of every pair + * of points with different x. A dip or a spike moves it far less than a + * least-squares fit. Above `THEIL_SEN_MAX_POINTS` points it uses that many, + * evenly spaced, so it stays fast (at most about 180,000 pairs). 0 with + * fewer than two distinct x. + */ +export function theilSenSlope( + points: ReadonlyArray<{ x: number; y: number }> +): number { + const step = Math.max(1, points.length / THEIL_SEN_MAX_POINTS); + const used = + step === 1 + ? points + : Array.from( + { length: THEIL_SEN_MAX_POINTS }, + (_, i) => points[Math.floor(i * step)] as { x: number; y: number } + ); + const slopes = used.flatMap((a, i) => + used + .slice(i + 1) + .filter((b) => b.x !== a.x) + .map((b) => (b.y - a.y) / (b.x - a.x)) + ); + if (slopes.length === 0) return 0; + const sorted = Float64Array.from(slopes).sort(); + const mid = Math.floor(sorted.length / 2); + return sorted.length % 2 === 1 + ? (sorted[mid] as number) + : ((sorted[mid - 1] as number) + (sorted[mid] as number)) / 2; +} diff --git a/src/live/buildLiveSoak.ts b/src/live/buildLiveSoak.ts index 98678c3..c4bf45d 100644 --- a/src/live/buildLiveSoak.ts +++ b/src/live/buildLiveSoak.ts @@ -3,6 +3,8 @@ */ import { durationToSeconds } from "../analyze/durationToSeconds.js"; import { memoryGrowthMBPerHour } from "../analyze/memoryGrowthMBPerHour.js"; +import { memoryGrowthWindow } from "../analyze/memoryGrowthWindow.js"; +import { settleSeconds } from "../analyze/settleSeconds.js"; import { LIVE_MAX_SAMPLES, type LiveSoak, @@ -20,9 +22,10 @@ const iso = (ms: number | null | undefined) => /** * A soak's entry in live.json: how far it is, the calls so far, and each * process's raw memory, at most 300 samples. Growth in MB/h, the same - * least-squares slope as the verdict, appears only once there is a full - * hour of samples after the warm-up: over minutes it overstates growth - * while the heap settles. + * Theil–Sen slope as the verdict (from the end of the warm-up plus the + * settle period to the last sample, or the end of the load), appears only + * once there is a full hour of samples after the settle period: over + * minutes it overstates growth. */ export function buildLiveSoak(input: { benchmark: SoakBenchmarkConfig; @@ -49,13 +52,20 @@ export function buildLiveSoak(input: { const warmupEndMs = loadStartedMs === null ? null : loadStartedMs + warmupS * 1000; const samples = progress?.samples ?? []; + const settleS = settleSeconds(benchmark); const memory = uutProcessNames(benchmark).map((name) => { const own = samples.filter((sample) => sample.name === name); const last = own.at(-1); - const hourAfterWarmup = - warmupEndMs !== null && - last !== undefined && - last.tMs - warmupEndMs >= HOUR_MS; + const window = + warmupEndMs === null || loadStartedMs === null || last === undefined + ? null + : memoryGrowthWindow({ + warmupEndMs, + settleS, + loadEndMs: Math.min(last.tMs, loadStartedMs + plannedS * 1000) + }); + const hourAfterSettle = + window !== null && window.toMs - window.fromMs >= HOUR_MS; return { name, samples: downsample(own, LIVE_MAX_SAMPLES).map((sample) => ({ @@ -63,8 +73,8 @@ export function buildLiveSoak(input: { rssMB: Math.round((sample.rssKB / 1024) * 10) / 10 })), growthMBPerHour: - hourAfterWarmup && warmupEndMs !== null - ? memoryGrowthMBPerHour(own, name, warmupEndMs) + hourAfterSettle && window !== null + ? memoryGrowthMBPerHour(own, name, window) : null }; }); diff --git a/src/render/live/renderLiveSoak.ts b/src/render/live/renderLiveSoak.ts index 3c226f4..243b389 100644 --- a/src/render/live/renderLiveSoak.ts +++ b/src/render/live/renderLiveSoak.ts @@ -9,12 +9,12 @@ import { liveSectionHead } from "./liveSectionHead.js"; import type { LiveViewContext } from "./types.js"; export const GROWTH_NOTE = - "Memory growth per hour appears once there is a full hour of data after the warm-up. Until then the page shows the raw curve only, because a slope over a few minutes overstates growth while the heap settles."; + "Memory growth per hour appears once there is a full hour of data after the warm-up and the settle period. Until then the page shows the raw curve only, because a slope over a few minutes overstates growth while the heap settles. Like the verdict, it is the Theil–Sen slope from the end of the settle period."; /** * A soak on the live page: how far it is and its calls so far, and the * raw memory per process with the warm-up shaded and a "now" line. MB/h - * shows only once a full hour after the warm-up is in. + * shows only once a full hour after the warm-up and the settle period is in. */ export function renderLiveSoak( benchmark: LiveSoak, diff --git a/src/render/sections/renderSoakSection.ts b/src/render/sections/renderSoakSection.ts index 19a53e1..976712f 100644 --- a/src/render/sections/renderSoakSection.ts +++ b/src/render/sections/renderSoakSection.ts @@ -13,7 +13,8 @@ import { formatMs, formatNumber, formatPct, - formatRate + formatRate, + formatTime } from "../format.js"; import { esc } from "../html.js"; import { targetLabel } from "../targetLabel.js"; @@ -114,6 +115,22 @@ function failedClause(rule: SoakRule, unit: string): string { return `${formatNumber(rule.measured, 0)} restarts (limit: ${formatNumber(rule.limit, 0)})`; } +/** + * Which samples the memory growth slope covers, from the rules' `window`; + * nothing for reports that predate it. + */ +function growthWindowNote(benchmark: MeasuredSoak): string { + const rule = benchmark.rules.find( + (r) => r.id === "memoryGrowthMBPerHour" && r.window !== undefined + ); + if (rule?.id !== "memoryGrowthMBPerHour" || !rule.window) return ""; + const settle = + benchmark.settleS === undefined + ? "" + : `, after a ${formatDuration(benchmark.settleS)} settle`; + return `

Memory growth: Theil–Sen slope from ${esc(formatTime(rule.window.from))} to ${esc(formatTime(rule.window.to))} UTC${esc(settle)}

`; +} + function measured(benchmark: MeasuredSoak): string { const { unit } = benchmark; const failed = benchmark.rules.filter((r) => !r.passed); @@ -154,7 +171,7 @@ ${narrativeCards(benchmark)} ${chart}

Stability checks

-

after the ${esc(formatDuration(benchmark.warmupS))} warm-up

+

after the ${esc(formatDuration(benchmark.warmupS))} warm-up

${growthWindowNote(benchmark)}
${checks}
diff --git a/src/schemas/checkSettleShorterThanDuration.ts b/src/schemas/checkSettleShorterThanDuration.ts new file mode 100644 index 0000000..e62be59 --- /dev/null +++ b/src/schemas/checkSettleShorterThanDuration.ts @@ -0,0 +1,26 @@ +/** + * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. + */ +import type { z } from "zod/v4"; +import { durationSeconds } from "./durationSeconds.js"; + +/** + * Adds an issue at `settle` when a soak's settle period is not shorter than + * its `duration`: memory growth would have no samples left to fit. No + * issue without one (the default fits any duration). For use + * inside the soak benchmark's `superRefine`. + */ +export function checkSettleShorterThanDuration( + benchmark: { settle?: string | undefined; duration: string }, + ctx: z.core.$RefinementCtx +): void { + if (benchmark.settle === undefined) return; + if (durationSeconds(benchmark.settle) >= durationSeconds(benchmark.duration)) { + ctx.addIssue({ + code: "custom", + path: ["settle"], + message: `expected a settle period shorter than duration (${benchmark.duration})`, + input: benchmark.settle + }); + } +} diff --git a/src/schemas/perfkitConfig.schema.ts b/src/schemas/perfkitConfig.schema.ts index 9494621..a9c77d1 100644 --- a/src/schemas/perfkitConfig.schema.ts +++ b/src/schemas/perfkitConfig.schema.ts @@ -14,6 +14,7 @@ import { checkUniqueIds } from "./checkUniqueIds.js"; import { projectLiveSchema } from "./report.schema.js"; import { checkStartCoversInstances } from "./checkStartCoversInstances.js"; import { checkStepEveryMultipleOfSippInterval } from "./checkStepEveryMultipleOfSippInterval.js"; +import { checkSettleShorterThanDuration } from "./checkSettleShorterThanDuration.js"; import { checkWarmupShorterThanStep } from "./checkWarmupShorterThanStep.js"; import { benchmarkIdSchema, @@ -341,18 +342,24 @@ export const capacityBenchmarkConfigSchema = z .superRefine(checkStepEveryMultipleOfSippInterval) .superRefine(checkStartCoversInstances); -export const soakBenchmarkConfigSchema = z.strictObject({ - id: benchmarkIdSchema, - kind: z.literal("soak"), - title: nonEmptyString, - narrative: narrativeSchema, - duration: durationSchema, - warmup: durationSchema, - uut: configUutSchema, - load: soakLoadSchema, - telemetry: telemetrySchema, - thresholds: soakThresholdsSchema -}); +export const soakBenchmarkConfigSchema = z + .strictObject({ + id: benchmarkIdSchema, + kind: z.literal("soak"), + title: nonEmptyString, + narrative: narrativeSchema, + duration: durationSchema, + warmup: durationSchema, + // Memory growth ignores this long after the warm-up while the heap + // settles; the other rules don't. Default: 30m, or half of duration + // when that is shorter (settleSeconds). + settle: durationSchema.optional(), + uut: configUutSchema, + load: soakLoadSchema, + telemetry: telemetrySchema, + thresholds: soakThresholdsSchema + }) + .superRefine(checkSettleShorterThanDuration); export const benchmarkConfigSchema = z.discriminatedUnion("kind", [ capacityBenchmarkConfigSchema, diff --git a/src/schemas/report.schema.ts b/src/schemas/report.schema.ts index 2c17623..3d0038a 100644 --- a/src/schemas/report.schema.ts +++ b/src/schemas/report.schema.ts @@ -159,7 +159,15 @@ export const soakRuleSchema = z.discriminatedUnion("id", [ z.strictObject({ id: z.literal("memoryGrowthMBPerHour"), process: nonEmptyString, - ...ruleFields + ...ruleFields, + /** + * The samples the slope covers: from the end of the warm-up plus the + * settle period to the end of the load. Optional: older reports have + * none. + */ + window: z + .strictObject({ from: timestampSchema, to: timestampSchema }) + .optional() }), z.strictObject({ id: z.enum(["failedCallsPct", "restarts"]), @@ -326,6 +334,11 @@ const soakMeasuredSchema = z reason: z.null(), durationS: nonNegativeNumber, warmupS: nonNegativeNumber, + /** + * How long memory growth ignored after the warm-up. Optional: older + * reports have none. + */ + settleS: nonNegativeNumber.optional(), tool: nonEmptyString, load: z.strictObject({ cps: positiveNumber, diff --git a/test/analyze/builders.ts b/test/analyze/builders.ts index a9757f9..762c1c9 100644 --- a/test/analyze/builders.ts +++ b/test/analyze/builders.ts @@ -97,6 +97,7 @@ export function soakConfig( overrides: { duration?: string; warmup?: string; + settle?: string; memoryGrowthMBPerHour?: number; otlp?: boolean; } = {} @@ -108,6 +109,7 @@ export function soakConfig( narrative, duration: overrides.duration ?? "2h", warmup: overrides.warmup ?? "30m", + ...(overrides.settle ? { settle: overrides.settle } : {}), uut: { compose: "one-soak/compose.yaml", processes: [ @@ -361,6 +363,8 @@ export interface SoakRawSpec { warmupS: number; /** How long the samples go on after the warm-up, s. */ afterS: number; + /** Samples taken after the load ended (meta.endedAt), s. Default 0. */ + tailS?: number; /** * A process's reading at `hours` since the warm-up ended (negative during * it), or null when the sampler did not find it. Default: pid 41, 100 MB. @@ -386,7 +390,8 @@ export function soakRaw(spec: SoakRawSpec): Record { const warmupEndMs = START_MS + spec.warmupS * 1000; const endMs = warmupEndMs + spec.afterS * 1000; const ticks = Math.floor((endMs - START_MS) / SOAK_TICK_MS); - const sampler = Array.from({ length: ticks + 1 }, (_, i) => { + const tailTicks = Math.floor(((spec.tailS ?? 0) * 1000) / SOAK_TICK_MS); + const sampler = Array.from({ length: ticks + tailTicks + 1 }, (_, i) => { const tMs = START_MS + i * SOAK_TICK_MS; const hours = (tMs - warmupEndMs) / 3_600_000; return samplerLines( diff --git a/test/analyze/createAnalyzeSoak.unit.test.ts b/test/analyze/createAnalyzeSoak.unit.test.ts index 8bd262b..b842603 100644 --- a/test/analyze/createAnalyzeSoak.unit.test.ts +++ b/test/analyze/createAnalyzeSoak.unit.test.ts @@ -5,7 +5,10 @@ import { expect } from "chai"; import { readFileSync } from "node:fs"; import { join } from "node:path"; import sinon from "sinon"; -import { createAnalyzeSoak } from "../../src/analyze/index.js"; +import { + createAnalyzeSoak, + leastSquaresSlope +} from "../../src/analyze/index.js"; import { ValidationError } from "../../src/errors/index.js"; import { soakBenchmarkResultSchema, @@ -23,6 +26,11 @@ import { } from "./builders.js"; const MB = 1024; +/** 2 h after a 30 min warm-up from 14:00, less the 30 min settle period. */ +const WINDOW = { + from: "2026-11-03T15:00:00.000Z", + to: "2026-11-03T16:30:00.000Z" +}; async function analyze( spec: Partial = {}, @@ -74,21 +82,24 @@ describe("createAnalyzeSoak", () => { process: "one", limit: 5, measured: 0, - passed: true + passed: true, + window: WINDOW }, { id: "memoryGrowthMBPerHour", process: "edgeport", limit: 5, measured: 0, - passed: true + passed: true, + window: WINDOW }, { id: "memoryGrowthMBPerHour", process: "postgres", limit: 5, measured: 0, - passed: true + passed: true, + window: WINDOW }, { id: "failedCallsPct", limit: 0.1, measured: 0, passed: true }, { id: "restarts", limit: 0, measured: 0, passed: true } @@ -117,11 +128,128 @@ describe("createAnalyzeSoak", () => { process: "edgeport", limit: 5, measured: 6, - passed: false + passed: false, + window: WINDOW } ]); }); + it("should pass RSS that climbs for 15 min after the warm-up, then stays flat within ±25 MB", async () => { + // Arrange: start-up climbs 2,200 to 2,310 MB in 15 min (the shape of + // Routr run 42699126885), then ±25 MB of noise; over the whole window + // the slope is about 20 MB/h + const noise = (hours: number) => + 25 * Math.sin(hours * 60 * 2.399) * Math.cos(hours * 60 * 0.7); + const rssAt = (hours: number) => + hours < 0.25 ? 2200 + Math.max(0, hours) * 440 : 2310 + noise(hours); + + // Act + const result = measured( + await analyze({ + processAt: (name, hours) => ({ + rssKB: Math.round((name === "one" ? rssAt(hours) : 100) * MB) + }) + }) + ); + + // Assert + const one = result.rules[0]; + expect(one).to.include({ process: "one", passed: true }); + expect(Math.abs(one?.measured ?? Infinity)).to.be.below(5); + expect(result.verdict).to.equal("passed"); + }); + + it("should fail a steady 10 MB/h rise", async () => { + // Act + const result = measured( + await analyze({ + processAt: (name, hours) => ({ + rssKB: Math.round((name === "one" ? 2300 + 10 * hours : 100) * MB) + }) + }) + ); + + // Assert + expect(result.verdict).to.equal("failed"); + expect(result.rules[0]).to.deep.equal({ + id: "memoryGrowthMBPerHour", + process: "one", + limit: 5, + measured: 10, + passed: false, + window: WINDOW + }); + }); + + it("should pass Routr run 42699126885's RoutrOne: start-up climb, then flat with a dip", async () => { + // Arrange: its RSS once a minute; it climbs until about minute 25, + // then stays at 2,280 to 2,352 MB with a dip at minutes 60 to 80 + const { rssMB } = JSON.parse( + readFileSync( + join(FIXTURES, "soak", "routr-42699126885-one-rss.json"), + "utf8" + ) + ) as { rssMB: number[] }; + const rssAt = (hours: number) => + rssMB[Math.min(130, Math.max(0, Math.round(hours * 60 + 10)))] ?? 0; + const wholeWindow = leastSquaresSlope( + rssMB.slice(10).map((y, i) => ({ x: i / 60, y })) + ); + + // Act + const result = measured( + await analyze( + { + warmupS: 600, + afterS: 7200, + processAt: (name, hours) => ({ + rssKB: Math.round((name === "one" ? rssAt(hours) : 100) * MB) + }) + }, + soakConfig({ warmup: "10m" }) + ) + ); + + // Assert: a least-squares fit from the warm-up on failed it + expect(wholeWindow).to.be.above(5); + expect(result.rules[0]).to.include({ process: "one", passed: true }); + expect(Math.abs(result.rules[0]?.measured ?? Infinity)).to.be.below(1); + }); + + it("should leave samples after the end of the load out of the growth", async () => { + // Arrange: flat during the load, then 500 MB/h for 10 min after it + const result = measured( + await analyze({ + tailS: 600, + processAt: (name, hours) => ({ + rssKB: Math.round((100 + Math.max(0, hours - 2) * 500) * MB) + }) + }) + ); + + // Assert + expect(result.rules[0]).to.deep.equal({ + id: "memoryGrowthMBPerHour", + process: "one", + limit: 5, + measured: 0, + passed: true, + window: WINDOW + }); + }); + + it("should start the window after the soak's own settle period", async () => { + // Act + const result = measured(await analyze({}, soakConfig({ settle: "45m" }))); + + // Assert + expect(result.settleS).to.equal(2700); + expect(result.rules[0]).to.have.deep.property("window", { + from: "2026-11-03T15:15:00.000Z", + to: "2026-11-03T16:30:00.000Z" + }); + }); + it("should fail at exactly 5.0 MB/h", async () => { // Act const result = measured( @@ -344,9 +472,9 @@ describe("createAnalyzeSoak", () => { soakConfig({ duration: "20m", warmup: "5m" }) ); - // Assert + // Assert: the default settle period is half of a short duration expect(result.verdict).to.equal("passed"); - expect(result).to.have.property("rules"); + expect(result).to.have.property("settleS", 600); }); it("should compute hops from the spans after the warm-up", async () => { diff --git a/test/analyze/memoryGrowthMBPerHour.unit.test.ts b/test/analyze/memoryGrowthMBPerHour.unit.test.ts index f3535a5..5de8a75 100644 --- a/test/analyze/memoryGrowthMBPerHour.unit.test.ts +++ b/test/analyze/memoryGrowthMBPerHour.unit.test.ts @@ -4,18 +4,22 @@ import { expect } from "chai"; import { memoryGrowthMBPerHour } from "../../src/analyze/index.js"; +const HOUR_MS = 3_600_000; + describe("memoryGrowthMBPerHour", () => { - it("should give one process's least-squares MB/h after the warm-up", () => { - // Arrange: 1 MB per hour after t=1000, a spike before it + it("should give one process's least-squares MB/h over the window", () => { + // Arrange: 1 MB per hour in [1000, 1000 + 1 h], spikes outside it const samples = [ { name: "one", tMs: 0, rssKB: 999_999 }, { name: "one", tMs: 1000, rssKB: 1024 }, - { name: "one", tMs: 1000 + 3_600_000, rssKB: 2048 }, + { name: "one", tMs: 1000 + HOUR_MS, rssKB: 2048 }, + { name: "one", tMs: 2000 + HOUR_MS, rssKB: 999_999 }, { name: "two", tMs: 2000, rssKB: 9 } ]; + const window = { fromMs: 1000, toMs: 1000 + HOUR_MS }; // Act & Assert - expect(memoryGrowthMBPerHour(samples, "one", 1000)).to.equal(1); - expect(memoryGrowthMBPerHour(samples, "two", 1000)).to.equal(0); + expect(memoryGrowthMBPerHour(samples, "one", window)).to.equal(1); + expect(memoryGrowthMBPerHour(samples, "two", window)).to.equal(0); }); }); diff --git a/test/analyze/memoryGrowthWindow.unit.test.ts b/test/analyze/memoryGrowthWindow.unit.test.ts new file mode 100644 index 0000000..4f89af8 --- /dev/null +++ b/test/analyze/memoryGrowthWindow.unit.test.ts @@ -0,0 +1,19 @@ +/** + * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. + */ +import { expect } from "chai"; +import { memoryGrowthWindow } from "../../src/analyze/index.js"; + +describe("memoryGrowthWindow", () => { + it("should start after the settle period and end at the end of the load", () => { + expect( + memoryGrowthWindow({ warmupEndMs: 1000, settleS: 2, loadEndMs: 9000 }) + ).to.deep.equal({ fromMs: 3000, toMs: 9000 }); + }); + + it("should be empty when the load ended within the settle period", () => { + expect( + memoryGrowthWindow({ warmupEndMs: 1000, settleS: 10, loadEndMs: 5000 }) + ).to.deep.equal({ fromMs: 5000, toMs: 5000 }); + }); +}); diff --git a/test/analyze/settleSeconds.unit.test.ts b/test/analyze/settleSeconds.unit.test.ts new file mode 100644 index 0000000..ad22c0b --- /dev/null +++ b/test/analyze/settleSeconds.unit.test.ts @@ -0,0 +1,19 @@ +/** + * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. + */ +import { expect } from "chai"; +import { DEFAULT_SETTLE_S, settleSeconds } from "../../src/analyze/index.js"; + +describe("settleSeconds", () => { + it("should take the soak's settle", () => { + expect(settleSeconds({ settle: "45m", duration: "2h" })).to.equal(2700); + }); + + it("should default to 30 minutes", () => { + expect(settleSeconds({ duration: "8h" })).to.equal(DEFAULT_SETTLE_S); + }); + + it("should default to half of a duration shorter than an hour", () => { + expect(settleSeconds({ duration: "20m" })).to.equal(600); + }); +}); diff --git a/test/analyze/theilSenSlope.unit.test.ts b/test/analyze/theilSenSlope.unit.test.ts new file mode 100644 index 0000000..ec40d25 --- /dev/null +++ b/test/analyze/theilSenSlope.unit.test.ts @@ -0,0 +1,60 @@ +/** + * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. + */ +import { expect } from "chai"; +import { + leastSquaresSlope, + THEIL_SEN_MAX_POINTS, + theilSenSlope +} from "../../src/analyze/index.js"; + +const line = (n: number, slope: number) => + Array.from({ length: n }, (_, i) => ({ x: i, y: 10 + slope * i })); + +describe("theilSenSlope", () => { + it("should give the slope of points on a line", () => { + expect(theilSenSlope(line(11, 2))).to.equal(2); + }); + + it("should give the median of the pairwise slopes, with an even count", () => { + // Pairs: (0,0)-(1,1): 1, (0,0)-(2,4): 2, (1,1)-(2,4): 3; with (3,9): 3, 4, 5 + const points = [0, 1, 2, 3].map((x) => ({ x, y: x * x })); + expect(theilSenSlope(points)).to.equal(3); + }); + + it("should barely move for a dip that pulls a least-squares fit", () => { + // Arrange: flat at 100, the first quarter dips to 90 + const points = Array.from({ length: 100 }, (_, x) => ({ + x, + y: x < 25 ? 90 : 100 + })); + + // Act & Assert + expect(theilSenSlope(points)).to.equal(0); + expect(leastSquaresSlope(points)).to.be.above(0.1); + }); + + it("should subsample a long series and still find its slope", () => { + // Arrange + const points = line(THEIL_SEN_MAX_POINTS * 5 + 3, 0.5); + + // Act + const started = Date.now(); + const slope = theilSenSlope(points); + + // Assert + expect(slope).to.be.closeTo(0.5, 1e-9); + expect(Date.now() - started).to.be.below(2000); + }); + + it("should give 0 with fewer than two distinct x", () => { + expect(theilSenSlope([])).to.equal(0); + expect(theilSenSlope([{ x: 1, y: 1 }])).to.equal(0); + expect( + theilSenSlope([ + { x: 1, y: 1 }, + { x: 1, y: 5 } + ]) + ).to.equal(0); + }); +}); diff --git a/test/fixtures/soak/routr-42699126885-one-rss.json b/test/fixtures/soak/routr-42699126885-one-rss.json new file mode 100644 index 0000000..85a1196 --- /dev/null +++ b/test/fixtures/soak/routr-42699126885-one-rss.json @@ -0,0 +1,21 @@ +{ + "run": "fonoster/routr actions run 42699126885, one-soak, process one", + "unit": "MB, one sample per minute from the load start (warm-up 10 min, load end at minute 130)", + "rssMB": [ + 69.8, 1256.3, 1433.8, 1805.7, 1897.3, 1997.8, 2072.3, 2116.4, 2159.1, + 2159.7, 2191.9, 2232.2, 2233.2, 2233.2, 2262.4, 2263.4, 2263.6, 2300.5, + 2300.5, 2301, 2280.6, 2280.6, 2280.6, 2280.6, 2301.2, 2301.2, 2301.2, + 2314.7, 2314.7, 2314.7, 2315.2, 2305.7, 2305.7, 2305.7, 2305.7, 2305.7, + 2305.7, 2305.7, 2305.7, 2305.7, 2305.7, 2329.9, 2329.9, 2329.9, 2329.9, + 2329.9, 2325.6, 2325.6, 2325.6, 2325.6, 2325.6, 2325.6, 2325.6, 2325.6, + 2325.6, 2325.6, 2325.6, 2325.6, 2325.6, 2325.6, 2325.6, 2312, 2328, 2329.9, + 2325.6, 2325.6, 2325.6, 2301.6, 2304.4, 2290.5, 2296.7, 2296.7, 2296.7, + 2296.7, 2296.7, 2296.7, 2296.7, 2296.7, 2280.5, 2280.5, 2280.5, 2280.5, + 2280.5, 2306.6, 2306.6, 2306.6, 2306.6, 2306.6, 2337.7, 2329.7, 2329.7, + 2329.7, 2314.7, 2314.7, 2314.7, 2314.7, 2322.7, 2322.7, 2322.7, 2322.7, + 2322.7, 2322.7, 2322.7, 2300.7, 2300.7, 2300.7, 2300.7, 2300.7, 2300.7, + 2300.7, 2313.6, 2314.6, 2314.6, 2314.6, 2352.2, 2352.2, 2352.2, 2317.8, + 2317.8, 2317.8, 2317.8, 2317.8, 2317.8, 2326, 2315.6, 2309.6, 2326.1, + 2322.4, 2323.4, 2326, 2326 + ] +} diff --git a/test/live/buildLiveSoak.unit.test.ts b/test/live/buildLiveSoak.unit.test.ts index 2decc10..384d80f 100644 --- a/test/live/buildLiveSoak.unit.test.ts +++ b/test/live/buildLiveSoak.unit.test.ts @@ -66,10 +66,10 @@ describe("buildLiveSoak", () => { ]); }); - it("should give the verdict's growth once a full hour after the warm-up is in", () => { - // Arrange: 30 min warm-up + 61 min + it("should give the verdict's growth once a full hour after the settle period is in", () => { + // Arrange: 30 min warm-up + 30 min settle + 61 min const progress = { - samples: samples("one", loadStartedMs, 91, 2), + samples: samples("one", loadStartedMs, 121, 2), placed: 1, failed: 0 }; @@ -84,7 +84,7 @@ describe("buildLiveSoak", () => { points: [] }, selected: true, - nowMs: loadStartedMs + 91 * MIN, + nowMs: loadStartedMs + 121 * MIN, estimatedStartMs: null, progress }); @@ -94,6 +94,36 @@ describe("buildLiveSoak", () => { expect(live.memory[1]?.growthMBPerHour).to.equal(null); }); + it("should leave the settle period out of the growth", () => { + // Arrange: 30 min warm-up + 120 min; 20 MB/h until the 30 min settle + // period after the warm-up ends, flat after + const rising = samples("one", loadStartedMs, 60, 20); + const top = rising.at(-1)?.rssKB ?? 0; + const flat = Array.from({ length: 90 * 6 }, (_, i) => ({ + name: "one", + tMs: loadStartedMs + 60 * MIN + (i + 1) * 10_000, + rssKB: top + })); + + // Act + const live = buildLiveSoak({ + benchmark: SOAK, + record: { + startedMs: loadStartedMs, + loadStartedMs, + endedMs: null, + points: [] + }, + selected: true, + nowMs: loadStartedMs + 150 * MIN, + estimatedStartMs: null, + progress: { samples: [...rising, ...flat], placed: 1, failed: 0 } + }); + + // Assert + expect(live.memory[0]?.growthMBPerHour).to.equal(0); + }); + it("should be queued with nothing measured before it starts", () => { const live = buildLiveSoak({ benchmark: SOAK, diff --git a/test/render/renderRunPage.unit.test.ts b/test/render/renderRunPage.unit.test.ts index 7bab373..a504025 100644 --- a/test/render/renderRunPage.unit.test.ts +++ b/test/render/renderRunPage.unit.test.ts @@ -180,6 +180,34 @@ describe("the report page", () => { expect(textOf($, "#one-soak .hop-legend")).to.contain("edgeport4.1 ms"); }); + it("should say which window the memory growth slope covers, when the report has it", async () => { + // Arrange + const report = loadReport("passed"); + const soak = report.benchmarks.find( + (b: { id: string }) => b.id === "one-soak" + ); + soak.settleS = 1800; + for (const rule of soak.rules) { + if (rule.id === "memoryGrowthMBPerHour") { + rule.window = { + from: "2026-11-03T15:00:00.000Z", + to: "2026-11-03T16:30:00.000Z" + }; + } + } + const site = await renderSite(report, loadHistory(RUNS_4)); + const old = await renderSite(loadReport("passed"), loadHistory(RUNS_4)); + + // Act + const $ = site.latest(); + + // Assert + expect(textOf($, "#one-soak .growth-window")).to.equal( + "Memory growth: Theil–Sen slope from 15:00 to 16:30 UTC, after a 30 min settle" + ); + expect(old.latest()("#one-soak .growth-window")).to.have.length(0); + }); + it("should list the test setup and the command", async () => { // Arrange const site = await renderSite(loadReport("passed"), loadHistory(RUNS_4)); diff --git a/test/schemas/perfkitConfig.unit.test.ts b/test/schemas/perfkitConfig.unit.test.ts index 4ca879d..b9efdb4 100644 --- a/test/schemas/perfkitConfig.unit.test.ts +++ b/test/schemas/perfkitConfig.unit.test.ts @@ -394,6 +394,20 @@ describe("perfkitConfigSchema", () => { ]); }); + it("should reject a soak settle period that is not shorter than its duration", () => { + const raw = loadConfig(); + raw.benchmarks[1].settle = raw.benchmarks[1].duration; + expect(errorsOf(perfkitConfigSchema, raw)).to.deep.equal([ + `benchmarks[1].settle: expected a settle period shorter than duration (${raw.benchmarks[1].duration})` + ]); + }); + + it("should accept a soak settle period shorter than its duration", () => { + const raw = loadConfig(); + raw.benchmarks[1].settle = "15m"; + expect(errorsOf(perfkitConfigSchema, raw)).to.deep.equal([]); + }); + it("should require the project, infra, image and benchmark fields", () => { const raw = loadConfig(); ["name", "repo", "siteUrl", "accent", "title", "lead"].forEach((key) => {