From cac1f1c673e5885ae6e5e76c30099d693ec27725 Mon Sep 17 00:00:00 2001 From: Pedro Sanders Date: Wed, 4 Nov 2026 05:48:19 -0400 Subject: [PATCH] fix: open every VPC port in the run's firewall and add a capacity warm-up On DigitalOcean the seed couldn't reach routr-one's API (51908) and Routr couldn't open new flows to the callees: the firewall allowed only SIP, OTLP and SSH from the VPC. The VPC is the run's own, so it now allows all tcp, udp and icmp. Capacity benchmarks get an optional warmup that the first step's measurement skips: EdgePort starts cold (JVM, GraalJS). --- README.md | 10 ++- docs/perfkit-yaml.md | 2 +- docs/project-setup.md | 4 +- package-lock.json | 4 +- package.json | 2 +- schemas/perfkit-v1.schema.json | 5 ++ src/analyze/capacityStepWindow.ts | 29 +++++++ src/analyze/index.ts | 1 + src/cli/formatPlan.ts | 3 +- src/infra/buildFirewall.ts | 17 ++-- src/infra/constants.ts | 5 -- src/orchestrate/createRunBenchmark.ts | 19 +++-- src/orchestrate/evaluateCapacityStep.ts | 11 ++- src/schemas/checkWarmupShorterThanStep.ts | 33 +++++++ src/schemas/perfkitConfig.schema.ts | 25 +++--- test/analyze/capacityStepWindow.unit.test.ts | 90 ++++++++++++++++++++ test/infra/helpers.unit.test.ts | 8 +- test/infra/run.unit.test.ts | 2 +- test/orchestrate/agent.unit.test.ts | 78 ++++++++++++++++- test/schemas/perfkitConfig.unit.test.ts | 20 +++++ 20 files changed, 313 insertions(+), 55 deletions(-) create mode 100644 src/analyze/capacityStepWindow.ts create mode 100644 src/schemas/checkWarmupShorterThanStep.ts create mode 100644 test/analyze/capacityStepWindow.unit.test.ts diff --git a/README.md b/README.md index fee40c9..ff23990 100644 --- a/README.md +++ b/README.md @@ -127,8 +127,8 @@ the end. In `infra.region` of `perfkit.yaml`, it creates, in this order: collide; 3. two `ubuntu-24-04-x64` droplets in it, `perfkit--uut` and `perfkit--loadgen`, tagged `perfkit` and `perfkit-run-`; -4. a firewall, `perfkit-`: SIP (5060-5063 tcp/udp) and OTLP (4318 tcp) from the VPC only, - SSH (22) from this machine's public IP (from `https://api.ipify.org`) and from the VPC, where the agent on the load generator reaches the server under test. +4. a firewall, `perfkit-`: from the VPC: all ports (tcp and udp) and icmp, so the load + generator reaches the server's API and the server reaches the callees; SSH (22) from this machine's public IP (from `https://api.ipify.org`) and from the VPC, where the agent on the load generator reaches the server under test. Both droplets get Docker Engine with the compose plugin, 25 MiB socket buffers (`net.core.rmem_max`, `wmem_max`, `rmem_default`, `wmem_default`) and 1048576 open files, through @@ -161,7 +161,7 @@ plan for Routr 2.31.6 (dry run, nothing created) region nyc3, new VPC (range assigned by DigitalOcean) uut c-8 ubuntu-24-04-x64 loadgen c-16 ubuntu-24-04-x64 - firewall SIP (5060-5063 tcp/udp) and OTLP (4318 tcp) from the VPC only; SSH (22) from 203.0.113.7 and the VPC only + firewall from the VPC: all ports (tcp/udp) and icmp; SSH (22) from 203.0.113.7 and the VPC only tags perfkit, perfkit-run-20261103T141210Z-bd3254a images fonoster/routr-edgeport:2.31.6@sha256:8bca…1866 fonoster/routr-echo:2.31.6@sha256:14a4…a43f @@ -364,6 +364,10 @@ SIPp adds load in steps, such as 50 more calls/s every minute. For each step: in the next one. - Its attempted, failed and retransmitted calls are SIPp's 10-second counters (`-trace_stat`) that fall in the step. Failed % is failed / attempted. +- **Warm-up.** With `warmup` set, the first step still runs its full `every`, but it is measured + only from `warmup` after it starts: calls sent, and 10-second counters whose period mid-point + falls, before that don't count. A cold JIT (EdgePort's JVM and GraalJS) makes the first seconds + slow; the later steps are measured whole. perfkit checks the steps in order: diff --git a/docs/perfkit-yaml.md b/docs/perfkit-yaml.md index f0d7f6a..a426364 100644 --- a/docs/perfkit-yaml.md +++ b/docs/perfkit-yaml.md @@ -65,7 +65,7 @@ the other kind rejects them. | `narrative.why` | string | yes | | Why it matters. | | `narrative.how` | string | yes | | How the load is driven and what the verdict checks. | | `duration` | duration | *soak*: yes | | How long the load runs. | -| `warmup` | duration | *soak*: yes | | Excluded from every verdict. | +| `warmup` | duration | *soak*: yes; *capacity*: no (`0s`) | | *soak*: excluded from every verdict. *capacity*: the first step still runs its full `every`, but its measurement starts `warmup` after the step starts, so a cold server's first seconds don't count. Shorter than `load.rate.every`. | | `uut.compose` | path | yes | | The Compose file the unit under test runs. | | `uut.setup` | path | no | none | A script that runs once the unit under test is up, for example to seed data. | | `uut.processes` | list | yes | | At least one. Sampled every 10 s. | diff --git a/docs/project-setup.md b/docs/project-setup.md index 54588b5..7887015 100644 --- a/docs/project-setup.md +++ b/docs/project-setup.md @@ -43,8 +43,8 @@ anything (exit 2): perfkit: sipp-uas/compose.yaml: service uas must use network_mode: host (Docker's UDP port publishing distorts the results); nothing was created ``` -The firewall lets SIP in on 5060 to 5063 (UDP and TCP) and OTLP on 4318 from the VPC, so the -server listens on a port in that range. +The firewall lets every TCP and UDP port (and ICMP) in from the VPC, so the server can listen on +any port, and SSH from the runner and the VPC. Nothing else reaches the droplets. **Healthchecks.** The UUT is up when every service is `running`, and `healthy` when it has a `healthcheck`, within 5 minutes; otherwise the run ends with exit 3. Give a service a healthcheck diff --git a/package-lock.json b/package-lock.json index c08c506..7ebc3e3 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "perfkit", - "version": "0.1.4", + "version": "0.1.5", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "perfkit", - "version": "0.1.4", + "version": "0.1.5", "license": "MIT", "dependencies": { "commander": "^14.0.3", diff --git a/package.json b/package.json index 80b5025..d5a6c0f 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "perfkit", - "version": "0.1.4", + "version": "0.1.5", "description": "Benchmark harness for Fonoster projects: DigitalOcean runs, one report.json schema, one branded report page", "license": "MIT", "author": "Pedro Sanders ", diff --git a/schemas/perfkit-v1.schema.json b/schemas/perfkit-v1.schema.json index 1bdf2c2..e141f80 100644 --- a/schemas/perfkit-v1.schema.json +++ b/schemas/perfkit-v1.schema.json @@ -152,6 +152,11 @@ ], "additionalProperties": false }, + "warmup": { + "default": "0s", + "type": "string", + "pattern": "^\\d+[smh]$" + }, "uut": { "type": "object", "properties": { diff --git a/src/analyze/capacityStepWindow.ts b/src/analyze/capacityStepWindow.ts new file mode 100644 index 0000000..8c342b5 --- /dev/null +++ b/src/analyze/capacityStepWindow.ts @@ -0,0 +1,29 @@ +/** + * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. + */ +import type { TimeWindow } from "./types.js"; + +/** + * The window a capacity step is measured over: step `index` (from 0) runs + * from `startedMs + index × everyMs` for `everyMs`. The first step's window + * starts `warmupMs` later, so a cold server's first seconds don't count; it + * still runs its full `everyMs`. `endedMs`, when given, cuts the window short + * (a ramp that ended early), and the window never starts after it ends. + */ +export function capacityStepWindow(input: { + startedMs: number; + index: number; + everyMs: number; + warmupMs: number; + endedMs?: number; +}): TimeWindow { + const { startedMs, index, everyMs, warmupMs } = input; + const stepStartMs = startedMs + index * everyMs; + const stepEndMs = stepStartMs + everyMs; + const toMs = + input.endedMs === undefined + ? stepEndMs + : Math.min(stepEndMs, input.endedMs); + const fromMs = Math.min(stepStartMs + (index === 0 ? warmupMs : 0), toMs); + return { fromMs, toMs }; +} diff --git a/src/analyze/index.ts b/src/analyze/index.ts index 769c153..1c100ee 100644 --- a/src/analyze/index.ts +++ b/src/analyze/index.ts @@ -15,6 +15,7 @@ export { } from "./analyzeSoak.schema.js"; export { computeHops } from "./computeHops.js"; export { countRestarts } from "./countRestarts.js"; +export { capacityStepWindow } from "./capacityStepWindow.js"; export { cpuPctOver } from "./cpuPctOver.js"; export { createAnalyzeCapacity } from "./createAnalyzeCapacity.js"; export { diff --git a/src/cli/formatPlan.ts b/src/cli/formatPlan.ts index 6f60a11..d312e01 100644 --- a/src/cli/formatPlan.ts +++ b/src/cli/formatPlan.ts @@ -1,7 +1,6 @@ /** * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. */ -import { OTLP_PORT, SIP_PORTS } from "../infra/constants.js"; import type { Plan } from "../infra/index.js"; import { parseDuration } from "../infra/parseDuration.js"; import type { BenchmarkConfig } from "../schemas/index.js"; @@ -60,7 +59,7 @@ export function formatPlan( `${plan.loadgen.size.padEnd(sizeWidth)}${plan.loadgen.image}` ]), ...field("firewall", [ - `SIP (${SIP_PORTS} tcp/udp) and OTLP (${OTLP_PORT} tcp) from the VPC only; SSH (22) from ${plan.sshSource.replace(/\/32$/, "")} and the VPC only` + `from the VPC: all ports (tcp/udp) and icmp; SSH (22) from ${plan.sshSource.replace(/\/32$/, "")} and the VPC only` ]), ...field("tags", [plan.tags.join(", ")]), ...field( diff --git a/src/infra/buildFirewall.ts b/src/infra/buildFirewall.ts index 657ff8c..755a455 100644 --- a/src/infra/buildFirewall.ts +++ b/src/infra/buildFirewall.ts @@ -1,16 +1,17 @@ /** * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. */ -import { OTLP_PORT, SIP_PORTS } from "./constants.js"; import type { DoFirewallInput } from "./types.js"; const ANYWHERE = { addresses: ["0.0.0.0/0", "::/0"] }; /** - * A run's firewall: SIP (tcp and udp) and OTLP (tcp) from the VPC's range - * only, SSH (22) from `sshSources` and from the VPC (the agent on the load - * generator drives the UUT over SSH at its private IP), and every outbound - * connection. It applies to the run's droplets. + * A run's firewall: every tcp and udp port and icmp from the VPC's range + * (the seed script reaches the UUT's API, and the UUT opens new flows to the + * callees on the load generator), SSH (22) from `sshSources` and from the + * VPC (the agent on the load generator drives the UUT over SSH at its + * private IP), and every outbound connection. It applies to the run's + * droplets. */ export function buildFirewall(input: { name: string; @@ -22,9 +23,9 @@ export function buildFirewall(input: { return { name: input.name, inbound_rules: [ - { protocol: "udp", ports: SIP_PORTS, sources: vpc }, - { protocol: "tcp", ports: SIP_PORTS, sources: vpc }, - { protocol: "tcp", ports: OTLP_PORT, sources: vpc }, + { protocol: "tcp", ports: "all", sources: vpc }, + { protocol: "udp", ports: "all", sources: vpc }, + { protocol: "icmp", sources: vpc }, { protocol: "tcp", ports: "22", diff --git a/src/infra/constants.ts b/src/infra/constants.ts index 650fdf0..f5c3279 100644 --- a/src/infra/constants.ts +++ b/src/infra/constants.ts @@ -12,11 +12,6 @@ export const PERFKIT_TAG = "perfkit"; /** Name of the SSH key that PERFKIT_SSH_KEY registers; never deleted. */ export const SHARED_KEY_NAME = "perfkit-shared"; -/** SIP from the VPC. */ -export const SIP_PORTS = "5060-5063"; -/** OTLP/HTTP from the VPC. */ -export const OTLP_PORT = "4318"; - /** Droplets active and cloud-init finished within this, or exit 3. */ export const READY_TIMEOUT_MS = 10 * 60 * 1000; /** How long teardown waits for a VPC to have no members. */ diff --git a/src/orchestrate/createRunBenchmark.ts b/src/orchestrate/createRunBenchmark.ts index 381f986..1b3263b 100644 --- a/src/orchestrate/createRunBenchmark.ts +++ b/src/orchestrate/createRunBenchmark.ts @@ -2,6 +2,7 @@ * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. */ import { basename } from "node:path"; +import { capacityStepWindow } from "../analyze/capacityStepWindow.js"; import { durationToSeconds } from "../analyze/durationToSeconds.js"; import type { Now, Sleep } from "../infra/index.js"; import type { BenchmarkConfig } from "../schemas/index.js"; @@ -543,18 +544,18 @@ export function createRunBenchmark(deps: AgentDeps, state: RunState) { ...base, kind: "capacity", steps: Array.from({ length: Math.max(steps, 1) }, (_, index) => { - const everyMs = - durationToSeconds(benchmark.load.rate.every) * 1000; + const window = capacityStepWindow({ + startedMs, + index, + everyMs: durationToSeconds(benchmark.load.rate.every) * 1000, + warmupMs: durationToSeconds(benchmark.warmup) * 1000, + endedMs: Math.max(endedMs, startedMs + 1) + }); return { offeredCps: benchmark.load.rate.start + index * benchmark.load.rate.step, - from: iso(startedMs + index * everyMs), - to: iso( - Math.min( - startedMs + (index + 1) * everyMs, - Math.max(endedMs, startedMs + 1) - ) - ) + from: iso(window.fromMs), + to: iso(window.toMs) }; }) } diff --git a/src/orchestrate/evaluateCapacityStep.ts b/src/orchestrate/evaluateCapacityStep.ts index de783d0..2105813 100644 --- a/src/orchestrate/evaluateCapacityStep.ts +++ b/src/orchestrate/evaluateCapacityStep.ts @@ -3,6 +3,7 @@ */ import { join } from "node:path"; import { + capacityStepWindow, measureCapacityStep, parseSamplerJsonl, parseUdpJsonl, @@ -52,10 +53,12 @@ export async function evaluateCapacityStep(input: { names.includes(file) ? deps.readFile(join(rawDir, file)) : ""; const measured = measureCapacityStep({ offeredCps, - window: { - fromMs: input.startMs + index * everyMs, - toMs: input.startMs + (index + 1) * everyMs - }, + window: capacityStepWindow({ + startedMs: input.startMs, + index, + everyMs, + warmupMs: durationMs(benchmark.warmup) + }), uac, samples: parseSamplerJsonl( await read("sampler-loadgen.jsonl"), diff --git a/src/schemas/checkWarmupShorterThanStep.ts b/src/schemas/checkWarmupShorterThanStep.ts new file mode 100644 index 0000000..ebe44a1 --- /dev/null +++ b/src/schemas/checkWarmupShorterThanStep.ts @@ -0,0 +1,33 @@ +/** + * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. + */ +import type { z } from "zod/v4"; + +const UNIT_S = { s: 1, m: 60, h: 3600 } as const; + +/** + * Adds an issue at `warmup` when a capacity benchmark's warm-up is not + * shorter than one step (`load.rate.every`): the first step would have + * nothing left to measure. A `0s` warm-up is always fine. For use inside the + * capacity benchmark's `superRefine`. + */ +export function checkWarmupShorterThanStep( + benchmark: { warmup: string; load: { rate: { every: string } } }, + ctx: z.core.$RefinementCtx +): void { + const warmupS = seconds(benchmark.warmup); + const everyS = seconds(benchmark.load.rate.every); + if (warmupS > 0 && warmupS >= everyS) { + ctx.addIssue({ + code: "custom", + path: ["warmup"], + message: `expected a warm-up shorter than load.rate.every (${benchmark.load.rate.every})`, + input: benchmark.warmup + }); + } +} + +function seconds(duration: string): number { + const match = /^(\d+)([smh])$/.exec(duration); + return match ? Number(match[1]) * UNIT_S[match[2] as keyof typeof UNIT_S] : 0; +} diff --git a/src/schemas/perfkitConfig.schema.ts b/src/schemas/perfkitConfig.schema.ts index dd63d3a..5588d60 100644 --- a/src/schemas/perfkitConfig.schema.ts +++ b/src/schemas/perfkitConfig.schema.ts @@ -9,6 +9,7 @@ */ import { z } from "zod/v4"; import { checkUniqueIds } from "./checkUniqueIds.js"; +import { checkWarmupShorterThanStep } from "./checkWarmupShorterThanStep.js"; import { benchmarkIdSchema, count, @@ -175,16 +176,20 @@ const narrativeSchema = z.strictObject({ how: nonEmptyString }); -export const capacityBenchmarkConfigSchema = z.strictObject({ - id: benchmarkIdSchema, - kind: z.literal("capacity"), - title: nonEmptyString, - narrative: narrativeSchema, - uut: configUutSchema, - load: capacityLoadSchema, - telemetry: telemetrySchema, - thresholds: capacityThresholdsSchema -}); +export const capacityBenchmarkConfigSchema = z + .strictObject({ + id: benchmarkIdSchema, + kind: z.literal("capacity"), + title: nonEmptyString, + narrative: narrativeSchema, + // The first step's measurement starts this long after the step does. + warmup: durationSchema.default("0s"), + uut: configUutSchema, + load: capacityLoadSchema, + telemetry: telemetrySchema, + thresholds: capacityThresholdsSchema + }) + .superRefine(checkWarmupShorterThanStep); export const soakBenchmarkConfigSchema = z.strictObject({ id: benchmarkIdSchema, diff --git a/test/analyze/capacityStepWindow.unit.test.ts b/test/analyze/capacityStepWindow.unit.test.ts new file mode 100644 index 0000000..becbe11 --- /dev/null +++ b/test/analyze/capacityStepWindow.unit.test.ts @@ -0,0 +1,90 @@ +/** + * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. + */ +import { expect } from "chai"; +import { + capacityStepWindow, + measureCapacityStep, + parseSamplerJsonl, + parseUdpJsonl, + readSippResults +} from "../../src/analyze/index.js"; +import { + capacityRaw, + fakeDeps, + RAW_DIR, + START_MS, + STEP_MS +} from "./builders.js"; + +describe("capacityStepWindow", () => { + it("should start only the first step's window warmup later", () => { + const window = (index: number) => + capacityStepWindow({ + startedMs: 0, + index, + everyMs: 60_000, + warmupMs: 20_000 + }); + expect(window(0)).to.deep.equal({ fromMs: 20_000, toMs: 60_000 }); + expect(window(1)).to.deep.equal({ fromMs: 60_000, toMs: 120_000 }); + }); + + it("should cut the window at endedMs and never start after it ends", () => { + expect( + capacityStepWindow({ + startedMs: 0, + index: 0, + everyMs: 60_000, + warmupMs: 20_000, + endedMs: 10_000 + }) + ).to.deep.equal({ fromMs: 10_000, toMs: 10_000 }); + }); + + it("should leave the warm-up's slow calls and stat rows out of the first step", async () => { + // Arrange: the first 30 of 100 calls (sent in the first 18 s) take 200 ms. + const latencies = [ + ...Array.from({ length: 30 }, () => 200), + ...Array.from({ length: 70 }, () => 5) + ]; + const files = capacityRaw([{ offeredCps: 100, latencies }]); + const uac = await readSippResults( + fakeDeps(files), + RAW_DIR, + Object.keys(files), + "uac_message.xml" + ); + const measure = (warmupMs: number) => + measureCapacityStep({ + offeredCps: 100, + window: capacityStepWindow({ + startedMs: START_MS, + index: 0, + everyMs: STEP_MS, + warmupMs + }), + uac: uac!, + samples: parseSamplerJsonl( + files["sampler.jsonl"] as string, + "sampler.jsonl" + ), + udp: parseUdpJsonl(files["udp.jsonl"] as string, "udp.jsonl"), + uutProcesses: ["edgeport"], + unit: "calls" + }).step; + + // Act + const cold = measure(0); + const warm = measure(20_000); + + // Assert: the stat rows at 10 s and 20 s (mid-points 5 s and 15 s) are out. + expect(cold.p99Ms).to.equal(200); + expect(warm).to.include({ + p99Ms: 5, + calls: 4000, + achievedCps: 100, + valid: true + }); + }); +}); diff --git a/test/infra/helpers.unit.test.ts b/test/infra/helpers.unit.test.ts index 8ce85a5..0879504 100644 --- a/test/infra/helpers.unit.test.ts +++ b/test/infra/helpers.unit.test.ts @@ -73,7 +73,7 @@ describe("buildCloudInit", () => { }); describe("buildFirewall", () => { - it("should open SIP and OTLP to the VPC, SSH to the sources, and all outbound", () => { + it("should open every tcp/udp port and icmp to the VPC, SSH to the sources, and all outbound", () => { // Act const firewall = buildFirewall({ name: "perfkit-x", @@ -89,9 +89,9 @@ describe("buildFirewall", () => { `${protocol} ${ports} ${sources.addresses?.join(",")}` ) ).to.deep.equal([ - "udp 5060-5063 10.1.0.0/20", - "tcp 5060-5063 10.1.0.0/20", - "tcp 4318 10.1.0.0/20", + "tcp all 10.1.0.0/20", + "udp all 10.1.0.0/20", + "icmp undefined 10.1.0.0/20", "tcp 22 203.0.113.7/32,10.1.0.0/20" ]); expect( diff --git a/test/infra/run.unit.test.ts b/test/infra/run.unit.test.ts index 86f45b1..807c0ad 100644 --- a/test/infra/run.unit.test.ts +++ b/test/infra/run.unit.test.ts @@ -24,7 +24,7 @@ const PLAN = `plan for Routr 2.31.5 (dry run, nothing created) region nyc3, new VPC (range assigned by DigitalOcean) uut c-8 ubuntu-24-04-x64 loadgen c-16 ubuntu-24-04-x64 - firewall SIP (5060-5063 tcp/udp) and OTLP (4318 tcp) from the VPC only; SSH (22) from 203.0.113.7 and the VPC only + firewall from the VPC: all ports (tcp/udp) and icmp; SSH (22) from 203.0.113.7 and the VPC only tags perfkit, perfkit-run-${RUN_ID} images fonoster/routr-edgeport:2.31.5@sha256:4f1c…e2a0 fonoster/routr-echo:2.31.5@sha256:41ab…77d0 diff --git a/test/orchestrate/agent.unit.test.ts b/test/orchestrate/agent.unit.test.ts index 98c59d5..1c34f5b 100644 --- a/test/orchestrate/agent.unit.test.ts +++ b/test/orchestrate/agent.unit.test.ts @@ -2,8 +2,9 @@ * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. */ import { expect } from "chai"; -import { readFile, writeFile } from "node:fs/promises"; -import { join } from "node:path"; +import { cp, mkdtemp, readFile, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; import { ValidationError } from "../../src/errors/index.js"; import { createNodeAnalyzeDeps } from "../../src/analyze/index.js"; import { analyzeBenchmark } from "../../src/orchestrate/analyzeBenchmark.js"; @@ -17,7 +18,13 @@ import { } from "../analyze/builders.js"; import { ROUTR_CONFIG } from "../fixtures.js"; import { fakeRegistry, perfkit } from "../infra/fakes.js"; -import { doSetup, MINIMAL, PASSING_STEPS, type RawFiles } from "./setup.js"; +import { + doSetup, + MINIMAL, + MINIMAL_CONFIG, + PASSING_STEPS, + type RawFiles +} from "./setup.js"; /** A detached run whose agent the test runs itself. */ async function detached(args: string[], files: RawFiles, routr = false) { @@ -87,6 +94,71 @@ describe("createAgent", () => { ); }); + it("should leave a capacity warm-up out of the first step, live and in the final verdict", async () => { + // Arrange: the first 30 of step 1's 100 messages (sent in its first 18 s) + // are slow, as with a cold JIT; a 20 s warm-up leaves them out. + const dir = await mkdtemp(join(tmpdir(), "perfkit-warmup-")); + await cp(dirname(MINIMAL_CONFIG), dir, { recursive: true }); + const config = join(dir, "perfkit.yaml"); + const yaml = await readFile(config, "utf8"); + await writeFile( + config, + yaml.replace("kind: capacity\n", "kind: capacity\n warmup: 20s\n") + ); + const cold = [ + ...Array.from({ length: 30 }, () => 200), + ...Array.from({ length: 70 }, () => 5) + ]; + const steps = [ + { offeredCps: 50, latencies: cold }, + ...PASSING_STEPS.slice(1) + ]; + const setup = await detached( + ["run", "--config", config, "--target-version", "3.7.3"], + capacityRaw(steps) + ); + + // Act + await setup.agent.run(setup.runId()); + + // Assert + const progress = join(setup.hosts.roots.loadgen, "progress.log"); + expect((await readFile(progress, "utf8")).trim().split("\n")).to.deep.equal( + [ + "[sipp-uas] step 50 messages/s p99 5.0 ms failed 0.00% ok", + "[sipp-uas] step 100 messages/s p99 5.0 ms failed 0.00% ok", + "[sipp-uas] step 150 messages/s p99 5.0 ms failed 0.00% ok", + "[sipp-uas] step 200 messages/s p99 5.0 ms failed 0.00% ok" + ] + ); + const raw = join(setup.hosts.roots.loadgen, "raw", "sipp-uas"); + const meta = JSON.parse(await readFile(join(raw, "meta.json"), "utf8")); + expect(meta.steps[0].from).to.equal( + new Date(START_MS + 20_000).toISOString() + ); + expect(meta.steps[1].from).to.equal( + new Date(START_MS + 60_000).toISOString() + ); + const state = runStateSchema.parse( + JSON.parse( + await readFile(join(setup.hosts.roots.loadgen, "run.json"), "utf8") + ) + ); + const result = await analyzeBenchmark( + createNodeAnalyzeDeps(), + state.plan.benchmarks[0]!, + raw, + null, + "SIPp 3.7.3, UDP" + ); + expect(result).to.include({ verdict: "passed" }); + expect("series" in result && result.series?.steps[0]).to.include({ + p99Ms: 5, + achievedCps: 50, + valid: true + }); + }); + it("should run compose down for the first benchmark before compose up for the second, and the second's setup once", async () => { // Arrange const setup = await detached( diff --git a/test/schemas/perfkitConfig.unit.test.ts b/test/schemas/perfkitConfig.unit.test.ts index 2efaad1..9c61564 100644 --- a/test/schemas/perfkitConfig.unit.test.ts +++ b/test/schemas/perfkitConfig.unit.test.ts @@ -41,6 +41,17 @@ describe("perfkitConfigSchema", () => { expect(config.benchmarks[0]?.load.uac.users).to.equal(undefined); }); + it("should default a capacity warm-up to 0s and accept one shorter than a step", () => { + const raw = loadConfig(); + expect(perfkitConfigSchema.parse(raw).benchmarks[0]?.warmup).to.equal( + "0s" + ); + raw.benchmarks[0].warmup = "20s"; + expect(perfkitConfigSchema.parse(raw).benchmarks[0]?.warmup).to.equal( + "20s" + ); + }); + it("should accept a project without a logo", () => { const raw = loadConfig(); delete raw.project.logo; @@ -65,6 +76,15 @@ describe("perfkitConfigSchema", () => { ]); }); + it("should reject a capacity warm-up as long as a step", () => { + const raw = loadConfig(); + raw.benchmarks[0].warmup = "1m"; + raw.benchmarks[0].load.rate.every = "60s"; + expect(errorsOf(perfkitConfigSchema, raw)).to.deep.equal([ + "benchmarks[0].warmup: expected a warm-up shorter than load.rate.every (60s)" + ]); + }); + it("should require duration and warmup in a soak benchmark", () => { const raw = loadConfig(); delete raw.benchmarks[1].duration;