diff --git a/README.md b/README.md index fee40c9..ff23990 100644 --- a/README.md +++ b/README.md @@ -127,8 +127,8 @@ the end. In `infra.region` of `perfkit.yaml`, it creates, in this order: collide; 3. two `ubuntu-24-04-x64` droplets in it, `perfkit--uut` and `perfkit--loadgen`, tagged `perfkit` and `perfkit-run-`; -4. a firewall, `perfkit-`: SIP (5060-5063 tcp/udp) and OTLP (4318 tcp) from the VPC only, - SSH (22) from this machine's public IP (from `https://api.ipify.org`) and from the VPC, where the agent on the load generator reaches the server under test. +4. a firewall, `perfkit-`: from the VPC: all ports (tcp and udp) and icmp, so the load + generator reaches the server's API and the server reaches the callees; SSH (22) from this machine's public IP (from `https://api.ipify.org`) and from the VPC, where the agent on the load generator reaches the server under test. Both droplets get Docker Engine with the compose plugin, 25 MiB socket buffers (`net.core.rmem_max`, `wmem_max`, `rmem_default`, `wmem_default`) and 1048576 open files, through @@ -161,7 +161,7 @@ plan for Routr 2.31.6 (dry run, nothing created) region nyc3, new VPC (range assigned by DigitalOcean) uut c-8 ubuntu-24-04-x64 loadgen c-16 ubuntu-24-04-x64 - firewall SIP (5060-5063 tcp/udp) and OTLP (4318 tcp) from the VPC only; SSH (22) from 203.0.113.7 and the VPC only + firewall from the VPC: all ports (tcp/udp) and icmp; SSH (22) from 203.0.113.7 and the VPC only tags perfkit, perfkit-run-20261103T141210Z-bd3254a images fonoster/routr-edgeport:2.31.6@sha256:8bca…1866 fonoster/routr-echo:2.31.6@sha256:14a4…a43f @@ -364,6 +364,10 @@ SIPp adds load in steps, such as 50 more calls/s every minute. For each step: in the next one. - Its attempted, failed and retransmitted calls are SIPp's 10-second counters (`-trace_stat`) that fall in the step. Failed % is failed / attempted. +- **Warm-up.** With `warmup` set, the first step still runs its full `every`, but it is measured + only from `warmup` after it starts: calls sent, and 10-second counters whose period mid-point + falls, before that don't count. A cold JIT (EdgePort's JVM and GraalJS) makes the first seconds + slow; the later steps are measured whole. perfkit checks the steps in order: diff --git a/docs/perfkit-yaml.md b/docs/perfkit-yaml.md index f0d7f6a..a426364 100644 --- a/docs/perfkit-yaml.md +++ b/docs/perfkit-yaml.md @@ -65,7 +65,7 @@ the other kind rejects them. | `narrative.why` | string | yes | | Why it matters. | | `narrative.how` | string | yes | | How the load is driven and what the verdict checks. | | `duration` | duration | *soak*: yes | | How long the load runs. | -| `warmup` | duration | *soak*: yes | | Excluded from every verdict. | +| `warmup` | duration | *soak*: yes; *capacity*: no (`0s`) | | *soak*: excluded from every verdict. *capacity*: the first step still runs its full `every`, but its measurement starts `warmup` after the step starts, so a cold server's first seconds don't count. Shorter than `load.rate.every`. | | `uut.compose` | path | yes | | The Compose file the unit under test runs. | | `uut.setup` | path | no | none | A script that runs once the unit under test is up, for example to seed data. | | `uut.processes` | list | yes | | At least one. Sampled every 10 s. | diff --git a/docs/project-setup.md b/docs/project-setup.md index 54588b5..7887015 100644 --- a/docs/project-setup.md +++ b/docs/project-setup.md @@ -43,8 +43,8 @@ anything (exit 2): perfkit: sipp-uas/compose.yaml: service uas must use network_mode: host (Docker's UDP port publishing distorts the results); nothing was created ``` -The firewall lets SIP in on 5060 to 5063 (UDP and TCP) and OTLP on 4318 from the VPC, so the -server listens on a port in that range. +The firewall lets every TCP and UDP port (and ICMP) in from the VPC, so the server can listen on +any port, and SSH from the runner and the VPC. Nothing else reaches the droplets. **Healthchecks.** The UUT is up when every service is `running`, and `healthy` when it has a `healthcheck`, within 5 minutes; otherwise the run ends with exit 3. Give a service a healthcheck diff --git a/package-lock.json b/package-lock.json index c08c506..7ebc3e3 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "perfkit", - "version": "0.1.4", + "version": "0.1.5", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "perfkit", - "version": "0.1.4", + "version": "0.1.5", "license": "MIT", "dependencies": { "commander": "^14.0.3", diff --git a/package.json b/package.json index 80b5025..d5a6c0f 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "perfkit", - "version": "0.1.4", + "version": "0.1.5", "description": "Benchmark harness for Fonoster projects: DigitalOcean runs, one report.json schema, one branded report page", "license": "MIT", "author": "Pedro Sanders ", diff --git a/schemas/perfkit-v1.schema.json b/schemas/perfkit-v1.schema.json index 1bdf2c2..e141f80 100644 --- a/schemas/perfkit-v1.schema.json +++ b/schemas/perfkit-v1.schema.json @@ -152,6 +152,11 @@ ], "additionalProperties": false }, + "warmup": { + "default": "0s", + "type": "string", + "pattern": "^\\d+[smh]$" + }, "uut": { "type": "object", "properties": { diff --git a/src/analyze/capacityStepWindow.ts b/src/analyze/capacityStepWindow.ts new file mode 100644 index 0000000..8c342b5 --- /dev/null +++ b/src/analyze/capacityStepWindow.ts @@ -0,0 +1,29 @@ +/** + * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. + */ +import type { TimeWindow } from "./types.js"; + +/** + * The window a capacity step is measured over: step `index` (from 0) runs + * from `startedMs + index × everyMs` for `everyMs`. The first step's window + * starts `warmupMs` later, so a cold server's first seconds don't count; it + * still runs its full `everyMs`. `endedMs`, when given, cuts the window short + * (a ramp that ended early), and the window never starts after it ends. + */ +export function capacityStepWindow(input: { + startedMs: number; + index: number; + everyMs: number; + warmupMs: number; + endedMs?: number; +}): TimeWindow { + const { startedMs, index, everyMs, warmupMs } = input; + const stepStartMs = startedMs + index * everyMs; + const stepEndMs = stepStartMs + everyMs; + const toMs = + input.endedMs === undefined + ? stepEndMs + : Math.min(stepEndMs, input.endedMs); + const fromMs = Math.min(stepStartMs + (index === 0 ? warmupMs : 0), toMs); + return { fromMs, toMs }; +} diff --git a/src/analyze/index.ts b/src/analyze/index.ts index 769c153..1c100ee 100644 --- a/src/analyze/index.ts +++ b/src/analyze/index.ts @@ -15,6 +15,7 @@ export { } from "./analyzeSoak.schema.js"; export { computeHops } from "./computeHops.js"; export { countRestarts } from "./countRestarts.js"; +export { capacityStepWindow } from "./capacityStepWindow.js"; export { cpuPctOver } from "./cpuPctOver.js"; export { createAnalyzeCapacity } from "./createAnalyzeCapacity.js"; export { diff --git a/src/cli/formatPlan.ts b/src/cli/formatPlan.ts index 6f60a11..d312e01 100644 --- a/src/cli/formatPlan.ts +++ b/src/cli/formatPlan.ts @@ -1,7 +1,6 @@ /** * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. */ -import { OTLP_PORT, SIP_PORTS } from "../infra/constants.js"; import type { Plan } from "../infra/index.js"; import { parseDuration } from "../infra/parseDuration.js"; import type { BenchmarkConfig } from "../schemas/index.js"; @@ -60,7 +59,7 @@ export function formatPlan( `${plan.loadgen.size.padEnd(sizeWidth)}${plan.loadgen.image}` ]), ...field("firewall", [ - `SIP (${SIP_PORTS} tcp/udp) and OTLP (${OTLP_PORT} tcp) from the VPC only; SSH (22) from ${plan.sshSource.replace(/\/32$/, "")} and the VPC only` + `from the VPC: all ports (tcp/udp) and icmp; SSH (22) from ${plan.sshSource.replace(/\/32$/, "")} and the VPC only` ]), ...field("tags", [plan.tags.join(", ")]), ...field( diff --git a/src/infra/buildFirewall.ts b/src/infra/buildFirewall.ts index 657ff8c..755a455 100644 --- a/src/infra/buildFirewall.ts +++ b/src/infra/buildFirewall.ts @@ -1,16 +1,17 @@ /** * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. */ -import { OTLP_PORT, SIP_PORTS } from "./constants.js"; import type { DoFirewallInput } from "./types.js"; const ANYWHERE = { addresses: ["0.0.0.0/0", "::/0"] }; /** - * A run's firewall: SIP (tcp and udp) and OTLP (tcp) from the VPC's range - * only, SSH (22) from `sshSources` and from the VPC (the agent on the load - * generator drives the UUT over SSH at its private IP), and every outbound - * connection. It applies to the run's droplets. + * A run's firewall: every tcp and udp port and icmp from the VPC's range + * (the seed script reaches the UUT's API, and the UUT opens new flows to the + * callees on the load generator), SSH (22) from `sshSources` and from the + * VPC (the agent on the load generator drives the UUT over SSH at its + * private IP), and every outbound connection. It applies to the run's + * droplets. */ export function buildFirewall(input: { name: string; @@ -22,9 +23,9 @@ export function buildFirewall(input: { return { name: input.name, inbound_rules: [ - { protocol: "udp", ports: SIP_PORTS, sources: vpc }, - { protocol: "tcp", ports: SIP_PORTS, sources: vpc }, - { protocol: "tcp", ports: OTLP_PORT, sources: vpc }, + { protocol: "tcp", ports: "all", sources: vpc }, + { protocol: "udp", ports: "all", sources: vpc }, + { protocol: "icmp", sources: vpc }, { protocol: "tcp", ports: "22", diff --git a/src/infra/constants.ts b/src/infra/constants.ts index 650fdf0..f5c3279 100644 --- a/src/infra/constants.ts +++ b/src/infra/constants.ts @@ -12,11 +12,6 @@ export const PERFKIT_TAG = "perfkit"; /** Name of the SSH key that PERFKIT_SSH_KEY registers; never deleted. */ export const SHARED_KEY_NAME = "perfkit-shared"; -/** SIP from the VPC. */ -export const SIP_PORTS = "5060-5063"; -/** OTLP/HTTP from the VPC. */ -export const OTLP_PORT = "4318"; - /** Droplets active and cloud-init finished within this, or exit 3. */ export const READY_TIMEOUT_MS = 10 * 60 * 1000; /** How long teardown waits for a VPC to have no members. */ diff --git a/src/orchestrate/createRunBenchmark.ts b/src/orchestrate/createRunBenchmark.ts index 381f986..1b3263b 100644 --- a/src/orchestrate/createRunBenchmark.ts +++ b/src/orchestrate/createRunBenchmark.ts @@ -2,6 +2,7 @@ * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. */ import { basename } from "node:path"; +import { capacityStepWindow } from "../analyze/capacityStepWindow.js"; import { durationToSeconds } from "../analyze/durationToSeconds.js"; import type { Now, Sleep } from "../infra/index.js"; import type { BenchmarkConfig } from "../schemas/index.js"; @@ -543,18 +544,18 @@ export function createRunBenchmark(deps: AgentDeps, state: RunState) { ...base, kind: "capacity", steps: Array.from({ length: Math.max(steps, 1) }, (_, index) => { - const everyMs = - durationToSeconds(benchmark.load.rate.every) * 1000; + const window = capacityStepWindow({ + startedMs, + index, + everyMs: durationToSeconds(benchmark.load.rate.every) * 1000, + warmupMs: durationToSeconds(benchmark.warmup) * 1000, + endedMs: Math.max(endedMs, startedMs + 1) + }); return { offeredCps: benchmark.load.rate.start + index * benchmark.load.rate.step, - from: iso(startedMs + index * everyMs), - to: iso( - Math.min( - startedMs + (index + 1) * everyMs, - Math.max(endedMs, startedMs + 1) - ) - ) + from: iso(window.fromMs), + to: iso(window.toMs) }; }) } diff --git a/src/orchestrate/evaluateCapacityStep.ts b/src/orchestrate/evaluateCapacityStep.ts index de783d0..2105813 100644 --- a/src/orchestrate/evaluateCapacityStep.ts +++ b/src/orchestrate/evaluateCapacityStep.ts @@ -3,6 +3,7 @@ */ import { join } from "node:path"; import { + capacityStepWindow, measureCapacityStep, parseSamplerJsonl, parseUdpJsonl, @@ -52,10 +53,12 @@ export async function evaluateCapacityStep(input: { names.includes(file) ? deps.readFile(join(rawDir, file)) : ""; const measured = measureCapacityStep({ offeredCps, - window: { - fromMs: input.startMs + index * everyMs, - toMs: input.startMs + (index + 1) * everyMs - }, + window: capacityStepWindow({ + startedMs: input.startMs, + index, + everyMs, + warmupMs: durationMs(benchmark.warmup) + }), uac, samples: parseSamplerJsonl( await read("sampler-loadgen.jsonl"), diff --git a/src/schemas/checkWarmupShorterThanStep.ts b/src/schemas/checkWarmupShorterThanStep.ts new file mode 100644 index 0000000..ebe44a1 --- /dev/null +++ b/src/schemas/checkWarmupShorterThanStep.ts @@ -0,0 +1,33 @@ +/** + * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. + */ +import type { z } from "zod/v4"; + +const UNIT_S = { s: 1, m: 60, h: 3600 } as const; + +/** + * Adds an issue at `warmup` when a capacity benchmark's warm-up is not + * shorter than one step (`load.rate.every`): the first step would have + * nothing left to measure. A `0s` warm-up is always fine. For use inside the + * capacity benchmark's `superRefine`. + */ +export function checkWarmupShorterThanStep( + benchmark: { warmup: string; load: { rate: { every: string } } }, + ctx: z.core.$RefinementCtx +): void { + const warmupS = seconds(benchmark.warmup); + const everyS = seconds(benchmark.load.rate.every); + if (warmupS > 0 && warmupS >= everyS) { + ctx.addIssue({ + code: "custom", + path: ["warmup"], + message: `expected a warm-up shorter than load.rate.every (${benchmark.load.rate.every})`, + input: benchmark.warmup + }); + } +} + +function seconds(duration: string): number { + const match = /^(\d+)([smh])$/.exec(duration); + return match ? Number(match[1]) * UNIT_S[match[2] as keyof typeof UNIT_S] : 0; +} diff --git a/src/schemas/perfkitConfig.schema.ts b/src/schemas/perfkitConfig.schema.ts index dd63d3a..5588d60 100644 --- a/src/schemas/perfkitConfig.schema.ts +++ b/src/schemas/perfkitConfig.schema.ts @@ -9,6 +9,7 @@ */ import { z } from "zod/v4"; import { checkUniqueIds } from "./checkUniqueIds.js"; +import { checkWarmupShorterThanStep } from "./checkWarmupShorterThanStep.js"; import { benchmarkIdSchema, count, @@ -175,16 +176,20 @@ const narrativeSchema = z.strictObject({ how: nonEmptyString }); -export const capacityBenchmarkConfigSchema = z.strictObject({ - id: benchmarkIdSchema, - kind: z.literal("capacity"), - title: nonEmptyString, - narrative: narrativeSchema, - uut: configUutSchema, - load: capacityLoadSchema, - telemetry: telemetrySchema, - thresholds: capacityThresholdsSchema -}); +export const capacityBenchmarkConfigSchema = z + .strictObject({ + id: benchmarkIdSchema, + kind: z.literal("capacity"), + title: nonEmptyString, + narrative: narrativeSchema, + // The first step's measurement starts this long after the step does. + warmup: durationSchema.default("0s"), + uut: configUutSchema, + load: capacityLoadSchema, + telemetry: telemetrySchema, + thresholds: capacityThresholdsSchema + }) + .superRefine(checkWarmupShorterThanStep); export const soakBenchmarkConfigSchema = z.strictObject({ id: benchmarkIdSchema, diff --git a/test/analyze/capacityStepWindow.unit.test.ts b/test/analyze/capacityStepWindow.unit.test.ts new file mode 100644 index 0000000..becbe11 --- /dev/null +++ b/test/analyze/capacityStepWindow.unit.test.ts @@ -0,0 +1,90 @@ +/** + * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. + */ +import { expect } from "chai"; +import { + capacityStepWindow, + measureCapacityStep, + parseSamplerJsonl, + parseUdpJsonl, + readSippResults +} from "../../src/analyze/index.js"; +import { + capacityRaw, + fakeDeps, + RAW_DIR, + START_MS, + STEP_MS +} from "./builders.js"; + +describe("capacityStepWindow", () => { + it("should start only the first step's window warmup later", () => { + const window = (index: number) => + capacityStepWindow({ + startedMs: 0, + index, + everyMs: 60_000, + warmupMs: 20_000 + }); + expect(window(0)).to.deep.equal({ fromMs: 20_000, toMs: 60_000 }); + expect(window(1)).to.deep.equal({ fromMs: 60_000, toMs: 120_000 }); + }); + + it("should cut the window at endedMs and never start after it ends", () => { + expect( + capacityStepWindow({ + startedMs: 0, + index: 0, + everyMs: 60_000, + warmupMs: 20_000, + endedMs: 10_000 + }) + ).to.deep.equal({ fromMs: 10_000, toMs: 10_000 }); + }); + + it("should leave the warm-up's slow calls and stat rows out of the first step", async () => { + // Arrange: the first 30 of 100 calls (sent in the first 18 s) take 200 ms. + const latencies = [ + ...Array.from({ length: 30 }, () => 200), + ...Array.from({ length: 70 }, () => 5) + ]; + const files = capacityRaw([{ offeredCps: 100, latencies }]); + const uac = await readSippResults( + fakeDeps(files), + RAW_DIR, + Object.keys(files), + "uac_message.xml" + ); + const measure = (warmupMs: number) => + measureCapacityStep({ + offeredCps: 100, + window: capacityStepWindow({ + startedMs: START_MS, + index: 0, + everyMs: STEP_MS, + warmupMs + }), + uac: uac!, + samples: parseSamplerJsonl( + files["sampler.jsonl"] as string, + "sampler.jsonl" + ), + udp: parseUdpJsonl(files["udp.jsonl"] as string, "udp.jsonl"), + uutProcesses: ["edgeport"], + unit: "calls" + }).step; + + // Act + const cold = measure(0); + const warm = measure(20_000); + + // Assert: the stat rows at 10 s and 20 s (mid-points 5 s and 15 s) are out. + expect(cold.p99Ms).to.equal(200); + expect(warm).to.include({ + p99Ms: 5, + calls: 4000, + achievedCps: 100, + valid: true + }); + }); +}); diff --git a/test/infra/helpers.unit.test.ts b/test/infra/helpers.unit.test.ts index 8ce85a5..0879504 100644 --- a/test/infra/helpers.unit.test.ts +++ b/test/infra/helpers.unit.test.ts @@ -73,7 +73,7 @@ describe("buildCloudInit", () => { }); describe("buildFirewall", () => { - it("should open SIP and OTLP to the VPC, SSH to the sources, and all outbound", () => { + it("should open every tcp/udp port and icmp to the VPC, SSH to the sources, and all outbound", () => { // Act const firewall = buildFirewall({ name: "perfkit-x", @@ -89,9 +89,9 @@ describe("buildFirewall", () => { `${protocol} ${ports} ${sources.addresses?.join(",")}` ) ).to.deep.equal([ - "udp 5060-5063 10.1.0.0/20", - "tcp 5060-5063 10.1.0.0/20", - "tcp 4318 10.1.0.0/20", + "tcp all 10.1.0.0/20", + "udp all 10.1.0.0/20", + "icmp undefined 10.1.0.0/20", "tcp 22 203.0.113.7/32,10.1.0.0/20" ]); expect( diff --git a/test/infra/run.unit.test.ts b/test/infra/run.unit.test.ts index 86f45b1..807c0ad 100644 --- a/test/infra/run.unit.test.ts +++ b/test/infra/run.unit.test.ts @@ -24,7 +24,7 @@ const PLAN = `plan for Routr 2.31.5 (dry run, nothing created) region nyc3, new VPC (range assigned by DigitalOcean) uut c-8 ubuntu-24-04-x64 loadgen c-16 ubuntu-24-04-x64 - firewall SIP (5060-5063 tcp/udp) and OTLP (4318 tcp) from the VPC only; SSH (22) from 203.0.113.7 and the VPC only + firewall from the VPC: all ports (tcp/udp) and icmp; SSH (22) from 203.0.113.7 and the VPC only tags perfkit, perfkit-run-${RUN_ID} images fonoster/routr-edgeport:2.31.5@sha256:4f1c…e2a0 fonoster/routr-echo:2.31.5@sha256:41ab…77d0 diff --git a/test/orchestrate/agent.unit.test.ts b/test/orchestrate/agent.unit.test.ts index 98c59d5..1c34f5b 100644 --- a/test/orchestrate/agent.unit.test.ts +++ b/test/orchestrate/agent.unit.test.ts @@ -2,8 +2,9 @@ * Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License. */ import { expect } from "chai"; -import { readFile, writeFile } from "node:fs/promises"; -import { join } from "node:path"; +import { cp, mkdtemp, readFile, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; import { ValidationError } from "../../src/errors/index.js"; import { createNodeAnalyzeDeps } from "../../src/analyze/index.js"; import { analyzeBenchmark } from "../../src/orchestrate/analyzeBenchmark.js"; @@ -17,7 +18,13 @@ import { } from "../analyze/builders.js"; import { ROUTR_CONFIG } from "../fixtures.js"; import { fakeRegistry, perfkit } from "../infra/fakes.js"; -import { doSetup, MINIMAL, PASSING_STEPS, type RawFiles } from "./setup.js"; +import { + doSetup, + MINIMAL, + MINIMAL_CONFIG, + PASSING_STEPS, + type RawFiles +} from "./setup.js"; /** A detached run whose agent the test runs itself. */ async function detached(args: string[], files: RawFiles, routr = false) { @@ -87,6 +94,71 @@ describe("createAgent", () => { ); }); + it("should leave a capacity warm-up out of the first step, live and in the final verdict", async () => { + // Arrange: the first 30 of step 1's 100 messages (sent in its first 18 s) + // are slow, as with a cold JIT; a 20 s warm-up leaves them out. + const dir = await mkdtemp(join(tmpdir(), "perfkit-warmup-")); + await cp(dirname(MINIMAL_CONFIG), dir, { recursive: true }); + const config = join(dir, "perfkit.yaml"); + const yaml = await readFile(config, "utf8"); + await writeFile( + config, + yaml.replace("kind: capacity\n", "kind: capacity\n warmup: 20s\n") + ); + const cold = [ + ...Array.from({ length: 30 }, () => 200), + ...Array.from({ length: 70 }, () => 5) + ]; + const steps = [ + { offeredCps: 50, latencies: cold }, + ...PASSING_STEPS.slice(1) + ]; + const setup = await detached( + ["run", "--config", config, "--target-version", "3.7.3"], + capacityRaw(steps) + ); + + // Act + await setup.agent.run(setup.runId()); + + // Assert + const progress = join(setup.hosts.roots.loadgen, "progress.log"); + expect((await readFile(progress, "utf8")).trim().split("\n")).to.deep.equal( + [ + "[sipp-uas] step 50 messages/s p99 5.0 ms failed 0.00% ok", + "[sipp-uas] step 100 messages/s p99 5.0 ms failed 0.00% ok", + "[sipp-uas] step 150 messages/s p99 5.0 ms failed 0.00% ok", + "[sipp-uas] step 200 messages/s p99 5.0 ms failed 0.00% ok" + ] + ); + const raw = join(setup.hosts.roots.loadgen, "raw", "sipp-uas"); + const meta = JSON.parse(await readFile(join(raw, "meta.json"), "utf8")); + expect(meta.steps[0].from).to.equal( + new Date(START_MS + 20_000).toISOString() + ); + expect(meta.steps[1].from).to.equal( + new Date(START_MS + 60_000).toISOString() + ); + const state = runStateSchema.parse( + JSON.parse( + await readFile(join(setup.hosts.roots.loadgen, "run.json"), "utf8") + ) + ); + const result = await analyzeBenchmark( + createNodeAnalyzeDeps(), + state.plan.benchmarks[0]!, + raw, + null, + "SIPp 3.7.3, UDP" + ); + expect(result).to.include({ verdict: "passed" }); + expect("series" in result && result.series?.steps[0]).to.include({ + p99Ms: 5, + achievedCps: 50, + valid: true + }); + }); + it("should run compose down for the first benchmark before compose up for the second, and the second's setup once", async () => { // Arrange const setup = await detached( diff --git a/test/schemas/perfkitConfig.unit.test.ts b/test/schemas/perfkitConfig.unit.test.ts index 2efaad1..9c61564 100644 --- a/test/schemas/perfkitConfig.unit.test.ts +++ b/test/schemas/perfkitConfig.unit.test.ts @@ -41,6 +41,17 @@ describe("perfkitConfigSchema", () => { expect(config.benchmarks[0]?.load.uac.users).to.equal(undefined); }); + it("should default a capacity warm-up to 0s and accept one shorter than a step", () => { + const raw = loadConfig(); + expect(perfkitConfigSchema.parse(raw).benchmarks[0]?.warmup).to.equal( + "0s" + ); + raw.benchmarks[0].warmup = "20s"; + expect(perfkitConfigSchema.parse(raw).benchmarks[0]?.warmup).to.equal( + "20s" + ); + }); + it("should accept a project without a logo", () => { const raw = loadConfig(); delete raw.project.logo; @@ -65,6 +76,15 @@ describe("perfkitConfigSchema", () => { ]); }); + it("should reject a capacity warm-up as long as a step", () => { + const raw = loadConfig(); + raw.benchmarks[0].warmup = "1m"; + raw.benchmarks[0].load.rate.every = "60s"; + expect(errorsOf(perfkitConfigSchema, raw)).to.deep.equal([ + "benchmarks[0].warmup: expected a warm-up shorter than load.rate.every (60s)" + ]); + }); + it("should require duration and warmup in a soak benchmark", () => { const raw = loadConfig(); delete raw.benchmarks[1].duration;