Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -390,7 +390,8 @@ SIPp adds load in steps, such as 50 more calls/s every minute. For each step:
- Its latency counts the calls whose request was **sent** in the step, even when the answer came
in the next one.
- Its attempted, failed and retransmitted calls are SIPp's 10-second counters (`-trace_stat`)
that fall in the step. Failed % is failed / attempted.
that fall in the step. Failed % is failed / attempted. So `every` must be a multiple of 10 s:
`perfkit validate` rejects a `15s` step, whose counters would mix two steps.
- **Warm-up.** With `warmup` set, the first step still runs its full `every`, but it is measured
only from `warmup` after it starts: calls sent, and 10-second counters whose period mid-point
falls, before that don't count. A cold JIT (EdgePort's JVM and GraalJS) makes the first seconds
Expand Down
4 changes: 3 additions & 1 deletion docs/perfkit-yaml.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ Rules for the whole file:
| `project.siteUrl` | URL | yes | | Where the report pages are published, like `https://routr.io/benchmarks`. |
| `project.accent` | string | yes | | 6-digit hex color, like `#5FCFCE`: the page's accent color. |
| `project.logo` | path | no | none | An `.svg` file. Without it, the nav shows `project.name`. |
| `project.favicon` | path | no | none | An `.svg` file: the browser tab's icon on every page. Read when the run starts and inlined into report.json and live.json. Without it, the pages have no icon. |
| `project.title` | string | yes | | The page's headline. |
| `project.lead` | string | yes | | The paragraph under the headline. |
| `project.live.url` | URL | no | `live.url`, else `https://<live.domain>/live.json` | Where the benchmarks page polls a run in progress. Goes into report.json, so the published page knows it. |
Expand Down Expand Up @@ -124,7 +125,7 @@ the other kind rejects them.
| `load.unit` | string | no | `calls` | What one SIPp call is, as a lowercase plural noun shown on the page (`calls`, `messages`). Copied into each report.json benchmark as `unit`. |
| `load.rate.start` | number (calls/s) | *capacity*: yes | | The first step's rate. |
| `load.rate.step` | number (calls/s) | *capacity*: yes | | Added at each step. |
| `load.rate.every` | duration | *capacity*: yes | | How long each step lasts. |
| `load.rate.every` | duration | *capacity*: yes | | How long each step lasts. A multiple of 10 s, because SIPp reports its counters every 10 s; `15s` is rejected (`expected a multiple of 10 s (SIPp reports every 10 s); got 15s`). |
| `load.rate.max` | number (calls/s) | *capacity*: yes | | The ramp stops here. Reaching it without breaking a rule makes the limit a lower bound. |
| `load.rate.cps` | number (calls/s) | *soak*: yes | | The fixed rate. Keep it well below saturation: a soak looks for leaks, not limits. |
| `telemetry` | object | no | `{ otlp: false }` | |
Expand Down Expand Up @@ -156,6 +157,7 @@ project:
siteUrl: https://routr.io/benchmarks
accent: "#5FCFCE" # 6-digit hex; the page's accent color
logo: logo.svg # optional SVG; without it the nav shows project.name
favicon: favicon.svg # optional SVG; the browser tab's icon
title: How much SIP traffic Routr handles.
lead: We run these tests after significant changes to the code, on dedicated DigitalOcean servers, one for Routr and one for the load generator. The EdgePort runs alone to find its limit; the whole stack runs for hours to catch leaks.
infra:
Expand Down
3 changes: 2 additions & 1 deletion docs/report-page.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ history and picker list every run published since, and a newer perfkit redraws i

## Nav and header

The nav has the project's logo, the run picker and a link to the run's `report.json`. The picker
With `project.faviconSvg` in report.json, every page and shell links it as the tab's icon. The
nav has the project's logo, the run picker and a link to the run's `report.json`. The picker
lists up to 8 runs, newest first, each a link to its page; "See all" jumps to Previous runs.

The header has the verdict pill, `N of M benchmarks passed`, followed by how many were not run,
Expand Down
1 change: 1 addition & 0 deletions docs/report-v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ Rules for both files:
| :--- | :--- | :--- |
| `project.name`, `repo`, `siteUrl`, `accent`, `title`, `lead` | string | Copied from perfkit.yaml. |
| `project.logoSvg` | string | The SVG markup of `project.logo`, inlined. Present only when perfkit.yaml has `logo`. |
| `project.faviconSvg` | string | The SVG markup of `project.favicon`, inlined. Every page links it as its icon (a URL-encoded `data:image/svg+xml` URI). Present only when perfkit.yaml has `favicon`. |
| `project.live.url` | URL | Where the latest page polls live.json: `project.live.url`, else `live.url`, else `https://<live.domain>/live.json` of perfkit.yaml. Absent without them. |

### run
Expand Down
4 changes: 4 additions & 0 deletions schemas/live-v1.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,10 @@
"type": "string",
"pattern": "<svg[\\s>]"
},
"faviconSvg": {
"type": "string",
"pattern": "<svg[\\s>]"
},
"title": {
"type": "string",
"minLength": 1
Expand Down
4 changes: 4 additions & 0 deletions schemas/perfkit-v1.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@
"type": "string",
"minLength": 1
},
"favicon": {
"type": "string",
"minLength": 1
},
"title": {
"type": "string",
"minLength": 1
Expand Down
4 changes: 4 additions & 0 deletions schemas/report-v1.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@
"type": "string",
"pattern": "<svg[\\s>]"
},
"faviconSvg": {
"type": "string",
"pattern": "<svg[\\s>]"
},
"title": {
"type": "string",
"minLength": 1
Expand Down
7 changes: 7 additions & 0 deletions src/cli/createStartExecutor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,12 @@ export function createStartExecutor(deps: CliDeps): BenchmarkExecutor {
"utf8"
)
: undefined;
const faviconSvg = config.project.favicon
? await readFile(
join(dirname(options.configPath), config.project.favicon),
"utf8"
)
: undefined;
await createStartRun({
remote: createSshRemote({
ssh: infra.ssh,
Expand All @@ -47,6 +53,7 @@ export function createStartExecutor(deps: CliDeps): BenchmarkExecutor {
plan,
config,
...(logoSvg ? { logoSvg: logoSvg.trim() } : {}),
...(faviconSvg ? { faviconSvg: faviconSvg.trim() } : {}),
hosts: {
uutIp: run.uut.privateIp,
loadgenIp: run.loadgen.privateIp,
Expand Down
8 changes: 8 additions & 0 deletions src/cli/runRunCommand.ts
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,13 @@ async function runLocal(
await deps.readFile(join(dirname(options.config), config.project.logo))
).trim()
: undefined;
const favicon = config.project.favicon
? (
await deps.readFile(
join(dirname(options.config), config.project.favicon)
)
).trim()
: undefined;
try {
await createStartRun({
remote: createLocalRemote({ run: infra.runProgram }),
Expand All @@ -284,6 +291,7 @@ async function runLocal(
plan,
config,
...(logo ? { logoSvg: logo } : {}),
...(favicon ? { faviconSvg: favicon } : {}),
hosts: {
uutIp: "127.0.0.1",
loadgenIp: "127.0.0.1",
Expand Down
1 change: 1 addition & 0 deletions src/orchestrate/createStartRun.ts
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,7 @@ export function createStartRun(deps: StartRunDeps) {
plan: input.plan,
config: input.config,
...(input.logoSvg ? { logoSvg: input.logoSvg } : {}),
...(input.faviconSvg ? { faviconSvg: input.faviconSvg } : {}),
hosts: input.hosts,
...(input.live ? { live: input.live } : {}),
machines: { uut, loadgen },
Expand Down
1 change: 1 addition & 0 deletions src/orchestrate/runState.schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ export const runStateSchema = z.strictObject({
plan: planSchema,
config: perfkitConfigSchema,
logoSvg: z.string().optional(),
faviconSvg: z.string().optional(),
hosts: z.strictObject({
uutIp: z.ipv4(),
loadgenIp: z.ipv4(),
Expand Down
1 change: 1 addition & 0 deletions src/orchestrate/startRun.schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ export const startRunInputSchema = z.strictObject({
plan: true,
config: true,
logoSvg: true,
faviconSvg: true,
hosts: true,
live: true
}).shape,
Expand Down
1 change: 1 addition & 0 deletions src/orchestrate/toReportProject.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ export function toReportProject(state: RunState): Report["project"] {
siteUrl: project.siteUrl,
accent: project.accent,
...(state.logoSvg ? { logoSvg: state.logoSvg } : {}),
...(state.faviconSvg ? { faviconSvg: state.faviconSvg } : {}),
title: project.title,
lead: project.lead,
...(liveUrl ? { live: { url: liveUrl } } : {})
Expand Down
8 changes: 6 additions & 2 deletions src/render/createRenderSite.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,8 @@ type History =
* (`app/<version>/app.js`), this run's report (`data/runs/<id>/report.json`,
* and a copy in `runs/<id>/` for the page's report.json link), the history
* (`data/runs.json`), and the shells (`index.html` and one
* `runs/<id>/index.html` per run in the history), which load the app. The
* `runs/<id>/index.html` per run in the history), which load the app and
* carry this report's favicon, so the tab has its icon before the app draws. The
* app draws each page from the data when it is viewed, so older reports are
* never read or rewritten, and a new perfkit redraws every run.
*
Expand Down Expand Up @@ -89,7 +90,10 @@ export function createRenderSite(deps: RenderSiteDeps) {
project: merged.project,
run,
page,
perfkitVersion: deps.version
perfkitVersion: deps.version,
...(report.project.faviconSvg
? { faviconSvg: report.project.faviconSvg }
: {})
});
const at = (path: string) => join(out, ...path.split("/"));

Expand Down
3 changes: 2 additions & 1 deletion src/render/live/renderLivePage.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import { esc } from "../html.js";
import { pageStyles } from "../pageStyles.js";
import { PICKER_SCRIPT } from "../pickerScript.js";
import { renderFaviconLink } from "../renderFaviconLink.js";
import { liveStyles } from "./liveStyles.js";
import { renderLiveView } from "./renderLiveView.js";
import type { LiveViewContext } from "./types.js";
Expand All @@ -25,7 +26,7 @@ export function renderLivePage(ctx: LiveViewContext): string {
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>${esc(`${live.project.name} benchmarks · running now · v${live.version}`)}</title>
<meta name="description" content="${esc(live.project.title)}">
${renderFaviconLink(live.project.faviconSvg)}<meta name="description" content="${esc(live.project.title)}">
<meta name="generator" content="perfkit ${esc(ctx.perfkitVersion)}">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
Expand Down
14 changes: 14 additions & 0 deletions src/render/renderFaviconLink.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
/**
* Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License.
*/

/**
* The page head's icon link for `project.faviconSvg`, as a URL-encoded data
* URI and ending in a newline, or an empty string without a favicon, so a
* page without one is unchanged.
*/
export function renderFaviconLink(faviconSvg: string | undefined): string {
return faviconSvg
? `<link rel="icon" type="image/svg+xml" href="data:image/svg+xml,${encodeURIComponent(faviconSvg)}">\n`
: "";
}
3 changes: 2 additions & 1 deletion src/render/renderRunPage.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
*/
import { esc } from "./html.js";
import { pageStyles } from "./pageStyles.js";
import { renderFaviconLink } from "./renderFaviconLink.js";
import { PICKER_SCRIPT } from "./pickerScript.js";
import { renderCapacitySection } from "./sections/renderCapacitySection.js";
import { renderFooter } from "./sections/renderFooter.js";
Expand Down Expand Up @@ -38,7 +39,7 @@ export function renderRunPage(ctx: PageContext): string {
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>${esc(title)}</title>
<meta name="description" content="${esc(report.project.title)}">
${renderFaviconLink(report.project.faviconSvg)}<meta name="description" content="${esc(report.project.title)}">
<meta name="generator" content="perfkit ${esc(ctx.perfkitVersion)}">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
Expand Down
7 changes: 5 additions & 2 deletions src/render/renderShell.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import type { RunEntry } from "../schemas/index.js";
import { SITE_PATHS } from "./app/sitePaths.js";
import { formatDate, formatTime, shortCommit } from "./format.js";
import { esc } from "./html.js";
import { renderFaviconLink } from "./renderFaviconLink.js";
import type { PageKind } from "./types.js";
import { summaryText } from "./verdicts.js";

Expand All @@ -21,8 +22,10 @@ export function renderShell(params: {
run: RunEntry;
page: PageKind;
perfkitVersion: string;
/** The project's favicon, from the report being rendered. */
faviconSvg?: string;
}): string {
const { project, run, page, perfkitVersion } = params;
const { project, run, page, perfkitVersion, faviconSvg } = params;
const base = page === "latest" ? "" : "../../";
const title = `${project} benchmarks · v${run.version}`;
const description = `${summaryText(run.verdict)}. ${project} v${run.version}, run of ${formatDate(run.startedAt)} ${formatTime(run.startedAt)} UTC, commit ${shortCommit(run.commit)}.`;
Expand All @@ -36,7 +39,7 @@ export function renderShell(params: {
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>${esc(title)}</title>
<meta name="description" content="${esc(description)}">
${renderFaviconLink(faviconSvg)}<meta name="description" content="${esc(description)}">
<meta property="og:type" content="website">
<meta property="og:site_name" content="${esc(`${project} benchmarks`)}">
<meta property="og:title" content="${esc(title)}">
Expand Down
2 changes: 1 addition & 1 deletion src/render/sections/renderHistory.ts
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ export function renderHistory(ctx: PageContext): string {
<div>
<p class="eyebrow">HISTORY</p>
<h2 class="section-title" id="previous-runs">Previous runs</h2>
<p class="section-sub">Every run keeps its own page and report.json. Open one to see it exactly as it was.</p>
<p class="section-sub">Every run keeps its own page and report.json.</p>
</div>
${trend}
</div>
Expand Down
29 changes: 29 additions & 0 deletions src/schemas/checkStepEveryMultipleOfSippInterval.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
/**
* Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License.
*/
import type { z } from "zod/v4";
import { durationSeconds } from "./durationSeconds.js";

/** How often SIPp writes its statistics (`-trace_stat`), in seconds. */
export const SIPP_STAT_INTERVAL_S = 10;

/**
* Adds an issue at `load.rate.every` when a capacity step is not a whole
* number of SIPp's 10-second statistics periods: a period would then span
* two steps, and the step's counters would mix both rates. For use inside
* the capacity benchmark's `superRefine`.
*/
export function checkStepEveryMultipleOfSippInterval(
benchmark: { load: { rate: { every: string } } },
ctx: z.core.$RefinementCtx
): void {
const { every } = benchmark.load.rate;
if (durationSeconds(every) % SIPP_STAT_INTERVAL_S !== 0) {
ctx.addIssue({
code: "custom",
path: ["load", "rate", "every"],
message: `expected a multiple of ${SIPP_STAT_INTERVAL_S} s (SIPp reports every ${SIPP_STAT_INTERVAL_S} s); got ${every}`,
input: every
});
}
}
12 changes: 3 additions & 9 deletions src/schemas/checkWarmupShorterThanStep.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,7 @@
* Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License.
*/
import type { z } from "zod/v4";

const UNIT_S = { s: 1, m: 60, h: 3600 } as const;
import { durationSeconds } from "./durationSeconds.js";

/**
* Adds an issue at `warmup` when a capacity benchmark's warm-up is not
Expand All @@ -15,8 +14,8 @@ export function checkWarmupShorterThanStep(
benchmark: { warmup: string; load: { rate: { every: string } } },
ctx: z.core.$RefinementCtx
): void {
const warmupS = seconds(benchmark.warmup);
const everyS = seconds(benchmark.load.rate.every);
const warmupS = durationSeconds(benchmark.warmup);
const everyS = durationSeconds(benchmark.load.rate.every);
if (warmupS > 0 && warmupS >= everyS) {
ctx.addIssue({
code: "custom",
Expand All @@ -26,8 +25,3 @@ export function checkWarmupShorterThanStep(
});
}
}

function seconds(duration: string): number {
const match = /^(\d+)([smh])$/.exec(duration);
return match ? Number(match[1]) * UNIT_S[match[2] as keyof typeof UNIT_S] : 0;
}
13 changes: 13 additions & 0 deletions src/schemas/durationSeconds.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
/**
* Copyright (C) 2026 by Fonoster Inc (https://fonoster.com). MIT License.
*/
const UNIT_S = { s: 1, m: 60, h: 3600 } as const;

/**
* The seconds in a perfkit duration (`30s`, `5m`, `2h`), or 0 when it is not
* one; the schema has already rejected a bad duration by then.
*/
export function durationSeconds(duration: string): number {
const match = /^(\d+)([smh])$/.exec(duration);
return match ? Number(match[1]) * UNIT_S[match[2] as keyof typeof UNIT_S] : 0;
}
9 changes: 8 additions & 1 deletion src/schemas/perfkitConfig.schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import { z } from "zod/v4";
import { checkUniqueIds } from "./checkUniqueIds.js";
import { projectLiveSchema } from "./report.schema.js";
import { checkStepEveryMultipleOfSippInterval } from "./checkStepEveryMultipleOfSippInterval.js";
import { checkWarmupShorterThanStep } from "./checkWarmupShorterThanStep.js";
import {
benchmarkIdSchema,
Expand Down Expand Up @@ -40,6 +41,11 @@ const configProjectSchema = z.strictObject({
error: "expected an .svg file"
})
.optional(),
favicon: relativePathSchema
.refine((value) => value.toLowerCase().endsWith(".svg"), {
error: "expected an .svg file"
})
.optional(),
title: nonEmptyString,
lead: nonEmptyString,
live: projectLiveSchema.optional()
Expand Down Expand Up @@ -230,7 +236,8 @@ export const capacityBenchmarkConfigSchema = z
telemetry: telemetrySchema,
thresholds: capacityThresholdsSchema
})
.superRefine(checkWarmupShorterThanStep);
.superRefine(checkWarmupShorterThanStep)
.superRefine(checkStepEveryMultipleOfSippInterval);

export const soakBenchmarkConfigSchema = z.strictObject({
id: benchmarkIdSchema,
Expand Down
4 changes: 4 additions & 0 deletions src/schemas/report.schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,10 @@ export const reportProjectSchema = z.strictObject({
.string()
.regex(/<svg[\s>]/, { error: "expected inline SVG markup" })
.optional(),
faviconSvg: z
.string()
.regex(/<svg[\s>]/, { error: "expected inline SVG markup" })
.optional(),
title: nonEmptyString,
lead: nonEmptyString,
live: projectLiveSchema.optional()
Expand Down
1 change: 1 addition & 0 deletions src/validate/checkConfigPaths.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ export async function checkConfigPaths(
): Promise<DocumentError[]> {
const named: Array<[string, string | undefined]> = [
["project.logo", config.project.logo],
["project.favicon", config.project.favicon],
...config.benchmarks.flatMap(
(benchmark, i): Array<[string, string | undefined]> => {
const at = `benchmarks[${i}]`;
Expand Down
Loading
Loading