[psanders]

Closes #24
Closes #25

perfkit now runs a project's benchmarks over TCP, on one server and on Kubernetes, in one run, and capacity benchmarks can register callees (#24, folded into #25).

What changed

perfkit.yaml v1 (additive; every existing config validates unchanged)

• load.transport: udp | tcp (default udp). TCP: every SIPp role runs -t tn with -max_socket = its -l + 100 (at most 60000) and --ulimit nofile=65536:65536.
• infra.cluster: { size, nodes, version = "latest" }, required when a benchmark has uut.helm.
• uut is either the existing { compose, setup?, processes } or { helm: { chart, values, entry: { service, port }, setup?, pods: [{ name, selector }] } }.
• load.register / load.uas on capacity benchmarks too (A capacity benchmark can't register callee agents or ramp calls to them #24): register role -l rows + ceil(max × (hold + 20)), uas-only -l ceil(max × (hold + 20)); every capacity uac gets -recv_timeout 10000 and -l ceil(max × (hold + 20)); the uac starts only after every callee registered (soak's timeout and abort reason); a step is invalid when any SIPp role (sipp-register, sipp-uas too) is over 80% of one core; raw capacity meta.json gains registerUsers.
• Soak rate is { cps } or { fromCapacity, fraction, fallbackCps }.

Runs

• DigitalOcean: a DOKS cluster perfkit-<run id lowercase> in the run's VPC, tagged perfkit + perfkit-run-<id>, created with the droplets and waited for in parallel (20 min). The firewall opens all tcp/udp from its node droplets; the kubeconfig (valid for maxLifetime) goes to /opt/perfkit/keys/kubeconfig (600) on the load generator only.
• Teardown deletes the cluster with destroy_with_associated_resources/dangerous in parallel with the droplets (VPC wait 10 min after a cluster). perfkit sweep finds clusters tagged perfkit named perfkit-<run id>; collect's teardown (sweep by run id) covers them.
• --local: kind perfkit-<run id> with infra.cluster.nodes nodes, MetalLB v0.14.9 L2, images the charts use (helm template) or images: names that exist locally loaded with kind load docker-image; deleted by the local teardown.
• Agent order: capacity benchmarks one after another; then soaks, different targets at the same time, one target one after another.
• Helm benchmark: metrics-server, chart install (PERFKIT_ENTRY_IP=0.0.0.0), wait for the entry Service's LB address, helm upgrade --wait with it resolved, every pod Ready, setup, then SIPp to <lb ip>:<entry.port>; helm uninstall + namespace delete at the end. Cluster health check is /readyz.
• Sampling on the cluster: a kubectl/jq sampler reads pods + Metrics API every 10 s into kube-uut.jsonl, converted to the usual sampler lines (summed CPU integrated into ticks, working-set memory, pid bumps on container restarts).

report.json v1 (optional fields): run.infra.cluster, benchmarks[].target, soak load.rateSource. The page puts the target in each section's eyebrow (01 · CAPACITY · Single server, 02 · SOAK · Kubernetes, 3 × c-8); old reports render with no label.

Decisions

• Identical SIPp args for UDP + Compose: soak args are byte-identical (tested) except -cid_str, which now carries the run id on every role, soak and capacity alike (<role>-<run id>-%u-%p@%s, requested mid-task so Call-IDs never repeat across runs). Capacity uac args differ further only by A capacity benchmark can't register callee agents or ramp calls to them #24's -recv_timeout 10000 and the new -l.
• Concurrent lanes: a cluster benchmark's SIPp roles listen 1000 ports up (6080/6070) with a k8s- Call-ID prefix, so the samplers of a server soak and a cluster soak on one load generator tell them apart.
• Where kube work runs: helm/kubectl run in alpine/k8s:1.33.4 (digest-pinned) on the load generator, not on the CLI machine, so the workflow needs no new tools. The helm setup script runs in that same alpine/k8s container on the load generator (not on the machine that runs perfkit run, whatever Routr's perf/k8s/seed.sh header says), as bash <setup> <namespace> <register users> <uac users> with KUBECONFIG and PERFKIT_* set.
• Release name perfkit, namespace = benchmark id. entry.service matches the Service name with or without perfkit- in front (Routr's routr-edgeport-tcp finds perfkit-routr-edgeport-tcp).
• DO SIPp address: the LB is public, so cluster-lane SIPp binds the load generator's public IP; the callees register with it.
• Local entry: MetalLB LoadBalancer (not NodePort), per the maintainer's kind test. Pool is .200-.250 of the kind network's second-to-last /24 (172.18.254.200-250 on 172.18.0.0/16), SIPp gets fixed .251-.253 there, to stay clear of another cluster's MetalLB in the last /24.
• metrics-server chart 3.12.2 with --kubelet-insecure-tls on both DOKS and kind (ephemeral clusters; it also works with valid kubelet certs).
• Helm 4: --force-conflicts is added only when helm version reports v4 (the pinned image ships Helm 3.18).
• fromCapacity: uses the limit of a passed capacity benchmark; for an inconclusive one (the load generator gave out) whose every step before the invalid one passed both rules, the highest such step, as a lower bound (rateSource.lowerBound: true, and the progress line says "the lower-bound limit ... at least N calls/s"). fallbackCps when it failed, was aborted, was inconclusive with no passing step, did not run, or floor(fraction × limit) < 1. rateSource.limit is null when there was no limit.
• Concurrent soaks: lanes run with Promise.allSettled; a failing cluster soak doesn't cut the server soak short (it keeps its own results), then the run fails with the first failure.
• Sweep without the kubernetes scope: a 403 from listing clusters counts as "no clusters", so sweep, collect's teardown and run.yaml's sweep --run keep working with a token scoped for droplets only.
• TCP on the load generator: cloud-init sets net.ipv4.ip_local_port_range = 1024 65535 and net.ipv4.tcp_tw_reuse = 1, so -t tn to one LB address doesn't run out of ephemeral ports (~470 calls/s with the defaults and 60 s TIME_WAIT).
• Values placeholders: PERFKIT_ENTRY_IP, PERFKIT_TARGET_VERSION, PERFKIT_IMAGE_<NAME>; any other ${PERFKIT_…} fails the benchmark naming the placeholder.
• Chart dependencies must be in the chart's charts/ (no helm dependency build at run time); perfkit validate checks <chart>/Chart.yaml.
• benchmarks[].target is written for every benchmark, server ones included, so the page always labels; the measurements of existing benchmarks don't change.
• Union errors in perfkit.yaml now point at the closest shape (benchmarks[0].uut.helm.pods: required) instead of "does not match any allowed shape" (ties keep the old message).
• The old test "should reject register and uas in a capacity benchmark" is replaced by "should accept…" / "should reject a capacity register without users" (A capacity benchmark can't register callee agents or ramp calls to them #24).

Verify

• npm test (735 passing), npm run typecheck, npm run lint, npm run schemas with no diff.
• perfkit validate on Routr's perf/perfkit.yaml, perf/perfkit.local.yaml, perfkit.smoke.yaml, perfkit.calibration.yaml (routr-v3 working copy): valid.
• Teardown of a cluster tested on success, failure (body throws; cluster in error state; delete that fails keeps the VPC) and SIGINT; sweep of clusters by age, by --run, --dry-run.
• Real perfkit run --local --benchmarks k8s-simple of Routr's local config on this Mac: kind (2 nodes) + MetalLB came up, 9 local images loaded, entry at 172.18.254.200:5060, 200 callees registered over TCP, one step measured, report written with infra.cluster and the pods' resources, kind cluster and containers deleted. No DigitalOcean resource was created.

Not done / follow-ups

• No DOKS run (not allowed here). The DOKS path is unit-tested against a fake API only.
• The full six-benchmark local run was not done; only k8s-simple. Its result was failed: 61.8% of calls at 20 calls/s got 503 Service unavailable on the BYE (then 481), and perfkit-routr-edgeport-0 grew to ~2.8 GB working set within a minute. That looks like Routr's multi-EdgePort TCP behavior (2 replicas behind the LB), not perfkit, but needs a look.
• Release notes for the next tag: perfkit.yaml gains transport: tcp, uut.helm, infra.cluster, rate.fromCapacity, capacity register/uas; capacity uacs get -recv_timeout 10000 and -l max × (hold + 20); every SIPp -cid_str carries the run id; report.json gains optional run.infra.cluster, benchmarks[].target, soak load.rateSource (with lowerBound); droplets get the full ephemeral port range and TIME_WAIT reuse; the DigitalOcean token needs the Kubernetes scopes for uut.helm; no new workflow inputs.

[psanders]

Review: Changes needed (round 1 of 2)

Committee: QA engineer, software architect, release engineer, security engineer. Left out: performance engineer, realtime-comms engineer (src/orchestrate, src/analyze), UX and visual designers (src/render), technical writer.
Verify: npm test, npm run typecheck, npm run lint and npm run schemas (no drift) all pass. #24's Verify (npx mocha --no-config ... perfkitConfig/buildSippRoles/agent/measureCapacityStep) gives 66 passing. perfkit validate passes on Routr's perf/perfkit.yaml and perf/perfkit.local.yaml. The full six-benchmark perfkit run --local was not run by the reviewer: one is already running on this machine. No DOKS run either: that would create DigitalOcean resources.
Tests: npm test gives 728 passing. CI's "Typecheck, lint and unit tests" and "Lint the workflows" pass.

Blocking:

1. Sweep and collect's teardown fail for a token without the Kubernetes scope, so droplets keep billing.
   • Evidence: the sweep always calls listKubernetesClusters() inside its Promise.all (src/infra/createSweep.ts:71). README.md:47 says the Kubernetes scopes are needed only "for uut.helm".
   • What breaks: a non-helm project whose custom-scoped token follows the README gets a 403 there. The whole sweep then throws before it deletes anything. That covers perfkit sweep, collect's teardown (src/cli/runCollectCommand.ts:151-160) and run.yaml's final sweep --run job.
   • Fix (one of):
     • Treat a 403 from listKubernetesClusters as no clusters. Without the scope no cluster can exist. Add a unit test.
     • Or move kubernetes:read into the always-required rows of the README scope table.
   • Expert: chair, kept by the pruner.
2. The TCP ramp on the cluster will likely hit the load generator's ephemeral-port limit, not Routr's.
   • Evidence: every TCP role runs -t tn, one connection per call (src/orchestrate/buildSippRoles.ts:104). All connections go from one load generator to one LB ip:5060, and SIPp closes each socket at call end, so each port then sits in TIME_WAIT for 60 s. The load generator's cloud-init (src/infra/buildCloudInit.ts:16-23) doesn't raise net.ipv4.ip_local_port_range, and doesn't turn on tcp_tw_reuse for non-loopback.
   • What breaks: on Ubuntu's default ~28k ports, new connections fail past ~470 calls/s. Routr's k8s-simple ramps 50→3000 calls/s, so its published limit, and the k8s-soak rate taken from it, would measure the load generator.
   • Fix: add net.ipv4.ip_local_port_range = 1024 65535 and net.ipv4.tcp_tw_reuse = 1 to /etc/sysctl.d/90-perfkit.conf in buildCloudInit.ts, and cover them in its unit test.
   • Expert: software architect, kept by the pruner.

[psanders]

Review: Ready for human review (round 2 of 2)

Committee: the chair re-checked round 1's two blocking findings and the new commits (96b65b7, 5ef5e61, 209a933) against the code. Round 1's committee: QA engineer, software architect, release engineer, security engineer.
Verify: at 209a933, npm test, npm run typecheck, npm run lint and npm run schemas all pass, with no schema drift. The full local run and a DOKS run were not done by the reviewer (as in round 1).
Tests: npm test gives 735 passing. CI's "Typecheck, lint and unit tests" and "Lint the workflows" pass.

Round 1's findings:

1. Sweep without the Kubernetes scope: fixed.
   • A DigitalOceanError with status 403 from listKubernetesClusters now counts as no clusters (src/infra/createSweep.ts:72-80). Any other error still fails the sweep.
   • A test covers both cases: "should sweep droplets as before when the token can't list clusters (403), and fail on other errors".
2. TCP ephemeral ports: fixed. buildCloudInit.ts now writes net.ipv4.ip_local_port_range = 1024 65535 and net.ipv4.tcp_tw_reuse = 1, and its test covers both lines.

Also checked:

• Concurrent soaks: soak lanes now run under Promise.allSettled, and the run fails with the first failure only after every lane ends (createAgent.ts:152-161). A test covers it.
• Lower bound for rate.fromCapacity (capacityLimitOf.ts):
  • An inconclusive capacity result gives a lower bound only when every step before the invalid one passed both rules. It uses the same strict < as the analyzer, and createAnalyzeCapacity returns inconclusive only in that case.
  • rateSource.lowerBound is optional in report v1, so it is additive.
  • It changes perfkit runs a project's benchmarks over TCP on one server and on Kubernetes in one run #25's "inconclusive → fallbackCps". The PR description states the new behavior.

Blocking: none.