feat: live results while a run is in progress

.github/workflows/run.yaml

@@ -3,7 +3,7 @@
 #
 #   uses: fonoster/perfkit/.github/workflows/run.yaml@v0.1.0
 #   with: { perfkit-ref: v0.1.0, target-version: 2.31.5, commit: ${{ github.sha }} }
-#   secrets: { DIGITALOCEAN_TOKEN: ..., PERFKIT_SSH_KEY: ... }
+#   secrets: { DIGITALOCEAN_TOKEN: ..., PERFKIT_SSH_KEY: ..., NGROK_AUTHTOKEN: ... (optional) }
 #
 # Secrets come only through `secrets:` below, never `secrets: inherit`.
 name: perfkit run
@@ -36,6 +36,9 @@ on:
         required: true
       PERFKIT_SSH_KEY:
         required: true
+      # Optional: opens the agent's ngrok tunnel to live.json (live.domain in perfkit.yaml).
+      NGROK_AUTHTOKEN:
+        required: false
     outputs:
       run-id:
         description: The run id.
@@ -77,6 +80,8 @@ jobs:
         env:
           DIGITALOCEAN_TOKEN: ${{ secrets.DIGITALOCEAN_TOKEN }}
           PERFKIT_SSH_KEY: ${{ secrets.PERFKIT_SSH_KEY }}
+          # Goes into the agent container's environment on the load generator only.
+          NGROK_AUTHTOKEN: ${{ secrets.NGROK_AUTHTOKEN }}
           CONFIG: ${{ inputs.config }}
           TARGET_VERSION: ${{ inputs.target-version }}
           COMMIT: ${{ inputs.commit }}

README.md

@@ -116,7 +116,7 @@ perfkit version.
 
 ```sh
 perfkit run --config <perfkit.yaml> --target-version <tag> [--commit <sha>] [--benchmarks <id,id>]
-            [--out <dir>] [--detach] [--dry-run] [--local]
+            [--out <dir>] [--detach] [--dry-run] [--local] [--live-port <port>]
 ```
 
 Creates the run's infrastructure on DigitalOcean, runs the benchmarks on it, and deletes it at
@@ -146,11 +146,14 @@ digest its tag had when the run started.
 | `--detach` | off | Return once the agent on the load generator has started; `perfkit collect` finishes the run. |
 | `--dry-run` | off | Print the plan and create nothing. Needs no token. |
 | `--local` | off | Run on this machine's Docker instead of DigitalOcean. Needs no token. |
+| `--live-port <port>` | `live.port` of perfkit.yaml, else `8787` | Where live.json is served during the run (see [Live results](#live-results)). |
 
 | Environment | Meaning |
 | :--- | :--- |
 | `DIGITALOCEAN_TOKEN` | Required, except with `--dry-run`. Never printed, never written to a file, `report.json` or a droplet. |
 | `PERFKIT_NAME_PREFIX` | Optional; containers and compose projects are named `<prefix>-<run id>-<what>`, `perfkit` by default. |
+| `NGROK_AUTHTOKEN` | Optional. Opens an ngrok tunnel to the run's live.json (see [Live results](#live-results)). On a DigitalOcean run it goes into the agent container's environment on the load generator only; never into run.json, a log or the report. |
+| `NGROK_DOMAIN` | Optional static ngrok domain when perfkit.yaml has no `live.domain`. |
 | `PERFKIT_SSH_KEY` | Optional ed25519 private key, in OpenSSH format, without a passphrase. When set, the run looks its public key up in the account by fingerprint and reuses it, or registers it as `perfkit-shared`. That key is never deleted, by teardown or by sweep, so parallel runs share it. Without it, the run creates `perfkit-<run id>` for this run only and deletes it at teardown. |
 
 `--dry-run` resolves the digests on Docker Hub and prints the plan:
@@ -440,6 +443,58 @@ The verdict is **failed** when any rule failed and **passed** otherwise. Also re
 part of the verdict: p99 in the first and the last hour after the warm-up, the most open file
 descriptors per process, the most concurrent calls, the registrations and their failures, and,
 with telemetry, each service's p50 and p99 self time (a span's time minus its direct children's).
+## Live results
+
+While a run goes on, the benchmarks page shows it live. Every run, published config or not:
+
+- **live.json.** The agent writes `<run root>/live.json` (`schema: perfkit/live/v1`, see
+  [docs/report-v1.md](docs/report-v1.md#livejson)) every 15 seconds: which benchmark runs, each
+  capacity step as the agent judged it, what held so far, a soak's calls and raw memory per
+  process, and when the run should end. Nothing in it is a verdict.
+- **Served read-only.** `GET /live.json` on `127.0.0.1:<live.port>` (8787 by default), never
+  cached, with CORS for the origin of `project.siteUrl` and for `http://localhost:*`; 503 before
+  the first snapshot, 404 for anything else. On a DigitalOcean run the agent serves it on the load
+  generator; with `--local` the `run` (or `collect`) process serves it on this machine.
+- **ngrok.** With `NGROK_AUTHTOKEN` set, the server is tunnelled through the static domain in
+  `live.domain` (or `NGROK_DOMAIN`) with the `@ngrok/ngrok` package, and the public URL is logged.
+  Without it there is no tunnel. The tunnel and the server close when the run ends.
+- **The page.** `project.live.url` (default `https://<live.domain>/live.json`) goes into
+  report.json, so the latest page (the site root) knows where to look. It polls every 15
+  seconds; while a run is in progress it shows it live, marked preliminary. When the run ends it
+  says the report is being published and shows it as soon as `data/runs.json` lists the run.
+  With no run in progress the page is the latest report, unchanged. Run pages never go live.
+  `?live=<url>` on the page overrides the URL.
+
+### Watch a local run live
+
+From a perfkit checkout (`npm ci && npm run build`); the minimal example takes about 4 minutes:
+
+```sh
+# 1. A site to watch: render any report into a directory (an example works).
+node dist/cli.js render --report examples/reports/passed.json \
+  --history examples/history/runs-14.json --out /tmp/perfkit-site
+# 2. Serve it (leave it running).
+node dist/cli.js serve --site /tmp/perfkit-site --port 8080
+# 3. In another terminal, a short local run; it serves live.json on http://127.0.0.1:8787.
+node dist/cli.js run --local --config examples/minimal/perfkit.yaml --target-version 3.7.3
+# 4. Open the page with the local live URL.
+open "http://localhost:8080/?live=http://localhost:8787/live.json"
+# 5. When the run ends, publish its report; the page switches to it.
+node dist/cli.js render --report .perfkit/<run id>/report.json \
+  --history /tmp/perfkit-site/data/runs.json --out /tmp/perfkit-site
+```
+
+## perfkit serve
+
+```sh
+perfkit serve --site <dir> [--port 8080] [--host 127.0.0.1]
+```
+
+A read-only static server for a directory `perfkit render` wrote, as GitHub Pages serves it,
+with nothing cached: to preview a site, or to watch a `--local` run live (above). It prints
+`serving <dir> at http://127.0.0.1:8080/ (Ctrl-C stops)` and runs until Ctrl-C. It exits 0 when
+stopped, 1 for a usage error and 2 when it can't listen (the port is taken).
+
 ## perfkit render
 
 ```sh

docs/github-workflow.md

@@ -44,6 +44,7 @@ adds that machine's IP to the run's firewall before it logs in, with `PERFKIT_SS
 | :--- | :--- |
 | `DIGITALOCEAN_TOKEN` | The DigitalOcean token, with the scopes in the README. It never reaches a droplet, a log or the report. |
 | `PERFKIT_SSH_KEY` | An ed25519 private key in OpenSSH format, without a passphrase. Every job logs in to the droplets with it, registered in the account as `perfkit-shared`. Make one with `ssh-keygen -t ed25519 -N "" -f perfkit`. |
+| `NGROK_AUTHTOKEN` | Optional. The agent opens an ngrok tunnel from `live.domain` of perfkit.yaml to its live.json server, so the benchmarks page shows the run live (see "Live results" in the README). It goes into the agent container's environment on the load generator and nowhere else: not run.json, a log or the report. Without it the run has no tunnel. |
 
 The workflow takes its secrets only through `secrets:`, so a caller has to name them: `secrets:
 inherit` would hand it every secret of the repository.
@@ -85,6 +86,7 @@ jobs:
     secrets:
       DIGITALOCEAN_TOKEN: ${{ secrets.DIGITALOCEAN_TOKEN }}
       PERFKIT_SSH_KEY: ${{ secrets.PERFKIT_SSH_KEY }}
+      NGROK_AUTHTOKEN: ${{ secrets.NGROK_AUTHTOKEN }} # optional: live results
 
   publish:
     needs: run

docs/perfkit-yaml.md

@@ -19,6 +19,7 @@ Rules for the whole file:
 | `apiVersion` | `perfkit/v1` | yes | | Tells perfkit which format this is. |
 | `project` | object | yes | | See [project](#project). |
 | `infra` | object | yes | | See [infra](#infra). |
+| `live` | object | no | none | Where the agent serves live.json during a run. See [live](#live). |
 | `images` | map of name to image | yes | | At least one. See [images](#images). |
 | `benchmarks` | list | yes | | At least one. See [benchmarks](#benchmarks). |
 
@@ -33,6 +34,7 @@ Rules for the whole file:
 | `project.logo` | path | no | none | An `.svg` file. Without it, the nav shows `project.name`. |
 | `project.title` | string | yes | | The page's headline. |
 | `project.lead` | string | yes | | The paragraph under the headline. |
+| `project.live.url` | URL | no | `live.url`, else `https://<live.domain>/live.json` | Where the benchmarks page polls a run in progress. Goes into report.json, so the published page knows it. |
 
 ## infra
 
@@ -44,6 +46,24 @@ Rules for the whole file:
 | `infra.loadgen.size` | string | yes | | Droplet size slug of the load generator, like `c-16`. |
 | `infra.maxLifetime` | duration | no | `9h` | The sweep deletes this run's droplets after this long. |
 
+## live
+
+Every run writes `live.json` (`schema: perfkit/live/v1`, see [report-v1.md](report-v1.md#livejson))
+every 15 seconds and serves it read-only on the load generator; with `NGROK_AUTHTOKEN` set, an
+ngrok tunnel makes it public. The page polls it; see "Live results" in the README.
+
+| Field | Type | Required | Default | Notes |
+| :--- | :--- | :--- | :--- | :--- |
+| `live.port` | number | no | `8787` | The port of the live.json server. `perfkit run --live-port` overrides it. |
+| `live.domain` | hostname | no | `NGROK_DOMAIN`, else none | The static ngrok domain of the tunnel, like `routr-benchmarks.ngrok-free.dev`. |
+| `live.url` | URL | no | `https://<live.domain>/live.json` | Where the page polls it; `project.live.url` wins. |
+
+```yaml
+live:
+  domain: routr-benchmarks.ngrok-free.dev
+  url: https://routr-benchmarks.ngrok-free.dev/live.json
+```
+
 ## images
 
 A map from a short name (the key) to an image. At least one image is required. Each entry is

docs/report-page.md

@@ -91,3 +91,28 @@ N runs" shows the rest.
 
 The droplets and region, each image with its digest, the tools, and the command to run the same
 benchmarks yourself.
+
+## The live view
+
+While a run is in progress, the latest page (the site root, never a run page) shows it live from
+the agent's live.json, polled every 15 seconds; see "Live results" in the README. The design is
+the "State / Live (preliminary)" frame of `designs/benchmarks.pen`.
+
+- **Nav and banner.** The picker reads "Running now · v<version>" with a live dot and badge.
+  Under the nav: "Running now. These numbers are preliminary and change every 15 seconds; the
+  verdict appears when the run ends." and "updated N s ago", which turns warning-coloured after a
+  minute and reads "connection lost, retrying" after 3 minutes or a failed poll.
+- **Header and progress.** "LIVE · RUN <id>", "<Project> <version> is under test right now.",
+  one card per step (setup, then each benchmark: done, running or queued with its expected start)
+  and a bar with when the run should end.
+- **Live numbers.** For the benchmark running now: the rate offered (and placed, last step),
+  p99 and failed % of the last step (a soak: calls placed and failed % so far), and the time into
+  the run with the time to go.
+- **Capacity.** The same "Latency as load rises" chart as the report, drawn up to the step
+  offered now, marked "now <rate>", the steps not run yet shaded; and "So far": what held, why the
+  limit is not known yet, and the last published limit from `data/runs.json`.
+- **Soak.** Raw memory per process with the warm-up shaded and a "now" line. Growth in MB/h shows
+  only once there is a full hour of data after the warm-up.
+
+When the run ends the page says the report is being published, and draws it once
+`data/runs.json` lists the run. Animations stop under `prefers-reduced-motion`.

docs/report-v1.md

@@ -41,6 +41,7 @@ Rules for both files:
 | :--- | :--- | :--- |
 | `project.name`, `repo`, `siteUrl`, `accent`, `title`, `lead` | string | Copied from perfkit.yaml. |
 | `project.logoSvg` | string | The SVG markup of `project.logo`, inlined. Present only when perfkit.yaml has `logo`. |
+| `project.live.url` | URL | Where the latest page polls live.json: `project.live.url`, else `live.url`, else `https://<live.domain>/live.json` of perfkit.yaml. Absent without them. |
 
 ### run
 
@@ -149,3 +150,34 @@ A benchmark's `headline` in runs.json is:
 - `passed` or `failed`, capacity: `{ limitCps, p99AtLimitMs }`, both `null` when no step passed;
 - `passed` or `failed`, soak: `{ memoryGrowthMBPerHour }`;
 - `not-run`, `inconclusive` and `aborted`: `null`.
+
+## live.json
+
+`schema: perfkit/live/v1`, JSON Schema in [schemas/live-v1.schema.json](../schemas/live-v1.schema.json).
+The agent rewrites it every 15 seconds for the whole run (a temporary file, then a rename) and
+serves it; nothing is kept after the run. Every number in it is preliminary: verdicts come only
+from report.json.
+
+| Field | Type | Notes |
+| :--- | :--- | :--- |
+| `runId`, `version`, `commit`, `startedAt` | | As in report.json. |
+| `updatedAt` | timestamp | When this snapshot was written. |
+| `status` | `running`, `finished` or `failed` | `failed` when the agent stopped the run; `reason` says why. |
+| `estimatedEndAt` | timestamp or null | While running: the planned time left of the running benchmark and every queued one, plus a fixed allowance per benchmark. |
+| `provider` | `digitalocean` or `local` | |
+| `project` | object | As report.json's `project`. |
+| `benchmarks` | list | Every benchmark of perfkit.yaml, in order. |
+
+Every benchmark has `id`, `title`, `unit`, `kind`, `state` (`queued`, `running`, `done` or
+`not-run`, the last for one not selected), `startedAt` (compose up), `loadStartedAt` (SIPp's
+start), `estimatedStartAt` (queued only) and `plannedS`. Then by kind:
+
+- **capacity:** `rate` (`start`, `step`, `everyS`, `max`), `thresholds`, `current` (`index`,
+  `total`, `offeredCps` of the step offered now, from the clock), `points` (each step as the
+  agent judged it when it ended: `offeredCps`, `achievedCps`, `calls`, `failed`, `p50Ms`,
+  `p95Ms`, `p99Ms`, `failedPct`, `ok`, `stopReason`), `heldCps` (the last step that passed) and,
+  once done, `result` (`limitCps`, `lowerBound`, `stopReason`).
+- **soak:** `cps`, `warmupS`, `durationS`, `elapsedS`, `thresholds`, `calls` (`placed`, `failed`
+  so far, SIPp's counters since the load started) and `memory`: per process, its raw RSS
+  (`samples` of `t`, `rssMB`, at most 300, evenly spaced) and `growthMBPerHour`, the verdict's
+  least-squares slope, `null` until there is a full hour of samples after the warm-up.