Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
ab73b5b
Plan bench's upgrade onto the site's toolchain
lwwmanning Oct 28, 2026
45e667a
Plan bench PR 1: the toolchain move
lwwmanning Oct 28, 2026
13609d9
Add a local parity check for bench builds
lwwmanning Oct 28, 2026
80602cd
Set the bench parity bar to no noticeable change
lwwmanning Oct 28, 2026
e53da05
Make the bench parity check fail on any non-200 and stale caches
lwwmanning Oct 28, 2026
b8562a0
Submit bun.lock to the dependency graph
lwwmanning Oct 28, 2026
5827ac4
Retry the snapshot submission once, group main pushes by commit
lwwmanning Oct 28, 2026
88afe40
Make bench a member of the site's Bun workspace
lwwmanning Oct 28, 2026
cb722e6
Upgrade bench to Next.js 16 and React 19.3
lwwmanning Oct 28, 2026
b9d9698
Format bench/scripts to bench's 100-column style
lwwmanning Oct 28, 2026
d92b74f
Lint and format bench with Biome
lwwmanning Oct 28, 2026
58e2900
Run Biome over all of bench in bench CI
lwwmanning Oct 28, 2026
6977969
Type-check bench with TypeScript 7 and a shared base config
lwwmanning Oct 28, 2026
091a757
Apply the site's a11y rules to bench, with inline exceptions
lwwmanning Oct 28, 2026
c9cca56
Compare bench's pages, error responses and dark theme in the parity c…
lwwmanning Oct 28, 2026
a810be6
Correct the linker and JSX comments for the shared toolchain
lwwmanning Oct 28, 2026
83d27f9
Cache Bun's packages and test the parity check in bench CI
lwwmanning Oct 28, 2026
bfd712c
Fail dependency review when the bun.lock snapshot is missing
lwwmanning Oct 28, 2026
f92f33a
Correct two bench notes in CLAUDE.md
lwwmanning Oct 28, 2026
ecfd462
Plan bench PR 2: clean up bench's code
lwwmanning Oct 29, 2026
dd82f68
Let bench's integration tests use a local Postgres
lwwmanning Oct 29, 2026
1d4e41b
Remove bench's unused IAM sign-in path
lwwmanning Oct 29, 2026
18b7a8d
Fetch a bench group's charts with two queries, not two per chart
lwwmanning Oct 29, 2026
f68093d
Build a bench group's payloads from one snapshot
lwwmanning Oct 29, 2026
c55792a
Split bench's chart store into modules
lwwmanning Oct 29, 2026
b4be6c6
Split bench's chart island into modules
lwwmanning Oct 29, 2026
e7858cc
Remove bench helpers that only tests used
lwwmanning Oct 29, 2026
a0198f8
Type-check bench with the site's stricter flags
lwwmanning Oct 29, 2026
31e5d0f
Point bench docs at the split modules
lwwmanning Oct 29, 2026
a2f133e
Keep null visible in two bench index reads
lwwmanning Oct 29, 2026
541fd77
Discard a bench client whose connection drops
lwwmanning Oct 29, 2026
84f8587
Correct bench's pool-size test and two stale notes
lwwmanning Oct 29, 2026
b4caf7c
Mark bench's fake Postgres credential for the secret scan
lwwmanning Oct 29, 2026
7f08ade
Merge main into claude/bench-cleanup
lwwmanning Oct 29, 2026
4365dee
Remove bench's working plans from the repository
lwwmanning Oct 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/actions/bench-checks/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,9 +70,9 @@ runs:
shell: bash
run: bun run --cwd bench/web build

# testcontainers self-skips the Postgres integration suite when no Docker daemon
# is available (a local-dev convenience); in CI that would silently drop the only
# Postgres integration coverage, so fail loudly if the daemon is missing.
# The Postgres integration suite runs on Docker here. Locally it skips when there
# is neither Docker nor BENCH_TEST_DATABASE_URL; with CI=true the test harness
# fails the suite instead. This step fails sooner, with a plainer cause.
- name: Require Docker for the integration suite
shell: bash
run: docker info > /dev/null
Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
- `bun lint` - Run Biome lint
- `bun start` - Start production server

bench is a member of the root Bun workspace. `bun install` at the root installs both apps; run bench's scripts with `bun run --cwd bench/web <script>`: `dev`, `build`, `lint`, `format`, `format:check`, `test`. `lint` and `format` are Biome, as at the root. The Postgres integration suite needs Docker and skips without it.
bench is a member of the root Bun workspace. `bun install` at the root installs both apps; run bench's scripts with `bun run --cwd bench/web <script>`: `dev`, `build`, `lint`, `format`, `format:check`, `test`. `lint` and `format` are Biome, as at the root. The Postgres integration suite needs Docker or a local Postgres named by `BENCH_TEST_DATABASE_URL` (`bench/web/README.md` has the commands), and skips without either, except when `CI=true`, where it fails.

## Repository layout

Expand All @@ -20,7 +20,7 @@ The repository holds two independent apps:

They share one Bun workspace and the CI policy, and deploy separately:

- **Toolchain.** `bench/web` is a member of the root Bun workspace: one `bun.lock`, one `bunfig.toml`, one `bun install` at the root. Bun's isolated linker gives each package its own dependencies; both apps run Next.js 16.4.1, React 19.3 and TypeScript 7. The root `biome.json` lints and formats both apps: a `bench/**` override keeps bench's old Prettier style (single quotes, trailing commas, 100 columns) and sets bench's lint rules (see `bench/AGENTS.md`), and `bun run check:ci` at the root checks all of it. Both `tsconfig.json` files extend the root `tsconfig.base.json`, which holds the compiler options the apps share, and each keeps its own target, paths and extra flags (the site's stricter `noUncheckedIndexedAccess` and `exactOptionalPropertyTypes`, bench's `noFallthroughCasesInSwitch` and `noUncheckedSideEffectImports`). The root `tsconfig.json` excludes `bench/`, and `bench/web/postcss.config.mjs` keeps Next.js from picking up the root's Tailwind config.
- **Toolchain.** `bench/web` is a member of the root Bun workspace: one `bun.lock`, one `bunfig.toml`, one `bun install` at the root. Bun's isolated linker gives each package its own dependencies; both apps run Next.js 16.4.1, React 19.3 and TypeScript 7. The root `biome.json` lints and formats both apps: a `bench/**` override keeps bench's old Prettier style (single quotes, trailing commas, 100 columns) and sets bench's lint rules (see `bench/AGENTS.md`), and `bun run check:ci` at the root checks all of it. Both `tsconfig.json` files extend the root `tsconfig.base.json`, which holds the compiler options the apps share, including `noUncheckedIndexedAccess` and `exactOptionalPropertyTypes`. Each app keeps its own target, paths and extra flags (bench's `noFallthroughCasesInSwitch` and `noUncheckedSideEffectImports`). The root `tsconfig.json` excludes `bench/`, and `bench/web/postcss.config.mjs` keeps Next.js from picking up the root's Tailwind config.
- **CI.** `.github/actions/changed-paths` reports which app a change touches. `ci.yml` skips its site jobs on bench-only changes, and `bench-ci.yml` skips on site-only changes. The workspace's root files (`package.json`, `bun.lock`, `bunfig.toml`, `biome.json`, `tsconfig*.json`) count as both. Both gate with `if:`, not `on.paths`, so a skipped required check still passes. Dependency review, the secret scan and REUSE run on every change.
- **Deploys.** The `vortex` Vercel project builds the site through the git integration, and `scripts/vercel-ignore-build.sh` skips bench-only commits. bench deploys to the `benchmarks-website` Vercel project (Root Directory `bench/web`) from `bench-deploy.yml`, with the Vercel CLI, on changes to `bench/` and to the root `package.json`, `bun.lock`, `bunfig.toml` and `tsconfig.base.json`. Schema migrations go through `bench-schema-deploy.yml`, which only runs manually from `main` because the AWS role trusts nothing else.

Expand Down
9 changes: 5 additions & 4 deletions bench/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,9 @@ rules below are the parts that were worth keeping.
LTTB on the visible commit range only.
- **Don't refetch on every scope change.** Once a chart's payload is in memory,
pan/zoom/slider/range-strip all rebuild in place via the in-memory LTTB pass on the cached
payload ([`web/components/Chart.tsx`](web/components/Chart.tsx)). The single exception is the
latest-100 to full-history zoom-out path: when the user first zooms past the materialized
window, the chart lazy-fetches `?n=all` once and replaces the payload in place.
payload ([`web/components/chart/controller.ts`](web/components/chart/controller.ts)). The single
exception is the latest-100 to full-history zoom-out path: when the user first zooms past the
materialized window, the chart lazy-fetches `?n=all` once and replaces the payload in place.
- **Group docs links mirror the monorepo's `Benchmark::doc_path`.**
[`web/lib/benchmark-docs.ts`](web/lib/benchmark-docs.ts) maps each SQL dataset name to the
benchmark's explainer doc in `vortex-data/vortex`, which is the same path that the monorepo's PR
Expand Down Expand Up @@ -71,7 +71,8 @@ rules below are the parts that were worth keeping.

## Footguns we have already hit

All in [`web/components/Chart.tsx`](web/components/Chart.tsx):
All in the chart island, [`web/components/Chart.tsx`](web/components/Chart.tsx) and the modules in
[`web/components/chart/`](web/components/chart/):

- **Reverse predecessor walk in the tooltip.** `payload.commits[]` is
sorted oldest-first by SQL - `commits[0]` is the oldest, `commits[N-1]`
Expand Down
6 changes: 3 additions & 3 deletions bench/docs/architecture/design-decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ v3 auto-queued a background `?n=all` fetch for every chart on group open; on a
the windowed fetches a user actually waits on. v4 removed the auto-warmup: full
history loads only on a deliberate per-chart signal — a window-chip click, a
~1000ms hover dwell, or a pan/zoom into the unloaded region.
→ `web/components/Chart.tsx`, `web/lib/chart-format.ts`.
→ `web/components/chart/controller.ts`, `web/lib/chart-format.ts`.

**Bounded windows render on the full-length virtual x-axis.**
Loading a small window and then "more" normally re-bases the Chart.js x-axis
Expand All @@ -181,7 +181,7 @@ Each fetch carries an `AbortController` and a 30s timeout; closing/reopening a
group cancels its in-flight fetches rather than piling load on the server, and a
stall aborts instead of spinning forever. 30s is headroom over the ~7.8s cold
first-hit, so a slow-but-live request is not falsely killed.
→ `web/components/Chart.tsx`, `web/lib/chart-store.ts`.
→ `web/components/chart/controller.ts`, `web/lib/chart-store/bundles.ts`.

**Expand All loads one bundle per group, not N per-chart requests.**
The viewport-gated per-chart path loads only what is on screen — the opposite of
Expand All @@ -190,7 +190,7 @@ bundle into a session-lifetime payload cache, priming every chart with a single
request; the `IntersectionObserver` still gates the CPU-bound Chart.js
construction. The cache is per-session by design (a close/reopen costs zero
fetches).
→ `web/lib/chart-store.ts` (`ensureGroupBundle`).
→ `web/lib/chart-store/bundles.ts` (`ensureGroupBundle`).

**A warmer cron keeps the function and its DB connections hot.**
Caches handle repeat reads but not the first read after idle. A Vercel-native
Expand Down
36 changes: 22 additions & 14 deletions bench/docs/architecture/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,17 +106,24 @@ fast.
scaling that is not full coverage — but a low-traffic site effectively runs one
instance, so the ping warms the path the typical first visitor actually takes.

### Parallel group fan-out

A group page builds one chart payload per chart (99 for TPC-DS, 43 for
Clickbench). The fan-out in `collectGroupCharts`
([`web/lib/queries.ts`](../../web/lib/queries.ts)) runs those queries with an
order-preserving `Promise.all` bounded by the existing `pg` pool (max 8
connections) rather than a sequential `await` loop, so a Data Cache *miss* on a
big group does not serialize into dozens of round-trips. The output (chart set,
order, shape) is unchanged and pinned by an integration test. This was a small,
low-risk change; batching a whole group into one SQL statement is the larger
win, deliberately left for later because the chart shapes are heterogeneous.
### One query per group

A group page inlines every chart's payload (99 for TPC-DS, 43 for Clickbench).
`groupChartPayloads` ([`web/lib/queries.ts`](../../web/lib/queries.ts)) builds
them from two statements instead of two per chart: the last `n` commits, and
every chart's rows in that window, one `LATERAL` join per chart (in slices of
25 charts, so a TPC-DS group runs five). They run one after another on one
connection, in one `REPEATABLE READ` snapshot. A chart's x-axis is the last
`min(n, total)` commits of the shared window, so the JSON is byte-identical to
`chartPayload`'s, which a test and a dump check pin. On one connection the
per-chart `MIN` over a query chart's whole history became the cost, so it is a
skip scan of `idx_query_measurements_summary`, and the commit count is a
binary search. `QUERY_CONCURRENCY` now bounds only the summaries in
`collectGroups`. On the 2026-10-28 dump at `n=100`, warm: TPC-DS (NVMe) SF=1
1.57 s → 0.31 s, Clickbench 0.82 s → 0.14 s, Compression 68 → 117 ms (its
`MIN` still reads every row). At `?n=all` the TPC-DS group takes 5.2 s against
4.7–4.9 s, and its peak memory stays under 1.5× the old fan-out's (1.2–1.8 GB
against 1.3 GB over four runs each).

### `query_measurements` reads filter on a denormalized timestamp

Expand Down Expand Up @@ -162,8 +169,9 @@ becomes the bottleneck.
The server makes a single payload cheap; the client decides *when and how many*
to ask for. The pure helpers below live in
[`web/lib/chart-format.ts`](../../web/lib/chart-format.ts); the fetch queues are
in [`web/lib/chart-store.ts`](../../web/lib/chart-store.ts) and the Chart.js
wiring in [`web/components/Chart.tsx`](../../web/components/Chart.tsx).
in [`web/lib/chart-store/queues.ts`](../../web/lib/chart-store/queues.ts), the
chart island in [`web/components/Chart.tsx`](../../web/components/Chart.tsx) and
the Chart.js wiring in [`web/components/chart/`](../../web/components/chart/).

### Lazy hydration, top first

Expand Down Expand Up @@ -222,7 +230,7 @@ chip's retry re-issues the *fetch* (an earlier version only retried chart
"Expand All" should load everything quickly, but the per-chart, viewport-gated
path does the opposite — it loads only what is on screen. So a toggle-open kicks a
single `GET /api/group/{slug}?n=100` **bundle** fetch
(`ensureGroupBundle`, [`web/lib/chart-store.ts`](../../web/lib/chart-store.ts))
(`ensureGroupBundle`, [`web/lib/chart-store/bundles.ts`](../../web/lib/chart-store/bundles.ts))
into a session-lifetime `payloadCache`, priming every chart in the group with one
request (~150–300 KB gzipped) instead of dozens. The `IntersectionObserver` still
gates the *Chart.js construction* (the actual CPU cost), and bundles run at
Expand Down
Loading
Loading