Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 84 additions & 6 deletions BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,21 @@ server, 7% on Hono, 30% smaller binaries and 0.9-2 MB less PSS.

## What is measured

Two applications, each compiled by geatsc and each also run under Node:
Three applications, each compiled by geatsc and each also run under Node:

- **Hono** (`apps/hono-hello/server.ts`): the real `hono` package with the
default `SmartRouter` stack, served through `@hono/node-server`. Under
geatsc the `serve` import resolves to node-compat's native answer for that
package (`runtime/node/node-server-adapter.ts`), so a request goes reactor →
`Request` → `app.fetch` → wire with no intermediate Node objects. The app
file is unchanged between the two runtimes.
- **NestJS** (`apps/hello-nest/server.ts`): `@nestjs/core` 12.5.1 with a
decorated controller, served through `@nestjs/platform-node`'s
`NodeAdapter` over `node:http`. The Node row runs the
`tsc` output of the same file (`dist-node/`), because Node's type stripping
has no decorators or decorator metadata; the gea row is `scripts/build.mjs`
on the file itself with the Nest AOT hook in `tools/nest/aot/`. The Nest
logger is off in both.
- **Raw `node:http`** (`apps/raw-http-hello/server.ts`): `createServer((req,
res) => ...)`, `writeHead`, `end`. Under geatsc this is node-compat's
`IncomingMessage`/`ServerResponse` implementation over the C++ reactor.
Expand Down Expand Up @@ -95,8 +102,8 @@ validated by the harness before any sample is taken (72 wire-contract checks,
capture group is `''`.

Routes: `GET /` returns `text/plain`, `GET /json` returns
`{"hello":"world"}`. Hono additionally serves `/missing` as a 404, which the
harness checks.
`{"hello":"world"}`. Hono and Nest additionally serve `/missing` as a 404,
which the harness checks.

## Machine and method

Expand Down Expand Up @@ -342,7 +349,7 @@ and relocated library. `bench/pss-breakdown.py` is the instrument: it sums

Run on an otherwise idle Linux host with at least eight logical CPUs. The
harness uses `taskset`, `/proc` and `lscpu`, so it does not run on macOS.
Ports 3000 (Node Hono), 3101 (raw servers) and 3900 (gea Hono) must be free.
Ports 3000 (Node Hono), 3101 (raw servers) and 3900 (gea Hono, Nest) must be free.

Requirements: Node.js and the repository's npm dependencies, Python 3, `wrk`,
`util-linux`, clang (18 or newer; `-flto` needs its gold plugin,
Expand All @@ -356,6 +363,7 @@ built from:
```sh
npm ci # @geastack/compiler + @geastack/node-compat from npm
npm ci --prefix apps/hono-hello # hono + @hono/node-server (no workspaces: the root install does not reach it)
npm ci --prefix apps/hello-nest # @nestjs/{common,core,platform-node} 12.5.1, reflect-metadata, rxjs, typescript
bash bench/goal-http-build.sh # gea servers + C++ and Rust controls, clang++ -Os -flto
```

Expand All @@ -382,6 +390,7 @@ script emit the servers. The gea servers are emitted by the one
```sh
cd /path/to/geastack/compiler && npm run build && cd ../node-compat
npm ci --prefix apps/hono-hello # needed on this path too
npm ci --prefix apps/hello-nest
bash bench/goal-http-build.sh
```

Expand Down Expand Up @@ -410,8 +419,8 @@ SCRIPTC_RAW=/path/to/raw-rp python3 bench/http-matrix.py --servers scriptc-raw \
check `/sys/devices/system/cpu/cpu*/topology/thread_siblings_list` before
choosing, because sibling numbering differs between machines and the wrong
choice silently measures a saturated host. `--servers` narrows a run to any of
`hono-gea`, `hono-node`, `gea-raw`, `node-raw`, `scriptc-raw`, `cpp-drogon`,
`rust-axum`, `rust-hyper`, `cpp-epoll`. The result JSON keeps every raw `wrk`
`hono-gea`, `hono-node`, `nest-gea`, `nest-node`, `gea-raw`, `node-raw`,
`scriptc-raw`, `cpp-drogon`, `rust-axum`, `rust-hyper`, `cpp-epoll`. The result JSON keeps every raw `wrk`
report, the percentile latencies, peak RSS and PSS per process, startup
samples, machine metadata, artifact SHA-256 hashes and the wire-contract
validation record.
Expand All @@ -428,3 +437,72 @@ package.json is read as a plain file, so an `exports` map that omits
two scripts that both say "build" can name different compilers: the 09-21
numbers were clang's while the document said g++, which was only caught by
reading the binaries' `.comment` sections.

## hello-nest and the full matrix on an arm64 Linux VM, 2026-11-21

Not the bench host. `bench/http-matrix.py` over all ten servers inside an
arm64 Linux container (Ubuntu 24.04, clang 18.1.3, Node v24.20.0) on a Colima
VM with 8 vCPUs on an Apple M1 (4 performance and 4 efficiency cores; the
VM's vCPUs are not pinned to physical cores, so `--server-cpus 0-3
--load-cpus 4-7` separates server and load generator inside the VM only).
The load generator is wrk 4.1.0 patched to time with the monotonic clock
(sha256 `6cd2ada4…`): the VM's guest agent steps its wall clock by about
100 ms, and stock wrk, which times with `gettimeofday`, reports false
timeouts after a backward step. The VM's other containers were stopped and
the macOS host ran nothing CPU-bound. Numbers here are comparable with each
other and with nothing in the tables above.

`apps/hello-nest` runs `@nestjs/{common,core,platform-node}` 12.5.1 from npm.
All three gea binaries were built inside the container by
`bench/goal-http-build.sh` with one compiler: `@geastack/compiler` main
`1ae87837` (1.0.61) plus the unreleased fixes hello-nest and Hono need
(geastack/compiler#21, #28, #29, #30, #31 and #33), packed and installed
through `GEA_COMPILER_DIR`. No released compiler builds this tree yet: main
alone refuses hello-nest at `instanceof Date` (#28) and does not compile
`apps/hono-hello` (#33). Ubuntu 24.04 ships libbrotlidec 1.1.0, which
`scripts/build.mjs` refuses, so brotli 1.2.1 was built into a prefix and
passed as `BROTLI_DIR`. hello-nest answered `scripts/test-hello-nest.mjs`
11 of 11 identical before the run, and all 72 wire checks passed. Binaries:
hono-gea 5,834,032 bytes, nest-gea 10,432,576, gea-raw 3,669,816; cpp-epoll
73,944, cpp-drogon 126,352, rust-hyper 1,029,480, rust-axum 1,218,160.

Records: `bench/results/http-colima-arm64-2026-11-21.json` (three complete
rounds per server, worker count and route) and its `-startup` file (five
launches each).

| Server | Single `/` | Single `/json` | Single RSS MiB | Single PSS MiB | Single startup ms | 4 workers `/` | 4 workers `/json` | 4 workers RSS MiB | 4 workers PSS MiB | 4 workers startup ms |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| **hono-gea** | **101,976** | **77,438** | 7.6 | 6.6 | 11 | **267,135** | **221,123** | 27.0 | 8.8 | 14 |
| hono-node | 19,602 | 18,276 | 106.2 | 105.1 | 93 | 57,472 | 52,756 | 432.8 | 254.7 | 136 |
| **nest-gea** | **67,391** | **64,149** | 12.9 | 11.5 | 19 | **201,071** | **185,376** | 37.8 | 13.6 | 21 |
| nest-node | 33,759 | 30,668 | 126.0 | 124.9 | 801 | 96,807 | 87,825 | 549.4 | 355.0 | 1,369 |
| **gea-raw** | **164,840** | **166,216** | 5.6 | 4.7 | 9 | **338,969** | **335,115** | 18.6 | 6.1 | 10 |
| node-raw | 35,551 | 35,323 | 87.2 | 86.1 | 69 | 103,145 | 100,339 | 357.0 | 180.5 | 113 |
| cpp-drogon | 120,560 | 123,346 | 11.6 | 10.1 | 20 | 285,978 | 286,891 | 11.8 | 10.3 | 12 |
| rust-axum | 141,322 | 145,564 | 4.1 | 3.2 | 6 | 227,313 | 227,929 | 5.0 | 4.1 | 6 |
| rust-hyper | 171,116 | 172,834 | 3.1 | 2.2 | 7 | 235,277 | 237,552 | 3.9 | 3.0 | 6 |
| cpp-epoll | 196,066 | 193,934 | 3.2 | 2.2 | 7 | 385,802 | 367,690 | 10.0 | 3.4 | 6 |

| Server | Single p50 | Single p99 | 4 workers p50 | 4 workers p99 |
| --- | ---: | ---: | ---: | ---: |
| hono-gea | 602 µs | 1.23 ms | 186 µs | 1.38 ms |
| hono-node | 3.15 ms | 6.43 ms | 1.02 ms | 2.71 ms |
| nest-gea | 0.90 ms | 1.81 ms | 266 µs | 1.24 ms |
| nest-node | 1.83 ms | 3.67 ms | 587 µs | 1.68 ms |
| gea-raw | 354 µs | 771 µs | 140 µs | 1.16 ms |
| node-raw | 1.76 ms | 3.53 ms | 549 µs | 1.63 ms |
| cpp-drogon | 488 µs | 1.14 ms | 167 µs | 1.05 ms |
| rust-axum | 436 µs | 614 µs | 237 µs | 748 µs |
| rust-hyper | 368 µs | 774 µs | 216 µs | 600 µs |
| cpp-epoll | 286 µs | 615 µs | 120 µs | 1.00 ms |

`GET /`, compiled over Node: hono-gea 5.20× at one worker and 4.65× at four,
nest-gea 2.00× and 2.08×, gea-raw 4.64× and 3.29×. At four workers nest-gea
is 0.75× hono-gea and 1.95× bare Node, and gea-raw is 0.88× cpp-epoll and
1.44× rust-hyper. PSS at four workers is 1/29 (Hono), 1/26 (Nest) and 1/30
(raw); nest-gea starts in 19-21 ms against nest-node's 801 ms and 1,369 ms.

The three rounds of each four-worker `GET /` cell stay within 2% for
nest-gea (199.9k-203.5k) and hono-gea (264.2k-271.1k) and within 6% for
gea-raw (331.0k-349.4k); rust-hyper spreads 16% (212.6k-247.2k), so its
ratio is the least firm one here.
10 changes: 10 additions & 0 deletions apps/hello-nest/cluster.node.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
import cluster from 'node:cluster'
import os from 'node:os'

if (cluster.isPrimary) {
const n = Number(process.env.NODE_WORKERS || os.availableParallelism())
for (let i = 0; i < n; i++) cluster.fork()
console.log(`nest-node cluster: ${n} workers on http://127.0.0.1:3900`)
} else {
await import('./dist-node/server.js')
}
2 changes: 1 addition & 1 deletion apps/hello-nest/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"type": "module",
"scripts": {
"build": "tsc -p tsconfig.json",
"start": "node dist/server.js"
"start": "node dist-node/server.js"
},
"dependencies": {
"@nestjs/common": "12.5.1",
Expand Down
2 changes: 1 addition & 1 deletion apps/hello-nest/tsconfig.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"experimentalDecorators": true,
"useDefineForClassFields": false,
"skipLibCheck": true,
"outDir": "./dist"
"outDir": "./dist-node"
},
"files": ["server.ts"]
}
7 changes: 7 additions & 0 deletions bench/goal-http-build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,12 @@ CXX="$CXX_BIN" node scripts/build.mjs \
apps/hono-hello/server.ts \
--out apps/hono-hello/dist

echo "=== Nest Gea: NestJS on @nestjs/platform-node through the shared compiler ==="
npm --prefix apps/hello-nest run build
CXX="$CXX_BIN" node scripts/build.mjs \
apps/hello-nest/server.ts \
--out apps/hello-nest/dist

echo "=== Gea raw: node:http source through the shared compiler ==="
CXX="$CXX_BIN" node scripts/build.mjs \
apps/raw-http-hello/server.ts \
Expand All @@ -77,6 +83,7 @@ echo "=== Rust Hyper and Axum controls ==="

artifacts=(
apps/hono-hello/dist/server
apps/hello-nest/dist/server
apps/raw-http-hello/dist/server
apps/raw-http-hello/dist/cpp-epoll
apps/raw-http-hello/dist/cpp-drogon
Expand Down
47 changes: 36 additions & 11 deletions bench/http-matrix.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
parser.add_argument('--rounds', type=int, default=2)
parser.add_argument('--duration', default='8s')
parser.add_argument('--servers', nargs='+', default=[
'hono-gea', 'hono-node', 'gea-raw', 'node-raw',
'hono-gea', 'hono-node', 'nest-gea', 'nest-node', 'gea-raw', 'node-raw',
'cpp-drogon', 'rust-axum', 'rust-hyper', 'cpp-epoll'])
parser.add_argument('--workers', nargs='+', type=int, default=[1, 8])
# `dist` is the historical build; `dist` is what `scripts/build.mjs` writes today.
Expand Down Expand Up @@ -56,10 +56,16 @@ def command(name, workers):
# adapter, kept beside the native build as `server-pkg`.
cmd = [f'apps/hono-hello/{args.gea_dist}/server-pkg']
env['GEA_WORKERS'] = str(workers)
elif name in ('hono-gea', 'gea-raw'):
app = 'hono-hello' if name == 'hono-gea' else 'raw-http-hello'
elif name in ('hono-gea', 'nest-gea', 'gea-raw'):
app = {'hono-gea': 'hono-hello', 'nest-gea': 'hello-nest', 'gea-raw': 'raw-http-hello'}[name]
cmd = [f'apps/{app}/{args.gea_dist}/server']
env['GEA_WORKERS'] = str(workers)
elif name == 'nest-node':
# The same server.ts the nest-gea row compiles, built by tsc because Node's
# type stripping has no decorators or decorator metadata.
entry = 'dist-node/server.js' if workers == 1 else 'cluster.node.mjs'
cmd = ['node', f'apps/hello-nest/{entry}']
env['NODE_WORKERS'] = str(workers)
elif name == 'hono-node-full':
# Node on the unchanged server.ts (type stripping), not the tiny bridge.
entry = 'server.ts' if workers == 1 else 'cluster.full.node.mjs'
Expand Down Expand Up @@ -98,6 +104,19 @@ def hono_gea_port():
return int(match[1]) if match else 3000


def nest_port():
source = Path('apps/hello-nest/server.ts').read_text()
match = re.search(r'listen\((\d+)\)', source)
return int(match[1]) if match else 3000


ROOT_BODY = {'hono-': 'Hello Hono!', 'nest-': 'Hello Nest!'}


def framework(name):
return next((prefix for prefix in ROOT_BODY if name.startswith(prefix)), None)


def port_free(port):
with socket.socket() as sock:
return sock.connect_ex(('127.0.0.1', port)) != 0
Expand Down Expand Up @@ -258,7 +277,13 @@ def wrk(port, path, duration, warmup=False):
artifacts = [
f'apps/hono-hello/{args.gea_dist}/server', f'apps/hono-hello/{args.gea_dist}/server-pkg',
f'apps/hono-hello/{args.gea_dist}/server.cpp',
f'apps/hono-hello/{args.gea_dist}/gea_runtime.h', f'apps/raw-http-hello/{args.gea_dist}/server',
f'apps/hono-hello/{args.gea_dist}/gea_runtime.h',
f'apps/hello-nest/{args.gea_dist}/server', f'apps/hello-nest/{args.gea_dist}/server.cpp',
f'apps/hello-nest/{args.gea_dist}/gea_runtime.h',
'apps/hello-nest/server.ts', 'apps/hello-nest/dist-node/server.js', 'apps/hello-nest/cluster.node.mjs',
'apps/hello-nest/node_modules/@nestjs/core/package.json',
'apps/hello-nest/node_modules/@nestjs/platform-node/package.json',
f'apps/raw-http-hello/{args.gea_dist}/server',
f'apps/raw-http-hello/{args.gea_dist}/server.cpp', f'apps/raw-http-hello/{args.gea_dist}/gea_runtime.h',
'apps/raw-http-hello/dist/cpp-drogon', 'apps/raw-http-hello/dist/cpp-epoll',
'runtime/gea_node.cpp', 'runtime/gea_node.hpp',
Expand Down Expand Up @@ -322,15 +347,15 @@ def save():
# 3900 (port 3000 is taken on the developer Mac), the node entry kept
# 3000, raw-http-hello listens on 3101.
port = hono_gea_port() if name in ('hono-gea', 'hono-gea-pkg', 'hono-node-full') else \
3000 if name == 'hono-node' else 3101
nest_port() if name.startswith('nest-') else 3000 if name == 'hono-node' else 3101
if not port_free(port):
raise RuntimeError(f'Port {port} occupied before {name}')
cmd, env = command(name, workers)
started = time.monotonic()
proc = subprocess.Popen(cmd, env=env, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True, start_new_session=True)
try:
expected = 'Hello Hono!' if name.startswith('hono-') else 'Hello, World! GET /'
expected = ROOT_BODY.get(framework(name), 'Hello, World! GET /')
for attempt in range(1000):
if proc.poll() is not None:
raise RuntimeError(f'{name} exited: {proc.stdout.read()}')
Expand All @@ -349,9 +374,9 @@ def save():
status, content_type, body = request(port, '/json')
if status != 200 or body != '{"hello":"world"}' or not content_type.startswith('application/json'):
raise RuntimeError(f'{name} wrong JSON: {status} {content_type} {body}')
if name.startswith('hono-') and request(port, '/missing')[0] != 404:
if framework(name) and request(port, '/missing')[0] != 404:
raise RuntimeError(f'{name} wrong missing-route status')
if not name.startswith('hono-'):
if not framework(name):
for path in ('/', '/json'):
contract = validate_raw_wire(port, path)
contract.update({'server': name, 'workers': workers})
Expand All @@ -363,9 +388,9 @@ def save():
# Let all cluster/fork workers finish startup before warming up.
time.sleep(1)
before = process_memory(proc.pid)
expected_processes = workers + (1 if name in ('hono-node', 'hono-node-full', 'node-raw') else 0)
if workers > 1 and name in ('hono-gea', 'hono-gea-pkg', 'gea-raw', 'hono-node', 'hono-node-full',
'node-raw', 'cpp-epoll'):
expected_processes = workers + (1 if name in ('hono-node', 'hono-node-full', 'nest-node', 'node-raw') else 0)
if workers > 1 and name in ('hono-gea', 'hono-gea-pkg', 'nest-gea', 'gea-raw', 'hono-node', 'hono-node-full',
'nest-node', 'node-raw', 'cpp-epoll'):
if len(before['processes']) != expected_processes:
raise RuntimeError(f'{name}: expected {expected_processes} processes, got {before}')
wrk(port, '/', '2s', warmup=True)
Expand Down
Loading
Loading