Skip to content

Commit 2fbe9cd

Browse files
1stvamp1stvamp
committed
docs(run-engine): make the e2e bench harness work on self-hosted
Enumerate runs from a trigger-time id manifest instead of runs.list (the list API can return empty on self-hosted, where it is ClickHouse-backed), add waitdrain and preflight helpers, pin the CLI and SDK to a matching version, and document the deploy invocation that actually works on self-hosted: no --self-hosted or --local-build, and --network host so the in-build indexer step can reach the instance API.
1 parent 0b40803 commit 2fbe9cd

6 files changed

Lines changed: 299 additions & 131 deletions

File tree

‎internal-packages/run-engine/design/benchmarks/e2e-tasks/README.md‎

Lines changed: 58 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,72 @@
11
# CK virtual-time end-to-end bench tasks
22

33
A tiny self-contained trigger.dev project used as the END-TO-END arm of the CK
4-
virtual-time A/B (see `../2026-07-26-ck-vtime-benchmark.md`). It is deployed to a
4+
virtual-time A/B (see `../2026-07-26-ck-vtime-benchmark.md`). It deploys to a
55
self-hosted instance and is not part of the monorepo build.
66

77
- `src/trigger/ckBench.ts` — one `ck-bench` task on a shared base queue; per-run
88
`concurrencyKey` makes the CK variants; a slot hold forces contention.
99
- `src/loadgen.ts` — noisy-neighbor load: tenant A floods across many keys,
10-
tenant B sends a few; each run is tagged and carries a per-run `region`.
11-
- `src/collect.ts` — reads per-run `createdAt`/`startedAt` by tag and reports
10+
tenant B sends a few; each run carries a per-run `region`. Captures every run
11+
id at trigger time and writes a manifest (`e2e-results/manifest-<batch>.json`).
12+
- `src/waitdrain.ts` — polls the manifest's run ids until all are terminal.
13+
- `src/collect.ts` — reads each run's `createdAt`/`startedAt` by id and reports
1214
per-tenant enqueue->start latency p50/p95/p99.
15+
- `src/preflight.ts` — fires one run per region to validate routing + access.
1316

1417
Everything reads credentials from the environment (`TRIGGER_API_URL`,
15-
`TRIGGER_SECRET_KEY`); nothing is hard-coded. Deploy with the CLI (`-p <ref>`).
16-
The feature flag is server-side and is flipped by the operator between arms, not
17-
by this project. Exact instance coordinates and the toggle live in the operator
18-
runbook, kept outside this public repo.
18+
`TRIGGER_SECRET_KEY`); nothing is hard-coded. The feature flag is server-side and
19+
is flipped by the operator between arms, not by this project. Exact instance
20+
coordinates and the toggle live in the operator runbook, kept outside this repo.
21+
22+
## Deploy from a standalone checkout, not inside the monorepo
23+
24+
This project must be installed and deployed from OUTSIDE the pnpm monorepo tree
25+
(copy it somewhere with no `pnpm-workspace.yaml` ancestor). Inside the workspace,
26+
`pnpm install` binds to the workspace and links the local SDK instead of the
27+
pinned published one. Use npm for the standalone copy (it avoids pnpm's
28+
build-script policy on esbuild):
29+
30+
```bash
31+
cp -r <this dir> ~/ck-vtime-e2e && cd ~/ck-vtime-e2e
32+
npm install # CLI + SDK pinned to the same version (4.5.7)
33+
```
34+
35+
## Deploy
36+
37+
The 4.5.7 CLI has no `--self-hosted` flag; self-hosted is implicit from the
38+
profile's API URL. Do NOT pass `--local-build` (it routes to a cloud-only ECR
39+
credential endpoint and fails on self-hosted). The push then rides the host's
40+
existing docker login to the registry. `--network host` is required so the
41+
in-build indexer step can reach the instance API (e.g. over a tailnet):
42+
43+
```bash
44+
TRIGGER_PROJECT_REF=<ref> \
45+
./node_modules/.bin/trigger deploy -e prod --network host --profile <profile> -p <ref>
46+
```
47+
48+
Deploy to the `prod` environment: the `dev` environment short-circuits
49+
worker-group routing, so dev runs never reach the managed regions.
50+
51+
## Run one A/B arm
52+
53+
The flag is flipped + redeployed server-side by the operator; run this once per
54+
arm with the matching `ARM`. This instance's `runs.list` (ClickHouse-backed) can
55+
be empty, so the harness enumerates runs from the trigger-time id manifest, not
56+
by tag.
1957

2058
```bash
21-
pnpm install
22-
# deploy (project ref on the CLI)
23-
TRIGGER_PROJECT_REF=<ref> npx trigger.dev@latest deploy --self-hosted --profile <profile> -p <ref>
24-
# generate load for one arm (flag already set + redeployed server-side)
25-
ARM=off BATCH=run1 pnpm loadgen
26-
# collect that arm
27-
ARM=off BATCH=run1 pnpm collect
59+
export TRIGGER_API_URL=<instance url>
60+
export TRIGGER_SECRET_KEY=<prod env secret key>
61+
62+
# validate region routing once
63+
./node_modules/.bin/tsx src/preflight.ts
64+
65+
# one arm (env knobs: HOLD_MS, A_KEYS, A_PER_KEY, B_KEYS, B_PER_KEY, REGIONS)
66+
ARM=off BATCH=off-1 ./node_modules/.bin/tsx src/loadgen.ts
67+
BATCH=off-1 ./node_modules/.bin/tsx src/waitdrain.ts
68+
ARM=off BATCH=off-1 ./node_modules/.bin/tsx src/collect.ts
2869
```
70+
71+
Results land in `e2e-results/` (`manifest-<batch>.json`, `e2e-<batch>-<arm>.json`,
72+
`e2e-summary.md`).

‎internal-packages/run-engine/design/benchmarks/e2e-tasks/package.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,10 @@
99
"collect": "tsx src/collect.ts"
1010
},
1111
"dependencies": {
12-
"@trigger.dev/sdk": "latest"
12+
"@trigger.dev/sdk": "4.5.7"
1313
},
1414
"devDependencies": {
15-
"trigger.dev": "latest",
15+
"trigger.dev": "4.5.7",
1616
"tsx": "^4.19.0",
1717
"typescript": "^5.5.0"
1818
}

‎internal-packages/run-engine/design/benchmarks/e2e-tasks/src/collect.ts‎

Lines changed: 50 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,15 @@
11
/**
2-
* Collect per-tenant enqueue->start latency for one END-TO-END arm.
2+
* Collect per-tenant enqueue->start latency for one END-TO-END arm, from a batch
3+
* manifest (retrieve by id; runs.list is unreliable on this instance).
34
*
4-
* Lists the runs for a batch by tag, reads each run's createdAt (enqueue) and
5-
* startedAt (execution start), and reports per-tenant p50/p95/p99 of
6-
* (startedAt - createdAt). Run once per arm; point --arm/BATCH at the tags the
7-
* loadgen used.
5+
* Metric per run = startedAt - createdAt (run-start latency: queue wait + dequeue
6+
* + worker pickup). Headline is tenant B (few keys): bounded under vtime, grows
7+
* with A's backlog under baseline.
88
*
9-
* The headline metric is tenant B's start latency: under the baseline it should
10-
* grow with tenant A's backlog (B waits behind the flood); under vtime it should
11-
* stay bounded (B takes its fair turn). Tenant A's latency is reported for
12-
* context and is expected to be similar or slightly higher under vtime.
13-
*
14-
* NOTE ON THE TIMESTAMP: startedAt - createdAt is the honest run-start latency a
15-
* reviewer cares about (queue wait + dequeue + worker pickup). For a tighter
16-
* dequeue-only number, use the Postgres/TRQL alternative in the benchmark doc.
17-
*
18-
* Auth + config (env):
19-
* TRIGGER_API_URL, TRIGGER_SECRET_KEY (prod env secret key)
20-
* ARM=off|on BATCH=<id> OUT=./e2e-results
9+
* Env: TRIGGER_API_URL, TRIGGER_SECRET_KEY. Args (env): BATCH ARM OUT POLL_CONCURRENCY
2110
*/
2211
import { configure, runs } from "@trigger.dev/sdk";
23-
import { appendFileSync, mkdirSync, writeFileSync } from "node:fs";
12+
import { appendFileSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
2413

2514
function pct(sorted: number[], p: number): number {
2615
if (sorted.length === 0) return NaN;
@@ -34,65 +23,70 @@ function summarize(xs: number[]) {
3423
}
3524

3625
async function main() {
37-
const apiURL = process.env.TRIGGER_API_URL;
38-
const accessToken = process.env.TRIGGER_SECRET_KEY;
39-
if (!apiURL || !accessToken) throw new Error("Set TRIGGER_API_URL and TRIGGER_SECRET_KEY.");
40-
configure({ baseURL: apiURL, accessToken });
41-
26+
configure({
27+
baseURL: process.env.TRIGGER_API_URL!,
28+
accessToken: process.env.TRIGGER_SECRET_KEY!,
29+
});
30+
const batch = process.env.BATCH!;
4231
const arm = (process.env.ARM ?? "off") as "off" | "on";
43-
const batch = process.env.BATCH;
4432
const outDir = process.env.OUT ?? "./e2e-results";
45-
if (!batch) throw new Error("Set BATCH=<id> to the loadgen batch id.");
33+
const conc = Number(process.env.POLL_CONCURRENCY ?? "20");
34+
const manifest = JSON.parse(readFileSync(`${outDir}/manifest-${batch}.json`, "utf8"));
35+
const entries: { id: string; tenant: string }[] = manifest.runs;
4636

4737
const waitsByTenant = new Map<string, number[]>();
48-
let total = 0;
4938
let missingStart = 0;
50-
51-
// Page through every run carrying this batch tag. BATCH is unique per arm
52-
// (e.g. off-1 / on-1), so the single batch tag identifies the arm; filtering
53-
// on one tag avoids any multi-tag AND/OR ambiguity in the runs filter.
54-
for await (const run of runs.list({ tag: `batch:${batch}`, limit: 100 })) {
55-
total++;
56-
const detail = await runs.retrieve(run.id);
57-
const createdAt = detail.createdAt?.getTime();
58-
const startedAt = detail.startedAt?.getTime();
59-
if (createdAt === undefined || startedAt === undefined) {
60-
missingStart++;
61-
continue;
39+
let i = 0;
40+
async function w() {
41+
while (i < entries.length) {
42+
const e = entries[i++]!;
43+
try {
44+
const r = await runs.retrieve(e.id);
45+
const c = r.createdAt?.getTime();
46+
const s = r.startedAt?.getTime();
47+
if (c === undefined || s === undefined) {
48+
missingStart++;
49+
continue;
50+
}
51+
const arr = waitsByTenant.get(e.tenant) ?? waitsByTenant.set(e.tenant, []).get(e.tenant)!;
52+
arr.push(s - c);
53+
} catch {
54+
missingStart++;
55+
}
6256
}
63-
const tenantTag = (detail.tags ?? []).find((t) => t.startsWith("tenant:")) ?? "tenant:?";
64-
const tenant = tenantTag.slice("tenant:".length);
65-
const wait = startedAt - createdAt;
66-
(waitsByTenant.get(tenant) ?? waitsByTenant.set(tenant, []).get(tenant)!).push(wait);
6757
}
58+
await Promise.all(Array.from({ length: Math.min(conc, entries.length) }, w));
6859

6960
const perTenant: Record<string, ReturnType<typeof summarize>> = {};
70-
for (const [tenant, xs] of waitsByTenant) perTenant[tenant] = summarize(xs);
71-
72-
const report = { arm, batch, total, missingStart, unit: "ms (startedAt - createdAt)", perTenant };
61+
for (const [t, xs] of waitsByTenant) perTenant[t] = summarize(xs);
62+
const report = {
63+
arm,
64+
batch,
65+
total: entries.length,
66+
missingStart,
67+
unit: "ms (startedAt - createdAt)",
68+
perTenant,
69+
};
7370

7471
mkdirSync(outDir, { recursive: true });
7572
writeFileSync(`${outDir}/e2e-${batch}-${arm}.json`, JSON.stringify(report, null, 2));
7673

77-
// Append a human row per tenant to a shared markdown file so OFF and ON land
78-
// in one table you can eyeball before running the joiner.
79-
const mdPath = `${outDir}/e2e-summary.md`;
8074
const rows = Object.entries(perTenant)
8175
.map(
82-
([tenant, s]) =>
83-
`| ${batch} | ${arm} | ${tenant} | ${s.count} | ${s.mean.toFixed(0)} | ${s.p50.toFixed(0)} | ${s.p95.toFixed(0)} | ${s.p99.toFixed(0)} |`
76+
([t, s]) =>
77+
`| ${batch} | ${arm} | ${t} | ${s.count} | ${s.mean.toFixed(0)} | ${s.p50.toFixed(0)} | ${s.p95.toFixed(0)} | ${s.p99.toFixed(0)} |`
8478
)
8579
.join("\n");
8680
appendFileSync(
87-
mdPath,
81+
`${outDir}/e2e-summary.md`,
8882
`\n<!-- batch ${batch} arm ${arm} -->\n| batch | arm | tenant | runs | mean ms | p50 | p95 | p99 |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n${rows}\n`
8983
);
90-
91-
console.log(`[collect] arm=${arm} batch=${batch} runs=${total} missingStart=${missingStart}`);
84+
console.log(
85+
`[collect] arm=${arm} batch=${batch} runs=${entries.length} missingStart=${missingStart}`
86+
);
9287
console.log(JSON.stringify(perTenant, null, 2));
9388
}
94-
95-
main().catch((err) => {
96-
console.error(err);
89+
main().catch((e) => {
90+
console.error(e);
9791
process.exit(1);
9892
});

0 commit comments

Comments
 (0)