Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions apps/desktop/bench/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# CF-8 retention reproduction

These opt-in benchmarks reproduce the reviewed history/log workloads for #281 through existing Electron/main/native-adapter fixtures. They are outside the ordinary E2E directory and do not run in routine CI. No real model or model server is used; the synthetic CLI reports command-shaped events without executing the displayed command. Installed apps and user profiles are not replaced.

From an owned worktree with the pinned pnpm dependencies installed, fetch both exact source objects before running. The current comparison source is PR #503’s pre-squash head, so a fresh clone of `main` may not contain it. These commands fetch source only; they do not switch or modify the worktree:

```sh
git fetch origin 89c98f280292b13fd9b1c7af1bf2fe65cd2fa0d1 44fb41eecf718cf6648b7c8ee95cab1e85e0e072
node scripts/cf8-retention.mjs \
--output /absolute/private/new-retention-directory \
--baseline 89c98f280292b13fd9b1c7af1bf2fe65cd2fa0d1 \
--current 44fb41eecf718cf6648b7c8ee95cab1e85e0e072
```

The output parent must exist; the final directory must be new. The CLI creates it privately, refuses existing files/directories and dangling symlinks, and never retries or deletes a failed run. Source exports and generated drivers stay in a new owned `apps/desktop/.probes/cf8-retention-*` directory. Each source’s seeded homes and repositories live under its private `fixtures/` directory, with locators written before further fixture setup. Worker exit cleanup is disabled only for this benchmark. The existing seeding helper uses that private root synchronously, then the ordinary runtime temporary-directory context is restored before app launch. All inputs, build logs, manifests, raw measurement logs and partial/final observations remain retained for review and guarded worktree cleanup.

Both full commit identities are mandatory. Sources are exported and verified against their Git blobs, built sequentially and rechecked before measurement. They share the driver's already-installed pinned dependencies and freshly built common core, so all three core source trees must match and the driver's core must be clean. Before building, the desktop’s installed core alias and resolved package entry must bind to that driver core; package metadata and the built entry are recorded and rechecked before launches. This compares rebuilt source, not historical shipped binaries or a recreation of the historical lockfile. These named-Mac workloads currently require macOS; offline hash verification is separate.

After both builds finish, current-source history/log cases run before baseline-source cases, serially, with one worker, no retries and a first-failure stop. Each source has three fresh-app history replicates (three 200-turn/1,400-item histories, three warm opens and 63 measured opens) and three fresh-app log replicates (one warm-up and 21 measured streams, each 3 MiB logical output in 48 deltas). Renderer GC is forced only at matched warm detached endpoints; intermediate renderer and all main snapshots use natural GC. The one/five-second detach samples are declared observation times, not readiness waits. The legacy-only shallow ASCII read and expected full-output violation remain explicit; current 256 KiB preview/IPC assertions are unchanged.

`--prepare-only` performs source/build/driver preparation without launching an app. Recheck retained inputs without executing a workload with:

```sh
node scripts/cf8-retention.mjs --verify /absolute/private/new-retention-directory/current-manifest.json
```

Verification binds source/build/common-core/dependency links, installed core package/entry identity and the original/generated driver files to their recorded hashes. Driver paths must remain regular, canonical paths under the declared driver root; matching-byte symlinks cannot redirect relative imports. Measurements separately report the target source and the diagnostic driver's checkout identity; the latter is never presented as the exported application's Git HEAD. The manifest records uncommitted driver bytes explicitly if preparation precedes an implementation commit.

Successful execution requires all twelve original cases to pass, all six observations per source to report complete state and returned app-close calls, and source/build/driver checks to remain valid afterwards. The receipts contain host/runtime identities, timestamps, load averages, per-replicate data and first failures. Local paths and process IDs remain private; public aggregates must omit them without changing numeric observations.

Numbers vary with host load, dependency/runtime versions and group ordering. Two animation frames are a sampling point rather than exhaustive settlement; sampled values are not operation high-water marks. There is no new memory or latency budget, universal ceiling, causal improvement, leak-freedom, real-provider, native-platform or human-accessibility claim. This tooling addresses the portable measurement procedure; #281's visual inventory and remaining native/device/human/child requirements remain separate.
3 changes: 3 additions & 0 deletions apps/desktop/bench/retention-env.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
import type { ModexBridge } from "../src/shared/types";
declare global { interface Window { modex?: ModexBridge; } }
export {};
21 changes: 21 additions & 0 deletions apps/desktop/bench/retention/history-fixture.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
const TURNS = 200;
export function longTranscript(threadId: string): Record<string, unknown>[] {
const out: Record<string, unknown>[] = [];
const at = new Date(Date.now() - TURNS * 60_000).toISOString();
for (let turn = 0; turn < TURNS; turn++) {
const user = `u-${turn}`;
const origin = (item: string) => ({ thread: threadId, turn: user, attempt: `a-${turn}`, backend: "codex", native: { runtime: "R", thread: "nt", turn: `nu-${turn}`, item } });
out.push({ id: user, kind: "user", text: `Request ${turn}: tidy the module and run its tests.`, at });
out.push({ id: `r-${turn}`, kind: "thinking", text: `Plan for ${turn}: ${"read the module, then test it. ".repeat(10)}`, reasoning: "summary", status: "done", outcome: "succeeded", durationMs: 2000, origin: origin(`r-${turn}`), at });
out.push({ id: `c-${turn}`, kind: "assistant", text: `Looking at module ${turn}. ${"Reading the files first. ".repeat(8)}`, phase: "commentary", nativePhase: "commentary", outcome: "succeeded", origin: origin(`c-${turn}`), at });
for (let k = 0; k < 3; k++) {
const id = `t-${turn}-${k}`;
out.push({ id, kind: "tool", name: "shell", title: `$ npm test -- part ${k}`, args: { command: `npm test -- part ${k}` }, status: "done", ok: true, outcome: "succeeded",
exec: { command: `/bin/zsh -lc 'npm test -- part ${k}'`, cwd: "/repo" }, exitCode: 0, output: `ok ${turn}.${k}\n`.repeat(400), outputBytes: 4000, durationMs: 1200, origin: origin(id), at });
}
out.push({ id: `f-${turn}`, kind: "assistant", text: `Answer ${turn}.\n\n- Tidied module ${turn}.\n- Tests pass.\n\n\`\`\`ts\nexport const value${turn} = ${turn};\n\`\`\`\n${"More detail on the change. ".repeat(20)}`,
phase: "final", nativePhase: "final_answer", outcome: "succeeded", origin: origin(`f-${turn}`), at });
}
return out;
}

98 changes: 98 additions & 0 deletions apps/desktop/bench/retention/history.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
import { test, expect } from "@playwright/test";
import fs from "node:fs";
import path from "node:path";
import os from "node:os";
import crypto from "node:crypto";
import { execFileSync } from "node:child_process";
import { pathToFileURL } from "node:url";
import { appDir, launch, seedHome, tid, items } from "./target-support";
import { longTranscript } from "./history-fixture";

import { root, sourceHead, driverHead, driverRoot, manifest, verifySource } from "./source-guard";
for (let replicate = 1; replicate <= 3; replicate++) {
test(`three-history natural-GC observations replicate ${replicate}`, async () => {
const { home, repo } = seedHome();
const receipt = path.join(root, `history-${replicate}.json`);
expect(fs.existsSync(receipt)).toBe(false);
fs.writeFileSync(receipt, JSON.stringify({ state: "fixture-seeded", sourceHead, sourceRole: manifest.role, fixture: { home, repo }, at: new Date().toISOString() }, null, 2) + "\n", { flag: "wx", mode: 0o600 });
const now = new Date().toISOString();
const ids = ["history-a", "history-b", "history-c"];
const files = ids.map(id => path.join(home, "app", "threads", id + ".json"));
const digest = (file: string) => crypto.createHash("sha256").update(fs.readFileSync(file)).digest("hex");
for (const [index, id] of ids.entries()) {
const saved = longTranscript(id);
saved[0]!.text = `Distinct history ${id}: tidy module group ${index}.`;
fs.writeFileSync(files[index]!, JSON.stringify(saved));
}
const beforeHashes = files.map(digest);
const statePath = path.join(home, "app", "state.json");
const state = JSON.parse(fs.readFileSync(statePath, "utf8"));
state.threads = ids.map(id => ({ id, projectId: "p1", title: id, titleSource: "manual", createdAt: now, updatedAt: now, cwd: repo, backend: "codex", mode: "chat", plan: false, model: "", status: "idle" }));
fs.writeFileSync(statePath, JSON.stringify(state));
const out: Record<string, any> = { diagnosticDriverHead: execFileSync("git", ["rev-parse", "HEAD"], { cwd: driverRoot, encoding: "utf8" }).trim(), sourceHead, sourceRole: manifest.role, fixture: { home, repo }, replicate, startedAt: now, state: "started", samples: [],
workload: { histories: 3, shape: "200 turns/1400 items each; distinct thread provenance and first request, otherwise same deterministic shape", warmupOpens: 3, measuredOpens: 63, opensPerHistory: 21, savedBytes: files.map(f => fs.statSync(f).size), savedSha256: beforeHashes },
host: { cpu: os.cpus()[0]?.model, cores: os.cpus().length, memoryBytes: os.totalmem(), release: os.release(), arch: os.arch(), node: process.version, loadAverage: os.loadavg() },
instrumentation: "Renderer forced GC only after three-history warmup and at final detached endpoint. All 63 per-open samples and timed post-detach observations are natural-GC snapshots. Main always natural GC. Two RAF plus positive last-answer/nondeferred checks are a sampling point, not exhaustive async settlement. Trace off. No latency/memory-ceiling/leak-freedom or historical-improvement claim." };
expect(out.diagnosticDriverHead).toBe(driverHead);
const persist = () => fs.writeFileSync(receipt, JSON.stringify(out, null, 2) + "\n"); persist();
let opened: Awaited<ReturnType<typeof launch>> | undefined;
try {
out.sourceVerification = verifySource();
expect(appDir).toBe(manifest.appDir);
opened = await launch(home);
const { app, page } = opened;
const cdp = await page.context().newCDPSession(page);
out.native = await app.evaluate(({ app, BrowserWindow }) => ({ appVersion: app.getVersion(), electron: process.versions.electron, chromium: process.versions.chrome, mainPid: process.pid, executablePath: process.execPath, appPath: app.getAppPath(), userData: app.getPath("userData"), rendererPids: BrowserWindow.getAllWindows().map(w => w.webContents.getOSProcessId()) }));
expect(out.native.appPath).toBe(manifest.appDir); expect(out.native.appVersion).toBe(manifest.appVersion);
out.document = await page.evaluate(() => location.href); expect(out.document).toBe(pathToFileURL(path.join(appDir, "dist/renderer/index.html")).href);
const observe = async (phase: string, forcedGc = false) => {
await page.evaluate(() => new Promise<void>(resolve => requestAnimationFrame(() => requestAnimationFrame(() => resolve()))));
if (forcedGc) await cdp.send("HeapProfiler.collectGarbage");
const heap = await cdp.send("Runtime.getHeapUsage");
const dom = await cdp.send("Memory.getDOMCounters");
const main = await app.evaluate(async ({ app }) => ({ pid: process.pid, node: process.memoryUsage(), nativeKiB: await process.getProcessMemoryInfo(), metrics: app.getAppMetrics() }));
return { at: new Date().toISOString(), phase, forcedGc, heap, dom, main, renderedItems: await items(page).count(), selectedThreadIds: await page.locator('[data-testid="thread-row"][aria-current="true"]').evaluateAll(rows => rows.map(row => row.getAttribute("data-thread-id"))) };
};
const detach = async () => { await page.keyboard.press("Meta+n"); await expect(tid(page, "draft-view")).toBeVisible(); return Date.now(); };
const open = async (id: string) => {
await detach();
const row = page.locator(`[data-testid="thread-row"][data-thread-id="${id}"]`);
await row.getByRole("button", { name: id, exact: true }).click();
await expect(row).toHaveAttribute("aria-current", "true");
await expect(items(page, "user").first()).toContainText(`Distinct history ${id}:`);
const last = items(page, "assistant").last();
await expect(last).toContainText("Answer 199.");
await expect(last.locator("[data-markdown-deferred]")).toHaveCount(0);
await expect(items(page)).toHaveCount(1400);
};
out.cold = await observe("cold draft natural");
for (const id of ids) await open(id);
await detach();
out.warmDetachedBefore = await observe("after warming all three histories, detached", true); persist();
for (let index = 0; index < 63; index++) {
const id = ids[index % ids.length]!;
await open(id);
out.samples.push({ history: id, open: index + 1, ...await observe(`natural open ${index + 1}`) }); persist();
}
const detachedAt = await detach();
out.detachTimeOrigin = "draft-view positively observed, before immediate measurement";
out.detachedNatural = await observe("detached, immediate natural"); persist();
out.detachedTimedNatural = [];
for (const intervalMs of [1000, 5000]) {
// A declared measurement time point, not a readiness wait or retry.
await new Promise(resolve => setTimeout(resolve, Math.max(0, detachedAt + intervalMs - Date.now())));
out.detachedTimedNatural.push({ requestedAfterDetachMs: intervalMs, actualAfterDetachMs: Date.now() - detachedAt, ...await observe(`detached natural +${intervalMs}ms`) }); persist();
}
out.warmDetachedAfter = await observe("final warm detached endpoint", true);
out.savedAfterSha256 = files.map(digest); expect(out.savedAfterSha256).toEqual(beforeHashes);
expect(out.samples).toHaveLength(63); expect(out.samples.every((s: any) => !s.forcedGc && s.renderedItems === 1400 && s.selectedThreadIds.length === 1 && s.selectedThreadIds[0] === s.history)).toBe(true);
expect(out.warmDetachedBefore.renderedItems).toBe(0); expect(out.warmDetachedAfter.renderedItems).toBe(0);
out.state = "observations-complete"; persist();
} catch (error) { out.state = "failed"; out.failure = String(error); persist(); throw error; }
finally {
try { if (opened) { await opened.app.close(); out.appClose = "returned"; } else out.appClose = "launch returned no app handle"; }
catch (error) { out.appClose = "failed"; out.closeError = String(error); throw error; }
finally { out.finishedAt = new Date().toISOString(); persist(); }
}
});
}
57 changes: 57 additions & 0 deletions apps/desktop/bench/retention/log-fixture.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
import fs from "node:fs";
import path from "node:path";
export const CHUNK = "x".repeat(64 * 1024 - 1) + "\n";
export const CHUNKS = 48;
export const PREVIEW = 256 * 1024;
// Canonical synthetic CLI plus local counter observations; never executes the emitted shell command.
export function fakeCodex(dir: string): string {
const cli = path.join(dir, "codex-stream.cjs");
fs.writeFileSync(cli, `#!${process.execPath}
if (process.argv.includes('--version')) { console.log('codex-cli 0.163.0'); process.exit(0); }
if (process.argv.includes('login') || process.argv.includes('auth')) { console.log('Logged in using ChatGPT'); process.exit(0); }
const emit = (o) => process.stdout.write(JSON.stringify(o) + '\\n');
const audit = (kind, fields) => require('node:fs').appendFileSync(${JSON.stringify(path.join(dir, "stream-calls.jsonl"))}, JSON.stringify({kind,pid:process.pid,...fields}) + '\\n');
const chunk = ${JSON.stringify(CHUNK)};
let turn = 0;
const cwds = new Map();
require('node:readline').createInterface({ input: process.stdin }).on('line', (line) => {
const msg = JSON.parse(line);
const p = msg.params || {};
if (msg.method === 'initialize') return emit({ id: msg.id, result: { userAgent: 'codex-stream/0.163.0' } });
if (msg.method === 'initialized') return;
if (msg.method === 'account/read') return emit({ id: msg.id, result: { account: { type: 'chatgpt' }, requiresOpenaiAuth: true } });
if (msg.method === 'model/list') return emit({ id: msg.id, result: { data: [{ id: 'fixture', model: 'fixture', displayName: 'Fixture', isDefault: true }], nextCursor: null } });
if (msg.method === 'thread/start' || msg.method === 'thread/resume') {
const id = p.threadId || 'thr-' + cwds.size;
cwds.set(id, p.cwd || '');
return emit({ id: msg.id, result: { thread: { id } } });
}
if (msg.method === 'turn/start') {
const id = 'turn-' + (++turn);
const n = (method, params) => emit({ method, params: { threadId: p.threadId, turnId: id, ...params } });
emit({ id: msg.id, result: { turn: { id } } });
if (/modex-title-/.test(cwds.get(p.threadId) || '')) {
audit('title-start', {threadId:p.threadId,turnId:id});
n('item/completed', { item: { id: 'name-' + turn, type: 'agentMessage', text: 'Stream', phase: 'final_answer' } });
return emit({ method: 'turn/completed', params: { threadId: p.threadId, turn: { id, status: 'completed', error: null } } });
}
audit('coding-start', {threadId:p.threadId,turnId:id});
emit({ method: 'turn/started', params: { threadId: p.threadId, turn: { id } } });
const command = "/bin/zsh -lc 'yes | head -c 3145728'";
n('item/started', { item: { type: 'commandExecution', id: 'c1', command, cwd: '/repo', status: 'inProgress' } });
let sent = 0;
const next = () => {
if (sent < ${CHUNKS}) { sent++; n('item/commandExecution/outputDelta', { itemId: 'c1', delta: chunk }); return setTimeout(next, 10); }
audit('coding-finish', {threadId:p.threadId,turnId:id,deltas:sent,rawBytes:Buffer.byteLength(chunk)*sent});
n('item/completed', { item: { type: 'commandExecution', id: 'c1', command, cwd: '/repo', status: 'completed', exitCode: 0, aggregatedOutput: chunk.repeat(${CHUNKS}) } });
n('item/completed', { item: { type: 'agentMessage', id: 'm1', text: 'Streamed the log.', phase: 'final_answer' } });
emit({ method: 'turn/completed', params: { threadId: p.threadId, turn: { id, status: 'completed', error: null } } });
};
return setTimeout(next, 10);
}
if (msg.id !== undefined && msg.method) emit({ id: msg.id, error: { code: -32601, message: 'fixture does not implement ' + msg.method } });
});
`, { mode: 0o755 });
return cli;
}

10 changes: 10 additions & 0 deletions apps/desktop/bench/retention/playwright.config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
import { defineConfig } from "@playwright/test";
import path from "node:path";
const output = process.env.MODEX_RETENTION_OUTPUT;
if (!output || !path.isAbsolute(output)) throw new Error("An explicit private MODEX_RETENTION_OUTPUT is required.");
export default defineConfig({
testDir: ".", testMatch: ["history.spec.ts", "stream.spec.ts"], workers: 1,
timeout: 180_000, expect: { timeout: 15_000 }, retries: 0, maxFailures: 1,
reporter: [["list"], ["json", { outputFile: path.join(output, "report.json") }]],
outputDir: path.join(output, "artifacts"), use: { trace: "off" },
});
Loading
Loading