From 54ad4da666b679aa6e10b6470e02383ad3300963 Mon Sep 17 00:00:00 2001 From: k-atana-agent Date: Mon, 26 Oct 2026 01:13:13 +0800 Subject: [PATCH] bench: scale benchmark script + report; free task worktrees as soon as they finish - script/bench/scale.ts: mock-agent benchmark over the real UnifiedScheduler (tasks, concurrency, overlap none|same-line, simulated agent time); SchedulerConfig.unitFactory is the injection point; the report now carries execMs / mergeMs / cleanupMs - docs/reports/scale-bench.md: results for 50/200/1000 tasks and the bottlenecks found - scheduler: remove each task's worktree right after the task finishes (branch and commit stay), instead of keeping every worktree alive until the end. 1000 tasks: cleanup 352 s -> 25 s, total 1206 s -> 462 s - README / CLAUDE.md: scale statement now cites measured numbers and what is not measured Closes #10 Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_016FSVfwwP6TVxGLUWatWM53 --- CLAUDE.md | 2 +- README.md | 2 +- docs/reports/scale-bench.md | 74 +++++++++++++ packages/core/src/execution/scheduler.ts | 31 +++++- packages/core/src/execution/worktree.ts | 13 +++ script/bench/scale.ts | 130 +++++++++++++++++++++++ 6 files changed, 247 insertions(+), 5 deletions(-) create mode 100644 docs/reports/scale-bench.md create mode 100644 script/bench/scale.ts diff --git a/CLAUDE.md b/CLAUDE.md index 9e035ef..2988746 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -3,7 +3,7 @@ # CLAUDE.md — full-self-coding (FSC) ## 项目概述 -FSC 核心引擎: 把代码任务拆解成原子级子任务，分发给多个 Agent 并行执行 (目标 100~1000, 尚无压测数据, 见 issue #10; 现状默认并发 5, 合并串行)。 +FSC 核心引擎: 把代码任务拆解成原子级子任务，分发给多个 Agent 并行执行 (mock agent 实测 1000 个独立任务可跑完, 见 docs/reports/scale-bench.md; 真实 agent 的内存/限流/费用未测; 默认并发 5, 合并串行)。 类比 GPU 并行计算 — CUDA Cores 换成 AI Agent 容器。 ## 架构 diff --git a/README.md b/README.md index cabb488..9dd8066 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Run many AI coding agents in parallel on one codebase, each in an isolated worktree, with scope contracts to keep their changes from colliding. -> **Scale**: the scheduler defaults to 5 concurrent agents (worktree pool of 24) and the merge step is serial. Running hundreds or thousands of agents is a *goal*, not a measured capability; see [#10](https://github.com/2233admin/full-self-coding/issues/10). +> **Scale**: defaults to 5 concurrent agents; merging is serial. With mock agents on one machine, 1000 independent tasks ran to completion in 7.7 minutes, and orchestration should not limit you below roughly 200 simultaneous agents. Real-agent memory, model rate limits and cost, and large repositories are not measured. Details and caveats: [docs/reports/scale-bench.md](docs/reports/scale-bench.md) (issue [#10](https://github.com/2233admin/full-self-coding/issues/10)). ## 🌟 Overview diff --git a/docs/reports/scale-bench.md b/docs/reports/scale-bench.md new file mode 100644 index 0000000..be0b7fc --- /dev/null +++ b/docs/reports/scale-bench.md @@ -0,0 +1,74 @@ +# Scale benchmark: native worktree pipeline with mock agents + +Date: 2026-10-26. Issue: #10. Script: `script/bench/scale.ts`. + +## What this measures, and what it does not + +Measured: the orchestration layer only, `UnifiedScheduler` -> `LocalExecution` (one git worktree + branch per task) -> `CommitSequencer` (serial cherry-pick onto the base branch), with a mock agent that writes one file (`bash -c`) and optionally sleeps. No model is called. + +Not measured: model latency, model rate limits or cost, agent memory, real repositories. + +Setup: one Windows Server 2025 machine, local NTFS, repository of 201 small files (about 40 KB of working tree), `git` on PATH. Other sessions were running on the same machine, so absolute numbers carry noise. Reproduce: + +``` +bun script/bench/scale.ts --n 200 --concurrency 16 --overlap none --agent-ms 0 +``` + +## Results + +All runs finished with `executionFailed = 0`, no leftover worktrees, and (for `overlap=none`) every commit merged. + +### Independent tasks (each task writes its own file), mock agent 0 ms + +| tasks | concurrency | total | exec | merge | cleanup | tasks/min | build | +|---:|---:|---:|---:|---:|---:|---:|---| +| 50 | 5 | 18.6 s | 12.9 s | 4.6 s | 1.1 s | 161 | after fix | +| 200 | 5 | 86.5 s | 63.0 s | 18.5 s | 5.0 s | 139 | after fix | +| 200 | 16 | 80.8 s | 57.5 s | 18.7 s | 4.6 s | 148 | after fix | +| 1000 | 16 | 461.9 s | 325.3 s | 111.7 s | 24.9 s | 130 | after fix | +| 200 | 16 | 113.8 s | 66.3 s | 28.1 s | 19.4 s | 105 | before fix | +| 1000 | 16 | 1206.2 s | 744.1 s | 110.4 s | 351.6 s | 50 | before fix | + +"Fix" = worktrees are removed as soon as a task finishes. Before it, every worktree stayed alive until the end of the run (1000 live checkouts at 1000 tasks) and the end-of-run cleanup took 352 s; after it, 25 s. + +### With simulated agent work (5 s per task) + +| tasks | concurrency | total | exec | merge | +|---:|---:|---:|---:|---:| +| 100 | 5 | 119.9 s | 107.7 s | 9.9 s | +| 100 | 16 | 49.2 s | 38.0 s | 8.9 s | +| 200 | 16 | 96.1 s | 71.4 s | 19.3 s | + +Ideal exec time is waves x 5 s: 100 s, 35 s and 65 s. Measured is within about 10 % of that, so with a 5 s agent the pipeline is agent-bound and scales with concurrency. + +### Tasks that all rewrite the same line of one file (worst-case overlap), 0 ms agent (before the cleanup fix) + +| tasks | concurrency | total | merged | conflicts | branches kept for recovery | +|---:|---:|---:|---:|---:|---:| +| 50 | 5 | 64.8 s | 1 | 49 | 49 | +| 200 | 16 | 136.1 s | 1 | 199 | 199 | + +One task wins, every other task is reported as `MERGE_CONFLICT` and kept on its `fsc/` branch. Each conflict costs about 130 to 320 ms in the merge step. Nothing is lost, but 99.5 % of the agent work in this scenario is wasted, and FSC does nothing to avoid it (conflict-aware scheduling is #6). + +## Findings + +1. **Starting tasks is serialized.** `WorktreeManager.create` runs under one mutex and takes about 300 ms here (a `git worktree add` of 201 files). Exec time per task is about 270 to 325 ms whether concurrency is 5 or 16 (rows 1 to 4). That caps task starts at roughly 190 per minute on this machine and repository. With real agents that run for tens of seconds the cap is hidden: a 60 s agent only reaches it at about 60 s / 0.3 s = 200 simultaneous agents. +2. **Merging is serial and linear**: about 90 to 110 ms per commit, about 9 per second, 112 s for 1000 commits. It does not grow with the number of tasks. At 200 simultaneous 60 s agents it would need about 3 commits per second, far under the cap. +3. **Cleanup was the superlinear cost** (22 ms per task at 50, 350 ms per task at 1000) because all worktrees were alive at once. Fixed. +4. **Scheduler memory is small**: 35 to 52 MB for the Bun process at 50 to 1000 tasks. Agent processes are not included. +5. **Throughput of useful work collapses under overlap** (above). The pipeline is correct under conflicts, but not efficient. + +## Answer to "how many agents can it run" + +On this machine, with a small repository and mock agents, the orchestration layer ran **1000 independent tasks to completion with no failures** in 7.7 minutes, and would not be the bottleneck for up to roughly **200 simultaneous real agents** (limits 1 and 2 above). It was never tested with more than 16 concurrent agents, and the default is 5 (`maxParallelWorktrees`). + +What actually limits "thousands of agents in parallel" is outside this benchmark and still unmeasured: + +- **Memory**: `AGENT_CATALOG` assumes 50 to 200 MB per agent; 200 Claude-class agents is on the order of 40 GB. +- **Model API rate limits and cost**: nothing in FSC currently caps spend (the README mentions token flow control; the code has none). Tracked in #6. +- **Real repositories**: `git worktree add` time grows with checkout size. A 50,000-file repository will make finding 1 much worse. +- **Overlap**: realistic task sets overlap far less than the worst case above, but how much is unknown. + +## Not covered + +Multi-machine runs (the Redis path), Docker mode, repositories larger than 201 files, anything above 16 concurrent agents, partial overlap (some tasks sharing files). diff --git a/packages/core/src/execution/scheduler.ts b/packages/core/src/execution/scheduler.ts index 83cb559..ba77f71 100644 --- a/packages/core/src/execution/scheduler.ts +++ b/packages/core/src/execution/scheduler.ts @@ -8,7 +8,7 @@ import { toWorkUnitByTier, toWorkUnitForAgent, toTaskResult } from "./mapping"; import { gitExec } from "./gitExec"; import type { Task, TaskResult } from "../task"; import type { CostTierStrategy } from "../modelRouter"; -import type { ExecutionResult } from "./types"; +import type { ExecutionResult, WorkUnit } from "./types"; import { SalaciaPlugin, type SalaciaConfig } from "../salacia/plugin"; import { ABExperiment, type ExperimentMetrics } from "../salacia/experiment"; @@ -18,6 +18,8 @@ export interface SchedulerConfig { costStrategy: CostTierStrategy; // default "minimize-cost" /** Run this agent for every task instead of cost-tier routing (e.g. "claude-code"). */ agentType?: string; + /** Build the work unit yourself (benchmarks and tests inject a mock agent here). */ + unitFactory?: (task: Task, repoPath: string) => WorkUnit; freeclawBase?: string; salacia?: SalaciaConfig & { experiment?: { enabled: boolean; splitRatio: number }; @@ -33,6 +35,12 @@ export interface SchedulerReport { tasks: TaskState[]; mergeResults: SequencerResult[]; durationMs: number; + /** Time spent running agents (all tasks, wall clock). */ + execMs?: number; + /** Time spent merging the finished commits onto the base branch. */ + mergeMs?: number; + /** Time spent removing task worktrees and branches. */ + cleanupMs?: number; zeroRegressionRate?: number; // SWE-CI inspired: fraction of merges where ALL pre-existing tests still pass } @@ -53,6 +61,7 @@ export class UnifiedScheduler { private config: SchedulerConfig; private stateStore: TaskStateStore; private pool: WorktreePool | null = null; + private worktrees: WorktreeManager; private concurrency: ConcurrencyPool; private execution: LocalExecution; private repoPath: string = ""; @@ -62,7 +71,8 @@ export class UnifiedScheduler { this.config = { ...DEFAULT_CONFIG, ...config }; this.stateStore = new TaskStateStore(); this.concurrency = new ConcurrencyPool(this.config.maxConcurrency); - this.execution = new LocalExecution(new WorktreeManager()); + this.worktrees = new WorktreeManager(); + this.execution = new LocalExecution(this.worktrees); } async execute(tasks: Task[], repoPath: string): Promise { @@ -88,6 +98,7 @@ export class UnifiedScheduler { // 4. Execute with concurrency control const executionResults: ExecutionResult[] = []; + const execStart = Date.now(); const execPromises = tasks.map((task) => this.concurrency.run(async () => { try { @@ -103,7 +114,9 @@ export class UnifiedScheduler { this.stateStore.transition(task.ID, "dispatched"); // Build work unit (with optional salacia enrichment) - const unit = this.config.agentType + const unit = this.config.unitFactory + ? this.config.unitFactory(task, repoPath) + : this.config.agentType ? toWorkUnitForAgent(task, this.config.agentType, repoPath, plugin) : toWorkUnitByTier(task, repoPath, this.config.costStrategy, this.config.freeclawBase, plugin); // LocalExecution creates the worktree under repoPath @@ -122,6 +135,10 @@ export class UnifiedScheduler { executionResults.push(result); + // The commit lives on the task branch; free the checkout now so live worktrees stay + // bounded by the concurrency limit instead of growing to the task count. + await this.worktrees.removeWorktree(result.worktreePath, repoPath); + // Record A/B metrics if (experiment) { const contract = plugin.getContract(task.ID); @@ -154,10 +171,13 @@ export class UnifiedScheduler { ); await Promise.allSettled(execPromises); + const execMs = Date.now() - execStart; // 4. Merge via sequencer const sequencer = new CommitSequencer(repoPath, this.baseBranch); + const mergeStart = Date.now(); const mergeResults = await sequencer.applyAll(executionResults); + const mergeMs = Date.now() - mergeStart; // Post-merge regression test (SWE-CI Zero-Regression Rate) // MEMO-inspired: failed merges are logged to an inverse-frequency replay buffer @@ -196,6 +216,7 @@ export class UnifiedScheduler { } // 5. Cleanup: task worktrees always; task branches only once their commit is on the base branch + const cleanupStart = Date.now(); for (const er of executionResults) { gitExec(["worktree", "remove", "--force", er.worktreePath], repoPath); if (mergeResults.some(m => m.taskId === er.workUnitId && m.success)) { @@ -203,6 +224,7 @@ export class UnifiedScheduler { } } + const cleanupMs = Date.now() - cleanupStart; const durationMs = Date.now() - startTime; const summary = this.stateStore.summary(); @@ -214,6 +236,9 @@ export class UnifiedScheduler { tasks: this.stateStore.list(), mergeResults, durationMs, + execMs, + mergeMs, + cleanupMs, zeroRegressionRate: mergedCount > 0 ? zeroRegressionCount / mergedCount : undefined, }; } diff --git a/packages/core/src/execution/worktree.ts b/packages/core/src/execution/worktree.ts index 0c1165d..8ad292e 100644 --- a/packages/core/src/execution/worktree.ts +++ b/packages/core/src/execution/worktree.ts @@ -89,6 +89,19 @@ export class WorktreeManager { } } + /** + * Remove only the worktree directory. The branch (and the commits on it) stays, so a finished + * task can free its checkout right away and still be merged or recovered later. + */ + async removeWorktree(worktreePath: string, repoPath: string): Promise { + await this.mutex.acquire(30_000); + try { + gitExec(["worktree", "remove", "--force", path.resolve(worktreePath)], path.resolve(repoPath)); + } finally { + this.mutex.release(); + } + } + /** * Remove a worktree and its branch. */ diff --git a/script/bench/scale.ts b/script/bench/scale.ts new file mode 100644 index 0000000..8dee541 --- /dev/null +++ b/script/bench/scale.ts @@ -0,0 +1,130 @@ +// Scale benchmark for the native pipeline: UnifiedScheduler -> LocalExecution (worktree per task) +// -> CommitSequencer, with a mock agent that makes no model calls. +// +// bun script/bench/scale.ts --n 50 --concurrency 5 --overlap none +// +// Flags +// --n number of tasks (default 10) +// --concurrency agents running at once (default 5) +// --overlap none: every task writes its own file. +// same-line: every task rewrites the same line of shared.txt. +// --agent-ms simulated agent work per task, via sleep (default 0) +// --files extra seed files in the repository (default 200) +// --root scratch directory (default D:/tmp/fsc-bench on this host, else the OS temp dir) +// --keep do not delete the scratch repository +// +// Prints one JSON object. This measures git, scheduling and merging, not model latency or cost. + +import { mkdirSync, rmSync, writeFileSync, existsSync } from "fs"; +import { join } from "path"; +import { tmpdir } from "os"; +import { UnifiedScheduler } from "../../packages/core/src/execution/scheduler"; +import { gitExec } from "../../packages/core/src/execution/gitExec"; +import type { WorkUnit } from "../../packages/core/src/execution/types"; +import type { Task } from "../../packages/core/src/task"; + +function arg(name: string, def: string): string { + const i = process.argv.indexOf(`--${name}`); + return i > -1 && process.argv[i + 1] ? process.argv[i + 1] : def; +} +const N = parseInt(arg("n", "10")); +const CONC = parseInt(arg("concurrency", "5")); +const OVERLAP = arg("overlap", "none"); +const AGENT_MS = parseInt(arg("agent-ms", "0")); +const FILES = parseInt(arg("files", "200")); +const KEEP = process.argv.includes("--keep"); +const ROOT = arg("root", existsSync("D:/tmp") ? "D:/tmp/fsc-bench" : join(tmpdir(), "fsc-bench")); + +if (!["none", "same-line"].includes(OVERLAP)) throw new Error(`--overlap must be none or same-line`); + +const dir = join(ROOT, `n${N}-c${CONC}-${OVERLAP}-${Date.now()}`); +const repo = join(dir, "repo"); +mkdirSync(repo, { recursive: true }); +const git = (...a: string[]) => { + const r = gitExec(a, repo); + if (r.exitCode !== 0) throw new Error(`git ${a.join(" ")}: ${r.stderr}`); + return r.stdout; +}; + +git("init", "-q", "-b", "main"); +git("config", "user.name", "bench"); +git("config", "user.email", "bench@local"); +git("config", "core.autocrlf", "false"); +writeFileSync(join(repo, "shared.txt"), "VALUE=base\n"); +mkdirSync(join(repo, "seed"), { recursive: true }); +for (let i = 0; i < FILES; i++) writeFileSync(join(repo, "seed", `f${i}.txt`), `seed file ${i}\n`.repeat(20)); +git("add", "-A"); +git("commit", "-q", "-m", "seed"); + +const sleep = AGENT_MS > 0 ? `sleep ${(AGENT_MS / 1000).toFixed(3)}; ` : ""; +const script = + OVERLAP === "none" + ? `${sleep}echo "$FSC_TASK_ID" > "out_$FSC_TASK_ID.txt"` + : `${sleep}printf 'VALUE=%s\\n' "$FSC_TASK_ID" > shared.txt`; + +const tasks: Task[] = Array.from({ length: N }, (_, i) => ({ + ID: `t${String(i).padStart(4, "0")}`, + title: `task ${i}`, + description: "mock", + priority: 1, +})); + +const unitFactory = (task: Task, repoPath: string): WorkUnit => ({ + id: task.ID, + repoPath, + baseBranch: "main", + taskPrompt: task.title, + agentConfig: { + type: "custom", + command: "bash", + buildArgs: () => ["-c", script], + costTier: "free", + memoryMB: 50, + }, + constraints: { timeoutMs: 120_000 }, +}); + +const realLog = console.log; +console.log = () => {}; // silence per-task chatter; the result is printed below +const t0 = Date.now(); +const scheduler = new UnifiedScheduler({ localPoolSize: CONC, maxConcurrency: CONC, unitFactory }); +const report = await scheduler.execute(tasks, repo); +const totalMs = Date.now() - t0; +console.log = realLog; + +const merged = report.mergeResults.filter((m) => m.success && m.resolvedTier).length; +const conflicts = report.mergeResults.filter((m) => !m.success && (m.error ?? "").includes("MERGE_CONFLICT")).length; +const otherMergeFailures = report.mergeResults.filter((m) => !m.success && !(m.error ?? "").includes("MERGE_CONFLICT")).length; +const rss = (process.resourceUsage() as { maxRSS?: number }).maxRSS; +const leftoverWorktrees = git("worktree", "list").trim().split("\n").length - 1; +const leftoverBranches = git("branch", "--list", "fsc/*").trim().split("\n").filter(Boolean).length; + +console.log( + JSON.stringify( + { + n: N, + concurrency: CONC, + overlap: OVERLAP, + agentMs: AGENT_MS, + seedFiles: FILES, + totalMs, + execMs: report.execMs, + mergeMs: report.mergeMs, + cleanupMs: report.cleanupMs, + executed: report.completed, + executionFailed: report.failed, + merged, + conflicts, + otherMergeFailures, + msPerMerge: merged > 0 && report.mergeMs ? Math.round(report.mergeMs / merged) : null, + tasksPerMinute: Math.round((N / totalMs) * 60_000), + schedulerMaxRssMB: rss ? Math.round(rss / 1024) : null, + leftoverWorktrees, + leftoverBranches, + }, + null, + 2, + ), +); + +if (!KEEP) rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 300 });