2233admin · Oct 25, 2026 · Oct 25, 2026 · Oct 25, 2026 · Oct 25, 2026 · coderabbitai
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -3,7 +3,7 @@
 # CLAUDE.md — full-self-coding (FSC)
 
 ## 项目概述
-FSC 核心引擎: 把代码任务拆解成原子级子任务，分发给多个 Agent 并行执行 (目标 100~1000, 尚无压测数据, 见 issue #10; 现状默认并发 5, 合并串行)。
+FSC 核心引擎: 把代码任务拆解成原子级子任务，分发给多个 Agent 并行执行 (mock agent 实测 1000 个独立任务可跑完, 见 docs/reports/scale-bench.md; 真实 agent 的内存/限流/费用未测; 默认并发 5, 合并串行)。
 类比 GPU 并行计算 — CUDA Cores 换成 AI Agent 容器。
 
 ## 架构

diff --git a/README.md b/README.md
@@ -6,7 +6,7 @@
 
 Run many AI coding agents in parallel on one codebase, each in an isolated worktree, with scope contracts to keep their changes from colliding.
 
-> **Scale**: the scheduler defaults to 5 concurrent agents (worktree pool of 24) and the merge step is serial. Running hundreds or thousands of agents is a *goal*, not a measured capability; see [#10](https://github.com/2233admin/full-self-coding/issues/10).
+> **Scale**: defaults to 5 concurrent agents; merging is serial. With mock agents on one machine, 1000 independent tasks ran to completion in 7.7 minutes, and orchestration should not limit you below roughly 200 simultaneous agents. Real-agent memory, model rate limits and cost, and large repositories are not measured. Details and caveats: [docs/reports/scale-bench.md](docs/reports/scale-bench.md) (issue [#10](https://github.com/2233admin/full-self-coding/issues/10)).
 
 ## 🌟 Overview
 

diff --git a/docs/reports/scale-bench.md b/docs/reports/scale-bench.md
@@ -0,0 +1,74 @@
+# Scale benchmark: native worktree pipeline with mock agents
+
+Date: 2026-10-26. Issue: #10. Script: `script/bench/scale.ts`.
+
+## What this measures, and what it does not
+
+Measured: the orchestration layer only, `UnifiedScheduler` -> `LocalExecution` (one git worktree + branch per task) -> `CommitSequencer` (serial cherry-pick onto the base branch), with a mock agent that writes one file (`bash -c`) and optionally sleeps. No model is called.
+
+Not measured: model latency, model rate limits or cost, agent memory, real repositories.
+
+Setup: one Windows Server 2025 machine, local NTFS, repository of 201 small files (about 40 KB of working tree), `git` on PATH. Other sessions were running on the same machine, so absolute numbers carry noise. Reproduce:
+
+```
+bun script/bench/scale.ts --n 200 --concurrency 16 --overlap none --agent-ms 0
+```
+
+## Results
+
+All runs finished with `executionFailed = 0`, no leftover worktrees, and (for `overlap=none`) every commit merged.
+
+### Independent tasks (each task writes its own file), mock agent 0 ms
+
+| tasks | concurrency | total | exec | merge | cleanup | tasks/min | build |
+|---:|---:|---:|---:|---:|---:|---:|---|
+| 50 | 5 | 18.6 s | 12.9 s | 4.6 s | 1.1 s | 161 | after fix |
+| 200 | 5 | 86.5 s | 63.0 s | 18.5 s | 5.0 s | 139 | after fix |
+| 200 | 16 | 80.8 s | 57.5 s | 18.7 s | 4.6 s | 148 | after fix |
+| 1000 | 16 | 461.9 s | 325.3 s | 111.7 s | 24.9 s | 130 | after fix |
+| 200 | 16 | 113.8 s | 66.3 s | 28.1 s | 19.4 s | 105 | before fix |
+| 1000 | 16 | 1206.2 s | 744.1 s | 110.4 s | 351.6 s | 50 | before fix |
+
+"Fix" = worktrees are removed as soon as a task finishes. Before it, every worktree stayed alive until the end of the run (1000 live checkouts at 1000 tasks) and the end-of-run cleanup took 352 s; after it, 25 s.
+
+### With simulated agent work (5 s per task)
+
+| tasks | concurrency | total | exec | merge |
+|---:|---:|---:|---:|---:|
+| 100 | 5 | 119.9 s | 107.7 s | 9.9 s |
+| 100 | 16 | 49.2 s | 38.0 s | 8.9 s |
+| 200 | 16 | 96.1 s | 71.4 s | 19.3 s |
+
+Ideal exec time is waves x 5 s: 100 s, 35 s and 65 s. Measured is within about 10 % of that, so with a 5 s agent the pipeline is agent-bound and scales with concurrency.
+
+### Tasks that all rewrite the same line of one file (worst-case overlap), 0 ms agent (before the cleanup fix)
+
+| tasks | concurrency | total | merged | conflicts | branches kept for recovery |
+|---:|---:|---:|---:|---:|---:|
+| 50 | 5 | 64.8 s | 1 | 49 | 49 |
+| 200 | 16 | 136.1 s | 1 | 199 | 199 |
+
+One task wins, every other task is reported as `MERGE_CONFLICT` and kept on its `fsc/<task>` branch. Each conflict costs about 130 to 320 ms in the merge step. Nothing is lost, but 99.5 % of the agent work in this scenario is wasted, and FSC does nothing to avoid it (conflict-aware scheduling is #6).
+
+## Findings
+
+1. **Starting tasks is serialized.** `WorktreeManager.create` runs under one mutex and takes about 300 ms here (a `git worktree add` of 201 files). Exec time per task is about 270 to 325 ms whether concurrency is 5 or 16 (rows 1 to 4). That caps task starts at roughly 190 per minute on this machine and repository. With real agents that run for tens of seconds the cap is hidden: a 60 s agent only reaches it at about 60 s / 0.3 s = 200 simultaneous agents.
+2. **Merging is serial and linear**: about 90 to 110 ms per commit, about 9 per second, 112 s for 1000 commits. It does not grow with the number of tasks. At 200 simultaneous 60 s agents it would need about 3 commits per second, far under the cap.
+3. **Cleanup was the superlinear cost** (22 ms per task at 50, 350 ms per task at 1000) because all worktrees were alive at once. Fixed.
+4. **Scheduler memory is small**: 35 to 52 MB for the Bun process at 50 to 1000 tasks. Agent processes are not included.
+5. **Throughput of useful work collapses under overlap** (above). The pipeline is correct under conflicts, but not efficient.
+
+## Answer to "how many agents can it run"
+
+On this machine, with a small repository and mock agents, the orchestration layer ran **1000 independent tasks to completion with no failures** in 7.7 minutes, and would not be the bottleneck for up to roughly **200 simultaneous real agents** (limits 1 and 2 above). It was never tested with more than 16 concurrent agents, and the default is 5 (`maxParallelWorktrees`).
+
+What actually limits "thousands of agents in parallel" is outside this benchmark and still unmeasured:
+
+- **Memory**: `AGENT_CATALOG` assumes 50 to 200 MB per agent; 200 Claude-class agents is on the order of 40 GB.
+- **Model API rate limits and cost**: nothing in FSC currently caps spend (the README mentions token flow control; the code has none). Tracked in #6.
+- **Real repositories**: `git worktree add` time grows with checkout size. A 50,000-file repository will make finding 1 much worse.
+- **Overlap**: realistic task sets overlap far less than the worst case above, but how much is unknown.
+
+## Not covered
+
+Multi-machine runs (the Redis path), Docker mode, repositories larger than 201 files, anything above 16 concurrent agents, partial overlap (some tasks sharing files).
diff --git a/packages/core/src/execution/scheduler.ts b/packages/core/src/execution/scheduler.ts
@@ -8,7 +8,7 @@ import { toWorkUnitByTier, toWorkUnitForAgent, toTaskResult } from "./mapping";
 import { gitExec } from "./gitExec";
 import type { Task, TaskResult } from "../task";
 import type { CostTierStrategy } from "../modelRouter";
-import type { ExecutionResult } from "./types";
+import type { ExecutionResult, WorkUnit } from "./types";
 import { SalaciaPlugin, type SalaciaConfig } from "../salacia/plugin";
 import { ABExperiment, type ExperimentMetrics } from "../salacia/experiment";
 
@@ -18,6 +18,8 @@ export interface SchedulerConfig {
   costStrategy: CostTierStrategy;  // default "minimize-cost"
   /** Run this agent for every task instead of cost-tier routing (e.g. "claude-code"). */
   agentType?: string;
+  /** Build the work unit yourself (benchmarks and tests inject a mock agent here). */
+  unitFactory?: (task: Task, repoPath: string) => WorkUnit;
   freeclawBase?: string;
   salacia?: SalaciaConfig & {
     experiment?: { enabled: boolean; splitRatio: number };
@@ -33,6 +35,12 @@ export interface SchedulerReport {
   tasks: TaskState[];
   mergeResults: SequencerResult[];
   durationMs: number;
+  /** Time spent running agents (all tasks, wall clock). */
+  execMs?: number;
+  /** Time spent merging the finished commits onto the base branch. */
+  mergeMs?: number;
+  /** Time spent removing task worktrees and branches. */
+  cleanupMs?: number;
   zeroRegressionRate?: number;  // SWE-CI inspired: fraction of merges where ALL pre-existing tests still pass
 }
 
@@ -53,6 +61,7 @@ export class UnifiedScheduler {
   private config: SchedulerConfig;
   private stateStore: TaskStateStore;
   private pool: WorktreePool | null = null;
+  private worktrees: WorktreeManager;
   private concurrency: ConcurrencyPool;
   private execution: LocalExecution;
   private repoPath: string = "";
@@ -62,7 +71,8 @@ export class UnifiedScheduler {
     this.config = { ...DEFAULT_CONFIG, ...config };
     this.stateStore = new TaskStateStore();
     this.concurrency = new ConcurrencyPool(this.config.maxConcurrency);
-    this.execution = new LocalExecution(new WorktreeManager());
+    this.worktrees = new WorktreeManager();
+    this.execution = new LocalExecution(this.worktrees);
   }
 
   async execute(tasks: Task[], repoPath: string): Promise<SchedulerReport> {
@@ -88,6 +98,7 @@ export class UnifiedScheduler {
     // 4. Execute with concurrency control
     const executionResults: ExecutionResult[] = [];
 
+    const execStart = Date.now();
     const execPromises = tasks.map((task) =>
       this.concurrency.run(async () => {
         try {
@@ -103,7 +114,9 @@ export class UnifiedScheduler {
           this.stateStore.transition(task.ID, "dispatched");
 
           // Build work unit (with optional salacia enrichment)
-          const unit = this.config.agentType
+          const unit = this.config.unitFactory
+            ? this.config.unitFactory(task, repoPath)
+            : this.config.agentType
             ? toWorkUnitForAgent(task, this.config.agentType, repoPath, plugin)
             : toWorkUnitByTier(task, repoPath, this.config.costStrategy, this.config.freeclawBase, plugin);
           // LocalExecution creates the worktree under repoPath
@@ -122,6 +135,10 @@ export class UnifiedScheduler {
 
           executionResults.push(result);
 
+          // The commit lives on the task branch; free the checkout now so live worktrees stay
+          // bounded by the concurrency limit instead of growing to the task count.
+          await this.worktrees.removeWorktree(result.worktreePath, repoPath);
+
           // Record A/B metrics
           if (experiment) {
             const contract = plugin.getContract(task.ID);
@@ -154,10 +171,13 @@ export class UnifiedScheduler {
     );
 
     await Promise.allSettled(execPromises);
+    const execMs = Date.now() - execStart;
 
     // 4. Merge via sequencer
     const sequencer = new CommitSequencer(repoPath, this.baseBranch);
+    const mergeStart = Date.now();
     const mergeResults = await sequencer.applyAll(executionResults);
+    const mergeMs = Date.now() - mergeStart;
 
     // Post-merge regression test (SWE-CI Zero-Regression Rate)
     // MEMO-inspired: failed merges are logged to an inverse-frequency replay buffer
@@ -196,13 +216,15 @@ export class UnifiedScheduler {
     }
 
     // 5. Cleanup: task worktrees always; task branches only once their commit is on the base branch
+    const cleanupStart = Date.now();
     for (const er of executionResults) {
       gitExec(["worktree", "remove", "--force", er.worktreePath], repoPath);
       if (mergeResults.some(m => m.taskId === er.workUnitId && m.success)) {
         gitExec(["branch", "-D", er.branch], repoPath);
       }
     }
 
+    const cleanupMs = Date.now() - cleanupStart;
     const durationMs = Date.now() - startTime;
     const summary = this.stateStore.summary();
 
@@ -214,6 +236,9 @@ export class UnifiedScheduler {
       tasks: this.stateStore.list(),
       mergeResults,
       durationMs,
+      execMs,
+      mergeMs,
+      cleanupMs,
       zeroRegressionRate: mergedCount > 0 ? zeroRegressionCount / mergedCount : undefined,
     };
   }

diff --git a/packages/core/src/execution/worktree.ts b/packages/core/src/execution/worktree.ts
@@ -89,6 +89,19 @@ export class WorktreeManager {
     }
   }
 
+  /**
+   * Remove only the worktree directory. The branch (and the commits on it) stays, so a finished
+   * task can free its checkout right away and still be merged or recovered later.
+   */
+  async removeWorktree(worktreePath: string, repoPath: string): Promise<void> {
+    await this.mutex.acquire(30_000);
+    try {
+      gitExec(["worktree", "remove", "--force", path.resolve(worktreePath)], path.resolve(repoPath));
+    } finally {
+      this.mutex.release();
+    }
+  }
+
   /**
    * Remove a worktree and its branch.
    */

diff --git a/script/bench/scale.ts b/script/bench/scale.ts
@@ -0,0 +1,130 @@
+// Scale benchmark for the native pipeline: UnifiedScheduler -> LocalExecution (worktree per task)
+// -> CommitSequencer, with a mock agent that makes no model calls.
+//
+//   bun script/bench/scale.ts --n 50 --concurrency 5 --overlap none
+//
+// Flags
+//   --n <int>            number of tasks (default 10)
+//   --concurrency <int>  agents running at once (default 5)
+//   --overlap <mode>     none: every task writes its own file.
+//                        same-line: every task rewrites the same line of shared.txt.
+//   --agent-ms <int>     simulated agent work per task, via sleep (default 0)
+//   --files <int>        extra seed files in the repository (default 200)
+//   --root <dir>         scratch directory (default D:/tmp/fsc-bench on this host, else the OS temp dir)
+//   --keep               do not delete the scratch repository
+//
+// Prints one JSON object. This measures git, scheduling and merging, not model latency or cost.
+
+import { mkdirSync, rmSync, writeFileSync, existsSync } from "fs";
+import { join } from "path";
+import { tmpdir } from "os";
+import { UnifiedScheduler } from "../../packages/core/src/execution/scheduler";
+import { gitExec } from "../../packages/core/src/execution/gitExec";
+import type { WorkUnit } from "../../packages/core/src/execution/types";
+import type { Task } from "../../packages/core/src/task";
+
+function arg(name: string, def: string): string {
+  const i = process.argv.indexOf(`--${name}`);
+  return i > -1 && process.argv[i + 1] ? process.argv[i + 1] : def;
+}
+const N = parseInt(arg("n", "10"));
+const CONC = parseInt(arg("concurrency", "5"));
+const OVERLAP = arg("overlap", "none");
+const AGENT_MS = parseInt(arg("agent-ms", "0"));
+const FILES = parseInt(arg("files", "200"));
+const KEEP = process.argv.includes("--keep");
+const ROOT = arg("root", existsSync("D:/tmp") ? "D:/tmp/fsc-bench" : join(tmpdir(), "fsc-bench"));
+
+if (!["none", "same-line"].includes(OVERLAP)) throw new Error(`--overlap must be none or same-line`);
+
+const dir = join(ROOT, `n${N}-c${CONC}-${OVERLAP}-${Date.now()}`);
+const repo = join(dir, "repo");
+mkdirSync(repo, { recursive: true });
+const git = (...a: string[]) => {
+  const r = gitExec(a, repo);
+  if (r.exitCode !== 0) throw new Error(`git ${a.join(" ")}: ${r.stderr}`);
+  return r.stdout;
+};
+
+git("init", "-q", "-b", "main");
+git("config", "user.name", "bench");
+git("config", "user.email", "bench@local");
+git("config", "core.autocrlf", "false");
+writeFileSync(join(repo, "shared.txt"), "VALUE=base\n");
+mkdirSync(join(repo, "seed"), { recursive: true });
+for (let i = 0; i < FILES; i++) writeFileSync(join(repo, "seed", `f${i}.txt`), `seed file ${i}\n`.repeat(20));
+git("add", "-A");
+git("commit", "-q", "-m", "seed");
+
+const sleep = AGENT_MS > 0 ? `sleep ${(AGENT_MS / 1000).toFixed(3)}; ` : "";
+const script =
+  OVERLAP === "none"
+    ? `${sleep}echo "$FSC_TASK_ID" > "out_$FSC_TASK_ID.txt"`
+    : `${sleep}printf 'VALUE=%s\\n' "$FSC_TASK_ID" > shared.txt`;
+
+const tasks: Task[] = Array.from({ length: N }, (_, i) => ({
+  ID: `t${String(i).padStart(4, "0")}`,
+  title: `task ${i}`,
+  description: "mock",
+  priority: 1,
+}));
+
+const unitFactory = (task: Task, repoPath: string): WorkUnit => ({
+  id: task.ID,
+  repoPath,
+  baseBranch: "main",
+  taskPrompt: task.title,
+  agentConfig: {
+    type: "custom",
+    command: "bash",
+    buildArgs: () => ["-c", script],
+    costTier: "free",
+    memoryMB: 50,
+  },
+  constraints: { timeoutMs: 120_000 },
+});
+
+const realLog = console.log;
+console.log = () => {}; // silence per-task chatter; the result is printed below
+const t0 = Date.now();
+const scheduler = new UnifiedScheduler({ localPoolSize: CONC, maxConcurrency: CONC, unitFactory });
+const report = await scheduler.execute(tasks, repo);
+const totalMs = Date.now() - t0;
+console.log = realLog;
+
+const merged = report.mergeResults.filter((m) => m.success && m.resolvedTier).length;
+const conflicts = report.mergeResults.filter((m) => !m.success && (m.error ?? "").includes("MERGE_CONFLICT")).length;
+const otherMergeFailures = report.mergeResults.filter((m) => !m.success && !(m.error ?? "").includes("MERGE_CONFLICT")).length;
+const rss = (process.resourceUsage() as { maxRSS?: number }).maxRSS;
+const leftoverWorktrees = git("worktree", "list").trim().split("\n").length - 1;
+const leftoverBranches = git("branch", "--list", "fsc/*").trim().split("\n").filter(Boolean).length;
+
+console.log(
+  JSON.stringify(
+    {
+      n: N,
+      concurrency: CONC,
+      overlap: OVERLAP,
+      agentMs: AGENT_MS,
+      seedFiles: FILES,
+      totalMs,
+      execMs: report.execMs,
+      mergeMs: report.mergeMs,
+      cleanupMs: report.cleanupMs,
+      executed: report.completed,
+      executionFailed: report.failed,
+      merged,
+      conflicts,
+      otherMergeFailures,
+      msPerMerge: merged > 0 && report.mergeMs ? Math.round(report.mergeMs / merged) : null,
+      tasksPerMinute: Math.round((N / totalMs) * 60_000),
+      schedulerMaxRssMB: rss ? Math.round(rss / 1024) : null,
+      leftoverWorktrees,
+      leftoverBranches,
+    },
+    null,
+    2,
+  ),
+);
+
+if (!KEEP) rmSync(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 300 });