Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions docs/reports/e2e-smoke-2026-10-25.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,18 @@ Change: `claude` is started with `--setting-sources project,local` (opt back in
- Trivial prompt, default vs isolated: 22 s vs 9 s. Default: the agent could recite the operator's tracker rules and the Stop hook; isolated: it knew neither.
- Same two-task repo as the first run, real `claude` agents: exec phase **20 s** (first run: 77 s). Both tasks merged at tier 1; `main` has the two implementations; agent logs contain no tracker or hook talk.
- **Correction:** this run took 15 min 28 s wall clock, not 20 s: analysis and execution finished within about a minute (file timestamps: 20 s + 27 s) and the process then idled for ~14 minutes. So the slow runs are not explained by a user-level hook (this run had them isolated). The cause is a different defect, found right after this report and fixed in a follow-up PR: the agent timeout timer was never cleared, so a successful run kept the process alive until the 15-minute timeout fired. Failing runs exited through `process.exit(1)` and never showed it. That also explains why only successful runs were slow.

## Fourth run: conflict retry, and the real cause of the 15-minute runs (2026-10-26)

### Conflict retry (`conflictRetries`, default 1)

Same three tasks as the second run, real `claude` agents, retry on:

- `double-c` and `greet-a` merged at tier 1. `greet-b` conflicted, was set aside as `fsc/greet-b-attempt1`, and was run again in a fresh worktree from the new `main` with a note naming `config.py`. The retry merged cleanly: **3/3 merged, 69 s**.
- **The outcome is not "both greetings applied".** `main` ended with `GREETING = "good day"`: the retry agent read its task as explicitly requiring that value and overwrote `greet-a`'s `"hi there"`. The two tasks contradict each other, and a clean merge does not mean both survived. The scheduler now lists such cases in `report.retryOverlaps` (retried tasks that touched a file another merged task also touched), and the CLI prints a WARNING for each. FSC cannot decide which task should win.
- Unit tests (mock agents): retry merges and cleans up the superseded branch; `conflictRetries: 0` keeps the old behavior; a retry that loses the race again is reported with both attempts kept.
- Cost: every retry is another agent run.

### The 15-minute runs

Two successful real runs had taken 16 min 10 s and 15 min 28 s although the work was done in about a minute. Cause: the per-agent timeout timer (15 min by default) was never cleared, so Bun kept the process alive until it fired. Runs that failed exited through `process.exit(1)` and never showed it. After clearing the timer, the same two-task run takes **1 min 18 s** total (analysis about 37 s, execution 40 s) and exits right after the report. A regression test starts a 1 s agent with a 10-minute timeout in a child process and requires it to exit within 30 s; it fails (60 s, killed) without the fix.
8 changes: 7 additions & 1 deletion packages/cli/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -112,9 +112,15 @@ export async function runFullAnalysis(): Promise<void> {
console.log('');
for (const t of report.tasks) {
const merged = report.mergeResults.find((m) => m.taskId === t.taskId);
const mergeNote = merged ? (merged.success ? ` (merged${merged.resolvedTier ? `, tier ${merged.resolvedTier}` : ''})` : ` (NOT merged: ${merged.error})`) : '';
const retried = report.retries?.[t.taskId];
const mergeNote = merged ? (merged.success ? ` (merged${retried ? ` after ${retried} retr${retried === 1 ? 'y' : 'ies'}` : ''}${merged.resolvedTier ? `, tier ${merged.resolvedTier}` : ''})` : ` (NOT merged: ${merged.error})`) : '';
console.log(` ${t.status === 'completed' && merged?.success ? 'ok ' : 'FAIL'} ${t.taskId}${mergeNote}${t.error ? ` - ${t.error}` : ''}`);
}
for (const [id, list] of Object.entries(report.retryOverlaps ?? {})) {
for (const o of list) {
console.log(` WARNING ${id} merged after a retry and changed files that ${o.task} also changed (${o.files.join(', ')}). Check that it kept ${o.task}'s change.`);
}
}
const merged = report.mergeResults.filter((m) => m.success && m.resolvedTier).length;
console.log(`
${report.completed}/${report.total} task(s) completed, ${merged} merged, ${report.failed} failed (${Math.round(report.durationMs / 1000)}s)`);
Expand Down
6 changes: 5 additions & 1 deletion packages/core/src/execution/local.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ export class LocalExecution implements ExecutionStrategy {
// 4. Spawn agent process
let timedOut = false;
let spawnError: string | null = null;
let timeoutTimer: ReturnType<typeof setTimeout> | undefined;
let exitCode: number | null = null;
let stdoutChunks: string[] = [];
let stderrChunks: string[] = [];
Expand All @@ -73,7 +74,7 @@ export class LocalExecution implements ExecutionStrategy {

// Race process completion against timeout
const timeoutPromise = new Promise<void>(resolve => {
setTimeout(() => {
timeoutTimer = setTimeout(() => {
timedOut = true;
try { proc.kill(); } catch {}
resolve();
Expand Down Expand Up @@ -104,6 +105,9 @@ export class LocalExecution implements ExecutionStrategy {
stderrChunks.push(spawnError!);
}
} finally {
// An uncleared timer keeps the whole process alive until it fires: a finished run used to
// hang for the full agent timeout (15 min) before the CLI could exit.
clearTimeout(timeoutTimer);
this.running.delete(unit.id);
}

Expand Down
105 changes: 97 additions & 8 deletions packages/core/src/execution/scheduler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,12 @@ export interface SchedulerConfig {
agentType?: string;
/** Build the work unit yourself (benchmarks and tests inject a mock agent here). */
unitFactory?: (task: Task, repoPath: string) => WorkUnit;
/**
* When a task's commit conflicts with what other tasks already merged, run the agent again on
* top of the merged result (new worktree from the current base branch, told which files
* conflicted). Each retry costs another agent run. Default 1; 0 disables it.
*/
conflictRetries?: number;
freeclawBase?: string;
salacia?: SalaciaConfig & {
experiment?: { enabled: boolean; splitRatio: number };
Expand All @@ -39,6 +45,16 @@ export interface SchedulerReport {
execMs?: number;
/** Time spent merging the finished commits onto the base branch. */
mergeMs?: number;
/** Time spent re-running conflicted tasks and merging them again. */
retryMs?: number;
/** taskId -> number of retry rounds it needed (only tasks that were retried). */
retries?: Record<string, number>;
/**
* For tasks that merged only after a retry: files they changed that another merged task also
* changed. The merge was clean, but a clean merge does not prove the retry kept the other
* task's change (two contradictory tasks both "succeed", the later one wins), so look at these.
*/
retryOverlaps?: Record<string, Array<{ task: string; files: string[] }>>;
/** Time spent removing task worktrees and branches. */
cleanupMs?: number;
zeroRegressionRate?: number; // SWE-CI inspired: fraction of merges where ALL pre-existing tests still pass
Expand Down Expand Up @@ -75,6 +91,20 @@ export class UnifiedScheduler {
this.execution = new LocalExecution(this.worktrees);
}

private buildUnit(task: Task, repoPath: string, plugin?: SalaciaPlugin, retryNote?: string): WorkUnit {
const unit = this.config.unitFactory
? this.config.unitFactory(task, repoPath)
: this.config.agentType
? toWorkUnitForAgent(task, this.config.agentType, repoPath, plugin)
: toWorkUnitByTier(task, repoPath, this.config.costStrategy, this.config.freeclawBase, plugin);
// LocalExecution creates the worktree under repoPath
unit.repoPath = repoPath;
if (retryNote) unit.taskPrompt += `

${retryNote}`;
return unit;
}

async execute(tasks: Task[], repoPath: string): Promise<SchedulerReport> {
const startTime = Date.now();
this.repoPath = repoPath;
Expand Down Expand Up @@ -114,13 +144,7 @@ export class UnifiedScheduler {
this.stateStore.transition(task.ID, "dispatched");

// Build work unit (with optional salacia enrichment)
const unit = this.config.unitFactory
? this.config.unitFactory(task, repoPath)
: this.config.agentType
? toWorkUnitForAgent(task, this.config.agentType, repoPath, plugin)
: toWorkUnitByTier(task, repoPath, this.config.costStrategy, this.config.freeclawBase, plugin);
// LocalExecution creates the worktree under repoPath
unit.repoPath = repoPath;
const unit = this.buildUnit(task, repoPath, plugin);

// Running — inject salacia plugin into execution
this.stateStore.transition(task.ID, "running", { agentType: unit.agentConfig.type });
Expand Down Expand Up @@ -176,9 +200,63 @@ export class UnifiedScheduler {
// 4. Merge via sequencer
const sequencer = new CommitSequencer(repoPath, this.baseBranch);
const mergeStart = Date.now();
const mergeResults = await sequencer.applyAll(executionResults);
let mergeResults = await sequencer.applyAll(executionResults);
const mergeMs = Date.now() - mergeStart;

// 4b. Conflicts: run those tasks again on top of what was merged (see conflictRetries)
const retries: Record<string, number> = {};
const retryStart = Date.now();
const taskById = new Map(tasks.map(t => [t.ID, t]));
for (let round = 1; round <= (this.config.conflictRetries ?? 1); round++) {
const conflicted = mergeResults.filter(m => !m.success && (m.error ?? "").includes("MERGE_CONFLICT"));
if (conflicted.length === 0) break;

const retryResults: ExecutionResult[] = [];
await Promise.allSettled(conflicted.map(c => this.concurrency.run(async () => {
const task = taskById.get(c.taskId);
if (!task) return;
try {
// Keep the conflicting attempt for inspection; the retry needs the branch name.
gitExec(["branch", "-m", `fsc/${c.taskId}`, `fsc/${c.taskId}-attempt${round}`], repoPath);
const files = (c.error ?? "").split("): ").slice(1).join("): ");
const note =
`## Retry note
An earlier attempt at this task could not be merged because other tasks ` +
`already changed the same code (${files || "unknown files"}). Those changes are in your ` +
`working tree now. Read the current code and do this task again on top of it. If another ` +
`task already made a conflicting decision, keep it unless this task explicitly requires ` +
`otherwise, and say what you did in your final message.`;
const unit = this.buildUnit(task, repoPath, undefined, note);
const r = await this.execution.execute({ ...unit, id: task.ID, repoPath });
Comment on lines +229 to +230

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restore the task-specific Salacia plugin for retries

When Salacia is enabled, retries are built without their scope contract and execute using whichever plugin remains in this.execution.salaciaPlugin from the initial batch. For example, with concurrency 1, if task b conflicts and task c executes afterward, b's retry uses c's plugin; SalaciaPlugin.checkDrift finds no contract for b and returns blocked: false. Consequently, a retry that commits changes to protected paths can bypass the protection enforced on its original attempt. Preserve or recreate the task-specific plugin and use it both when building and executing the retry.

Useful? React with 👍 / 👎.

await this.worktrees.removeWorktree(r.worktreePath, repoPath);
retryResults.push(r);
executionResults.push(r);
retries[task.ID] = round;
Comment on lines +230 to +234

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Update task state from the retry execution result

If a retry exits nonzero or times out after changing files, LocalExecution can return a failure result with an auto-committed partial change. This loop merges that commit without updating the task's original completed state. Reproducing with a retry that appends a line and exits 1 yields completed: 2, failed: 0, and two successful merges, so the CLI reports success and exits 0 despite the failed agent. Update the state and error from the latest execution result so retry failures cannot be reported as successful completion.

Useful? React with 👍 / 👎.

} catch (e: any) {
console.error(`[Retry] ${c.taskId}: ${e?.message ?? e}`);
}
})));

const retryMerge = await sequencer.applyAll(retryResults);
const byTask = new Map(retryMerge.map(m => [m.taskId, m]));
mergeResults = mergeResults.map(m => byTask.get(m.taskId) ?? m);
}
const retryMs = Date.now() - retryStart;

const latest = new Map<string, ExecutionResult>();
for (const r of executionResults) latest.set(r.workUnitId, r);
const mergedIds = new Set(mergeResults.filter(m => m.success).map(m => m.taskId));
const retryOverlaps: Record<string, Array<{ task: string; files: string[] }>> = {};
for (const taskId of Object.keys(retries)) {
if (!mergedIds.has(taskId)) continue;
const mine = new Set(latest.get(taskId)?.filesChanged ?? []);
for (const [other, r] of latest) {
if (other === taskId || !mergedIds.has(other)) continue;
const files = r.filesChanged.filter(f => mine.has(f));
if (files.length > 0) (retryOverlaps[taskId] ??= []).push({ task: other, files });
}
}

// Post-merge regression test (SWE-CI Zero-Regression Rate)
// MEMO-inspired: failed merges are logged to an inverse-frequency replay buffer
// so retryFailed() can prioritize rare/hard failures.
Expand Down Expand Up @@ -223,6 +301,14 @@ export class UnifiedScheduler {
gitExec(["branch", "-D", er.branch], repoPath);
}
}
// A retry that merged supersedes the conflicting attempts that were kept for inspection.
for (const taskId of Object.keys(retries)) {
if (!mergeResults.some(m => m.taskId === taskId && m.success)) continue;
const old = gitExec(["branch", "--list", `fsc/${taskId}-attempt*`, "--format=%(refname:short)"], repoPath);
for (const b of old.stdout.split("\n").map(s => s.trim()).filter(Boolean)) {
gitExec(["branch", "-D", b], repoPath);
}
}

const cleanupMs = Date.now() - cleanupStart;
const durationMs = Date.now() - startTime;
Expand All @@ -238,6 +324,9 @@ export class UnifiedScheduler {
durationMs,
execMs,
mergeMs,
retryMs,
retries,
retryOverlaps,
cleanupMs,
zeroRegressionRate: mergedCount > 0 ? zeroRegressionCount / mergedCount : undefined,
};
Expand Down
21 changes: 21 additions & 0 deletions packages/core/test/execution/fixtures/run-once.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
// Used by local.test.ts: run one mock agent through LocalExecution, print the status, return.
import { LocalExecution } from "../../../src/execution/local";
import { WorktreeManager } from "../../../src/execution/worktree";

const repo = process.argv[2];
const execution = new LocalExecution(new WorktreeManager());
const result = await execution.execute({
id: "timer-test",
repoPath: repo,
baseBranch: "main",
taskPrompt: "x",
agentConfig: {
type: "custom",
command: process.platform === "win32" ? "bash" : "/bin/sh",
buildArgs: () => ["-c", "echo hi > timer.txt"],
costTier: "free",
memoryMB: 50,
},
constraints: { timeoutMs: 600_000 },
});
console.log(result.status);
30 changes: 30 additions & 0 deletions packages/core/test/execution/local.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -157,3 +157,33 @@ describe("LocalExecution", () => {
expect(result.commitHash).toBeNull();
});
});

describe("LocalExecution timers", () => {
test("a finished run leaves no pending timeout timer behind", () => {
// run-once.ts executes a 1 s agent with a 10-minute timeout, then returns. If the timeout
// timer is not cleared the process stays alive for the full 10 minutes.
const repo = path.resolve(os.tmpdir(), `fsc-timer-${Date.now()}`);
fs.mkdirSync(repo, { recursive: true });
const g = (...a: string[]) => gitExec(a, repo);
g("init"); g("checkout", "-b", "main"); g("config", "user.name", "t"); g("config", "user.email", "t@t");
fs.writeFileSync(path.join(repo, "a.txt"), "a");
g("add", "-A"); g("commit", "-m", "init");

const started = Date.now();
const proc = Bun.spawnSync({
cmd: ["bun", path.resolve(import.meta.dir, "fixtures", "run-once.ts"), repo],
timeout: 60_000,
});
const elapsed = Date.now() - started;

expect(new TextDecoder().decode(proc.stdout)).toContain("success");
expect(elapsed).toBeLessThan(30_000); // about 600 s without clearTimeout

new WorktreeManager().cleanup(repo);
fs.rmSync(repo, { recursive: true, force: true });
const parent = path.dirname(repo);
for (const e of fs.readdirSync(parent)) {
if (e.startsWith(`${path.basename(repo)}-fsc-`)) fs.rmSync(path.join(parent, e), { recursive: true, force: true });
}
}, 90_000);
});
Loading
Loading