Skip to content

orch: serialize stale-lock takeover, and resolve a reused branch to its open PR - #38

Open
J0UH wants to merge 2 commits into
chhoumann:mainfrom
J0UH:fix/orch-store-lock-and-branch
Open

J0UH wants to merge 2 commits into
chhoumann:mainfrom
J0UH:fix/orch-store-lock-and-branch

Conversation

@J0UH

@J0UH J0UH commented Oct 21, 2026

Copy link
Copy Markdown

Two bugs in the orchestrate store, plugins/pstack/skills/poteto-mode/scripts/orch/store.ts. Both are in Lauren's current store.ts (unchanged upstream since ce649c0), so the fix makes store.ts and orch.test.ts declared forks in tools/forks.json.

1. Two writers can both take a stale lock

When .orch.lock names a dead pid, acquireLock reads the holder, then runs unlink and an exclusive create. Two writers can both read the dead holder. If the second one's unlink runs after the first one's create, it deletes the first writer's live lock and creates its own. Both then hold the lock, and their read-modify-write of units.tsv or ledger.tsv drops rows.

Reproduction

A harness writes a dead pid into .orch.lock, starts 6 processes that each open the store and run units.add with their own id, and counts the rows in units.tsv against the processes that reported success. On chhoumann/main, 200 trials:

{"trials":200,"writers":6,"succeeded":217,"lostUnitRows":17,"refused":983}

17 successful units.add calls left no row, so two writers held the lock at once. With this branch:

{"trials":200,"writers":6,"succeeded":200,"lostUnitRows":0,"refused":1000}

The new test "keeps a lock another writer took over from the same dead holder" pins the interleaving deterministically: in onStaleLock, which runs after the late writer reads the dead holder, it lets a rival finish its takeover. On main the late writer then deletes the rival's lock and succeeds.

Fix

A takeover now runs behind a claim: a directory, .orch.lock.takeover, that holds one file named for the claimant's pid. The writer stages that directory and renames it into place. rename cannot replace a directory that holds a file, so one writer claims at a time and the others fail with store lock held by pid <pid> is being replaced by another writer; retry. Under the claim the writer reads the lock again and unlinks it only if it still names the holder it judged replaceable. A claim left by a dead writer is removed by its pid-named file, which cannot remove a live claimant's file. The forced takeover goes through the same claim. The lock file format does not change.

I chose this over renaming the stale lock aside first. A rename takes whatever file is at the path, including a lock another writer just created, and putting it back leaves a gap in which a third writer can create the lock. An exclusive-create claim file would avoid that, but a writer that dies holding it leaves nothing that says whether the claimant is alive. The directory claim is the smallest form I found that recovers from a dead claimant.

2. A reused branch fails frontier resolution

orch frontier set lists each branch's PRs with --state all, because a merged or closed PR still anchors the chain below the current branch. A branch that had a merged or closed PR and now has a new open one returns all of them, and branchPullRequest refuses:

error: branch stack/open has multiple pull requests (#11, #9, #8); close the stale ones first

Closing does not help: #9 and #8 are already closed or merged.

Fix

When a branch has an open PR, that PR is the branch's current one, and its merged and closed PRs are ignored. Two open PRs on one branch still fail, and the error names only the open ones. A branch with no open PR keeps the old rule of exactly one PR.

Tests

Six new tests in orch/orch.test.ts: the two lock tests, and for both gh and origin, a reused branch resolving to its open PR and a branch with two open PRs failing. On chhoumann/main's store.ts, all six fail:

(fail) Store > keeps a lock another writer took over from the same dead holder
(fail) Store > serializes stale-lock takeovers behind a claim a dead writer cannot hold
error: branch stack/open has multiple pull requests (#11, #9, #8); close the stale ones first
(fail) Store > resolves a gh branch reused after a merged and a closed PR to its open PR
Received message: "branch stack/open has multiple pull requests (#12, #11, #9); close the stale ones first"
(fail) Store > rejects a gh branch with more than one open PR
(fail) Store > resolves a origin branch reused after a merged and a closed PR to its open PR
(fail) Store > rejects a origin branch with more than one open PR
 17 pass
 6 fail

On this branch:

  • GIT_CONFIG_GLOBAL=/dev/null bun test orch watch-pr check-plan.test.ts: 125 pass, 0 fail
  • bun run typecheck: clean
  • bunx prettier@3.6.2 --check watch-pr/settle.ts watch-pr/settle.test.ts resolve-codex-transcript.ts: clean
  • GIT_CONFIG_GLOBAL=/dev/null bun test tests/ --timeout 30000: 550 pass, 46 skip, 0 fail
  • bun tools/generate.mjs --check: 17 generated files current, all checks pass
  • bun tools/sync.mjs {pstack,cursor-team-kit,thermos} 88d4751 --dry-run: exit 0 for each; pstack lists orch/store.ts and orch/orch.test.ts as declared policy forks

Release

This PR does not bump VERSION or add a CHANGES.md entry, so the fix is inert on installed copies until a release. I left the release to you, since other fix PRs are open at the same time. The since field in tools/forks.json says 2.0.0-rc.16; change it if the fix ships in a different release.

Race harness

race.ts:

// Usage: bun race.ts <store.ts> <trials> <writers>
import { mkdtemp, writeFile, readFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
const [storePath, trialsArg, writersArg] = process.argv.slice(2);
const trials = Number(trialsArg), writers = Number(writersArg);
const worker = join(import.meta.dir, "worker.ts");
const { openStore } = await import(storePath);
let succeeded = 0, lost = 0, refused = 0;
for (let t = 0; t < trials; t++) {
  const dir = await mkdtemp(join(tmpdir(), "orch-race-"));
  const init = openStore(dir); await init.init(); await init.close();
  const dead = Bun.spawn(["true"]); await dead.exited;
  await writeFile(join(dir, ".orch.lock"), `${dead.pid}\n`);
  const procs = Array.from({ length: writers }, (_, i) =>
    Bun.spawn([process.execPath, worker, storePath, dir, `u${i}`], { stdout: "pipe", stderr: "pipe" }));
  const outs = await Promise.all(procs.map(async (p) => { await p.exited; return (await new Response(p.stdout).text()).trim(); }));
  const ok = outs.filter((o) => o === "ok").length;
  refused += outs.filter((o) => o.startsWith("refused")).length;
  const rows = (await readFile(join(dir, "units.tsv"), "utf8")).trim().split("\n").length - 1;
  succeeded += ok;
  if (rows < ok) lost += ok - rows;
  await rm(dir, { recursive: true, force: true });
}
console.log(JSON.stringify({ trials, writers, succeeded, lostUnitRows: lost, refused }));

worker.ts:

const [storePath, dir, id] = process.argv.slice(2);
const { openStore } = await import(storePath);
const store = openStore(dir);
try {
  await store.units.add({ id, track: "race" });
  await new Promise((r) => setTimeout(r, 50));
  console.log("ok");
} catch (e) { console.log("refused " + (e as Error).message); }
finally { await store.close().catch(() => {}); }

J0UH added 2 commits October 21, 2026 10:52
Two writers that both found the store lock naming a dead pid could both
take it. Each read the dead holder, then ran unlink and an exclusive
create. When the second writer's unlink ran after the first writer's
create, it deleted the first writer's live lock and created its own, so
both held the lock and their read-modify-write of units.tsv or
ledger.tsv could drop rows.

A takeover now runs behind a claim: a directory, .orch.lock.takeover,
holding one file named for the claimant's pid. The claimant stages that
directory and renames it into place. rename cannot replace a directory
that holds a file, so only one writer claims at a time and the others
fail with "being replaced by another writer; retry". Under the claim,
the writer reads the lock again and unlinks it only if it still names
the holder it judged replaceable. A claim left by a dead writer is
removed by its pid-named file, which cannot remove a live claimant's.
The forced takeover goes through the same claim.

store.ts and orch.test.ts are now declared forks in tools/forks.json.
orch frontier set lists each branch's pull requests with --state all,
because a merged or closed PR still anchors the chain below the current
branch. A branch that had a merged or closed PR and now has a new open
one returned all of them, and frontier resolution failed with "has
multiple pull requests (chhoumann#11, chhoumann#9, chhoumann#8); close the stale ones first".
Closing does not help, since the old PRs are already closed or merged.

When a branch has an open PR, that PR is the branch's current one, and
its merged and closed PRs are ignored. Two open PRs on one branch still
fail, and the error now names only the open ones. A branch with no open
PR keeps the old rule: exactly one PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant