Skip to content

fix(ai-harness): a second instance must not stop or interleave the owner's turn - #1810

Open
jherr wants to merge 3 commits into
feat/harness-corefrom
fix/harness-session-events-owner
Open

jherr wants to merge 3 commits into
feat/harness-corefrom
fix/harness-session-events-owner

Conversation

@jherr

@jherr jherr commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

On a durable host with more than one instance, a second instance could stop the turn that the owner runs, or run inputs in the middle of it. This PR fixes two ways that happened. Both were found while hosting the harness on Netlify Play.

  1. A second instance's plugin state stopped the owner's turn. A host that opened the thread wrote its plugin state (for example the MCP connection status) to the shared log. The owner's next write conflicted, and its turn stopped. Then the turn ran again.
  2. A slow store let a second host run inputs in the middle of a turn. On Play, Blobs requests stalled for ~10 s. The owner's claim expired while its run lease still renewed. Another host then ran the queued follow-ups in the middle of the owner's tool call. The transcript got a user message between a tool call and its result, and the model refused it with a 400.

Each fix is its own commit:

Commit What it fixes
a host that does not own a thread keeps its session events off the log Fix 1
a live run lease of another host means the thread is owned elsewhere Fix 2
cover the inbox drain in the live-lease test, document both rules Smaller diff, the inbox case of fix 2 in the test, docs

🎯 Changes

  • A durable session reads the work claim before its plugins mount. While another host owns the thread, its session events (plugin state, plugin events, warnings) are not written to the log.
  • isOwnedElsewhere also reads the thread's active run. A live run lease of another host counts, also when that host's claim lapsed. The inbox drain checks it before it takes the claim.
  • Docs: durable-sessions.md (a live run lease makes the owner, and a host that does not own a thread writes nothing to its log).

✅ Checklist

  • I have followed the steps in the Contributing guide.
  • I have tested code changes locally with pnpm run test:pr, or these tests do not apply to this pull request. (See Testing: the targets that apply were run one by one. test:pr compares against main, which this stacked PR is not based on.)
  • I fully understand the code in this pull request, including any code generated with AI assistance.
  • Docs: I updated docs/ for this change, or this change is not user-facing.
  • Changeset: I added a changeset (pnpm changeset), or this PR does not change a published package.

🚀 Release Impact

  • This change affects published code, and I have generated a changeset.
  • This change is docs/CI/dev-only (no release).

Root cause

Issue. On a durable host with workClaims and several instances (serverless, no sticky routing), a second instance can write to the log of a thread while the owner runs a turn. Fix 1: the owner's turn stops with "Another host wrote to this log" and runs again from the start. Fix 2: the second host runs queued inputs mid-turn, and the transcript is invalid for the model.

Cause.

  • Fix 1. pluginState().publish, emitPluginEvent and warnSession publish to the session feed. On a durable host that feed is the log writer. open() mounts the plugins before anything checks the claim, so a second host appends to the log. The owner's next append fails the position check, and stopOnLogFailure stops its session.
  • Fix 2. isOwnedElsewhere() reads only the work claim. A claim renewal that stalls (the @netlify/blobs client retries 5 times, 5 s apart) lets the claim expire while the run lease still renews. Then recoverFromLog sees the live lease and skips that one input (continue), and runs the later pending inputs. drainOnce takes the expired claim and applies the inbox entries.

Fix. A session keeps its session events off the log while another host owns the thread. A live run lease of another host means the thread is owned elsewhere. So recovery is deferred, new inputs go to the inbox, and the inbox drain leaves them for the owner.

Possible alternatives

  • Pass the second host's session events to the owner through the inbox. Plugin state of one host (an MCP connection) is not the owner's state. So the events are dropped, and the doc says so.
  • Stop recovery at the first input whose lease is live, instead of a lease check in isOwnedElsewhere. It fixes recovery only. A new input on the second host would still be accepted and run mid-turn, so the check is where every path reads ownership.
  • Make the claim renewal faster or retry less. That narrows the window, and the store decides it. Any store can stall longer than the claim TTL.

Testing

Commands run (on this branch, current feat/harness-core at d7e490af0):

  • nx run-many -t build for all packages: pass.
  • ai-harness: vitest run 1556 passed, tsc --noEmit, oxlint on the changed files: pass.
  • E2E: tsc --noEmit in testing/e2e, the full suite: 626 passed. After the last commit, playwright test --grep harness: 57 passed.
  • pnpm test:docs and pnpm test:kiira (1896 snippets): pass.
  • Not run: pnpm test:pr, test:knip, test:sherif (they compare against main).

Repro before and after. Both tests are in tests/forwarding.test.ts. Run with src/ at feat/harness-core, then at this branch:

=== base (feat/harness-core src)
 × keeps the plugin state of another host off the log while the owner runs a tool
 × leaves a turn alone when its host claim lapsed but its run lease is live
Error: Another host wrote to this log, so this session stopped. Open the thread again.
AssertionError: expected [ Array(3) ] to have a length of 1 but got 3
=== fix
 Tests  2 passed | 15 skipped (17)

Each new case in testing/e2e/tests/harness-forwarding.spec.ts (?plugin=1, ?stall=1) fails on the base and passes here.

Also tested on Netlify Play. A Netlify Server app on this branch with Netlify Blobs stores ran these tests:

  • A simulated 12 s Blobs outage on the owner actor during a tool call, 3 runs. Every transcript was valid, and every follow-up was answered.
  • 16 follow-ups that reached a non-owner instance. All 16 ran, in order.
  • A browser test of a chat UI, with an approval.

Before the fix, the same outage left a user message between a tool call and its result.

Manual test

  1. Run npx vitest run tests/forwarding.test.ts -t "run lease is live" in packages/ai-harness. It passes.
  2. Revert src/session/base.ts to feat/harness-core, then run the same command. It fails: the second host made 3 model calls while the owner's turn ran.
  3. Do the same with -t "plugin state of another host". It fails on the base with "Another host wrote to this log".

How this PR makes testing easy: two cases in tests/forwarding.test.ts, and two in testing/e2e/tests/harness-forwarding.spec.ts (?plugin=1, ?stall=1).

Risk / rollback

  • More reads. A durable host with workClaims reads runs.findActiveRun(threadId) when the claim is free. That happens once per input and once per open.
  • Plugin events. A host that does not own the thread no longer shows its own plugin events to its clients through the log.
  • Rollback: revert the PR.

🤖 Generated with Claude Code

jherr and others added 3 commits October 16, 2026 03:35
…events off the log

A durable host that opened a thread while another host ran a turn wrote
its plugin state (for example the MCP plugin's connection status) to the
shared log. The owner's next append conflicted, and its turn stopped. The
session now reads the claim before its plugins mount, and drops session
events while another host owns the thread.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… owned elsewhere

On Netlify Play, Blobs requests stalled for ~10 s (the client retries for
up to 25 s). The owner's claim renewals did not land and its claim expired,
while its run lease renewed and its turn ran a tool. Another host opened
the thread, took the claim, skipped the live turn in recovery, and ran the
queued follow-ups. The transcript got a user message between a tool call
and its result, and the model refused it.

`isOwnedElsewhere` also reads the thread's active run: a live lease of
another host counts. The inbox drain checks it before it claims.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nt both rules

The inbox drain guard is now proven by the test: an input that waits in
the inbox for an owner whose claim lapsed stays there until the owner's
turn ends. Drop the ownerElsewhere resets after a claim: every accept()
reads ownership again. Docs: a live run lease makes the owner, and a host
that does not own a thread writes nothing to its log.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Oct 16, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: d6d2400

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 7 packages
Name Type
@tanstack/ai-harness Patch
@tanstack/ai-acp Patch
@tanstack/ai-code-mode Patch
@tanstack/ai-harness-cli Patch
@tanstack/ai-mcp Patch
@tanstack/ai-sandbox Patch
@tanstack/ai-skills Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai

coderabbitai Bot commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: TanStack/ai/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: b95821a8-77b4-4b07-a778-4ee8537ae8d4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@nx-cloud

nx-cloud Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

View your CI Pipeline Execution ↗ for commit d6d2400

Command Status Duration Result
nx affected --targets=test:sherif,test:knip,tes... ✅ Succeeded 5m 19s View ↗
nx run-many --targets=test:coverage --projects=... ✅ Succeeded 2s View ↗
nx run-many --targets=build --exclude=examples/... ✅ Succeeded 56s View ↗

☁️ Nx Cloud last updated this comment at 2026-10-16 21:50:48 UTC

@pkg-pr-new

pkg-pr-new Bot commented Oct 16, 2026

Copy link
Copy Markdown

Open in StackBlitz

@tanstack/ai

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai@1810

@tanstack/ai-acp

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-acp@1810

@tanstack/ai-angular

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-angular@1810

@tanstack/ai-anthropic

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-anthropic@1810

@tanstack/ai-bedrock

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-bedrock@1810

@tanstack/ai-byteplus

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-byteplus@1810

@tanstack/ai-claude-code

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-claude-code@1810

@tanstack/ai-client

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-client@1810

@tanstack/ai-cloudflare

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-cloudflare@1810

@tanstack/ai-code-mode

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-code-mode@1810

@tanstack/ai-code-mode-snippets

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-code-mode-snippets@1810

@tanstack/ai-codex

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-codex@1810

@tanstack/ai-cohere

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-cohere@1810

@tanstack/ai-compaction

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-compaction@1810

@tanstack/ai-devtools-core

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-devtools-core@1810

@tanstack/ai-durable-stream

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-durable-stream@1810

@tanstack/ai-elevenlabs

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-elevenlabs@1810

@tanstack/ai-event-client

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-event-client@1810

@tanstack/ai-fal

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-fal@1810

@tanstack/ai-gemini

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-gemini@1810

@tanstack/ai-grok

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-grok@1810

@tanstack/ai-grok-build

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-grok-build@1810

@tanstack/ai-groq

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-groq@1810

@tanstack/ai-harness

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-harness@1810

@tanstack/ai-harness-cli

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-harness-cli@1810

@tanstack/ai-isolate-cloudflare

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-isolate-cloudflare@1810

@tanstack/ai-isolate-daytona

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-isolate-daytona@1810

@tanstack/ai-isolate-e2b

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-isolate-e2b@1810

@tanstack/ai-isolate-node

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-isolate-node@1810

@tanstack/ai-isolate-quickjs

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-isolate-quickjs@1810

@tanstack/ai-isolate-quickjs-bun

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-isolate-quickjs-bun@1810

@tanstack/ai-llmgateway

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-llmgateway@1810

@tanstack/ai-lovable

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-lovable@1810

@tanstack/ai-mcp

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-mcp@1810

@tanstack/ai-memory

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-memory@1810

@tanstack/ai-mistral

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-mistral@1810

@tanstack/ai-octane

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-octane@1810

@tanstack/ai-ollama

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-ollama@1810

@tanstack/ai-ollaya

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-ollaya@1810

@tanstack/ai-openai

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-openai@1810

@tanstack/ai-opencode

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-opencode@1810

@tanstack/ai-openrouter

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-openrouter@1810

@tanstack/ai-perplexity

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-perplexity@1810

@tanstack/ai-persistence

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-persistence@1810

@tanstack/ai-preact

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-preact@1810

@tanstack/ai-react

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-react@1810

@tanstack/ai-react-ui

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-react-ui@1810

@tanstack/ai-reactor

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-reactor@1810

@tanstack/ai-remix

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-remix@1810

@tanstack/ai-sandbox

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox@1810

@tanstack/ai-sandbox-blaxel

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-blaxel@1810

@tanstack/ai-sandbox-boxd

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-boxd@1810

@tanstack/ai-sandbox-cloudflare

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-cloudflare@1810

@tanstack/ai-sandbox-daytona

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-daytona@1810

@tanstack/ai-sandbox-docker

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-docker@1810

@tanstack/ai-sandbox-e2b

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-e2b@1810

@tanstack/ai-sandbox-local-process

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-local-process@1810

@tanstack/ai-sandbox-railway

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-railway@1810

@tanstack/ai-sandbox-sprites

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-sprites@1810

@tanstack/ai-sandbox-upstash-box

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-upstash-box@1810

@tanstack/ai-sandbox-vercel

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-vercel@1810

@tanstack/ai-skills

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-skills@1810

@tanstack/ai-solid

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-solid@1810

@tanstack/ai-solid-ui

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-solid-ui@1810

@tanstack/ai-svelte

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-svelte@1810

@tanstack/ai-typesafe

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-typesafe@1810

@tanstack/ai-utils

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-utils@1810

@tanstack/ai-vercel-gateway

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-vercel-gateway@1810

@tanstack/ai-vertex

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-vertex@1810

@tanstack/ai-vue

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-vue@1810

@tanstack/ai-vue-ui

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-vue-ui@1810

@tanstack/ai-worldlabs

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-worldlabs@1810

@tanstack/openai-base

npm i https://pkg.pr.new/TanStack/ai/@tanstack/openai-base@1810

@tanstack/preact-ai-devtools

npm i https://pkg.pr.new/TanStack/ai/@tanstack/preact-ai-devtools@1810

@tanstack/react-ai-devtools

npm i https://pkg.pr.new/TanStack/ai/@tanstack/react-ai-devtools@1810

@tanstack/solid-ai-devtools

npm i https://pkg.pr.new/TanStack/ai/@tanstack/solid-ai-devtools@1810

@tanstack/svelte-ai-devtools

npm i https://pkg.pr.new/TanStack/ai/@tanstack/svelte-ai-devtools@1810

commit: d6d2400

@github-actions

Copy link
Copy Markdown
Contributor

Coverage

✅ Coverage held across 30 compared package(s).

Each package is measured twice in this job — on this PR and on its merge-base with main — and compared. A metric fails only when it is at or above 60% on the merge-base and under 60% on this PR. Other drops are listed but do not fail. Packages your PR doesn't affect are not measured and not listed.

package statements branches functions lines
ai-acp 82.30% (0.00) 71.49% (0.00) 77.20% (0.00) 84.13% (0.00) ok
ai-claude-code 88.93% (0.00) 76.88% (0.00) 86.88% (0.00) 90.40% (0.00) ok
ai-code-mode 92.71% (0.00) 85.66% (0.00) 89.06% (0.00) 93.51% (0.00) ok
ai-code-mode-snippets 87.13% (0.00) 75.67% (0.00) 91.11% (0.00) 88.39% (0.00) ok
ai-codex 94.55% (0.00) 85.10% (0.00) 89.28% (0.00) 95.56% (0.00) ok
ai-durable-stream 86.70% (0.00) 82.25% (0.00) 92.30% (0.00) 88.72% (0.00) ok
ai-grok-build 79.19% (0.00) 63.56% (0.00) 77.39% (0.00) 81.02% (0.00) ok
ai-harness 94.34% (0.00) 88.69% (0.00) 94.29% (+0.06) 95.70% (+0.02) ok
ai-harness-cli 84.31% (0.00) 80.27% (0.00) 81.03% (0.00) 84.90% (0.00) ok
ai-isolate-cloudflare 94.11% (0.00) 70.66% (0.00) 100.00% (0.00) 93.96% (0.00) ok
ai-isolate-daytona 90.25% (0.00) 80.59% (0.00) 100.00% (0.00) 90.15% (0.00) ok
ai-isolate-e2b 92.11% (0.00) 81.48% (0.00) 97.36% (0.00) 95.05% (0.00) ok
ai-isolate-node 84.00% (0.00) 43.39% (0.00) 100.00% (0.00) 83.78% (0.00) ok
ai-isolate-quickjs 89.07% (0.00) 76.41% (0.00) 90.90% (0.00) 90.30% (0.00) ok
ai-isolate-quickjs-bun 7.76% (0.00) 8.08% (0.00) 5.00% (0.00) 8.33% (0.00) ok
ai-mcp 96.69% (0.00) 91.65% (0.00) 96.88% (0.00) 98.62% (0.00) ok
ai-opencode 69.04% (0.00) 60.82% (0.00) 65.26% (0.00) 70.87% (0.00) ok
ai-sandbox 77.96% (-0.06) 83.06% (-0.04) 75.22% (-0.11) 78.52% (-0.05) ok
ai-sandbox-blaxel 87.64% (0.00) 77.18% (0.00) 82.26% (0.00) 89.71% (0.00) ok
ai-sandbox-boxd 95.86% (0.00) 87.12% (0.00) 95.31% (0.00) 98.17% (0.00) ok
ai-sandbox-cloudflare 67.91% (0.00) 57.47% (0.00) 71.79% (0.00) 69.62% (0.00) ok
ai-sandbox-daytona 82.70% (0.00) 81.89% (0.00) 67.14% (0.00) 85.18% (0.00) ok
ai-sandbox-docker 80.15% (0.00) 71.49% (0.00) 77.38% (0.00) 82.13% (0.00) ok
ai-sandbox-e2b 94.42% (0.00) 88.97% (0.00) 93.24% (0.00) 98.63% (0.00) ok
ai-sandbox-local-process 68.00% (0.00) 53.59% (0.00) 75.86% (0.00) 68.88% (0.00) ok
ai-sandbox-railway 94.75% (0.00) 92.68% (0.00) 90.90% (0.00) 97.59% (0.00) ok
ai-sandbox-sprites 73.23% (0.00) 59.70% (0.00) 73.83% (0.00) 76.17% (0.00) ok
ai-sandbox-upstash-box 94.82% (0.00) 91.52% (0.00) 91.04% (0.00) 97.76% (0.00) ok
ai-sandbox-vercel 44.67% (0.00) 38.18% (0.00) 35.18% (0.00) 45.76% (0.00) ok
ai-skills 83.59% (0.00) 68.58% (0.00) 87.36% (0.00) 85.63% (0.00) ok

@github-actions
github-actions Bot requested a review from tombeckenham October 16, 2026 23:37
@github-actions github-actions Bot added the waiting-on: maintainer The ball is in the maintainers’ court label Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-on: maintainer The ball is in the maintainers’ court

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants