refactor(chat): run recovery continuations on Tasks

[mattzcarey]

Base

Rebased onto main after #2190 merged. The rebase also incorporates #2191's Tasks write optimizations and #2192's deterministic Durable Object test helpers.

Why this changed

Root AI Chat and Think recovery still used Scheduler rows even though Tasks had become the durable replay primitive. Moving recovery onto Tasks exposed a broader alarm-lifetime issue: a job can return at a bounded handoff so Lifecycle can continue its due batch while the promise it started is still running. A memory reset from that promise still belongs to the alarm that started it, but it no longer reaches the alarm's call stack unless Lifecycle retains that relationship.

This is an alarm-domain concern, not Task or Scheduler policy.

Final architecture

Lifecycle owns bounded alarm work

• Lifecycle pre-arms its existing deadman alarm before driving due jobs.
• trackAlarmWork(promise) lets a bounded job handoff register the work it started with the current alarm.
• Lifecycle continues driving the due batch, runs the host alarm hook, then joins the dynamically growing registered-work batch before classifying the alarm.
• Every registered promise settling without a memory reset makes the alarm clean and clears prior strikes.
• Any registered promise reporting a memory reset enters the existing alarm breaker once for the whole batch, using the job that registered it as the executing context.
• Physical alarm re-arms requested while the batch is running are coalesced until the batch settles, so they cannot overwrite the pre-armed deadman.

Tasks is only an adapter to that boundary

• Task wakes use one JobDriver attempt because ReplayStep owns their durable retry policy.
• When Tasks reaches its five-second dispatch handoff, it registers the still-running attempt with Lifecycle and returns the run's authoritative wake outcome.
• Tasks still updates authoritative run rows on breaker backoff/sealing, preventing startup reconciliation from resurrecting contained work.
• Framework recovery definitions remain internal; there is no public grouping or recovery flag.
• retain: false removes completed, failed, and cancelled runs, journals, wakes, and idempotency keys.

Chat recovery uses Tasks without losing Scheduler compatibility

• Root AI Chat and Think recovery attempts are chained __cf_internal_chat_recovery Task runs.
• Initial attempts deduplicate by incident; delayed retries use durable Task sleeps.
• _chatRecoveryContinue and _chatRecoveryRetry return at model handoff, preserving queue liveness.
• At that handoff AI Chat and Think register the model promise with Lifecycle. This applies identically whether the callback arrived through a Task or the temporary Scheduler compatibility transport.
• Pre-handoff failures remain owned by the current Task/Scheduler execution; only post-handoff platform failures enqueue one replacement attempt.
• Think startup recovery detects active Task runs through tasks.list().

Upgrade and breaker correctness

• Existing cf_agents_jobs tables are upgraded idempotently with recovery_loop before any query references it.
• Existing Task wake rows are reconciled to the one-attempt JobDriver policy.
• Condemned-isolate errors escape step.do immediately.
• Connection-loss failures use ReplayStep's durable retry budget and remain unsettled after exhaustion.
• Recovery-loop wakes still back off and seal as a group; unrelated jobs remain untouched.

What we found during review

The first downstream approach used Task-specific late-OOM and clean-settlement marker jobs. It handled one detached Task, but it made Tasks responsible for alarm policy and failed when several promises belonged to the same alarm: one clean Task could clear strikes before a sibling reported OOM. It also left Scheduler-originated recovery as a separate case.

The final version removes those Task marker jobs and moves promise retention and group classification into Lifecycle. Tasks, Scheduler-driven chat recovery, and future alarm-backed capabilities now use one small core boundary instead of reimplementing breaker behavior downstream.

Compatibility and release

• Existing pending _chatRecoveryContinue / _chatRecoveryRetry Scheduler rows still dispatch through the retained methods and drain normally.
• Routed dynamic agents temporarily keep the root-owned Scheduler transport until Tasks supports routed child wakes. The refactor(lifecycle): dissolve alarm memory-limit policy into the jobs domain #2190 sealing bridge protects that path.
• AI Chat and Think require agents >=0.23.0, the pending changesets release batch containing the shared runtime support.

Verification

Current head: 56e4635f.

• Changed files formatted with Oxfmt.
• git diff --check passes.
• GitHub CI and Devin review are running for the current head.

Regression coverage in the branch includes:

• existing job-table schema upgrade and repeated startup
• final-step platform failure deferring after one Task wake dispatch
• queued and warm Task attempts remaining in the alarm breaker after handoff
• all-clean registered alarm work clearing stale strikes
• concurrent clean and OOM work producing one alarm-level breaker outcome
• Task run-row backoff surviving startup reconciliation
• sealing at the strike budget and non-retained terminal cleanup
• pre-handoff ownership versus post-handoff replacement for both chat callbacks
• root AI Chat and Think recovery using Tasks rather than Scheduler rows
• legacy and routed Scheduler recovery compatibility

---

[changeset-bot]

🦋 Changeset detected

Latest commit: 56e4635

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
agents	Minor
@cloudflare/ai-chat	Patch
@cloudflare/think	Patch
@cloudflare/agent-think	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

Open in StackBlitz

agents

npm i https://pkg.pr.new/agents@2194

@cloudflare/ai-chat

npm i https://pkg.pr.new/@cloudflare/ai-chat@2194

@cloudflare/codemode

npm i https://pkg.pr.new/@cloudflare/codemode@2194

hono-agents

npm i https://pkg.pr.new/hono-agents@2194

@cloudflare/shell

npm i https://pkg.pr.new/@cloudflare/shell@2194

@cloudflare/think

npm i https://pkg.pr.new/@cloudflare/think@2194

@cloudflare/voice

npm i https://pkg.pr.new/@cloudflare/voice@2194

@cloudflare/worker-bundler

npm i https://pkg.pr.new/@cloudflare/worker-bundler@2194

commit: 1f98ea4