fix(cloud-agent-next): recover sandbox after wrapper cleanup exhaustion

[eshurakov]

Summary

When wrapper stop/cleanup attempts are exhausted (wrapper unreachable, sandbox likely wedged), the lease previously stayed exhausted forever and queued message delivery remained held. This adds durable post-exhaustion sandbox recovery:

• New postExhaustionRecovery entry in the DO sandbox-recovery state, classified from session metadata as isolated-destroy (delete the session sandbox) or shared-failover (record failover on the route, resolve the replacement assignment, persist it into session metadata).
• Retried with backoff via alarms and idempotent reducers keyed by the exhausted lease identity; the exhausted lease settles only after the sandbox action completes, then pending delivery drain is released.
• Unhealthy-wrapper terminalization now persists terminal transitions with idle-batch allowance and defers finalization to the outbox, which flushes terminal wrapper-run callbacks once the terminal wait is released.
• The alarm handler defers terminal-effect repair while queued delivery is held and cleanup is not yet exhausted, so repair no longer terminalizes messages that recovery can still deliver.

Verification

• No manual verification; covered by targeted unit tests (wrapper-supervisor, wrapper-runtime-state, message-settlement-outbox) and Workers-runtime integration tests (execute-directly-failure, start-execution-v2).

Visual Changes

N/A

Reviewer Notes

Focus on the new reducer transitions in wrapper-runtime-state.ts and the alarm-driven reconcilePostExhaustionSandboxRecovery loop in wrapper-supervisor.ts.

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The previously flagged double-destroy risk in the isolated-destroy post-exhaustion recovery path is fixed by a destroyedAt checkpoint that mirrors the shared-failover assignment checkpoint, and is covered by new reducer and supervisor tests; no new issues found in the incremental changes.

Files Reviewed (4 files)

• services/cloud-agent-next/src/session/wrapper-runtime-state.ts
• services/cloud-agent-next/src/session/wrapper-runtime-state.test.ts
• services/cloud-agent-next/src/session/wrapper-supervisor.ts
• services/cloud-agent-next/src/session/wrapper-supervisor.test.ts

Previous Review Summary (commit fe16a0c)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit fe16a0c)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

The isolated-destroy post-exhaustion recovery path in wrapper-supervisor.ts can re-invoke sandbox destruction on an already-destroyed sandbox if settlement fails after a successful destroy, unlike the checkpointed shared-failover path.

Overview

Severity	Count
CRITICAL	0
WARNING	1
SUGGESTION	0

Issue Details (click to expand)

WARNING

File	Line	Issue
services/cloud-agent-next/src/session/wrapper-supervisor.ts	1249	destroyIsolatedSandbox can be invoked a second time on an already-destroyed sandbox if settlePostExhaustionRecovery fails afterward, with no checkpoint (unlike the shared-failover branch) and no verified idempotency in CloudflareAgentSandbox.delete('recovery')

Files Reviewed (9 files)

• services/cloud-agent-next/src/persistence/CloudAgentSession.ts
• services/cloud-agent-next/src/session/message-settlement-outbox.ts - 0 issues
• services/cloud-agent-next/src/session/message-settlement-outbox.test.ts
• services/cloud-agent-next/src/session/wrapper-runtime-state.ts - 0 issues (checked settle_post_exhaustion coupling with failoverPublication; consistent with existing recordSharedSandboxFailover call sites, not a defect)
• services/cloud-agent-next/src/session/wrapper-runtime-state.test.ts
• services/cloud-agent-next/src/session/wrapper-supervisor.ts - 1 issue
• services/cloud-agent-next/src/session/wrapper-supervisor.test.ts
• services/cloud-agent-next/test/integration/session/execute-directly-failure.test.ts
• services/cloud-agent-next/test/integration/session/start-execution-v2.test.ts

Fix these issues in Kilo Cloud

---

_{Reviewed by claude-sonnet-5 · Input: 26 · Output: 4K · Cached: 585.4K}

_{Review guidance: REVIEW.md from base branch main}