Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -365,12 +365,42 @@ A {{< gloss "Worker" >}}Worker{{< /gloss >}} is an ordinary Kubernetes pod, so e
* A suspended Actor costs nothing. Its state is already written to snapshot storage, and it occupies no Worker while it waits, so the next message resumes it onto whichever Worker in the pool has availability.
* An Actor that is still running a turn is lost. The Session's in-flight execution state lives in the Actor's sandbox memory on the Worker, and Agent Substrate writes nothing new to snapshot storage until this Actor suspends.

{{< version include-if="latest,0.x" >}}
A lost Actor moves to `ACTOR_STATE_CRASHED`, and Agent Substrate refuses the `resume` and `suspend` operation on an Actor in this state. kagent does not recover the Actor on its own, so the {{< gloss "Session" >}}Session{{< /gloss >}} stays crashed until a client asks for recovery.
{{< /version >}}
{{< version exclude-if="latest,0.x" >}}
A lost Actor moves to `ACTOR_STATE_CRASHED`, and Agent Substrate refuses the `resume` and `suspend` operation on an Actor in this state. By default, the {{< gloss "Session" >}}Session{{< /gloss >}} stays crashed until a client asks for recovery.
{{< /version >}}

{{< version include-if="latest,0.x" >}}
A client with update access to the Session can recover a crashed agent by calling `RevertSession` on the `SessionService` gRPC API. The call restores the Actor to its last durable snapshot and marks the turn that was running as failed. If the Actor is missing or was replaced, the call fails with `FAILED_PRECONDITION`, and starting a new Session is the only option. Neither the `kagent` CLI nor the UI offers a way to revert a Session.
{{< /version >}}
{{< version exclude-if="latest,0.x" >}}
A client with update access to the Session can recover a crashed agent by calling `RevertSession` on the `SessionService` gRPC API. The call restores the Actor to its last durable snapshot and marks the turn that was running as failed. The UI offers a Recover action when a recovery is pending, when a send is refused for an open turn, or after a send retry window runs out. If the Actor is missing or was replaced, the Session is recorded as `FAILED` with operation `NONE` and failure reason `RuntimeLost`. Start a new Session, or fork one from a checkpoint.
{{< /version >}}

Many Actors on a node are suspended at any moment, so losing the node crashes only the ones that are mid-turn. Those Actors lose everything since their last snapshot, and a node pool that runs Workers therefore needs stricter rules than the rest of the cluster. A Worker pod that Kubernetes deletes gives its Actors time to suspend. Node auto-upgrade, a spot reclaim, and a pool scale-down do not.

{{< version exclude-if="latest,0.x" >}}
### Recover crashed Sessions from sends {#recover-crashed-sessions-from-sends}

You can let send requests attempt the same recovery that `RevertSession` performs. The sender still needs Session update access, because the send can start or continue a runtime revert.

Set `config.sessions.automaticRecovery` in the controller configuration.

```yaml
config:
sessions:
automaticRecovery: true
```

| Setting | Description |
| ------- | ----------- |
| `config.sessions.automaticRecovery` | Set to `true` to let a send attempt recovery when the send reaches a crashed runtime, is refused because a crashed turn is still open, or is blocked by a suspension whose Worker was lost. The setting defaults to `false`, so existing installations require an explicit `RevertSession` call or UI Recover action. |

A send that starts recovery waits at most 10 seconds. If recovery is still running, the send returns `KAGENT_SEND_NOT_ACCEPTED` with `state: RECOVERING` and `retryAfterMs: 1000`. Retry the send with the same message ID to continue the recovery. A send while recovery is pending joins that recovery.
{{< /version >}}

### Graceful pod deletion

Deleting a Worker pod is safe. Kubernetes forwards `SIGTERM` into the Actor's containers, the Worker drains for up to 30 minutes inside a pod termination grace period of 3600 seconds, and an Actor that suspends inside this window keeps its state and stays resumable. Rolling a `workerImage` change through a pool therefore finishes any in-flight turns rather than cutting them off.
Expand Down