Skip to content
122 changes: 120 additions & 2 deletions src/app/docs/kagent/operations/upgrade/page.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,15 @@ Follow these steps to upgrade kagent to the latest version and keep your cluster
3. Back up your current configuration, including the following:
- Agent definitions
- Any custom settings
- PostgreSQL database
- PostgreSQL database: You can take a snapshot now so that you have a restore point if the upgrade fails. For the database connection string, see [Database configuration](/docs/kagent/operations/operational-considerations#database-configuration).

4. **v0.9.0 and later**: You must be running at least v0.8.0 before upgrading to v0.9.0. Check the [release notes](/docs/kagent/resources/release-notes#v09) for 0.9-specific upgrades related to database migrations and RBAC scope.
```bash
pg_dump "postgres://<user>:<password>@<host>:5432/<dbname>" \
--format=custom \
--file=kagent-pre-upgrade-snapshot.dump
```

4. **v0.9.0 and later**: You must be running at least v0.8.0 before upgrading to v0.9.0. Check the [release notes](/docs/kagent/resources/release-notes#v09) for 0.9-specific upgrades related to database migrations and RBAC scope.

## Upgrade kagent

Expand Down Expand Up @@ -81,3 +87,115 @@ After upgrading, verify that kagent is running.
```bash
kubectl get pods -n kagent
```

## Roll back kagent

If you need to roll back to a previous version after a successful upgrade, use the following steps.

### Rollback compatibility window

As of v0.9, kagent guarantees `n-1` application compatibility with the database: an application at version `n-1` can start against a database schema applied by version `n`. This means that you can roll back the application without manually resetting the database first, as long as you stay within the supported window:

- **Supported**: Roll back one minor version (for example, `v0.10.x` → `v0.9.x`).
- **Requires DB reset, one minor at a time**: Roll back more than one minor version. Be sure to roll back only one minor version at a time, as skipping a minor version can result in data loss.

When the controller detects that the database schema is ahead of its own migration files, it starts in compatibility mode and skips migration application. The database schema is left unchanged.

### Steps to roll back

1. Save the kagent version you want to roll back to in an environment variable.
```bash
export ROLLBACK_VERSION=<previous-version>
Comment thread
Rachael-Graham marked this conversation as resolved.
```

2. Roll back the kagent chart.
```bash
helm upgrade kagent oci://ghcr.io/kagent-dev/kagent/helm/kagent \
--namespace kagent \
-f values.yaml \
--version $ROLLBACK_VERSION
```

3. Roll back the kagent-crds chart.
```bash
helm upgrade kagent-crds oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \
--namespace kagent \
--version $ROLLBACK_VERSION
```

4. Verify that the rollback succeeded.
```bash
kubectl get pods -n kagent
```

### Reset the database before rolling back further

If you need to roll back more than one minor kagent version, be sure to roll back one minor version at a time. Kagent guarantees backwards compatibility of only one minor version, so skipping minor versions can result in data loss.

You have two options for resetting the database schema at each step.

#### Option 1: Restore from snapshot

If you took a snapshot before upgrading, you can restore it directly. This is simpler than running down migrations, but any data written after the snapshot was taken is lost.

```bash
pg_restore \
--clean \
--dbname="postgres://<user>:<password>@<host>:5432/<dbname>" \
kagent-pre-upgrade-snapshot.dump
```

After restoring, follow the steps to [roll back the kagent application](#steps-to-roll-back).

#### Option 2: Run down migrations

Use `golang-migrate` to run down migrations one minor version at a time. This preserves data written after the snapshot but requires more steps.

The source must be the current (newer) version that you are rolling back from, because it contains the down migrations needed to reverse the schema changes. The `goto` target is the highest migration sequence number present in the version that you are rolling back to.

For example, `v0.9.9` has migrations up to `000005_a2a_protocol_version.up.sql` and `v0.9.3` has migrations up to `000004_feedback_single_pk.up.sql`. To roll back from `v0.9.9` to `v0.9.3`, you set `CURRENT_VERSION=0.9.9`, `ROLLBACK_VERSION=0.9.3`, and run `goto 4` because you want to go back to migration sequence 4 (v0.9.3's `000004`).

1. Save your current kagent version and the kagent version you want to roll back to in environment variables.
```bash
export CURRENT_VERSION=<current-version>
export ROLLBACK_VERSION=<previous-version>
```

2. Install [`golang-migrate`](https://github.com/golang-migrate/migrate/tree/master/cmd/migrate).

3. Stop the kagent controller.
```bash
kubectl -n kagent scale deploy/kagent-controller --replicas=0
```

4. Open the core migration directory for your rollback version and save the sequence number of the highest-numbered file in an environment variable, such as `4` from the previous v0.9.3 `goto 4` example.
```bash
open "https://github.com/kagent-dev/kagent/tree/v${ROLLBACK_VERSION}/go/core/pkg/migrations/core/"
```
```bash
export ROLLBACK_MIGRATION_VERSION=<sequence-number>
```

5. Reset the core track. The `github://` source references the migration files directly from the release tag without a local checkout. For the database connection string, see [Database configuration](/docs/kagent/operations/operational-considerations#database-configuration).
```bash
migrate \
-source "github://kagent-dev/kagent/go/core/pkg/migrations/core#v$CURRENT_VERSION" \
-database "postgres://<user>:<password>@<host>:5432/<dbname>?sslmode=require&x-migrations-table=schema_migrations" \
goto $ROLLBACK_MIGRATION_VERSION
```

6. If vector features are enabled, reset the vector track as well.
1. Open the vector migration directory for your rollback version and save the sequence number of the highest-numbered file in an environment variable.
```bash
open "https://github.com/kagent-dev/kagent/tree/v${ROLLBACK_VERSION}/go/core/pkg/migrations/vector/"
export ROLLBACK_VECTOR_MIGRATION_VERSION=<sequence-number>
```
2. Reset the vector track.
```bash
migrate \
-source "github://kagent-dev/kagent/go/core/pkg/migrations/vector#v$CURRENT_VERSION" \
-database "postgres://<user>:<password>@<host>:5432/<dbname>?sslmode=require&x-migrations-table=vector_schema_migrations" \
goto $ROLLBACK_VECTOR_MIGRATION_VERSION
```

7. After the database is at the correct schema version, follow the steps to [roll back the kagent application](#steps-to-roll-back).
Loading