From 07ca2d2f3aa40b5d253cb174d9d9462380c1e3df Mon Sep 17 00:00:00 2001 From: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Mon, 3 Aug 2026 08:38:56 +0100 Subject: [PATCH 1/4] docs: restructure self-hosting kubernetes guide into evaluation and production paths --- docs/self-hosting/docker.mdx | 10 +- docs/self-hosting/kubernetes.mdx | 529 +++++++++++++++---------------- 2 files changed, 262 insertions(+), 277 deletions(-) diff --git a/docs/self-hosting/docker.mdx b/docs/self-hosting/docker.mdx index 065b4ad55da..a4fabfcc08d 100644 --- a/docs/self-hosting/docker.mdx +++ b/docs/self-hosting/docker.mdx @@ -365,12 +365,10 @@ We patch the latest released version line only, so keep an eye on new releases t You can also lock the versions of the bundled services, for example with `CLICKHOUSE_IMAGE_TAG`. If you do, or if you bring your own ClickHouse via `CLICKHOUSE_URL`, note that Trigger.dev requires ClickHouse 25.8 or newer. - The bundled ClickHouse now uses the official `clickhouse/clickhouse-server` image. Your existing - data volume carries over automatically. If you previously pinned `CLICKHOUSE_IMAGE_TAG` to a - Bitnami tag (for example `25.7.5-debian-12-r0`), update it to an official image tag such as - `26.2` — Bitnami tags don't exist in the official repository. Note the switch is one-way: the - official image takes ownership of the data files, so rolling back to the Bitnami image requires - manually restoring their previous owner (`chown -R 1001:1001` on the volume). + The bundled ClickHouse uses the official `clickhouse/clickhouse-server` image, and your existing + data volume carries over automatically on upgrade. If you pinned `CLICKHOUSE_IMAGE_TAG` to a + Bitnami tag, switch it to an official tag (for example `26.2`) — Bitnami tags don't exist in the + official repository. diff --git a/docs/self-hosting/kubernetes.mdx b/docs/self-hosting/kubernetes.mdx index 1939ff5dfb8..fa225c158e0 100644 --- a/docs/self-hosting/kubernetes.mdx +++ b/docs/self-hosting/kubernetes.mdx @@ -1,15 +1,41 @@ --- title: "Kubernetes" -description: "You can self-host Trigger.dev in Kubernetes using our official Helm chart." +description: "Deploy Trigger.dev to Kubernetes with our official Helm chart - one command for evaluation, external datastores for production." --- -The following instructions will help you deploy Trigger.dev to Kubernetes using our official Helm chart. Make sure to read the self-hosting [overview](/self-hosting/overview) first. +The official Helm chart installs the full Trigger.dev stack into a Kubernetes namespace. Read the self-hosting [overview](/self-hosting/overview) first. -As self-hosted deployments tend to have unique requirements and configurations, we don't provide specific advice for securing your deployment, scaling up, or improving reliability. +Pick a path and stay on it. Run the **evaluation install** to try Trigger.dev with everything bundled in-cluster - one `helm install`, default values, no external dependencies. Run the **production install** for real workloads: external Postgres, Redis, ClickHouse, and object storage, your own secrets, and HA on the datastores you already operate. Each configuration decision below belongs to one path or the other. -Should the burden ever get too much, we'd be happy to see you on [Trigger.dev cloud](https://trigger.dev/pricing) where we deal with these concerns for you. +This guide alone is unlikely to produce a hardened production deployment - securing, scaling, and reliability-tuning your cluster remain yours to own. Should the burden get too much, we'd be happy to see you on [Trigger.dev cloud](https://trigger.dev/pricing) where we deal with these concerns for you. -**Warning:** This guide alone is unlikely to result in a production-ready deployment. Security, scaling, and reliability concerns are not fully addressed here. +## Architecture + +The chart deploys a webapp, a supervisor, and the datastores they depend on. The supervisor pulls runs from the webapp and schedules each one onto a worker node, which executes the task in a container sized by its machine preset. + +```mermaid +flowchart TD + webapp["webapp
API + dashboard"] + supervisor["supervisor"] + workers["worker nodes
task containers"] + pg[("Postgres
primary DB")] + redis[("Redis
queue / state")] + ch[("ClickHouse
task events v2")] + obj[("object storage
MinIO / external S3")] + reg[("registry
deploy images")] + s2[("s2 / s2-lite
realtime streams v2")] + + webapp --> pg + webapp --> redis + webapp --> ch + webapp --> obj + webapp --> s2 + webapp -->|worker token| supervisor + supervisor --> workers + workers --> reg +``` + +Each component maps to a top-level key in `values.yaml`: `webapp`, `supervisor`, `postgres`, `redis`, `clickhouse`, `minio`/`s3`, `registry`, and `s2`. Bundled datastores carry a `deploy: true` default; setting `deploy: false` and an `external` block swaps in your own. Webapp and supervisor behaviour is driven by `config` and `extraEnvVars`. ## Requirements @@ -52,94 +78,108 @@ webapp: memory: 4Gi ``` -## Installation - -### Quick start - -1. Install with default values (for testing only): - -```bash -helm upgrade -n trigger --install trigger \ - oci://ghcr.io/triggerdotdev/charts/trigger \ - --version "^4.5.0" \ - --create-namespace -``` +## Evaluation install -2. Access the webapp: +Bundled datastores let you run the whole stack with a single command and no external services. The default values are insecure and are only suitable for testing - for real workloads, follow the [production install](#production-install) instead. -```bash -kubectl port-forward svc/trigger-webapp 3040:3030 -n trigger -``` + + + ```bash + helm upgrade -n trigger --install trigger \ + oci://ghcr.io/triggerdotdev/charts/trigger \ + --version "^4.5.0" \ + --create-namespace + ``` -3. Open the dashboard: `http://localhost:3040` + The chart deploys the webapp, supervisor, and bundled Postgres, Redis, ClickHouse, MinIO, registry, and s2-lite. + -4. Login with the magic link: + + ```bash + kubectl port-forward svc/trigger-webapp 3040:3030 -n trigger + ``` -```bash -# Check the webapp logs -kubectl logs -n trigger deployment/trigger-webapp | grep -A1 "magic link" -``` + The dashboard is now reachable at `http://localhost:3040`. + -## Upgrading + + Default installs print the login link to the webapp logs instead of sending email. -`helm upgrade` keeps generated secrets, PVCs, and datastore volumes, so most upgrades need no preparation. These chart versions are the exceptions. + ```bash + kubectl logs -n trigger deployment/trigger-webapp | grep -A1 "magic link" + ``` -### 4.5.6: two new required secret keys + Open the printed link to sign in. + + -The webapp now reads `PROVIDER_SECRET` and `COORDINATOR_SECRET`. When you supply `secrets.existingSecret` the chart generates nothing and reads every key from your Secret, so both keys must exist there before you upgrade. Inline `secrets.*` values are ignored while `existingSecret` is set. +### Verify it -Add them with two fresh 32-character hex values: +Confirm every component reached a healthy state: ```bash -kubectl patch secret my-trigger-secrets -n trigger --type merge -p \ - "{\"stringData\":{\"PROVIDER_SECRET\":\"$(openssl rand -hex 16)\",\"COORDINATOR_SECRET\":\"$(openssl rand -hex 16)\"}}" +kubectl get pods -n trigger ``` -The chart checks your Secret in a pre-upgrade hook and aborts with the list of missing keys, leaving the running release untouched. Without that check a missing key surfaces as a `CreateContainerConfigError` partway through the webapp rollout. +The webapp, supervisor, and each bundled datastore pod should report `Running` with all containers `READY`, and the bootstrap and migration jobs should show `Completed`. If a pod is stuck in `Pending` or `CrashLoopBackOff`, jump to [Troubleshooting](#troubleshooting). - - GitOps tools that render with `helm template` have no cluster access, so the check is - skipped and a missing key still reaches the rollout. Verify the keys yourself before syncing. - +Once you can sign in and trigger a run, you've validated the stack. For a deployment that survives restarts, upgrades, and load, move on to the production install. -### 4.5.6: ClickHouse credentials moved +## Production install -The bundled ClickHouse password moved out of the subchart-generated Secret and into the chart-managed datastore Secret: +Running external datastores is the requirement that separates a production install from an evaluation one. You supply your own Postgres, Redis, ClickHouse, and object storage, manage secrets yourself, and point the chart at each service. The steps below layer onto the same `helm upgrade` command, driven by a custom values file. -| | Up to 4.5.5 | 4.5.6 and later | -| ------ | ---------------------- | --------------------------- | -| Secret | `-clickhouse` | `trigger-datastore` | -| Key | `admin-password` | `clickhouse-admin-password` | +### Secrets -The webapp and the ClickHouse server both read the new location, so the upgrade itself needs no action. Repoint anything outside the chart that reads the old Secret — a maintenance CronJob, a Grafana datasource, an external secret sync: - -```bash -kubectl get secret trigger-datastore -n trigger \ - -o jsonpath='{.data.clickhouse-admin-password}' | base64 -d -``` +Application, control-plane, and bundled-datastore secrets left unset are generated on first install and **retained across `helm upgrade`** - they are never rotated automatically, so sessions, encrypted data, and datastore volumes survive upgrades. This auto-generation is the default and needs no configuration. -The same move applies to the bundled PostgreSQL (`postgres-password`) and MinIO (`minio-root-user`, `minio-root-password`). +For production, supply your own Secret instead so credentials live outside the release. With `secrets.existingSecret` set, the chart generates nothing and reads every key from your Secret; a preflight check fails the install - before touching a running release - and lists any keys it can't find. -### Bitnami ClickHouse to the official image +```yaml +# Leave these unset to have the chart auto-generate strong values on first +# install (retained across upgrades). Set them explicitly only if you need to +# control the value - e.g. sharing MANAGED_WORKER_SECRET with an external +# supervisor - or use an existingSecret. +secrets: + enabled: true + sessionSecret: "your-32-char-hex-secret-1" + magicLinkSecret: "your-32-char-hex-secret-2" + # ... -The bundled ClickHouse runs the official `clickhouse/clickhouse-server` image instead of the Bitnami subchart. The chart adopts your existing data volume automatically, with three caveats covered under [ClickHouse](#clickhouse) in external services: rendering without cluster access, pinned Bitnami image tags, and storage that doesn't support `fsGroup`. +# Recommended: existingSecret, must contain at least the following keys: +# - SESSION_SECRET +# - MAGIC_LINK_SECRET +# - ENCRYPTION_KEY +# - PROVIDER_SECRET +# - COORDINATOR_SECRET +# - MANAGED_WORKER_SECRET +# Plus s3-auth-access-key-id and s3-auth-secret-access-key if you deploy the +# bundled MinIO with s3.auth.existingSecret cleared. The chart lists any keys +# it can't find and fails the install before touching a running release. +secrets: + enabled: false + existingSecret: "your-existing-secret" +``` - - The data volume is `ReadWriteOnce`, so during the upgrade the new ClickHouse pod may sit in - `ContainerCreating` with a multi-attach warning for a minute while the old pod still holds the - volume. This clears itself once the old pod is removed later in the same upgrade - don't abort. - +`PROVIDER_SECRET` and `COORDINATOR_SECRET` are not new secrets. Chart 4.5.6 made the webapp reject the shared, previously-published defaults these used to fall back to, so an existing Secret must now carry strong, unique values for both: -## Configuration +```bash +kubectl patch secret my-trigger-secrets -n trigger --type merge -p \ + "{\"stringData\":{\"PROVIDER_SECRET\":\"$(openssl rand -hex 16)\",\"COORDINATOR_SECRET\":\"$(openssl rand -hex 16)\"}}" +``` -Most values map directly to the environment variables documented in the [webapp](/self-hosting/env/webapp) and [supervisor](/self-hosting/env/supervisor) environment variable overview. +There is no clean migration for a compromised `ENCRYPTION_KEY`: changing it makes existing encrypted data unreadable. If a deployment is still running a previously published default and cannot rotate yet, set `ALLOW_INSECURE_DEFAULT_SECRETS=true` on the webapp to keep booting while you plan a migration. -**Naming convention:** + + GitOps tools that render with `helm template` (e.g. Argo CD) cannot read the existing + secret, so they regenerate auto-generated values on every sync - which rotates them. If you + deploy via GitOps, always supply your own `secrets.existingSecret` (and datastore + credentials) so nothing is generated in-cluster. + -- Environment variables use `UPPER_SNAKE_CASE` -- Helm values use `camelCase` +### Custom values -**Example mapping:** +Most values map directly to the environment variables documented in the [webapp](/self-hosting/env/webapp) and [supervisor](/self-hosting/env/supervisor) environment variable overview. Environment variables use `UPPER_SNAKE_CASE`; Helm values use `camelCase`: ```bash # Environment variable @@ -150,9 +190,7 @@ config: appOrigin: "https://trigger.example.com" ``` -### Default values - -The following commands will display the default values: +Print the full default surface for the version you're deploying: ```bash # Specific version @@ -164,37 +202,9 @@ helm show values oci://ghcr.io/triggerdotdev/charts/trigger \ --version "^4.5.0" ``` -### Custom values - -The default values are insecure and are only suitable for testing. You will need to configure your own secrets as a bare minimum. - -Create a `values-custom.yaml` file to override the defaults. For example: +Create a `values-custom.yaml` that overrides the defaults with your URLs and resource requests: ```yaml -# Leave these unset to have the chart auto-generate strong values on first -# install (retained across upgrades). Set them explicitly only if you need to -# control the value - e.g. sharing MANAGED_WORKER_SECRET with an external -# supervisor - or use an existingSecret. -secrets: - enabled: true - sessionSecret: "your-32-char-hex-secret-1" - magicLinkSecret: "your-32-char-hex-secret-2" - # ... - -# Recommended: existingSecret, must contain at least the following keys: -# - SESSION_SECRET -# - MAGIC_LINK_SECRET -# - ENCRYPTION_KEY -# - PROVIDER_SECRET -# - COORDINATOR_SECRET -# - MANAGED_WORKER_SECRET -# Plus s3-auth-access-key-id and s3-auth-secret-access-key if you deploy the -# bundled MinIO with s3.auth.existingSecret cleared. The chart lists any keys -# it can't find and fails the install before touching a running release. -secrets: - enabled: false - existingSecret: "your-existing-secret" - # Application URLs config: appOrigin: "https://trigger.example.com" @@ -231,27 +241,7 @@ helm upgrade -n trigger --install trigger \ -f values-custom.yaml ``` -### Secret generation and rotation - -Application, control-plane, and bundled-datastore secrets left unset are generated on -first install and **retained across `helm upgrade`** - they are never rotated -automatically, so sessions, encrypted data, and datastore volumes survive upgrades. - - - GitOps tools that render with `helm template` (e.g. Argo CD) cannot read the existing - secret, so they regenerate these values on every sync - which rotates them. If you - deploy via GitOps, always supply your own `secrets.existingSecret` (and datastore - credentials) so nothing is generated in-cluster. - - -There is no clean migration for a compromised `ENCRYPTION_KEY`: changing it makes -existing encrypted data unreadable. If a deployment is still running a previously -published default and cannot rotate yet, set `ALLOW_INSECURE_DEFAULT_SECRETS=true` on -the webapp to keep booting while you plan a migration. - -### Extra env - -You can set extra environment variables on all services. For example: +Set extra environment variables on any service: ```yaml webapp: @@ -260,9 +250,7 @@ webapp: value: "extra-value" ``` -### Extra annotations - -You can set extra annotations on all services. For example: +Set extra annotations on any service: ```yaml webapp: @@ -272,7 +260,7 @@ webapp: ### External services -You can disable the built-in services and use external services instead. The chart supports both direct configuration and existing Kubernetes secrets for secure credential management. +Disable each built-in datastore and point the chart at your own. Every service supports direct configuration or an existing Kubernetes secret; use existing secrets to keep credentials out of `values.yaml` and Helm releases. #### PostgreSQL @@ -330,35 +318,11 @@ redis: #### ClickHouse -Trigger.dev requires ClickHouse 25.8 or newer. - - - When upgrading from a chart version that bundled ClickHouse via the Bitnami subchart, the chart - automatically adopts the existing data volume, so no manual migration is needed. If you render - manifests without cluster access (for example with GitOps tools that use `helm template`), set - `clickhouse.persistence.existingClaim` to the old PVC name - (`data--clickhouse-shard0-0`) to keep your data — auto-detection can't run there, and - skipping this starts ClickHouse on a fresh empty volume. If that happened, your old data is - still on the old PVC: delete the ClickHouse StatefulSet with `--cascade=orphan` (its volume - configuration is immutable), set `existingClaim`, and sync again. If you render without cluster - access, pin `clickhouse.persistence.existingClaim` for good once the volume is adopted: a later - render where the lookup can't see the PVC would otherwise try to re-add `volumeClaimTemplates`, - which the API server rejects on an existing StatefulSet. - - - - If you pinned `clickhouse.image` to a Bitnami repository or tag in your values, update it to - the official `clickhouse/clickhouse-server` image — Bitnami tags don't exist there. The - bundled ClickHouse is single-node: the old Bitnami subchart keys (`shards`, `replicaCount`, - `keeper`) are no longer supported, so use an external ClickHouse for clustered setups. - +Migration from the old Bitnami subchart to the official `clickhouse/clickhouse-server` image is automatic - a normal upgrade moves you across with no manual step. - When a volume is adopted from the Bitnami-based chart, a one-time init container fixes its - ownership for the non-root ClickHouse server. This runs automatically on `helm upgrade` (and - when you set `clickhouse.persistence.existingClaim`), so storage that doesn't support `fsGroup` - ownership changes (NFS, hostPath, local-path) works without manual steps. Set - `clickhouse.volumePermissions.enabled: true` to force it in other cases. + Trigger.dev requires ClickHouse 25.8 or newer. The bundled ClickHouse is single-node — run + ClickHouse externally for a clustered or replicated setup. **Direct configuration:** @@ -375,8 +339,7 @@ clickhouse: An inline external `username`/`password` is percent-encoded into the connection URL for you, so - store the **raw** value - special characters like `@ : / %` are handled automatically. If you - previously percent-encoded the password by hand to work around this, switch back to the raw value. + store the **raw** value - special characters like `@ : / %` are handled automatically. (Credentials from `existingSecret` are injected at runtime and are unaffected.) @@ -393,7 +356,9 @@ clickhouse: # existingSecretKey: "clickhouse-password" # default (optional) ``` -#### S3 Object Storage +#### Object storage + +Object storage holds deploy artifacts and large payloads. See the [Docker object storage setup](/self-hosting/docker#object-storage) for conceptual information. Disable the bundled MinIO and point at any S3-compatible endpoint. **Direct configuration:** @@ -424,7 +389,7 @@ s3: ### PostgreSQL SSL with custom CA certificates -When connecting to PostgreSQL instances that require custom CA certificates (such as AWS RDS with SSL verification), you can mount the CA certificate as a volume and configure the webapp to use it: +When connecting to PostgreSQL instances that require custom CA certificates (such as AWS RDS with SSL verification), mount the CA certificate as a volume and configure the webapp to use it: ```yaml postgres: @@ -465,99 +430,9 @@ webapp: - Compatible with secret management tools (External Secrets Operator, etc.) - Follows Kubernetes security best practices -## DNS performance - -For production clusters we recommend deploying [NodeLocal DNSCache](https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/). DNS queries — especially to managed Postgres or Redis endpoints — can be very slow under Kubernetes' default resolver, and a node-local cache typically gives a large step change in latency and throughput across the cluster. - -The default `ndots: 5` setting also forces every cluster search domain to be tried before resolving hostnames with fewer dots (the case for most external database hosts). Lowering `ndots` to `1` on the webapp and supervisor pods avoids those extra round-trips. +### Registry setup -## Task events - -By default, task events (timeline, logs, spans) are stored in PostgreSQL. For production deployments we recommend storing them in ClickHouse instead, it scales to much higher volumes and avoids unbounded growth of the `TaskEvent` table. - -ClickHouse is already deployed by the chart, so no extra services are required. To enable, set `EVENT_REPOSITORY_DEFAULT_STORE` on the webapp via `extraEnvVars`: - -```yaml -webapp: - extraEnvVars: - - name: EVENT_REPOSITORY_DEFAULT_STORE - value: "clickhouse_v2" -``` - -This only affects new runs; existing runs continue to read from wherever their events were originally stored. - -## Realtime streams - -Realtime streams power AI-agent token streaming and run streams. They default to **v2**, backed by the bundled `s2` deployment — [s2-lite](https://s2.dev), the open-source, self-hostable S2 server. The chart deploys it with a persistent volume, so no extra services are required. - -To fall back to the Redis-backed **v1** streams, set the default version to `v1`: - -```yaml -s2: - defaultStreamVersion: "v1" -``` - -To use a hosted S2 at [s2.dev](https://s2.dev) instead of the bundled s2-lite, disable the bundled deployment and point at your basin. Supply the access token via an existing secret: - -```yaml -s2: - deploy: false - skipAccessTokens: false - external: - endpoint: "https://your-basin.b.aws.s2.dev/v1" - existingSecret: "s2-credentials" - existingSecretAccessTokenKey: "access-token" - basin: "your-basin" -``` - -To disable realtime streams v2 entirely and use v1, set `s2.deploy: false` with no external endpoint. See `helm show values` for all `s2` options. - -## Worker token - -When using the default bootstrap configuration, worker creation and authentication is handled automatically. The webapp generates a worker token and makes it available to the supervisor via a shared volume. - -### Bootstrap (default) - -```yaml -webapp: - bootstrap: - enabled: true - workerGroupName: "bootstrap" -``` - -### Manual - -If you need to set up workers separately or use a custom token: - -1. Get the worker token from the webapp logs: - -```bash -kubectl logs deployment/trigger-webapp -n trigger | grep -A15 "Worker Token" -``` - -2. Create a secret with the token: - -```bash -kubectl create secret generic worker-token \ - --from-literal=token=tr_wgt_your_token_here \ - -n trigger -``` - -3. Configure the supervisor to use the secret: - -```yaml -supervisor: - bootstrap: - enabled: false - workerToken: - secret: - name: "worker-token" - key: "token" -``` - -## Registry setup - -See the [Docker registry setup](/self-hosting/docker#registry-setup) for conceptual information. The configuration is specified in your `values.yaml`: +Built deploy images are pushed to a registry. See the [Docker registry setup](/self-hosting/docker#registry-setup) for conceptual information. Configure an external registry in your `values.yaml`: ```yaml # Use external registry (recommended) @@ -579,28 +454,15 @@ registry: and additional cluster configuration. Use an external registry for production. -## Object storage - -See the [Docker object storage setup](/self-hosting/docker#object-storage) for conceptual information. The defaults will use built-in MinIO, but you can use an external S3-compatible storage. The configuration is specified in your `values.yaml`: +### DNS performance -```yaml -# Use external S3-compatible storage -minio: - deploy: false - external: - url: "https://s3.amazonaws.com" - # or: "https://your-minio.com:9000" +For production clusters we recommend deploying [NodeLocal DNSCache](https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/). DNS queries — especially to managed Postgres or Redis endpoints — can be very slow under Kubernetes' default resolver, and a node-local cache typically gives a large step change in latency and throughput across the cluster. -# Configure credentials -secrets: - objectStore: - accessKeyId: "admin" - secretAccessKey: "very-safe-password" -``` +The default `ndots: 5` setting also forces every cluster search domain to be tried before resolving hostnames with fewer dots (the case for most external database hosts). Lowering `ndots` to `1` on the webapp and supervisor pods avoids those extra round-trips. -## Authentication +### Authentication -Authentication options are identical to the [Docker-based installation](/self-hosting/docker#authentication). The configuration is specified in your `values.yaml`: +Authentication options are identical to the [Docker-based installation](/self-hosting/docker#authentication). Configure the provider in your `values.yaml`: **GitHub OAuth:** @@ -637,7 +499,92 @@ webapp: value: "^(user1@company\\.com|user2@company\\.com)$" ``` -## Version locking +### Worker token + +The supervisor authenticates to the webapp with a worker token. The default bootstrap path generates one for you; supply your own token to manage workers separately. + +**Bootstrap (default):** the webapp generates a worker token and makes it available to the supervisor via a shared volume. + +```yaml +webapp: + bootstrap: + enabled: true + workerGroupName: "bootstrap" +``` + +**Manual:** to set up workers separately or use a custom token: + + + + ```bash + kubectl logs deployment/trigger-webapp -n trigger | grep -A15 "Worker Token" + ``` + + + + ```bash + kubectl create secret generic worker-token \ + --from-literal=token=tr_wgt_your_token_here \ + -n trigger + ``` + + + + ```yaml + supervisor: + bootstrap: + enabled: false + workerToken: + secret: + name: "worker-token" + key: "token" + ``` + + + +With external datastores, secrets, registry, and authentication in place, you have a production install. If you only wanted to try Trigger.dev, the [evaluation install](#evaluation-install) gets you there in one command. + +## Operating your deployment + +### Upgrading + +`helm upgrade` is safe to run as-is. Your generated secrets, data volumes, and the bundled ClickHouse are preserved and migrated automatically, so a normal upgrade needs no preparation. A few situations need one manual step first — expand any that apply to you. + + + + With `secrets.existingSecret` set, the chart reads every key from your Secret and generates none, so any key the app needs must already be present there. + + Chart 4.5.6 hardened this: the webapp now requires strong, unique `PROVIDER_SECRET` and `COORDINATOR_SECRET` values — it previously fell back to a shared, published default for these, which it now rejects at startup. Add both to your Secret before upgrading: + + ```bash + kubectl patch secret my-trigger-secrets -n trigger --type merge -p \ + "{\"stringData\":{\"PROVIDER_SECRET\":\"$(openssl rand -hex 16)\",\"COORDINATOR_SECRET\":\"$(openssl rand -hex 16)\"}}" + ``` + + If a required key is missing, the upgrade stops before touching your running release and tells you which to add. + + + + The bundled datastore passwords live in the `trigger-datastore` Secret. If a maintenance job, a dashboard, or a secret sync reads one directly, point it there: + + ```bash + kubectl get secret trigger-datastore -n trigger \ + -o jsonpath='{.data.clickhouse-admin-password}' | base64 -d + ``` + + The keys are `clickhouse-admin-password`, `postgres-password`, and `minio-root-user` / `minio-root-password`. + + + + During an upgrade the chart reads the cluster to keep your data and to check your Secret. Renderers that run offline can't do that, so set these yourself: + + - Point `clickhouse.persistence.existingClaim` at your ClickHouse data volume (`data--clickhouse-shard0-0`) and keep it set on future syncs, so ClickHouse reuses its data instead of starting on an empty volume. + - Confirm `PROVIDER_SECRET` and `COORDINATOR_SECRET` are already in your Secret before syncing. + - If you pinned `clickhouse.image` to a Bitnami tag, switch it to an official `clickhouse/clickhouse-server` tag. + + + +### Version locking Trigger.dev 4.5.0 is the last version we officially support for running v3 (SDK v3) tasks. If @@ -645,7 +592,7 @@ webapp: later reject v3 triggers and deploys with an upgrade message. -You can lock versions in two ways: +Lock versions in two ways. **Helm chart version (recommended):** @@ -677,7 +624,7 @@ supervisor: The chart version's `appVersion` field determines the default image tags. Newer image tags may be incompatible with older chart versions and vice versa. -## Troubleshooting +### Troubleshooting **Check logs:** @@ -722,18 +669,58 @@ kubectl delete namespace trigger See the [Docker troubleshooting](/self-hosting/docker#troubleshooting) section for more information. -## CLI usage +### Task events + +By default, task events (timeline, logs, spans) are stored in PostgreSQL. For production deployments we recommend storing them in ClickHouse instead, it scales to much higher volumes and avoids unbounded growth of the `TaskEvent` table. + +ClickHouse is already deployed by the chart, so no extra services are required. To enable, set `EVENT_REPOSITORY_DEFAULT_STORE` on the webapp via `extraEnvVars`: + +```yaml +webapp: + extraEnvVars: + - name: EVENT_REPOSITORY_DEFAULT_STORE + value: "clickhouse_v2" +``` + +This only affects new runs; existing runs continue to read from wherever their events were originally stored. + +### Realtime streams + +Realtime streams power AI-agent token streaming and run streams. They default to **v2**, backed by the bundled `s2` deployment — [s2-lite](https://s2.dev), the open-source, self-hostable S2 server. The chart deploys it with a persistent volume, so no extra services are required. + +To fall back to the Redis-backed **v1** streams, set the default version to `v1`: + +```yaml +s2: + defaultStreamVersion: "v1" +``` + +To use a hosted S2 at [s2.dev](https://s2.dev) instead of the bundled s2-lite, disable the bundled deployment and point at your basin. Supply the access token via an existing secret: + +```yaml +s2: + deploy: false + skipAccessTokens: false + external: + endpoint: "https://your-basin.b.aws.s2.dev/v1" + existingSecret: "s2-credentials" + existingSecretAccessTokenKey: "access-token" + basin: "your-basin" +``` + +To disable realtime streams v2 entirely and use v1, set `s2.deploy: false` with no external endpoint. See `helm show values` for all `s2` options. + +### CLI usage See the [Docker CLI usage](/self-hosting/docker#cli-usage) section, the commands are identical regardless of deployment method. -## CI / GitHub Actions +### CI / GitHub Actions -When running the CLI in a CI environment, your login profiles won't be available. Instead, you can use the `TRIGGER_API_URL` and `TRIGGER_ACCESS_TOKEN` environment -variables to point at your self-hosted instance and authenticate. +When running the CLI in a CI environment, your login profiles won't be available. Instead, use the `TRIGGER_API_URL` and `TRIGGER_ACCESS_TOKEN` environment variables to point at your self-hosted instance and authenticate. For more detailed instructions, see the [GitHub Actions guide](/github-actions). -## Telemetry +### Telemetry By default, the Trigger.dev webapp sends telemetry data to our servers. This data is used to improve the product and is not shared with third parties. To disable telemetry, set in your `values.yaml`: From 05b6f73cef55e128d45b998613eeff06abb8c4c0 Mon Sep 17 00:00:00 2001 From: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Mon, 3 Aug 2026 08:57:46 +0100 Subject: [PATCH 2/4] docs: correct bundled-component defaults and object-storage config keys --- docs/self-hosting/kubernetes.mdx | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/docs/self-hosting/kubernetes.mdx b/docs/self-hosting/kubernetes.mdx index fa225c158e0..bff0eb5ff8c 100644 --- a/docs/self-hosting/kubernetes.mdx +++ b/docs/self-hosting/kubernetes.mdx @@ -35,7 +35,7 @@ flowchart TD workers --> reg ``` -Each component maps to a top-level key in `values.yaml`: `webapp`, `supervisor`, `postgres`, `redis`, `clickhouse`, `minio`/`s3`, `registry`, and `s2`. Bundled datastores carry a `deploy: true` default; setting `deploy: false` and an `external` block swaps in your own. Webapp and supervisor behaviour is driven by `config` and `extraEnvVars`. +Each component maps to a top-level key in `values.yaml`: `webapp`, `supervisor`, `postgres`, `redis`, `electric`, `clickhouse`, `s3` (object storage), `s2`, and `registry`. The datastores default to `deploy: true` and run in-cluster; set `deploy: false` with an `external` block to bring your own. The registry is the exception - it defaults to `deploy: false`, so you point it at an external one. Webapp and supervisor behaviour is driven by `config` and `extraEnvVars`. ## Requirements @@ -91,7 +91,7 @@ Bundled datastores let you run the whole stack with a single command and no exte --create-namespace ``` - The chart deploys the webapp, supervisor, and bundled Postgres, Redis, ClickHouse, MinIO, registry, and s2-lite. + The chart deploys the webapp, supervisor, and bundled Postgres, Redis, Electric, ClickHouse, MinIO, and s2-lite. The registry is not deployed by default - configure an external one before deploying tasks. @@ -121,7 +121,7 @@ Confirm every component reached a healthy state: kubectl get pods -n trigger ``` -The webapp, supervisor, and each bundled datastore pod should report `Running` with all containers `READY`, and the bootstrap and migration jobs should show `Completed`. If a pod is stuck in `Pending` or `CrashLoopBackOff`, jump to [Troubleshooting](#troubleshooting). +The webapp, supervisor, and each bundled datastore pod should report `Running` with all containers `READY`. Migrations and bootstrap run inside the webapp pod rather than as separate jobs, so if the webapp stays unready, check its logs. If a pod is stuck in `Pending` or `CrashLoopBackOff`, jump to [Troubleshooting](#troubleshooting). Once you can sign in and trigger a run, you've validated the stack. For a deployment that survives restarts, upgrades, and load, move on to the production install. @@ -363,9 +363,8 @@ Object storage holds deploy artifacts and large payloads. See the [Docker object **Direct configuration:** ```yaml -minio: - deploy: false s3: + deploy: false external: endpoint: "https://s3.amazonaws.com" accessKeyId: "my-access-key" @@ -375,16 +374,14 @@ s3: **Using existing secrets (recommended):** ```yaml -minio: - deploy: false s3: + deploy: false external: endpoint: "https://s3.amazonaws.com" existingSecret: "s3-credentials" - # Optional: Use secretKeys to specify the key names in the secret - # secretKeys: - # accessKeyIdKey: "access-key-id" # default - # secretAccessKeyKey: "secret-access-key" # default + # Optional: override the key names in the secret + # existingSecretAccessKeyIdKey: "access-key-id" # default + # existingSecretSecretAccessKeyKey: "secret-access-key" # default ``` ### PostgreSQL SSL with custom CA certificates From c5f5710552e80f47344a39c8c3f63af39954563d Mon Sep 17 00:00:00 2001 From: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Mon, 3 Aug 2026 10:07:23 +0100 Subject: [PATCH 3/4] docs: correct helm value paths and fix remaining self-hosting doc errors --- docs/self-hosting/env/webapp.mdx | 2 +- docs/self-hosting/kubernetes.mdx | 39 +++++++++++--------------------- 2 files changed, 14 insertions(+), 27 deletions(-) diff --git a/docs/self-hosting/env/webapp.mdx b/docs/self-hosting/env/webapp.mdx index 09416f0f38f..82dd6958a60 100644 --- a/docs/self-hosting/env/webapp.mdx +++ b/docs/self-hosting/env/webapp.mdx @@ -12,7 +12,7 @@ mode: "wide" | `MAGIC_LINK_SECRET` | Yes | — | Magic link encryption secret. Run: `openssl rand -hex 16` | | `ENCRYPTION_KEY` | Yes | — | Secret store encryption key. Run: `openssl rand -hex 16` | | `MANAGED_WORKER_SECRET` | Yes | — | Managed worker secret. Must be set and match supervisor. Run: `openssl rand -hex 32` | -| `ALLOW_INSECURE_DEFAULT_SECRETS` | No | false | Boot even if a secret is still a known-insecure published default. Temporary escape hatch for values you can't safely rotate yet (see [Secret generation and rotation](/self-hosting/kubernetes#secret-generation-and-rotation)). | +| `ALLOW_INSECURE_DEFAULT_SECRETS` | No | false | Boot even if a secret is still a known-insecure published default. Temporary escape hatch for values you can't safely rotate yet (see [Secrets](/self-hosting/kubernetes#secrets)). | | **Domains & ports** | | | | | `REMIX_APP_PORT` | No | 3030 | Remix app port. | | `APP_ORIGIN` | Yes | http://localhost:3030 | App origin URL. | diff --git a/docs/self-hosting/kubernetes.mdx b/docs/self-hosting/kubernetes.mdx index bff0eb5ff8c..0326ec94ded 100644 --- a/docs/self-hosting/kubernetes.mdx +++ b/docs/self-hosting/kubernetes.mdx @@ -35,7 +35,7 @@ flowchart TD workers --> reg ``` -Each component maps to a top-level key in `values.yaml`: `webapp`, `supervisor`, `postgres`, `redis`, `electric`, `clickhouse`, `s3` (object storage), `s2`, and `registry`. The datastores default to `deploy: true` and run in-cluster; set `deploy: false` with an `external` block to bring your own. The registry is the exception - it defaults to `deploy: false`, so you point it at an external one. Webapp and supervisor behaviour is driven by `config` and `extraEnvVars`. +Each component maps to a top-level key in `values.yaml`: `webapp`, `supervisor`, `postgres`, `redis`, `electric`, `clickhouse`, `s3` (object storage), `s2`, and `registry`. The datastores default to `deploy: true` and run in-cluster; set `deploy: false` with an `external` block to bring your own. The registry is the exception - it defaults to `deploy: false`, so you point it at an external one. Webapp settings live under `webapp`, supervisor settings under `supervisor.config`, and each service takes `extraEnvVars` for anything else. ## Requirements @@ -135,27 +135,19 @@ Application, control-plane, and bundled-datastore secrets left unset are generat For production, supply your own Secret instead so credentials live outside the release. With `secrets.existingSecret` set, the chart generates nothing and reads every key from your Secret; a preflight check fails the install - before touching a running release - and lists any keys it can't find. +To pin values inline instead of letting the chart generate them - for example to share `MANAGED_WORKER_SECRET` with an external supervisor - set them under `secrets`: + ```yaml -# Leave these unset to have the chart auto-generate strong values on first -# install (retained across upgrades). Set them explicitly only if you need to -# control the value - e.g. sharing MANAGED_WORKER_SECRET with an external -# supervisor - or use an existingSecret. secrets: enabled: true sessionSecret: "your-32-char-hex-secret-1" magicLinkSecret: "your-32-char-hex-secret-2" # ... +``` -# Recommended: existingSecret, must contain at least the following keys: -# - SESSION_SECRET -# - MAGIC_LINK_SECRET -# - ENCRYPTION_KEY -# - PROVIDER_SECRET -# - COORDINATOR_SECRET -# - MANAGED_WORKER_SECRET -# Plus s3-auth-access-key-id and s3-auth-secret-access-key if you deploy the -# bundled MinIO with s3.auth.existingSecret cleared. The chart lists any keys -# it can't find and fails the install before touching a running release. +To use an existing Secret instead, leave the values unset and set `existingSecret`. It must contain at least `SESSION_SECRET`, `MAGIC_LINK_SECRET`, `ENCRYPTION_KEY`, `PROVIDER_SECRET`, `COORDINATOR_SECRET`, and `MANAGED_WORKER_SECRET` - plus `s3-auth-access-key-id` and `s3-auth-secret-access-key` if you deploy the bundled MinIO with `s3.auth.existingSecret` cleared: + +```yaml secrets: enabled: false existingSecret: "your-existing-secret" @@ -186,7 +178,7 @@ Most values map directly to the environment variables documented in the [webapp] APP_ORIGIN=https://trigger.example.com # Becomes Helm value -config: +webapp: appOrigin: "https://trigger.example.com" ``` @@ -205,14 +197,11 @@ helm show values oci://ghcr.io/triggerdotdev/charts/trigger \ Create a `values-custom.yaml` that overrides the defaults with your URLs and resource requests: ```yaml -# Application URLs -config: +# Application URLs and resource limits +webapp: appOrigin: "https://trigger.example.com" loginOrigin: "https://trigger.example.com" apiOrigin: "https://trigger.example.com" - -# Resource limits -webapp: resources: requests: cpu: 1000m @@ -318,7 +307,7 @@ redis: #### ClickHouse -Migration from the old Bitnami subchart to the official `clickhouse/clickhouse-server` image is automatic - a normal upgrade moves you across with no manual step. +Migration from the old Bitnami subchart to the official `clickhouse/clickhouse-server` image is automatic - a normal upgrade moves you across with no manual step. The one exception: if you pinned `clickhouse.image` to a Bitnami tag, switch it to an official `clickhouse/clickhouse-server` tag first, since Bitnami tags don't exist in the official repository. Trigger.dev requires ClickHouse 25.8 or newer. The bundled ClickHouse is single-node — run @@ -438,8 +427,7 @@ registry: # Part of deployment image ref, for example: your-registry.example.com/your-company/proj_123:20250625.1.prod repositoryNamespace: "your-company" external: - host: "your-registry.example.com" - port: 5000 + host: "your-registry.example.com:5000" auth: enabled: true username: "your-username" @@ -447,7 +435,7 @@ registry: ``` - The internal registry (`registry.external: false`) is experimental and requires proper TLS setup + The internal registry (`registry.deploy: true`) is experimental and requires proper TLS setup and additional cluster configuration. Use an external registry for production. @@ -577,7 +565,6 @@ With external datastores, secrets, registry, and authentication in place, you ha - Point `clickhouse.persistence.existingClaim` at your ClickHouse data volume (`data--clickhouse-shard0-0`) and keep it set on future syncs, so ClickHouse reuses its data instead of starting on an empty volume. - Confirm `PROVIDER_SECRET` and `COORDINATOR_SECRET` are already in your Secret before syncing. - - If you pinned `clickhouse.image` to a Bitnami tag, switch it to an official `clickhouse/clickhouse-server` tag. From bc9a752e8ef0d4be7e208b0bf11e41dfc483a469 Mon Sep 17 00:00:00 2001 From: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Mon, 3 Aug 2026 10:29:17 +0100 Subject: [PATCH 4/4] docs: correct why an existing secret must carry the provider and coordinator keys --- docs/self-hosting/kubernetes.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/self-hosting/kubernetes.mdx b/docs/self-hosting/kubernetes.mdx index 0326ec94ded..2299373fd16 100644 --- a/docs/self-hosting/kubernetes.mdx +++ b/docs/self-hosting/kubernetes.mdx @@ -153,7 +153,7 @@ secrets: existingSecret: "your-existing-secret" ``` -`PROVIDER_SECRET` and `COORDINATOR_SECRET` are not new secrets. Chart 4.5.6 made the webapp reject the shared, previously-published defaults these used to fall back to, so an existing Secret must now carry strong, unique values for both: +The chart wires `PROVIDER_SECRET` and `COORDINATOR_SECRET` into the webapp deployment, and the pre-upgrade check requires both, so an existing Secret must include them. For a deployment that already runs, add them before upgrading: ```bash kubectl patch secret my-trigger-secrets -n trigger --type merge -p \ @@ -539,7 +539,7 @@ With external datastores, secrets, registry, and authentication in place, you ha With `secrets.existingSecret` set, the chart reads every key from your Secret and generates none, so any key the app needs must already be present there. - Chart 4.5.6 hardened this: the webapp now requires strong, unique `PROVIDER_SECRET` and `COORDINATOR_SECRET` values — it previously fell back to a shared, published default for these, which it now rejects at startup. Add both to your Secret before upgrading: + As of chart 4.5.6 the webapp deployment references `PROVIDER_SECRET` and `COORDINATOR_SECRET`, and the pre-upgrade check requires both, so an existing Secret must carry them. Add both before upgrading: ```bash kubectl patch secret my-trigger-secrets -n trigger --type merge -p \