-
Notifications
You must be signed in to change notification settings - Fork 2
docs: restructure self-hosting sidebar; add Production subsection and Support #705
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: docs/self-hosting
Are you sure you want to change the base?
Changes from all commits
354b76a
53eb7ab
07b23e7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,137 +1,35 @@ | ||
| --- | ||
| title: "Production Hardening & Operations" | ||
| description: "Production readiness checklist — replace secrets, configure TLS, set up managed data stores, run Postgres/ClickHouse/MinIO backups, and follow the upgrade runbook." | ||
| title: "Production" | ||
| description: "What to harden before a self-hosted Future AGI instance goes live" | ||
| --- | ||
|
|
||
| ## About | ||
| Everything past a local trial happens here. The default Docker Compose stack boots with placeholder secrets, no TLS, and compose-managed data stores. That's fine on a laptop. Before real traffic reaches the instance, work through the flow below in order, then keep each page as a runbook. | ||
|
|
||
| Run through this before exposing the stack to real users. Covers secrets, TLS, swapping in managed data stores, backup commands for Postgres/ClickHouse/MinIO, Prometheus monitoring, and the upgrade and rollback runbook. | ||
| ## In this page | ||
|
|
||
| ## Hardening checklist | ||
| Production readiness for a self-hosted instance breaks into five steps. Do them in order the first time, then keep each page as a runbook. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 'keep each page as a runbook' is already in the hero four lines up. Say it once |
||
|
|
||
| **Secrets** — replace all `CHANGEME` values before going live: | ||
| **Before you go live** | ||
|
|
||
| ```bash | ||
| openssl rand -hex 32 # SECRET_KEY, AGENTCC_INTERNAL_API_KEY | ||
| openssl rand -base64 24 # PG_PASSWORD, MINIO_ROOT_PASSWORD | ||
| ``` | ||
|
|
||
| **Runtime flags** in `.env`: | ||
| - `ENV_TYPE=prod` | ||
| - `FAST_STARTUP=false` | ||
| - `GRANIAN_WORKERS=<your CPU count>` | ||
|
|
||
| **TLS** — the frontend and backend don't terminate TLS. Put Caddy, nginx, or Traefik in front: | ||
|
|
||
| ``` | ||
| # Caddyfile (simplest — auto-issues Let's Encrypt certs) | ||
| app.yourcompany.com { reverse_proxy localhost:3000 } | ||
| api.yourcompany.com { reverse_proxy localhost:8000 } | ||
| ``` | ||
|
|
||
| After setting up TLS, set `VITE_HOST_API=https://api.yourcompany.com` in `.env` and rebuild: | ||
|
|
||
| ```bash | ||
| docker compose build frontend && docker compose up -d frontend | ||
| ``` | ||
|
|
||
| **Managed data stores** — for production, replace compose-managed services: | ||
|
|
||
| | Replace | With | Change | | ||
| |---|---|---| | ||
| | `postgres` | RDS / Aurora / Cloud SQL | Set `PG_*` vars to managed endpoint | | ||
| | `clickhouse` | ClickHouse Cloud | Set `CH_HOST`, `CH_PORT`, etc. | | ||
| | `redis` | ElastiCache / Upstash | Set `REDIS_URL` | | ||
| | `minio` | AWS S3 | Set `S3_ENDPOINT_URL=https://s3.amazonaws.com` + AWS creds | | ||
|
|
||
| <Note> | ||
| `code-executor` requires `privileged: true`. Run on EC2 / GCE instances — not Fargate or Cloud Run. | ||
| </Note> | ||
|
|
||
| **Secrets manager** — use AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager instead of a plain `.env` file. | ||
|
|
||
| --- | ||
|
|
||
| ## Backups | ||
|
|
||
| ### PostgreSQL | ||
|
|
||
| ```bash | ||
| # Backup | ||
| docker compose exec postgres \ | ||
| pg_dump -U futureagi -d futureagi --format=custom \ | ||
| > backup-$(date +%F).dump | ||
|
|
||
| # Restore | ||
| docker compose exec -T postgres \ | ||
| pg_restore -U futureagi -d futureagi --clean --if-exists \ | ||
| < backup-2026-04-22.dump | ||
| ``` | ||
|
|
||
| Volumes: `future-agi_postgres-data` · `future-agi_clickhouse-data` · `future-agi_redis-data` · `future-agi_minio-data` · `future-agi_peerdb-catalog-data` · `future-agi_peerdb-minio-data` | ||
|
|
||
| ### ClickHouse | ||
|
|
||
| ```sql | ||
| BACKUP TABLE default.traces TO S3('s3://your-bucket/ch-backup/', 'KEY', 'SECRET'); | ||
| ``` | ||
|
|
||
| ClickHouse data can also be rebuilt from scratch by re-running PeerDB init since it replicates from Postgres. | ||
|
|
||
| ### MinIO | ||
|
|
||
| ```bash | ||
| mc alias set local http://localhost:9005 futureagi <MINIO_ROOT_PASSWORD> | ||
| mc alias set s3 https://s3.amazonaws.com <AWS_KEY> <AWS_SECRET> | ||
| mc mirror local/ s3/your-bucket/ | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Monitoring | ||
|
|
||
| Backend exposes Prometheus metrics at `http://localhost:8000/metrics`. Add a scraper: | ||
|
|
||
| ```yaml | ||
| # prometheus.yml | ||
| scrape_configs: | ||
| - job_name: futureagi | ||
| static_configs: | ||
| - targets: ['localhost:8000'] | ||
| metrics_path: /metrics | ||
| ``` | ||
|
|
||
| Key signals: backend error rate, Temporal workflow success/failure, Postgres WAL lag (PeerDB health), ClickHouse query latency, PeerDB mirror status at [localhost:3001](http://localhost:3001). | ||
|
|
||
| --- | ||
|
|
||
| ## Upgrades | ||
|
|
||
| ```bash | ||
| git pull | ||
| docker compose build | ||
| docker compose up -d | ||
| ``` | ||
|
|
||
| Migrations run automatically. If a migration fails: `docker compose exec backend python manage.py migrate` | ||
|
|
||
| If release notes mention PeerDB changes: `docker compose run --rm peerdb-init bash /setup.sh` | ||
|
|
||
| **Rollback:** | ||
|
|
||
| ```bash | ||
| git log --oneline -5 | ||
| git checkout <previous-hash> | ||
| docker compose build && docker compose up -d | ||
| ``` | ||
| <CardGroup cols={2}> | ||
| <Card title="Checklist" icon="list-check" href="/docs/self-hosting/production/checklist"> | ||
| The go-live pass: secrets, prod runtime flags, and managed data stores | ||
| </Card> | ||
| <Card title="Security & TLS" icon="shield" href="/docs/self-hosting/production/security-tls"> | ||
| Terminate TLS in front of the stack and lock down secrets | ||
| </Card> | ||
| </CardGroup> | ||
|
|
||
| ## Next Steps | ||
| **Operating it** | ||
|
|
||
| <CardGroup cols={2}> | ||
| <Card title="Troubleshooting" icon="wrench" href="/docs/self-hosting/troubleshooting"> | ||
| Symptoms, causes, and fixes for common errors. | ||
| <CardGroup cols={3}> | ||
| <Card title="Backups & restore" icon="database" href="/docs/self-hosting/production/backups-restore"> | ||
| Back up and restore Postgres, ClickHouse, and MinIO | ||
| </Card> | ||
| <Card title="Monitoring" icon="gauge" href="/docs/self-hosting/production/monitoring"> | ||
| Watch the health signals the stack actually exposes | ||
| </Card> | ||
| <Card title="System Configuration" icon="sliders" href="/docs/self-hosting/configuration"> | ||
| Tune the LLM gateway, PeerDB mirrors, and Temporal workers. | ||
| <Card title="Upgrades & rollback" icon="arrows-rotate" href="/docs/self-hosting/production/upgrades-rollback"> | ||
| Pull a release, run migrations, and roll back safely | ||
| </Card> | ||
| </CardGroup> | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| --- | ||
| title: "Backups & restore" | ||
| description: "Back up and restore the data stores behind a self-hosted instance" | ||
| --- | ||
|
|
||
| A self-hosted instance keeps state in a few stores: Postgres for application data, ClickHouse for the observability records (spans and traces), and MinIO for object storage. Redis is a cache (sessions, locks, rate limits, pub/sub), so it rebuilds on its own and doesn't need a backup. RabbitMQ holds the task queue: losing it drops in-flight background jobs, so drain it before planned downtime rather than backing it up. | ||
|
|
||
| ## Postgres | ||
|
|
||
| Postgres holds the application data, so back it up on a schedule. Use the custom format, and pass `-T` so `docker compose exec` doesn't allocate a TTY and mangle the binary dump: | ||
|
|
||
| ```bash | ||
| # Backup | ||
| docker compose exec -T postgres \ | ||
| pg_dump -U futureagi -d futureagi --format=custom \ | ||
| > backup-$(date +%F).dump | ||
|
|
||
| # Restore | ||
| docker compose exec -T postgres \ | ||
| pg_restore -U futureagi -d futureagi --clean --if-exists \ | ||
| < backup-2026-04-22.dump | ||
| ``` | ||
|
|
||
| The named volumes that hold state. The compose project name is `futureagi`, so every volume is prefixed `futureagi_`: | ||
|
|
||
| | Volume | Holds | | ||
| |---|---| | ||
| | `futureagi_postgres-data` | Postgres application data | | ||
| | `futureagi_clickhouse-data` | ClickHouse spans and traces | | ||
| | `futureagi_minio-data` | MinIO objects | | ||
| | `futureagi_rabbitmq-data` | RabbitMQ task queue | | ||
| | `futureagi_redis-data` | Redis cache (rebuildable) | | ||
| | `futureagi_peerdb-catalog-data` | PeerDB replication catalog | | ||
| | `futureagi_peerdb-minio-data` | PeerDB staging objects | | ||
| | `futureagi_fi-collector-data` | fi-collector buffer | | ||
|
|
||
| ## ClickHouse | ||
|
|
||
| ClickHouse is not just a replica anymore. Since the CH25 cutover (`CH25_DROP_LEGACY_CDC_CHAIN` defaults to `true`), the fi-collector writes `spans` straight to ClickHouse and Django dual-writes `traces`. PeerDB only rebuilds the tables it mirrors from Postgres, so if you lose ClickHouse the observability data does **not** come back from a PeerDB re-init. Back it up on its own: | ||
|
|
||
| ```sql | ||
| BACKUP DATABASE default TO S3('s3://your-bucket/ch-backup/', 'KEY', 'SECRET'); | ||
| ``` | ||
|
|
||
| <Warning> | ||
| Don't rely on PeerDB init to rebuild ClickHouse. It restores the mirrored Postgres tables, not the `spans` the collector writes directly. ClickHouse needs a real backup on its own schedule. | ||
| </Warning> | ||
|
|
||
| ## MinIO | ||
|
|
||
| Mirror the MinIO bucket to S3 with the MinIO client: | ||
|
|
||
| ```bash | ||
| mc alias set local http://localhost:9005 futureagi <MINIO_ROOT_PASSWORD> | ||
| mc alias set s3 https://s3.amazonaws.com <AWS_KEY> <AWS_SECRET> | ||
| mc mirror local/ s3/your-bucket/ | ||
| ``` | ||
|
|
||
| If you've already moved to [managed data stores](/docs/self-hosting/production/checklist), your provider's own backup tooling replaces these commands. | ||
|
|
||
| ## Dive deeper | ||
|
|
||
| <CardGroup cols={2}> | ||
| <Card title="Monitoring" icon="gauge" href="/docs/self-hosting/production/monitoring"> | ||
| Watch store health and replication lag | ||
| </Card> | ||
| <Card title="Upgrades & rollback" icon="arrows-rotate" href="/docs/self-hosting/production/upgrades-rollback"> | ||
| Roll back releases without losing data | ||
| </Card> | ||
| </CardGroup> |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| --- | ||
| title: "Checklist" | ||
| description: "The go-live pass before a self-hosted instance takes real traffic" | ||
| --- | ||
|
|
||
| Run through this once before the stack is reachable by anyone else. Three things separate a laptop trial from a real deployment: | ||
|
|
||
| - Replace the dev-only secret defaults | ||
| - Bring the stack up with the production overlay, so it refuses to boot until those secrets are set | ||
| - Move the compose-managed data stores to managed services | ||
|
|
||
| ## Replace the dev-only secrets | ||
|
|
||
| The stack boots with dev-only placeholders baked into `docker-compose.yml`, values like `local-dev-only-not-for-production-replace-me`, and `futureagi` for the database passwords. It runs fine with them, which is the point on a laptop and the danger in production. | ||
|
|
||
| What forces real secrets is the production overlay, `deploy/docker-compose.production.yml`. It re-binds each one with `${VAR:?}`, so the stack won't start until you've set them. Bring the stack up with that overlay and set, at minimum: | ||
|
|
||
| - `SECRET_KEY` | ||
| - `AGENTCC_INTERNAL_API_KEY` | ||
| - `AGENTCC_ADMIN_TOKEN` | ||
| - `PG_PASSWORD` | ||
| - `MINIO_ROOT_PASSWORD` | ||
| - `RABBITMQ_USER` and `RABBITMQ_PASSWORD` | ||
| - `FRONTEND_URL` and `VITE_HOST_API` | ||
|
|
||
| ```bash | ||
| openssl rand -hex 32 # SECRET_KEY, AGENTCC_INTERNAL_API_KEY, AGENTCC_ADMIN_TOKEN | ||
| openssl rand -base64 24 # PG_PASSWORD, MINIO_ROOT_PASSWORD, RABBITMQ_PASSWORD | ||
| ``` | ||
|
|
||
| <Warning> | ||
| `PG_PASSWORD` is baked into the Postgres volume on the **first** boot, so set it before your first `docker compose up`. `MINIO_ROOT_PASSWORD` is read from the environment on every boot, so that one you can change and restart. The full field list is in [Environment variables](/docs/self-hosting/environment). | ||
| </Warning> | ||
|
|
||
| ## Switch the backend to production mode | ||
|
|
||
| Set these runtime flags before going live: | ||
|
|
||
| | Variable | Go-live value | Why | | ||
| |---|---|---| | ||
| | `ENV_TYPE` | `prod` | Disables debug output and runs Django `check --deploy` | | ||
| | `FAST_STARTUP` | `false` | Always applies migrations on restart | | ||
| | `GRANIAN_WORKERS` | your CPU count | One worker per core, up from the default `1` | | ||
|
|
||
| ## Move to managed data stores | ||
|
|
||
| Compose-managed Postgres, ClickHouse, Redis, and MinIO are fine for a trial. For production, point the stack at managed services. The catch: the backend reads these hosts from **hardcoded values in the `backend` env block of `docker-compose.yml`** (`PG_HOST: postgres`, `CH_HOST: clickhouse`, `REDIS_URL: redis://redis:6379/0`, `S3_ENDPOINT_URL: http://minio:9000`), not from `.env`. Setting them in `.env` does nothing. You edit the compose file. | ||
|
|
||
| | Replace | With | Edit in `docker-compose.yml` | | ||
| |---|---|---| | ||
| | `postgres` | RDS, Aurora, or Cloud SQL | `PG_HOST` / `PG_PORT` to the managed endpoint | | ||
| | `clickhouse` | ClickHouse Cloud | `CH_HOST` / `CH_PORT` and the credentials | | ||
| | `redis` | ElastiCache or Upstash | `REDIS_URL` | | ||
| | `minio` | AWS S3 | `STORAGE_BACKEND: s3` and the S3 credentials | | ||
|
|
||
| <Note> | ||
| `code-executor` runs with `privileged: true`, so it can't run on ECS Fargate or Cloud Run. Put it on an EC2 or GCE instance. The platform matrix is in [Requirements](/docs/self-hosting/requirements). | ||
| </Note> | ||
|
|
||
| ## Dive deeper | ||
|
|
||
| <CardGroup cols={2}> | ||
| <Card title="Security & TLS" icon="shield" href="/docs/self-hosting/production/security-tls"> | ||
| Put TLS in front of the stack and move secrets into a manager | ||
| </Card> | ||
| <Card title="Backups & restore" icon="database" href="/docs/self-hosting/production/backups-restore"> | ||
| Set up backups before the instance holds real data | ||
| </Card> | ||
| </CardGroup> |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| --- | ||
| title: "Monitoring" | ||
| description: "Watch a self-hosted instance with the health signals it actually exposes" | ||
| --- | ||
|
|
||
| The stack has no Prometheus `/metrics` endpoint yet (the fi-collector lists a metrics exporter as a TODO), so monitoring today is built from what the containers already expose: their Docker health checks, the fi-collector's admin health endpoint, and the PeerDB console. This page covers those and the signals worth watching. | ||
|
|
||
| ## Container health | ||
|
|
||
| Every service ships a Docker health check, so `docker compose ps` is the fastest read on what's up: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not every service. Only the data stores (postgres, clickhouse, redis, rabbitmq, minio, temporal) define healthchecks in compose. frontend, backend, worker, code-executor and fi-collector don't, so |
||
|
|
||
| ```bash | ||
| docker compose ps # STATUS shows healthy / unhealthy per service | ||
| docker stats # live CPU and memory per container | ||
| ``` | ||
|
|
||
| Watch memory on `clickhouse` and the Temporal `worker` first. They're the resource drivers, and an OOM there is the most common cause of a stall. | ||
|
|
||
| ## fi-collector health | ||
|
|
||
| The fi-collector exposes an admin endpoint on `127.0.0.1:9464` (`FI_COLLECTOR_ADMIN_PORT`), which serves a health check. Hit it to confirm the collector is up: | ||
|
|
||
| ```bash | ||
| curl -s http://localhost:9464/healthz | ||
| ``` | ||
|
|
||
| ## PeerDB replication | ||
|
|
||
| The Postgres-to-ClickHouse pipeline has its own console at [localhost:3001](http://localhost:3001). Mirror status there tells you whether trace analytics are keeping up with Postgres. A mirror in anything other than `running` means the dashboard is reading stale. | ||
|
|
||
| <Note> | ||
| A Prometheus metrics exporter is on the fi-collector roadmap. When it lands, scrape it here. Until then, the checks above are what the stack actually exposes. | ||
| </Note> | ||
|
|
||
| ## Dive deeper | ||
|
|
||
| <CardGroup cols={2}> | ||
| <Card title="Upgrades & rollback" icon="arrows-rotate" href="/docs/self-hosting/production/upgrades-rollback"> | ||
| Keep the stack current without downtime | ||
| </Card> | ||
| <Card title="Troubleshooting & FAQs" icon="wrench" href="/docs/self-hosting/troubleshooting"> | ||
| Symptoms, causes, and fixes for common errors | ||
| </Card> | ||
| </CardGroup> | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Production folder needs an Overview child pointing at
/docs/self-hosting/production, same as the Explore dashboard folder in Observe. Right now the overview page isn't reachable from the sidebar