Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/docs/content/alpine/how-to/customize-runtime.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ Zerops caches custom runtime images to optimize deployment times. Learn about [c
For information about managing builds and deployments, see [managing builds and deployments](/features/pipeline#manage-builds-and-deployments).

:::warning
Shared storage mounts are not available during the runtime prepare phase.
Shared storage mounts and Local Storage volumes are not available during the runtime prepare phase.
:::

## Troubleshooting
Expand Down
2 changes: 1 addition & 1 deletion apps/docs/content/deno/how-to/customize-runtime.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ Zerops caches custom runtime images to optimize deployment times. Learn about [c
For information about managing builds and deployments, see [managing builds and deployments](/features/pipeline#manage-builds-and-deployments).

:::warning
Shared storage mounts are not available during the runtime prepare phase.
Shared storage mounts and Local Storage volumes are not available during the runtime prepare phase.
:::

## Troubleshooting
Expand Down
2 changes: 1 addition & 1 deletion apps/docs/content/features/backup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Zerops provides an automated, secure backup system for supported services. This

## Supported Services

Zerops provides automated backup functionality for the following services. For specific backup format details and restore instructions, visit each service's documentation: [MariaDB](/mariadb/how-to/backup), [PostgreSQL](/postgresql/how-to/manage#backups), [Qdrant](/qdrant/overview), [Elasticsearch](/elasticsearch/overview), [NATS](/nats/overview), [Meilisearch](/meilisearch/overview), and [Shared Storage](/shared-storage/how-to/manage#backups).
Zerops provides automated backup functionality for the following services. For specific backup format details and restore instructions, visit each service's documentation: [MariaDB](/mariadb/how-to/backup), [PostgreSQL](/postgresql/how-to/manage#backups), [Qdrant](/qdrant/overview), [Elasticsearch](/elasticsearch/overview), [NATS](/nats/overview), [Meilisearch](/meilisearch/overview), [Local Storage](/local-storage/how-to/manage#backups), and [Shared Storage](/shared-storage/how-to/manage#backups).

## Managing Backups in the UI

Expand Down
2 changes: 1 addition & 1 deletion apps/docs/content/features/coding-agents.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ ZCP supports multiple coding agents including Claude Code (Anthropic), Codex (Op

### An ordinary Zerops project underneath

The agent operates inside a normal Zerops project — same shape as production. Managed databases (PostgreSQL, MariaDB, ClickHouse), key-value stores (KeyDB, Valkey), search (Elasticsearch, Meilisearch, Typesense), vector store (Qdrant), message queues (NATS, Kafka), object and shared storage, managed Nginx — all on a private network, addressable by hostname. The same pipeline that deploys here can deploy to a separate HA production project with no `zcp` service attached. Not a sandbox the work outgrows.
The agent operates inside a normal Zerops project — same shape as production. Managed databases (PostgreSQL, MariaDB, ClickHouse), key-value stores (KeyDB, Valkey), search (Elasticsearch, Meilisearch, Typesense), vector store (Qdrant), message queues (NATS, Kafka), object, local and shared storage, managed Nginx — all on a private network, addressable by hostname. The same pipeline that deploys here can deploy to a separate HA production project with no `zcp` service attached. Not a sandbox the work outgrows.

### Human ↔ agent handover

Expand Down
2 changes: 1 addition & 1 deletion apps/docs/content/features/pipeline.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -308,7 +308,7 @@ Learn more about building custom runtime images:
:::warning
Do not include your application code in the custom runtime image, as your built application code is deployed automatically into fresh containers.

Shared storage mounts are also not available during the runtime prepare phase.
Shared storage mounts and Local Storage volumes are also not available during the runtime prepare phase.
:::

## Deploy phase
Expand Down
6 changes: 5 additions & 1 deletion apps/docs/content/features/scaling.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Unlike traditional hosting where resources must be predicted and pre-provisioned
Zerops uses two fundamentally different approaches for optimizing infrastructure:

#### **Resource Management (Vertical Scaling)**
- **Applies to:** Runtime services, databases, shared storage, and Linux containers (Alpine and Ubuntu)
- **Applies to:** Runtime services, databases, local and shared storage, and Linux containers (Alpine and Ubuntu)
- **What it does:** Adjusts CPU, RAM, and disk resources within individual containers
- **Management:** Automated by Zerops, but customizable by users

Expand Down Expand Up @@ -306,6 +306,10 @@ For databases and shared storage services, Zerops offers two deployment modes fo
Deployment mode cannot be changed after creation.
:::

:::note
[Local Storage](/local-storage/overview) is currently available in single container mode only — its volume lives on a single physical machine by design. See its [technical details](/local-storage/tech-details) for how availability and scaling work there.
:::

#### Single Container Mode

Single Container Mode provides one container with vertical scaling only. This is suitable for development environments or non-critical data storage.
Expand Down
3 changes: 2 additions & 1 deletion apps/docs/content/guides/backup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: "Zerops auto-backs up databases and storage daily (00:00-01:00 UTC)
Zerops auto-backs up databases and storage daily (00:00-01:00 UTC) with X25519 encryption; backups are retained for 7 days minimum after service/project deletion.

## Supported Services
MariaDB, PostgreSQL, Qdrant, Elasticsearch, NATS, Meilisearch, Shared Storage.
MariaDB, PostgreSQL, Qdrant, Elasticsearch, NATS, Meilisearch, Local Storage, Shared Storage.

**ClickHouse**: not on the standard auto-backup path — back it up with the native `BACKUP ALL ...` SQL command (super user), stored as `.tar.gz`.

Expand Down Expand Up @@ -50,6 +50,7 @@ End-to-end with X25519 per-project keys. Decrypted only on download.
| Meilisearch | `.dump` |
| Qdrant | `.snapshot` |
| NATS | `.tar.gz` |
| Local Storage | tar → `.tar.gz` (archive of the live volume — pause database writes or use an application-level backup for guaranteed consistency) |
| Shared Storage | tar → `.tar.gz` |

## Gotchas
Expand Down
2 changes: 1 addition & 1 deletion apps/docs/content/guides/choose-runtime-base.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ description: "**Use Alpine** as the default base for all services. Use Ubuntu on

- Size: ~100MB base
- Package manager: `sudo apt-get update && sudo apt-get install -y <pkg>` (sudo required)
- Version: 24.04 LTS (22.04 also available)
- Version: 26.04 LTS (24.04 and 22.04 also available)
- Use when: you need glibc (musl incompatibility) — CGO-linked Go, glibc-built C-extensions, or the Deno runtime (no Alpine build). Needing a package is NOT a reason — both bases install packages
- Example: Go apps with CGO, Python packages with C extensions that don't compile on musl

Expand Down
34 changes: 34 additions & 0 deletions apps/docs/content/guides/choose-storage.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
title: "Choosing Storage on Zerops"
description: "**Use Object Storage** for uploads, media, and backups; **use Local Storage** for anything that needs a real filesystem (SQLite, Prometheus, Gitea, shared certs). Shared Storage is deprecated."
---

**Use Object Storage** (S3) for uploads, media, and backups. **Use Local Storage** (`run.volume`) for anything that needs a real filesystem — SQLite and other embedded databases, single-node stateful apps, filesystem state shared between services. **Shared Storage is deprecated** — don't use it in new projects. Human-facing comparison: [Storage on Zerops](/storage/overview).

## Decision Matrix

| Need | Choice | Why |
|------|--------|-----|
| User uploads, media, backups, logs | **Object Storage** (default) | S3 API, HA, cheapest per GB, survives independent of any service |
| SQLite / Prometheus / filesystem-based databases | **Local Storage** | Single-kernel POSIX: locking and mmap are correct — the only Zerops storage where these workloads are safe |
| Single-node stateful app (Gitea, etc.) | **Local Storage** | Persistent volume survives deploys and container replacement |
| Filesystem state shared between services (certs, caches) | **Local Storage** | One volume mountable by multiple runtime services, writes visible immediately |
| Temp/scratch files | Container disk | Fine as-is; lost on deploy |
| Existing `mount:`-based setups | Shared Storage (deprecated) | Keeps working; [migrate to Local Storage](/local-storage/how-to/migrate-from-shared-storage) or Object Storage |

## Local Storage

- Service type `local-storage:single@1`; runtime services mount it via `run.volume` in zerops.yaml: `volume: {hostname: <storage>, mountPath: /path (default /mnt/{hostname}), readOnly: false}`. One volume per runtime service; multiple services can mount the same volume.
- The volume is local to one physical machine: **all containers of all connected services are co-located on the volume's machine**. Not HA today (an HA mode may be added in the future) — on hardware failure or maintenance, Zerops moves the volume to a healthy server and every connected service goes through a small outage (stop, move, start); volume recovery is not guaranteed, so backups are strongly recommended. Horizontal scaling of connected services is capped by that machine's capacity.
- The volume survives deploys: new containers get the same volume attached. Mounted in runtime containers only (not build/prepare).
- Capacity = the service's max disk in `verticalAutoscaling`. Usage is billed as the service's disk usage.
- The service's own container has the volume at `/data` with SSH/web-shell access for direct inspection. Daily `.tar.gz` backups of the live volume are on by default — they can capture database files mid-write, so for SQLite schedule a `run.crontab` job that writes a consistent copy onto the volume (`sqlite3 <db> ".backup <path-on-volume>"`, `allContainers: false`) and let the archive pick that copy up. Snapshot-based point-in-time backups are planned.
- SQLite single-writer restrictions still apply: locking is correct, but only one writer at a time — keep writes in one service/container and mount `readOnly: true` elsewhere.

## Object Storage

See the [object-storage integration guide](/guides/object-storage-integration): S3-compatible (MinIO), `objectStorageSize` quota, `AWS_USE_PATH_STYLE_ENDPOINT: true`, `${storage_*}` env var references. Not a filesystem — no locking, no partial writes, no mmap.

## Shared Storage (deprecated)

A managed SeaweedFS cluster mounted at `/mnt/<hostname>` via the import-level `mount:` field. Deprecated: it will be replaced by a standalone SeaweedFS managed service mounted with `zsc shared-storage mount` in `initCommands`. Its per-mount-only locks make databases unsafe on it — that workload belongs on Local Storage. Existing services keep working.
7 changes: 4 additions & 3 deletions apps/docs/content/guides/deployment-lifecycle.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -132,10 +132,11 @@ Zerops keeps **10 most recent versions**. Older auto-deleted. Any archived versi

When using SSHFS (`zerops_mount`) for dev workflows, deploy replaces the container. This has important consequences:

1. **After deploy, run container only has `deployFiles` content.** All other files (including zerops.yml if not in deployFiles) are gone. Use `deployFiles: [.]` for dev services to ensure zerops.yml and source files survive the deploy cycle.
1. **After deploy, run container only has `deployFiles` content.** All other files (including zerops.yml if not in deployFiles) are gone — persistent data belongs in a [storage service](/storage/overview). Use `deployFiles: [.]` for dev services to ensure zerops.yml and source files survive the deploy cycle.
2. **SSHFS mount auto-reconnects only while the service is running.** Usually no remount is needed, but if the mount goes stale after a deploy (stat/ls returns empty, writes hang), recover it explicitly with `zerops_mount action="mount"`. A stopped service has no live mount until it starts again.
3. **zerops.yml must be in deployFiles** for dev self-deploy lifecycle. Without it, subsequent deploys from the container fail because zerops.yml is missing.

**Two kinds of "mount" (disambiguation):**
**Three kinds of "mount" (disambiguation):**
- `zerops_mount` -- SSHFS tool, mounts service `/var/www` locally for development. This is a dev workflow tool.
- Shared storage mount -- platform feature, attaches a shared-storage volume at `/mnt/{hostname}` via the service-level `mount:` parameter in import.yml. There is no `mount` field in zerops.yml (a `run.mount` is silently ignored). These are completely unrelated features.
- Shared storage mount (deprecated) -- platform feature, attaches a shared-storage volume at `/mnt/{hostname}` via the service-level `mount:` parameter in import.yml. There is no `mount` field in zerops.yml (a `run.mount` is silently ignored). These are completely unrelated features.
- Local Storage volume -- platform feature, mounts a Local Storage service's persistent volume via the `run.volume` field in zerops.yml (`volume: {hostname, mountPath, readOnly}`). The volume survives deploys: new containers get the same volume attached at the configured `mountPath`.
2 changes: 1 addition & 1 deletion apps/docs/content/guides/object-storage-integration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ Each service = one bucket (auto-named, immutable). Need multiple buckets? Create

## Gotchas
1. **`forcePathStyle: true` / `AWS_USE_PATH_STYLE_ENDPOINT: true` is REQUIRED**: Zerops uses MinIO which doesn't support virtual-hosted style
2. **Container filesystem is replaced on deploy**: Files on disk survive restarts but are lost when a new container is created (deploy, scale-up). Always use Object Storage for persistent data
2. **Container filesystem is replaced on deploy**: Files on disk survive restarts but are lost when a new container is created (deploy, scale-up). Use Object Storage for persistent data, or a Local Storage volume (`run.volume`) when the workload needs a real filesystem (SQLite, Prometheus, Gitea)
3. **Region is required but ignored**: Set `us-east-1` — MinIO ignores it but SDKs require it
4. **Public URL format**: `{apiUrl}/{bucketName}/path/to/file`
5. **Independent infrastructure**: Object Storage runs on separate infra from other services — accessible from Zerops and remotely over internet
Expand Down
6 changes: 4 additions & 2 deletions apps/docs/content/guides/production-checklist.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -64,12 +64,14 @@ Remove entirely or disable `enableSubdomainAccess`. Use VPN + pgAdmin/DBeaver lo

## File Storage

**Container filesystem survives restarts but is replaced on every deploy** — files stored on disk persist through reload/restart/stop+start but are lost on deploy or container replacement (scale-up/down).
**Container filesystem survives restarts but is replaced on every deploy** — files stored on disk persist through reload/restart/stop+start but are lost on deploy or container replacement (scale-up/down). See [Storage on Zerops](/storage/overview).

| Use case | Solution |
|----------|----------|
| User uploads | Object Storage (S3) |
| Media files | Object Storage (S3) |
| SQLite / single-node stateful apps (Prometheus, Gitea) | Local Storage (persistent volume, survives deploys) |
| Shared filesystem state between services (certs, caches) | Local Storage |
| Temp files | Container disk (OK) |
| Build artifacts | Deploy via zerops.yaml |

Expand Down Expand Up @@ -147,7 +149,7 @@ Production services should pair a `deploy.readinessCheck` (gates traffic during

## Gotchas
1. **HA is immutable**: Must delete and recreate service to switch modes
2. **Container filesystem survives restarts but is replaced on every deploy**: use external storage for persistent data
2. **Container filesystem survives restarts but is replaced on every deploy**: use [external storage](/storage/overview) for persistent data
3. **File sessions break with scaling**: Multiple containers don't share filesystem
4. **Mailpit is not production SMTP**: Only for dev — no delivery guarantees
5. **Debug mode leaks secrets**: Disable APP_DEBUG in production
Expand Down
8 changes: 5 additions & 3 deletions apps/docs/content/guides/shared-storage-integration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@ title: Shared Storage Integration on Zerops
description: "Guide: Shared Storage Integration on Zerops"
---

Shared storage is a managed SeaweedFS volume mounted as a POSIX filesystem at `/mnt/<storageHostname>` into one or more runtime services — for files that must be shared *between containers/services* (shared config, plugin directories, a common working set). It is mounted via the import.yaml `mount:` field; there is **no `zerops.yaml` mount**. For high-write workloads or user uploads, prefer Object Storage (S3) instead — shared storage is POSIX/NFS-style and not built for high-throughput churn.
**DEPRECATED**: Shared Storage is deprecated and will be replaced by a standalone SeaweedFS managed service mounted with `zsc shared-storage mount` in `initCommands`. For new projects, prefer **[Local Storage](/local-storage/overview)** (`run.volume` in zerops.yaml) for a persistent shared filesystem, or **Object Storage** for large or highly available data. Existing Shared Storage services keep working; move them with the [migration guide](/local-storage/how-to/migrate-from-shared-storage).

Shared storage is a managed SeaweedFS volume mounted as a POSIX filesystem at `/mnt/<storageHostname>` into one or more runtime services — for files that must be shared *between containers/services* (shared config, plugin directories, a common working set). It is mounted via the import.yaml `mount:` field; there is **no `zerops.yaml` mount** (do not confuse it with `run.volume`, which mounts a Local Storage volume). For high-write workloads or user uploads, prefer Object Storage (S3) instead — shared storage is POSIX/NFS-style and not built for high-throughput churn.

## Mounting — import.yaml `mount:` is the only config-file mechanism

Expand Down Expand Up @@ -43,6 +45,6 @@ This registers the connection, but the FUSE mount materializes **only on the nex
- **POSIX**: standard filesystem ops (with minor permission-setting limits). Filesystem operations are logged to runtime logs tagged `zerops-mount-<serviceHostname>`. `df` can report misleading numbers — use the Zerops GUI for accurate storage metrics.
- **No env vars**: shared storage exposes no connection variables — it's a filesystem, not a networked service.

## Shared storage vs object storage
## Shared storage vs local storage vs object storage

Use **shared storage** when you need a POSIX filesystem shared across services (shared config, plugin/extension directories, a common scratch area). Use **object storage** (S3/MinIO) for user uploads, media, and any high-throughput or write-heavy file operations — and for anything that must survive independent of any single service. Don't reach for shared storage as a generic uploads bucket.
Use **local storage** (`run.volume`) when you need a persistent POSIX filesystem — it is a real single-node local filesystem with correct locking and mmap semantics, so it also covers SQLite and other filesystem-based databases that are unsafe on shared storage. Use **shared storage** (deprecated) only for existing setups that need the filesystem replicated across nodes. Use **object storage** (S3/MinIO) for user uploads, media, and any high-throughput or write-heavy file operations — and for anything that must survive independent of any single service. Don't reach for shared storage as a generic uploads bucket.
6 changes: 4 additions & 2 deletions apps/docs/content/homepage.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,10 @@ export const databases = [
]

export const storages = [
{ name: "Storage overview", link: "/storage/overview", icon: <Icons.archive-box/> },
{ name: "Object storage", link: "/object-storage/overview", icon: <Icons.cube-solid/> },
{ name: "Shared storage", link: "/shared-storage/overview", icon: <Icons.server/> },
{ name: "Local storage", link: "/local-storage/overview", icon: <Icons.circle-stack/> },
{ name: "Shared storage", link: "/shared-storage/overview", icon: <Icons.servers-connected/> },
]

<div className="!max-w-xxl !mx-none">
Expand Down Expand Up @@ -90,7 +92,7 @@ items={databases} />

### Storages

Fully managed S3 compatible storage running on a separate infrastructure and persistent disk that can be mounted to multiple services.
Fully managed S3 compatible storage running on a separate infrastructure and persistent disk volumes that can be mounted to multiple services. Runtime containers are replaced on every deploy, so persistent data belongs in one of these — see [Storage on Zerops](/storage/overview).

<TechCard
items={storages} />
Expand Down
Loading
Loading