diff --git a/base.yml b/base.yml index c88ed002..fb70f185 100644 --- a/base.yml +++ b/base.yml @@ -239,6 +239,7 @@ nav: - docs/operation/opa-integration.md - docs/operation/tuning.md - docs/operation/global-resources.md + - docs/operation/health.md - docs/operation/administration.md - docs/observation.md - Internals: diff --git a/docs/docs/operation/health.md b/docs/docs/operation/health.md new file mode 100644 index 00000000..925c8777 --- /dev/null +++ b/docs/docs/operation/health.md @@ -0,0 +1,300 @@ +--- +title: Health Endpoint +--- + +## `GET /api/v1/health` + +The health endpoint is an HTTP endpoint exposed by the [`web` node](../install/running-web.md) that reports whether +the Concourse deployment is in a healthy state. Operators and monitoring tools can poll it to get a fast, structured +answer about the state of the system — without needing to log in or use `fly`. + +The endpoint requires no authentication and returns a JSON body along with an HTTP status code that reflects the +current health state. + +## Health states + +The endpoint reports one of three states: + +**Ok** — everything is working as expected. The number of connected workers meets the configured minimum, and +all internal components are keeping up with their scheduled work. + +**Degraded** — the system is running but something needs attention. Either the number of connected workers has +dropped below [`--health-min-worker-count`](#--health-min-worker-count), or one or more internal components (such as +the build scheduler, resource checker, or build tracker) have fallen behind and are considered stale. Builds can +still run. This is a warning state — the system is operational, but the operator should investigate. + +**Failing** — the system is in a critical state. Either the database is unreachable or read-only, or there are no +connected workers. Builds cannot run. + +## Response format + +The response body is a JSON object with the following top-level fields: + +| Field | Description | +|---|---| +| `status` | One of `"ok"`, `"degraded"`, or `"failing"`. | +| `timestamp` | RFC 3339 timestamp of when the response was generated. | +| `database` | Database connectivity status. See below for details. | +| `workers` | Connected worker counts. See below for details. | +| `components` | Array of internal background components and their health. See below for details. | + +**`database` fields:** + +| Field | Description | +|---|---| +| `status` | `"healthy"` or `"unhealthy"`. | +| `error` | Human-readable error message. Only present when `status` is `"unhealthy"`. | + +**`workers` fields:** + +| Field | Description | +|---|---| +| `status` | One of `"healthy"`, `"degraded"`, or `"unhealthy"`. | +| `total` | Total number of registered workers. | +| `running` | Number of workers currently in the `running` state. | +| `unhealthy_workers` | Array of the names of workers not in the `running` state. Not present if all workers are healthy. | + +**`components` fields (one object per component):** + +| Field | Description | +|---|---| +| `name` | Component identifier (e.g. `"scheduler"`, `"tracker"`, `"collector_volumes"`) | +| `status` | `"healthy"` if the component is keeping up with its scheduled work, `"unhealthy"` if stale. | +| `paused` | `true` if the component has been administratively paused. | +| `last_ran` | RFC 3339 timestamp of the last completed run. | + +### Ok + +``` +HTTP 200 OK +``` + +```json +{ + "status": "ok", + "timestamp": "2026-06-08T10:05:00Z", + "database": { "status": "healthy" }, + "workers": { "status": "healthy", "total": 3, "running": 3 }, + "components": [ + { "name": "scheduler", "status": "healthy", "paused": false, "last_ran": "2026-06-08T10:05:00Z" }, + { "name": "tracker", "status": "healthy", "paused": false, "last_ran": "2026-06-08T10:05:00Z" }, + { "name": "collector_volumes", "status": "healthy", "paused": false, "last_ran": "2026-06-08T10:05:00Z" } + ] +} +``` + +### Degraded + +``` +HTTP 200 OK +``` + +```json +{ + "status": "degraded", + "timestamp": "2026-06-08T10:05:00Z", + "database": { "status": "healthy" }, + "workers": { "status": "healthy", "total": 3, "running": 3 }, + "components": [ + { "name": "scheduler", "status": "unhealthy", "paused": false, "last_ran": "2026-06-08T10:00:00Z" }, + { "name": "tracker", "status": "healthy", "paused": false, "last_ran": "2026-06-08T10:05:00Z" }, + { "name": "collector_volumes", "status": "unhealthy", "paused": false, "last_ran": "2026-06-07T08:00:00Z" } + ] +} +``` + +In this example the `scheduler` component is stale, which drives the top-level +`degraded` status. `collector_volumes` is also stale, but because it is not a +runtime-critical component, it does not contribute to the top-level status on +its own. + +### Failing + +``` +HTTP 503 Service Unavailable +``` + +```json +{ + "status": "failing", + "timestamp": "2026-06-08T10:05:00Z", + "database": { "status": "unhealthy", "error": "database unreachable" }, + "workers": { "status": "unhealthy", "total": 0, "running": 0 }, + "components": [ + { "name": "scheduler", "status": "unhealthy", "paused": false, "last_ran": "2026-06-08T10:00:00Z" }, + { "name": "tracker", "status": "unhealthy", "paused": false, "last_ran": "2026-06-08T10:00:00Z" }, + { "name": "collector_volumes", "status": "unhealthy", "paused": false, "last_ran": "2026-06-07T08:00:00Z" } + ] +} +``` + +## What makes a component stale? + +Every component runs on a fixed interval. If a component hasn't completed a run for longer than its interval +multiplied by `--health-component-stale-multiplier`, it is marked stale (`"status": "unhealthy"`) in the +`components` array. + +For example: if the build scheduler runs every 10 seconds and the stale multiplier is `2.0`, the scheduler is +considered stale after 20 seconds without a completed run. + +A stale component usually means the `web` node is under heavy load or is stuck. Check the `web` node logs for errors +if a component stays stale. + +### Which components affect the top-level status? + +Not all stale components drive the top-level `status` field. Only three are +considered critical for scheduling and running builds: + +| Component | Description | +|---|---| +| `scheduler` | Determines when new job builds should run. See [Build Scheduler](../internals/scheduler.md). | +| `tracker` | Picks up and executes started builds. See [Build Tracker](../internals/build-tracker.md). | +| `scanner` | Schedules resource checks. See [Resource Checker](../internals/checker.md). | + +If any of these three fall behind, the top-level `status` becomes `degraded` (or `failing` if workers are also gone). + +All other components — the garbage collection collectors, `pipeline_pauser`, `being_watched_build_marker`, +`signing_key_lifecycler`, and others — appear in the `components` array with their own `status` field, +but a stale value for any of them does not change the top-level `status`. They are worth monitoring, but a stale GC +collector does not by itself indicate that builds are at risk of not running. + +## Web Configuration flags + +The following flags can be set on the `concourse web` command to change the behaviour of the health endpoint. + +### `--health-min-worker-count` + +Controls how many connected workers are required for the endpoint to report `ok`. When the number of connected +workers in the `running` state drops below this value, the endpoint reports `degraded`. + +Setting this to `0` disables the worker count check entirely — the endpoint will never report `degraded` due to +worker count, and will only transition to `failing` when there are no running workers at all. + +| | | +|---|---| +| **Flag** | `--health-min-worker-count` | +| **Env var** | `CONCOURSE_HEALTH_MIN_WORKER_COUNT` | +| **Type** | integer | +| **Default** | `1` | + +```properties +CONCOURSE_HEALTH_MIN_WORKER_COUNT=2 +``` + +### `--health-component-stale-multiplier` + +Controls how tolerant the endpoint is of slow or delayed background components. A component is considered stale when +it hasn't run for longer than its interval multiplied by this value. + +Increasing the multiplier gives components more time before they are marked stale, which reduces false positives in +environments where the `web` node is occasionally slow. Decreasing it makes the endpoint more sensitive — useful if +you want earlier warnings when a component falls behind. + +| | | +|---|---| +| **Flag** | `--health-component-stale-multiplier` | +| **Env var** | `CONCOURSE_HEALTH_COMPONENT_STALE_MULTIPLIER` | +| **Type** | float | +| **Default** | `2.0` | + +```properties +CONCOURSE_HEALTH_COMPONENT_STALE_MULTIPLIER=3.0 +``` + +## Related `fly` commands + +### `fly workers` + +To see the number of connected workers and their current state, run: + +```shell +fly -t example workers +``` + +This can help explain why the health endpoint is reporting `degraded` or `failing`. A worker in the `stalled` or +`landing` state does not count toward the `workers.running` total. Only workers in the `running` state are counted. + +``` +name platform tags team state version +worker-1 linux none none running 2.3 +worker-2 linux none none stalled 2.3 +``` + +In this example, `workers.running` would be `1`. If `--health-min-worker-count` is `2`, the health endpoint would +report `degraded`. + +See [Administration](administration.md#fly-workers) for more on managing workers. + +### `fly health` + +To check the health of a Concourse deployment, run: + +```shell +fly -t example health +``` + +This prints a summary table with one row per subsystem and component: + +``` +subsystem status detail +overall ok 2026-06-08T10:05:00Z +database healthy +workers healthy 3/3 running +scheduler healthy last ran: 2026-06-08T10:04:50Z +tracker healthy last ran: 2026-06-08T10:04:55Z +scanner healthy last ran: 2026-06-08T10:04:45Z +collector_volumes unhealthy last ran: 2026-06-07T08:00:00Z +pipeline_pauser healthy paused +``` + +The `overall` row shows the top-level status and the timestamp from the response. The `workers` row shows how many +workers are running out of how many are registered; if any are not running, their names appear in the detail column. +Each component row shows when it last ran, or `paused` if the component has been administratively paused. + +Status values are color-coded: green for `ok` / `healthy`, yellow for `degraded`, red for `failing` / `unhealthy`. + +!!! note + + The exit code is always `0`, even when the cluster is failing. `fly health` is informational, consistent with + `fly status`. + +To get the raw JSON response instead — useful for scripting or piping to `jq`: + +```shell +fly -t example health --json +``` + +`fly h` is a supported alias of `fly health`. + +## Usage as a monitoring probe + +The health endpoint is well suited for use as an external monitoring probe or a Kubernetes health check. + +**Kubernetes liveness and readiness probes:** + +```yaml +livenessProbe: + httpGet: + path: /api/v1/health + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 15 + +readinessProbe: + httpGet: + path: /api/v1/health + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 15 +``` + +Note that both `ok` and `degraded` return HTTP `200`, so both probes will pass in either state. Only `failing` +returns HTTP `503`, which will cause the probe to fail. If you want readiness to fail while the system is degraded, +you will need a wrapper script or a custom probe that inspects the `status` field in the response body. + +**Alerting pipelines:** + +If you have an external monitor (such as Prometheus Blackbox Exporter, Nagios, or a Concourse pipeline itself), +pointing it at `/api/v1/health` and alerting on anything other than `HTTP 200` gives you a fast signal for failing +states. The `components` field in the response body provides the detail needed to start investigating alerts easily. +For example, if the `scheduler` is stale, you can check the `web` node logs for errors. \ No newline at end of file