Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions .github/workflows/managed-n8n.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@ on:
pull_request:
paths:
- "services/workflow-automation/n8n/**"
- "services/workflow-automation/n8n-prod/**"
- "lib/managed_updates/**"
- "tests/test-shell-resolvable.sh"
- "scripts/validate-managed-updates.sh"
- "scripts/install-managed-update-systemd.sh"
- "systemd/**"
Expand All @@ -21,7 +23,9 @@ on:
- "renovate/**"
paths:
- "services/workflow-automation/n8n/**"
- "services/workflow-automation/n8n-prod/**"
- "lib/managed_updates/**"
- "tests/test-shell-resolvable.sh"
- "scripts/validate-managed-updates.sh"
- "scripts/install-managed-update-systemd.sh"
- "systemd/**"
Expand Down Expand Up @@ -51,12 +55,19 @@ jobs:
run: >-
shellcheck --severity=error corekit.sh
lib/managed_updates/managed-update.sh
services/workflow-automation/n8n/managed/backup.sh
services/workflow-automation/n8n/managed/strict-healthcheck.sh
services/workflow-automation/n8n/managed/*.sh
services/workflow-automation/n8n-prod/managed/*.sh
services/workflow-automation/n8n-prod/cli.sh
tests/test-shell-resolvable.sh
scripts/validate-managed-updates.sh
scripts/install-managed-update-systemd.sh
tests/test-managed-update.sh
tests/test-n8n-candidate.sh
- name: Every called command resolves
# Catches a call left behind when a function is deleted. shellcheck does
# not report undefined functions and `bash -n` cannot see them, which is
# how one reached this branch and broke `managed-update apply`.
run: tests/test-shell-resolvable.sh
- name: Validate policy, stable channel, Compose, and Git diff
run: ./scripts/validate-managed-updates.sh --online
- uses: actions/setup-node@v4
Expand Down
1 change: 0 additions & 1 deletion config/stacks/core.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -77,5 +77,4 @@ services:
- mysql
- postgres
- redis
- caddy
- watchtower
108 changes: 108 additions & 0 deletions docs/MANAGED_UPDATES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# Managed updates

`corekit managed-update` deploys a committed, tested version of a service. It is
service-agnostic: everything it knows about a particular service comes from that
service's own directory.

## Registering a service

A service is registered by a `managed_update` block in its `service.json`:

```json
"managed_update": {
"enabled": true,
"deployment_branch": "main",
"channel": "stable",
"minimum_release_age_days": 7,
"automatic_update_types": ["patch", "minor"],
"major_updates_require_manual_approval": true,
"backup_required": true,
"minimum_free_space_gb": 15,
"components": ["service", "service-runner"]
}
```

`enabled: false`, or no block at all, means the service is never touched. List
what is registered with:

```bash
corekit managed-update list
```

## Hooks

Anything service-specific lives in `<service>/managed/`. The driver runs these
and treats a non-zero exit as a failed deployment. Three are optional and are
skipped when absent: `validate.sh`, `counts.sh` and `stop.sh`. The rest are
called unconditionally, and a missing one fails the run.

| Hook | Required | Purpose | Contract |
|---|---|---|---|
| `version.sh` | yes | What version is committed, and what is live | Prints one JSON object: `{"target_version","live_version","identity"}`. `identity` is any string that changes when the committed build would change |
| `validate.sh` | no | Refuse an incoherent committed bundle | Exit non-zero to abort. Message on stderr |
| `build.sh` | yes | Build or pull the candidate and verify it is what was asked for | Receives the target version as `$1`. Prints JSON `{"images":{...}}` on the last line. A service that builds nothing still needs one, to verify the image it adopts |
| `backup.sh` | yes | Create and verify a recovery set | Prints the manifest path on the last line. The manifest must report `verified: true`, `restore_drill.result: "passed"` and `candidate_migration_drill.result: "passed"` — see below |
| `counts.sh` | in practice yes | Entity counts and in-flight executions | Prints `{"workflows","credentials","active_executions"}`. Must exit non-zero if it cannot read them: reporting zero would let a deployment proceed during a database outage. Without the hook, the post-deployment count check fails the deployment |
| `stop.sh` | no | Stop this service's containers after a failed health check | Without it the driver stops the containers named in `components` |
| `deploy.sh` | yes | Recreate the service with the candidate | Receives the target version as `$1`, and the candidate images as `COREKIT_CANDIDATE_IMAGES` |
| `strict-healthcheck.sh` | yes | Readiness, counts, canary | Exit non-zero if the deployment should be treated as failed |

Hooks run with `COREKIT_PROJECT_ROOT`, `COREKIT_SERVICE_DIR` and
`COREKIT_SERVICE_NAME` in the environment. `backup.sh` also receives
`COREKIT_MANAGED_BACKUP_ROOT`, and `deploy.sh` receives
`COREKIT_CANDIDATE_IMAGES`, the JSON that `build.sh` printed.

`components` is ordered and load-bearing: the first entry is the container whose
image is recorded as the rollback target, the second is its task runner. Recovery
hands both to the operator, so listing them the other way round would tell them
to restart the wrong image.

### The backup manifest

The driver refuses to deploy unless the manifest reports all three of:

```json
{ "verified": true,
"restore_drill": { "result": "passed" },
"candidate_migration_drill":{ "result": "passed" } }
```

`verified` is an assertion the hook makes about itself, so it is worth no more
than the work behind it. A hook that writes `verified: true` without restoring
anything turns the gate into decoration. Restore the dump somewhere isolated and
compare what comes back against the live instance before claiming it.

Where a step does not apply — a service that adopts a prebuilt image has no
candidate whose migrations need rehearsing — say so explicitly in the manifest
with a `note`, rather than omitting the key. An absent key fails the gate, which
is correct: silence should not pass.

`counts.sh` matters more than it looks. A service with its own database must
report from that database: measuring one instance against another instance's
numbers would let a deployment that silently lost data pass its checks.

## Why hooks rather than configuration

Services differ in ways configuration cannot express. One builds an image from a
Dockerfile; another adopts an image another service already built. One holds its
data in a shared database; another in its own. A hook is a small script in the
service's own directory, next to the Compose file it belongs to, which is where
someone changing that service will look.

## Gates

Ordered, each stopping the run:

1. The service is registered and enabled
2. The worktree is clean and on the deployment branch
3. The committed bundle validates
4. The change is not a downgrade, and a major change has explicit approval
5. The release-age gate is met, or is explicitly overridden
6. Free space is above the floor
7. The candidate builds and verifies
8. A recovery set is created and verified
9. The deployment succeeds
10. Strict health passes

Only then is the run recorded as successful. A deployment that leaves the
service unhealthy is a failed deployment whatever the build reported.
149 changes: 149 additions & 0 deletions docs/PROMOTION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
# Promotion

Imports workflow definitions into an n8n service from a commit that is already
merged into a protected branch.

It is exposed on the service rather than as a top-level corekit verb, because it
is n8n-specific: it drives `n8n-git` and the n8n CLI inside the container, and
nothing else in the catalogue could use it as written.

corekit run <service> promote [options]

## The problem it solves

A service where definitions can be authored freely is a poor place to keep
production credentials: anything that can write a definition and run it can use
those credentials, without ever reading the secret itself. A service that holds
production credentials is therefore a poor place to author freely.

Splitting authoring from production is the usual answer, and it leaves one gap:
how a change crosses from one to the other. Get that crossing wrong and it
becomes the very thing the split was meant to prevent — a way to run an
arbitrary definition against real credentials.

## The shape

```
authoring instance --push--> branch --review, merge--> protected branch
|
corekit run <svc> promote (host)
v
production instance
```

Four properties make the crossing safe:

| Property | How |
|---|---|
| The artefact is fixed | A full 40-character commit SHA. A branch tip moves; a commit does not |
| It has been reviewed | The commit must be an ancestor of the protected branch. Whatever review that branch requires has therefore happened |
| It is recoverable | The target is exported before the import, and the run is recorded with the previous commit |
| It cannot be self-triggered | It is a host command. Nothing running inside either instance can invoke it |

That last one matters most. An approval step that the authoring instance can
trigger is not a control — it becomes a prompt that gets waved through. Keep the
decision on the host, with a human.

## Credentials are never imported

`--credentials 0` is passed unconditionally. Credentials are created once, by
hand, on the production instance. Two consequences worth stating:

- A promoted definition can only use credentials that already exist there, so
promotion cannot introduce new reach.
- Definitions exported from another instance are encrypted with *that*
instance's key and are inert anywhere else. Never export decrypted
credentials to work around this; that would defeat the entire arrangement.

## Configuration

Split by whether it is safe to publish.

**`service.json`** — policy, committed:

```json
"promotion": {
"enabled": true,
"protected_branch": "main",
"repository_path": "workflows",
"preserve_ids": true,
"backup_required": true,
"credentials": "never"
}
```

**The service `.env`** — site-specific, not tracked:

```
PROMOTION_REPOSITORY='owner/repo'
PROMOTION_TOKEN='...'
# Optional; these override the policy in service.json.
# No inline comments: the value is read literally, so a trailing comment
# would become part of the branch name or path.
PROMOTION_BRANCH='main'
PROMOTION_PATH='workflows'
```

Give the token **read access only**. The control is the branch protection on the
repository, not the secrecy of this token: a read-only token cannot promote
anything that has not been merged, so leaking it costs you nothing that matters.

This is the right way round deliberately. If the authoring instance can run
shell commands or read files — and most can — then any token it can reach is
one an injected instruction can steal. Design so that stealing it is not worth
anything.

## Use

```bash
corekit run <service> promote --plan # show what would happen
corekit run <service> promote # promote the protected branch tip
corekit run <service> promote --commit <sha> # promote one exact commit
corekit run <service> promote --dry-run # import in the tool's dry-run mode
corekit run <service> status # what is deployed, and from where
corekit run <service> rollback-plan # recovery procedure
```

A service opts in by shipping a `cli.sh` that delegates to the shared
implementation in `the service's managed/promote.sh`, and a `promotion` block in its
`service.json`.

`--plan` changes nothing and needs no lock, so it is safe to run at any time.

## A note on `counts.sh`

Promotion reads the target's workflow count through the service's `counts.sh`
hook, to tell a genuinely empty target — where the first promotion happens and
an empty recovery set is expected — from one whose count cannot be read, where a
backup matters most. Without that hook the count is unknown, so the first
promotion into an empty target is refused. That is the safe default rather than a
fault, but it is worth knowing before adding promotion to a service that has no
counts hook.

## Gates

A promotion stops at the first failure:

1. Promotion is enabled for the service in `service.json`
2. `PROMOTION_REPOSITORY` and `PROMOTION_TOKEN` are set
3. The target container exists and is running
4. The commit resolves, and is a full SHA if given explicitly
5. **The commit is an ancestor of the protected branch** — enforced by the
import tool, not by this script
6. A pre-promotion export succeeds and is non-empty
7. The import succeeds
8. The container is healthy afterwards

Only then is the run recorded. A promotion that leaves the service unhealthy is
a failed promotion, whatever the import reported.

## Recovery

`corekit run <service> rollback-plan` prints the procedure and the recorded
previous commit. Rolling forward to the previous commit is usually right and is
the same deterministic path. Restoring the pre-promotion export is for when the
definitions are not the problem.

Restoring is refused automatically, by design: if the service may have accepted
writes since the promotion, an automatic restore would destroy them. Establish
that first.
Loading
Loading