Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 83 additions & 12 deletions .agents/skills/layer-nerdctl/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
---
name: layer-nerdctl
description: 'nerdctl (containerd CLI) + the rootless containerd/buildkit/CNI stack
— native Arch/Alpine packages, the pinned sha256 nerdctl-full tarball on Fedora/Debian/Ubuntu,
nerdctl.toml + the charly CNI conflist + the nested-pod posture. Use when working
with the nerdctl engine, the layer-nerdctl candy, or engine: nerdctl deploys.'
— native Arch/Alpine packages, the pinned FATALLY sha256-verified nerdctl-full tarball
on Fedora/Debian/Ubuntu, nerdctl.toml with namespace=default, the charly CNI conflist,
the written rootless containerd/buildkit user units, and the CONTAINERD.md §2/§4
checklist (uidmap helpers, subuid/subgid, delegation, /dev/fuse, user D-Bus,
slirp4netns). Use when working with the nerdctl engine, the layer-nerdctl candy, or
engine: nerdctl deploys.'
---

# layer-nerdctl
Expand All @@ -14,18 +17,72 @@ engine.

## What it installs

- **Native packages** on Arch/CachyOS/Omarchy and Alpine: `nerdctl`,
- **Native packages** on Arch/CachyOS/Omarchy and Alpine: `nerdctl`, `containerd`,
`cni-plugins`, `rootlesskit`, `slirp4netns`, `buildkit`, `fuse-overlayfs`,
`crun`, `iptables-nft` (arch) / `iptables` (alpine).
`crun`, `iptables-nft` (arch) / `iptables` (alpine), plus `shadow`
(`newuidmap`/`newgidmap`) and `dbus` (the user `dbus.socket`).
- **Non-native distros** (Fedora/Debian/Ubuntu) install the pinned,
sha256-verified `nerdctl-full` tarball (step `nerdctl-full-tarball`), which
supplies nerdctl + containerd + `containerd-fuse-overlayfs-grpc` + CNI +
buildkit + rootlesskit in one archive.
- `/etc/nerdctl/nerdctl.toml` (charly namespace + CNI path) and
the `charly` CNI network conflist at `/etc/cni/net.d/charly.conflist`.
**fatally** sha256-verified `nerdctl-full` tarball (step
`nerdctl-full-tarball`), which supplies nerdctl + containerd +
`containerd-fuse-overlayfs-grpc` + CNI + buildkit + rootlesskit in one archive.
The helpers come from `shadow-utils` (Fedora) or `uidmap`
(Debian/Ubuntu); the user bus from `dbus-daemon` or `dbus-user-session`.
- `/etc/nerdctl/nerdctl.toml` with `namespace = "default"` (the ONE spelling
the programme uses) and `cni_path = "/opt/cni/bin"`. The BuildKit socket is
deliberately NOT pinned: it is a runtime value resolved through the ONE roots
resolver (`$XDG_RUNTIME_DIR/charly/buildkitd.sock`).
- The `charly` CNI network conflist at `/etc/cni/net.d/charly.conflist`, and the CNI
plugins at the ONE `cni_path` the toml declares (`/opt/cni/bin`): Arch's package
installs there, the nerdctl-full tarball extracts to `/usr/local/libexec/cni`, and
a step links every real plugin to the declared path (no per-distro `cni_path`).
- The rootless **containerd and buildkit user units**
(`~/.config/systemd/user/{containerd,buildkit}.service`), written from the
upstream setuptool templates. `ExecStart` goes through `/usr/bin/env` so one
unit works for both the `/usr/local/bin` (tarball) and `/usr/bin` (package)
install locations.
- The **nested-pod posture**: userns-scoped `cap_add: ALL`, `/dev/fuse`,
`/dev/net/tun`, `unmask=/proc/*`.

## The rootless checklist (CONTAINERD.md §2 and §4)

Every item below is a real plan step with a `check:`; the runtime half is
asserted by the `check-nerdctl-rootless` disposable-VM bed.

| Item | Where | Why it exists |
|---|---|---|
| `newuidmap` / `newgidmap` | distro packages (`shadow`, `uidmap`, `shadow-utils`) | rootlesskit execs them to write the UID/GID map; the setuptool checks only the binaries **it** installs, never these two |
| `user:100000:65536` in `/etc/subuid` and `/etc/subgid` | plan step `rootless-subuid` | the ranges the helpers consume |
| `user@.service.d/delegate.conf` with `Delegate=cpu cpuset io memory pids` | plan step `rootless-delegate-controllers` | memory/cpuset/io limits and `--cpuset-cpus`; a new login picks it up |
| linger | bed, PERFORM (`loginctl enable-linger`) | the user manager and its units must outlive the login session |
| `/dev/fuse` | bed, PRESENCE (opens the node) | the fuse-overlayfs fallback is a libfuse3 client; the setuptool checks the binaries and never the node |
| user D-Bus (`dbus-user-session` / `dbus-daemon` / `dbus`) | package + bed (`dbus.socket`) | the rootless unit has `Requires=dbus.socket` |
| `slirp4netns` as the network driver | the containerd unit sets `CONTAINERD_ROOTLESS_ROOTLESSKIT_NET=slirp4netns` | the documented default |
| the setuptool run | bed, PERFORM (`install` + `install-buildkit`) | starts and enables the written units |
| `namespace = "default"` | `nerdctl.toml` | one namespace spelling, no per-layer name |

### Network drivers — the measured state

- **`slirp4netns` is the default** and is pinned in the unit.
- **`pasta` is an option, not a recommendation.** Measured on rootlesskit 3.3.0
with pasta `2026_10_02.cba3570`: `rootlesskit --net=pasta true` returns rc 0.
The deployment's networking properties (published ports bind `0.0.0.0`,
container→host loopback is blocked, no ephemeral host port) are measured on
slirp4netns, so pasta is not recommended.
- **`bypass4netns` is an option only where rootlesskit serves it.** The setuptool
can install `bypass4netnsd`, and nerdctl annotates a container with
`nerdctl/bypass4netns=true`. Measured on rootlesskit 3.3.0, both
`--net=bypass4netns` and `--port-driver=bypass4netns` are refused
(`unknown network mode` / `unknown port driver`), so this layer does not set it.

### Low ports — a documented option, never forced

Do **not** lower `net.ipv4.ip_unprivileged_port_start` for this layer. A rootless
**publish** of a port below 1024 fails `bind: permission denied` until the host
lowers it; the **container-side** threshold is the CNI netns's own `0`, so a
process inside a rootless lease binds a low port with no sysctl at all. If a
specific deploy needs the publish side, set it deliberately in
`/etc/sysctl.d/` on that host — never here, and never as a blanket default.

## Engine capability

The `engine: nerdctl` word is served by `opencharly/plugin-nerdctl`
Expand All @@ -35,7 +92,21 @@ The `engine: nerdctl` word is served by `opencharly/plugin-nerdctl`
host user under rootless), guaranteeing uid-identical sharing; a non-zero uid
needs the subuid caveat.

## Checks and the R10 bed

- The candy's `check:` steps prove the static half on every distro arm: the
helpers, the subuid ranges, the delegation drop-in, D-Bus, slirp4netns, the
written units (marker + `systemd-analyze --user verify`), `namespace =
"default"`, and a **negative** test that the checksum verifier rejects a
corrupted archive.
- `check-nerdctl-rootless` (root `charly.yml`, disposable VM) is the runtime
gate: ABSENCE (no rootless daemon) → PERFORM (daemon-reload, linger, the
setuptool) → PRESENCE (`nerdctl info` reports `name=rootless`, the helpers
resolve, `/dev/fuse` opens, the units are active). It records the rootful
system socket's inode/mtime before the daemon starts and asserts it is
unchanged after, and that the rootless daemon listens on the user socket.

## Use when

Working with the nerdctl engine, the layer-nerdctl candy, or an
`engine: nerdctl` deploy.
Working with the nerdctl engine, the layer-nerdctl candy, the rootless
containerd/buildkit checklist, or an `engine: nerdctl` deploy.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Build artefacts
.build/
# The VM bed's rendered disk + seed (charly vm build) — never authored, never tracked.
/image/

# Editor / OS
.DS_Store
Expand Down
11 changes: 9 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,15 @@ bed image in `box/check-nerdctl-app/charly.yml`. The candy carries the embedded

Canonical files:

- `charly.yml` — the project root: `discover:`, `defaults:`, and the
`check-nerdctl-nested:` disposable `pod:` R10 bed.
- `charly.yml` — the project root: `discover:`, `defaults:`, the
`check-nerdctl-nested:` disposable `pod:` bed (nested-pod posture), and the
`check-nerdctl-rootless:` disposable `vm:` bed (the CONTAINERD.md §2/§4
rootless checklist, whose runtime half needs a real systemd).
- `candy/layer-nerdctl/charly.yml` — the `layer-nerdctl:` candy entity (the
`NERDCTL_FULL_VERSION` var, the `security:` posture, the plan steps, the
`check:` assertions) and the `layer-nerdctl-skill:` skill entity.
- `.agents/skills/layer-nerdctl/SKILL.md` — the skill body (authored content,
discovered from the repo root).
- `box/check-nerdctl-app/charly.yml` — the disposable bed image composing the candy.
- `README.md` — user overview only; never agent guidance.

Expand Down Expand Up @@ -44,6 +48,9 @@ Canonical files:
`pod:` bed in the root `charly.yml` — a FRESH deploy asserting `/dev/fuse`,
`/dev/net/tun`, the full `CapEff` mask, the userns uid map, and `nerdctl` on
PATH. It fails without the `security:` block.
- The R10 for the rootless checklist is `check-nerdctl-rootless:` — a disposable
`vm:` bed, because the checklist's runtime half (`systemctl --user`, linger,
the units) cannot be proven in a supervisord pod.

## Modify this repo

Expand Down
39 changes: 28 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,39 @@ stack, as a standalone OpenCharly layer repo.

Composed by a box that wants the opt-in `engine: nerdctl` engine. On
Arch/CachyOS/Omarchy and Alpine the candy uses native distro packages; on Fedora,
Debian, and Ubuntu it installs the pinned, sha256-verified `nerdctl-full` tarball
(which supplies nerdctl + containerd + `containerd-fuse-overlayfs-grpc` + CNI +
buildkit + rootlesskit in one archive). It writes `/etc/nerdctl/nerdctl.toml`
(the `charly` namespace + rootless buildkit host) and the `charly` CNI network
conflist at `/etc/cni/net.d/charly.conflist`.
Debian, and Ubuntu it installs the pinned, **fatally** sha256-verified
`nerdctl-full` tarball (which supplies nerdctl + containerd +
`containerd-fuse-overlayfs-grpc` + CNI + buildkit + rootlesskit in one archive).
It writes `/etc/nerdctl/nerdctl.toml` (`namespace = "default"`), the `charly` CNI
network conflist at `/etc/cni/net.d/charly.conflist`, and the rootless
containerd/buildkit user units.

## What it provides

| Property | Value |
|---|---|
| Layer / candy | `layer-nerdctl` |
| Pinned version | `NERDCTL_FULL_VERSION` `2.1.2` |
| Config | `/etc/nerdctl/nerdctl.toml`, `/etc/cni/net.d/charly.conflist` |
| Pinned version | `NERDCTL_FULL_VERSION` `2.1.2`, sha256-verified fatally |
| Config | `/etc/nerdctl/nerdctl.toml` (`namespace = "default"`), `/etc/cni/net.d/charly.conflist`, CNI plugins at `/opt/cni/bin` |
| Rootless units | `~/.config/systemd/user/containerd.service`, `~/.config/systemd/user/buildkit.service` |
| Rootless checklist | uidmap helpers, `/etc/subuid` + `/etc/subgid` (`user:100000:65536`), `user@.service.d/delegate.conf`, linger, `/dev/fuse`, user D-Bus, slirp4netns (CONTAINERD.md §2, §4) |
| Nested-pod posture | userns-scoped `cap_add: ALL`, `/dev/fuse`, `/dev/net/tun`, `unmask=/proc/*` |
| Service / port | none |

The engine word is served by `opencharly/plugin-nerdctl` (out-of-process) or
compiled in; every engine op delegates to `container.InvokeEngineOp("nerdctl", …)`.

## Rootless networking

`slirp4netns` is the default and is pinned in the containerd unit. `pasta` and
`bypass4netns` are documented options, not recommendations: on rootlesskit 3.3.0
pasta starts (`rootlesskit --net=pasta true`, rc 0) but the deployment's
networking properties are measured on slirp4netns, and bypass4netns is refused by
that rootlesskit (`unknown network mode` / `unknown port driver`). Publishing a
port below 1024 is governed by the host's
`net.ipv4.ip_unprivileged_port_start`; this layer never sets it — see the skill
for when to do it deliberately.

## How to use it

This repo uses the multi-directory layout: the root `charly.yml` is the project
Expand All @@ -41,17 +55,20 @@ my-nested-box:

## Layout

- `charly.yml` — the project root: `discover:`, `defaults:`, and the
`check-nerdctl-nested:` disposable `pod:` R10 bed.
- `charly.yml` — the project root: `discover:`, `defaults:`, the
`check-nerdctl-nested:` disposable `pod:` bed (nested-pod posture), and the
`check-nerdctl-rootless:` disposable VM bed (the §4 rootless checklist).
- `candy/layer-nerdctl/charly.yml` — the `layer-nerdctl:` candy entity, its
`security:` posture, and the embedded `layer-nerdctl-skill:` skill entity.
`security:` posture, the plan steps/checks, and the `layer-nerdctl-skill:`
skill entity.
- `.agents/skills/layer-nerdctl/SKILL.md` — the skill body.
- `box/check-nerdctl-app/charly.yml` — the disposable bed image composing the candy.
- `README.md` — this user overview.

## Related

- Owning skill: `/charly-distros:layer-nerdctl` — the nerdctl engine, the
`layer-nerdctl` candy, and `engine: nerdctl` deploys.
`layer-nerdctl` candy, the rootless checklist, and `engine: nerdctl` deploys.
- `/charly-internals:plugin-nerdctl` — the out-of-tree engine plugin.
- `/charly-image:layer` — candy authoring reference.
- [`opencharly/charly`](https://github.com/opencharly/charly) — the charly CLI and image builder
Expand Down
4 changes: 4 additions & 0 deletions box/check-nerdctl-app/charly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ check-nerdctl-app:
Disposable bed image for the layer-nerdctl R10 — a Fedora base composing
layer-nerdctl, so the nested-pod posture is emitted and can be asserted.
base: quay.io/fedora/fedora:43
# The image's distro tag chain. Without it `img.DistroDef` is nil and the generator
# emits NO declarative package installs, so a candy's `distro.fedora.package:` list
# silently does nothing (the tarball still installs, which is how this went unseen).
distro: ["fedora:43", fedora]
candy:
- layer-nerdctl
plan:
Expand Down
Loading