From 82d2301435d70d8541216f2f57b7e09bb174fce9 Mon Sep 17 00:00:00 2001 From: Andreas Trawoeger Date: Fri, 23 Oct 2026 21:14:46 +0200 Subject: [PATCH] docs: regenerate the site at marketplace 76b8c06 and fix the kubernetes engine mapping (#177) The marketplace gitlink sat at b879d711 (about twenty corpus commits behind), so the published pages still carried the retired engine word and the pre-rename `charly fleet` vocabulary. Advanced to 76b8c06 (v2026.296.1903) and regenerated with the deploy workflow's exact inputs: charly bafd8dbc, narrative umbrella e631aacf, plugin-docs v2026.289.0814, and the docs-corpus wiring candy built from pins.tsv. guides/kubernetes.md is hand-authored: the engine word maps to kind's KIND_EXPERIMENTAL_PROVIDER through the engine capability data (containerd -> nerdctl), not 1:1, and the third bed is rootless containerd (the entity name keeps kind's nerdctl provider word). 38 generated pages change, including one new page (recipes/check/clawsweeper-review.md). Five pages lose net lines; each is a faithful projection of its current source (page lines = source lines + 1), and the dropped text is absent from the corpus. A second regeneration is a no-op. Assisted-by: Claude Code deepseek-v4.1-flash:cloud (fully tested and validated) --- marketplace | 2 +- src/content/docs/guides/kubernetes.md | 15 +- src/content/docs/recipes/build/build.md | 168 ++++++++++------ src/content/docs/recipes/build/docs.md | 40 +--- src/content/docs/recipes/build/generate.md | 6 +- src/content/docs/recipes/build/migrate.md | 113 ++++++----- src/content/docs/recipes/build/pull.md | 4 +- src/content/docs/recipes/build/settings.md | 6 +- .../docs/recipes/check/clawsweeper-review.md | 76 +++++++ src/content/docs/recipes/check/crabbox.md | 73 ++++++- .../docs/recipes/core/charly-doctor.md | 2 +- .../docs/recipes/core/charly-status.md | 8 +- src/content/docs/recipes/core/clean.md | 30 +-- src/content/docs/recipes/core/deploy.md | 131 ++++++------ .../docs/recipes/distros/container-nesting.md | 45 ++++- .../recipes/distros/githubrunner-ubuntu.md | 59 ++++-- .../docs/recipes/distros/layer-nerdctl.md | 98 +++++++-- src/content/docs/recipes/distros/nvidia.md | 43 ++++ .../docs/recipes/hermes/playwright-layer.md | 2 + src/content/docs/recipes/index.md | 3 +- .../docs/recipes/infrastructure/k3s.md | 66 +++++- .../docs/recipes/infrastructure/tmux-layer.md | 1 + .../docs/recipes/internals/generate-source.md | 2 +- .../multistage-and-intermediates.md | 8 +- .../generate-source/task-emission.md | 6 +- .../git-workflow/umbrella-mechanics.md | 20 ++ .../docs/recipes/internals/plugin-nerdctl.md | 39 +++- .../docs/recipes/kubernetes/check-k8s.md | 3 +- src/content/docs/recipes/kubernetes/helm.md | 16 +- src/content/docs/recipes/kubernetes/kind.md | 10 +- .../docs/recipes/local/local-deploy.md | 8 +- .../docs/recipes/openwebui/openwebui.md | 2 +- .../docs/recipes/strata/strata-layer.md | 54 ++++- src/content/docs/recipes/strata/strata.md | 189 ++++++++++++++++-- src/content/docs/recipes/tools/crabbox-cli.md | 86 +++++--- src/content/docs/recipes/tools/docs-site.md | 17 ++ src/content/docs/recipes/tools/goplaces.md | 4 +- src/content/docs/recipes/tools/himalaya.md | 2 - src/content/docs/recipes/vm/vm.md | 54 ++--- 39 files changed, 1086 insertions(+), 425 deletions(-) create mode 100644 src/content/docs/recipes/check/clawsweeper-review.md diff --git a/marketplace b/marketplace index b879d711..76b8c065 160000 --- a/marketplace +++ b/marketplace @@ -1 +1 @@ -Subproject commit b879d71102631f928349558aebcbdee7c359765d +Subproject commit 76b8c0653965f0e0b9c396cedddd7fc8f21829b9 diff --git a/src/content/docs/guides/kubernetes.md b/src/content/docs/guides/kubernetes.md index a9937d85..43aceb37 100644 --- a/src/content/docs/guides/kubernetes.md +++ b/src/content/docs/guides/kubernetes.md @@ -34,11 +34,14 @@ A local-cluster sibling reuses the same Kustomize + `kubectl` machinery without upstream `kind` tool, served by the `deploy:kindcluster` provider beside `deploy:kubernetes` in `plugin-kube` ([kind](/recipes/kubernetes/kind/)). -kind provisions its node containers on the operator's own container engine, and charly's engine -word maps 1:1 to kind's `KIND_EXPERIMENTAL_PROVIDER`. Three disposable beds witness the three -engines — podman (`check-kindcluster`), docker (`check-kindcluster-docker`) and rootless nerdctl -(`check-kindcluster-nerdctl`) — differing only in `engine:` and a collision-free cluster name and -kubeconfig context, since a kind cluster name is unique only within an engine. +kind provisions its node containers on the operator's own container engine. charly's engine +word maps to kind's `KIND_EXPERIMENTAL_PROVIDER` through the engine capability data +(`podman` → `podman`, `docker` → `docker`, `containerd` → `nerdctl`), and any word outside +the engine vocabulary is refused. Three disposable beds witness the three engines — podman +(`check-kindcluster`), docker (`check-kindcluster-docker`) and rootless containerd +(`check-kindcluster-nerdctl`, whose entity name keeps kind's `nerdctl` provider word) — differing +only in `engine:` and a collision-free cluster name and kubeconfig context, since a kind cluster +name is unique only within an engine. ## k3s runs in a VM, not a pod @@ -69,7 +72,7 @@ Every bed below is declared in charly's own `charly.yml`: | `check-kubevirt-operator` | the KubeVirt + CDI platform install on a disposable k3s VM | | `check-kindcluster` | kind on the podman engine | | `check-kindcluster-docker` | kind on the docker engine | -| `check-kindcluster-nerdctl` | kind on the rootless nerdctl engine | +| `check-kindcluster-nerdctl` | kind on the rootless containerd engine (the entity name keeps kind's `nerdctl` provider word) | | `check-kindcluster-workload` | the `deploy:kindcluster` workload leg — a locally-built image loaded into the kind node and applied through the shared Kustomize generator | | `check-kindcluster-lock` | teardown lock inheritance — a kindcluster root with a deploy-level member, torn down together | | `check-kind-host-vm` | the kind-host profile applied into a disposable eval-vm guest, off the operator's workstation | diff --git a/src/content/docs/recipes/build/build.md b/src/content/docs/recipes/build/build.md index 01e1ca4c..52b8a690 100644 --- a/src/content/docs/recipes/build/build.md +++ b/src/content/docs/recipes/build/build.md @@ -13,13 +13,13 @@ Invoked as `charly box build`. See [`/charly-image:image`](/recipes/image/image/ ## Overview -`charly box build` generates Containerfiles from `charly.yml` and layer definitions, then builds images in dependency order with **BuildKit**, charly's only image builder. Images at the same dependency level are built in parallel (up to `--jobs` concurrent builds). +`charly box build` generates Containerfiles from `charly.yml` and layer definitions, then builds images in dependency order using the configured build engine (Docker or Podman). Images at the same dependency level are built in parallel (up to `--jobs` concurrent builds). **Mode purity**: `charly box build` reads `charly.yml` only. `charly.yml` is never read during build — this is enforced by `LoadConfig` in `charly/loader_threaded.go` (it reads the PROJECT `charly.yml` only and never merges the per-host `charly.yml` overlay — no `MergeDeployOverlay`), to guarantee OCI labels are baked strictly from authored configuration, never from local deploy-time overrides. See [`/charly-internals:go`](/recipes/internals/go/) "Mode purity" for the architectural invariant this protects and the bug it prevents. -**Build dispatch**: `charly box build` does not construct a generator host-side at all (the former `NewGenerator` is DELETED) — `BuildCmd.Run` computes the build tag, holds the build-activity retention lock, and dispatches through the COMPILED-IN `build:box` plugin (candy/plugin-build) over the F10 reverse channel; on return it runs the retention prune host-side. The plugin's `runBoxBuild` → `resolveBuildEngine` (candy/plugin-build/resolve.go) runs the ENTIRE build RESOLVE plugin-side — loader, vocab, scan, connect, validate, box resolve, intermediates, order, effective versions, host-fs prep, render-seam-cache prep, render-prep, user-context, envelope projection, and drive-model — and returns `spec.BuildResolveReply` (the resolved-project envelope + a serializable drive-model: engine/platform, build order, per-image descriptors, resolved tunables; NO Containerfile content). This is the plugin-side replacement for the DELETED host `build-prep` fat seam (`charly/build_resolve_host.go`'s `hostBuildBuildResolve`). The plugin **OWNS the BuildKit DRIVE** (the build-order loop, per-image build lock, the `buildctl` argv, the store bridge, push, and the inline-merge gate) IN the candy — it imports only the sdk module. It reaches the host ONLY for thin `buildengine-*` shards a sdk-only candy cannot hold: `buildengine-scan-local` (local candy scan), `buildengine-connect-plugins` (registry plugin connect), and `buildengine-prep` (the render-seam-floor `renderGenCache` — `hostBuildPrep` stores a CHEAP candy-scan-only Generator for the render-seam floor's reverse-channel consumers and returns an EMPTY map; the host-fs PREP itself moved plugin-side in the K3 host-prep move — `runHostFSPrep` in candy/plugin-build/host_prep.go). The resolved-project envelope + drive-model are NOT returned by `buildengine-prep` — they come from `resolveBuildEngine`'s reply. The render DRIVE is `sdk/deploykit.Generator`, driven by the candy over the envelope + `HostBuild("render-seam")` for the host-coupled render seams (#67 — the host no longer renders the `.build/` tree); `bake_plugin:` binary baking is INLINE via `deploykit.EmitBakedPlugins` (no `HostBuild("bake-plugins")` round-trip — that host-builder + `host_build_bake_plugins.go` are DELETED). `charly box generate` mirrors this via `build:generate` → `runBoxGenerate` → `resolveBuildEngine(..., generateOnly=true)`: `generateOnly` is a PLUGIN-SIDE parameter (NOT a `HostBuild("buildengine-prep")` argument — that host leg takes `spec.ResolvedProjectRequest`, which has no `GenerateOnly` field) that returns after the envelope projection (no engine/order drive-model); the candy renders the `.build/` tree itself via `deploykit.Generator` + returns the written Containerfile paths; no podman. (The resolved-project envelope that OTHER verbs fetch — `charly box inspect`, `charly check`, … — comes over the separate `InvokeProvider("build","project",OpResolve)` peer seam, the `build:project` word; the build/generate path projects its own envelope inside `resolveBuildEngine`.) See [`/charly-internals:plugin`](/recipes/internals/plugin/) (the `build` provider class + the in-proc reverse channel) and [`/charly-build:generate`](/recipes/build/generate/). +**Build-engine dispatch**: `charly box build` does not construct a generator host-side at all (the former `NewGenerator` is DELETED) — `BuildCmd.Run` computes the build tag, holds the build-activity retention lock, and dispatches through the COMPILED-IN `build:box` plugin (candy/plugin-build) over the F10 reverse channel; on return it runs the retention prune host-side. The plugin's `runBoxBuild` → `resolveBuildEngine` (candy/plugin-build/resolve.go) runs the ENTIRE build-engine RESOLVE plugin-side — loader, vocab, scan, connect, validate, box resolve, intermediates, order, effective versions, host-fs prep, render-seam-cache prep, render-prep, user-context, envelope projection, and drive-model — and returns `spec.BuildResolveReply` (the resolved-project envelope + a serializable drive-model: engine/platform, build order, per-image descriptors, resolved tunables; NO Containerfile content). This is the plugin-side replacement for the DELETED host `build-prep` fat seam (`charly/build_resolve_host.go`'s `hostBuildBuildResolve`). The plugin **OWNS the podman DRIVE** (the build-order loop, per-image build lock, `podman build`, push, and the inline-merge gate) IN the candy — it imports only the sdk module. It reaches the host ONLY for thin `buildengine-*` shards a sdk-only candy cannot hold: `buildengine-scan-local` (local candy scan), `buildengine-connect-plugins` (registry plugin connect), and `buildengine-prep` (the render-seam-floor `renderGenCache` — `hostBuildPrep` stores a CHEAP candy-scan-only Generator for the render-seam floor's reverse-channel consumers and returns an EMPTY map; the host-fs PREP itself moved plugin-side in the K3 host-prep move — `runHostFSPrep` in candy/plugin-build/host_prep.go). The resolved-project envelope + drive-model are NOT returned by `buildengine-prep` — they come from `resolveBuildEngine`'s reply. The render DRIVE is `sdk/deploykit.Generator`, driven by the candy over the envelope + `HostBuild("render-seam")` for the host-coupled render seams (#67 — the host no longer renders the `.build/` tree); `bake_plugin:` binary baking is INLINE via `deploykit.EmitBakedPlugins` (no `HostBuild("bake-plugins")` round-trip — that host-builder + `host_build_bake_plugins.go` are DELETED). `charly box generate` mirrors this via `build:generate` → `runBoxGenerate` → `resolveBuildEngine(..., generateOnly=true)`: `generateOnly` is a PLUGIN-SIDE parameter (NOT a `HostBuild("buildengine-prep")` argument — that host leg takes `spec.ResolvedProjectRequest`, which has no `GenerateOnly` field) that returns after the envelope projection (no engine/order drive-model); the candy renders the `.build/` tree itself via `deploykit.Generator` + returns the written Containerfile paths; no podman. (The resolved-project envelope that OTHER verbs fetch — `charly box inspect`, `charly fleet compile`, `charly check`, … — comes over the separate `InvokeProvider("build","project",OpResolve)` peer seam, the `build:project` word; the build/generate path projects its own envelope inside `resolveBuildEngine`.) See [`/charly-internals:plugin`](/recipes/internals/plugin/) (the `build` provider class + the in-proc reverse channel) and [`/charly-build:generate`](/recipes/build/generate/). -**Build-mode emission**: the build-mode Containerfile render DRIVE is `sdk/deploykit.Generator.Generate` / `generateContainerfile` (relocated from `charly/generate.go` in #67), driven by `candy/plugin-build` over the resolved-project envelope + `HostBuild("render-seam")`; the `charly/generate.go` top-level `Generate`/`writeCandySteps` orchestrator is DELETED. the pod-overlay deploy path reaches `deploykit.Generator.EmitTasks` DIRECTLY — `charly/tasks.go` holds no shim and cannot hold one (P16 import purity forbids `charly/` importing the sdk); that path stays for the pod-overlay deploy path (the candy `plugin-deploy-pod` constructs `deploykit.OCITarget` + renders via `deploykit.NewRenderGeneratorFromProject`, the per-step fragment over the `HostBuild("step-emit","oci-emit-step")` seam); the build-mode render (in `sdk/deploykit`) drives `WriteCandySteps` → `EmitTasks` directly. The render walks each layer's ops directly to write Containerfile text — NOT the `InstallPlan` IR. It shares the package-cascade / shell-snippet / localpkg compiler helpers with the IR (one source of truth, R2), but the overall walk is the direct generator. The `InstallPlan` IR + `deploykit.OCITarget.Emit` is the DEPLOY-mode path used by `charly deploy add` (the candy `plugin-deploy-pod`'s `add_candy:` overlay synthesis, rendered in its own code via `deploykit.OCITarget` + `deploykit.NewRenderGeneratorFromProject` + the external out-of-process deploys; the local/vm/kubernetes/android substrates are external plugins via `pluginDeployTarget` (S3b, dispatched through `candy/plugin-fleet`'s `Invoke(OpDeployDispatch)`) — `deploy:local` via candy/plugin-deploy-local and `deploy:vm` via candy/plugin-deploy-vm DO consume the IR (the plugin walks it via `kit.WalkPlans` over the reverse channel, the vm one over the guest `SSHExecutor`), while `deploy:kubernetes` via candy/plugin-kube does NOT, materializing its Kustomize tree PLUGIN-side (`candy/plugin-kube/materialize.go`'s `materializeKustomize`, reaching `candy/plugin-k8sgen` peer-to-peer over `verb:k8sgen`/`OpEmit`)). See [`/charly-internals:install-plan`](/recipes/internals/install-plan/) for the IR and [`/charly-internals:generate-source`](/recipes/internals/generate-source/) for the Go call graph. +**Build-mode emission**: the build-mode Containerfile render DRIVE is `sdk/deploykit.Generator.Generate` / `generateContainerfile` (relocated from `charly/generate.go` in #67), driven by `candy/plugin-build` over the resolved-project envelope + `HostBuild("render-seam")`; the `charly/generate.go` top-level `Generate`/`writeCandySteps` orchestrator is DELETED. the pod-overlay deploy path reaches `deploykit.Generator.EmitTasks` DIRECTLY — `charly/tasks.go` holds no shim and cannot hold one (P16 import purity forbids `charly/` importing the sdk); that path stays for the pod-overlay deploy path (the candy `plugin-deploy-pod` constructs `deploykit.OCITarget` + renders via `deploykit.NewRenderGeneratorFromProject`, the per-step fragment over the `HostBuild("step-emit","oci-emit-step")` seam); the build-mode render (in `sdk/deploykit`) drives `WriteCandySteps` → `EmitTasks` directly. The render walks each layer's ops directly to write Containerfile text — NOT the `InstallPlan` IR. It shares the package-cascade / shell-snippet / localpkg compiler helpers with the IR (one source of truth, R3), but the overall walk is the direct generator. The `InstallPlan` IR + `deploykit.OCITarget.Emit` is the DEPLOY-mode path used by `charly fleet add` (the candy `plugin-deploy-pod`'s `add_candy:` overlay synthesis, rendered in its own code via `deploykit.OCITarget` + `deploykit.NewRenderGeneratorFromProject` + the external out-of-process deploys; the local/vm/kubernetes/android substrates are external plugins via `pluginDeployTarget` (S3b, dispatched through `candy/plugin-fleet`'s `Invoke(OpDeployDispatch)`) — `deploy:local` via candy/plugin-deploy-local and `deploy:vm` via candy/plugin-deploy-vm DO consume the IR (the plugin walks it via `kit.WalkPlans` over the reverse channel, the vm one over the guest `SSHExecutor`), while `deploy:kubernetes` via candy/plugin-kube does NOT, materializing its Kustomize tree PLUGIN-side (`candy/plugin-kube/materialize.go`'s `materializeKustomize`, reaching `candy/plugin-k8sgen` peer-to-peer over `verb:k8sgen`/`OpEmit`)). See [`/charly-internals:install-plan`](/recipes/internals/install-plan/) for the IR and [`/charly-internals:generate-source`](/recipes/internals/generate-source/) for the Go call graph. **Three-phase templates**: the embedded build vocabulary's format and builder definitions split each install operation into `phases.{prepare, install, cleanup}.{container, host}` — three phases × two venues. Build-mode emission reads the `container` cell; local deploys read `host` — the `(install, container)` cell is the single source of truth for build-mode package installation. See [`/charly-image:layer`](/recipes/image/layer/) "Service Declaration" for the analogue at the init-system level (`init..service_schema`). @@ -44,7 +44,7 @@ charly box build --cache image [image...] # Image cache (read-only, de charly box build --cache gha [image...] # GitHub Actions cache charly box build --no-cache [image...] # Disable cache entirely charly box build --jobs N [image...] # Max concurrent images per DAG level (0=auto from defaults.jobs, else 4) -charly box build --stage-parallelism N [image...] # Max concurrent stages within a single build (0=auto: min(NCPU, defaults.stage_parallelism_cap)) +charly box build --podman-jobs N [image...] # Max concurrent stages within a single podman build (0=auto: min(NCPU, defaults.podman_jobs_cap)) charly box build --include-disabled # Build an `enabled: false` image without flipping authored config charly box build --dev-local-pkg # EVAL-BED ONLY: build localpkg candies (the charly toolchain) from LOCAL in-dev source, not the published release ``` @@ -59,7 +59,7 @@ in-development binary + plugins — a disposable check bed always tests the in-development charly, never a stale release. Generic across all kinds + all localpkg candies, one decision point (`deploykit.RenderLocalPkgImageInstall`); a production box build omits the flag. A dev build that cannot build via the plugin -HARD-errors (no silent release fallback, R3). +HARD-errors (no silent release fallback, R4). Full mechanics: [`/charly-internals:install-plan`](/recipes/internals/install-plan/) "Check-vs-production charly toolchain"; the candy view: [`/charly-tools:charly`](/recipes/tools/charly/). @@ -164,46 +164,48 @@ cat .build/my-image/Containerfile # Inspect generated output ## Build Flow 1. Run `charly box generate` internally (produces Containerfiles in `.build/`) -2. Resolve runtime config (there is no build-engine selection: BuildKit is the builder) +2. Resolve runtime config to get build engine (`engine.build`) 3. Resolve image build order (dependency ordering, grouped by level) 4. Filter to requested images (and their base dependencies) 5. For each level: build images in parallel (up to `--jobs` concurrent, default 4) 6. After all builds: `charly box merge --all` (if `merge.auto` enabled, skipped for `--push`) -## Parallelism: `--jobs` vs `--stage-parallelism` +## Parallelism: `--jobs` vs `--podman-jobs` charly exposes **two** parallelism knobs with distinct meanings: | Flag | Env var | `defaults:` key | What it controls | |---|---|---|---| | `--jobs N` | `CHARLY_BUILD_JOBS` | `jobs` | **Outer** concurrency: how many charly-level images to build in parallel within a DAG level (e.g., when `charly box build` rebuilds the whole graph). | -| `--stage-parallelism N` | `CHARLY_STAGE_PARALLELISM` | `stage_parallelism` (+ `stage_parallelism_cap`) | **Inner** concurrency: how many stages of a *single* multi-stage build run concurrently. It becomes the BuildKit daemon's `[worker.oci] max-parallelism`, applied when the daemon starts. | +| `--podman-jobs N` | `CHARLY_PODMAN_JOBS` | `podman_jobs` (+ `podman_jobs_cap`) | **Inner** concurrency: passed to `podman build --jobs N`, controls how many stages of a *single* multi-stage build run concurrently. | Both knobs are **config-driven**: precedence is **CLI flag → env → `defaults:` in `charly.yml` → built-in fallback** (4). The inner auto value (when -`--stage-parallelism` / `CHARLY_STAGE_PARALLELISM` / `defaults.stage_parallelism` are all unset) is -CPU-proportional, capped at `defaults.stage_parallelism_cap`: `min(NCPU, cap)`. -The cap is the operative ceiling, and it is the project's own to set. (The 20-run -race gate required before raising it on a new engine version is in `CHANGELOG/`.) +`--podman-jobs` / `CHARLY_PODMAN_JOBS` / `defaults.podman_jobs` are all unset) is +CPU-proportional, capped at `defaults.podman_jobs_cap`: `min(NCPU, cap)`. The +cap is the operative ceiling; the repo ships `podman_jobs_cap: 8`. (A +high-concurrency `--cache-from` SIGABRT race in podman ≤ 5.7.x originally +motivated a hard cap of 4 — see `CHANGELOG/` for that history and the 20-run +race gate required before raising the cap on a new podman version.) ```yaml # charly.yml — defaults: block defaults: jobs: 4 # outer: images per DAG level - stage_parallelism: 0 # inner: 0 = auto = min(NCPU, stage_parallelism_cap) - stage_parallelism_cap: 8 # ceiling for the auto calc + podman_jobs: 0 # inner: 0 = auto = min(NCPU, podman_jobs_cap) + podman_jobs_cap: 8 # ceiling for the auto calc ``` ```bash -charly box build --stage-parallelism 16 # fully parallel stages (override) -CHARLY_STAGE_PARALLELISM=8 charly update --build # via env -charly box build --stage-parallelism 1 # fully serialised, worst-case debugging +charly box build --podman-jobs 16 # fully parallel stages (override) +CHARLY_PODMAN_JOBS=8 charly update --build # via env +charly box build --podman-jobs 1 # fully serialised, worst-case debugging ``` -Source: `candy/plugin-build/resolve.go` (`resolveDriveStageParallelism`, feeding -`buildkit.ResolveStageParallelism`; the host-resolved `StageParallelism` value is the -`MaxParallelism` the drive hands the daemon in `candy/plugin-build/buildkit.go`). -Covered by `buildkit/build_helpers_test.go` (`TestResolveStageParallelism`). The outer `--jobs` knob and the inner `--stage-parallelism` +Source: `candy/plugin-build/resolve.go` (`resolveDrivePodmanJobs`, feeding +`buildkit.ResolvePodmanJobs`; the host-resolved `PodmanJobs` value feeds `--jobs` +directly in `candy/plugin-build/podman.go`). Covered by +`candy/plugin-build/podman_test.go`. The outer `--jobs` knob and the inner `--podman-jobs` are separate fields so the two semantics don't get conflated. ## Build-context excludes (`defaults.context_ignore`) @@ -227,9 +229,8 @@ defaults: Why it matters: the build context tar is streamed to the engine on **every** build regardless of cache state, so excluding large never-COPYed directories is -the dominant warm-rebuild win. BuildKit's dockerfile frontend reads `.dockerignore`; -`.containerignore` is the engine-native name kept for engine-native builds — emitting both -keeps the two readers in lockstep. Only add a +the dominant warm-rebuild win. podman reads `.containerignore`; docker reads +`.dockerignore` — emitting both keeps the two engines in lockstep. Only add a directory you've confirmed no Containerfile COPY/ADDs from (generated Containerfiles COPY only from `candy/`, `templates/`, `.build/`). Source: `candy/plugin-build/host_prep.go:writeContextIgnore` + `charly/host_build_buildengine.go` @@ -287,8 +288,8 @@ what actually orders them; **within one image's tag rows**, the *"creation time, not the tag"* across to the tag budget — it is true of the first and false of the second. -Images referenced by a running container are skipped, and removal runs without `-f` as a -backstop (`charly clean` owns the prune; never call the engine's prune directly). +Images referenced by a container (`podman ps -a`) are skipped, and `rmi` runs without +`-f` as a backstop. ```yaml # charly.yml — defaults: @@ -333,8 +334,10 @@ OCI labels. The CLI accepts an explicit `--tag ` for pinning; an empty `--tag` resolves to newest-local automatically. Rationale: -1. **Nothing can pin a stale float tag.** charly never emits `:latest`, so a build can never - resolve to a float tag a previous build left behind. +1. **No stale-`:latest`-under-`--cache-from` bug.** The documented + caveat where podman's `--cache-from` silently pulls a stale + `:latest` from the remote registry is unreachable when charly never + emits `:latest` in the first place. 2. **No "ambiguous short name" errors under multiple tags.** Previously the resolver listed every `:latest` + `:` tag of the same image as competing candidates; the CalVer sort picks the newest @@ -344,7 +347,7 @@ Rationale: a CalVer tag is self-describing and rotates naturally. Any stray `:latest` tag in local storage is never refreshed by a new -build; prune it with `charly clean` if you want it gone. +build; prune it with `podman image prune` if you want it gone. ## Cache Efficiency @@ -364,7 +367,7 @@ Three kinds of source changes are real cache invalidators — if you see a long ### What does NOT invalidate cache -- **CalVer tag shifts alone.** The Containerfile emits `ARG BASE_IMAGE=/:` with a fresh CalVer on every generate. That ARG default appears in the Containerfile text but is not part of the cache key for subsequent RUN/COPY steps — BuildKit resolves the FROM to an image digest first, and caches downstream steps off that digest. If the SHA is unchanged, cache hits. The cache cost comes from content changes in the layer itself, not the tag. +- **CalVer tag shifts alone.** The Containerfile emits `ARG BASE_IMAGE=/:` with a fresh CalVer on every generate. That ARG default appears in the Containerfile text but is not part of the cache key for subsequent RUN/COPY steps — podman/buildah resolves the FROM to an image SHA first, and caches downstream steps off that SHA. If the SHA is unchanged, cache hits. The cache cost comes from content changes in the layer itself, not the tag. - **Baked `plan:` edits.** LABEL directives are emitted last in every final stage (after the last USER). Editing a baked step (a `check:`/`agent-check:` step, or a runtime-context `run:` step) — a common layer mutation — only re-runs the final LABEL block (~2 seconds on a 138-step stack like `immich-ml`). (Editing a build/deploy-context `run:` step changes the install timeline, so it re-runs that RUN/COPY instead.) See [`/charly-internals:generate-source`](/recipes/internals/generate-source/) for the rationale. - **Re-running `charly box build` without source changes.** Fully cached; seconds to complete. @@ -388,15 +391,16 @@ Three kinds of source changes are real cache invalidators — if you see a long ## Build Flow Details -**Internal base images** use exact CalVer tags in Containerfiles (`FROM ghcr.io/opencharly/fedora:2026.46.1415`), so each image names the precise version of its parent. **The builder does not resolve that tag through a local-image lookup**: BuildKit's OCI worker has no local-first resolution. The built image is stored in the configured RUN engine's store (`LoadExport`: load → tag → verify), where every later read of it — capability labels, deploy, status, inline merge, `charly clean` — resolves it; it is ALSO pushed to charly's local OCI registry, which the buildkitd mirrors `ghcr.io/opencharly` to, so a dependent box's `FROM ${BASE_IMAGE}` resolves a locally built base instead of the upstream registry (`opencharly/charly#937`, `opencharly/layer-charly-build#25`). +**Internal base images** use exact CalVer tags in Containerfiles (`FROM ghcr.io/opencharly/fedora:2026.46.1415`). This ensures each image references the precise version of its parent. Both Docker and Podman resolve local images before pulling from registry. ## Push Mode -BuildKit's image exporter is the ONE push path — `--output type=image,name=,push=true` — -and the only one that can push the multi-platform image a `--push` build produces. It -replaces the engine-native push drivers — no per-engine push path remains. +- **Docker:** `docker buildx build --push` for multi-platform builds +- **Podman:** `podman build --manifest` + `podman manifest push` -Source: `candy/plugin-build/buildkit.go` (`pushOutput`). +Podman manifest push uses retry with exponential backoff (3 attempts, 5s/10s/20s delays) to handle transient registry errors (e.g., GHCR 500 errors after long builds). + +Source: `candy/plugin-build/podman.go` (`retryCmd`). ## Layer Merging @@ -443,17 +447,13 @@ For filtered builds (`charly box build `), only the built images are merg ## Engine Configuration ```bash -charly settings set engine.run docker # or podman — which engine RUNS containers +charly settings set engine.build docker # or podman +charly settings set engine.run docker # or podman ``` -**There is no `engine.build`**: the image builder is BuildKit, which is not a container -engine — it cannot run a container and therefore cannot be selected like one. `engine.run` -is unaffected and still selectable. - ## Host Prerequisites (First Time) -Building images needs the `charly` binary and a container engine — `docker` or `podman`, whichever `engine.run` names (BuildKit builds the image; the drive bridges it into -that engine's store). Nothing else, and no project checkout: `charly box build` runs against the `charly.yml` in the current directory, one started by `charly box new project ` or read from a published project with `--repo /`. +Building images needs the `charly` binary and a container engine — `docker` or `podman`, whichever `engine.build` names. Nothing else, and no project checkout: `charly box build` runs against the `charly.yml` in the current directory, one started by `charly box new project ` or read from a published project with `--repo /`. Install `charly` as a distro-native package with your OWN package manager (`pacman -U` / `dnf install` / `apt install` / `apk add`) — the `charly generate-packages` plugin (nFPM) builds the `.pkg.tar.zst`/`.rpm`/`.deb`/`.apk`/`.ipk` published to the per-distro package repos. Obtaining the binary is covered end to end at [Install](https://opencharly.ai/start/install/); it is the one page that describes building from a source tree. @@ -516,13 +516,9 @@ Beyond the YAML-unmarshal symptom above, a stale `charly` binary can produce *sy If the symptom persists after upgrading, the copy on `$PATH` may not be the one you just installed — `which charly` reports which wins. -### Cache-mount corruption (pixi tzdata et al.) +### Buildah cache-mount corruption (pixi tzdata et al.) -Cache mounts are declared as `--mount=type=cache,id=charly-,dst=,sharing=locked` (pixi, npm, cargo, -rattler and dnf all use one — `buildkit.SharedCacheMount` emits the format). Their **id is stable on purpose**: -that is what makes the cache survive layer-hash churn, which is the entire reason cache mounts exist -(`candy/plugin-build/cache_mount_test.go`). A cache mount is therefore not the build cache: `charly box build ---no-cache` disables the build cache for that build, and the mount's contents persist regardless. A disk-full event mid-download can leave a partial package in the cache, and every subsequent build re-hits the same broken file: +Cache mounts declared via `--mount=type=cache,dst=,uid=,gid=` (used for pixi, npm, cargo, rattler, dnf) persist across builds and are **not** evicted by `charly box build --no-cache` (which only suppresses `--cache-from`; see above). A disk-full event mid-download can leave a partial package in the cache, and every subsequent build re-hits the same broken file: ``` error: × failed to link tzdata-2025c-hc9c84f9_1.conda @@ -533,27 +529,77 @@ Options, in order of least to most disruptive: 1. **Just retry the build.** Often the cache clears itself once the partial package finishes downloading or gets superseded by a checksum mismatch. 2. **Bump a content-hash on the builder layer** (e.g. `echo "" >> candy/pixi/charly.yml`) to evict the cache mount associated with that build-stage's hash, then revert the cosmetic change. Same workaround as the scratch-stage cache issue above. -3. **The engine's `image prune -af` is NOT the right hammer** — use `charly clean`; see next note. +3. **`podman image prune -af` is NOT the right hammer** — see next note. + +### `podman image prune -af` caveat — removes tagged-but-idle images + +`podman image prune -a -f` will delete tagged images that are **not referenced by a running container**. That means the image you just built via `charly box build` — and haven't started as a container yet — is eligible for prune. Observed sequence: `charly box build` → `podman image prune -af` → `sudo podman load` (manual rootful refresh) fails with `image not known`. Protect the fresh build by starting a container or use `prune` with explicit filters instead. + +### `--no-cache` does not invalidate intermediate scratch-stage caches + +`charly box build --no-cache ` and `charly box build --cache none ` reliably disable the +cache for the **final image stage**, but in observed behavior they do **not** propagate +to intermediate scratch stages produced by `COPY candy// /` instructions +(`[15/25] STEP 2/2: COPY candy/labwc/ /` style). Editing a single file inside a layer +directory and rebuilding with `--no-cache` may still pull the labwc scratch stage from +cache, leaving the new file content out of the rebuilt image. + +**Workaround:** force a content-hash bump on the layer's `charly.yml`. The simplest is +adding (or removing) a trailing comment line: -### Engine-level `image prune -a -f` caveat — removes tagged-but-idle images +```bash +echo "" >> candy/labwc/charly.yml # bump content hash +charly box build selkies-desktop # now invalidates the labwc scratch stage +git checkout -- candy/labwc/charly.yml # revert the cosmetic change +``` + +This was discovered while shipping commit `febb9bd` (labwc autostart race fix): the +edit to `candy/labwc/autostart` did not propagate to the rebuilt image until +`candy/labwc/charly.yml` itself was touched. Two consecutive `--no-cache` rebuilds +produced the same image hash (`502c8012c7a5`) until the charly.yml content changed. + +Symptom: `charly box list tags ` shows a new tag, but `charly shell -c "cat /path/to/changed/file"` +returns the **old** content. -An engine-level `image prune -a -f` deletes tagged images that are **not referenced by a running container**. That means the image you just built via `charly box build` — and haven't started as a container yet — is eligible for prune. Observed sequence: `charly box build` → an engine-level `image prune -af` → a manual rootful reload fails with `image not known`. Protect the fresh build by starting a container or use `prune` with explicit filters instead. +### Known caveat: stale `:latest` under `--cache-from` -### Forcing a rebuild of one layer's scratch stage +charly's Containerfile generator currently emits `FROM /:latest` +for parent builder images (not pinned CalVer). When charly invokes +`podman build --cache-from /`, podman resolves those `FROM` +clauses at parse time and will **pull `:latest` from the remote registry**, +silently clobbering any locally-rebuilt `:latest` tag. For images that rebuild +their builder stages in the same invocation this can lead to the later +stages using the **stale registry builder** instead of the freshly-built one. -A layer's scratch stage is keyed by the layer directory's own content — the cache key is -`(parent-image-SHA, instruction-text, COPY-source-content)` (see "Cache Efficiency" above) — so editing any -file under `candy//` rebuilds that stage and everything downstream of it. To force that rebuild without -changing the file that matters, bump the layer's `charly.yml`: +**Symptom observed on this project:** `charly update selkies-desktop --build` +fails deep inside selkies' `pixi install && bash build.sh` step with +`error: can't find Rust compiler`. The remote +`ghcr.io/opencharly/fedora-builder:latest` predates the `build-toolchain` +layer adding `cargo` as an RPM, so its pixi env has no rustc — even though +the current local `build-toolchain/charly.yml` lists `cargo`. The build phase +that rebuilds fedora-builder locally *does* run, but parent-stage FROM +resolution happens before that stage exists, and podman pulls the stale +remote image. + +**Workaround until the generator is fixed:** ```bash -echo "" >> candy/labwc/charly.yml # bump the layer's content hash -charly box build selkies-desktop # rebuilds the labwc scratch stage -git diff candy/labwc/charly.yml # the appended line, and nothing else +charly box build --cache=none # or equivalently --no-cache at the charly level ``` -Remove the added line to revert it. Do not reach for `git checkout -- ` to undo it: that restores the -file from the index and discards any other uncommitted work it holds. +Both `--cache=none` and `--no-cache` short-circuit `cacheArgs()` in +`candy/plugin-build/podman.go` (`driveConfig.cacheArgs`) and do NOT pass `--cache-from` to podman, so +the broken resolution path never fires. `--no-cache` is charly-level only — it +does *not* pass `--no-cache` to podman, it just skips `--cache-from`. + +**Proper fix (not yet implemented):** charly's generator should emit pinned +CalVer tags for builder `FROM` clauses (e.g., +`FROM ghcr.io/opencharly/fedora-builder:2026.105.0128`) or pass +`--pull=never` to podman so local tags aren't resolved from the remote. +Tracked as a follow-up. + +See also: [`/charly-build:generate`](/recipes/build/generate/) for the Containerfile generation path, [`/charly-core:charly-update`](/recipes/core/charly-update/) +for the `--build` flag that also picks up this caveat. ## Project directory override diff --git a/src/content/docs/recipes/build/docs.md b/src/content/docs/recipes/build/docs.md index 33f66bac..f98c3f19 100644 --- a/src/content/docs/recipes/build/docs.md +++ b/src/content/docs/recipes/build/docs.md @@ -13,35 +13,6 @@ description: "The `charly docs generate` verb — the generator rendering the re **opencharly.ai** from the sources that already exist in this repository. It is served by `candy/plugin-docs`. -## R4 — documentation never routes around a defect - -The rulebook states R4 as a mandate (umbrella `AGENTS.md`, "The ground-truth rules -R1–R10"); this is its operationalization for every page this skill renders and every -hand-authored page beside it. - -- **Fix the product first.** When documented behaviour and the code disagree, establish - which side holds the INTENT, then fix the code BEFORE touching the prose. Editing docs - to match a bug is forbidden; so is editing them to avoid one. -- **The tells of a routed-around defect:** a `git clone`, a `cd`, or a `-C` standing in - for a flag that should work; an "except on…" caveat; a "known limitation" note in place - of a fix. Each describes the defect instead of removing it — and makes it permanent by - making it look intended. -- **Binary-only reader.** Every command a reader is told to run MUST work with nothing but - the `charly` binary installed — no clone, no `cd`, no pre-existing project, no unnamed - companion tool. Write every page for a reader who installed `charly` as a native package - with NO charly checkout on the machine: no `task` target, no `./bin/charly`, no - repo-relative path, no verb supplied by a candy that exists only in this repository. - `--repo /` reads a published project without a clone; - `charly box new project ` starts a local one. -- **The one exception is the INSTALL page**, where the reader obtains charly: it leads - with the native package and scopes the checkout to working ON charly. A - repository-maintenance command is named as maintenance this project performs, never as - a step the reader runs. -- **Anywhere else, needing more than the binary is a PRODUCT DEFECT** — the missing - capability is the bug, and the fix belongs in `charly`. If that fix is genuinely out of - scope, file it with its root cause and let the page say nothing rather than something - false: silence is recoverable, a documented workaround is taught. - ## Placement: a RUNTIME plugin, deliberately `plugin-docs` is **not** listed in `charly/charly.yml`'s `compiled_plugins:`. A dev-time @@ -82,8 +53,8 @@ charly docs generate \ ## What it emits -The generator owns seven top-level entries — `index.md`, `grievances.md`, `vision.md`, -`liberation.md`, `soul.md`, `reference/` and `recipes/` — and rewrites them wholesale each run, so a deleted +The generator owns six top-level entries — `index.md`, `grievances.md`, `vision.md`, +`liberation.md`, `reference/` and `recipes/` — and rewrites them wholesale each run, so a deleted source entity disappears from the site instead of lingering as an orphan page. (The table below expands `reference/` into its sub-trees plus the single file `reference/providers.md`, so it runs to many rows.) What stays hand-authored is the teaching narrative this repository has no @@ -108,7 +79,6 @@ not the page.** | `grievances.md` | `GRIEVANCES.md`, H1 dropped, repo-relative links rewritten for a web reader | | `vision.md` | `VISION.md`, H1 dropped, repo-relative links rewritten for a web reader | | `liberation.md` | `LIBERATION.md`, H1 dropped, repo-relative links rewritten — the deliberate easter-egg page, entered last in the sidebar as "Liberation" | -| `soul.md` | `SOUL.md`, H1 dropped, published verbatim — the identity of charly and every charly agent; self-contained, so there are no repo-relative links to rewrite | | `reference/cli/` | one page per `command:` provider word | | `reference/candy/` | every defined candy: packages, services, and its `plan:` as an acceptance spec | | `reference/box/` | every defined box | @@ -205,7 +175,7 @@ the pinned charly) belongs to the same cutover as the candy edit. `candy/*/charly.yml` — but the generator DOES project candy `skill:` entities directly from its own remote-aware walk (`collectCandySkills`), the complement that keeps a moved candy's recipe references resolving on the site even before the marketplace regeneration -lands (the stale-marketplace gap [`/charly-tools:crabbox-cli`](/recipes/tools/crabbox-cli/) measures — see the tests +lands (the stale-marketplace gap [`/charly-tools:crabbox`](/recipes/tools/crabbox/) measures — see the tests section). The long path still holds: a skill is authored as a `skill:` entity in its owning candy; `charly marketplace generate` projects that into the opencharly/marketplace repo (`/skills//SKILL.md`), and @@ -237,7 +207,7 @@ stylistic preference; the obvious surfaces are actively wrong for the purpose: - A plugin absent from `compiled_plugins:` still loads out-of-process when a plan references its word, so "in the binary" is not the same set as "exists". -So discovery is not the walked trees alone: the catalog is ONE assembly (R2/R5, +So discovery is not the walked trees alone: the catalog is ONE assembly (R3/R5, `assembleCatalog`) unioning three sets — 1. the **walked closure** — remote-ref-aware (`candywalk.CollectEntitiesRemote`): @@ -325,7 +295,7 @@ describe the bad shape rather than spelling it. The host renders every dynamic command word with a generic stub description, intercepts the depth-1 `--help` itself, and plugin-served help arrives in at least three mutually incompatible -formats with no machine-readable dump. Parsing that would be the fragile shim R3 forbids. Every +formats with no machine-readable dump. Parsing that would be the fragile shim R4 forbids. Every fact the site needs is already declared: `plugin.providers`, the per-plugin `schema/*.cue`, and each candy's `description:`. diff --git a/src/content/docs/recipes/build/generate.md b/src/content/docs/recipes/build/generate.md index f3c48707..40e72fdc 100644 --- a/src/content/docs/recipes/build/generate.md +++ b/src/content/docs/recipes/build/generate.md @@ -17,13 +17,13 @@ Parses `charly.yml`, scans `candy/`, resolves the dependency graph, and emits al The generator is **config-driven** — distro format templates, builder stage templates, and init system fragments all come from the build vocabulary embedded in the `charly` binary at `charly/charly.yml` (three top-level sections: `distro:`, `builder:`, `init:`) — and **declarative per-task** for install logic — each task verb (see [`/charly-image:layer`](/recipes/image/layer/)) has a dedicated emitter that writes the right Containerfile directive. -**Build dispatch**: `charly box generate` does not construct a generator host-side at all (the former `NewGenerator` is DELETED) — it is served by the COMPILED-IN `command:box` plugin (candy/plugin-box, nested under the `box` grammar group), whose generate handler invokes the peer `build:generate` plugin (candy/plugin-build) over the F10 reverse channel (InvokeProvider). The plugin's `runBoxGenerate` → `resolveBuildEngine(..., generateOnly=true)` (candy/plugin-build/resolve.go) runs the ENTIRE build RESOLVE plugin-side — loader, vocab, scan, connect, validate, box resolve, intermediates, order, effective versions, host-fs prep, render-seam-cache prep, render-prep, user-context, and envelope projection — and returns the resolved-project envelope (`spec.BuildResolveReply.ResolvedProject`); `generateOnly` is a PLUGIN-SIDE parameter that returns after the envelope projection (no engine/order drive-model). The host is reached ONLY for thin `buildengine-*` shards (`buildengine-scan-local`, `buildengine-connect-plugins`, and `buildengine-prep` — the render-seam-floor `renderGenCache` populate, which returns an EMPTY map; the host-fs PREP itself moved plugin-side); the envelope + drive-model come from `resolveBuildEngine`'s reply, NOT from `HostBuild("buildengine-prep")` (that host leg takes `spec.ResolvedProjectRequest`, which has no `GenerateOnly` field). The candy renders the Containerfiles itself via `sdk/deploykit.Generator` over the envelope + `HostBuild("render-seam")` (#67 — the host no longer renders); the written paths are returned — no podman (generate builds nothing). `charly box build` uses the SAME resolve pipeline (`runBoxBuild` → `resolveBuildEngine(..., generateOnly=false)`, which additionally computes the drive-model), then the candy additionally OWNS the BuildKit DRIVE (the `buildctl` argv / push / inline-merge) IN the candy. See [`/charly-internals:plugin`](/recipes/internals/plugin/) (the `build` provider class + the in-proc reverse channel) and [`/charly-build:build`](/recipes/build/build/). +**Build-engine dispatch**: `charly box generate` does not construct a generator host-side at all (the former `NewGenerator` is DELETED) — it is served by the COMPILED-IN `command:box` plugin (candy/plugin-box, nested under the `box` grammar group), whose generate handler invokes the peer `build:generate` plugin (candy/plugin-build) over the F10 reverse channel (InvokeProvider). The plugin's `runBoxGenerate` → `resolveBuildEngine(..., generateOnly=true)` (candy/plugin-build/resolve.go) runs the ENTIRE build-engine RESOLVE plugin-side — loader, vocab, scan, connect, validate, box resolve, intermediates, order, effective versions, host-fs prep, render-seam-cache prep, render-prep, user-context, and envelope projection — and returns the resolved-project envelope (`spec.BuildResolveReply.ResolvedProject`); `generateOnly` is a PLUGIN-SIDE parameter that returns after the envelope projection (no engine/order drive-model). The host is reached ONLY for thin `buildengine-*` shards (`buildengine-scan-local`, `buildengine-connect-plugins`, and `buildengine-prep` — the render-seam-floor `renderGenCache` populate, which returns an EMPTY map; the host-fs PREP itself moved plugin-side); the envelope + drive-model come from `resolveBuildEngine`'s reply, NOT from `HostBuild("buildengine-prep")` (that host leg takes `spec.ResolvedProjectRequest`, which has no `GenerateOnly` field). The candy renders the Containerfiles itself via `sdk/deploykit.Generator` over the envelope + `HostBuild("render-seam")` (#67 — the host no longer renders); the written paths are returned — no podman (generate builds nothing). `charly box build` uses the SAME resolve pipeline (`runBoxBuild` → `resolveBuildEngine(..., generateOnly=false)`, which additionally computes the drive-model), then the candy additionally OWNS the podman DRIVE (`podman build` / push / inline-merge) IN the candy. See [`/charly-internals:plugin`](/recipes/internals/plugin/) (the `build` provider class + the in-proc reverse channel) and [`/charly-build:build`](/recipes/build/build/). -**Build-mode emission is `WriteCandySteps` → `EmitTasks`** in `sdk/deploykit` (`deploykit.Generator`, relocated from `charly/generate.go` in #67, driven by `candy/plugin-build` over the resolved-project envelope + `HostBuild("render-seam")`; the `charly/generate.go` top-level `Generate`/`writeCandySteps` orchestrator is DELETED). the pod-overlay deploy path reaches `deploykit.Generator.EmitTasks` DIRECTLY — `charly/tasks.go` holds no shim and cannot hold one (P16 import purity forbids `charly/` importing the sdk); that path stays for the pod-overlay deploy path (the candy `plugin-deploy-pod` constructs `deploykit.OCITarget` + renders via `deploykit.NewRenderGeneratorFromProject`, the per-step fragment over the `HostBuild("step-emit","oci-emit-step")` seam). The render walks each layer's ops directly to write Containerfile text — NOT the `InstallPlan` IR. Build shares the package-cascade, shell-snippet, and localpkg compiler helpers with the IR (`ResolveCascadePackages`, `CompileShellSnippetSteps`, `deploykit.RenderLocalPkgImageInstall` — one source of truth, R2, relocated to sdk/deploykit in W3), but the overall walk is the direct generator, not an `EmitTarget`. The `InstallPlan` IR + `deploykit.OCITarget.Emit` is the DEPLOY-mode path: the candy `plugin-deploy-pod`'s overlay-Containerfile synthesis (rendered in its own code via `deploykit.OCITarget` + `deploykit.NewRenderGeneratorFromProject`) when `add_candy:` is set, plus the external execution targets. (The local/vm/kubernetes/android substrates are external out-of-process plugins via `pluginDeployTarget` (S3b, dispatched through `candy/plugin-fleet`'s `Invoke(OpDeployDispatch)`): `deploy:local` (candy/plugin-deploy-local) and `deploy:vm` (candy/plugin-deploy-vm) DO consume the IR — the plugin walks it via `kit.WalkPlans` over the reverse channel, the vm one over the guest `SSHExecutor` so the walk runs inside the guest — while `deploy:kubernetes` (candy/plugin-kube) does NOT, materializing its Kustomize tree PLUGIN-side (`candy/plugin-kube/materialize.go`'s `materializeKustomize`, reaching `candy/plugin-k8sgen` peer-to-peer over `verb:k8sgen`/`OpEmit`).) See [`/charly-internals:install-plan`](/recipes/internals/install-plan/) for the IR catalog and [`/charly-internals:generate-source`](/recipes/internals/generate-source/) for the Go-level call graph. +**Build-mode emission is `WriteCandySteps` → `EmitTasks`** in `sdk/deploykit` (`deploykit.Generator`, relocated from `charly/generate.go` in #67, driven by `candy/plugin-build` over the resolved-project envelope + `HostBuild("render-seam")`; the `charly/generate.go` top-level `Generate`/`writeCandySteps` orchestrator is DELETED). the pod-overlay deploy path reaches `deploykit.Generator.EmitTasks` DIRECTLY — `charly/tasks.go` holds no shim and cannot hold one (P16 import purity forbids `charly/` importing the sdk); that path stays for the pod-overlay deploy path (the candy `plugin-deploy-pod` constructs `deploykit.OCITarget` + renders via `deploykit.NewRenderGeneratorFromProject`, the per-step fragment over the `HostBuild("step-emit","oci-emit-step")` seam). The render walks each layer's ops directly to write Containerfile text — NOT the `InstallPlan` IR. Build shares the package-cascade, shell-snippet, and localpkg compiler helpers with the IR (`ResolveCascadePackages`, `CompileShellSnippetSteps`, `deploykit.RenderLocalPkgImageInstall` — one source of truth, R3, relocated to sdk/deploykit in W3), but the overall walk is the direct generator, not an `EmitTarget`. The `InstallPlan` IR + `deploykit.OCITarget.Emit` is the DEPLOY-mode path: the candy `plugin-deploy-pod`'s overlay-Containerfile synthesis (rendered in its own code via `deploykit.OCITarget` + `deploykit.NewRenderGeneratorFromProject`) when `add_candy:` is set, plus the external execution targets. (The local/vm/kubernetes/android substrates are external out-of-process plugins via `pluginDeployTarget` (S3b, dispatched through `candy/plugin-fleet`'s `Invoke(OpDeployDispatch)`): `deploy:local` (candy/plugin-deploy-local) and `deploy:vm` (candy/plugin-deploy-vm) DO consume the IR — the plugin walks it via `kit.WalkPlans` over the reverse channel, the vm one over the guest `SSHExecutor` so the walk runs inside the guest — while `deploy:kubernetes` (candy/plugin-kube) does NOT, materializing its Kustomize tree PLUGIN-side (`candy/plugin-kube/materialize.go`'s `materializeKustomize`, reaching `candy/plugin-k8sgen` peer-to-peer over `verb:k8sgen`/`OpEmit`).) See [`/charly-internals:install-plan`](/recipes/internals/install-plan/) for the IR catalog and [`/charly-internals:generate-source`](/recipes/internals/generate-source/) for the Go-level call graph. **Build-time plugin execution**: the build path connects the project's external (out-of-tree) plugin candies via `loadProjectPlugins` (`charly/host_build_buildengine.go:51`, and `charly/check_cmd.go:70` for the check path), so a `run:` plugin verb — and a plugin builder — is registered + dialable DURING image generation, the SAME loader the deploy/check paths use. The host-side `NewGenerator` that used to perform this connect is DELETED. A builtin is already registered via `init()` and needs no connect; only an external one is host-built + connected here. `EmitTasks` then dispatches a `plugin:` verb step PLACEMENT-AGNOSTICALLY above the registry: a builtin `ProvisionActor` renders an act shell `RUN` in-proc (the zero-JSON fast path), while ANY other resolved provider — an external `grpcProvider`, or a builtin emitting a richer fragment — renders via `invokeOpEmitFragmentOpt` → `Invoke(OpEmit)` (in-proc for a builtin, go-plugin gRPC for an external), and the returned `spec.EmitReply.Fragment` is spliced verbatim into the Containerfile (egress-validated with the rest before write). This is operator-authorized build-time execution of host-built plugin code — a project's composed external plugins run as host code during its image builds. Detail → [`/charly-internals:plugin`](/recipes/internals/plugin/) (placement) + [`/charly-internals:generate-source`](/recipes/internals/generate-source/). -**Build-time BUILDER leg (all via `OpResolve`)**: the builder stage emission is `deploykit.Generator.EmitBuilderStages` / `EmitBuilderArtifacts` / `EmitExternalBuilderStages` / `EmitExternalBuilderArtifacts` (`sdk/deploykit/builders_render.go` — relocated from `charly/generate.go` in #67, driven by `candy/plugin-build` over the envelope + `HostBuild("render-seam")`); the shared builder `OpResolve` helper (`renderSeamCaller.resolveBuilderStage`, `sdk/deploykit/render_generator_from_project.go`) drives both legs (the former `charly/generate.go` is DELETED, K-wave 2). These render the four DETECTION-builders' stages (pixi/npm/aur — selected by a candy's `pixi.toml` / `package.json` / `aur:` section; cargo is INLINE) by `Invoke(OpResolve)`ing their plugins — NOT from an in-core `builders.` vocabulary (C10 moved the stage templates into the plugins' `kit.BuilderResolve`). For each detected `(candy, builder)`, `EmitBuilderStages` connects the plugin on-demand (`ensureBuildersConnected`) and calls the shared `resolveBuilderStage` → `prov.Invoke(OpResolve)` with a `spec.BuilderResolveInput` (the host-computed render context — builder ref, stage name, copy src, uid/gid/home, detected manifest/lockfile/build-script, aur packages/options, PRE-RENDERED cache-mount flags) as `op.Params` + a `spec.BuildEnv` as `op.Env`; the returned `spec.BuilderResolveReply.Stage` is written verbatim PRE-main-FROM (cached per `(candy, builder)`), `EmitBuilderArtifacts` writes the cached `CopyArtifacts` + once-per-builder `CopyBinary` POST-main-FROM, and cargo's `InlineFragment` splices in `WriteCandySteps` (deploykit). A candy that instead selects an OUT-OF-TREE builder via its `external_builder: ` field gets its multi-stage emitted by `EmitExternalBuilderStages` / `EmitExternalBuilderArtifacts` (run right after their detection counterparts), sharing the SAME `resolveBuilderStage` but sending a MINIMAL input (candy name only): the word resolves through `providerRegistry.ResolveBuilder` to an EXTERNAL `*grpcProvider`, and `EmitExternalBuilderStages` (the BUILDER-leg analogue of `invokeOpEmitFragmentOpt`) requires a non-empty `Stage`. An empty `Stage` (or empty cargo `InlineFragment`), an unresolvable word, or an Invoke error fails LOUDLY (R3) — never a silently-dropped builder stage. Every half is egress-validated with the rest of the Containerfile before write. Detail → [`/charly-internals:plugin`](/recipes/internals/plugin/) (placement) + [`/charly-internals:generate-source`](/recipes/internals/generate-source/). +**Build-time BUILDER leg (all via `OpResolve`)**: the builder stage emission is `deploykit.Generator.EmitBuilderStages` / `EmitBuilderArtifacts` / `EmitExternalBuilderStages` / `EmitExternalBuilderArtifacts` (`sdk/deploykit/builders_render.go` — relocated from `charly/generate.go` in #67, driven by `candy/plugin-build` over the envelope + `HostBuild("render-seam")`); the shared builder `OpResolve` helper (`renderSeamCaller.resolveBuilderStage`, `sdk/deploykit/render_generator_from_project.go`) drives both legs (the former `charly/generate.go` is DELETED, K-wave 2). These render the four DETECTION-builders' stages (pixi/npm/aur — selected by a candy's `pixi.toml` / `package.json` / `aur:` section; cargo is INLINE) by `Invoke(OpResolve)`ing their plugins — NOT from an in-core `builders.` vocabulary (C10 moved the stage templates into the plugins' `kit.BuilderResolve`). For each detected `(candy, builder)`, `EmitBuilderStages` connects the plugin on-demand (`ensureBuildersConnected`) and calls the shared `resolveBuilderStage` → `prov.Invoke(OpResolve)` with a `spec.BuilderResolveInput` (the host-computed render context — builder ref, stage name, copy src, uid/gid/home, detected manifest/lockfile/build-script, aur packages/options, PRE-RENDERED cache-mount flags) as `op.Params` + a `spec.BuildEnv` as `op.Env`; the returned `spec.BuilderResolveReply.Stage` is written verbatim PRE-main-FROM (cached per `(candy, builder)`), `EmitBuilderArtifacts` writes the cached `CopyArtifacts` + once-per-builder `CopyBinary` POST-main-FROM, and cargo's `InlineFragment` splices in `WriteCandySteps` (deploykit). A candy that instead selects an OUT-OF-TREE builder via its `external_builder: ` field gets its multi-stage emitted by `EmitExternalBuilderStages` / `EmitExternalBuilderArtifacts` (run right after their detection counterparts), sharing the SAME `resolveBuilderStage` but sending a MINIMAL input (candy name only): the word resolves through `providerRegistry.ResolveBuilder` to an EXTERNAL `*grpcProvider`, and `EmitExternalBuilderStages` (the BUILDER-leg analogue of `invokeOpEmitFragmentOpt`) requires a non-empty `Stage`. An empty `Stage` (or empty cargo `InlineFragment`), an unresolvable word, or an Invoke error fails LOUDLY (R4) — never a silently-dropped builder stage. Every half is egress-validated with the rest of the Containerfile before write. Detail → [`/charly-internals:plugin`](/recipes/internals/plugin/) (placement) + [`/charly-internals:generate-source`](/recipes/internals/generate-source/). **Three-phase templates**: the embedded build vocabulary's format (`formats.`) and builder (`builders.`) definitions carry a `phases: { prepare, install, cleanup }.{ container, host }` structure. The generator reads `phases.install.container` as the single source for build-mode package installation. The `host:` cell is consumed only when a `SystemPackagesStep` renders on a host venue (`RenderHostPackageCommand`) — for the external `local:` AND `vm:` deploys the plugin drives the `SystemPackages` host render via the host's `RunHostStep` channel — never by the generator. diff --git a/src/content/docs/recipes/build/migrate.md b/src/content/docs/recipes/build/migrate.md index 0967cb33..f1fb6e9e 100644 --- a/src/content/docs/recipes/build/migrate.md +++ b/src/content/docs/recipes/build/migrate.md @@ -1,119 +1,124 @@ --- title: "migrate" -description: "The `charly migrate` command — the single idempotent reshape pass bringing any opencharly config forward to the current schema." +description: "The `charly migrate` command — the single idempotent migration bringing any opencharly config to the latest schema CalVer." --- *Recipe card from the `charly-build` plugin (Commands — runtime CLI verbs).* -# charly migrate — the version-free reshape pass +# charly migrate — single-command schema migration -`charly migrate` is **one idempotent command** that applies the declarative migration table to an opencharly config on disk. There are no sub-verbs to choose between: it walks the whole table in declaration order, and because every step is idempotent a re-run on an already-current project is a no-op. +`charly migrate` is **one idempotent command** that brings any opencharly config — as far back as the supported schema floor — up to the **latest schema CalVer**, and only to the latest. There are no sub-verbs to choose between: `charly migrate` always applies every migration-table step newer than the config, up to HEAD. ```bash -charly migrate # apply the table to every reachable config -charly migrate --dry-run # print every change migrate would make; touch nothing -charly migrate --project-only # skip the per-host overlay +charly migrate # migrate every reachable config to the latest schema CalVer +charly migrate --dry-run # print every change migrate would make; touch nothing ``` The project directory is the current working directory; use the top-level `-C` / `--dir` / `CHARLY_PROJECT_DIR` global to point elsewhere (`main()` chdir's before dispatch). -## There is no schema version +## CalVer schema versioning -A config carries **no `version:` field**, and the schema is CLOSED: a `charly.yml` still carrying a retired key — `version:` above all — is an unknown field and **fails closed-CUE when it loads**. That load failure is the trigger to run `charly migrate`, and `charly migrate` is what removes the retired keys. +The YAML schema version is a **CalVer string** — `version: YYYY.DDD.HHMM`, the same fixed-width scheme as image tags (e.g. a HEAD like `version: 2026.174.1100`). It is CANONICAL fixed-width: a 4-digit year, a 3-digit zero-padded day-of-year, and a 4-digit zero-padded HHMM, so a plain alphanumeric sort of CalVer strings is chronological. `ParseCalVer` is EXTREMELY STRICT — it accepts ONLY that exact form (no `version: 4`, no non-padded `2026.45.830`); a non-canonical value is "not a CalVer", which the load gate treats as older-than-HEAD so it flows into `charly migrate`. `charly migrate` re-stamps every versioned file it brings forward to the canonical HEAD (`universalStamp`); a value that predates the supported floor — or is not a CalVer at all — is unmigratable and refused untouched (see "How it runs"). Every versioned file carries the stamp: -There is no schema HEAD, no floor, and no version comparison anywhere in the engine. `charly migrate` does not decide *whether* a project needs migrating — it reshapes every project it is pointed at, and idempotency is what makes that safe. A universal final step, **`strip-version-stamp`**, removes a leftover top-level or per-entity `version:` key: from the root `charly.yml`, from every other project document, and — unless `--project-only` — from the per-host overlay. +- every project `charly.yml` (splitting into per-kind `vm.yml` / `local.yml` / … siblings is an optional `import:` convenience a project MAY use — never the default; see [`/charly-image:image`](/recipes/image/image/)) +- the per-host `~/.config/charly/charly.yml` -### Per-merge release git tags +The parsed `ParsedCalVer` type, `ParseCalVer(string) (ParsedCalVer, bool)`, `ParsedCalVer.Less`, and `MustCalVer` live in **`spec/spec/calver_parse.go`** (the parsed type is named `ParsedCalVer` because `spec` already binds `CalVer = string`, the CUE wire scalar); `sdk/kit/calver.go` re-exports them for kit consumers. **CUE owns the HEAD and floor:** `spec/schema/version.cue` defines `#SchemaVersion` (the HEAD CalVer every file is stamped to) and `#SchemaFloor` (the oldest version migratable FROM). The spec repo's `charly task cue-gen` emits `spec/spec/version_gen.go` with the `SchemaVersion` / `SchemaFloor` string consts; `spec.LatestSchemaCalVer()` and `spec.SchemaFloorCalVer()` parse them, re-exported as `kit.LatestSchemaVersion()` / `kit.SchemaFloor()` (what `candy/plugin-migrate/engine.go` calls) and shimmed in core by `charly/version.go`'s `LatestSchemaVersion()`. There is NO hand-maintained Go HEAD literal — the load-time gate requires `kit.LatestSchemaVersion()`, and `charly migrate` refuses anything below `kit.SchemaFloor()`. -Every landing of ANY repo carries a **fresh** annotated release git tag `v` — ONE per merge, so a repo accumulates MULTIPLE `v…` tags over time. The tag is **generated and applied by the fresh `pr-validator` at MERGE**, from the current UTC merge time — never by the author at push time (author-time stamps collide and mis-order across concurrent out-of-sync PRs; see [`/charly-internals:git-workflow`](/recipes/internals/git-workflow/) "CalVer"). Tag EVERY landing — a content change, a submodule extraction, an image drop. Tags are **immutable**: only ever ADD new ones, never move or force-push an existing tag. Every component is fixed-width zero-padded — 4-digit year, 3-digit day-of-year, 4-digit HHMM (`v2026.064.1937`, `v2026.142.1640`) — so tags sort chronologically under a plain alphanumeric sort; the evaluator computes `v$(date -u +%Y.%j.%H%M)` at merge (`%j` and `%H%M` are already zero-padded). Each repo — the superproject, every `box/` submodule, `plugins`, and `docs` — is tagged independently on its merged `main` HEAD (the tag marks the merge, so a repo needs no `charly.yml` to be tagged); a producer repo's PR merges + tags BEFORE its consumer's pointer bump. The SOLE exception is `sdk`, which tags under its Go-module `v0..` scheme (a semver requirement, not an exemption). The tag is independent of the migration table: the tag marks the MERGE, a migration entry marks the SCHEMA. See the project rulebook "Commit, push, land" (`AGENTS.md`) and [`/charly-internals:git-workflow`](/recipes/internals/git-workflow/). +### Per-merge release git tags (decoupled from `version:`) + +Every landing of ANY repo carries a **fresh** annotated release git tag `v` — ONE per merge, so a repo accumulates MULTIPLE `v…` tags over time. The tag is **generated and applied by the fresh `pr-validator` at MERGE**, from the current UTC merge time — never by the author at push time (author-time stamps collide and mis-order across concurrent out-of-sync PRs; see [`/charly-internals:git-workflow`](/recipes/internals/git-workflow/) "CalVer"). This is **decoupled** from the `charly.yml` `version:` field: `version:` is the SCHEMA version (bumped only when a cutover raises `#SchemaVersion`), whereas the tag marks the merge moment. Tag EVERY landing — including one at an unchanged `version:` (a content change, submodule extraction, image drop). Tags are **immutable**: only ever ADD new ones, never move or force-push an existing tag (so the load-time gate never sees a "newer than supported" CalVer from a re-tag). Every component is fixed-width zero-padded — 4-digit year, 3-digit day-of-year, 4-digit HHMM (`v2026.064.1937`, `v2026.142.1640`) — so tags sort chronologically under a plain alphanumeric sort; the evaluator computes `v$(date -u +%Y.%j.%H%M)` at merge (`%j` and `%H%M` are already zero-padded). Each repo — the superproject, every `box/` submodule, `plugins`, and `docs` — is tagged independently on its merged `main` HEAD (the tag marks the merge, so a repo needs no `charly.yml` to be tagged); a producer repo's PR merges + tags BEFORE its consumer's pointer bump. The SOLE exception is `sdk`, which tags under its Go-module `v0..` scheme (a semver requirement, not an exemption). See the project rulebook "Post-Execution Policies" (`AGENTS.md`) and [`/charly-internals:git-workflow`](/recipes/internals/git-workflow/). ## The migration table (declarative data) -The migration steps are **declarative DATA, not code**: an ordered list embedded from **`candy/plugin-migrate/migrations.cue`**, validated at process start against the `#Migration` schema in **`candy/plugin-migrate/schema/migration.cue`**. That schema is `@go(-)` — it generates no Go type — and lives IN the plugin: it is validation-only, read by this engine and by no charly core code, so per the kernel/plugin boundary law it stays out of the SDK contract. The list is ORDERED — the order the cutovers were authored in, which is the only correct replay order — and `charly migrate` applies every step, in order. +The migration steps are **declarative DATA, not code**: an ordered list embedded from **`candy/plugin-migrate/migrations.cue`** (the TABLE lives in candy/plugin-migrate), validated at process start against the `#Migration` schema (`candy/plugin-migrate/schema/migration.cue` — a plugin-only validation schema that lives in the plugin per the kernel/plugin boundary law, concatenated at compile with the SDK's `version.cue` for `#CanonCalVer`). Each step is stamped with the CalVer of the date it landed and listed **chronologically** — the order the cutovers were authored in, which is the only correct replay order for an arbitrarily-old config. `charly migrate` applies every step whose version is newer than the config's stamp; each step is idempotent, so applying the whole set is safe (an already-current file is a no-op). -A step is: +A step is a small record: ```cue -{name: string, touches_host?: bool, ops?: [...#MigrationOp] | apply?: string} +{version: "YYYY.DDD.HHMM", name: "", touches_host?: bool, ops?: [...#Op] | apply?: ""} ``` -`name` labels the step for `--dry-run` output and progress. `touches_host: true` flags a step that mutates per-host state (`~/.config/charly`, quadlets, `.secrets`) — those steps are skipped by remote-cache auto-migration (see below). A step carries EITHER an `ops:` list OR a single `apply:` hook, never both; that "exactly one of the two" rule is enforced in Go, since CUE's field-presence comparison is too fragile for the gate. +`touches_host: true` flags a step that mutates per-host state (`~/.config/charly`, quadlets, `.secrets`) — those steps are skipped by remote-cache auto-migration (see below). A step carries EITHER an `ops:` list OR a single `apply:` hook, never both. **The op vocabulary — each op is DATA, zero Go.** Four generic key transforms cover the common cutovers, all applied by ONE comment-preserving yaml.v3 interpreter (the op-walker in `candy/plugin-migrate/engine.go`): | Op | Shape | Effect | |---|---|---| -| `rename_key` | `{from, to, scope?="any", under_kind?}` | rename a mapping key `from` → `to` | -| `delete_key` | `{key, scope?="any", under_kind?}` | drop a mapping key | +| `rename_key` | `{from, to, scope, under_kind?}` | rename a mapping key `from` → `to` | +| `delete_key` | `{key, scope, under_kind?}` | drop a mapping key | | `remap_scalar` | `{key, from, to, under_kind?}` | rewrite a scalar VALUE `from` → `to` at `key` | | `move_key` | `{key, from_parent, to_parent, under_kind?}` | relocate a key between parent mappings | -`scope` is `root` (top-level keys only) or `any` (every depth, the default; only `rename_key` / `delete_key` take it); `under_kind` further scopes an op to entities of a named kind (e.g. only `vm:` nodes). None of the four ops needs a line of new Go — the walker interprets the data. +`scope` is `root` (top-level keys only) or `any` (every depth); `under_kind` further scopes an op to entities of a named kind (e.g. only `vm:` nodes). None of the four ops needs a line of new Go — the walker interprets the data. -**The `apply:` hook — the deliberate, visible EXCEPTION.** A structural reshape the four ops can't express (splitting one node into siblings, folding a list into a map) sets `apply: ""` naming a Go function registered in the plugin's `goHooks` map. This is the ONLY path that runs bespoke Go for a migration, and it is intentionally conspicuous: a table entry carrying `apply:` is the signal that this step needed real code. +**The `apply:` hook — the deliberate, visible EXCEPTION.** A structural reshape the four ops can't express (splitting one node into siblings, folding a list into a map) sets `apply: ""` naming a Go function registered in the core `goHooks` map. This is the ONLY path that runs bespoke Go for a migration, and it is intentionally conspicuous: a table entry carrying `apply:` is the signal that this step needed real code. -**CUE has NO transformation capability.** CUE is monotone unification — it can validate, default, require, and close, and nothing more; it CANNOT rewrite a config. So CUE owns exactly ONE thing here: the SHAPE of the declarative table (`#Migration` validates every entry at startup). Every actual transform is Go — the generic op-walker, or an `apply:` hook. Do not expect CUE to migrate anything. +**CUE has NO transformation capability.** CUE is monotone unification — it can validate, default, require, and close, and nothing more; it CANNOT rewrite a config. So CUE owns exactly two things here: the version pins (`#SchemaVersion` / `#SchemaFloor`) and the SHAPE of the declarative table (`#Migration` validates every entry at startup). Every actual transform is Go — the generic op-walker, or an `apply:` hook. Do not expect CUE to migrate anything. -**The current table.** `candy/plugin-migrate/migrations.cue` is the authoritative list — read it there; each entry carries its own comment explaining the reshape and why the declarative ops could not express it. The entries run in the order the cutovers were authored in (the only correct replay order) and are: +**The current table.** The floor sits at the migration-baseline reset version (`2026.174.1100`), so any config predating it is unmigratable — the accepted clean-slate consequence. The table carries ONE entry, the schema-compaction cutover: -| name | declarative ops | `apply:` hook | `touches_host` | +| version | name | touches_host | apply | |---|---|---|---| -| `compact-node-form` | — | `compactNodeForm` | `true` | -| `strip-candy-libvirt-field` | — | `stripCandyLibvirtField` | — | -| `strip-deploy-shell-overlay` | — | `stripDeployShellOverlay` | `true` | -| `k8s-to-kubernetes` | 2 × `rename_key` | — | — | -| `remove-candy-localpkg` | `delete_key` | — | — | -| `install-template-to-phases` | — | `installTemplateToPhases` | — | -| `reshape-graphics-gl` | — | `reshapeGraphicsGL` | — | -| `record-field-to-instrument` | — | `recordFieldToInstrument` | — | -| `unroll-group-deploy` | — | `unrollGroupDeploy` | — | -| `deploy-cpu-spelling` | — | `reshapeDeployCPU` | `true` | -| `rekey-legacy-vm-overlay` | — | `rekeyLegacyVMOverlay` | `true` | - -`compact-node-form` is the schema-compaction cutover: it folds the former named data/step child-node grammar into the compact node form — collections (`env`/`volume`/`port`/`service`/…) inline in the kind value, steps as the ordered `plan:` list (a meaningful former step-node name becomes the step's `id:`), deploy data (`add_candy`/`install_opts`/`vm_state`/…) inline in the substrate node, and every authored step `plugin:`+`plugin_input:` pair rewritten to the `: ` verb sugar. Member children (sub-entities under a deployable kind) stay named children. Remote layer caches auto-migrate on fetch (see "Remote-cache auto-migration"). - -On top of the table the engine ALWAYS applies one further step, **`strip-version-stamp`** — it is not a table entry, so it needs no author (see "There is no schema version"). +| `2026.186.2323` | `compact-node-form` | `true` | `compactNodeForm` (goHook) | + +The `compactNodeForm` hook is a structural reshape the four generic ops can't express: it folds the former named data/step child-node grammar into the compact node form — collections (`env`/`volume`/`port`/`service`/…) inline in the kind value, steps as the ordered `plan:` list (a meaningful former step-node name becomes the step's `id:`), deploy data (`add_candy`/`install_opts`/`vm_state`/…) inline in the substrate node, and every authored step `plugin:`+`plugin_input:` pair rewritten to the `: ` verb sugar. Member children (sub-entities under a deployable kind) stay named children. Remote layer caches auto-migrate on fetch (see "Remote-cache auto-migration"). The same cutover DELETED the Calamares `target:` / `module:` / `package-group:` kinds (and their plugins) — they had zero core readers; they return as their own cutover when the installer feature is real. ## How it runs -The migration engine is a **COMPILED-IN command plugin** (`candy/plugin-migrate`, `command:migrate`) — the engine, table, and CLI live in the plugin, out of charly core; core keeps only the load-gate HINT (the `Run: charly migrate` message a closed-schema load failure carries). `charly migrate` dispatches to the compiled-in `command:migrate` provider; the remote-cache auto-migration resolves the same provider and Invokes it (`OpRun --project-only`) in-proc. Compiled-in placement is load-bearing: migrate is whole-project file-based and must run when the config is exactly what cannot load, so it can never be discovered out-of-process. +The migration engine is a **COMPILED-IN command plugin** (`candy/plugin-migrate`, `command:migrate`) — the engine, table, and CLI live in the plugin, out of charly core; core keeps only the below-floor/behind-head load-gate HINTS (the `Run: charly migrate` messages). `charly migrate` dispatches to the compiled-in `command:migrate` provider; the remote-cache auto-migration (`refs.go`) resolves the same provider and Invokes it (`OpRun --project-only`) in-proc. -`runMigrations` (`candy/plugin-migrate/engine.go`) walks the table in declaration order: +`runMigrations` (`candy/plugin-migrate/engine.go`) is **floor-gated**, comparing the config's `version:` stamp against `kit.SchemaFloor()` and `kit.LatestSchemaVersion()`: -- **no `charly.yml` in the directory** → reports `no in — nothing to migrate`. -- **otherwise** → applies every step (the generic op-walker or an `apply:` hook) in order, then the universal `strip-version-stamp`, and reports `nothing to migrate` when nothing changed. +- **at HEAD** → no-op; prints `nothing to migrate (already at schema )`. +- **below the floor, or not a CalVer at all** → **unmigratable**: refused with an actionable error (`predates the supported floor … re-author against the current schema`) and **NO filesystem change**. Configs from before the baseline reset fall here. A stranded **per-host** config (`~/.config/charly/charly.yml`) at a below-floor version is doubly stuck: it can neither migrate NOR be WRITTEN — `saveDeployState` refuses to overwrite a config it cannot load (`refusing to overwrite … the existing per-host config fails to load … fix it (or remove it to regenerate) first`), so every pod/vm/local deploy — including a `charly check run ` at its `config`/`deploy-add` step — fails until it is reset. Since it holds only regenerable deploy STATE (no authored intent), back it up and reset it to a bare `version: ` stub (or remove it); the state rebuilds on the next deploy. +- **in `[floor, HEAD)`** → apply every migration-table step newer than the stamp (in order, via the op-walker / `apply:` hooks), then re-stamp every versioned file to HEAD (`universalStamp`). -**Backups are gitignored, never adjacent.** A step's per-file backup goes to `/.charly/backups/.`, and the tree is kept out of git by a `/.charly/.gitignore` containing a bare `*` (created on demand). A migration therefore never leaves `*.bak` files beside the originals and never dirties a working tree. +Per-step backups follow the established `.bak.` convention. ### Remote-cache auto-migration (project-only) -`sdk/loaderkit`'s refs seams auto-run the remote-cache migration on a freshly-cloned remote-repo cache so external repos pull through at the current schema. It skips every `touches_host` step and leaves the host-deploy path empty, so a remote fetch **never mutates the user's per-host state** — even the final `strip-version-stamp` touches only the cache's project files. +`sdk/loaderkit`'s refs seams (the former `charly/refs.go` is DELETED, K-wave 2) auto-run the remote-cache migration (`migrateCacheViaPeer`, `sdk/loaderkit/refs_seams_executor.go`) on a freshly-cloned remote-repo cache so external repos pull through at the latest schema. It skips every `touches_host` step and leaves the host-deploy path empty, so a remote fetch **never mutates the user's per-host state** — even the final re-stamp touches only the cache's project files. A remote whose config predates the floor **fails the fetch** with the same predates-floor error (an old remote is unmigratable — the accepted clean-slate consequence). + +## Load-time gate + +The load-time gate `GateSchemaVersion` (`sdk/loaderkit/load_unified.go` — the former `charly/unified.go` is DELETED, K-wave 2) is unchanged: `LoadUnified` parses the merged `version:` and rejects anything below HEAD (or absent, or non-CalVer): + +``` +charly.yml: schema 2026.186.2323 is required (found "4"). Run: charly migrate +``` + +A non-CalVer value (a legacy integer, empty, or garbage) parses as "older than every real CalVer", so a stale config trips the gate with a uniform `Run: charly migrate` hint — the forward-compat trigger that makes a future migration fire. Whether that migrate then SUCCEEDS depends on the floor: a config in `[floor, HEAD)` migrates; one below the floor is refused (see "How it runs"). Residual-key checks (e.g. `kind: deployment`, `target: host`, `secret_backend: kdbx`) remain as defense-in-depth, but every remediation hint points uniformly at bare `charly migrate`. -## Load-time enforcement: closedness +## Adding a future cutover -There is no version gate to trip. Compatibility has ONE mechanism — the **CLOSED CUE schema**: a retired key (a leftover `version:`, `kind: deployment`, `target: host`, `secret_backend: kdbx`, …) is an unknown field, and the load fails with a hard `field not allowed` diagnostic naming it. The same failure carries the `Run: charly migrate` hint, and `charly migrate` is what strips the key (see "There is no schema version"). Residual-key checks remain as defense-in-depth alongside the closedness gate. +**From here to the end of "Standing rule" is maintenance the charly project performs on its OWN schema**, run from checkouts of the spec repo and the superproject. Nothing in these two sections is a step an charly user runs — the user-facing surface of a schema cutover is the single command `charly migrate`, unchanged. They are recorded here because this skill owns the migration table; the commands are named as this project's maintenance, not as instructions for the reader. -## Adding a future migration +The payoff of the declarative engine: the common case needs **zero new Go** — but since the schema lives in the spec module, a schema-version bump is **CROSS-REPO**: the spec repo lands + tags FIRST, then the superproject consumes the published module at the bumped require version (no submodule, no local checkout). -**This section is maintenance the charly project performs on its OWN schema.** Nothing in it is a step a charly user runs — the user-facing surface is the single command `charly migrate`, unchanged. It is recorded here because this skill owns the migration table. +1. Bump `#SchemaVersion` in **`spec/schema/version.cue`** to the new CalVer. +2. Run the spec repo's **`charly task cue-gen`** to regenerate `spec/spec/version_gen.go`. +3. **Land + tag the spec repo** (its Go-module tag scheme `v0..` — see [`/charly-internals:git-workflow`](/recipes/internals/git-workflow/)). +4. In the superproject: bump the `github.com/opencharly/spec` require version in `charly/go.mod` and every lockstep go.mod (the canonical-go.mod gate in `tools/gomod-canonical` asserts one shared pin; `charly task mods-tidy` re-syncs the go.sum files), append the matching entry to **`candy/plugin-migrate/migrations.cue`** (the TABLE lives in candy/plugin-migrate) with the same `version:` — **strictly greater** than the previous HEAD, expressing the transform as an `ops:` list (`rename_key` / `delete_key` / `remap_scalar` / `move_key`) — and run the superproject's own gates. -The declarative engine's payoff: the common case needs **zero new Go**. Append ONE entry to **`candy/plugin-migrate/migrations.cue`**, expressing the transform as an `ops:` list (`rename_key` / `delete_key` / `remap_scalar` / `move_key`). Because the table is order-driven and version-free, there is no CalVer to bump and no cross-repo sequence — the entry IS the whole change, and it is idempotent by contract. A **structural reshape the four ops can't express** additionally registers ONE `goHooks` entry (a Go function named by the step's `apply:` field). Ship the entry in the same change that alters the schema, and update the repo's own versioned YAML. +That is the whole common case — no bespoke migrator type, no registry function, no per-migrator Go file. Only a **structural reshape the four ops can't express** additionally registers ONE `goHooks` entry (a Go function named by the step's `apply:` field). Update the HEAD-CalVer fixtures + the repo's own versioned YAML in the same change. The operator command never changes — it stays `charly migrate`. -**The validated schema is CUE-single-source.** The `@go()`-annotated `spec/schema/*.cue` defs are the sole source for the Go param structs (generated into `spec/spec` by `charly task cue-gen`); a wire-key change is a CUE edit first, then `charly task cue-gen`, then — only if it breaks existing on-disk configs — a migration-table entry here. A pure codegen refactor that leaves every authored wire key untouched needs NO migration entry. See the [`/charly-internals:go`](/recipes/internals/go/) recipe "How to change the charly.yml schema (CUE is the single source of truth)". +**The validated schema is CUE-single-source.** The `@go()`-annotated `spec/schema/*.cue` defs are the sole source for the Go param structs (generated into `spec/spec` by `charly task cue-gen`); a wire-key change is a CUE edit first, then `charly task cue-gen`, then — only if it breaks existing on-disk configs — a migration-table entry here (plus the `#SchemaVersion` bump above). A pure codegen refactor that leaves every authored wire key untouched needs NO migration entry and NO `version:` bump. See the [`/charly-internals:go`](/recipes/internals/go/) recipe "How to change the charly.yml schema (CUE is the single source of truth)". -### Standing rule: a schema/format change ships its migration entry AND mints a git tag +### Standing rule: a schema/format change bumps `version:` AND mints a git tag -Any change to the YAML schema or composition format (a key rename, a deleted key, a new key shape) is a hard-cutover that MUST ship a matching migration-table entry in the same change, so `charly migrate` can carry an existing config across. A migration entry WITHOUT the matching schema change (or vice-versa) is forbidden. +Any change to the YAML schema or composition format (a key rename, a deleted key, a new key shape) is a hard-cutover that MUST: -Separately, **the fresh `pr-validator` mints the per-merge git tag** — `v` computed from the MERGE moment (see "Per-merge release git tags" above). The tag marks the MERGE and every landing gets one, schema change or not; it is independent of the migration table, which marks the SCHEMA. +1. **Bump the `#SchemaVersion` CalVer** — edit `spec/schema/version.cue`, run `charly task cue-gen`, and append the matching entry to `candy/plugin-migrate/migrations.cue` (the cross-repo sequence under "Adding a future cutover"), then land the sdk PR (the fresh `pr-validator` merges + tags it), and in the superproject adopt the tagged sdk release as the new shared pinned require (the `charly task mods-tidy` sweep + canonical-go.mod gate keep every module on the one pin). The load-time gate then rejects any not-yet-migrated config with a `Run: charly migrate` hint, so every reader sees the new format. Raising `#SchemaVersion` WITHOUT the migration-table entry (or vice-versa) is forbidden — and `version:` is NEVER set above `LatestSchemaVersion()` (newer configs hard-fail at load). **The author writes a PLACEHOLDER `#SchemaVersion` / `version:` / `migrations.cue` CalVer; the fresh `pr-validator` FINALIZES it at MERGE** — re-stamping it strictly ABOVE the current `main` HEAD's schema version (two concurrent schema PRs would otherwise collide or mis-order; see [`/charly-internals:git-workflow`](/recipes/internals/git-workflow/) "CalVer"). +2. **The fresh `pr-validator` mints the per-merge git tag** at merge — `v` from the merge moment (see "Per-merge release git tags" above). The tag and the `version:` bump are decoupled: the tag marks the merge, `version:` marks the schema. A schema cutover happens to do BOTH at once, but a content-only landing (no schema change) still mints a tag at an unchanged `version:`. ## Idempotency -Running `charly migrate` twice is a no-op: every step's old shape is absent the second time, so each does nothing. An op with no matching key, a `strip-version-stamp` on a file with no stamp, and an `apply:` hook on an already-reshaped node are all inert. The table's own invariant — every entry a valid `#Migration` — is validated against CUE at process start. +Running `charly migrate` twice is a no-op: after the first run the config is stamped at HEAD, so the second run hits the floor gate's "at HEAD → nothing to migrate" branch. Every op is itself idempotent (a rename with no matching key does nothing). The migration table's own invariants — every entry a valid `#Migration`, versions strictly ascending and unique, HEAD == `#SchemaVersion` — are validated against CUE at process start. ## See Also @@ -122,5 +127,5 @@ Running `charly migrate` twice is a no-op: every step's old shape is absent the - [`/charly-core:deploy`](/recipes/core/deploy/) — the deploy entries a migration may rewrite - [`/charly-local:local-spec`](/recipes/local/local-spec/) — `kind: local` templates - [`/charly-build:secrets`](/recipes/build/secrets/), [`/charly-build:settings`](/recipes/build/settings/) — the credential schema -- [`/charly-internals:go`](/recipes/internals/go/) — loader internals (`LoadUnified` and the closedness gate; the `candy/plugin-migrate/engine.go` op-walker) +- [`/charly-internals:go`](/recipes/internals/go/) — loader internals (`LoadUnified`, `ParseCalVer`, the `gateSchemaVersion` load gate, the `candy/plugin-migrate/engine.go` op-walker) - [`/charly-internals:cutover-policy`](/recipes/internals/cutover-policy/) — why hard-cutover + a single idempotent `charly migrate` is the required shape diff --git a/src/content/docs/recipes/build/pull.md b/src/content/docs/recipes/build/pull.md index f7bc0612..519ed24e 100644 --- a/src/content/docs/recipes/build/pull.md +++ b/src/content/docs/recipes/build/pull.md @@ -96,8 +96,8 @@ per-call-site code: from local storage. 2. `dispatchBuildEnsure` (`charly/dispatch_build_ensure.go`) dispatches to candy/plugin-build's `build:ensure` word (`candy/plugin-build/ensure.go`), - which returns the same sentinel when the image is absent from the RUN engine's - store (the one every later read resolves it through). + which returns the same sentinel when the image is absent from both run and + build engines. 3. `FormatCLIError` in `charly/image.go` unwraps the sentinel at the top-level error boundary in `main()` and renders: diff --git a/src/content/docs/recipes/build/settings.md b/src/content/docs/recipes/build/settings.md index 1dc07898..d36384f8 100644 --- a/src/content/docs/recipes/build/settings.md +++ b/src/content/docs/recipes/build/settings.md @@ -30,6 +30,7 @@ Manage charly's runtime configuration stored in `~/.config/charly/config.yml`. C | Key | Default | Env Var | Description | |-----|---------|---------|-------------| +| `engine.build` | `auto` | `CHARLY_BUILD_ENGINE` | Build engine (docker/podman) | | `engine.run` | `auto` | `CHARLY_RUN_ENGINE` | Run engine (docker/podman) | | `run_mode` | `auto` | `CHARLY_RUN_MODE` | Deployment mode (quadlet/direct) | | `bind_address` | `127.0.0.1` | `CHARLY_BIND_ADDRESS` | Default bind address for ports | @@ -46,11 +47,12 @@ Manage charly's runtime configuration stored in `~/.config/charly/config.yml`. C ### Engine Selection ```bash -# Which engine RUNS containers (the builder is BuildKit, and is not selectable) +# Switch to podman for both build and run +charly settings set engine.build podman charly settings set engine.run podman # Check current engine -charly settings get engine.run +charly settings get engine.build ``` ### Storage Paths diff --git a/src/content/docs/recipes/check/clawsweeper-review.md b/src/content/docs/recipes/check/clawsweeper-review.md new file mode 100644 index 00000000..11dd1409 --- /dev/null +++ b/src/content/docs/recipes/check/clawsweeper-review.md @@ -0,0 +1,76 @@ +--- +title: "clawsweeper-review" +description: "The ClawSweeper review engine as a disposable container: a pinned upstream checkout, Node 24 and pnpm 12.4.1, the OpenAI Codex CLI on the Ollama Cloud `responses` endpoint, the `opencharly` target-profile fallback, and a one-shot no-token `--local-range` review whose JSON verdict the caller posts." +--- + + + +*Recipe card from the `charly-check` plugin (Commands — runtime CLI verbs).* + +# clawsweeper-review — the review half of PR validation + +## The contract + +`clawsweeper-review` runs ONE review of the tree mounted at `/target` against the base ref in +`BASE_SHA`, and prints the engine's JSON to stdout. It is **advisory**: it never gates a merge, +never writes to GitHub, and is given no token. + +```bash +docker run --rm --read-only \ + -e OLLAMA_API_KEY -e CLAWSWEEPER_TARGET_REPO -e BASE_SHA \ + -v "$PWD:/target:ro" \ + ghcr.io/opencharly/clawsweeper-review: +``` + +`--local-range` is the only mode this image offers. It reviews `merge-base(base, HEAD)..HEAD`, +refuses a dirty tree, and makes no GitHub reads or writes — which is what lets it run with the +PR tree mounted read-only and no credential. + +## The pins and why each one exists + +| Pin | Value source | What breaks without it | +|---|---|---| +| `CLAWSWEEPER_COMMIT` | this candy's `var:` | the engine and its prompt/schema would float | +| `NODE_VERSION` | `24.x` | the engine requires Node >= 24 | +| `PNPM_VERSION` | `12.4.1` | the lockfile is frozen against it | +| `CODEX_VERSION` | `0.162.1` | Codex is the engine's only runner; a different version changes the wire contract | + +## The model path (measured) + +Codex accepts **only** `wire_api = "responses"`. Ollama Cloud serves stateless Responses at +`https://ollama.com/v1`, so the image writes: + +```toml +model = "deepseek-v4.1-flash" +model_provider = "ollama-cloud" +[model_providers.ollama-cloud] +base_url = "https://ollama.com/v1" +env_key = "OLLAMA_API_KEY" +wire_api = "responses" +requires_openai_auth = false +``` + +Proven with a tool call, not a completion: `codex exec` was asked to run `uname -s && uname -m` +and returned `Linux` / `x86_64` from inside its sandbox. If a future Codex or endpoint change +contradicts this, re-run that probe before touching the candy. + +## The target profile + +ClawSweeper refuses a repository it has no profile for. The candy appends a **generic fallback +for owner `opencharly`**, so any `opencharly/` resolves without a per-repo entry and an +upstream sync never conflicts with a local config edit. The candy asserts the entry exists. + +## Sandbox prerequisites + +Codex's read-only review sandbox uses bubblewrap, which needs unprivileged user namespaces. The +runner image (`github-runner-ubuntu`) sets +`kernel.apparmor_restrict_unprivileged_userns=0` and `kernel.unprivileged_userns_clone=1`; a host +without those fails at browser-free sandbox setup, not in this engine. + +## Failure classes, kept distinct + +- **Engine produced a verdict** → the caller posts it. +- **Provider never answered / empty completion** → no verdict; the caller reports it as + inconclusive and stays advisory. +- **A missing tool in the image** → the build-time `check:` steps name it (`node`, `pnpm`, + `codex`, the engine's `--help`, the profile, the config, the entrypoint). diff --git a/src/content/docs/recipes/check/crabbox.md b/src/content/docs/recipes/check/crabbox.md index 6efd9265..0e2ea867 100644 --- a/src/content/docs/recipes/check/crabbox.md +++ b/src/content/docs/recipes/check/crabbox.md @@ -32,7 +32,8 @@ and the coordinator with [`/charly-crabbox:crabbox-coordinator`](/recipes/crabbo | Group | Methods | |---|---| -| In-venue CLI — credential-free | `version` (`crabbox --version`) `providers` (`crabbox providers`) `config` (`crabbox config show`) `doctor` (`crabbox doctor` — needs NO broker configured, or a REACHABLE one) | +| In-venue CLI — credential-free | `version` (`crabbox --version`) `providers` (`crabbox providers`) `describe` (`crabbox providers describe --json`) `config` (`crabbox config show`) `doctor` (`crabbox doctor` — needs NO broker configured, or a REACHABLE one) | +| In-venue gate — containerd | `containerd-preflight` (read-only dependency probe for a containerd lease; PASS or a visible SKIP naming the exact absent dependency) | | Coordinator HTTP probe (host-side, port 8080) | `health` (`/v1/health`) `ready` (`/v1/ready`) | | In-venue CLI — broker-backed (requires `crabbox login` in the venue) | `leases` (`crabbox list` — the ID/NAME/STATE/TYPE/IP inventory) `usage` (`crabbox usage`) `events` (`crabbox events `) `logs` (`crabbox logs `) | @@ -43,7 +44,42 @@ on the pinned v0.79.0: `crabbox status --all` exits *"flag provided but not defined: -all"*), so the earlier prose that documented `leases` as `crabbox status --all` named a command that cannot run. -All methods are read-only — there are no mutating methods in this first cut. +All methods are read-only with respect to LEASES — none creates, mutates, or +releases one. `containerd-preflight`'s only write is a `nerdctl pull` of the +configured lease image, which proves the ref exists in a registry instead of +trusting a local cache. + +## `describe` — the provider's own description + +`describe` takes the provider as its **positional** argument (that CLI command has +no `--provider` flag) and asks for JSON: + +``` +crabbox providers describe containerd --json +{"provider":{"requested":"containerd","canonical":"containerd",...},"kind":"ssh-lease","runnable":true,...} +``` + +Assert one field with `json_path:` (e.g. `provider.canonical`, `kind`, `runnable`) +rather than pattern-matching the document. A provider the CLI does not have exits +non-zero and the step FAILS — an unknown name is an authoring error, not a skip. + +## `containerd-preflight` — the visible skip for an absent dependency + +A containerd lease needs a CLI that carries the provider, a reachable daemon, the +kernel's `xt_multiport` match (the CNI portmap hook publishes the lease SSH port), +and a pullable lease image. When any is absent the preflight returns `skip` with the +first missing dependency named: + +- `the in-venue crabbox CLI (version …) does not list the containerd provider` — the + venue's `layer-crabbox` predates the source build of `opencharly/crabbox`; +- `the containerd daemon at
(namespace …) is not reachable: …`; +- `the kernel has no xt_multiport match … (opencharly/layer-containerd#5)`; +- `the lease image is not present and cannot be pulled: …`. + +A provider that IS present but cannot describe itself, or a CLI whose `providers` +command fails, is a FAIL, not a skip. Never author a bed that runs a lease without +this gate: a `run:` step has no conditional SKIP, so the gate is what keeps the leg +honest while a dependency is missing. ## Input fields @@ -54,7 +90,9 @@ All methods are read-only — there are no mutating methods in this first cut. | `timeout` | HTTP probe timeout (default 10s) | | `json_path` | dotted path into the response; stdout becomes that value | | `run_id` | run identifier for the `events`/`logs` methods | -| `provider` | the provider to select for the methods whose CLI command DEFINES `--provider` (`doctor`, `leases`) | +| `provider` | the provider to select for the methods whose CLI command DEFINES `--provider` (`doctor`, `leases`), and the positional name for `describe` | +| `containerd` | the containerd provider's own keys, passed as `--containerd- ` on `doctor`/`leases`, and read by `containerd-preflight` (address, namespace, runtime, image) | +| `firecracker` | the firecracker provider's own keys, passed as `--firecracker- ` on `doctor`/`leases` | ## Scalar shorthand @@ -80,17 +118,40 @@ coordinator readiness first. ## Selecting a provider `provider:` is passed through as `--provider ` for the two methods whose CLI -command defines the flag — `doctor` and `leases` (`crabbox list`) — and nowhere -else, because attaching it to a command that does not define it is an argument -error, not a selection. Measured on the pinned v0.79.0: +command defines the flag — `doctor` and `leases` (`crabbox list`) — and becomes the +POSITIONAL argument for `describe` (`providers describe `); it is attached +nowhere else, because attaching it to a command that does not define it is an +argument error, not a selection. Measured on the release the org previously pinned +(v0.79.0) and on the source build of `opencharly/crabbox` b7b5aaa3: ``` $ crabbox doctor --provider local-container -> ok provider-selection provider=local-container source=flag selected=true $ crabbox list --provider aws -> the ID/NAME/STATE/TYPE/IP inventory $ crabbox usage --provider aws -> flag provided but not defined: -provider $ crabbox providers --provider aws -> flag provided but not defined: -provider +$ crabbox providers describe containerd --json -> {"provider":{"canonical":"containerd",...},"kind":"ssh-lease",...} ``` +The `containerd:` block rides the SAME allowlist as `--provider` (`doctor`, +`leases`), because those are the commands that define `--containerd-*`; it is never +appended to `providers` or `describe`. + Unset, the CLI keeps its own configured/ambient selection — which is what every step got before this input existed, so an authored step that does not name a provider behaves exactly as it did. + +## The config candy + +`opencharly/layer-crabbox` ships a SEPARATE candy, `crabbox-containerd-config`, +that renders `~/.config/crabbox/config.yaml`: + +```yaml +provider: containerd +containerd: + image: ghcr.io/opencharly/pod-crabbox:2026.293.1655 +``` + +It is referenced by path +(`@github.com/opencharly/layer-crabbox/candy/crabbox-containerd-config:`) so +the root `crabbox` candy keeps its meaning for every current consumer, and the +rendered bytes are validated against plugin-egress's `crabbox-yaml` schema. diff --git a/src/content/docs/recipes/core/charly-doctor.md b/src/content/docs/recipes/core/charly-doctor.md index f9cf92fb..3a0eff25 100644 --- a/src/content/docs/recipes/core/charly-doctor.md +++ b/src/content/docs/recipes/core/charly-doctor.md @@ -130,7 +130,7 @@ Each check shows the binary path and version when available, or an install hint ## Cross-References - [`/charly-automation:udev`](/recipes/automation/udev/) — install udev rules for GPU device access -- [`/charly-core:charly-config`](/recipes/core/charly-config/) — `engine.run`, `secret_backend` settings, `--no-auto-detect` flag, DRINODE injection via `appendAutoDetectedEnv()` +- [`/charly-core:charly-config`](/recipes/core/charly-config/) — `engine.build`, `engine.run`, `secret_backend` settings, `--no-auto-detect` flag, DRINODE injection via `appendAutoDetectedEnv()` - [`/charly-automation:enc`](/recipes/automation/enc/) — credential lookup path behind the Secret Service collection + keyring-index checks; iteration-capable ssClient; broken-collection troubleshooting - [`/charly-build:secrets`](/recipes/build/secrets/) — `charly secrets set/list/prune` commands referenced by the keyring-index remediation hint - [`/charly-build:settings`](/recipes/build/settings/) — `keyring_collection_label`, `secret_backend`, and other runtime config keys surfaced by the Secret Storage checks diff --git a/src/content/docs/recipes/core/charly-status.md b/src/content/docs/recipes/core/charly-status.md index 282fb12a..ccc0af3a 100644 --- a/src/content/docs/recipes/core/charly-status.md +++ b/src/content/docs/recipes/core/charly-status.md @@ -69,10 +69,10 @@ former `status-substrate` HostBuild seam is DELETED, K-wave 2). - `candy/plugin-status/nested_tree.go` — the declared-nested-tree pre-resolution (K5, relocated from charly/status_nested.go's buildStatusRootsTree): resolves the merged project + per-machine deploy tree directly (`HostBuild("resolved- - project")` + `deploykit.LoadDeployConfig`) into the wire-safe + project")` + `deploykit.LoadFleetConfig`) into the wire-safe `[]spec.StatusNestedNode` shape the overlay folds, including the `--nested` live-probe leg (`ResolveDeployChain` + `NestedExecutor`, the SAME primitive - `charly deploy add` / `charly check live parent.child` use). + `charly fleet add` / `charly check live parent.child` use). - `candy/plugin-status/command.go` (`hostStatusSubstrate`) — the generic `status-substrate` F10 forward is DELETED with `charly/status_substrate_host.go` (K-wave 2): `command:status` now `InvokeProvider(verb:status-fanout)` DIRECTLY over @@ -200,7 +200,7 @@ match keeps the synthesized declared row (`Source="nested"`). subprocesses. - **`charly status --nested`**: each child's LIVE venue is probed through the real multi-hop chain (`ResolveDeployChain` → `NestedExecutor`, the SAME primitive - `charly deploy add` and `charly check live parent.child` use — no bespoke nested dial) + `charly fleet add` and `charly check live parent.child` use — no bespoke nested dial) under a STRICT 4-second per-child context deadline. A timed-out / failing child renders `unreachable`; the table is NEVER blocked. The deadline is a context cancellation, never a sleep/retry loop. `--nested` also turns on live @@ -318,7 +318,7 @@ Authoritative direct queries (when you need the raw mount data): ```bash charly status --json # volumes[] carries the live mounts -charly deploy show +charly fleet show ``` ## Usage diff --git a/src/content/docs/recipes/core/clean.md b/src/content/docs/recipes/core/clean.md index 498bb491..c9e08c9e 100644 --- a/src/content/docs/recipes/core/clean.md +++ b/src/content/docs/recipes/core/clean.md @@ -1,6 +1,6 @@ --- title: "clean" -description: "Prune reusable build artifacts to defaults: retention (images, check runs) and sweep one-time build-candy leftovers — `charly clean`, keep_images/keep_check_runs, image-tag pruning, .check run cleanup." +description: "Prune reusable build artifacts to defaults: retention (images, check runs) and sweep one-time makepkg leftovers — `charly clean`, keep_images/keep_check_runs, image-tag pruning, .check run cleanup." --- @@ -39,7 +39,7 @@ opts in; third-party configs get no surprise pruning until they set a value. ## Commands ```bash -charly clean # apply retention now: prune images + check runs + build-candy leftovers +charly clean # apply retention now: prune images + check runs + makepkg leftovers charly clean --dry-run # print everything that WOULD be removed; touch nothing charly clean --images # only image-tag retention charly clean --check # only check-run retention @@ -50,28 +50,12 @@ charly clean --deep --dry-run # the safe default probe: report the would-remov charly clean --keep N # override the retention count for this run (0 = use defaults:) charly clean --invalidate '' # remove charly-labeled image tags matching the glob # (full ref or last segment; in-use skipped; runs ONLY this) -charly clean --cache # reclaim unreferenced blobs from every named charly cache (the - # content-addressed ArtifactStore's own GC); reports each store's live - # entries + the blobs/bytes it would reclaim; runs ONLY this category -charly clean --scopes # reap HUNG transient build scopes — the systemd user scopes - # buildah/podman put a build container into (runc-buildah-*.scope). - # A scope is stopped only when it has hung; a live one is left alone ``` -With **none** of `--images` / `--check` / `--deep` / `--cache` / `--scopes`, the **four** DEFAULT categories -run (image-tag + check-run + build-candy + scopes) — unchanged, and `--deep` and `--cache` NEVER fire -implicitly. `--scopes` IS in that list because it is itself a category flag: the test the code applies is -`anyCategory := images || check || deep || cacheGC || scopes`, so naming ANY one of the five suppresses the -default set and runs only what was named (plus, for `--deep`, the scope reap below). -**`--scopes` is a default and `--deep` implies it:** `cleanCategories` computes -`doScopes = scopes || deep || !anyCategory`, so a bare `charly clean` reaps hung build scopes -and `--deep` does too, while `--deep` on its own runs only the image half plus that scope reap — and -`--cache` on its own runs only the cache sweep, with `doScopes` false. -`--keep N` overrides both the image and check-run retention counts for the invocation (it does -not affect `--deep`, which has no keep-N — it purges every untagged image). -**Category names, from the engine's own header** (`retention.go`): *image-tag / build-candy / check-run -pruning*, plus the `--deep` store-wide purge and the `--cache` blob sweep. The middle one prunes -`.build/_candy/./` per candy (`pruneBuildCandyDirs`). +With neither `--images` nor `--check` nor `--deep`, the three DEFAULT categories run (images + +check + makepkg) — unchanged, `--deep` NEVER fires implicitly. `--keep N` overrides both the +image and check-run retention counts for the invocation (it does not affect `--deep`, which has +no keep-N — it purges every untagged image). **`--deep` — the store-wide untagged/dangling-image purge.** The default image-tag sweep above only ever touches images carrying the `ai.opencharly.box` label. A multi-stage build's @@ -211,7 +195,7 @@ importing only `sdk/kit` + `spec/spec`). `--deep` shares its engine with the def charly-labeled dangling sweep via `pruneDanglingImages`/`selectDanglingImages` (`candy/plugin-clean/retention.go`) parameterized by an `onlyCharly` bool: `pruneDanglingCharlyImages` (onlyCharly=true, the default sweep) and `pruneDeepDanglingImages` (onlyCharly=false, `--deep`) -are both thin wrappers over the ONE shared selection + removal engine (R2 — no duplicated +are both thin wrappers over the ONE shared selection + removal engine (R3 — no duplicated listing/removal logic between the two categories). The engine is reached two ways: `charly clean`'s own CLI calls `runRetention` in-package (no wire hop); the three OTHER callers reach `verb:retention` — the build command's post-build prune (`candy/plugin-box/box.go`'s diff --git a/src/content/docs/recipes/core/deploy.md b/src/content/docs/recipes/core/deploy.md index 0cf16930..8c19e8c6 100644 --- a/src/content/docs/recipes/core/deploy.md +++ b/src/content/docs/recipes/core/deploy.md @@ -1,6 +1,6 @@ --- title: "deploy" -description: "`charly deploy add`/`charly deploy del`, quadlet generation, volume backing, tunnels (Tailscale/Cloudflare), `add_candy:` overlay, or per-machine deploy overlays." +description: "`charly fleet add`/`charly fleet del`, quadlet generation, volume backing, tunnels (Tailscale/Cloudflare), `add_candy:` overlay, or per-machine deploy overlays." --- @@ -15,26 +15,26 @@ description: "`charly deploy add`/`charly deploy del`, quadlet generation, volum - **The substrate node at the edge** — `vm: {from: }` for a VM target, `pod: {image: }` for a pod target, `kubernetes: {from: }` for a cluster target, `local: {from: