From 67a7eaa546134bb7eae29dd99c63f537f3f78581 Mon Sep 17 00:00:00 2001 From: Andreas Trawoeger Date: Thu, 29 Oct 2026 01:13:16 +0100 Subject: [PATCH] docs(corpus)!: advance the corpus and the plugin-build pin, and regenerate the site on --stage-parallelism MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This repo's `marketplace` pin moves from 99a1bb4a6f10112f67e3ddd73d3e86b3f6bc3417 to 5feb9c8b8350849fd2d46d252b486844c18bd0a4 — the corpus's merged `main` tip, advanced through the submodule's own remote-tracking path, never a hand-set gitlink. The corpus is what decides which projection of the guidance corpus the site serves, and at this tip it names the `build`-family owners DIRECTLY — plugin-box at v2026.301.1542 among them — so the recipe pages it projects teach the renamed knob. The wiring candy's own `plugin-build` pin moves from v2026.242.0525 to v2026.296.1104 — the newest plugin-build tag, and the first one whose `candy/plugin-build/schema/build.cue` carries `stage_parallelism?: int`. That is what the site's `reference/plugin/**` and `reference/candy/**` sections project, so the site's last `podman_jobs?: int` goes with the old pin. THE REGRESSION THIS REPAIRS, AND ITS ROOT CAUSE. The published site taught the retired flag before, then stopped, then taught it again — measured on this repo's own history: $ for c in 81da73e c0e7db9 98a5f6c; do git show $c:src/content/docs/recipes/build/build.md \ | grep -cE 'stage-parallelism|podman-jobs'; done 7 0 # 81da73e — PR #167, the corpus pin at layer-charly-build:v2026.292.2017 0 7 # c0e7db9 — PR #176, the corpus pin at layer-charly-build:v2026.296.0919 0 7 # 98a5f6c — PR #197, the corpus pin at the corpus's merged main of Oct 28 The pin moved FORWARD in time every time, and the CONTENT moved BACKWARD. The mechanism, measured on the corpus's own refs list and on the owner's tree at the tag the corpus named: $ for c in 602324e 1c15651 99a1bb4a 5feb9c8; do git -C marketplace show $c:candy/charly-marketplace/charly.yml \ | grep -oE '@github.com/opencharly/layer-charly-build:v[0-9.]+'; done 602324e layer-charly-build:v2026.292.2017 # the body teaches --stage-parallelism 1c15651 layer-charly-build:v2026.296.0919 # the body teaches --podman-jobs 99a1bb4a layer-charly-build:v2026.296.0919 # the body teaches --podman-jobs 5feb9c8 (no ref — the build family is named from plugin-box:v2026.301.1542) `layer-charly-build` was DRAINED by the build-family split, and at its tag v2026.296.0919 it has no `.agents/` tree at all — its manifest says it "owns NO `skill:` entity any more". So the corpus's `build/skills/build/SKILL.md` at 1c15651 and 99a1bb4a COULD NOT HAVE BEEN SOURCED from the tag the corpus's own refs list names: it was an ORPHAN body, carried in the corpus's tree from before the drain. The corpus published a body that no tag it named could supply, and the site published that body faithfully. The MISSED CONTROL is exactly there. A provenance check that compares a projected body against the owner's file AT THE NAMED TAG has nothing to compare an orphan against, so it is silent precisely where the defect lived. The site's drift gate is downstream of the same blindness: it compares the committed tree to the pinned inputs, and a faithful projection of a regressed input IS "no drift". The corpus currency report asks a third question — is the PIN behind the corpus's merged main — and answered `cur` throughout. The BLAST RADIUS: the published `build` family and every page naming the flag. The site told its readers to pass `--podman-jobs`, a flag the shipped binary REJECTS with rc=80 since opencharly/charly#1082 landed (opencharly/plugin-box#66 renamed it). THE NEW PIN HAS NO ORPHAN. The corpus's body at 5feb9c8 is plugin-box's file at v2026.301.1542 plus the generator's own header, and nothing else: $ git -C marketplace show 5feb9c8:build/skills/build/SKILL.md > corpus.md $ git --git-dir=$GD/plugin-box show v2026.301.1542:.agents/skills/build/SKILL.md > owner.md $ diff owner.md corpus.md 6a7,8 > THE CLAIM, MEASURED. The retired name is gone from the published tree, and the new one is carried: $ grep -rnE "podman-jobs|podman_jobs|PODMAN_JOBS" src/content/docs (no output; exit 1) $ git grep --cached -nE "podman-jobs|podman_jobs|PODMAN_JOBS" -- src/content/docs (no output; exit 1) Four carriers carried it at HEAD: the `build` recipe (the flag line, the `Parallelism: --jobs vs --podman-jobs` section, the `defaults.podman_jobs` / `podman_jobs_cap` example), the `beds-and-r10` recipe, the `parallel-bed-testing` recipe, and the plugin-build reference page's schema block. The last is the pin advance; the first three are the corpus advance. No page is hand-edited. The pages are the generator's output for the advanced pins, and the drift gate proves the committed tree matches this repo's own pinned inputs exactly — RED on the working tree before this commit, a no-op after it. Refs opencharly/docs#196 Refs opencharly/opencharly#785 Assisted-by: Claude Code minimax/minimax-m3.1-flash:cloud (fully tested and validated) --- .github/docs-corpus/pins.tsv | 2 +- marketplace | 2 +- src/content/docs/recipes/build/build.md | 176 +++++++----------- src/content/docs/recipes/build/docs.md | 40 +++- src/content/docs/recipes/build/generate.md | 6 +- src/content/docs/recipes/build/migrate.md | 113 ++++++----- src/content/docs/recipes/build/pull.md | 15 +- src/content/docs/recipes/build/settings.md | 38 +++- .../docs/recipes/check/check/beds-and-r10.md | 2 +- src/content/docs/recipes/internals/agents.md | 2 +- .../internals/agents/parallel-bed-testing.md | 9 +- .../docs/recipes/internals/git-workflow.md | 6 +- .../git-workflow/branch-and-pr-loop.md | 4 +- .../git-workflow/pin-intent-and-currency.md | 4 +- .../git-workflow/umbrella-mechanics.md | 9 +- .../docs/recipes/internals/marketplace.md | 39 ++-- .../docs/recipes/internals/repo-setup.md | 5 +- .../plugin-build.md | 49 ----- .../plugin-build.md | 56 ++++++ .../plugin-build.md | 29 +-- src/content/docs/reference/providers.md | 6 +- 21 files changed, 325 insertions(+), 287 deletions(-) delete mode 100644 src/content/docs/reference/candy/github-com-opencharly-plugin-build-v2026-242-0525/plugin-build.md create mode 100644 src/content/docs/reference/candy/github-com-opencharly-plugin-build-v2026-296-1104/plugin-build.md rename src/content/docs/reference/plugin/{github-com-opencharly-plugin-build-v2026-242-0525 => github-com-opencharly-plugin-build-v2026-296-1104}/plugin-build.md (69%) diff --git a/.github/docs-corpus/pins.tsv b/.github/docs-corpus/pins.tsv index 6d3e0e78..e2868b94 100644 --- a/.github/docs-corpus/pins.tsv +++ b/.github/docs-corpus/pins.tsv @@ -33,7 +33,7 @@ plugin-sidecar v2026.242.0532 plugin-substrate v2026.242.0533 plugin-candy-kind v2026.242.0527 plugin-harness-kind v2026.242.0529 -plugin-build v2026.242.0525 +plugin-build v2026.296.1104 plugin-installstep v2026.242.0532 plugin-tunnel v2026.242.0609 plugin-tmux v2026.242.0534 diff --git a/marketplace b/marketplace index 99a1bb4a..5feb9c8b 160000 --- a/marketplace +++ b/marketplace @@ -1 +1 @@ -Subproject commit 99a1bb4a6f10112f67e3ddd73d3e86b3f6bc3417 +Subproject commit 5feb9c8b8350849fd2d46d252b486844c18bd0a4 diff --git a/src/content/docs/recipes/build/build.md b/src/content/docs/recipes/build/build.md index 52b8a690..5f441401 100644 --- a/src/content/docs/recipes/build/build.md +++ b/src/content/docs/recipes/build/build.md @@ -13,13 +13,13 @@ Invoked as `charly box build`. See [`/charly-image:image`](/recipes/image/image/ ## Overview -`charly box build` generates Containerfiles from `charly.yml` and layer definitions, then builds images in dependency order using the configured build engine (Docker or Podman). Images at the same dependency level are built in parallel (up to `--jobs` concurrent builds). +`charly box build` generates Containerfiles from `charly.yml` and layer definitions, then builds images in dependency order with **BuildKit**, charly's only image builder. Images at the same dependency level are built in parallel (up to `--jobs` concurrent builds). **Mode purity**: `charly box build` reads `charly.yml` only. `charly.yml` is never read during build — this is enforced by `LoadConfig` in `charly/loader_threaded.go` (it reads the PROJECT `charly.yml` only and never merges the per-host `charly.yml` overlay — no `MergeDeployOverlay`), to guarantee OCI labels are baked strictly from authored configuration, never from local deploy-time overrides. See [`/charly-internals:go`](/recipes/internals/go/) "Mode purity" for the architectural invariant this protects and the bug it prevents. -**Build-engine dispatch**: `charly box build` does not construct a generator host-side at all (the former `NewGenerator` is DELETED) — `BuildCmd.Run` computes the build tag, holds the build-activity retention lock, and dispatches through the COMPILED-IN `build:box` plugin (candy/plugin-build) over the F10 reverse channel; on return it runs the retention prune host-side. The plugin's `runBoxBuild` → `resolveBuildEngine` (candy/plugin-build/resolve.go) runs the ENTIRE build-engine RESOLVE plugin-side — loader, vocab, scan, connect, validate, box resolve, intermediates, order, effective versions, host-fs prep, render-seam-cache prep, render-prep, user-context, envelope projection, and drive-model — and returns `spec.BuildResolveReply` (the resolved-project envelope + a serializable drive-model: engine/platform, build order, per-image descriptors, resolved tunables; NO Containerfile content). This is the plugin-side replacement for the DELETED host `build-prep` fat seam (`charly/build_resolve_host.go`'s `hostBuildBuildResolve`). The plugin **OWNS the podman DRIVE** (the build-order loop, per-image build lock, `podman build`, push, and the inline-merge gate) IN the candy — it imports only the sdk module. It reaches the host ONLY for thin `buildengine-*` shards a sdk-only candy cannot hold: `buildengine-scan-local` (local candy scan), `buildengine-connect-plugins` (registry plugin connect), and `buildengine-prep` (the render-seam-floor `renderGenCache` — `hostBuildPrep` stores a CHEAP candy-scan-only Generator for the render-seam floor's reverse-channel consumers and returns an EMPTY map; the host-fs PREP itself moved plugin-side in the K3 host-prep move — `runHostFSPrep` in candy/plugin-build/host_prep.go). The resolved-project envelope + drive-model are NOT returned by `buildengine-prep` — they come from `resolveBuildEngine`'s reply. The render DRIVE is `sdk/deploykit.Generator`, driven by the candy over the envelope + `HostBuild("render-seam")` for the host-coupled render seams (#67 — the host no longer renders the `.build/` tree); `bake_plugin:` binary baking is INLINE via `deploykit.EmitBakedPlugins` (no `HostBuild("bake-plugins")` round-trip — that host-builder + `host_build_bake_plugins.go` are DELETED). `charly box generate` mirrors this via `build:generate` → `runBoxGenerate` → `resolveBuildEngine(..., generateOnly=true)`: `generateOnly` is a PLUGIN-SIDE parameter (NOT a `HostBuild("buildengine-prep")` argument — that host leg takes `spec.ResolvedProjectRequest`, which has no `GenerateOnly` field) that returns after the envelope projection (no engine/order drive-model); the candy renders the `.build/` tree itself via `deploykit.Generator` + returns the written Containerfile paths; no podman. (The resolved-project envelope that OTHER verbs fetch — `charly box inspect`, `charly fleet compile`, `charly check`, … — comes over the separate `InvokeProvider("build","project",OpResolve)` peer seam, the `build:project` word; the build/generate path projects its own envelope inside `resolveBuildEngine`.) See [`/charly-internals:plugin`](/recipes/internals/plugin/) (the `build` provider class + the in-proc reverse channel) and [`/charly-build:generate`](/recipes/build/generate/). +**Build dispatch**: `charly box build` does not construct a generator host-side at all (the former `NewGenerator` is DELETED) — `BuildCmd.Run` computes the build tag, holds the build-activity retention lock, and dispatches through the COMPILED-IN `build:box` plugin (candy/plugin-build) over the F10 reverse channel; on return it runs the retention prune host-side. The plugin's `runBoxBuild` → `resolveBuildEngine` (candy/plugin-build/resolve.go) runs the ENTIRE build RESOLVE plugin-side — loader, vocab, scan, connect, validate, box resolve, intermediates, order, effective versions, host-fs prep, render-seam-cache prep, render-prep, user-context, envelope projection, and drive-model — and returns `spec.BuildResolveReply` (the resolved-project envelope + a serializable drive-model: engine/platform, build order, per-image descriptors, resolved tunables; NO Containerfile content). This is the plugin-side replacement for the DELETED host `build-prep` fat seam (`charly/build_resolve_host.go`'s `hostBuildBuildResolve`). The plugin **OWNS the BuildKit DRIVE** (the build-order loop, per-image build lock, the `buildctl` argv, the store bridge, push, and the inline-merge gate) IN the candy — it imports only the sdk module. It reaches the host ONLY for thin `buildengine-*` shards a sdk-only candy cannot hold: `buildengine-scan-local` (local candy scan), `buildengine-connect-plugins` (registry plugin connect), and `buildengine-prep` (the render-seam-floor `renderGenCache` — `hostBuildPrep` stores a CHEAP candy-scan-only Generator for the render-seam floor's reverse-channel consumers and returns an EMPTY map; the host-fs PREP itself moved plugin-side in the K3 host-prep move — `runHostFSPrep` in candy/plugin-build/host_prep.go). The resolved-project envelope + drive-model are NOT returned by `buildengine-prep` — they come from `resolveBuildEngine`'s reply. The render DRIVE is `sdk/deploykit.Generator`, driven by the candy over the envelope + `HostBuild("render-seam")` for the host-coupled render seams (#67 — the host no longer renders the `.build/` tree); `bake_plugin:` binary baking is INLINE via `deploykit.EmitBakedPlugins` (no `HostBuild("bake-plugins")` round-trip — that host-builder + `host_build_bake_plugins.go` are DELETED). `charly box generate` mirrors this via `build:generate` → `runBoxGenerate` → `resolveBuildEngine(..., generateOnly=true)`: `generateOnly` is a PLUGIN-SIDE parameter (NOT a `HostBuild("buildengine-prep")` argument — that host leg takes `spec.ResolvedProjectRequest`, which has no `GenerateOnly` field) that returns after the envelope projection (no engine/order drive-model); the candy renders the `.build/` tree itself via `deploykit.Generator` + returns the written Containerfile paths; no podman. (The resolved-project envelope that OTHER verbs fetch — `charly box inspect`, `charly check`, … — comes over the separate `InvokeProvider("build","project",OpResolve)` peer seam, the `build:project` word; the build/generate path projects its own envelope inside `resolveBuildEngine`.) See [`/charly-internals:plugin`](/recipes/internals/plugin/) (the `build` provider class + the in-proc reverse channel) and [`/charly-build:generate`](/recipes/build/generate/). -**Build-mode emission**: the build-mode Containerfile render DRIVE is `sdk/deploykit.Generator.Generate` / `generateContainerfile` (relocated from `charly/generate.go` in #67), driven by `candy/plugin-build` over the resolved-project envelope + `HostBuild("render-seam")`; the `charly/generate.go` top-level `Generate`/`writeCandySteps` orchestrator is DELETED. the pod-overlay deploy path reaches `deploykit.Generator.EmitTasks` DIRECTLY — `charly/tasks.go` holds no shim and cannot hold one (P16 import purity forbids `charly/` importing the sdk); that path stays for the pod-overlay deploy path (the candy `plugin-deploy-pod` constructs `deploykit.OCITarget` + renders via `deploykit.NewRenderGeneratorFromProject`, the per-step fragment over the `HostBuild("step-emit","oci-emit-step")` seam); the build-mode render (in `sdk/deploykit`) drives `WriteCandySteps` → `EmitTasks` directly. The render walks each layer's ops directly to write Containerfile text — NOT the `InstallPlan` IR. It shares the package-cascade / shell-snippet / localpkg compiler helpers with the IR (one source of truth, R3), but the overall walk is the direct generator. The `InstallPlan` IR + `deploykit.OCITarget.Emit` is the DEPLOY-mode path used by `charly fleet add` (the candy `plugin-deploy-pod`'s `add_candy:` overlay synthesis, rendered in its own code via `deploykit.OCITarget` + `deploykit.NewRenderGeneratorFromProject` + the external out-of-process deploys; the local/vm/kubernetes/android substrates are external plugins via `pluginDeployTarget` (S3b, dispatched through `candy/plugin-fleet`'s `Invoke(OpDeployDispatch)`) — `deploy:local` via candy/plugin-deploy-local and `deploy:vm` via candy/plugin-deploy-vm DO consume the IR (the plugin walks it via `kit.WalkPlans` over the reverse channel, the vm one over the guest `SSHExecutor`), while `deploy:kubernetes` via candy/plugin-kube does NOT, materializing its Kustomize tree PLUGIN-side (`candy/plugin-kube/materialize.go`'s `materializeKustomize`, reaching `candy/plugin-k8sgen` peer-to-peer over `verb:k8sgen`/`OpEmit`)). See [`/charly-internals:install-plan`](/recipes/internals/install-plan/) for the IR and [`/charly-internals:generate-source`](/recipes/internals/generate-source/) for the Go call graph. +**Build-mode emission**: the build-mode Containerfile render DRIVE is `sdk/deploykit.Generator.Generate` / `generateContainerfile` (relocated from `charly/generate.go` in #67), driven by `candy/plugin-build` over the resolved-project envelope + `HostBuild("render-seam")`; the `charly/generate.go` top-level `Generate`/`writeCandySteps` orchestrator is DELETED. the pod-overlay deploy path reaches `deploykit.Generator.EmitTasks` DIRECTLY — `charly/tasks.go` holds no shim and cannot hold one (P16 import purity forbids `charly/` importing the sdk); that path stays for the pod-overlay deploy path (the candy `plugin-deploy-pod` constructs `deploykit.OCITarget` + renders via `deploykit.NewRenderGeneratorFromProject`, the per-step fragment over the `HostBuild("step-emit","oci-emit-step")` seam); the build-mode render (in `sdk/deploykit`) drives `WriteCandySteps` → `EmitTasks` directly. The render walks each layer's ops directly to write Containerfile text — NOT the `InstallPlan` IR. It shares the package-cascade / shell-snippet / localpkg compiler helpers with the IR (one source of truth, R2), but the overall walk is the direct generator. The `InstallPlan` IR + `deploykit.OCITarget.Emit` is the DEPLOY-mode path used by `charly deploy add` (the candy `plugin-deploy-pod`'s `add_candy:` overlay synthesis, rendered in its own code via `deploykit.OCITarget` + `deploykit.NewRenderGeneratorFromProject` + the external out-of-process deploys; the local/vm/kubernetes/android substrates are external plugins via `pluginDeployTarget` (S3b, dispatched through `candy/plugin-fleet`'s `Invoke(OpDeployDispatch)`) — `deploy:local` via candy/plugin-deploy-local and `deploy:vm` via candy/plugin-deploy-vm DO consume the IR (the plugin walks it via `kit.WalkPlans` over the reverse channel, the vm one over the guest `SSHExecutor`), while `deploy:kubernetes` via candy/plugin-kube does NOT, materializing its Kustomize tree PLUGIN-side (`candy/plugin-kube/materialize.go`'s `materializeKustomize`, reaching `candy/plugin-k8sgen` peer-to-peer over `verb:k8sgen`/`OpEmit`)). See [`/charly-internals:install-plan`](/recipes/internals/install-plan/) for the IR and [`/charly-internals:generate-source`](/recipes/internals/generate-source/) for the Go call graph. **Three-phase templates**: the embedded build vocabulary's format and builder definitions split each install operation into `phases.{prepare, install, cleanup}.{container, host}` — three phases × two venues. Build-mode emission reads the `container` cell; local deploys read `host` — the `(install, container)` cell is the single source of truth for build-mode package installation. See [`/charly-image:layer`](/recipes/image/layer/) "Service Declaration" for the analogue at the init-system level (`init..service_schema`). @@ -44,7 +44,7 @@ charly box build --cache image [image...] # Image cache (read-only, de charly box build --cache gha [image...] # GitHub Actions cache charly box build --no-cache [image...] # Disable cache entirely charly box build --jobs N [image...] # Max concurrent images per DAG level (0=auto from defaults.jobs, else 4) -charly box build --podman-jobs N [image...] # Max concurrent stages within a single podman build (0=auto: min(NCPU, defaults.podman_jobs_cap)) +charly box build --stage-parallelism N [image...] # Max concurrent stages within a single build (0=auto: min(NCPU, defaults.stage_parallelism_cap)) charly box build --include-disabled # Build an `enabled: false` image without flipping authored config charly box build --dev-local-pkg # EVAL-BED ONLY: build localpkg candies (the charly toolchain) from LOCAL in-dev source, not the published release ``` @@ -59,7 +59,7 @@ in-development binary + plugins — a disposable check bed always tests the in-development charly, never a stale release. Generic across all kinds + all localpkg candies, one decision point (`deploykit.RenderLocalPkgImageInstall`); a production box build omits the flag. A dev build that cannot build via the plugin -HARD-errors (no silent release fallback, R4). +HARD-errors (no silent release fallback, R3). Full mechanics: [`/charly-internals:install-plan`](/recipes/internals/install-plan/) "Check-vs-production charly toolchain"; the candy view: [`/charly-tools:charly`](/recipes/tools/charly/). @@ -164,48 +164,46 @@ cat .build/my-image/Containerfile # Inspect generated output ## Build Flow 1. Run `charly box generate` internally (produces Containerfiles in `.build/`) -2. Resolve runtime config to get build engine (`engine.build`) +2. Resolve runtime config (there is no build-engine selection: BuildKit is the builder) 3. Resolve image build order (dependency ordering, grouped by level) 4. Filter to requested images (and their base dependencies) 5. For each level: build images in parallel (up to `--jobs` concurrent, default 4) 6. After all builds: `charly box merge --all` (if `merge.auto` enabled, skipped for `--push`) -## Parallelism: `--jobs` vs `--podman-jobs` +## Parallelism: `--jobs` vs `--stage-parallelism` charly exposes **two** parallelism knobs with distinct meanings: | Flag | Env var | `defaults:` key | What it controls | |---|---|---|---| | `--jobs N` | `CHARLY_BUILD_JOBS` | `jobs` | **Outer** concurrency: how many charly-level images to build in parallel within a DAG level (e.g., when `charly box build` rebuilds the whole graph). | -| `--podman-jobs N` | `CHARLY_PODMAN_JOBS` | `podman_jobs` (+ `podman_jobs_cap`) | **Inner** concurrency: passed to `podman build --jobs N`, controls how many stages of a *single* multi-stage build run concurrently. | +| `--stage-parallelism N` | `CHARLY_STAGE_PARALLELISM` | `stage_parallelism` (+ `stage_parallelism_cap`) | **Inner** concurrency: how many stages of a *single* multi-stage build run concurrently. It becomes the BuildKit daemon's `[worker.oci] max-parallelism`, applied when the daemon starts. | Both knobs are **config-driven**: precedence is **CLI flag → env → `defaults:` in `charly.yml` → built-in fallback** (4). The inner auto value (when -`--podman-jobs` / `CHARLY_PODMAN_JOBS` / `defaults.podman_jobs` are all unset) is -CPU-proportional, capped at `defaults.podman_jobs_cap`: `min(NCPU, cap)`. The -cap is the operative ceiling; the repo ships `podman_jobs_cap: 8`. (A -high-concurrency `--cache-from` SIGABRT race in podman ≤ 5.7.x originally -motivated a hard cap of 4 — see `CHANGELOG/` for that history and the 20-run -race gate required before raising the cap on a new podman version.) +`--stage-parallelism` / `CHARLY_STAGE_PARALLELISM` / `defaults.stage_parallelism` are all unset) is +CPU-proportional, capped at `defaults.stage_parallelism_cap`: `min(NCPU, cap)`. +The cap is the operative ceiling, and it is the project's own to set. (The 20-run +race gate required before raising it on a new engine version is in `CHANGELOG/`.) ```yaml # charly.yml — defaults: block defaults: jobs: 4 # outer: images per DAG level - podman_jobs: 0 # inner: 0 = auto = min(NCPU, podman_jobs_cap) - podman_jobs_cap: 8 # ceiling for the auto calc + stage_parallelism: 0 # inner: 0 = auto = min(NCPU, stage_parallelism_cap) + stage_parallelism_cap: 8 # ceiling for the auto calc ``` ```bash -charly box build --podman-jobs 16 # fully parallel stages (override) -CHARLY_PODMAN_JOBS=8 charly update --build # via env -charly box build --podman-jobs 1 # fully serialised, worst-case debugging +charly box build --stage-parallelism 16 # fully parallel stages (override) +CHARLY_STAGE_PARALLELISM=8 charly update --build # via env +charly box build --stage-parallelism 1 # fully serialised, worst-case debugging ``` -Source: `candy/plugin-build/resolve.go` (`resolveDrivePodmanJobs`, feeding -`buildkit.ResolvePodmanJobs`; the host-resolved `PodmanJobs` value feeds `--jobs` -directly in `candy/plugin-build/podman.go`). Covered by -`candy/plugin-build/podman_test.go`. The outer `--jobs` knob and the inner `--podman-jobs` +Source: `candy/plugin-build/resolve.go` (`resolveDriveStageParallelism`, feeding +`buildkit.ResolveStageParallelism`; the host-resolved `StageParallelism` value is the +`MaxParallelism` the drive hands the daemon in `candy/plugin-build/buildkit.go`). +Covered by `buildkit/build_helpers_test.go` (`TestResolveStageParallelism`). The outer `--jobs` knob and the inner `--stage-parallelism` are separate fields so the two semantics don't get conflated. ## Build-context excludes (`defaults.context_ignore`) @@ -229,8 +227,9 @@ defaults: Why it matters: the build context tar is streamed to the engine on **every** build regardless of cache state, so excluding large never-COPYed directories is -the dominant warm-rebuild win. podman reads `.containerignore`; docker reads -`.dockerignore` — emitting both keeps the two engines in lockstep. Only add a +the dominant warm-rebuild win. BuildKit's dockerfile frontend reads `.dockerignore`; +`.containerignore` is the engine-native name kept for engine-native builds — emitting both +keeps the two readers in lockstep. Only add a directory you've confirmed no Containerfile COPY/ADDs from (generated Containerfiles COPY only from `candy/`, `templates/`, `.build/`). Source: `candy/plugin-build/host_prep.go:writeContextIgnore` + `charly/host_build_buildengine.go` @@ -288,8 +287,8 @@ what actually orders them; **within one image's tag rows**, the *"creation time, not the tag"* across to the tag budget — it is true of the first and false of the second. -Images referenced by a container (`podman ps -a`) are skipped, and `rmi` runs without -`-f` as a backstop. +Images referenced by a running container are skipped, and removal runs without `-f` as a +backstop (`charly clean` owns the prune; never call the engine's prune directly). ```yaml # charly.yml — defaults: @@ -334,10 +333,8 @@ OCI labels. The CLI accepts an explicit `--tag ` for pinning; an empty `--tag` resolves to newest-local automatically. Rationale: -1. **No stale-`:latest`-under-`--cache-from` bug.** The documented - caveat where podman's `--cache-from` silently pulls a stale - `:latest` from the remote registry is unreachable when charly never - emits `:latest` in the first place. +1. **Nothing can pin a stale float tag.** charly never emits `:latest`, so a build can never + resolve to a float tag a previous build left behind. 2. **No "ambiguous short name" errors under multiple tags.** Previously the resolver listed every `:latest` + `:` tag of the same image as competing candidates; the CalVer sort picks the newest @@ -347,7 +344,7 @@ Rationale: a CalVer tag is self-describing and rotates naturally. Any stray `:latest` tag in local storage is never refreshed by a new -build; prune it with `podman image prune` if you want it gone. +build; prune it with `charly clean` if you want it gone. ## Cache Efficiency @@ -367,7 +364,7 @@ Three kinds of source changes are real cache invalidators — if you see a long ### What does NOT invalidate cache -- **CalVer tag shifts alone.** The Containerfile emits `ARG BASE_IMAGE=/:` with a fresh CalVer on every generate. That ARG default appears in the Containerfile text but is not part of the cache key for subsequent RUN/COPY steps — podman/buildah resolves the FROM to an image SHA first, and caches downstream steps off that SHA. If the SHA is unchanged, cache hits. The cache cost comes from content changes in the layer itself, not the tag. +- **CalVer tag shifts alone.** The Containerfile emits `ARG BASE_IMAGE=/:` with a fresh CalVer on every generate. That ARG default appears in the Containerfile text but is not part of the cache key for subsequent RUN/COPY steps — BuildKit resolves the FROM to an image digest first, and caches downstream steps off that digest. If the SHA is unchanged, cache hits. The cache cost comes from content changes in the layer itself, not the tag. - **Baked `plan:` edits.** LABEL directives are emitted last in every final stage (after the last USER). Editing a baked step (a `check:`/`agent-check:` step, or a runtime-context `run:` step) — a common layer mutation — only re-runs the final LABEL block (~2 seconds on a 138-step stack like `immich-ml`). (Editing a build/deploy-context `run:` step changes the install timeline, so it re-runs that RUN/COPY instead.) See [`/charly-internals:generate-source`](/recipes/internals/generate-source/) for the rationale. - **Re-running `charly box build` without source changes.** Fully cached; seconds to complete. @@ -391,16 +388,19 @@ Three kinds of source changes are real cache invalidators — if you see a long ## Build Flow Details -**Internal base images** use exact CalVer tags in Containerfiles (`FROM ghcr.io/opencharly/fedora:2026.46.1415`). This ensures each image references the precise version of its parent. Both Docker and Podman resolve local images before pulling from registry. +**Internal base images** use exact CalVer tags in Containerfiles (`FROM ghcr.io/opencharly/fedora:2026.46.1415`), so each image names the precise version of its parent. **The builder does not resolve that tag through a local-image lookup**: BuildKit's OCI worker has no local-first resolution. -## Push Mode +Every build's export is NAMED — `io.containerd.image.name=` plus `org.opencontainers.image.created` on the manifest — and written to the **charly image store**, one engine-neutral OCI layout at `$XDG_DATA_HOME/charly/images` (`CHARLY_IMAGE_STORE_DIR` relocates it for a disposable target). The store is the build's canonical artefact, and it is what a dependent box's parent is solved from: when `ARG BASE_IMAGE=` names an image already in the store, the drive hands the store to the daemon as an `--oci-layout` and binds that ref to the parent's manifest digest as a named context, so the parent is read **from the store and not from the registry** (`opencharly/opencharly#481`). The registry remains the fallback when the parent is absent locally, and the buildkitd mirrors `ghcr.io/opencharly` to charly's local OCI registry, so any base the store does not hold resolves internally rather than against the upstream registry (`opencharly/charly#937`). + +The image is then materialised into the configured RUN engine's store with a plain `load` — **no `tag` step**: the naming annotation is what applies the reference, and the drive verifies the reference resolves before the build is reported successful. Every later read of it — capability labels, deploy, status, inline merge, `charly clean` — resolves it through that engine. -- **Docker:** `docker buildx build --push` for multi-platform builds -- **Podman:** `podman build --manifest` + `podman manifest push` +## Push Mode -Podman manifest push uses retry with exponential backoff (3 attempts, 5s/10s/20s delays) to handle transient registry errors (e.g., GHCR 500 errors after long builds). +BuildKit's image exporter is the ONE push path — `--output type=image,name=,push=true` — +and the only one that can push the multi-platform image a `--push` build produces. It +replaces the engine-native push drivers — no per-engine push path remains. -Source: `candy/plugin-build/podman.go` (`retryCmd`). +Source: `candy/plugin-build/buildkit.go` (`pushOutput`). ## Layer Merging @@ -447,13 +447,21 @@ For filtered builds (`charly box build `), only the built images are merg ## Engine Configuration ```bash -charly settings set engine.build docker # or podman -charly settings set engine.run docker # or podman +charly settings set engine.run docker # or podman / containerd — which engine RUNS containers ``` +`engine.run` accepts `auto` (detect podman, then docker), `podman`, `docker`, and +`containerd`; `containerd` is opt-in and driven through the `nerdctl` CLI (`nerdctl` is the +binary, not an engine value). See [`/charly-build:settings`](/recipes/build/settings/). + +**There is no `engine.build`**: the image builder is BuildKit, which is not a container +engine — it cannot run a container and therefore cannot be selected like one. `engine.run` +is unaffected and still selectable. + ## Host Prerequisites (First Time) -Building images needs the `charly` binary and a container engine — `docker` or `podman`, whichever `engine.build` names. Nothing else, and no project checkout: `charly box build` runs against the `charly.yml` in the current directory, one started by `charly box new project ` or read from a published project with `--repo /`. +Building images needs the `charly` binary and a container engine — `podman`, `docker`, or `containerd`, whichever `engine.run` names (BuildKit builds the image; the drive bridges it into +that engine's store). Nothing else, and no project checkout: `charly box build` runs against the `charly.yml` in the current directory, one started by `charly box new project ` or read from a published project with `--repo /`. Install `charly` as a distro-native package with your OWN package manager (`pacman -U` / `dnf install` / `apt install` / `apk add`) — the `charly generate-packages` plugin (nFPM) builds the `.pkg.tar.zst`/`.rpm`/`.deb`/`.apk`/`.ipk` published to the per-distro package repos. Obtaining the binary is covered end to end at [Install](https://opencharly.ai/start/install/); it is the one page that describes building from a source tree. @@ -516,9 +524,13 @@ Beyond the YAML-unmarshal symptom above, a stale `charly` binary can produce *sy If the symptom persists after upgrading, the copy on `$PATH` may not be the one you just installed — `which charly` reports which wins. -### Buildah cache-mount corruption (pixi tzdata et al.) +### Cache-mount corruption (pixi tzdata et al.) -Cache mounts declared via `--mount=type=cache,dst=,uid=,gid=` (used for pixi, npm, cargo, rattler, dnf) persist across builds and are **not** evicted by `charly box build --no-cache` (which only suppresses `--cache-from`; see above). A disk-full event mid-download can leave a partial package in the cache, and every subsequent build re-hits the same broken file: +Cache mounts are declared as `--mount=type=cache,id=charly-,dst=,sharing=locked` (pixi, npm, cargo, +rattler and dnf all use one — `buildkit.SharedCacheMount` emits the format). Their **id is stable on purpose**: +that is what makes the cache survive layer-hash churn, which is the entire reason cache mounts exist +(`candy/plugin-build/cache_mount_test.go`). A cache mount is therefore not the build cache: `charly box build +--no-cache` disables the build cache for that build, and the mount's contents persist regardless. A disk-full event mid-download can leave a partial package in the cache, and every subsequent build re-hits the same broken file: ``` error: × failed to link tzdata-2025c-hc9c84f9_1.conda @@ -529,77 +541,27 @@ Options, in order of least to most disruptive: 1. **Just retry the build.** Often the cache clears itself once the partial package finishes downloading or gets superseded by a checksum mismatch. 2. **Bump a content-hash on the builder layer** (e.g. `echo "" >> candy/pixi/charly.yml`) to evict the cache mount associated with that build-stage's hash, then revert the cosmetic change. Same workaround as the scratch-stage cache issue above. -3. **`podman image prune -af` is NOT the right hammer** — see next note. +3. **The engine's `image prune -af` is NOT the right hammer** — use `charly clean`; see next note. -### `podman image prune -af` caveat — removes tagged-but-idle images +### Engine-level `image prune -a -f` caveat — removes tagged-but-idle images -`podman image prune -a -f` will delete tagged images that are **not referenced by a running container**. That means the image you just built via `charly box build` — and haven't started as a container yet — is eligible for prune. Observed sequence: `charly box build` → `podman image prune -af` → `sudo podman load` (manual rootful refresh) fails with `image not known`. Protect the fresh build by starting a container or use `prune` with explicit filters instead. +An engine-level `image prune -a -f` deletes tagged images that are **not referenced by a running container**. That means the image you just built via `charly box build` — and haven't started as a container yet — is eligible for prune. Observed sequence: `charly box build` → an engine-level `image prune -af` → a manual rootful reload fails with `image not known`. Protect the fresh build by starting a container or use `prune` with explicit filters instead. -### `--no-cache` does not invalidate intermediate scratch-stage caches +### Forcing a rebuild of one layer's scratch stage -`charly box build --no-cache ` and `charly box build --cache none ` reliably disable the -cache for the **final image stage**, but in observed behavior they do **not** propagate -to intermediate scratch stages produced by `COPY candy// /` instructions -(`[15/25] STEP 2/2: COPY candy/labwc/ /` style). Editing a single file inside a layer -directory and rebuilding with `--no-cache` may still pull the labwc scratch stage from -cache, leaving the new file content out of the rebuilt image. - -**Workaround:** force a content-hash bump on the layer's `charly.yml`. The simplest is -adding (or removing) a trailing comment line: +A layer's scratch stage is keyed by the layer directory's own content — the cache key is +`(parent-image-SHA, instruction-text, COPY-source-content)` (see "Cache Efficiency" above) — so editing any +file under `candy//` rebuilds that stage and everything downstream of it. To force that rebuild without +changing the file that matters, bump the layer's `charly.yml`: ```bash -echo "" >> candy/labwc/charly.yml # bump content hash -charly box build selkies-desktop # now invalidates the labwc scratch stage -git checkout -- candy/labwc/charly.yml # revert the cosmetic change +echo "" >> candy/labwc/charly.yml # bump the layer's content hash +charly box build selkies-desktop # rebuilds the labwc scratch stage +git diff candy/labwc/charly.yml # the appended line, and nothing else ``` -This was discovered while shipping commit `febb9bd` (labwc autostart race fix): the -edit to `candy/labwc/autostart` did not propagate to the rebuilt image until -`candy/labwc/charly.yml` itself was touched. Two consecutive `--no-cache` rebuilds -produced the same image hash (`502c8012c7a5`) until the charly.yml content changed. - -Symptom: `charly box list tags ` shows a new tag, but `charly shell -c "cat /path/to/changed/file"` -returns the **old** content. - -### Known caveat: stale `:latest` under `--cache-from` - -charly's Containerfile generator currently emits `FROM /:latest` -for parent builder images (not pinned CalVer). When charly invokes -`podman build --cache-from /`, podman resolves those `FROM` -clauses at parse time and will **pull `:latest` from the remote registry**, -silently clobbering any locally-rebuilt `:latest` tag. For images that rebuild -their builder stages in the same invocation this can lead to the later -stages using the **stale registry builder** instead of the freshly-built one. - -**Symptom observed on this project:** `charly update selkies-desktop --build` -fails deep inside selkies' `pixi install && bash build.sh` step with -`error: can't find Rust compiler`. The remote -`ghcr.io/opencharly/fedora-builder:latest` predates the `build-toolchain` -layer adding `cargo` as an RPM, so its pixi env has no rustc — even though -the current local `build-toolchain/charly.yml` lists `cargo`. The build phase -that rebuilds fedora-builder locally *does* run, but parent-stage FROM -resolution happens before that stage exists, and podman pulls the stale -remote image. - -**Workaround until the generator is fixed:** - -```bash -charly box build --cache=none # or equivalently --no-cache at the charly level -``` - -Both `--cache=none` and `--no-cache` short-circuit `cacheArgs()` in -`candy/plugin-build/podman.go` (`driveConfig.cacheArgs`) and do NOT pass `--cache-from` to podman, so -the broken resolution path never fires. `--no-cache` is charly-level only — it -does *not* pass `--no-cache` to podman, it just skips `--cache-from`. - -**Proper fix (not yet implemented):** charly's generator should emit pinned -CalVer tags for builder `FROM` clauses (e.g., -`FROM ghcr.io/opencharly/fedora-builder:2026.105.0128`) or pass -`--pull=never` to podman so local tags aren't resolved from the remote. -Tracked as a follow-up. - -See also: [`/charly-build:generate`](/recipes/build/generate/) for the Containerfile generation path, [`/charly-core:charly-update`](/recipes/core/charly-update/) -for the `--build` flag that also picks up this caveat. +Remove the added line to revert it. Do not reach for `git checkout -- ` to undo it: that restores the +file from the index and discards any other uncommitted work it holds. ## Project directory override diff --git a/src/content/docs/recipes/build/docs.md b/src/content/docs/recipes/build/docs.md index f98c3f19..33f66bac 100644 --- a/src/content/docs/recipes/build/docs.md +++ b/src/content/docs/recipes/build/docs.md @@ -13,6 +13,35 @@ description: "The `charly docs generate` verb — the generator rendering the re **opencharly.ai** from the sources that already exist in this repository. It is served by `candy/plugin-docs`. +## R4 — documentation never routes around a defect + +The rulebook states R4 as a mandate (umbrella `AGENTS.md`, "The ground-truth rules +R1–R10"); this is its operationalization for every page this skill renders and every +hand-authored page beside it. + +- **Fix the product first.** When documented behaviour and the code disagree, establish + which side holds the INTENT, then fix the code BEFORE touching the prose. Editing docs + to match a bug is forbidden; so is editing them to avoid one. +- **The tells of a routed-around defect:** a `git clone`, a `cd`, or a `-C` standing in + for a flag that should work; an "except on…" caveat; a "known limitation" note in place + of a fix. Each describes the defect instead of removing it — and makes it permanent by + making it look intended. +- **Binary-only reader.** Every command a reader is told to run MUST work with nothing but + the `charly` binary installed — no clone, no `cd`, no pre-existing project, no unnamed + companion tool. Write every page for a reader who installed `charly` as a native package + with NO charly checkout on the machine: no `task` target, no `./bin/charly`, no + repo-relative path, no verb supplied by a candy that exists only in this repository. + `--repo /` reads a published project without a clone; + `charly box new project ` starts a local one. +- **The one exception is the INSTALL page**, where the reader obtains charly: it leads + with the native package and scopes the checkout to working ON charly. A + repository-maintenance command is named as maintenance this project performs, never as + a step the reader runs. +- **Anywhere else, needing more than the binary is a PRODUCT DEFECT** — the missing + capability is the bug, and the fix belongs in `charly`. If that fix is genuinely out of + scope, file it with its root cause and let the page say nothing rather than something + false: silence is recoverable, a documented workaround is taught. + ## Placement: a RUNTIME plugin, deliberately `plugin-docs` is **not** listed in `charly/charly.yml`'s `compiled_plugins:`. A dev-time @@ -53,8 +82,8 @@ charly docs generate \ ## What it emits -The generator owns six top-level entries — `index.md`, `grievances.md`, `vision.md`, -`liberation.md`, `reference/` and `recipes/` — and rewrites them wholesale each run, so a deleted +The generator owns seven top-level entries — `index.md`, `grievances.md`, `vision.md`, +`liberation.md`, `soul.md`, `reference/` and `recipes/` — and rewrites them wholesale each run, so a deleted source entity disappears from the site instead of lingering as an orphan page. (The table below expands `reference/` into its sub-trees plus the single file `reference/providers.md`, so it runs to many rows.) What stays hand-authored is the teaching narrative this repository has no @@ -79,6 +108,7 @@ not the page.** | `grievances.md` | `GRIEVANCES.md`, H1 dropped, repo-relative links rewritten for a web reader | | `vision.md` | `VISION.md`, H1 dropped, repo-relative links rewritten for a web reader | | `liberation.md` | `LIBERATION.md`, H1 dropped, repo-relative links rewritten — the deliberate easter-egg page, entered last in the sidebar as "Liberation" | +| `soul.md` | `SOUL.md`, H1 dropped, published verbatim — the identity of charly and every charly agent; self-contained, so there are no repo-relative links to rewrite | | `reference/cli/` | one page per `command:` provider word | | `reference/candy/` | every defined candy: packages, services, and its `plan:` as an acceptance spec | | `reference/box/` | every defined box | @@ -175,7 +205,7 @@ the pinned charly) belongs to the same cutover as the candy edit. `candy/*/charly.yml` — but the generator DOES project candy `skill:` entities directly from its own remote-aware walk (`collectCandySkills`), the complement that keeps a moved candy's recipe references resolving on the site even before the marketplace regeneration -lands (the stale-marketplace gap [`/charly-tools:crabbox`](/recipes/tools/crabbox/) measures — see the tests +lands (the stale-marketplace gap [`/charly-tools:crabbox-cli`](/recipes/tools/crabbox-cli/) measures — see the tests section). The long path still holds: a skill is authored as a `skill:` entity in its owning candy; `charly marketplace generate` projects that into the opencharly/marketplace repo (`/skills//SKILL.md`), and @@ -207,7 +237,7 @@ stylistic preference; the obvious surfaces are actively wrong for the purpose: - A plugin absent from `compiled_plugins:` still loads out-of-process when a plan references its word, so "in the binary" is not the same set as "exists". -So discovery is not the walked trees alone: the catalog is ONE assembly (R3/R5, +So discovery is not the walked trees alone: the catalog is ONE assembly (R2/R5, `assembleCatalog`) unioning three sets — 1. the **walked closure** — remote-ref-aware (`candywalk.CollectEntitiesRemote`): @@ -295,7 +325,7 @@ describe the bad shape rather than spelling it. The host renders every dynamic command word with a generic stub description, intercepts the depth-1 `--help` itself, and plugin-served help arrives in at least three mutually incompatible -formats with no machine-readable dump. Parsing that would be the fragile shim R4 forbids. Every +formats with no machine-readable dump. Parsing that would be the fragile shim R3 forbids. Every fact the site needs is already declared: `plugin.providers`, the per-plugin `schema/*.cue`, and each candy's `description:`. diff --git a/src/content/docs/recipes/build/generate.md b/src/content/docs/recipes/build/generate.md index 40e72fdc..f3c48707 100644 --- a/src/content/docs/recipes/build/generate.md +++ b/src/content/docs/recipes/build/generate.md @@ -17,13 +17,13 @@ Parses `charly.yml`, scans `candy/`, resolves the dependency graph, and emits al The generator is **config-driven** — distro format templates, builder stage templates, and init system fragments all come from the build vocabulary embedded in the `charly` binary at `charly/charly.yml` (three top-level sections: `distro:`, `builder:`, `init:`) — and **declarative per-task** for install logic — each task verb (see [`/charly-image:layer`](/recipes/image/layer/)) has a dedicated emitter that writes the right Containerfile directive. -**Build-engine dispatch**: `charly box generate` does not construct a generator host-side at all (the former `NewGenerator` is DELETED) — it is served by the COMPILED-IN `command:box` plugin (candy/plugin-box, nested under the `box` grammar group), whose generate handler invokes the peer `build:generate` plugin (candy/plugin-build) over the F10 reverse channel (InvokeProvider). The plugin's `runBoxGenerate` → `resolveBuildEngine(..., generateOnly=true)` (candy/plugin-build/resolve.go) runs the ENTIRE build-engine RESOLVE plugin-side — loader, vocab, scan, connect, validate, box resolve, intermediates, order, effective versions, host-fs prep, render-seam-cache prep, render-prep, user-context, and envelope projection — and returns the resolved-project envelope (`spec.BuildResolveReply.ResolvedProject`); `generateOnly` is a PLUGIN-SIDE parameter that returns after the envelope projection (no engine/order drive-model). The host is reached ONLY for thin `buildengine-*` shards (`buildengine-scan-local`, `buildengine-connect-plugins`, and `buildengine-prep` — the render-seam-floor `renderGenCache` populate, which returns an EMPTY map; the host-fs PREP itself moved plugin-side); the envelope + drive-model come from `resolveBuildEngine`'s reply, NOT from `HostBuild("buildengine-prep")` (that host leg takes `spec.ResolvedProjectRequest`, which has no `GenerateOnly` field). The candy renders the Containerfiles itself via `sdk/deploykit.Generator` over the envelope + `HostBuild("render-seam")` (#67 — the host no longer renders); the written paths are returned — no podman (generate builds nothing). `charly box build` uses the SAME resolve pipeline (`runBoxBuild` → `resolveBuildEngine(..., generateOnly=false)`, which additionally computes the drive-model), then the candy additionally OWNS the podman DRIVE (`podman build` / push / inline-merge) IN the candy. See [`/charly-internals:plugin`](/recipes/internals/plugin/) (the `build` provider class + the in-proc reverse channel) and [`/charly-build:build`](/recipes/build/build/). +**Build dispatch**: `charly box generate` does not construct a generator host-side at all (the former `NewGenerator` is DELETED) — it is served by the COMPILED-IN `command:box` plugin (candy/plugin-box, nested under the `box` grammar group), whose generate handler invokes the peer `build:generate` plugin (candy/plugin-build) over the F10 reverse channel (InvokeProvider). The plugin's `runBoxGenerate` → `resolveBuildEngine(..., generateOnly=true)` (candy/plugin-build/resolve.go) runs the ENTIRE build RESOLVE plugin-side — loader, vocab, scan, connect, validate, box resolve, intermediates, order, effective versions, host-fs prep, render-seam-cache prep, render-prep, user-context, and envelope projection — and returns the resolved-project envelope (`spec.BuildResolveReply.ResolvedProject`); `generateOnly` is a PLUGIN-SIDE parameter that returns after the envelope projection (no engine/order drive-model). The host is reached ONLY for thin `buildengine-*` shards (`buildengine-scan-local`, `buildengine-connect-plugins`, and `buildengine-prep` — the render-seam-floor `renderGenCache` populate, which returns an EMPTY map; the host-fs PREP itself moved plugin-side); the envelope + drive-model come from `resolveBuildEngine`'s reply, NOT from `HostBuild("buildengine-prep")` (that host leg takes `spec.ResolvedProjectRequest`, which has no `GenerateOnly` field). The candy renders the Containerfiles itself via `sdk/deploykit.Generator` over the envelope + `HostBuild("render-seam")` (#67 — the host no longer renders); the written paths are returned — no podman (generate builds nothing). `charly box build` uses the SAME resolve pipeline (`runBoxBuild` → `resolveBuildEngine(..., generateOnly=false)`, which additionally computes the drive-model), then the candy additionally OWNS the BuildKit DRIVE (the `buildctl` argv / push / inline-merge) IN the candy. See [`/charly-internals:plugin`](/recipes/internals/plugin/) (the `build` provider class + the in-proc reverse channel) and [`/charly-build:build`](/recipes/build/build/). -**Build-mode emission is `WriteCandySteps` → `EmitTasks`** in `sdk/deploykit` (`deploykit.Generator`, relocated from `charly/generate.go` in #67, driven by `candy/plugin-build` over the resolved-project envelope + `HostBuild("render-seam")`; the `charly/generate.go` top-level `Generate`/`writeCandySteps` orchestrator is DELETED). the pod-overlay deploy path reaches `deploykit.Generator.EmitTasks` DIRECTLY — `charly/tasks.go` holds no shim and cannot hold one (P16 import purity forbids `charly/` importing the sdk); that path stays for the pod-overlay deploy path (the candy `plugin-deploy-pod` constructs `deploykit.OCITarget` + renders via `deploykit.NewRenderGeneratorFromProject`, the per-step fragment over the `HostBuild("step-emit","oci-emit-step")` seam). The render walks each layer's ops directly to write Containerfile text — NOT the `InstallPlan` IR. Build shares the package-cascade, shell-snippet, and localpkg compiler helpers with the IR (`ResolveCascadePackages`, `CompileShellSnippetSteps`, `deploykit.RenderLocalPkgImageInstall` — one source of truth, R3, relocated to sdk/deploykit in W3), but the overall walk is the direct generator, not an `EmitTarget`. The `InstallPlan` IR + `deploykit.OCITarget.Emit` is the DEPLOY-mode path: the candy `plugin-deploy-pod`'s overlay-Containerfile synthesis (rendered in its own code via `deploykit.OCITarget` + `deploykit.NewRenderGeneratorFromProject`) when `add_candy:` is set, plus the external execution targets. (The local/vm/kubernetes/android substrates are external out-of-process plugins via `pluginDeployTarget` (S3b, dispatched through `candy/plugin-fleet`'s `Invoke(OpDeployDispatch)`): `deploy:local` (candy/plugin-deploy-local) and `deploy:vm` (candy/plugin-deploy-vm) DO consume the IR — the plugin walks it via `kit.WalkPlans` over the reverse channel, the vm one over the guest `SSHExecutor` so the walk runs inside the guest — while `deploy:kubernetes` (candy/plugin-kube) does NOT, materializing its Kustomize tree PLUGIN-side (`candy/plugin-kube/materialize.go`'s `materializeKustomize`, reaching `candy/plugin-k8sgen` peer-to-peer over `verb:k8sgen`/`OpEmit`).) See [`/charly-internals:install-plan`](/recipes/internals/install-plan/) for the IR catalog and [`/charly-internals:generate-source`](/recipes/internals/generate-source/) for the Go-level call graph. +**Build-mode emission is `WriteCandySteps` → `EmitTasks`** in `sdk/deploykit` (`deploykit.Generator`, relocated from `charly/generate.go` in #67, driven by `candy/plugin-build` over the resolved-project envelope + `HostBuild("render-seam")`; the `charly/generate.go` top-level `Generate`/`writeCandySteps` orchestrator is DELETED). the pod-overlay deploy path reaches `deploykit.Generator.EmitTasks` DIRECTLY — `charly/tasks.go` holds no shim and cannot hold one (P16 import purity forbids `charly/` importing the sdk); that path stays for the pod-overlay deploy path (the candy `plugin-deploy-pod` constructs `deploykit.OCITarget` + renders via `deploykit.NewRenderGeneratorFromProject`, the per-step fragment over the `HostBuild("step-emit","oci-emit-step")` seam). The render walks each layer's ops directly to write Containerfile text — NOT the `InstallPlan` IR. Build shares the package-cascade, shell-snippet, and localpkg compiler helpers with the IR (`ResolveCascadePackages`, `CompileShellSnippetSteps`, `deploykit.RenderLocalPkgImageInstall` — one source of truth, R2, relocated to sdk/deploykit in W3), but the overall walk is the direct generator, not an `EmitTarget`. The `InstallPlan` IR + `deploykit.OCITarget.Emit` is the DEPLOY-mode path: the candy `plugin-deploy-pod`'s overlay-Containerfile synthesis (rendered in its own code via `deploykit.OCITarget` + `deploykit.NewRenderGeneratorFromProject`) when `add_candy:` is set, plus the external execution targets. (The local/vm/kubernetes/android substrates are external out-of-process plugins via `pluginDeployTarget` (S3b, dispatched through `candy/plugin-fleet`'s `Invoke(OpDeployDispatch)`): `deploy:local` (candy/plugin-deploy-local) and `deploy:vm` (candy/plugin-deploy-vm) DO consume the IR — the plugin walks it via `kit.WalkPlans` over the reverse channel, the vm one over the guest `SSHExecutor` so the walk runs inside the guest — while `deploy:kubernetes` (candy/plugin-kube) does NOT, materializing its Kustomize tree PLUGIN-side (`candy/plugin-kube/materialize.go`'s `materializeKustomize`, reaching `candy/plugin-k8sgen` peer-to-peer over `verb:k8sgen`/`OpEmit`).) See [`/charly-internals:install-plan`](/recipes/internals/install-plan/) for the IR catalog and [`/charly-internals:generate-source`](/recipes/internals/generate-source/) for the Go-level call graph. **Build-time plugin execution**: the build path connects the project's external (out-of-tree) plugin candies via `loadProjectPlugins` (`charly/host_build_buildengine.go:51`, and `charly/check_cmd.go:70` for the check path), so a `run:` plugin verb — and a plugin builder — is registered + dialable DURING image generation, the SAME loader the deploy/check paths use. The host-side `NewGenerator` that used to perform this connect is DELETED. A builtin is already registered via `init()` and needs no connect; only an external one is host-built + connected here. `EmitTasks` then dispatches a `plugin:` verb step PLACEMENT-AGNOSTICALLY above the registry: a builtin `ProvisionActor` renders an act shell `RUN` in-proc (the zero-JSON fast path), while ANY other resolved provider — an external `grpcProvider`, or a builtin emitting a richer fragment — renders via `invokeOpEmitFragmentOpt` → `Invoke(OpEmit)` (in-proc for a builtin, go-plugin gRPC for an external), and the returned `spec.EmitReply.Fragment` is spliced verbatim into the Containerfile (egress-validated with the rest before write). This is operator-authorized build-time execution of host-built plugin code — a project's composed external plugins run as host code during its image builds. Detail → [`/charly-internals:plugin`](/recipes/internals/plugin/) (placement) + [`/charly-internals:generate-source`](/recipes/internals/generate-source/). -**Build-time BUILDER leg (all via `OpResolve`)**: the builder stage emission is `deploykit.Generator.EmitBuilderStages` / `EmitBuilderArtifacts` / `EmitExternalBuilderStages` / `EmitExternalBuilderArtifacts` (`sdk/deploykit/builders_render.go` — relocated from `charly/generate.go` in #67, driven by `candy/plugin-build` over the envelope + `HostBuild("render-seam")`); the shared builder `OpResolve` helper (`renderSeamCaller.resolveBuilderStage`, `sdk/deploykit/render_generator_from_project.go`) drives both legs (the former `charly/generate.go` is DELETED, K-wave 2). These render the four DETECTION-builders' stages (pixi/npm/aur — selected by a candy's `pixi.toml` / `package.json` / `aur:` section; cargo is INLINE) by `Invoke(OpResolve)`ing their plugins — NOT from an in-core `builders.` vocabulary (C10 moved the stage templates into the plugins' `kit.BuilderResolve`). For each detected `(candy, builder)`, `EmitBuilderStages` connects the plugin on-demand (`ensureBuildersConnected`) and calls the shared `resolveBuilderStage` → `prov.Invoke(OpResolve)` with a `spec.BuilderResolveInput` (the host-computed render context — builder ref, stage name, copy src, uid/gid/home, detected manifest/lockfile/build-script, aur packages/options, PRE-RENDERED cache-mount flags) as `op.Params` + a `spec.BuildEnv` as `op.Env`; the returned `spec.BuilderResolveReply.Stage` is written verbatim PRE-main-FROM (cached per `(candy, builder)`), `EmitBuilderArtifacts` writes the cached `CopyArtifacts` + once-per-builder `CopyBinary` POST-main-FROM, and cargo's `InlineFragment` splices in `WriteCandySteps` (deploykit). A candy that instead selects an OUT-OF-TREE builder via its `external_builder: ` field gets its multi-stage emitted by `EmitExternalBuilderStages` / `EmitExternalBuilderArtifacts` (run right after their detection counterparts), sharing the SAME `resolveBuilderStage` but sending a MINIMAL input (candy name only): the word resolves through `providerRegistry.ResolveBuilder` to an EXTERNAL `*grpcProvider`, and `EmitExternalBuilderStages` (the BUILDER-leg analogue of `invokeOpEmitFragmentOpt`) requires a non-empty `Stage`. An empty `Stage` (or empty cargo `InlineFragment`), an unresolvable word, or an Invoke error fails LOUDLY (R4) — never a silently-dropped builder stage. Every half is egress-validated with the rest of the Containerfile before write. Detail → [`/charly-internals:plugin`](/recipes/internals/plugin/) (placement) + [`/charly-internals:generate-source`](/recipes/internals/generate-source/). +**Build-time BUILDER leg (all via `OpResolve`)**: the builder stage emission is `deploykit.Generator.EmitBuilderStages` / `EmitBuilderArtifacts` / `EmitExternalBuilderStages` / `EmitExternalBuilderArtifacts` (`sdk/deploykit/builders_render.go` — relocated from `charly/generate.go` in #67, driven by `candy/plugin-build` over the envelope + `HostBuild("render-seam")`); the shared builder `OpResolve` helper (`renderSeamCaller.resolveBuilderStage`, `sdk/deploykit/render_generator_from_project.go`) drives both legs (the former `charly/generate.go` is DELETED, K-wave 2). These render the four DETECTION-builders' stages (pixi/npm/aur — selected by a candy's `pixi.toml` / `package.json` / `aur:` section; cargo is INLINE) by `Invoke(OpResolve)`ing their plugins — NOT from an in-core `builders.` vocabulary (C10 moved the stage templates into the plugins' `kit.BuilderResolve`). For each detected `(candy, builder)`, `EmitBuilderStages` connects the plugin on-demand (`ensureBuildersConnected`) and calls the shared `resolveBuilderStage` → `prov.Invoke(OpResolve)` with a `spec.BuilderResolveInput` (the host-computed render context — builder ref, stage name, copy src, uid/gid/home, detected manifest/lockfile/build-script, aur packages/options, PRE-RENDERED cache-mount flags) as `op.Params` + a `spec.BuildEnv` as `op.Env`; the returned `spec.BuilderResolveReply.Stage` is written verbatim PRE-main-FROM (cached per `(candy, builder)`), `EmitBuilderArtifacts` writes the cached `CopyArtifacts` + once-per-builder `CopyBinary` POST-main-FROM, and cargo's `InlineFragment` splices in `WriteCandySteps` (deploykit). A candy that instead selects an OUT-OF-TREE builder via its `external_builder: ` field gets its multi-stage emitted by `EmitExternalBuilderStages` / `EmitExternalBuilderArtifacts` (run right after their detection counterparts), sharing the SAME `resolveBuilderStage` but sending a MINIMAL input (candy name only): the word resolves through `providerRegistry.ResolveBuilder` to an EXTERNAL `*grpcProvider`, and `EmitExternalBuilderStages` (the BUILDER-leg analogue of `invokeOpEmitFragmentOpt`) requires a non-empty `Stage`. An empty `Stage` (or empty cargo `InlineFragment`), an unresolvable word, or an Invoke error fails LOUDLY (R3) — never a silently-dropped builder stage. Every half is egress-validated with the rest of the Containerfile before write. Detail → [`/charly-internals:plugin`](/recipes/internals/plugin/) (placement) + [`/charly-internals:generate-source`](/recipes/internals/generate-source/). **Three-phase templates**: the embedded build vocabulary's format (`formats.`) and builder (`builders.`) definitions carry a `phases: { prepare, install, cleanup }.{ container, host }` structure. The generator reads `phases.install.container` as the single source for build-mode package installation. The `host:` cell is consumed only when a `SystemPackagesStep` renders on a host venue (`RenderHostPackageCommand`) — for the external `local:` AND `vm:` deploys the plugin drives the `SystemPackages` host render via the host's `RunHostStep` channel — never by the generator. diff --git a/src/content/docs/recipes/build/migrate.md b/src/content/docs/recipes/build/migrate.md index f1fb6e9e..0967cb33 100644 --- a/src/content/docs/recipes/build/migrate.md +++ b/src/content/docs/recipes/build/migrate.md @@ -1,124 +1,119 @@ --- title: "migrate" -description: "The `charly migrate` command — the single idempotent migration bringing any opencharly config to the latest schema CalVer." +description: "The `charly migrate` command — the single idempotent reshape pass bringing any opencharly config forward to the current schema." --- *Recipe card from the `charly-build` plugin (Commands — runtime CLI verbs).* -# charly migrate — single-command schema migration +# charly migrate — the version-free reshape pass -`charly migrate` is **one idempotent command** that brings any opencharly config — as far back as the supported schema floor — up to the **latest schema CalVer**, and only to the latest. There are no sub-verbs to choose between: `charly migrate` always applies every migration-table step newer than the config, up to HEAD. +`charly migrate` is **one idempotent command** that applies the declarative migration table to an opencharly config on disk. There are no sub-verbs to choose between: it walks the whole table in declaration order, and because every step is idempotent a re-run on an already-current project is a no-op. ```bash -charly migrate # migrate every reachable config to the latest schema CalVer -charly migrate --dry-run # print every change migrate would make; touch nothing +charly migrate # apply the table to every reachable config +charly migrate --dry-run # print every change migrate would make; touch nothing +charly migrate --project-only # skip the per-host overlay ``` The project directory is the current working directory; use the top-level `-C` / `--dir` / `CHARLY_PROJECT_DIR` global to point elsewhere (`main()` chdir's before dispatch). -## CalVer schema versioning +## There is no schema version -The YAML schema version is a **CalVer string** — `version: YYYY.DDD.HHMM`, the same fixed-width scheme as image tags (e.g. a HEAD like `version: 2026.174.1100`). It is CANONICAL fixed-width: a 4-digit year, a 3-digit zero-padded day-of-year, and a 4-digit zero-padded HHMM, so a plain alphanumeric sort of CalVer strings is chronological. `ParseCalVer` is EXTREMELY STRICT — it accepts ONLY that exact form (no `version: 4`, no non-padded `2026.45.830`); a non-canonical value is "not a CalVer", which the load gate treats as older-than-HEAD so it flows into `charly migrate`. `charly migrate` re-stamps every versioned file it brings forward to the canonical HEAD (`universalStamp`); a value that predates the supported floor — or is not a CalVer at all — is unmigratable and refused untouched (see "How it runs"). Every versioned file carries the stamp: +A config carries **no `version:` field**, and the schema is CLOSED: a `charly.yml` still carrying a retired key — `version:` above all — is an unknown field and **fails closed-CUE when it loads**. That load failure is the trigger to run `charly migrate`, and `charly migrate` is what removes the retired keys. -- every project `charly.yml` (splitting into per-kind `vm.yml` / `local.yml` / … siblings is an optional `import:` convenience a project MAY use — never the default; see [`/charly-image:image`](/recipes/image/image/)) -- the per-host `~/.config/charly/charly.yml` +There is no schema HEAD, no floor, and no version comparison anywhere in the engine. `charly migrate` does not decide *whether* a project needs migrating — it reshapes every project it is pointed at, and idempotency is what makes that safe. A universal final step, **`strip-version-stamp`**, removes a leftover top-level or per-entity `version:` key: from the root `charly.yml`, from every other project document, and — unless `--project-only` — from the per-host overlay. -The parsed `ParsedCalVer` type, `ParseCalVer(string) (ParsedCalVer, bool)`, `ParsedCalVer.Less`, and `MustCalVer` live in **`spec/spec/calver_parse.go`** (the parsed type is named `ParsedCalVer` because `spec` already binds `CalVer = string`, the CUE wire scalar); `sdk/kit/calver.go` re-exports them for kit consumers. **CUE owns the HEAD and floor:** `spec/schema/version.cue` defines `#SchemaVersion` (the HEAD CalVer every file is stamped to) and `#SchemaFloor` (the oldest version migratable FROM). The spec repo's `charly task cue-gen` emits `spec/spec/version_gen.go` with the `SchemaVersion` / `SchemaFloor` string consts; `spec.LatestSchemaCalVer()` and `spec.SchemaFloorCalVer()` parse them, re-exported as `kit.LatestSchemaVersion()` / `kit.SchemaFloor()` (what `candy/plugin-migrate/engine.go` calls) and shimmed in core by `charly/version.go`'s `LatestSchemaVersion()`. There is NO hand-maintained Go HEAD literal — the load-time gate requires `kit.LatestSchemaVersion()`, and `charly migrate` refuses anything below `kit.SchemaFloor()`. +### Per-merge release git tags -### Per-merge release git tags (decoupled from `version:`) - -Every landing of ANY repo carries a **fresh** annotated release git tag `v` — ONE per merge, so a repo accumulates MULTIPLE `v…` tags over time. The tag is **generated and applied by the fresh `pr-validator` at MERGE**, from the current UTC merge time — never by the author at push time (author-time stamps collide and mis-order across concurrent out-of-sync PRs; see [`/charly-internals:git-workflow`](/recipes/internals/git-workflow/) "CalVer"). This is **decoupled** from the `charly.yml` `version:` field: `version:` is the SCHEMA version (bumped only when a cutover raises `#SchemaVersion`), whereas the tag marks the merge moment. Tag EVERY landing — including one at an unchanged `version:` (a content change, submodule extraction, image drop). Tags are **immutable**: only ever ADD new ones, never move or force-push an existing tag (so the load-time gate never sees a "newer than supported" CalVer from a re-tag). Every component is fixed-width zero-padded — 4-digit year, 3-digit day-of-year, 4-digit HHMM (`v2026.064.1937`, `v2026.142.1640`) — so tags sort chronologically under a plain alphanumeric sort; the evaluator computes `v$(date -u +%Y.%j.%H%M)` at merge (`%j` and `%H%M` are already zero-padded). Each repo — the superproject, every `box/` submodule, `plugins`, and `docs` — is tagged independently on its merged `main` HEAD (the tag marks the merge, so a repo needs no `charly.yml` to be tagged); a producer repo's PR merges + tags BEFORE its consumer's pointer bump. The SOLE exception is `sdk`, which tags under its Go-module `v0..` scheme (a semver requirement, not an exemption). See the project rulebook "Post-Execution Policies" (`AGENTS.md`) and [`/charly-internals:git-workflow`](/recipes/internals/git-workflow/). +Every landing of ANY repo carries a **fresh** annotated release git tag `v` — ONE per merge, so a repo accumulates MULTIPLE `v…` tags over time. The tag is **generated and applied by the fresh `pr-validator` at MERGE**, from the current UTC merge time — never by the author at push time (author-time stamps collide and mis-order across concurrent out-of-sync PRs; see [`/charly-internals:git-workflow`](/recipes/internals/git-workflow/) "CalVer"). Tag EVERY landing — a content change, a submodule extraction, an image drop. Tags are **immutable**: only ever ADD new ones, never move or force-push an existing tag. Every component is fixed-width zero-padded — 4-digit year, 3-digit day-of-year, 4-digit HHMM (`v2026.064.1937`, `v2026.142.1640`) — so tags sort chronologically under a plain alphanumeric sort; the evaluator computes `v$(date -u +%Y.%j.%H%M)` at merge (`%j` and `%H%M` are already zero-padded). Each repo — the superproject, every `box/` submodule, `plugins`, and `docs` — is tagged independently on its merged `main` HEAD (the tag marks the merge, so a repo needs no `charly.yml` to be tagged); a producer repo's PR merges + tags BEFORE its consumer's pointer bump. The SOLE exception is `sdk`, which tags under its Go-module `v0..` scheme (a semver requirement, not an exemption). The tag is independent of the migration table: the tag marks the MERGE, a migration entry marks the SCHEMA. See the project rulebook "Commit, push, land" (`AGENTS.md`) and [`/charly-internals:git-workflow`](/recipes/internals/git-workflow/). ## The migration table (declarative data) -The migration steps are **declarative DATA, not code**: an ordered list embedded from **`candy/plugin-migrate/migrations.cue`** (the TABLE lives in candy/plugin-migrate), validated at process start against the `#Migration` schema (`candy/plugin-migrate/schema/migration.cue` — a plugin-only validation schema that lives in the plugin per the kernel/plugin boundary law, concatenated at compile with the SDK's `version.cue` for `#CanonCalVer`). Each step is stamped with the CalVer of the date it landed and listed **chronologically** — the order the cutovers were authored in, which is the only correct replay order for an arbitrarily-old config. `charly migrate` applies every step whose version is newer than the config's stamp; each step is idempotent, so applying the whole set is safe (an already-current file is a no-op). +The migration steps are **declarative DATA, not code**: an ordered list embedded from **`candy/plugin-migrate/migrations.cue`**, validated at process start against the `#Migration` schema in **`candy/plugin-migrate/schema/migration.cue`**. That schema is `@go(-)` — it generates no Go type — and lives IN the plugin: it is validation-only, read by this engine and by no charly core code, so per the kernel/plugin boundary law it stays out of the SDK contract. The list is ORDERED — the order the cutovers were authored in, which is the only correct replay order — and `charly migrate` applies every step, in order. -A step is a small record: +A step is: ```cue -{version: "YYYY.DDD.HHMM", name: "", touches_host?: bool, ops?: [...#Op] | apply?: ""} +{name: string, touches_host?: bool, ops?: [...#MigrationOp] | apply?: string} ``` -`touches_host: true` flags a step that mutates per-host state (`~/.config/charly`, quadlets, `.secrets`) — those steps are skipped by remote-cache auto-migration (see below). A step carries EITHER an `ops:` list OR a single `apply:` hook, never both. +`name` labels the step for `--dry-run` output and progress. `touches_host: true` flags a step that mutates per-host state (`~/.config/charly`, quadlets, `.secrets`) — those steps are skipped by remote-cache auto-migration (see below). A step carries EITHER an `ops:` list OR a single `apply:` hook, never both; that "exactly one of the two" rule is enforced in Go, since CUE's field-presence comparison is too fragile for the gate. **The op vocabulary — each op is DATA, zero Go.** Four generic key transforms cover the common cutovers, all applied by ONE comment-preserving yaml.v3 interpreter (the op-walker in `candy/plugin-migrate/engine.go`): | Op | Shape | Effect | |---|---|---| -| `rename_key` | `{from, to, scope, under_kind?}` | rename a mapping key `from` → `to` | -| `delete_key` | `{key, scope, under_kind?}` | drop a mapping key | +| `rename_key` | `{from, to, scope?="any", under_kind?}` | rename a mapping key `from` → `to` | +| `delete_key` | `{key, scope?="any", under_kind?}` | drop a mapping key | | `remap_scalar` | `{key, from, to, under_kind?}` | rewrite a scalar VALUE `from` → `to` at `key` | | `move_key` | `{key, from_parent, to_parent, under_kind?}` | relocate a key between parent mappings | -`scope` is `root` (top-level keys only) or `any` (every depth); `under_kind` further scopes an op to entities of a named kind (e.g. only `vm:` nodes). None of the four ops needs a line of new Go — the walker interprets the data. +`scope` is `root` (top-level keys only) or `any` (every depth, the default; only `rename_key` / `delete_key` take it); `under_kind` further scopes an op to entities of a named kind (e.g. only `vm:` nodes). None of the four ops needs a line of new Go — the walker interprets the data. -**The `apply:` hook — the deliberate, visible EXCEPTION.** A structural reshape the four ops can't express (splitting one node into siblings, folding a list into a map) sets `apply: ""` naming a Go function registered in the core `goHooks` map. This is the ONLY path that runs bespoke Go for a migration, and it is intentionally conspicuous: a table entry carrying `apply:` is the signal that this step needed real code. +**The `apply:` hook — the deliberate, visible EXCEPTION.** A structural reshape the four ops can't express (splitting one node into siblings, folding a list into a map) sets `apply: ""` naming a Go function registered in the plugin's `goHooks` map. This is the ONLY path that runs bespoke Go for a migration, and it is intentionally conspicuous: a table entry carrying `apply:` is the signal that this step needed real code. -**CUE has NO transformation capability.** CUE is monotone unification — it can validate, default, require, and close, and nothing more; it CANNOT rewrite a config. So CUE owns exactly two things here: the version pins (`#SchemaVersion` / `#SchemaFloor`) and the SHAPE of the declarative table (`#Migration` validates every entry at startup). Every actual transform is Go — the generic op-walker, or an `apply:` hook. Do not expect CUE to migrate anything. +**CUE has NO transformation capability.** CUE is monotone unification — it can validate, default, require, and close, and nothing more; it CANNOT rewrite a config. So CUE owns exactly ONE thing here: the SHAPE of the declarative table (`#Migration` validates every entry at startup). Every actual transform is Go — the generic op-walker, or an `apply:` hook. Do not expect CUE to migrate anything. -**The current table.** The floor sits at the migration-baseline reset version (`2026.174.1100`), so any config predating it is unmigratable — the accepted clean-slate consequence. The table carries ONE entry, the schema-compaction cutover: +**The current table.** `candy/plugin-migrate/migrations.cue` is the authoritative list — read it there; each entry carries its own comment explaining the reshape and why the declarative ops could not express it. The entries run in the order the cutovers were authored in (the only correct replay order) and are: -| version | name | touches_host | apply | +| name | declarative ops | `apply:` hook | `touches_host` | |---|---|---|---| -| `2026.186.2323` | `compact-node-form` | `true` | `compactNodeForm` (goHook) | - -The `compactNodeForm` hook is a structural reshape the four generic ops can't express: it folds the former named data/step child-node grammar into the compact node form — collections (`env`/`volume`/`port`/`service`/…) inline in the kind value, steps as the ordered `plan:` list (a meaningful former step-node name becomes the step's `id:`), deploy data (`add_candy`/`install_opts`/`vm_state`/…) inline in the substrate node, and every authored step `plugin:`+`plugin_input:` pair rewritten to the `: ` verb sugar. Member children (sub-entities under a deployable kind) stay named children. Remote layer caches auto-migrate on fetch (see "Remote-cache auto-migration"). The same cutover DELETED the Calamares `target:` / `module:` / `package-group:` kinds (and their plugins) — they had zero core readers; they return as their own cutover when the installer feature is real. +| `compact-node-form` | — | `compactNodeForm` | `true` | +| `strip-candy-libvirt-field` | — | `stripCandyLibvirtField` | — | +| `strip-deploy-shell-overlay` | — | `stripDeployShellOverlay` | `true` | +| `k8s-to-kubernetes` | 2 × `rename_key` | — | — | +| `remove-candy-localpkg` | `delete_key` | — | — | +| `install-template-to-phases` | — | `installTemplateToPhases` | — | +| `reshape-graphics-gl` | — | `reshapeGraphicsGL` | — | +| `record-field-to-instrument` | — | `recordFieldToInstrument` | — | +| `unroll-group-deploy` | — | `unrollGroupDeploy` | — | +| `deploy-cpu-spelling` | — | `reshapeDeployCPU` | `true` | +| `rekey-legacy-vm-overlay` | — | `rekeyLegacyVMOverlay` | `true` | + +`compact-node-form` is the schema-compaction cutover: it folds the former named data/step child-node grammar into the compact node form — collections (`env`/`volume`/`port`/`service`/…) inline in the kind value, steps as the ordered `plan:` list (a meaningful former step-node name becomes the step's `id:`), deploy data (`add_candy`/`install_opts`/`vm_state`/…) inline in the substrate node, and every authored step `plugin:`+`plugin_input:` pair rewritten to the `: ` verb sugar. Member children (sub-entities under a deployable kind) stay named children. Remote layer caches auto-migrate on fetch (see "Remote-cache auto-migration"). + +On top of the table the engine ALWAYS applies one further step, **`strip-version-stamp`** — it is not a table entry, so it needs no author (see "There is no schema version"). ## How it runs -The migration engine is a **COMPILED-IN command plugin** (`candy/plugin-migrate`, `command:migrate`) — the engine, table, and CLI live in the plugin, out of charly core; core keeps only the below-floor/behind-head load-gate HINTS (the `Run: charly migrate` messages). `charly migrate` dispatches to the compiled-in `command:migrate` provider; the remote-cache auto-migration (`refs.go`) resolves the same provider and Invokes it (`OpRun --project-only`) in-proc. +The migration engine is a **COMPILED-IN command plugin** (`candy/plugin-migrate`, `command:migrate`) — the engine, table, and CLI live in the plugin, out of charly core; core keeps only the load-gate HINT (the `Run: charly migrate` message a closed-schema load failure carries). `charly migrate` dispatches to the compiled-in `command:migrate` provider; the remote-cache auto-migration resolves the same provider and Invokes it (`OpRun --project-only`) in-proc. Compiled-in placement is load-bearing: migrate is whole-project file-based and must run when the config is exactly what cannot load, so it can never be discovered out-of-process. -`runMigrations` (`candy/plugin-migrate/engine.go`) is **floor-gated**, comparing the config's `version:` stamp against `kit.SchemaFloor()` and `kit.LatestSchemaVersion()`: +`runMigrations` (`candy/plugin-migrate/engine.go`) walks the table in declaration order: -- **at HEAD** → no-op; prints `nothing to migrate (already at schema )`. -- **below the floor, or not a CalVer at all** → **unmigratable**: refused with an actionable error (`predates the supported floor … re-author against the current schema`) and **NO filesystem change**. Configs from before the baseline reset fall here. A stranded **per-host** config (`~/.config/charly/charly.yml`) at a below-floor version is doubly stuck: it can neither migrate NOR be WRITTEN — `saveDeployState` refuses to overwrite a config it cannot load (`refusing to overwrite … the existing per-host config fails to load … fix it (or remove it to regenerate) first`), so every pod/vm/local deploy — including a `charly check run ` at its `config`/`deploy-add` step — fails until it is reset. Since it holds only regenerable deploy STATE (no authored intent), back it up and reset it to a bare `version: ` stub (or remove it); the state rebuilds on the next deploy. -- **in `[floor, HEAD)`** → apply every migration-table step newer than the stamp (in order, via the op-walker / `apply:` hooks), then re-stamp every versioned file to HEAD (`universalStamp`). +- **no `charly.yml` in the directory** → reports `no in — nothing to migrate`. +- **otherwise** → applies every step (the generic op-walker or an `apply:` hook) in order, then the universal `strip-version-stamp`, and reports `nothing to migrate` when nothing changed. -Per-step backups follow the established `.bak.` convention. +**Backups are gitignored, never adjacent.** A step's per-file backup goes to `/.charly/backups/.`, and the tree is kept out of git by a `/.charly/.gitignore` containing a bare `*` (created on demand). A migration therefore never leaves `*.bak` files beside the originals and never dirties a working tree. ### Remote-cache auto-migration (project-only) -`sdk/loaderkit`'s refs seams (the former `charly/refs.go` is DELETED, K-wave 2) auto-run the remote-cache migration (`migrateCacheViaPeer`, `sdk/loaderkit/refs_seams_executor.go`) on a freshly-cloned remote-repo cache so external repos pull through at the latest schema. It skips every `touches_host` step and leaves the host-deploy path empty, so a remote fetch **never mutates the user's per-host state** — even the final re-stamp touches only the cache's project files. A remote whose config predates the floor **fails the fetch** with the same predates-floor error (an old remote is unmigratable — the accepted clean-slate consequence). - -## Load-time gate - -The load-time gate `GateSchemaVersion` (`sdk/loaderkit/load_unified.go` — the former `charly/unified.go` is DELETED, K-wave 2) is unchanged: `LoadUnified` parses the merged `version:` and rejects anything below HEAD (or absent, or non-CalVer): - -``` -charly.yml: schema 2026.186.2323 is required (found "4"). Run: charly migrate -``` - -A non-CalVer value (a legacy integer, empty, or garbage) parses as "older than every real CalVer", so a stale config trips the gate with a uniform `Run: charly migrate` hint — the forward-compat trigger that makes a future migration fire. Whether that migrate then SUCCEEDS depends on the floor: a config in `[floor, HEAD)` migrates; one below the floor is refused (see "How it runs"). Residual-key checks (e.g. `kind: deployment`, `target: host`, `secret_backend: kdbx`) remain as defense-in-depth, but every remediation hint points uniformly at bare `charly migrate`. +`sdk/loaderkit`'s refs seams auto-run the remote-cache migration on a freshly-cloned remote-repo cache so external repos pull through at the current schema. It skips every `touches_host` step and leaves the host-deploy path empty, so a remote fetch **never mutates the user's per-host state** — even the final `strip-version-stamp` touches only the cache's project files. -## Adding a future cutover +## Load-time enforcement: closedness -**From here to the end of "Standing rule" is maintenance the charly project performs on its OWN schema**, run from checkouts of the spec repo and the superproject. Nothing in these two sections is a step an charly user runs — the user-facing surface of a schema cutover is the single command `charly migrate`, unchanged. They are recorded here because this skill owns the migration table; the commands are named as this project's maintenance, not as instructions for the reader. +There is no version gate to trip. Compatibility has ONE mechanism — the **CLOSED CUE schema**: a retired key (a leftover `version:`, `kind: deployment`, `target: host`, `secret_backend: kdbx`, …) is an unknown field, and the load fails with a hard `field not allowed` diagnostic naming it. The same failure carries the `Run: charly migrate` hint, and `charly migrate` is what strips the key (see "There is no schema version"). Residual-key checks remain as defense-in-depth alongside the closedness gate. -The payoff of the declarative engine: the common case needs **zero new Go** — but since the schema lives in the spec module, a schema-version bump is **CROSS-REPO**: the spec repo lands + tags FIRST, then the superproject consumes the published module at the bumped require version (no submodule, no local checkout). +## Adding a future migration -1. Bump `#SchemaVersion` in **`spec/schema/version.cue`** to the new CalVer. -2. Run the spec repo's **`charly task cue-gen`** to regenerate `spec/spec/version_gen.go`. -3. **Land + tag the spec repo** (its Go-module tag scheme `v0..` — see [`/charly-internals:git-workflow`](/recipes/internals/git-workflow/)). -4. In the superproject: bump the `github.com/opencharly/spec` require version in `charly/go.mod` and every lockstep go.mod (the canonical-go.mod gate in `tools/gomod-canonical` asserts one shared pin; `charly task mods-tidy` re-syncs the go.sum files), append the matching entry to **`candy/plugin-migrate/migrations.cue`** (the TABLE lives in candy/plugin-migrate) with the same `version:` — **strictly greater** than the previous HEAD, expressing the transform as an `ops:` list (`rename_key` / `delete_key` / `remap_scalar` / `move_key`) — and run the superproject's own gates. +**This section is maintenance the charly project performs on its OWN schema.** Nothing in it is a step a charly user runs — the user-facing surface is the single command `charly migrate`, unchanged. It is recorded here because this skill owns the migration table. -That is the whole common case — no bespoke migrator type, no registry function, no per-migrator Go file. Only a **structural reshape the four ops can't express** additionally registers ONE `goHooks` entry (a Go function named by the step's `apply:` field). Update the HEAD-CalVer fixtures + the repo's own versioned YAML in the same change. +The declarative engine's payoff: the common case needs **zero new Go**. Append ONE entry to **`candy/plugin-migrate/migrations.cue`**, expressing the transform as an `ops:` list (`rename_key` / `delete_key` / `remap_scalar` / `move_key`). Because the table is order-driven and version-free, there is no CalVer to bump and no cross-repo sequence — the entry IS the whole change, and it is idempotent by contract. A **structural reshape the four ops can't express** additionally registers ONE `goHooks` entry (a Go function named by the step's `apply:` field). Ship the entry in the same change that alters the schema, and update the repo's own versioned YAML. The operator command never changes — it stays `charly migrate`. -**The validated schema is CUE-single-source.** The `@go()`-annotated `spec/schema/*.cue` defs are the sole source for the Go param structs (generated into `spec/spec` by `charly task cue-gen`); a wire-key change is a CUE edit first, then `charly task cue-gen`, then — only if it breaks existing on-disk configs — a migration-table entry here (plus the `#SchemaVersion` bump above). A pure codegen refactor that leaves every authored wire key untouched needs NO migration entry and NO `version:` bump. See the [`/charly-internals:go`](/recipes/internals/go/) recipe "How to change the charly.yml schema (CUE is the single source of truth)". +**The validated schema is CUE-single-source.** The `@go()`-annotated `spec/schema/*.cue` defs are the sole source for the Go param structs (generated into `spec/spec` by `charly task cue-gen`); a wire-key change is a CUE edit first, then `charly task cue-gen`, then — only if it breaks existing on-disk configs — a migration-table entry here. A pure codegen refactor that leaves every authored wire key untouched needs NO migration entry. See the [`/charly-internals:go`](/recipes/internals/go/) recipe "How to change the charly.yml schema (CUE is the single source of truth)". -### Standing rule: a schema/format change bumps `version:` AND mints a git tag +### Standing rule: a schema/format change ships its migration entry AND mints a git tag -Any change to the YAML schema or composition format (a key rename, a deleted key, a new key shape) is a hard-cutover that MUST: +Any change to the YAML schema or composition format (a key rename, a deleted key, a new key shape) is a hard-cutover that MUST ship a matching migration-table entry in the same change, so `charly migrate` can carry an existing config across. A migration entry WITHOUT the matching schema change (or vice-versa) is forbidden. -1. **Bump the `#SchemaVersion` CalVer** — edit `spec/schema/version.cue`, run `charly task cue-gen`, and append the matching entry to `candy/plugin-migrate/migrations.cue` (the cross-repo sequence under "Adding a future cutover"), then land the sdk PR (the fresh `pr-validator` merges + tags it), and in the superproject adopt the tagged sdk release as the new shared pinned require (the `charly task mods-tidy` sweep + canonical-go.mod gate keep every module on the one pin). The load-time gate then rejects any not-yet-migrated config with a `Run: charly migrate` hint, so every reader sees the new format. Raising `#SchemaVersion` WITHOUT the migration-table entry (or vice-versa) is forbidden — and `version:` is NEVER set above `LatestSchemaVersion()` (newer configs hard-fail at load). **The author writes a PLACEHOLDER `#SchemaVersion` / `version:` / `migrations.cue` CalVer; the fresh `pr-validator` FINALIZES it at MERGE** — re-stamping it strictly ABOVE the current `main` HEAD's schema version (two concurrent schema PRs would otherwise collide or mis-order; see [`/charly-internals:git-workflow`](/recipes/internals/git-workflow/) "CalVer"). -2. **The fresh `pr-validator` mints the per-merge git tag** at merge — `v` from the merge moment (see "Per-merge release git tags" above). The tag and the `version:` bump are decoupled: the tag marks the merge, `version:` marks the schema. A schema cutover happens to do BOTH at once, but a content-only landing (no schema change) still mints a tag at an unchanged `version:`. +Separately, **the fresh `pr-validator` mints the per-merge git tag** — `v` computed from the MERGE moment (see "Per-merge release git tags" above). The tag marks the MERGE and every landing gets one, schema change or not; it is independent of the migration table, which marks the SCHEMA. ## Idempotency -Running `charly migrate` twice is a no-op: after the first run the config is stamped at HEAD, so the second run hits the floor gate's "at HEAD → nothing to migrate" branch. Every op is itself idempotent (a rename with no matching key does nothing). The migration table's own invariants — every entry a valid `#Migration`, versions strictly ascending and unique, HEAD == `#SchemaVersion` — are validated against CUE at process start. +Running `charly migrate` twice is a no-op: every step's old shape is absent the second time, so each does nothing. An op with no matching key, a `strip-version-stamp` on a file with no stamp, and an `apply:` hook on an already-reshaped node are all inert. The table's own invariant — every entry a valid `#Migration` — is validated against CUE at process start. ## See Also @@ -127,5 +122,5 @@ Running `charly migrate` twice is a no-op: after the first run the config is sta - [`/charly-core:deploy`](/recipes/core/deploy/) — the deploy entries a migration may rewrite - [`/charly-local:local-spec`](/recipes/local/local-spec/) — `kind: local` templates - [`/charly-build:secrets`](/recipes/build/secrets/), [`/charly-build:settings`](/recipes/build/settings/) — the credential schema -- [`/charly-internals:go`](/recipes/internals/go/) — loader internals (`LoadUnified`, `ParseCalVer`, the `gateSchemaVersion` load gate, the `candy/plugin-migrate/engine.go` op-walker) +- [`/charly-internals:go`](/recipes/internals/go/) — loader internals (`LoadUnified` and the closedness gate; the `candy/plugin-migrate/engine.go` op-walker) - [`/charly-internals:cutover-policy`](/recipes/internals/cutover-policy/) — why hard-cutover + a single idempotent `charly migrate` is the required shape diff --git a/src/content/docs/recipes/build/pull.md b/src/content/docs/recipes/build/pull.md index 519ed24e..2133b896 100644 --- a/src/content/docs/recipes/build/pull.md +++ b/src/content/docs/recipes/build/pull.md @@ -31,7 +31,6 @@ recommendation pointing here. | Pull fully-qualified ref | `charly box pull ghcr.io/opencharly/jupyter:2026.108.56` | Pulls as-is, works from anywhere | | Pull remote project ref | `charly box pull @github.com/org/repo/image:v1` | Downloads repo, reads its `charly.yml`, pulls registry ref | | Override tag (short name) | `charly box pull jupyter --tag 2026.108.1` | Pull a specific CalVer tag | -| Override platform | `charly box pull jupyter --platform linux/arm64` | Pull a specific platform | ## Three Input Forms @@ -58,8 +57,9 @@ charly box inspect jupyter --format tag # ghcr.io/opencharly/jupyter:2026. charly box pull ghcr.io/opencharly/jupyter:2026.108.56 ``` -Runs `podman pull ` directly. Useful from any directory, including -`/tmp`. Supports any registry that the underlying engine (`podman`/`docker`) +Runs the configured engine's `pull` directly (`podman pull` / `docker pull`, or +`nerdctl pull` when `engine.run: containerd`). Useful from any directory, including +`/tmp`. Supports any registry that the underlying engine (`podman`/`docker`/`containerd`) can authenticate against. ### 3. Remote project ref (`@github.com/...`) @@ -82,7 +82,8 @@ refs are accepted in `charly`. Deploy-mode commands (`charly shell`, `charly sta not restart, reconfigure, or touch any running container. - **Prints the resolved ref** on success. One line to stderr. - **Respects engine selection.** Uses the run engine from - `ResolveRuntime` (`podman` by default; `docker` if configured). + `ResolveRuntime` (`engine.run`, default `auto`: detects podman, then docker; + `containerd` is opt-in and never auto-detected). ## Interaction with `ErrImageNotLocal` (the sentinel pattern) @@ -96,8 +97,8 @@ per-call-site code: from local storage. 2. `dispatchBuildEnsure` (`charly/dispatch_build_ensure.go`) dispatches to candy/plugin-build's `build:ensure` word (`candy/plugin-build/ensure.go`), - which returns the same sentinel when the image is absent from both run and - build engines. + which returns the same sentinel when the image is absent from the RUN engine's + store (the one every later read resolves it through). 3. `FormatCLIError` in `charly/image.go` unwraps the sentinel at the top-level error boundary in `main()` and renders: @@ -130,8 +131,6 @@ Use `charly box pull ` once to seed local storage; use `charly update - `--tag ` (default: `latest`) — Image tag when resolving a short name. Ignored for fully-qualified or remote-project refs (the tag is embedded in the ref). -- `--platform ` — Target platform (default: host). Passed to - `podman pull --platform`. ## Typical Workflows diff --git a/src/content/docs/recipes/build/settings.md b/src/content/docs/recipes/build/settings.md index d36384f8..2fd189d0 100644 --- a/src/content/docs/recipes/build/settings.md +++ b/src/content/docs/recipes/build/settings.md @@ -30,9 +30,9 @@ Manage charly's runtime configuration stored in `~/.config/charly/config.yml`. C | Key | Default | Env Var | Description | |-----|---------|---------|-------------| -| `engine.build` | `auto` | `CHARLY_BUILD_ENGINE` | Build engine (docker/podman) | -| `engine.run` | `auto` | `CHARLY_RUN_ENGINE` | Run engine (docker/podman) | -| `run_mode` | `auto` | `CHARLY_RUN_MODE` | Deployment mode (quadlet/direct) | +| `engine.run` | `auto` | `CHARLY_RUN_ENGINE` | Run engine (`auto`, `podman`, `docker`, `containerd`). `auto` detects podman, then docker; `containerd` is opt-in and never auto-detected. `nerdctl` is containerd's CLI binary, not an engine value. | +| `containerd.namespace` | `default` | — | containerd namespace for the deploy's images and containers. A per-deploy `namespace:` overrides this setting; this setting overrides the engine capability's built-in default. There is no env mirror. | +| `run_mode` | `auto` | `CHARLY_RUN_MODE` | Deployment mode (`auto`, `quadlet`, `systemd-unit`, `direct`). `quadlet` is podman's, `systemd-unit` wraps the engine CLI (containerd's `nerdctl`), `direct` is an ephemeral argv launch. | | `bind_address` | `127.0.0.1` | `CHARLY_BIND_ADDRESS` | Default bind address for ports | | `encrypted_storage_path` | `~/.local/share/charly/encrypted` | `CHARLY_ENCRYPTED_STORAGE_PATH` | Base path for gocryptfs volumes | | `volumes_path` | `~/.local/share/charly/volumes` | `CHARLY_VOLUMES_PATH` | Base path for bind-mounted volumes | @@ -46,13 +46,35 @@ Manage charly's runtime configuration stored in `~/.config/charly/config.yml`. C ### Engine Selection +`engine.run` selects the engine that RUNS containers; the image builder is BuildKit and is +not selectable. The closed engine vocabulary is `podman`, `docker`, and `containerd`. +`auto` (the default) resolves it by detecting podman, then docker; `containerd` is opt-in +and is never auto-detected. `containerd` is driven through the `nerdctl` CLI, so `nerdctl` +is the binary, not an engine value. + ```bash -# Switch to podman for both build and run -charly settings set engine.build podman +# Which engine RUNS containers (the builder is BuildKit, and is not selectable) charly settings set engine.run podman +# containerd: opt-in, never auto-detected (driven through the nerdctl CLI) +charly settings set engine.run containerd + # Check current engine -charly settings get engine.build +charly settings get engine.run +``` + +### Containerd Namespace + +The namespace a containerd deploy's images and containers live in. The default is the +engine capability's namespace fact (`default`). A deploy's own `namespace:` still wins +over this setting; this setting wins over the capability default. + +```bash +# Pin the namespace for this host (default: the engine capability's namespace) +charly settings set containerd.namespace k8s.io + +# Which namespace is in effect +charly settings get containerd.namespace ``` ### Storage Paths @@ -86,6 +108,10 @@ charly secrets migrate-secrets --dry-run Settings resolve in this order: environment variable > config.yml > default value. +`containerd.namespace` is the one setting with no environment mirror: its levels are a +per-deploy `namespace:` (highest) > `config.yml` (`containerd.namespace`) > the engine +capability's built-in namespace default. + ## Cross-References - [`/charly-core:charly-config`](/recipes/core/charly-config/) -- deployment configuration (uses settings) diff --git a/src/content/docs/recipes/check/check/beds-and-r10.md b/src/content/docs/recipes/check/check/beds-and-r10.md index e58b97ef..da4c919f 100644 --- a/src/content/docs/recipes/check/check/beds-and-r10.md +++ b/src/content/docs/recipes/check/check/beds-and-r10.md @@ -146,7 +146,7 @@ canary", "to fit session bounds", "shorten this run", "skip the heavy leg", "faster iteration cycle" — are confessions, not defences. The `iterate:` block's `plateau_iteration` and the AI's `progress_no_improvement_timeout` together define the AI's recovery budget per phase; narrow neither without explicit -authorization. Speed levers that do not shrink scope (e.g. `--podman-jobs`, `--jobs`) need +authorization. Speed levers that do not shrink scope (e.g. `--stage-parallelism`, `--jobs`) need no authorization — the lever catalog is [`/charly-internals:agents`](/recipes/internals/agents/) "Speed levers". diff --git a/src/content/docs/recipes/internals/agents.md b/src/content/docs/recipes/internals/agents.md index 4ee6b36b..b76fe7b7 100644 --- a/src/content/docs/recipes/internals/agents.md +++ b/src/content/docs/recipes/internals/agents.md @@ -127,7 +127,7 @@ A plugin is not one artifact and does not resolve from one ref. Its **code** may **The rule.** Before assuming a plugin works, verify the pin its BACKEND resolves from — not only that the plugin file loaded. A tool failure of the form **"X not found — sync the pin"** is a **PIN-LAG**, not a plugin defect: the tool is present in the plugin, but the script/submodule it dispatches to is older than the commit that added it. Do NOT debug the plugin's code for it, and do NOT reimplement or hand-copy the missing script into the pinned checkout (that is an R3 workaround and a generated-copy edit). -**Remedy.** Identify the submodule / ref the backend script resolves from; compare the pinned gitlink (or `@github` ref) against the commit that merged the script; then advance the pin the sanctioned way — `charly task sync` for an umbrella gitlink, or a refs-list bump + `charly marketplace generate` for the marketplace corpus — never a hand-pin and never a checkout into the submodule. +**Remedy.** Identify the submodule / ref the backend script resolves from; compare the pinned gitlink (or `@github` ref) against the commit that merged the script; then advance the pin the sanctioned way — `charly task sync` for an umbrella gitlink (a wave: it advances EVERY pin the umbrella declares), or a refs-list bump + `charly marketplace generate` for the marketplace corpus — never a hand-pin and never a checkout into the submodule. **Concrete case.** In one session `coord_comment` failed while `coord_watch` worked. Both are registered by the SAME plugin, loaded from the workspace — but `coord_comment` shells out to `marketplace/scripts/coord.sh` while `coord_watch` shells out to `marketplace/scripts/gh_watch.sh`. `gh_watch.sh` was already in the pinned `marketplace` gitlink; `coord.sh` had merged LATER and the gitlink had not advanced, so the tool reported "coord.sh not found — sync the marketplace pin". The failure was a stale `marketplace` gitlink, not a defect in the plugin that bound the script. diff --git a/src/content/docs/recipes/internals/agents/parallel-bed-testing.md b/src/content/docs/recipes/internals/agents/parallel-bed-testing.md index 3efa9ce3..343643a1 100644 --- a/src/content/docs/recipes/internals/agents/parallel-bed-testing.md +++ b/src/content/docs/recipes/internals/agents/parallel-bed-testing.md @@ -602,12 +602,13 @@ in how `charly check run` actually behaves: the Containerfile, so a check-only edit rebuilds in seconds (every upstream RUN/COPY cache-hits). Write check coverage aggressively; only layer/package edits pay a full rebuild. -- **`context_ignore` and `--podman-jobs` / `--jobs`** are the legitimate - build-speed levers (trim the build-context tar; parallelize stages - within one build and images across a DAG level). On by default. +- **`context_ignore` and `--stage-parallelism` / `--jobs`** are the legitimate + build-speed levers (trim the build-context tar; raise the limit on how + many build steps the builder runs at once, and how many images are built + per DAG level). On by default. **Flag discipline — speed levers vs. scope-shrinking.** A "go faster" -mandate tempts the forbidden shortcut. Legitimate: `--podman-jobs`, +mandate tempts the forbidden shortcut. Legitimate: `--stage-parallelism`, `--jobs`, `context_ignore`, pre-warming, agent-layer parallelism, longest-pole scheduling. R10-scope-shrinking (need explicit per-turn operator authorization, the project rulebook's R10 flag-override clause): diff --git a/src/content/docs/recipes/internals/git-workflow.md b/src/content/docs/recipes/internals/git-workflow.md index b210d3d8..4445c3b0 100644 --- a/src/content/docs/recipes/internals/git-workflow.md +++ b/src/content/docs/recipes/internals/git-workflow.md @@ -258,6 +258,7 @@ commit — never re-dispatch the same head. Detail: the reference + the script h A push that skips any step is an R1 violation. A partial fix set or a skipped RCA is one too. The validator re-reviews the diff + body + the FULL live thread on every run, so pushing with a stale read OR a stale body re-reviews the wrong state and can re-ship a defect an existing comment already named. A hook/classifier block is never reshaped-and-retried (B5). - **Assert the body edit LANDED, then read the LIVE body back — a silently-failed edit is the same defect as a stale body.** A PR body is rewritten by script far more often than by hand, and its failure mode is silent: `str.replace`/`sed` whose anchor stopped matching (a renamed heading, a reflowed line, a re-quoted string) returns the input UNCHANGED and exits 0, so the script reports success while the prose keeps describing an edit the artefact never received — and the next verdict blocks on evidence attributed to a head or a symbol that is not there. Do BOTH whenever a body is rewritten by a script: (a) **assert every insertion** — `assert s.count(anchor) == 1` (or an explicit `if anchor not in s: raise`) around EACH replacement, so a no-op cannot pass silently; and (b) **read the live body back** — `gh api repos///pulls/ --jq .body`, then grep it for a NAMED marker unique to each edit (a symbol, a line number, a section heading) BEFORE firing any re-run, because the validator reads the body GitHub is serving, not the file on disk. Two supporting habits close the same class from the other end: put the head SHA and the command INSIDE every evidence fence (`# head (git rev-parse HEAD)` as the fence's first line) so a paste carries its own provenance instead of depending on the prose around it; and put a NUMBER on every deferred debt (an issue number, a run id, a count) rather than a phrase like "later", so the record says what is owed and to whom. - **A fence that names its head is self-ATTRIBUTING, not self-VERIFYING — add one command whose output would DIFFER if the paste were from another tree.** Naming the head inside the fence is necessary and not sufficient: the label can be right while the content is stale. Measured (`opencharly/plugin-clean#13`): a fence headed "every command below ran on this tree" printed `git rev-parse HEAD` → `8c67522` and then a `git log` listing the PREDECESSOR's commits — correct head, stale body, and the contradiction sat in plain sight. The instrument is a single command: `git rev-parse ^{tree} ^{tree}` printing TWO EQUAL hashes establishes tree identity, and two equal tree hashes cannot both be true of different content. Use it (or any command whose result changes with the tree) instead of arguing the trees are equivalent; when the paste IS from an earlier head, say which head it is from and back the delta with the same instrument. +- **A pasted commit id is the 40-character output of the command that produced it — a prefix is quoted as a prefix, never completed, and never under a fence that claims to be a command's output.** A 40-hex string is well-formed long before it is TRUE, and a fence labelled with a command is a claim about that command's result: a reader is entitled to read the 40 characters under that label as the command's own. Measured (`opencharly/opencharly#762`): a body carried `160000 commit b920b01854379fb4b4d16a0d4a06b7a6a3ff1ba0` under the label `git ls-tree origin/main marketplace`, where that command's real output was `160000 commit b920b01854f2981d645cac45920091f0166e20ca` — the first ten characters are the tree's own and the rest was COMPLETED BY PATTERN, because the only measured forms of that id were the tools' own ten-character abbreviations (`recorded b920b01854` in the currency output, a prefix in a `git log` line), and a prefix is not an id. The invented tail then supported a claim that was true of the REAL commit, so no gate could catch it: the claim was TRUE and its evidence was FALSE. So — paste the FULL 40 characters the command printed; when only a prefix was measured, paste the PREFIX and say that it is one (`git log --oneline` prints prefixes, and quoting them under that label is honest); and never leave a reader to complete the tail. The umbrella's own reconciler now resolves every 40-hex id a body pastes and names one that exists in no store the tree can see, but it can only judge the ids a body DOES paste — the discipline above is what keeps the fence true in the first place. - **A claim about THE object is only as good as the string you hand the probe — paste the exact string you fetched, and use a check whose output would differ for a different object.** The falsifier can be sitting in the object's OWN NAME. Measured (`charly#843`): a body claimed an image-filename shape was gone; the probe fetched `…cloudimg-v20261001.604814.qcow2`, while the pin's real name carries NO `v` — re-run with the real string, the shape returns `QFI3` on the newest AND the oldest release, and only the `v`-prefixed name returns `` from the umbrella root, no worktrees inside submodules, pin only MERGED refs, run -`charly task sync` on a base current with `origin/main` (its answer is base-relative), -and bound every command's output (SIGPIPE is ignored — `grep` floods on `Broken +`charly task sync` (it advances EVERY pin the umbrella declares) on a base current +with `origin/main` (its answer is base-relative), and bound every command's output +(SIGPIPE is ignored — `grep` floods on `Broken pipe`). Full detail: [`references/umbrella-mechanics.md`](/recipes/internals/git-workflow/umbrella-mechanics/). ## Reference Index diff --git a/src/content/docs/recipes/internals/git-workflow/branch-and-pr-loop.md b/src/content/docs/recipes/internals/git-workflow/branch-and-pr-loop.md index 51899776..8d0ffb09 100644 --- a/src/content/docs/recipes/internals/git-workflow/branch-and-pr-loop.md +++ b/src/content/docs/recipes/internals/git-workflow/branch-and-pr-loop.md @@ -361,7 +361,9 @@ checkouts stay DETACHED at their recorded gitlinks. 5. Remove your umbrella worktree after its PR merged: `git -C worktree remove .worktrees//`. 6. Fast-forward the umbrella's own `main` (same ff-only rule). -7. Advance pins ONLY with `charly task sync`, prove with `charly task verify`, +7. Advance pins ONLY with `charly task sync` — it advances EVERY pin the umbrella + declares, rolls the checkouts that exist and stages the gitlinks, so read its + scope before running it — prove with `charly task verify`, and land the bump as a normal `feat/` PR (`chore(sync): bump submodule pins …`). A submodule pin is a GITLINK: its checkout stays DETACHED at the recorded commit — NEVER `git -C diff --git a/src/content/docs/recipes/internals/git-workflow/pin-intent-and-currency.md b/src/content/docs/recipes/internals/git-workflow/pin-intent-and-currency.md index f345c662..ed71fc22 100644 --- a/src/content/docs/recipes/internals/git-workflow/pin-intent-and-currency.md +++ b/src/content/docs/recipes/internals/git-workflow/pin-intent-and-currency.md @@ -14,7 +14,9 @@ DETACHED and clean. What that recorded value is ALLOWED to be, and whether it is the value the owning repo actually SERVES, is the **pin contract**. It lives in ONE verb — `verb: git-submodules`, owned by `opencharly/plugin-task` — whose `mode` is `status` | `bump` | `verify` | `currency`. The first three are the -maintenance modes (the task table above; "`task sync` and the commit gate" below). +maintenance modes (the task table above; "`task sync` and the commit gate" below), and +`bump` is the ADVANCE — it moves EVERY pin the umbrella declares, and it is the only +mode of the verb that writes. **`currency` is a first-class mode of the SAME verb**, not a second implementation, and it is the mode that answers "is the umbrella current?". diff --git a/src/content/docs/recipes/internals/git-workflow/umbrella-mechanics.md b/src/content/docs/recipes/internals/git-workflow/umbrella-mechanics.md index 94774fda..35d99885 100644 --- a/src/content/docs/recipes/internals/git-workflow/umbrella-mechanics.md +++ b/src/content/docs/recipes/internals/git-workflow/umbrella-mechanics.md @@ -26,13 +26,13 @@ umbrella root; `charly task list` enumerates them, and the umbrella README's |---|---| | `charly task map` | STATE — list every submodule with its pin and its branch (`mode: status`) | | `charly task init` | MATERIALIZE the umbrella's submodules ON DEMAND: the derived WORKING SET by default, `CHARLY_INIT_PATHS=` for named paths, `CHARLY_INIT_MODE=all` for every declared path. Blobless, never shallow; the mode travels in the ENVIRONMENT, never as a task param — see "The clone shape is a working set" below | -| `charly task sync` | bump pins per Policy B. A path WITH a checkout is MATERIALISED (`switch --detach`) and its gitlink STAGED (`git add`); a path with NO checkout is advanced IN THE INDEX, because the pin IS the index entry — so **a PIN WAVE needs no full set** (`plugin-task#37`). It does not commit and does not open a PR, and it is **not read-only** — it moves checkouts (`plugin-task#26`). A wave's order is **sync -> init -> verify** | +| `charly task sync` | advance EVERY pin the umbrella declares per Policy B — an ORG-WIDE cutover, not a way to move one pin. A path WITH a checkout is MATERIALISED (`switch --detach`) and its gitlink STAGED (`git add`); a path with NO checkout is advanced IN THE INDEX, because the pin IS the index entry — so **a PIN WAVE needs no full set** (`plugin-task#37`). It does not commit and does not open a PR, and it is **not read-only** — it moves checkouts (`plugin-task#26`). A wave's order is **sync -> init -> verify** | | `charly task hooks` | install `hooks/pre-commit` for this clone (once; sets `core.hooksPath hooks`) | -| `charly task verify` | CONSISTENCY — the FULL pinning audit on demand: policy B, then every checkout clean and at its recorded gitlink, then the per-submodule `git ls-remote` default-branch audit. There is **no CI gate**; this IS the gate. Run it on the final tree and paste the output (R7: a green `git status` proves nothing). **The checkout half covers the MATERIALIZED paths only** — an unmaterialized path is *uninitialized (pin audited, no checkout)*, a notice, not a failure | +| `charly task verify` | CONSISTENCY — the FULL pinning audit on demand: policy B, then every checkout clean and at its recorded gitlink, then the CURRENCY question (a behind pin is REPORTED, not failed), then the per-submodule `git ls-remote` default-branch audit. There is **no CI gate**; this IS the gate. Run it on the final tree and paste the output (R7: a green `git status` proves nothing). **The checkout half covers the MATERIALIZED paths only** — an unmaterialized path is *uninitialized (pin audited, no checkout)*, a notice, not a failure | | `charly task org-map` | reconcile the README org-map TABLES against `.gitmodules`. It reads ONLY those two surfaces, so it is NOT an org-vs-umbrella census — see "`org-map` answers the wrong question" below | | `charly task skills` | `scripts/sync-dispatcher.sh`: splice the generated R0 dispatcher into `AGENTS.md` (no-op while the table is hand-curated outside the markers) | | `charly task pin-check` | assert the pin contract's consistency half — the EDGE (every `pin_map` entry equals its twin in the `pinned_from` checkout) and the CHECKOUT AUDIT (every declared path clean and at its recorded gitlink). It READS the index's gitlinks, the MANIFEST `charly.yml`'s `pins:` block and the DECLARED PATH SET `.gitmodules`, so a commit can violate it by moving a gitlink **or** by editing the manifest **or** by editing the declared path set — a commit that moves NO gitlink can still violate it. It is the half of `verify` that needs no remote (the per-submodule `git ls-remote` default-branch audit is the rest), and it is what `hooks/pre-commit` runs. Checkout DRIFT on its own — a checkout dirty or at another commit, caused by NO commit — stays with `charly task verify` | -| `charly task pins` | the Policy-B pin operation over the `distro-*` set (mode via param) | +| `charly task pins` | the pin-contract pin operation over EVERY declared path (mode via param) — the `distro-*` paths are its EDGE subset (Policy B's `pin_map`); `MODE=bump` advances EVERY pin the umbrella declares | | `charly task self-test` | self-test the committed CI body/pin-evidence builders + the dispatcher splice + the submodule initializer | | `charly task prune` | reap merged-upstream session worktrees + branches. The declared `MODE` DEFAULTS to `report`, so **plain `charly task prune` IS the dry run**; a positional `MODE=report` is rejected (`unexpected extra argument`), and the removal is selected through the task's own param (`--param MODE=prune`) | @@ -192,7 +192,8 @@ them — its builds resolve them from the Go proxy at pinned `go.mod` requires. `marketplace`, `docs`, and every `plugin-*` repo are recorded at their own default-branch HEAD. Pin only MERGED refs (default branches or gitlinks charly records), never a PR branch; a dangling pin is a failure. If pinning changed, the -fix is `charly task sync` + a PR — never a hand-pin or a branch checkout to +fix is `charly task sync` + a PR — it advances EVERY pin the umbrella declares, so +read its scope before running it — never a hand-pin or a branch checkout to "catch up". **Policy B is a CONSISTENCY contract, not a CURRENCY one: it holds while both diff --git a/src/content/docs/recipes/internals/marketplace.md b/src/content/docs/recipes/internals/marketplace.md index e58891fb..63fd2a53 100644 --- a/src/content/docs/recipes/internals/marketplace.md +++ b/src/content/docs/recipes/internals/marketplace.md @@ -13,8 +13,9 @@ description: "How the OpenCharly skill/marketplace corpus is GENERATED, refreshe The opencharly/marketplace repo's root (family dirs: internals/, build/, check/, core/, vm/, automation/, kubernetes/, local/, image/, pod/, ...) is the GENERATED corpus. Every SKILL.md -carries a `GENERATED by charly marketplace generate — DO NOT EDIT` banner; agent -definitions under `/agents/` are HAND-AUTHORED (no banner). +and every agent definition under `/agents/` carries a +`GENERATED by charly marketplace generate — DO NOT EDIT` banner: both are projections of an +owning repo's file, so neither is edited in the corpus. `candy/charly-marketplace/charly.yml` is the refs list: `@github.com/opencharly/:v` pins for every standalone candy repo that owns `skill:`/`hook:`/`marketplace:` entities. It is the ONE generation input — change a skill at its SOURCE, never the generated copy. @@ -22,23 +23,23 @@ is the ONE generation input — change a skill at its SOURCE, never the generate ## The ownership split (post cutover) The charly repo owns NO skill: entities (its candy/ dir was emptied by the de-submodule -cutover). Each family lives in its standalone owning candy repo: - -| Family | Owning repo | -|---|---| -| internals | layer-charly-internals (+ layer-charly-internals-extra for strict-policy and vm-spec) | -| build | layer-charly-build | -| check | layer-charly-check | -| kubernetes | layer-charly-kubernetes | -| image | layer-charly-image | -| core | pod-charly-core | -| automation | pod-charly-automation | -| local | layer-charly-local | -| pod | layer-charly-pod | -| hooks / marketplace verb / hook: entities | pod-charly-hooks | - -A skill edit lands in the OWNING repo as a FILE — `/.agents/skills//SKILL.md`, -or `/.agents/agents/.md` for a sub-agent definition — while the `skill:` node +cutover). A FAMILY is not the unit of ownership: the drains moved each skill to the repo +that holds the verb it documents, so one family spans several repos, and the split moves +with every such move — so the map is not restated here. + +**The map is the refs list.** `candy/charly-marketplace/charly.yml` is the ONE generation +input: it names the sources that own `skill:`/`hook:`/`marketplace:` entities, and a source +it does not name reaches the corpus only through the closure of a ref that does. A published +body is the byte-identical copy of ONE source's body — the text after the frontmatter of +that source's `.agents/skills//SKILL.md` file, or the inline body of its manifest's +`skill:` node. The generator writes that body with the ONE trailing newline the EOF +convention prescribes — and an authored file already ends with one — so the source's body +and the published body (the one that follows the generated header) agree byte for byte. So +the owner of a published skill is the source whose body matches the published one +byte for byte: read the refs list for the sources, then match. + +A skill edit lands in the OWNING repo as a FILE — `/.agents/skills//SKILL.md`, +or `/.agents/agents/.md` for a sub-agent definition — while the `skill:` node beside it carries only routing (`family`/`owner`/`triggers`). Then the marketplace refs list is bumped and the corpus regenerated. Never hand-edit a generated SKILL.md or a marketplace family dir. The authoring contract — the file's shape, the routing-only node, the prohibition on an image `owner:`, and diff --git a/src/content/docs/recipes/internals/repo-setup.md b/src/content/docs/recipes/internals/repo-setup.md index 4b68d485..dbab0ac0 100644 --- a/src/content/docs/recipes/internals/repo-setup.md +++ b/src/content/docs/recipes/internals/repo-setup.md @@ -135,7 +135,10 @@ an appliance is the defect this step exists to catch. for the new repo. 4. Land the first commit via a PR — never a direct push to `main`. 5. Add it to the umbrella as a submodule and advance the pin via - `charly task sync` + PR (see [`/charly-internals:git-workflow`](/recipes/internals/git-workflow/) B8). + `charly task sync` + PR (see [`/charly-internals:git-workflow`](/recipes/internals/git-workflow/) B8). **That verb + is an ORG-WIDE wave, not a one-repo operation: it advances EVERY pin the + umbrella declares** — read its scope before running it, and expect the PR's + diff to carry every pin it moved, not only the new repo's. ## The measured gap this checklist prevents diff --git a/src/content/docs/reference/candy/github-com-opencharly-plugin-build-v2026-242-0525/plugin-build.md b/src/content/docs/reference/candy/github-com-opencharly-plugin-build-v2026-242-0525/plugin-build.md deleted file mode 100644 index 267afc61..00000000 --- a/src/content/docs/reference/candy/github-com-opencharly-plugin-build-v2026-242-0525/plugin-build.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -title: "plugin-build" -description: "The BUILD-DRIVE plugin: it OWNS the podman build drive (the build-order loop, the per-image build lock, the push, and the merge gate) for `build:box` (the `charly box build` engine) and `build:generate` (the `charly box generate` engine), AND the ensure-image ORCHESTRATION for `build:ensure` (core-min wave 3, build-engine cluster relocation — the former core ensure-image + cross-engine-transfer helpers): pull an image, falling back to a local (or remote-cached) build reached via the SAME in-process build:box drive this candy already owns — no new build seam." ---- - - - -| | | -|---|---| -| **Version** | `2026.206.1217` | -| **Repo** | `box/github.com/opencharly/plugin-build:v2026.242.0525` | -| **Plugin** | yes — see the [plugin reference](/reference/plugin/github-com-opencharly-plugin-build-v2026-242-0525/plugin-build/) | - -The BUILD-DRIVE plugin: it OWNS the podman build drive (the build-order loop, the -per-image build lock, the push, and the merge gate) for `build:box` (the `charly box -build` engine) and `build:generate` (the `charly box generate` engine), AND the -ensure-image ORCHESTRATION for `build:ensure` (core-min wave 3, build-engine cluster -relocation — the former core ensure-image + cross-engine-transfer helpers): pull an -image, falling back to a local (or remote-cached) build reached via the SAME in-process -build:box drive this candy already owns — no new build seam. The heavy loader/render -RESOLVE runs PLUGIN-SIDE ITSELF (resolveBuildEngine, K3 U6 — the former host-side -NewGenerator this candy used to call over HostBuild is DELETED, #55 step3 3-II), reaching -the host only for what a sdk-only candy structurally cannot do (the bootstrap-delicate -local scan, the git fetch, the build-time plugin connect, the host-fs prep) over a small -`buildengine-*` HostBuild leg family; the layer MERGE crosses to verb:oci via -InvokeProvider (no HostBuild("merge") seam). The candy then runs podman directly — -building each image (Containerfile piped over stdin), gating the inline merge on the -box's MergeAuto, and pushing (podman) after merge; build:generate renders the .build/ -Containerfile tree in its own code (via sdk/deploykit, #67) and returns the written paths -(no podman, no merge); build:ensure resolves a project-authored image identifier -against charly.yml PLUGIN-SIDE (loaderkit.LoadUnified via the K1 loader reverse legs — -no "box-ref-resolve" HostBuild seam, shed #55 coneK1 #8) and a remote -@github.com/... identifier via HostBuild("remote-image-resolve", …) (git clone/cache -ONLY, then plugin-side resolve), then execs `podman pull`/`podman tag` directly (no -seam needed for that — same host, same privilege as the build drive's own podman exec). -Only the wire envelopes cross the seam; the podman exec happens IN the candy. COMPILED-IN (in the -embedded compiled_plugins:) — `charly box build` / `charly box generate` / every -dispatchBuildEnsure caller dispatch it IN-PROCESS, the host threading the reverse -channel onto the Invoke context so HostBuild is reachable without a go-plugin broker. -Also serves out-of-process via cmd/serve for module-shape parity (one provider, two -placements). - -## Acceptance plan - -This candy's `plan:` — the runnable spec `charly check` executes against a live deployment. `check:` steps are idempotent probes; `run:` steps change state. - -| Intent | Step | -|---|---| -| `check` | the build-drive plugin ships a buildable Go module the host compiles in (default) or serves out-of-process; its build:box / build:generate / build:ensure words own the podman build drive + the ensure-image orchestration and resolve/merge over the F10 HostBuild seam, exercised by EVERY box build and every image-ensure fallback across the whole ecosystem (every `charly box build` / `charly box generate` / `charly box pull` / deploy image-preflight invocation) | diff --git a/src/content/docs/reference/candy/github-com-opencharly-plugin-build-v2026-296-1104/plugin-build.md b/src/content/docs/reference/candy/github-com-opencharly-plugin-build-v2026-296-1104/plugin-build.md new file mode 100644 index 00000000..8f77b083 --- /dev/null +++ b/src/content/docs/reference/candy/github-com-opencharly-plugin-build-v2026-296-1104/plugin-build.md @@ -0,0 +1,56 @@ +--- +title: "plugin-build" +description: "The BUILD-DRIVE plugin: it OWNS the build drive — BuildKit is the builder (the build-order loop, the per-image build lock, the push, and the merge gate) for `build:box` (the `charly box build` engine) and `build:generate` (the `charly box generate` engine), AND the ensure-image ORCHESTRATION for `build:ensure` (core-min wave 3, build-engine cluster relocation — the former core ensure-image + cross-engine-transfer helpers): pull an image, falling back to a local (or remote-cached) build reached via the SAME in-process build:box drive this candy already owns — no new build seam." +--- + + + +| | | +|---|---| +| **Repo** | `box/github.com/opencharly/plugin-build:v2026.296.1104` | +| **Plugin** | yes — see the [plugin reference](/reference/plugin/github-com-opencharly-plugin-build-v2026-296-1104/plugin-build/) | + +The BUILD-DRIVE plugin: it OWNS the build drive — BuildKit is the builder (the build-order loop, the +per-image build lock, the push, and the merge gate) for `build:box` (the `charly box +build` engine) and `build:generate` (the `charly box generate` engine), AND the +ensure-image ORCHESTRATION for `build:ensure` (core-min wave 3, build-engine cluster +relocation — the former core ensure-image + cross-engine-transfer helpers): pull an +image, falling back to a local (or remote-cached) build reached via the SAME in-process +build:box drive this candy already owns — no new build seam. The heavy loader/render +RESOLVE runs PLUGIN-SIDE ITSELF (resolveBuildEngine, K3 U6 — the former host-side +NewGenerator this candy used to call over HostBuild is DELETED, #55 step3 3-II), reaching +the host only for what a sdk-only candy structurally cannot do (the bootstrap-delicate +local scan, the git fetch, the build-time plugin connect, the host-fs prep) over a small +`buildengine-*` HostBuild leg family; the layer MERGE crosses to verb:oci via +InvokeProvider (no HostBuild("merge") seam). Each image is built with BuildKit, its +export NAMED with `io.containerd.image.name=`, written to the engine-neutral +CHARLY IMAGE STORE (one OCI layout at $XDG_DATA_HOME/charly/images, see imagestore.go), +and materialised into the configured container backend with NO tag step — the +annotation names the image on load. A parent box present in the store is supplied to +the solve as an `--oci-layout` named context (buildkitdrive.Build.OCILayouts), so a +box-on-box build does not reach the registry for it; each image is ALSO pushed to the +LOCAL CHARLY OCI REGISTRY under the same FullTag, which the daemon mirrors +`ghcr.io/opencharly` to, so any base the store does not hold resolves internally +rather than against the upstream registry. build:generate renders the .build/ +Containerfile tree in its own code (via sdk/deploykit, #67) and returns the written paths +(no podman, no merge); build:ensure resolves a project-authored image identifier +against charly.yml PLUGIN-SIDE (loaderkit.LoadUnified via the K1 loader reverse legs — +no "box-ref-resolve" HostBuild seam, shed #55 coneK1 #8) and a remote +@github.com/... identifier via HostBuild("remote-image-resolve", …) (git clone/cache +ONLY, then plugin-side resolve), then pulls/tags through the resolved run engine's +binary from the engine capability table (no engine word literal, no seam needed — +same host, same privilege as the build drive). +Only the wire envelopes cross the seam; the engine exec happens IN the candy. COMPILED-IN (in the +embedded compiled_plugins:) — `charly box build` / `charly box generate` / every +dispatchBuildEnsure caller dispatch it IN-PROCESS, the host threading the reverse +channel onto the Invoke context so HostBuild is reachable without a go-plugin broker. +Also serves out-of-process via cmd/serve for module-shape parity (one provider, two +placements). + +## Acceptance plan + +This candy's `plan:` — the runnable spec `charly check` executes against a live deployment. `check:` steps are idempotent probes; `run:` steps change state. + +| Intent | Step | +|---|---| +| `check` | the build-drive plugin ships a buildable Go module the host compiles in (default) or serves out-of-process; its build:box / build:generate / build:ensure words own the build drive (BuildKit via the buildkitdrive kit, bridged into the run engine's store) + the ensure-image orchestration and resolve/merge over the F10 HostBuild seam, exercised by EVERY box build and every image-ensure fallback across the whole ecosystem (every `charly box build` / `charly box generate` / `charly box pull` / deploy image-preflight invocation) | diff --git a/src/content/docs/reference/plugin/github-com-opencharly-plugin-build-v2026-242-0525/plugin-build.md b/src/content/docs/reference/plugin/github-com-opencharly-plugin-build-v2026-296-1104/plugin-build.md similarity index 69% rename from src/content/docs/reference/plugin/github-com-opencharly-plugin-build-v2026-242-0525/plugin-build.md rename to src/content/docs/reference/plugin/github-com-opencharly-plugin-build-v2026-296-1104/plugin-build.md index 2716e7b6..bb672ccf 100644 --- a/src/content/docs/reference/plugin/github-com-opencharly-plugin-build-v2026-242-0525/plugin-build.md +++ b/src/content/docs/reference/plugin/github-com-opencharly-plugin-build-v2026-296-1104/plugin-build.md @@ -1,6 +1,6 @@ --- title: "plugin-build" -description: "The BUILD-DRIVE plugin: it OWNS the podman build drive (the build-order loop, the per-image build lock, the push, and the merge gate) for `build:box` (the `charly box build` engine) and `build:generate` (the `charly box generate` engine), AND the ensure-image ORCHESTRATION for `build:ensure` (core-min wave 3, build-engine cluster relocation — the former core ensure-image + cross-engine-transfer helpers): pull an image, falling back to a local (or remote-cached) build reached via the SAME in-process build:box drive this candy already owns — no new build seam." +description: "The BUILD-DRIVE plugin: it OWNS the build drive — BuildKit is the builder (the build-order loop, the per-image build lock, the push, and the merge gate) for `build:box` (the `charly box build` engine) and `build:generate` (the `charly box generate` engine), AND the ensure-image ORCHESTRATION for `build:ensure` (core-min wave 3, build-engine cluster relocation — the former core ensure-image + cross-engine-transfer helpers): pull an image, falling back to a local (or remote-cached) build reached via the SAME in-process build:box drive this candy already owns — no new build seam." --- @@ -9,7 +9,6 @@ description: "The BUILD-DRIVE plugin: it OWNS the podman build drive (the build- |---|---| | **Placement** | compiled-in (in-process) | | **Source** | `github.com/opencharly/plugin-build/candy/plugin-build` | -| **Version** | `2026.206.1217` | | **Candy** | `plugin-build` | This plugin is listed in `charly/charly.yml`'s `compiled_plugins:`, so its providers are compiled into the `charly` binary and register in-process. @@ -24,7 +23,7 @@ The reserved words this plugin serves: ## What it does -The BUILD-DRIVE plugin: it OWNS the podman build drive (the build-order loop, the +The BUILD-DRIVE plugin: it OWNS the build drive — BuildKit is the builder (the build-order loop, the per-image build lock, the push, and the merge gate) for `build:box` (the `charly box build` engine) and `build:generate` (the `charly box generate` engine), AND the ensure-image ORCHESTRATION for `build:ensure` (core-min wave 3, build-engine cluster @@ -36,17 +35,25 @@ NewGenerator this candy used to call over HostBuild is DELETED, #55 step3 3-II), the host only for what a sdk-only candy structurally cannot do (the bootstrap-delicate local scan, the git fetch, the build-time plugin connect, the host-fs prep) over a small `buildengine-*` HostBuild leg family; the layer MERGE crosses to verb:oci via -InvokeProvider (no HostBuild("merge") seam). The candy then runs podman directly — -building each image (Containerfile piped over stdin), gating the inline merge on the -box's MergeAuto, and pushing (podman) after merge; build:generate renders the .build/ +InvokeProvider (no HostBuild("merge") seam). Each image is built with BuildKit, its +export NAMED with `io.containerd.image.name=`, written to the engine-neutral +CHARLY IMAGE STORE (one OCI layout at $XDG_DATA_HOME/charly/images, see imagestore.go), +and materialised into the configured container backend with NO tag step — the +annotation names the image on load. A parent box present in the store is supplied to +the solve as an `--oci-layout` named context (buildkitdrive.Build.OCILayouts), so a +box-on-box build does not reach the registry for it; each image is ALSO pushed to the +LOCAL CHARLY OCI REGISTRY under the same FullTag, which the daemon mirrors +`ghcr.io/opencharly` to, so any base the store does not hold resolves internally +rather than against the upstream registry. build:generate renders the .build/ Containerfile tree in its own code (via sdk/deploykit, #67) and returns the written paths (no podman, no merge); build:ensure resolves a project-authored image identifier against charly.yml PLUGIN-SIDE (loaderkit.LoadUnified via the K1 loader reverse legs — no "box-ref-resolve" HostBuild seam, shed #55 coneK1 #8) and a remote @github.com/... identifier via HostBuild("remote-image-resolve", …) (git clone/cache -ONLY, then plugin-side resolve), then execs `podman pull`/`podman tag` directly (no -seam needed for that — same host, same privilege as the build drive's own podman exec). -Only the wire envelopes cross the seam; the podman exec happens IN the candy. COMPILED-IN (in the +ONLY, then plugin-side resolve), then pulls/tags through the resolved run engine's +binary from the engine capability table (no engine word literal, no seam needed — +same host, same privilege as the build drive). +Only the wire envelopes cross the seam; the engine exec happens IN the candy. COMPILED-IN (in the embedded compiled_plugins:) — `charly box build` / `charly box generate` / every dispatchBuildEnsure caller dispatch it IN-PROCESS, the host threading the reverse channel onto the Invoke context so HostBuild is reachable without a go-plugin broker. @@ -85,11 +92,11 @@ The CUE schema below is the authoritative grammar for this plugin's input. It is cache?: string no_cache?: bool jobs?: int - podman_jobs?: int + stage_parallelism?: int } ``` --- -See also the [candy reference](/reference/candy/github-com-opencharly-plugin-build-v2026-242-0525/plugin-build/) for this candy's install surface. +See also the [candy reference](/reference/candy/github-com-opencharly-plugin-build-v2026-296-1104/plugin-build/) for this candy's install surface. diff --git a/src/content/docs/reference/providers.md b/src/content/docs/reference/providers.md index 0c9d826e..4e5b7239 100644 --- a/src/content/docs/reference/providers.md +++ b/src/content/docs/reference/providers.md @@ -21,12 +21,12 @@ description: "Every reserved word — verb, kind, deploy target, step, builder, | Word | Served by | |---|---| -| `box` | [plugin-build](/reference/plugin/github-com-opencharly-plugin-build-v2026-242-0525/plugin-build/) | | `box` | [plugin-build](/reference/plugin/github-com-opencharly-plugin-build-v2026-287-2220/plugin-build/) | -| `ensure` | [plugin-build](/reference/plugin/github-com-opencharly-plugin-build-v2026-242-0525/plugin-build/) | +| `box` | [plugin-build](/reference/plugin/github-com-opencharly-plugin-build-v2026-296-1104/plugin-build/) | | `ensure` | [plugin-build](/reference/plugin/github-com-opencharly-plugin-build-v2026-287-2220/plugin-build/) | -| `generate` | [plugin-build](/reference/plugin/github-com-opencharly-plugin-build-v2026-242-0525/plugin-build/) | +| `ensure` | [plugin-build](/reference/plugin/github-com-opencharly-plugin-build-v2026-296-1104/plugin-build/) | | `generate` | [plugin-build](/reference/plugin/github-com-opencharly-plugin-build-v2026-287-2220/plugin-build/) | +| `generate` | [plugin-build](/reference/plugin/github-com-opencharly-plugin-build-v2026-296-1104/plugin-build/) | ## `builder` — 6 words