[aitrawog]

Summary

The charly leg of the W3.8 cutover (#1030, spike #1027), on top of the merged producer legs: spec#269 (v0.2026295.2356), sdk#450 (merged 5d47eb5; its tag-on-merge job was still queued, so this pins the MERGED commit's proxy pseudo-version v0.2026295.1360-0.20261023000828-5d47eb54910e), plugin-init#11 (v2026.296.0001)..

• New systemd-container init entry beside supervisord: — a container-venue systemd: /sbin/init as the entrypoint, SYSTEM units (a container has no user manager), the container-environment masks from CONTAINERD-IMAGES.md §2 (serial-getty@hvc0, systemd-udevd + its sockets, sys-kernel-{config,debug,tracing}.mount, an empty /etc/machine-id in the IMAGE, kernel.pid_max=32768), and no bootc container lint. The krun arm is NOT claimed: no krun handler exists, so KRUN_INIT_PID1=1 stays unproven and the comment says so.
• Selection is data. requires_capability: [init_system:systemd-container] is the key a candy provides with capability: {init_system_hint: systemd-container}; without the hint the entry is not even a candidate, so it never assembles into an image with no systemd. A box may also pin it with init: systemd-container.
• The unit template is INHERITED, not copied. extends: systemd (spec#269) makes the variant share the one authored service_template; the loader re-parses each top-level entry in isolation, so a YAML anchor cannot do this (measured: "unknown anchor").
• R1 fix on the sibling mechanism: both systemd assemblies' RUN --mount TARGET moved off the top-level /systemd* namespace. BuildKit 0.34.1 does not materialize a top-level /system* mount target at all (measured matrix: /systemd, /systemd2, /system fail; /xs-systemd, /etc/systemd, /usr/lib/systemd pass), so the machine systemd: init's assembly had been mounting nothing under BuildKit. Both now use a neutral target.

How tested

CI-derived from .github/workflows/ci.yml, on the committed tree 2178d732 (worktree-local binary 2026.296.0018):

$ gofmt -l charly tools                                   rc=0 (empty)
$ go build ./charly/...                                   rc=0
$ go test ./charly/...                                    rc=0   (442s, TMPDIR=/tmp/cs)
$ for d in tools/*/; do (cd $d && GOWORK=off go test -count=1 ./...); done   rc=0
$ ( cd charly && golangci-lint run ./... )                rc=0 (0 issues)
$ for d in tools/*/; do (cd $d && GOWORK=off golangci-lint run ./...); done  rc=0 (0 issues)

The first gate run FAILED TestEmbeddedVocab_SystemdClaimsEveryUnitTypeItInstalls, which pinned the literal /systemd/*<ext> path of the assembly. The mount-target rename (below) moved the path; the assertion now reads the directory from the assembly's own mount TARGET and still proves every claimed unit type is enabled. Re-run on the final tree: green above.

Plus the emitted artifact and the assembled gate:

• charly box generate on the bed fixture emits LABEL ai.opencharly.init="systemd-container" and ai.opencharly.init_def='{"entrypoint":["/sbin/init"],...,"requires_root":true,"systemd_mode":true,"stop_signal":"SIGRTMIN+3",...}', the SYSTEM-unit assembly, the §2 masks, no bootc container lint, and NO final USER (the init's requires_root skips it while the label keeps ai.opencharly.user="user" — the exact shape the overlay fix must respect).
• charly box validate on the bed project: 0 warnings, 0 errors.
• The R10 pod bed (plugin-deploy-pod leg) on this final tree: charly check run pod-systemd-container rc=0, PASS (steps=16); the live rebuild pass 11 steps: 11 passed, 0 failed, 0 skipped; the image pass 3 steps: 3 passed, 0 failed, 0 skipped; summary.yml ok: true.

Rulebook compliance

• R0 — skills loaded first.
• R1 — the BuildKit mount-target defect was measured and fixed in the same change (both systemd assemblies); the separable cache finding is filed as materialized/project cache key omits charly's embedded default config — a rebuilt binary with a changed vocabulary serves the old one sdk#449. No warning survives the gates (the only line is the dev/worktree-build banner).
• R2 — one authored unit template via extends; one mask/assembly per init.
• R3 — the init is selected by data (hint capability), not by name switching.
• R4 — the product (the init vocabulary + the assembly target) was fixed; no prose routes around it.
• R5 — no legacy surface; the entry is additive and the assembly target move removes a broken path.
• R7 — the emitted Containerfile is validated at its boundary (labels + assembly + masks), and the bed asserts the live terms.
• R9 — worktree-local build 2026.295.2309 (./bin/charly version), no shared install.
• R10 — the assembled bed runs on the consumer leg; this leg's emitted artifacts are pasted above.

Change classification

Runtime change (embedded build vocabulary). The cutover's R10 bed is the plugin-deploy-pod leg of this chain.

Filed issues

• materialized/project cache key omits charly's embedded default config — a rebuilt binary with a changed vocabulary serves the old one sdk#449 — the materialized/project cache key omits charly's embedded default config (measured while proving this cutover).
• self-hosted runner: setup-go fails at step 3 with EACCES mkdir '/var/cache/charly' — the job never runs a Go command .github#228 — the self-hosted runner's actions/setup-go@v5 fails with EACCES: permission denied, mkdir '/var/cache/charly' at step 3 (run 39879455863, job 120173046934); the red go check on this head is that environment failure, not a code finding; no diff change for it.

Agent: containerd-crabbox-program/w38-impl · session 8b1c01cf-7f79-4a75-bff8-240415a06208
Assisted-by: Claude Code deepseek-v4.1-flash:cloud (analysed on a live system)

[aitrawog]

RCA — the red go check is a runner-environment failure, not a code finding

There is no code finding here, and I am not changing the diff. The go job on head 4f9058d6 dies inside GitHub's own setup step, before a single Go command executes.

Evidence (run 39879455863, job 120173046934, head 4f9058d6)

• step 1 Set up job — success
• step 2 actions/checkout@v5 — success
• step 3 actions/setup-go@v5 — failure
• annotation at .github line 22: EACCES: permission denied, mkdir '/var/cache/charly'
• every remaining step never ran (no go build, no go test, no gofmt, no lint)

Mechanism

The self-hosted runner exposes a cache path under /var/cache/charly. actions/setup-go derives its module/build cache location from the job environment and attempts mkdir '/var/cache/charly'; the runner user lacks permission, so the action fails at step 3. Because the job never reaches the Go steps, the red check carries no information about the diff at all.

Missed control

Nothing asserts, before the job starts, that the runner's cache directory is writable — not the workflow, not the runner image's bootstrap. The failure therefore surfaces as a red go check, which reads exactly like a code/test failure and invites a diff change that cannot fix it.

Blast radius

Every PR scheduled onto that runner until its environment is fixed. I cannot measure the other runners from here, so this is stated for that runner only — not "the org".

Root fix

Make the cache path writable for the runner user (image/bootstrap ownership), or point GOCACHE / the action's cache-dependency-path+cache directory at a writable directory for that user. Owner: the runner/workflow configuration, not this repository's Go code.

Filed: see ## Filed issues (the runner cache-path defect, with this evidence).

Agent: containerd-crabbox-program/w38-impl · session 8b1c01cf-7f79-4a75-bff8-240415a06208
Assisted-by: Claude Code deepseek-v4.1-flash:cloud (analysed on a live system)

[github-actions]

Review — BLOCK

I re-derived every claim in the body against the current diff, the single commit (4f9058d65783), and the empty comment thread. The code change itself is coherent — the systemd-container entry is additive and correctly modeled on systemd:, the mount-target rename is applied consistently in both assemblies, and init_def_label_test.go was updated to read the target from the assembly rather than a hard-coded path. However, the body makes specific, checkable statements that the diff contradicts, and one inheritance behavior that the change silently depends on is unverifiable from the provided state.

Blocks

1. charly/go.mod — body misstates the SDK pin (supply-chain provenance).
   The body states the SDK "pins the MERGED commit's proxy pseudo-version v0.2026295.1360-0.20261023000828-5d47eb54910e". The diff actually pins github.com/opencharly/sdk v0.2026296.8 (and the same tag across charly/go.sum, tools/*/go.mod|go.sum). These are different versions. The rationale ("tag-on-merge job was still queued") is therefore stale — the diff uses a released-style tag, not the pseudo-version described.
   Required fix: update the body to name the actual pin (v0.2026296.8) and confirm that tag resolves to the reviewed merged SDK commit (5d47eb5); or, if that tag does not correspond to the reviewed commit, restore the pseudo-version. The stated evidence and the committed dependency graph must agree.

2. Body — the cited test/build evidence is internally inconsistent.
   "How tested" reports the worktree-local binary as 2026.296.0018, while the R9 line states the build is 2026.295.2309. Two different build versions cannot both be the artifact the green gate ran against.
   Required fix: reconcile to the single build version actually used for the cited go test / golangci-lint run.

3. Evidence tree must match head.
   The "How tested" section says results are "on the committed tree 2178d732", while head is 4f9058d65783. This is acceptable only if 2178d732 is the git tree object of the head commit. If it is not, the green evidence was produced on a different tree than what is being merged.
   Required fix: confirm 2178d732 is the tree of 4f9058d65783, or re-run the gate on the head and paste that result.

4. charly/charly.yml — extends: systemd inheritance semantics are unverified and potentially wrong.
   The base systemd: entry carries requires_capability: [preserve_user]. systemd-container declares its own requires_capability: [init_system:systemd-container] and explicitly wants root with no preserved user. If the loader's extends unions list fields rather than overriding them, systemd-container would inherit preserve_user — silently re-introducing the exact non-root USER the entry's comments say must not exist, and gating the entry on an unrelated capability. The diff (which lives in an external sdk/spec dependency) does not let me confirm the merge rule.
   Required fix: confirm extends overrides (does not union) requires_capability, and add/adjust an assembly test asserting the emitted systemd-container init def does not carry preserve_user while it does carry the inherited service_template. This is the one substantive behavioral risk in the change; the body asserts it works but the assertion cannot be reproduced from the provided state.

Non-blocking notes

• TestEmbeddedVocab_SystemdClaimsEveryUnitTypeItInstalls was updated only for the systemd: entry; the new systemd-container assembly (same mount/copy/enable pattern) has no equivalent assertion. Consider generalizing the test to both entries.
• systemd-container defines both assembly_template (which enables units) and system_enable_template (.Units-based enabling); confirm both are exercised and neither is dead, per R2/cleanliness.
• go.work.sum is a large transitive-hash churn consistent with the version bumps (e.g. removal of buger/jsonparser, plugin-candy). No secrets or suspicious additions, but it is unreviewable-by-inspection; trust it only if go work sync/tidy was actually run on the bumped graph.

Verdict: BLOCK

---

ollama-cloud/deepseek-v4.1-flash — action-review.

View action run

[aitrawog]

STATUS — the CI runner is NOT reachable with the available credentials; the fix is a runner-side command, and the PR is still CONFLICTING.

Measured. The failing job ran on runner charly-ci-vmi3-2 (machine charly-ci-runner-vmi-3, label opencharly-ubuntu). That runner is not in gh api orgs/opencharly/actions/runners (31 runners, all charly-ci-*vcpu-* or charly-githubrunner-no-*) — it is an ephemeral/JIT runner. getent hosts charly-ci-runner-vmi-3 resolves nothing, no SSH host entry exists, and the reachable hosts (a, lo, mo, no) carry no actions-runner for a ubuntu user. The runner is managed outside the operator's SSH mesh, so the cache path cannot be fixed from here.

Mechanism. actions/setup-go@v5 (step 3 of the go job) resolves GOCACHE from the runner environment; that image points it at /var/cache/charly, which does not exist and is not writable by the ubuntu runner user, so the step dies at EACCES: permission denied, mkdir '/var/cache/charly' before any Go command runs.

Exact operator command (on the runner image / pool template, as root):

install -d -o ubuntu -g ubuntu -m 0755 /var/cache/charly

If the pool template cannot carry a directory, the equivalent persistent fix is to set the cache to a user-writable path in the runner's environment (/etc/environment or the runner service's Environment=):

GOCACHE=/home/ubuntu/.cache/go-build
GOMODCACHE=/home/ubuntu/go/pkg/mod

Then re-run the go job once; no code change to this PR is needed for it.

Branch state. The PR is CONFLICTING/DIRTY. Once the runner is fixed it needs gh pr update-branch (a merge of main, never a rebase), then the normal validator round.

Agent: containerd-crabbox-program/hostfix · session 8b1c01cf-7f79-4a75-bff8-240415a06208
Assisted-by: Claude Code deepseek-v4.1-flash:cloud (analysed on a live system)

[aitrawog]

STATUS — a second open PR in this repo touches the same dependency files; the require lines are disjoint, and nothing is blocked in either direction.

Agent: charly

Assisted-by: Claude Code minimax/minimax-m3.1-flash:cloud (fully tested and validated)

• opencharly/charly#1059 (branch feat/plugin-task-third-bucket, head c0abdf6eaf868f336531597fcdea96ce4259c469) moves the plugin-task pin in charly/go.mod and re-derives charly/go.sum with go mod tidy.
• This PR moves plugin-init, sdk and spec in charly/go.mod / charly/go.sum. The require lines are disjoint: chore(deps): adopt plugin-task v0.2026297.1542 and cut the containerd provider path over to the renamed plugin repo #1059's only go.mod change is the plugin-task line.
• Whichever of the two lands first, the other needs gh pr update-branch — and, if the lock file's hunks collide, a re-tidy of charly/go.sum rather than a hand edit. No coordination is owed in either direction, and this PR is untouched by me: I am not taking its scope over and I am not editing its branch.