Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions .agents/skills/test-release-canary/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,21 +92,29 @@ Note `install.sh` is pulled from `raw.githubusercontent.com/ai-openshell/openshe

`Release Dev` publishes two chart versions for every dev build (see the push steps in `.github/actions/release-helm-oci/action.yml`):

- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` — floating, overwritten on every main push.
- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev.<sha>` — immutable, `appVersion` set to the same SHA so it pulls the matching `gateway`, `sandbox`, and `supervisor` images.
- `oci://ghcr.io/ai-openshell/openshell/helm-chart:0.0.0-dev` — floating, overwritten on every main push.
- `oci://ghcr.io/ai-openshell/openshell/helm-chart:0.0.0-dev.<sha>` — SHA-pinned, with `appVersion` set to the same SHA so it pulls the matching `gateway`, `sandbox`, and `supervisor` images.

To smoke-test the chart for a specific dev build, dispatch `Release Dev` on the branch first, then run the kind canary steps locally pointed at the SHA-pinned chart (see "Local kind reproduction" below). The release-canary workflow itself does not currently expose `chart_version` / `image_tag` inputs.

## Local kind reproduction

The `kubernetes` job can be reproduced on any machine with Docker and `mise install`-provided `kubectl` + `helm`:

Authenticate Docker and Helm to `ghcr.io` with credentials that have pull access to the dev packages before running these commands.

```shell
kind create cluster --name release-canary-local

bash e2e/support/install-agent-sandbox.sh

helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
for component in gateway sandbox supervisor; do
image="ghcr.io/ai-openshell/openshell/${component}:dev"
docker pull "$image"
kind load docker-image "$image" --name release-canary-local
done

helm install openshell oci://ghcr.io/ai-openshell/openshell/helm-chart \
--version 0.0.0-dev \
--namespace openshell --create-namespace \
--set server.disableTls=true \
Expand All @@ -127,7 +135,7 @@ Keep `pkiInitJob.enabled=true` (the chart default), even when
`server.disableTls=true`. The hook also generates the sandbox JWT signing
secret that the gateway pod always mounts.

Swap `0.0.0-dev` for `0.0.0-dev.<sha>` to pin to a specific dev build. Tear down with `kind delete cluster --name release-canary-local`.
Swap `0.0.0-dev` for `0.0.0-dev.<sha>` and `:dev` for `:<sha>` to pin to a specific dev build. Tear down with `kind delete cluster --name release-canary-local`.

Loopback registration auto-derives the gateway name to `openshell` if `--name` is omitted, which collides with the `install.sh`-installed local gateway — always pass `--name kind` (or another distinct name) when registering in addition to a local install.

Expand All @@ -142,7 +150,7 @@ Loopback registration auto-derives the gateway name to `openshell` if `--name` i
| `ubuntu-snap-system-docker` fails during the prover checks | The prover artifact is missing or packaged for the wrong architecture, `openshell.prover` is not exposed, missing the alias to `openshell-prover`, or not permitted to read the test policies, or its solver linkage is not runnable. | The `Install and check status` and `Check a policy boundary with the snap prover` steps, plus `snap info openshell` and `snap connections openshell`. |
| `ubuntu-snap-docker-preflight` unexpectedly succeeds | The installer no longer fails before installing the OpenShell snap when Docker is absent or supplied by the Docker snap. | Inspect `install.log`, `docker-snap.log`, `snap list`, and snapd changes. |
| `kubernetes` job fails on `helm install --wait` | Chart did not deploy in 5 min — usually image pull failure or readiness probe failing. | "Diagnostics on failure" step dumps `helm status`, manifest, pod describe, pod logs. |
| `kubernetes` job fails on `kubectl wait` | Gateway pod stuck `CrashLoopBackOff` or `ImagePullBackOff`. | Diagnostics dump; check `:dev` image existence at `ghcr.io/nvidia/openshell/gateway`. |
| `kubernetes` job fails on `kubectl wait` | Gateway pod stuck `CrashLoopBackOff` or `ImagePullBackOff`. | Diagnostics dump; check that `ghcr.io/ai-openshell/openshell/gateway:dev` was loaded into kind. |
| `kubernetes` job fails on `openshell gateway add` or `status` | Port-forward not reachable, or CLI/gateway proto mismatch. | `port-forward.log` and `openshell gateway list` in the diagnostics dump. |

The `kubernetes` job's diagnostics step (only runs `if: failure()`) emits, in order: helm status, rendered manifest, `kubectl get all`, pod descriptions, pod logs (200 lines per container), port-forward log, gateway list, CLI version. Read it top-to-bottom — most failures fall out by the manifest or pod logs.
Expand Down
25 changes: 21 additions & 4 deletions .github/actions/release-helm-oci/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,14 @@ inputs:
release-kind:
description: "dev or public — controls the duplicate-version guard."
required: true
registry-namespace:
description: "GHCR namespace without the oci:// prefix."
required: false
default: ghcr.io/nvidia/openshell
image-registry:
description: "Override global.image.registry in the packaged gateway chart."
required: false
default: ""
pin-sha:
description: >
Full commit SHA. When set on dev releases, a second chart is also pushed
Expand Down Expand Up @@ -48,6 +56,7 @@ runs:
env:
CHART_VERSION: ${{ inputs.chart-version }}
APP_VERSION: ${{ inputs.app-version }}
IMAGE_REGISTRY: ${{ inputs.image-registry }}
shell: bash
run: |
set -euo pipefail
Expand All @@ -59,6 +68,10 @@ runs:
sed -i "s/^version:.*/version: ${CHART_VERSION}/" "${chart_dir}/Chart.yaml"
sed -i "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" "${chart_dir}/Chart.yaml"
done
if [[ -n "${IMAGE_REGISTRY}" ]]; then
sed -i "s|^ registry: ghcr.io/nvidia$| registry: ${IMAGE_REGISTRY}|" "${GATEWAY_CHART_DIR}/values.yaml"
grep -Fxq " registry: ${IMAGE_REGISTRY}" "${GATEWAY_CHART_DIR}/values.yaml"
fi
echo "gateway_chart_dir=${GATEWAY_CHART_DIR}" >> "$GITHUB_OUTPUT"
echo "workspace_chart_dir=${WORKSPACE_CHART_DIR}" >> "$GITHUB_OUTPUT"
echo "chart_version=${CHART_VERSION}" >> "$GITHUB_OUTPUT"
Expand All @@ -70,17 +83,18 @@ runs:
if: inputs.release-kind == 'public'
env:
CHART_VERSION: ${{ steps.prep.outputs.chart_version }}
REGISTRY_NAMESPACE: ${{ inputs.registry-namespace }}
shell: bash
run: |
set -euo pipefail
found=0
for chart in helm-chart openshell-workspace; do
OCI_CHART="oci://ghcr.io/nvidia/openshell/${chart}"
OCI_CHART="oci://${REGISTRY_NAMESPACE}/${chart}"
# Only "not found" means unpublished; any other lookup error fails
# rather than risk overwriting.
if lookup_error=$(helm show chart "${OCI_CHART}" --version "${CHART_VERSION}" 2>&1 >/dev/null); then
found=$((found + 1))
elif [[ "${lookup_error}" != *"ghcr.io/nvidia/openshell/${chart}:${CHART_VERSION}: not found"* ]]; then
elif [[ "${lookup_error}" != *"${REGISTRY_NAMESPACE}/${chart}:${CHART_VERSION}: not found"* ]]; then
echo "::error::Could not check whether ${chart}:${CHART_VERSION} is published: ${lookup_error}"
exit 1
fi
Expand Down Expand Up @@ -108,6 +122,8 @@ runs:

- name: Push Helm chart to GHCR OCI
if: steps.published.outputs.published != 'true'
env:
REGISTRY_NAMESPACE: ${{ inputs.registry-namespace }}
shell: bash
run: |
set -euo pipefail
Expand All @@ -117,7 +133,7 @@ runs:
push_chart() {
local attempt
for attempt in 1 2 3 4; do
if helm push "$1" oci://ghcr.io/nvidia/openshell; then
if helm push "$1" "oci://${REGISTRY_NAMESPACE}"; then
return 0
fi
if [ "${attempt}" -lt 4 ]; then
Expand All @@ -135,6 +151,7 @@ runs:
if: inputs.pin-sha != ''
env:
PIN_SHA: ${{ inputs.pin-sha }}
REGISTRY_NAMESPACE: ${{ inputs.registry-namespace }}
GATEWAY_CHART_DIR: ${{ steps.prep.outputs.gateway_chart_dir }}
WORKSPACE_CHART_DIR: ${{ steps.prep.outputs.workspace_chart_dir }}
shell: bash
Expand All @@ -144,7 +161,7 @@ runs:
push_chart() {
local attempt
for attempt in 1 2 3 4; do
if helm push "$1" oci://ghcr.io/nvidia/openshell; then
if helm push "$1" "oci://${REGISTRY_NAMESPACE}"; then
return 0
fi
if [ "${attempt}" -lt 4 ]; then
Expand Down
18 changes: 17 additions & 1 deletion .github/workflows/release-canary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -381,6 +381,9 @@ jobs:
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
runs-on: ubuntu-latest
timeout-minutes: 20
permissions:
contents: read
packages: read
env:
KIND_CLUSTER_NAME: release-canary-${{ github.run_id }}
RELEASE_NAME: openshell
Expand Down Expand Up @@ -408,10 +411,23 @@ jobs:
- name: Install Agent Sandbox controller
run: bash e2e/support/install-agent-sandbox.sh

- name: Pull dev runtime images into kind
env:
GHCR_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
printf '%s' "$GHCR_TOKEN" | docker login ghcr.io -u "$GITHUB_ACTOR" --password-stdin
printf '%s' "$GHCR_TOKEN" | helm registry login ghcr.io -u "$GITHUB_ACTOR" --password-stdin
for component in gateway sandbox supervisor; do
image="ghcr.io/ai-openshell/openshell/${component}:dev"
docker pull "$image"
kind load docker-image "$image" --name "$KIND_CLUSTER_NAME"
done

- name: Install OpenShell Helm chart from GHCR OCI
run: |
set -euo pipefail
helm install "$RELEASE_NAME" oci://ghcr.io/nvidia/openshell/helm-chart \
helm install "$RELEASE_NAME" oci://ghcr.io/ai-openshell/openshell/helm-chart \
--version 0.0.0-dev \
--namespace "$RELEASE_NAMESPACE" --create-namespace \
--set server.disableTls=true \
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/release-dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -553,6 +553,8 @@ jobs:
chart-version: 0.0.0-dev
app-version: dev
release-kind: dev
registry-namespace: ghcr.io/ai-openshell/openshell
image-registry: ghcr.io/ai-openshell
pin-sha: ${{ github.sha }}

publish-fern-docs:
Expand Down
5 changes: 5 additions & 0 deletions skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -513,6 +513,11 @@ When `userns` is configured (e.g. `userns = "auto"` or `userns = "keep-id"`):

### Step 6: Check Kubernetes Helm Gateways

For a dev Helm install, verify that the cluster has pull access to the
`ghcr.io/ai-openshell/openshell` runtime images, or that matching images were
preloaded on the nodes. An `ImagePullBackOff` after a successful chart download
can indicate that Helm had registry access but the cluster did not.

```bash
helm -n openshell status openshell
helm -n openshell get values openshell
Expand Down
Loading