Skip to content

feat: kubernetes runner with workflow steps as native pods - #40

Draft
hllvc wants to merge 48 commits into
mainfrom
feat/k8s-run-controller
Draft

hllvc wants to merge 48 commits into
mainfrom
feat/k8s-run-controller

Conversation

@hllvc

@hllvc hllvc commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a Kubernetes runner prototype under k8s/: Private Runners as Kubernetes pods, installed with a Helm chart and scaled by a small Go controller. The run controller still runs as an ECS task through ECS Anywhere, but workflow steps run as their own Kubernetes pods. Prototype only, not supported for customer use; it needs a run controller release with step broker support, which hasn't shipped yet.

Changes

  • Runner pod (k8s/runner.sh, k8s/Dockerfile): registration via main.sh, nested Docker for run controllers, SSM and ECS agents, drain and park through SG, Docker cache pruning, re-registration when the advertised size changes.
  • Step broker (k8s/scaler/internal/steps, Go): runs each step as a pod on the runner's node, mounting the run's workspace through subPath; a step wrapper turns exit codes 11/12 into Completed pods; finished pods keep an outcome label for an hour.
  • Guard: a Docker authorization plugin refuses step containers in the nested daemon, so a run controller without broker support fails clearly instead of running steps there.
  • Scaler (k8s/scaler, Go): parks idle runners, wakes or adds runners on queued workflows, reaps long-parked ones; nudges node autoscalers that won't scale up for pods with persistent disks.
  • Helm chart (k8s/chart/sg-runner, Kubernetes 1.29+): size presets, broker as a native sidecar, optional NetworkPolicy, uninstall hook that deregisters every runner.
  • CI (.github/workflows/k8s.yml): Go tests, chart lint and kubeconform, multi-arch image builds, Trivy, and a kind smoke install with a broker stage.
  • main.sh (affects VM runners too): private IP detection from the outbound route, and error reports no longer retry themselves when the API is down.

Testing

  • Go unit tests, bats tests, chart checks, kind smoke (including the broker and guard)
  • End to end on QA from OrbStack: plan/apply, approval and resume, early finish, failing step, cancel, timeout, bad image, concurrent runs, drain mid-step, guard refusing an unsupported run controller
  • CI on GitHub (never run there yet)
  • GKE/EKS: NetworkPolicy enforcement, multi-node disks, node autoscaling sized for steps

Risks

  • The main.sh changes reach VM runners: private IP detection and error-report handling. Covered by new bats tests.

Deployment Notes

  • Release ordering: the run controller change must ship before Kubernetes runners can run workflows.

hllvc added 30 commits October 12, 2026 22:10
# Conflicts:
#	k8s/Makefile
#	k8s/README.md
#	k8s/manifests/scaler.yaml
@hllvc hllvc self-assigned this Oct 14, 2026
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant