atelet: pull image credentials from the kubelet credential provider

[Benjamin Elder (BenTheElder)]

atelet authenticated image pulls with Google application default credentials, applied to gcr.io and pkg.dev only. That compiles one cloud's registry story into the binary and leaves every other cloud unserved.

Every managed node already answers "which credentials does this registry want": the kubelet ships a credential provider executable and a CredentialProviderConfig naming the registries it handles. Reuse that contract instead. internal/credentialprovider is an authn.Keychain that speaks the kubelet's exec-plugin protocol -- config parsing and validation, the kubelet's image match rules, the request/response exchange over stdio, and per-cacheKeyType response caching. atelet resolves credentials through it when --image-credential-provider-config and --image-credential-provider-bin-dir are set (the same flags the kubelet takes), and the DaemonSet mounts the node's plugin dir and config read-only.

The result is registry-agnostic: the same atelet authenticates to Artifact Registry on GKE and to ECR on EKS, given each node's own plugin. The GCP application default credentials path stays for installs that do not set the new flags.

POC for proposal at https://docs.google.com/document/d/1rWkmIWTAo_MUGleZ48k3py9jmL7Yfjdf1qffdyiLJCw/edit?resourcekey=0-z9N8Ee108biqI4CsTTjQUg&tab=t.0#heading=h.6vr9xodr524z (shared with the mailinglist from the README.md for access)

[Steven Jin (Stevenjin8)]

I have two thoughts:

1. In various discussions, I've seen the credential-provider-bin-dir be described as a host mount. I have nothing against it being a host mount, I just don't want to make it a requirement. For example, AKS' credential provider is dynamically linked:

# ldd acr-credential-provider
	linux-vdso.so.1 (0x00007fffef38b000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe670585000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe67079b000)

and the last thing any of us wants to deal with is libc compatibility shenanigans.

2. In the CredentialProviderRequest we pass into the credential provider, we are missing serviceAccountToken and serviceAccountAnnotations. serviceAccountToken isn't too bad. It's usually a file mounted on the pod, so we can just pass in the filename as an arg.

serviceAccountAnnotations is much more annoying but still important. Ideally, we look it up on the kubeapiserver and maybe have some caching in place. It puts the api server in the path, but I think its fine because it only happens on golden image creation (and we can cache/have watchers)?

[Benjamin Elder (BenTheElder)]

In various discussions, I've seen the credential-provider-bin-dir be described as a host mount. I have nothing against it being a host mount, I just don't want to make it a requirement. For example, AKS' credential provider is dynamically linked:

Yeah, I expected this to happen with some providers, the most obvious workaround I have to suggest is a custom atelet image with the appropriate binary + libraries added directly to the image. That will add some friction to deploying to that environment but it doesn't seem terrible.

We can make the mount customizable as well, and we can consider using a minimal libc base image.

Hopefully more vendors can leverage GOFIPS140 for static binaries in the near future ...

serviceAccountAnnotations is much more annoying but still important. Ideally, we look it up on the kubeapiserver and maybe have some caching in place. It puts the api server in the path, but I think its fine because it only happens on golden image creation (and we can cache/have watchers)?

We really don't want apiserver in the path, or the need to have that many objects, like, we want O(100ms) end to end spawning lightweight actors, churning them multiple times per second. We're also targeting ~1B actors with ~1M active (which will require some other changes ...).

We absolutely do not have that today, e.g. the sandbox setup is too slow, but we're treating every performance gap as a bug to burn down. We're basically using Kubernetes for compute. Actors do not have KSA.

This also needs to be not just on golden image creation, we have to pull the image again to another node if the actor resumes there.

The current inbuilt GCP support is atelet wide, not per-actor, so this is not a regression on that front, it just opens it up to more vendors without compiling them in (and we can look at dropping the built-in GCP support).