Adding CAIOS example pod to SkyPilot reference architectures

[ntnadkarni]

Added file to /skypilot/config-examples for a test pod that demonstrates CAIOS integration and runs a quick test on reading, writing, and listing. Changes added to README.md

[narenandu]

Is it worth adding sample AWS CONFIG file to show the format? Possibly in README.md or as a .dotfile? I find it useful when dealing with AI Object Storage. Something on the lines of

[default]
endpoint_url = https://cwobject.com
s3 =
   addressing_style = virtual
region = us-west-13b
output = json

[ntnadkarni]

@tmadhyastha-cw has a file which configures credentials for storage in https://github.com/coreweave/reference-architecture/blob/tmadhyastha/caios-credential-setup/storage/caios-credentials/configure_caios_credentials.sh perhaps I can add a pre-requisite to run this file first?

[narenandu]

Cross-linking existing resource in the same repo works too 👍

[radu-malliu]

fwiw this should be the preferred mechanism for getting CAIOS credentials once it's released, just requires a service account being specified for pods: https://github.com/coreweave/kabinet-charts/blob/main/charts/pod-identity-webhook/README.md
AWS SDK-based tools that can use creds files can also use the variables injected by the webhook to get short-lived, auto-rotated creds.

[tmadhyastha-cw]

@radu-malliu I do not understand how this webhook mechanism helps. Can you help explain how you would envision this flow changing with this? CAIOS credentials are currently handled by skypilot in the AWS config style format, and are copied automatically to the pods created by skypilot.

[tmadhyastha-cw]

Regarding cross-linking, yes I think one of us should definitely put a sample config file in the examples - I think it would make sense to go here. You can say, if you do not have a profile "cw" defined, you can run the other script to set that up.

[radu-malliu]

@tmadhyastha-cw based on this SkyPilot page, it's possible to specify a service account for pods that SkyPilot launches.
The mutating webhook I mentioned is responsible for injecting an OIDC token issued by the cluster and env variables for an endpoint where such token can be exchanged for CAIOS credentials, assuming the setup described in the webhook docs exists. The mechanism for the exchange is built into the AWS SDK. In other words, if the env vars are there, there is a credential provider in the provider chain which can be called to obtain credentials.

[ntnadkarni]

@radu-malliu is this released? Would be happy to test it out with SP if so. It is neat, and would simplify the credential configuration process greatly.