Skip to content

feat(chart): let object-store workloads mount extra volumes - #872

Draft
scalejeff wants to merge 1 commit into
mainfrom
feat/inference-pod-extra-volumes
Draft

feat(chart): let object-store workloads mount extra volumes#872
scalejeff wants to merge 1 commit into
mainfrom
feat/inference-pod-extra-volumes

Conversation

@scalejeff

@scalejeff scalejeff commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

The pods the service template renders have a fixed volume list. The existing hooks do not reach them: extraVolumes/extraVolumeMounts apply to the control-plane deployments, and extraPodEnvFrom carries env only. Nor can either mounted ConfigMap smuggle a file in -- config-volume uses subPath and infra-service-config-volume restricts itself to one key via items.

So a site that has to hand these workloads a FILE has no way in at all. The case that surfaced this: an on-prem S3 endpoint whose certificate is signed by the customer's own CA. Verification needs AWS_CA_BUNDLE (and its requests/openssl equivalents) pointing at a real path, and there was no way to put the CA at one short of baking it into the image.

extraPodVolumes and extraPodVolumeMounts are added everywhere the template renders something that talks to object storage: the endpoint Deployment, both halves of a LeaderWorkerSet, batch-job-orchestration, the docker-image batch jobs, and the input-downloader initContainer -- which takes ${S3_FILE} as its argument and so fails against a private CA exactly like the rest.

In every case the block is the FIRST entry under volumeMounts/volumes, never inside a conditional. That is load-bearing rather than stylistic: the pod-level volume is unconditional, so a mount nested under an unrelated if yields a CA present in the pod but absent from the container that reads it -- the exact failure this change exists to prevent, and one that hides whenever that condition happens to be true.

The LWS worker and both batch-job templates also gain extraPodEnvFrom, which they did not have. A CA file with no AWS_CA_BUNDLE beside it is dead weight, so without this the volume would land somewhere nothing reads it. That is a behaviour change for anyone already setting extraPodEnvFrom -- those workloads will now receive it too, which is almost certainly what was wanted, but it is a change and reviewers should say so if not.

An earlier revision scoped coverage to "inference pods" on the grounds that it matched extraPodEnvFrom's existing boundary. That was a consistency argument dressed up as a functional one: the batch jobs mount the same AWS config and read the same store, so they break the same way. Coverage now follows what reads object storage rather than what the older hook happened to touch.

Both new values default to [] and render nothing when unset.

Verified against values_circleci.yaml, and again with config.values nulled so config_values is falsy: the embedded templates parse identically in both modes and against the unmodified chart (17 parse, 15 fail on unsubstituted ${STORAGE_DICT}/${NODE_PORT_DICT} placeholders throughout), and the volume, mount and envFrom reach every rendered template including the LWS worker and the input-downloader initContainer. Rendering only the config-populated case would have missed a mount nested under $config_values.

Pull Request Summary

What is this PR changing? Why is this change being made? Any caveats you'd like to highlight? Link any relevant documents, links, or screenshots here if applicable.

Test Plan and Usage Guide

How did you validate that your PR works correctly? How do you run or demo the code? Provide enough detail so a reviewer can reasonably reproduce the testing procedure. Paste example command line invocations if applicable.

The pods the service template renders have a fixed volume list. The existing
hooks do not reach them: extraVolumes/extraVolumeMounts apply to the
control-plane deployments, and extraPodEnvFrom carries env only. Nor can either
mounted ConfigMap smuggle a file in -- config-volume uses subPath and
infra-service-config-volume restricts itself to one key via items.

So a site that has to hand these workloads a FILE has no way in at all. The
case that surfaced this: an on-prem S3 endpoint whose certificate is signed by
the customer's own CA. Verification needs AWS_CA_BUNDLE (and its
requests/openssl equivalents) pointing at a real path, and there was no way to
put the CA at one short of baking it into the image.

extraPodVolumes and extraPodVolumeMounts are added everywhere the template
renders something that talks to object storage: the endpoint Deployment, both
halves of a LeaderWorkerSet, batch-job-orchestration, the docker-image batch
jobs, and the input-downloader initContainer -- which takes ${S3_FILE} as its
argument and so fails against a private CA exactly like the rest.

In every case the block is the FIRST entry under volumeMounts/volumes, never
inside a conditional. That is load-bearing rather than stylistic: the pod-level
volume is unconditional, so a mount nested under an unrelated `if` yields a CA
present in the pod but absent from the container that reads it -- the exact
failure this change exists to prevent, and one that hides whenever that
condition happens to be true.

The LWS worker and both batch-job templates also gain extraPodEnvFrom, which
they did not have. A CA file with no AWS_CA_BUNDLE beside it is dead weight, so
without this the volume would land somewhere nothing reads it. That is a
behaviour change for anyone already setting extraPodEnvFrom -- those workloads
will now receive it too, which is almost certainly what was wanted, but it is a
change and reviewers should say so if not.

An earlier revision scoped coverage to "inference pods" on the grounds that it
matched extraPodEnvFrom's existing boundary. That was a consistency argument
dressed up as a functional one: the batch jobs mount the same AWS config and
read the same store, so they break the same way. Coverage now follows what
reads object storage rather than what the older hook happened to touch.

Both new values default to [] and render nothing when unset.

Verified against values_circleci.yaml, and again with config.values nulled so
$config_values is falsy: the embedded templates parse identically in both modes
and against the unmodified chart (17 parse, 15 fail on unsubstituted
${STORAGE_DICT}/${NODE_PORT_DICT} placeholders throughout), and the volume,
mount and envFrom reach every rendered template including the LWS worker and
the input-downloader initContainer. Rendering only the config-populated case
would have missed a mount nested under $config_values.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant