feat(chart): let object-store workloads mount extra volumes - #872
Draft
scalejeff wants to merge 1 commit into
Draft
feat(chart): let object-store workloads mount extra volumes#872scalejeff wants to merge 1 commit into
scalejeff wants to merge 1 commit into
Conversation
The pods the service template renders have a fixed volume list. The existing
hooks do not reach them: extraVolumes/extraVolumeMounts apply to the
control-plane deployments, and extraPodEnvFrom carries env only. Nor can either
mounted ConfigMap smuggle a file in -- config-volume uses subPath and
infra-service-config-volume restricts itself to one key via items.
So a site that has to hand these workloads a FILE has no way in at all. The
case that surfaced this: an on-prem S3 endpoint whose certificate is signed by
the customer's own CA. Verification needs AWS_CA_BUNDLE (and its
requests/openssl equivalents) pointing at a real path, and there was no way to
put the CA at one short of baking it into the image.
extraPodVolumes and extraPodVolumeMounts are added everywhere the template
renders something that talks to object storage: the endpoint Deployment, both
halves of a LeaderWorkerSet, batch-job-orchestration, the docker-image batch
jobs, and the input-downloader initContainer -- which takes ${S3_FILE} as its
argument and so fails against a private CA exactly like the rest.
In every case the block is the FIRST entry under volumeMounts/volumes, never
inside a conditional. That is load-bearing rather than stylistic: the pod-level
volume is unconditional, so a mount nested under an unrelated `if` yields a CA
present in the pod but absent from the container that reads it -- the exact
failure this change exists to prevent, and one that hides whenever that
condition happens to be true.
The LWS worker and both batch-job templates also gain extraPodEnvFrom, which
they did not have. A CA file with no AWS_CA_BUNDLE beside it is dead weight, so
without this the volume would land somewhere nothing reads it. That is a
behaviour change for anyone already setting extraPodEnvFrom -- those workloads
will now receive it too, which is almost certainly what was wanted, but it is a
change and reviewers should say so if not.
An earlier revision scoped coverage to "inference pods" on the grounds that it
matched extraPodEnvFrom's existing boundary. That was a consistency argument
dressed up as a functional one: the batch jobs mount the same AWS config and
read the same store, so they break the same way. Coverage now follows what
reads object storage rather than what the older hook happened to touch.
Both new values default to [] and render nothing when unset.
Verified against values_circleci.yaml, and again with config.values nulled so
$config_values is falsy: the embedded templates parse identically in both modes
and against the unmodified chart (17 parse, 15 fail on unsubstituted
${STORAGE_DICT}/${NODE_PORT_DICT} placeholders throughout), and the volume,
mount and envFrom reach every rendered template including the LWS worker and
the input-downloader initContainer. Rendering only the config-populated case
would have missed a mount nested under $config_values.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The pods the service template renders have a fixed volume list. The existing hooks do not reach them: extraVolumes/extraVolumeMounts apply to the control-plane deployments, and extraPodEnvFrom carries env only. Nor can either mounted ConfigMap smuggle a file in -- config-volume uses subPath and infra-service-config-volume restricts itself to one key via items.
So a site that has to hand these workloads a FILE has no way in at all. The case that surfaced this: an on-prem S3 endpoint whose certificate is signed by the customer's own CA. Verification needs AWS_CA_BUNDLE (and its requests/openssl equivalents) pointing at a real path, and there was no way to put the CA at one short of baking it into the image.
extraPodVolumes and extraPodVolumeMounts are added everywhere the template renders something that talks to object storage: the endpoint Deployment, both halves of a LeaderWorkerSet, batch-job-orchestration, the docker-image batch jobs, and the input-downloader initContainer -- which takes ${S3_FILE} as its argument and so fails against a private CA exactly like the rest.
In every case the block is the FIRST entry under volumeMounts/volumes, never inside a conditional. That is load-bearing rather than stylistic: the pod-level volume is unconditional, so a mount nested under an unrelated
ifyields a CA present in the pod but absent from the container that reads it -- the exact failure this change exists to prevent, and one that hides whenever that condition happens to be true.The LWS worker and both batch-job templates also gain extraPodEnvFrom, which they did not have. A CA file with no AWS_CA_BUNDLE beside it is dead weight, so without this the volume would land somewhere nothing reads it. That is a behaviour change for anyone already setting extraPodEnvFrom -- those workloads will now receive it too, which is almost certainly what was wanted, but it is a change and reviewers should say so if not.
An earlier revision scoped coverage to "inference pods" on the grounds that it matched extraPodEnvFrom's existing boundary. That was a consistency argument dressed up as a functional one: the batch jobs mount the same AWS config and read the same store, so they break the same way. Coverage now follows what reads object storage rather than what the older hook happened to touch.
Both new values default to [] and render nothing when unset.
Verified against values_circleci.yaml, and again with config.values nulled so config_values is falsy: the embedded templates parse identically in both modes and against the unmodified chart (17 parse, 15 fail on unsubstituted ${STORAGE_DICT}/${NODE_PORT_DICT} placeholders throughout), and the volume, mount and envFrom reach every rendered template including the LWS worker and the input-downloader initContainer. Rendering only the config-populated case would have missed a mount nested under $config_values.
Pull Request Summary
What is this PR changing? Why is this change being made? Any caveats you'd like to highlight? Link any relevant documents, links, or screenshots here if applicable.
Test Plan and Usage Guide
How did you validate that your PR works correctly? How do you run or demo the code? Provide enough detail so a reviewer can reasonably reproduce the testing procedure. Paste example command line invocations if applicable.