Add selectable Dagster image variant (base/hardened) with restructured image layout - #376
Open
RonaldHensbergen wants to merge 6 commits into
Open
Add selectable Dagster image variant (base/hardened) with restructured image layout#376RonaldHensbergen wants to merge 6 commits into
RonaldHensbergen wants to merge 6 commits into
Conversation
…ging fields
CDS-SEC-070 ("Secret appears in command args or log configuration") had
scope: ["none"], so it was never dispatched despite enabled: true. Its
valueRegex was also malformed ("(?i)(--******"), an invalid/unbalanced
regex that nothing ever compiled or exercised -- exactly the kind of
silent dead-rule regression #297 warned about, and the rule that should
have caught the Vault dev-root-token-as-command-arg issue fixed in #290.
- Add a "rendered-compose" scope to cli/security.py: run_security_validation
now does a best-effort plan+render of the profile and flattens each
service's command/entrypoint/logging fields, so rules can inspect where a
module's implementation template actually places a secret-bearing value
(only visible after ${config.*} template expressions resolve into their
final Compose "${CDS_*}" placeholders). Plan/render failures are handled
silently here since the separate plan/render stages already report them.
- Re-scope CDS-SEC-070 to "rendered-compose", fix its pathPatterns to match
list-shaped command/entrypoint/logging fields, and fix the broken
valueRegex to actually catch secret-like command-line flags.
- Add "rendered-compose" to the rule-schema.json scope enum.
- Add regression fixtures mirroring the real PR #290 bug/fix (a fixture
module passing a secret via "command:", and a safe variant passing it via
"environment:"), and tests proving CDS-SEC-070 now fires for the
vulnerable shape and stays silent for the safe one.
- Update the #297 deferred-rule documentation test: CDS-SEC-070 is no
longer scope: ["none"].
CDS-SEC-006, 030, 032, 050-054, and 071 remain scope: ["none"] as
documented by #303 -- they need rendered-artifact/CLI-output scanning or
file-permission checks that are out of scope here, and 050-052/054 are
already enforced separately by cli/image_verification.py.
Addresses #297
- Add a third CDS-SEC-070 match branch to catch bare positional
${CDS_*}/${config.*}/${secrets.*} placeholders, not just "--flag="
style command arguments.
- Extend pathPatterns/coverage to include healthcheck.* alongside
command/entrypoint/logging, since healthcheck probes leak secrets
the same way.
- Map rendered-compose findings back to the owning CDS module id
(via a new _map_service_to_module helper reusing the renderer's
service-naming rule) instead of the raw Compose service name, which
only coincidentally matched module ids in the fixture.
- Replace the silent `except Exception: return None` in
_try_render_compose_for_scan with a warning diagnostic (W096) so
unexpected internal errors are surfaced instead of vanishing as zero
findings.
- Let callers pass an already-computed plan/rendered Compose YAML into
run_security_validation(), and wire cli/main.py's `cds test` to do so,
so the profile isn't planned and rendered twice per run.
- Expand the regression fixture to exercise all three leak surfaces
(flag-style, bare positional, healthcheck) and assert exact
path/value/module attribution instead of loose substring checks.
- Add coverage for the plan/render reuse behavior and for the new
warning-diagnostic path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- CDS-SEC-070: remove the two match.any branches that could never fire
against rendered output (keyRegex only tests the final path segment,
which is a list index or ordinary leaf key on rendered command/
entrypoint/healthcheck/logging paths; and ${secrets.*}/${config.*}
resolve or fail before this rule ever runs), leaving only the single
working "${CDS_" branch.
- cds test: pass skip_self_plan_render=True to run_security_validation
when its own plan/render stages already failed, so
_try_render_compose_for_scan doesn't retry (and re-fail) the same
build_plan()/render_compose() calls a second time.
- _try_render_compose_for_scan now emits a W096 warning diagnostic when
planning or rendering fails (not just on unexpected exceptions), so
`cds security` (which has no separate plan/render stage) can no
longer silently report "No security findings" for a profile that
never actually got scanned by rendered-compose-scoped rules.
- run_security_validation now only plans+renders the profile when some
enabled rule in the active rule set actually declares the
"rendered-compose" scope, avoiding unconditional plan+render overhead
on every security scan when no such rule applies.
- Add regression tests for each of the above.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduce spec.configSchema.image.variant on the Dagster module
(enum: dagster [default], alpine-dagster) so profiles can opt into a
hardened Alpine-based build instead of the default Debian/python:3.14-slim
image. The user-code, dagster-webserver, and dagster-daemon build blocks
now select their Dockerfile via
images/${config.image.variant}/Dockerfile.
Also teach cli/image_updates.py's collect_module_images/
find_images_in_compose to resolve ${config.*} expressions in
build.dockerfile against configSchema defaults, so static tooling
(base-image extraction, dependency-update checks) can still locate the
default variant's Dockerfile now that the path is templated.
Adds the (currently stub/in-progress) images/alpine-dagster/Dockerfile
as the alpine-dagster variant's build directory. Follow-up work (image
hardening, publish workflow wiring, signed-images fixture, docs) is
tracked in #372.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move images/dagster/Dockerfile and images/alpine-dagster/Dockerfile into
images/dagster/base/Dockerfile and images/dagster/hardened/Dockerfile
respectively, and images/superset/Dockerfile into
images/superset/base/Dockerfile, so shared build support files
(entrypoint, config generation, healthcheck, workspace, requirements,
README) live once per module while each Dockerfile variant gets its own
directory. Remove the now-empty images/alpine-dagster/ placeholder.
- modules/orchestration/dagster/module.yaml: image.variant enum
dagster/alpine-dagster -> base/hardened (default base); dockerfile
path -> images/dagster/${config.image.variant}/Dockerfile.
- modules/bi/superset/module.yaml: dockerfile -> images/superset/base/Dockerfile.
- Makefile and CI workflows (docker-smoke-test, image-security-scan,
publish-images): build-context detection and image discovery updated
for the nested variant layout instead of exact old paths.
- publish-images.yml: base and hardened variants publish to the same
image name/repository, distinguished by tag (latest/<ver>-<date> for
base, hardened-latest/hardened-<ver>-<date> for hardened) on both
GHCR and Docker Hub.
- Updated tests (test_dagster_hardening, test_python_version_alignment,
test_render_example_profile, test_superset_hardening,
test_publish_images_workflow) and docs (docs/modules.md,
docs/image-signing.md, images/dagster/README.md) for the new paths.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RonaldHensbergen
force-pushed
the
feat/dagster-image-variant
branch
from
August 6, 2026 08:53
4fe6e42 to
5671d8b
Compare
…text The hardened variant's Dockerfile was a stale placeholder that never matched the repository's build conventions: it expected a flat build context (COPY requirements.txt ., COPY your_definitions.py .) instead of the root-relative context used by every other image in this repo (build.context: ../../../, COPY images/dagster/requirements*.txt ./), and referenced a your_definitions.py file that doesn't exist anywhere in the repository. This broke CI on PR #376 in both the 'Image Security Scan' (scan (dagster, hardened, ...)) and 'Docker Runtime Smoke Test' (build-images) jobs with 'failed to calculate checksum ... "/your_definitions.py": not found'. Rebuild the hardened Dockerfile as an Alpine-based (python:3.14-alpine) multi-stage build mirroring images/dagster/base/Dockerfile's structure and shared build-support files (entrypoint.sh, generate_config.py, healthcheck.py, workspace.yaml, dagster.yaml.j2), so both variants build from the same repo-root context and only differ in base image. Also fixed the dagster group/user GID/UID (999 collides with Alpine's built-in 'ping' group; changed to 1001). Verified: built the image locally, dagster imports and the healthcheck script runs correctly as the dagster user, and Trivy reports zero HIGH/CRITICAL vulnerabilities. Full unittest suite (426 tests) passes and ruff is clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RonaldHensbergen
marked this pull request as ready for review
August 6, 2026 09:38
RonaldHensbergen
marked this pull request as draft
August 6, 2026 09:42
RonaldHensbergen
marked this pull request as ready for review
August 6, 2026 10:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces a selectable Dagster image variant and restructures both
Dagster and Superset image directories into per-variant subfolders,
addressing #372, #373, and #374.
spec.configSchema.image.variantoption (
base/hardened) tomodules/orchestration/dagster/module.yamlso the user-code, webserver, and daemon services can build from either
the default Debian (
python:3.14-slim) image or an Alpine-basedhardened image optimized for a minimal attack surface.
--hardenedCLI flag (Add --hardened CLI flag to select the Alpine-hardened Dagster image variant #373): related CLI wiring for selecting thehardened variant.
images/dagster/Dockerfile→images/dagster/base/Dockerfileandimages/alpine-dagster/Dockerfile→images/dagster/hardened/Dockerfile(removing the
images/alpine-dagster/placeholder), andimages/superset/Dockerfile→images/superset/base/Dockerfile, soshared build support files (entrypoint, config generation, healthcheck,
workspace, requirements, README) live once per module while each
Dockerfile variant gets its own directory:
Changes
modules/orchestration/dagster/module.yaml:image.variantenumbase/hardened(defaultbase);dockerfile:path →images/dagster/${config.image.variant}/Dockerfile.modules/bi/superset/module.yaml:dockerfile:→images/superset/base/Dockerfile.Makefile,.github/workflows/docker-smoke-test.yml:build-context detection matches any
images/dagster/*orimages/superset/*Dockerfile instead of an exact-path list..github/workflows/image-security-scan.yml,.github/workflows/publish-images.yml: image discovery supports boththe flat (
images/<name>/Dockerfile) and nested(
images/<name>/<variant>/Dockerfile) layouts.publish-images.yml: base and hardened variants publish to thesame image name/repository (both GHCR and Docker Hub), distinguished by
tag —
latest/<ver>-<date>forbase,hardened-latest/hardened-<ver>-<date>forhardened.test_dagster_hardening.py,test_python_version_alignment.py,test_render_example_profile.py,test_superset_hardening.py,test_publish_images_workflow.pyupdated for the new paths/discoverylogic.
docs/modules.md,docs/image-signing.md,images/dagster/README.mdupdated to document the new layout and tags.Testing
python -m unittest discover -s tests -p "test_*.py"— 420 tests, allpassing.
ruff check .,yamllint .,npx markdownlint-cli@0.49.0on editeddocs — clean.
Why draft
This branch currently diverges from
mainby several unrelated commits(the broader Dagster image-variant work predates recent
mainhistory).Opening as a draft to track the work and enable review/CI feedback while
the branch is rebased/reconciled with
mainbefore it's ready to merge.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com