From 30bd129d59c2a92f0d6febd311ce7bbc20c37bd1 Mon Sep 17 00:00:00 2001 From: Tomasz Janiszewski Date: Tue, 10 Nov 2026 13:42:57 +0100 Subject: [PATCH 1/4] ci: pin functional tests to the minimal vulnerability bundle Loading the full vulnerability feed adds Scanner initialization work to each Jenkins plugin test run and makes vulnerability-dependent behavior depend on changing production definitions. Use the frozen CI dataset introduced by [StackRox PR #23544](https://github.com/stackrox/stackrox/pull/23544) to reduce that work and keep vulnerability inputs reproducible. ## Deployment - Pin SCANNER_V4_MATCHER_VULNERABILITY_VERSION to the published bundle's content hash in the Tests workflow. - Pass the version through roxie's central.spec.customize.envVars so Scanner requests the pinned definitions through Central. Exporting the variable on the GitHub runner alone would not configure the deployment. - Keep vulnerability-based Matcher readiness, --early-readiness=false, and the existing 60-minute Central wait. - Apply the pin to all Tests workflow triggers, including scheduled runs. This workflow invokes roxie directly and does not use StackRox's shared full-bundle/nightly selection helper. The artifact is already published, so this configuration does not require the unmerged PR's helper scripts in the StackRox checkout: - [Immutable vulnerability bundle](https://definitions.stackrox.io/v4/vulnerability-bundles/ci-minimal-bab3cec6e362f74bbe0c4f6a2e552ce1bac983feb836c449929f20cfd88bf4ce/vulnerabilities.zip) - [Bundle provenance, maintenance, and publication](https://github.com/stackrox/stackrox/blob/a1a20a3af3c7f68207292e0b32ea3daced54284c/scanner/updater/ci/README.md) - [Operator environment configuration example](https://github.com/stackrox/stackrox/blob/a1a20a3af3c7f68207292e0b32ea3daced54284c/operator/tests/common/central-cr.yaml) ## Test coverage The functional suite checks latest-tag policy enforcement, scan failures, and a fixable CVSS >= 7 finding for nginx-alpine:1.2.1. It does not assert production-feed CVE counts. The bundle's independent matching tests cover Alpine nginx findings with fixed versions and CVSS >= 7; the actual Jenkins image and both functional test variants still need end-to-end validation. Java unit tests use local fixtures and mocked API responses. - [Jenkins functional scenarios](https://github.com/stackrox/jenkins-plugin/blob/ad42508f1d15a156ffa9b7b0b39ebd1ae86026e0/functionaltest-jenkins-plugin/src/test/groovy/ImageScanningTest.groovy) - [Independent bundle matching expectations](https://github.com/stackrox/stackrox/blob/a1a20a3af3c7f68207292e0b32ea3daced54284c/scanner/updater/ci/generate/matching_test.go#L267) ## Validation - Downloaded the published ZIP successfully (HTTP 200, 216257 bytes) and verified SHA-256 matches the pinned version: bab3cec6e362f74bbe0c4f6a2e552ce1bac983feb836c449929f20cfd88bf4ce. - Confirmed the response includes Last-Modified. - actionlint -shellcheck= .github/workflows/tests.yaml passed. - Full actionlint reports the same pre-existing SC1090 and SC2086 ShellCheck findings as the unchanged workflow; no new findings. - git diff --check passed. - End-to-end tests and startup timing measurements were not run locally. --- .github/workflows/tests.yaml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml index 49f3f672..24847296 100644 --- a/.github/workflows/tests.yaml +++ b/.github/workflows/tests.yaml @@ -13,6 +13,8 @@ on: env: # StackRox internal certificate for central is issued for this name. ROX_INTERNAL_CENTRAL_NAME: 'central.stackrox' + # Frozen test data published by https://github.com/stackrox/stackrox/pull/23544. + SCANNER_V4_MATCHER_VULNERABILITY_VERSION: ci-minimal-bab3cec6e362f74bbe0c4f6a2e552ce1bac983feb836c449929f20cfd88bf4ce jobs: build: @@ -74,6 +76,7 @@ jobs: --set roxie.imageRegistry=quay.io/stackrox-io \ --set central.namespace=stackrox \ --set 'central.spec.customize.envVars[0].name=SCANNER_V4_MATCHER_READINESS,central.spec.customize.envVars[0].value=vulnerability' \ + --set "central.spec.customize.envVars[1].name=SCANNER_V4_MATCHER_VULNERABILITY_VERSION,central.spec.customize.envVars[1].value=$SCANNER_V4_MATCHER_VULNERABILITY_VERSION" \ --early-readiness=false \ --central-wait 60m source "$envrc_file" From e285f7d4a61458622c326b6c0b705ba12bb2603a Mon Sep 17 00:00:00 2001 From: Tomasz Janiszewski Date: Tue, 10 Nov 2026 14:11:55 +0100 Subject: [PATCH 2/4] ci: load the minimal bundle directly and preserve Scanner env vars Separate roxie --set expressions replace the envVars list rather than merging its indexed entries. Setting index 1 in a second expression drops the readiness entry at index 0, leaving null and causing Kubernetes to reject the Central resource before deployment. Pass both entries in one --set expression and document the list replacement behavior. Keep vulnerability-based Matcher readiness and the existing Central readiness wait. Set SCANNER_V4_MATCHER_VULNERABILITIES_URL to the immutable published ZIP instead of selecting a version through Central. Scanner now downloads the same pinned dataset directly from definitions.stackrox.io, bypassing Central's definitions handler. References: - [Minimal bundle implementation and provenance](https://github.com/stackrox/stackrox/pull/23544) - [Pinned vulnerability bundle](https://definitions.stackrox.io/v4/vulnerability-bundles/ci-minimal-bab3cec6e362f74bbe0c4f6a2e552ce1bac983feb836c449929f20cfd88bf4ce/vulnerabilities.zip) - [Roxie version used for validation](https://github.com/stackrox/roxie/releases/tag/v0.4.17) Validation: - Verified the roxie v0.4.17 binary against its published release checksum. - Ran its verbose dry run with the workflow's --set arguments, a fixed image tag, Kind configuration, and an empty kubeconfig. Both environment entries render correctly with no null list element. The optional cluster manifest lookup failed without a cluster; configuration rendering and the dry run completed successfully. - actionlint -shellcheck= .github/workflows/tests.yaml passed. - git diff --check passed. - Live deployment and functional tests remain unverified. --- .github/workflows/tests.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml index 24847296..2ebb9042 100644 --- a/.github/workflows/tests.yaml +++ b/.github/workflows/tests.yaml @@ -14,7 +14,7 @@ env: # StackRox internal certificate for central is issued for this name. ROX_INTERNAL_CENTRAL_NAME: 'central.stackrox' # Frozen test data published by https://github.com/stackrox/stackrox/pull/23544. - SCANNER_V4_MATCHER_VULNERABILITY_VERSION: ci-minimal-bab3cec6e362f74bbe0c4f6a2e552ce1bac983feb836c449929f20cfd88bf4ce + SCANNER_V4_MATCHER_VULNERABILITIES_URL: https://definitions.stackrox.io/v4/vulnerability-bundles/ci-minimal-bab3cec6e362f74bbe0c4f6a2e552ce1bac983feb836c449929f20cfd88bf4ce/vulnerabilities.zip jobs: build: @@ -70,13 +70,13 @@ jobs: run: | envrc_file="$(mktemp)" cd stackrox + # Keep envVars in one --set expression; separate flags replace the list. roxie deploy central \ --envrc "$envrc_file" \ --resources=tiny \ --set roxie.imageRegistry=quay.io/stackrox-io \ --set central.namespace=stackrox \ - --set 'central.spec.customize.envVars[0].name=SCANNER_V4_MATCHER_READINESS,central.spec.customize.envVars[0].value=vulnerability' \ - --set "central.spec.customize.envVars[1].name=SCANNER_V4_MATCHER_VULNERABILITY_VERSION,central.spec.customize.envVars[1].value=$SCANNER_V4_MATCHER_VULNERABILITY_VERSION" \ + --set "central.spec.customize.envVars[0].name=SCANNER_V4_MATCHER_READINESS,central.spec.customize.envVars[0].value=vulnerability,central.spec.customize.envVars[1].name=SCANNER_V4_MATCHER_VULNERABILITIES_URL,central.spec.customize.envVars[1].value=$SCANNER_V4_MATCHER_VULNERABILITIES_URL" \ --early-readiness=false \ --central-wait 60m source "$envrc_file" From 81a7c129e3c1771ab4746198201547ebb6ac138c Mon Sep 17 00:00:00 2001 From: Tomasz Janiszewski Date: Tue, 10 Nov 2026 15:09:07 +0100 Subject: [PATCH 3/4] test: align CVSS enforcement with the minimal bundle fixture The existing nginx-alpine:1.2.1 image has fixable CVE-2022-2097 with a CVSS score of 5.3 in the frozen vulnerability bundle. Its other applicable Alpine fixture records are already fixed, so the default CVSS >= 7 policy does not trigger and both Jenkins functional test variants unexpectedly report SUCCESS. Override the CVSS criterion to >= 5 in the shared test helper while preserving the fixability criterion, enforcement settings, and expected Jenkins failure. Keep the default policy name for repeated lookups and document the test-only override. Read the saved policy back and assert its threshold so a rejected or lost update cannot silently weaken coverage. This keeps the test focused on configured vulnerability-policy enforcement without changing its image, regenerating the shared bundle, or changing production policy defaults. References: - [Failing functional test run](https://github.com/stackrox/jenkins-plugin/actions/runs/42822378589/job/129850880233) - [Minimal vulnerability bundle](https://github.com/stackrox/stackrox/pull/23544) Validation: - Java 11: ./gradlew --no-daemon compileTestGroovy codenarcMain codenarcTest passed, matching the CI Java version. - git diff --check passed. - End-to-end tests were not run locally because the Central endpoint and credentials are not configured. All 14 scenarios still require CI verification, including policy-driven failures in both test variants. --- .../src/test/groovy/ImageScanningTest.groovy | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/functionaltest-jenkins-plugin/src/test/groovy/ImageScanningTest.groovy b/functionaltest-jenkins-plugin/src/test/groovy/ImageScanningTest.groovy index 00f2c47a..d3e02722 100644 --- a/functionaltest-jenkins-plugin/src/test/groovy/ImageScanningTest.groovy +++ b/functionaltest-jenkins-plugin/src/test/groovy/ImageScanningTest.groovy @@ -114,8 +114,20 @@ class ImageScanningTest extends BaseSpecification { setExclusions([]) setScope([]) } + if (policyName == "Fixable CVSS >= 7") { + // Keep the default lookup name, but use a test-only threshold for the minimal bundle. + // The image has fixable CVE-2022-2097 with CVSS 5.3. + def cvssGroup = policy.policySections.collectMany { it.policyGroups }.find { it.fieldName == "CVSS" } + assert cvssGroup != null + cvssGroup.values[0].value = ">= 5" + } restApiClient.updatePolicy(policy, policyId) - return restApiClient.getPolicy(policyId) + def updatedPolicy = restApiClient.getPolicy(policyId) + if (policyName == "Fixable CVSS >= 7") { + assert updatedPolicy.policySections.collectMany { it.policyGroups } + .find { it.fieldName == "CVSS" }.values*.value == [">= 5"] + } + return updatedPolicy } } From 35a1226982f9a297d239ce3617e252fb7e380f23 Mon Sep 17 00:00:00 2001 From: Tomasz Janiszewski Date: Tue, 10 Nov 2026 16:35:22 +0100 Subject: [PATCH 4/4] test: create a dedicated fixable CVSS >= 5 policy Overriding the default CVSS >= 7 policy leaves its name inconsistent with the threshold exercised by the Jenkins functional tests. Define a separate Fixable CVSS >= 5 policy directly and use it in both inherited test variants. The minimal bundle supplies fixable CVE-2022-2097 with CVSS 5.3 for the existing test image, so this retains vulnerability-policy enforcement coverage without changing the shipped default policy's criteria. Create the policy only when the existing list lookup does not find it. Supply the required category, severity, policy version, lifecycle stages, fixability criterion, and CVSS criterion, and enable strict validation as the StackRox test policy service does. Return the POST response directly: initial creation needs only the list lookup and POST, with no source-policy fetch or cloning. Subsequent calls reuse the existing enforcement-update path. References: - [Minimal-bundle Jenkins PR](https://github.com/stackrox/jenkins-plugin/pull/521) - [Original failing test run](https://github.com/stackrox/jenkins-plugin/actions/runs/42822378589/job/129850880233) - [StackRox test policy creation pattern](https://github.com/stackrox/stackrox/blob/a1a20a3af3c7f68207292e0b32ea3daced54284c/qa-tests-backend/src/main/groovy/services/PolicyService.groovy#L27) Validation: - Java 11: ./gradlew --no-daemon compileTestGroovy codenarcMain codenarcTest passed. - git diff --check passed. - End-to-end execution remains unverified locally because the Central endpoint and credentials are not configured. --- .../src/test/groovy/ImageScanningTest.groovy | 33 ++++++++++--------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/functionaltest-jenkins-plugin/src/test/groovy/ImageScanningTest.groovy b/functionaltest-jenkins-plugin/src/test/groovy/ImageScanningTest.groovy index d3e02722..048f89de 100644 --- a/functionaltest-jenkins-plugin/src/test/groovy/ImageScanningTest.groovy +++ b/functionaltest-jenkins-plugin/src/test/groovy/ImageScanningTest.groovy @@ -9,6 +9,10 @@ import com.offbytwo.jenkins.model.BuildResult import com.stackrox.model.StorageEnforcementAction import com.stackrox.model.StorageListPolicy import com.stackrox.model.StoragePolicy +import com.stackrox.model.StoragePolicyGroup +import com.stackrox.model.StoragePolicySection +import com.stackrox.model.StoragePolicyValue +import com.stackrox.model.StorageSeverity import util.Config @@ -22,7 +26,7 @@ class ImageScanningTest extends BaseSpecification { @Unroll def "image scanning test with toggle enforcement(#imageName, #policyName, #enforcements, #endStatus)"() { given: - updatePolicy("Fixable CVSS >= 7", []) + updatePolicy("Fixable CVSS >= 5", []) updatePolicy("Fixable Severity at least Important", []) when: @@ -66,7 +70,7 @@ class ImageScanningTest extends BaseSpecification { where: "data inputs are: " imageName | policyName | tag - "nginx-alpine:1.2.1" | "Fixable CVSS >= 7" | "1.2.1" + "nginx-alpine:1.2.1" | "Fixable CVSS >= 5" | "1.2.1" "nginx-alpine:latest" | "Latest tag" | "latest" } @@ -104,6 +108,17 @@ class ImageScanningTest extends BaseSpecification { StoragePolicy updatePolicy(String policyName, List enforcements) { List policies = restApiClient.policies def policyId = policies.find { it.name == policyName }?.id + if (policyId == null && policyName == "Fixable CVSS >= 5") { + // The minimal bundle includes fixable CVE-2022-2097 with CVSS 5.3 for the test image. + return restApiClient.policyServiceApi.policyServicePostPolicy(new StoragePolicy( + name: policyName, policyVersion: "1.1", severity: StorageSeverity.MEDIUM_SEVERITY, + categories: ["Vulnerability Management"], disabled: false, + lifecycleStages: [BUILD, DEPLOY], enforcementActions: enforcements, + policySections: [new StoragePolicySection(policyGroups: + ["Fixed By": ".*", "CVSS": ">= 5"].collect { field, value -> + new StoragePolicyGroup(fieldName: field, values: [new StoragePolicyValue(value: value)]) + })]), true) + } assert policyId != null def policy = restApiClient.getPolicy(policyId) @@ -114,20 +129,8 @@ class ImageScanningTest extends BaseSpecification { setExclusions([]) setScope([]) } - if (policyName == "Fixable CVSS >= 7") { - // Keep the default lookup name, but use a test-only threshold for the minimal bundle. - // The image has fixable CVE-2022-2097 with CVSS 5.3. - def cvssGroup = policy.policySections.collectMany { it.policyGroups }.find { it.fieldName == "CVSS" } - assert cvssGroup != null - cvssGroup.values[0].value = ">= 5" - } restApiClient.updatePolicy(policy, policyId) - def updatedPolicy = restApiClient.getPolicy(policyId) - if (policyName == "Fixable CVSS >= 7") { - assert updatedPolicy.policySections.collectMany { it.policyGroups } - .find { it.fieldName == "CVSS" }.values*.value == [">= 5"] - } - return updatedPolicy + return restApiClient.getPolicy(policyId) } }