OCPNODE-3983: Add e2e tests for KubeletEnsureSecretPulledImages feature gate#31102
OCPNODE-3983: Add e2e tests for KubeletEnsureSecretPulledImages feature gate#31102Chandan9112 wants to merge 1 commit intoopenshift:release-4.22from
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: automatic mode |
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughAdds a new Ginkgo node test that validates kubelet image-pull credential verification and caching across scenarios: multi-tenant isolation, secret rotation, imagePullPolicy modes, KubeletConfig verification policy changes with MCP rollout, and registry availability impacts. Changes
Sequence Diagram(s)sequenceDiagram
participant Test as Test Harness
participant APIServer as Kubernetes API
participant Kubelet as Node Kubelet
participant Registry as Internal Registry
participant MCP as MachineConfigPool
Test->>APIServer: create namespaces, imagestream tag, extract secret
Test->>APIServer: create pods referencing secrets / imagePullPolicy
APIServer->>Kubelet: schedule Pod spec
Kubelet->>Registry: attempt image pull (uses secret or cached pull-record)
Registry-->>Kubelet: respond (success / unauthorized / unavailable)
alt KubeletConfig change
Test->>APIServer: apply KubeletConfig (NeverVerify/AlwaysVerify)
APIServer->>MCP: rollout config
MCP-->>Kubelet: kubelet restarts with new policy
end
Kubelet-->>APIServer: report Pod status (Running / ErrImagePull / ImagePullBackOff)
Test->>APIServer: scale registry (to zero) and validate cached vs new pulls
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 8 | ❌ 4❌ Failed checks (4 warnings)
✅ Passed checks (8 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
test/extended/node/kubelet_secret_pulled_images.go (1)
409-425: ⚡ Quick winConsider folding the MCP transition polling into the shared helper.
This local helper duplicates MCP condition polling that already exists in
test/extended/node/node_utils.go:521-570. Extending the shared wait path to support “must observeUpdating=Truebefore waiting for ready” would keep rollout semantics in one place and avoid the two implementations drifting on degraded/updated handling.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@test/extended/node/kubelet_secret_pulled_images.go` around lines 409 - 425, The function credVerifyWaitForMCPUpdating duplicates MCP polling; remove it and extend the existing shared helper waitForMCP (in node_utils.go) to accept an option/flag (e.g., requireObserveUpdating bool or a policy enum) that first polls for condition Type "Updating" == True and only then proceeds with the existing wait-for-ready semantics; update callers (including where credVerifyWaitForMCPUpdating was used) to call waitForMCP with the new option so all MCP transition logic remains centralized and avoids duplicated/ drifting implementations.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@test/extended/node/kubelet_secret_pulled_images.go`:
- Around line 27-30: Replace the Docker Hub mutable image references with
pinned, internally-hosted images: change the constant credVerifyPublicImage from
"docker.io/library/nginx:alpine" to a registry-controlled, immutable image
(preferably using a digest) served by the cluster image-registry (use
internalRegistryPrefix as the prefix and point to a specific repo@sha256:...
digest); update the seed import lines referenced around 71-74 likewise to pull
from the internal registry and pin by digest (do not use :latest or other
mutable tags) so all pulls come from the environment-controlled registry and are
immutable.
- Around line 251-257: Before scaling the image-registry down, query and store
its current replica count (e.g., via
oc.AsAdmin().WithoutNamespace().Run("get").Args("deployment/image-registry",
"-n", "openshift-image-registry", "-o", "jsonpath={.spec.replicas}") or
equivalent) and then use that stored value in the g.DeferCleanup closure instead
of the hard-coded "--replicas=2"; in the cleanup closure call
oc.AsAdmin().WithoutNamespace().Run("scale").Args("deployment/image-registry",
"-n", "openshift-image-registry", fmt.Sprintf("--replicas=%d",
originalReplicas)) and then wait for completion with
oc.AsAdmin().WithoutNamespace().Run("rollout").Args("status",
"deployment/image-registry", "-n", "openshift-image-registry",
"--timeout=2m").Execute() to ensure the registry is restored to its exact prior
size and the test waits for rollout.
---
Nitpick comments:
In `@test/extended/node/kubelet_secret_pulled_images.go`:
- Around line 409-425: The function credVerifyWaitForMCPUpdating duplicates MCP
polling; remove it and extend the existing shared helper waitForMCP (in
node_utils.go) to accept an option/flag (e.g., requireObserveUpdating bool or a
policy enum) that first polls for condition Type "Updating" == True and only
then proceeds with the existing wait-for-ready semantics; update callers
(including where credVerifyWaitForMCPUpdating was used) to call waitForMCP with
the new option so all MCP transition logic remains centralized and avoids
duplicated/ drifting implementations.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 6c10e20a-d25e-473b-9257-be65b6405b54
📒 Files selected for processing (1)
test/extended/node/kubelet_secret_pulled_images.go
| // credVerifyPublicImage is a publicly available image used in multi-tenancy tests | ||
| credVerifyPublicImage = "docker.io/library/nginx:alpine" | ||
| // internalRegistryPrefix is the OpenShift internal registry service address | ||
| internalRegistryPrefix = "image-registry.openshift-image-registry.svc:5000" |
There was a problem hiding this comment.
Avoid Docker Hub and mutable tags in this suite.
credVerifyPublicImage and the seed import both depend on anonymous pulls from Docker Hub, and busybox:latest also makes the test behavior drift as that tag moves. For a disruptive long-running e2e, that is an easy source of flakes from rate limits, outages, or image changes unrelated to this feature. Please pin immutable images and source them from a registry the test environment controls.
Also applies to: 71-74
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@test/extended/node/kubelet_secret_pulled_images.go` around lines 27 - 30,
Replace the Docker Hub mutable image references with pinned, internally-hosted
images: change the constant credVerifyPublicImage from
"docker.io/library/nginx:alpine" to a registry-controlled, immutable image
(preferably using a digest) served by the cluster image-registry (use
internalRegistryPrefix as the prefix and point to a specific repo@sha256:...
digest); update the seed import lines referenced around 71-74 likewise to pull
from the internal registry and pin by digest (do not use :latest or other
mutable tags) so all pulls come from the environment-controlled registry and are
immutable.
Chandan9112
changed the title
Chandan9112
changed the title
openshift-ci-robot
added
the
jira/valid-reference
|
@Chandan9112: This pull request references OCPNODE-3983 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@Chandan9112: This pull request references OCPNODE-3983 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target either version "5.0." or "openshift-5.0.", but it targets "openshift-4.22" instead. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Chandan9112
force-pushed
the
add-kubelet-secret-pulled-images-tests
branch
from
179ba2d to
57ed3a2
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Chandan9112 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
openshift-ci
Bot
removed
the
do-not-merge/work-in-progress
Chandan9112
force-pushed
the
add-kubelet-secret-pulled-images-tests
branch
from
57ed3a2 to
63b9239
Compare
|
/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-disruptive-longrunning |
|
@ngopalak-redhat: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/3cb375d0-44a6-11f1-837c-4e7f0d7ef13a-0 |
| credVerifyCreateSecret(ctx, oc, ns, "secret-v1", pullSecret) | ||
|
|
||
| g.By("Pulling private image with secret-v1 to establish pull record on the node") | ||
| credVerifyRunPod(ctx, oc, credVerifyPod(ns, "pod-initial-pull", privateImage, workerNode, corev1.PullIfNotPresent, "secret-v1")) |
There was a problem hiding this comment.
If this fails is the secret deleted?
There was a problem hiding this comment.
Why would the secret need to be deleted?
| }, | ||
| } | ||
| _, err := oc.AdminKubeClient().CoreV1().Namespaces().Create(ctx, ns, metav1.CreateOptions{}) | ||
| if apierrors.IsAlreadyExists(err) { |
There was a problem hiding this comment.
If the namespace exists ensure that it has the required labels
There was a problem hiding this comment.
Added in latest commit. credVerifyEnsureNamespace now applies the required labels when namespace already exist.
…re gate Automates 14 test scenarios from OCPNODE-3982 covering: - Multi-tenancy isolation for private and public images - Credential rotation (secret hash and coordinates) - ImagePullPolicy (IfNotPresent, Never, Always) - Credential verification policy (NeverVerify, AlwaysVerify) - Registry availability (cached pull-records vs new credentials)
Chandan9112
force-pushed
the
add-kubelet-secret-pulled-images-tests
branch
from
63b9239 to
7e09a28
Compare
|
@Chandan9112: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
@Chandan9112 Can you add the link to the job that actually runs this test case? |
The tests are skipped in this job because the cluster doesn't have the KubeletEnsureSecretPulledImages feature gate enabled (requires TechPreviewNoUpgrade or CustomNoUpgrade FeatureSet). |
|
/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-disruptive-longrunning-techpreview |
|
@Chandan9112: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/39bdf220-484c-11f1-96b2-66597a73dafb-0 |
|
/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-disruptive-longrunning-techpreview |
|
@Chandan9112: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/0a664830-4868-11f1-9015-6470e27966d2-0 |
|
/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-disruptive-longrunning-techpreview |
|
@Chandan9112: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/6a85c3b0-4879-11f1-9d80-723c8c0f052f-0 |
4 participants
Summary
Test Cases
Prerequisites
KubeletEnsureSecretPulledImagesfeature gate to be enabledTechPreviewNoUpgradeorCustomNoUpgradeFeatureSet[OCPFeatureGate:KubeletEnsureSecretPulledImages]and will be skipped automatically on clusters without the feature gate enabledTesting Result
Tested on latest 4.22 OCP cluster
References
Summary by CodeRabbit