Skip to content

Add S2I speculative deploy testing for telemetry-operator - #992

Open
danpawlik wants to merge 1 commit into
openstack-k8s-operators:mainfrom
danpawlik:s2i-test
Open

Add S2I speculative deploy testing for telemetry-operator#992
danpawlik wants to merge 1 commit into
openstack-k8s-operators:mainfrom
danpawlik:s2i-test

Conversation

@danpawlik

@danpawlik danpawlik commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Implement similar S2I testing workflow as watcher-operator [1] that adds:

  • s2i-speculative-deploy-test-telemetry job - validates speculatively-built S2I telemetry-operator container images against a live OpenStack deployment with telemetry enabled
  • s2i-openstack-container-content-provider job - builds S2I containers
  • Non-voting github-check jobs to validate S2I builds before merging

This enables testing whether speculatively-rebuilt telemetry container images function correctly in real deployments.

Cloudkitty s2i (cloudkitty/cloudkitty-api) deferred: the cloudkitty deploy path pulls the meta content provider, which conflicts with s2i image injection. Add when a clean deploy path exists.

[1] openstack-k8s-operators/watcher-operator#443

Related-To: OSPRH-34558
Related-To: OSPRH-34557
Assisted-By: Claude Haiku 4.5 noreply@anthropic.com

Depends-On: openstack-k8s-operators/ci-framework#4131
Depends-On: #993

optional

Depends-On: openstack-k8s-operators/ci-framework#4152

@openshift-ci

openshift-ci Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Team

Run ID: 69ff4044-97b7-4dc5-819f-3c9486c7898b


Comment @coderabbitai help to get the list of available commands.

Comment thread zuul.d/projects.yaml
@rebtoor

rebtoor commented Aug 25, 2026

Copy link
Copy Markdown

@danpawlik hey Daniel until they get merged, you need depends-on to openstack-k8s-operators/s2i-openstack-containers#91 and openstack-k8s-operators/ci-framework#4131

@centosinfra-prod-github-app

Copy link
Copy Markdown

This change depends on a change that failed to merge.

Change openstack-k8s-operators/ci-framework#4131 is needed.

@centosinfra-prod-github-app

Copy link
Copy Markdown

This change depends on a change that failed to merge.

Change openstack-k8s-operators/ci-framework#4131 is needed.

@rebtoor

rebtoor commented Aug 25, 2026

Copy link
Copy Markdown

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/37eadcdc908d456d97af96645e11a044

✔️ telemetry-openstack-meta-content-provider-master SUCCESS in 2h 45m 17s
✔️ telemetry-operator-multinode-cloudkitty SUCCESS in 1h 42m 50s
✔️ telemetry-openstack-meta-content-provider-all-services-master SUCCESS in 2h 45m 44s
✔️ telemetry-operator-multinode-master SUCCESS in 1h 37m 24s
✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 10m 41s
✔️ telemetry-operator-multinode-default-telemetry SUCCESS in 1h 33m 32s
✔️ s2i-openstack-container-content-provider SUCCESS in 2h 52m 33s (non-voting)
✔️ s2i-speculative-deploy-test-telemetry SUCCESS in 1h 28m 06s (non-voting)
✔️ functional-tests-osp18 SUCCESS in 2h 23m 05s
functional-periodic-telemetry-with-ceph FAILURE in 2h 21m 22s

@rebtoor

rebtoor commented Aug 26, 2026

Copy link
Copy Markdown

@danpawlik

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/logs//1e3/rdoproject.org/1e38091c651948179e937784174f260e/controller/ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7c4e74411335a883ab800a2f1812e725a15ea42bdf17bba8192fc315f7f72332/namespaces/openstack/crs/openstackversions.core.openstack.org/controlplane.yaml < patched images have been used

spec:
  customContainerImages:
    aodhAPIImage: xxx.xxx.xxx.xxx:5001/openstack/openstack-aodh-api:992-ce9bfaeed0ec4a4f7230a88e9effa4de8fec7fa6
    aodhEvaluatorImage: xxx.xxx.xxx.xxx:5001/openstack/openstack-aodh-api:992-ce9bfaeed0ec4a4f7230a88e9effa4de8fec7fa6
    aodhListenerImage: xxx.xxx.xxx.xxx:5001/openstack/openstack-aodh-api:992-ce9bfaeed0ec4a4f7230a88e9effa4de8fec7fa6
    aodhNotifierImage: xxx.xxx.xxx.xxx:5001/openstack/openstack-aodh-api:992-ce9bfaeed0ec4a4f7230a88e9effa4de8fec7fa6

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/logs//1e3/rdoproject.org/1e38091c651948179e937784174f260e/controller/ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7c4e74411335a883ab800a2f1812e725a15ea42bdf17bba8192fc315f7f72332/namespaces/openstack/pods/aodh-0/aodh-0-describe

(The ceph failure is well known =( )

Comment thread zuul.d/projects.yaml

- job:
name: s2i-speculative-deploy-test-telemetry
parent: telemetry-operator-multinode-autoscaling

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This parent doesn't actually run any tests.
Better parent is: functional-tests-osp18, which runs tempest and FVTs for logging, autoscaling.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, let's try

@openshift-ci

openshift-ci Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: danpawlik, rebtoor
Once this PR has been reviewed and has the lgtm label, please assign paramite for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@danpawlik
danpawlik marked this pull request as ready for review August 31, 2026 12:35
@openshift-ci
openshift-ci Bot requested review from abays and mgirgisf August 31, 2026 12:35
@vyzigold

Copy link
Copy Markdown
Contributor

This seems OK, as long as the CI passes. One thing though... I see this is testing Aodh and Cloudkitty is deferred, what about Ceilometer?

@danpawlik

danpawlik commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

This seems OK, as long as the CI passes. One thing though... I see this is testing Aodh and Cloudkitty is deferred, what about Ceilometer?

With ceilometer would be done in another story - OSPRH-34556 - as Ceilometer would use 2 images not 4.

@danpawlik

Copy link
Copy Markdown
Contributor Author

/test precommit-check

@danpawlik

Copy link
Copy Markdown
Contributor Author

/test telemetry-operator-build-deploy

@openshift-ci

openshift-ci Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

New changes are detected. LGTM label has been removed.

@centosinfra-prod-github-app

Copy link
Copy Markdown

This change depends on a change that failed to merge.

Change #993 is needed.

@danpawlik

Copy link
Copy Markdown
Contributor Author

recheck

@danpawlik

Copy link
Copy Markdown
Contributor Author

@vyzigold added change as depends-on #993 + updated images to use s2i ceilometer images. Let's see

@danpawlik

danpawlik commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

@rebtoor @elfiesmelfie fun fact: since when I changed in s2i-speculative-deploy-test-telemetry parent to functional-tests-osp18 it was never executed.
When I go back to telemetry-operator-multinode-autoscaling it works normally. In the past, when there was some issue, Zuul raises something. Here is nothing. If the parent really needs to be functional-tests-osp18 then I need to co-op with infra folks and check whats going on

Implement similar S2I testing workflow as watcher-operator [1]
that adds:

- s2i-speculative-deploy-test-telemetry job - validates speculatively-built
  S2I telemetry-operator container images against a live OpenStack deployment
  with telemetry enabled
- s2i-openstack-container-content-provider job - builds S2I containers
- Non-voting github-check jobs to validate S2I builds before merging

This enables testing whether speculatively-rebuilt telemetry container images
function correctly in real deployments.

Cloudkitty s2i (cloudkitty/cloudkitty-api) deferred: the cloudkitty deploy
path pulls the meta content provider, which conflicts with s2i image
injection. Add when a clean deploy path exists.

[1] openstack-k8s-operators/watcher-operator#443

Related-To: OSPRH-34558
Assisted-By: Claude Haiku 4.5 <noreply@anthropic.com>

Depends-On: openstack-k8s-operators/ci-framework#4131
Depends-On: openstack-k8s-operators#993

Signed-off-by: Daniel Pawlik <dpawlik@redhat.com>
@danpawlik

Copy link
Copy Markdown
Contributor Author

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/logs//146/rdoproject.org/146103f1d7da4711a958eba1e7a7f287/controller/ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-7c4e74411335a883ab800a2f1812e725a15ea42bdf17bba8192fc315f7f72332/namespaces/openstack/crs/openstackversions.core.openstack.org/controlplane.yaml

spec:
  customContainerImages:
    aodhAPIImage: 38.102.83.20:5001/openstack/openstack-aodh-api:992-22723563aa3d4d40d36e0167b6ba01fbc8ac3b02
    aodhEvaluatorImage: 38.102.83.20:5001/openstack/openstack-aodh-api:992-22723563aa3d4d40d36e0167b6ba01fbc8ac3b02
    aodhListenerImage: 38.102.83.20:5001/openstack/openstack-aodh-api:992-22723563aa3d4d40d36e0167b6ba01fbc8ac3b02
    aodhNotifierImage: 38.102.83.20:5001/openstack/openstack-aodh-api:992-22723563aa3d4d40d36e0167b6ba01fbc8ac3b02
    ceilometerCentralImage: 38.102.83.20:5001/openstack/openstack-ceilometer-central:992-22723563aa3d4d40d36e0167b6ba01fbc8ac3b02
    ceilometerComputeImage: 38.102.83.20:5001/openstack/openstack-ceilometer-compute:992-22723563aa3d4d40d36e0167b6ba01fbc8ac3b02
    ceilometerIpmiImage: 38.102.83.20:5001/openstack/openstack-ceilometer-compute:992-22723563aa3d4d40d36e0167b6ba01fbc8ac3b02
    ceilometerNotificationImage: 38.102.83.20:5001/openstack/openstack-ceilometer-central:992-22723563aa3d4d40d36e0167b6ba01fbc8ac3b02
  targetVersion: 19.0.0-1788261999

@danpawlik

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/3689ed894dc7402199698798b4d4bcf0

✔️ telemetry-openstack-meta-content-provider-master SUCCESS in 3h 51m 29s
✔️ telemetry-operator-multinode-cloudkitty SUCCESS in 1h 35m 59s
✔️ telemetry-openstack-meta-content-provider-all-services-master SUCCESS in 2h 17m 41s
✔️ telemetry-operator-multinode-master SUCCESS in 1h 33m 29s
✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 00m 02s
✔️ telemetry-operator-multinode-default-telemetry SUCCESS in 1h 41m 46s
✔️ s2i-openstack-container-content-provider SUCCESS in 1h 45m 04s (non-voting)
✔️ s2i-speculative-deploy-test-telemetry SUCCESS in 1h 26m 51s (non-voting)
functional-tests-osp18 TIMED_OUT in 3h 26m 36s
functional-periodic-telemetry-with-ceph FAILURE in 49m 56s

@danpawlik

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

This change depends on a change that failed to merge.

Change openstack-k8s-operators/ci-framework#4152 is needed.

@danpawlik

Copy link
Copy Markdown
Contributor Author

recheck

@danpawlik

Copy link
Copy Markdown
Contributor Author

/test telemetry-operator-build-deploy

@danpawlik

Copy link
Copy Markdown
Contributor Author

recheck

@danpawlik

Copy link
Copy Markdown
Contributor Author

/test telemetry-operator-build-deploy

@openshift-ci

openshift-ci Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

@danpawlik: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/telemetry-operator-build-deploy 2272356 link false /test telemetry-operator-build-deploy

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/5d3048379bcf47f88c47dd485b899703

telemetry-openstack-meta-content-provider-master FAILURE in 12m 31s
⚠️ telemetry-operator-multinode-cloudkitty SKIPPED Skipped due to failed job telemetry-openstack-meta-content-provider-master
telemetry-openstack-meta-content-provider-all-services-master FAILURE in 8m 57s
⚠️ telemetry-operator-multinode-master SKIPPED Skipped due to failed job telemetry-openstack-meta-content-provider-all-services-master
✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 02m 19s
✔️ telemetry-operator-multinode-default-telemetry SUCCESS in 1h 41m 52s
✔️ s2i-openstack-container-content-provider SUCCESS in 1h 51m 16s (non-voting)
✔️ s2i-speculative-deploy-test-telemetry SUCCESS in 1h 30m 53s (non-voting)
⚠️ functional-tests-osp18 SKIPPED Skipped due to failed job telemetry-openstack-meta-content-provider-master
⚠️ functional-periodic-telemetry-with-ceph SKIPPED Skipped due to failed job telemetry-openstack-meta-content-provider-master

@danpawlik

Copy link
Copy Markdown
Contributor Author

recheck

2 similar comments
@danpawlik

Copy link
Copy Markdown
Contributor Author

recheck

@danpawlik

Copy link
Copy Markdown
Contributor Author

recheck

@danpawlik

danpawlik commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Centos stream container images were broken

[zuul@np0000163341 base]$ podman run --rm quay.io/centos/centos:stream10 ls -l /bin/sh /usr/bin/sh
Error: crun: executable file `ls` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

new images seems to work:

[zuul@np0000163341 base]$ podman run --rm  quay.io/centos/centos@sha256:301bc4e6d5af2b6e707bec46aa71c21dc91ebc97422373cca37ea98d0e1aed63 ls -l /bin/sh /usr/bin/sh
lrwxrwxrwx. 1 root root 4 Oct 29  2024 /bin/sh -> bash
lrwxrwxrwx. 1 root root 4 Oct 29  2024 /usr/bin/sh -> bash

@danpawlik

Copy link
Copy Markdown
Contributor Author

recheck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants