Add CI job for lightspeed operator deployment - #65
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: malingatembo The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Central YAML (base), Organization UI (inherited) Review profile: CHILL Plan: Team Run ID: Comment |
|
/recheck |
|
Build failed (check pipeline). Post Warning: |
|
/recheck |
|
recheck |
1 similar comment
|
recheck |
0c3b371 to
cecc1eb
Compare
|
Unable to freeze job graph: Job lightspeed-operator-deployment-crc does not specify a run playbook |
|
Build failed (check pipeline). Post ❌ lightspeed-operator-deployment-crc FAILURE in 29m 14s |
| # CI job for deploying OpenStack Lightspeed operator | ||
| - job: | ||
| name: lightspeed-operator-deployment-crc | ||
| parent: cifmw-base-crc |
There was a problem hiding this comment.
question (non-blocking): Does this job deploy OpenStack as well? I think it does not but I am not 100 % certain about it. In a long term it would be good if the job deployed OpenStack as well because then we can do some basic checks that involve MCP.
There was a problem hiding this comment.
No no, this one only deploys OpenStack Lightspeed operator using the CRC ( so no full deployment of OpenStack control plane). for full OpenStack deployment + MCP integration testing, i think we'd need to add that to a seperate job or enhance this one - maybe we can keep track and have a seperate ticket for it
| - job: | ||
| name: lightspeed-operator-deployment-crc | ||
| parent: cifmw-base-crc | ||
| attempts: 1 |
There was a problem hiding this comment.
suggestion (blocking): I think you want to add openstack-k8s-operators-content-provider job as a dependency [1]. This job builds the operator and then provides it to the "child" jobs that depend on it. This allows you to test the operator from a content in a PR.
There was a problem hiding this comment.
Thanks for the pointer to this feature.. Yeah, I agree we should require a build from pr. I have make the change in ebf2560
| --- | ||
| # CI job for deploying OpenStack Lightspeed operator | ||
| - job: | ||
| name: lightspeed-operator-deployment-crc |
There was a problem hiding this comment.
suggestion (non-blocking): If I were working on this I would be using autohold to investigate the job from the inside.
|
Build failed (check pipeline). Post ❌ lightspeed-operator-deployment-crc FAILURE in 20m 52s |
|
Unable to freeze job graph: Job lightspeed-operator-deployment-crc depends on openstack-k8s-operators-content-provider which was not run. |
|
Build failed (check pipeline). Post ❌ openstack-k8s-operators-content-provider FAILURE in 7m 04s |
| @@ -0,0 +1,17 @@ | |||
| --- | |||
There was a problem hiding this comment.
@malingatembo just FYI, you want to set the CONTAINER_TOOL to podman in Makefile. I've encountered it too:).
Lines 81 to 85 in 8eded89
Because of the failure in the job:
2026-08-26 17:01:59.168382 | controller | TASK [operator_build : lightspeed-operator - Call docker-build dry_run={{ cifmw_operator_build_dryrun|bool }}, chdir={{ operator.src }}, output_dir={{ cifmw_operator_build_basedir }}/artifacts, script=make docker-build, extra_args={'IMG': '{{ operator_img }}'}] ***
2026-08-26 17:01:59.168387 | controller | Wednesday 26 August 2026 17:01:59 +0000 (0:00:00.025) 0:02:31.959 ******
2026-08-26 17:01:59.168391 | controller | Wednesday 26 August 2026 17:01:59 +0000 (0:00:00.025) 0:02:31.958 ******
2026-08-26 17:01:59.315000 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_002_lightspeed_operator_call_docke.log
2026-08-26 17:01:59.315047 | controller | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: NoneType: None
2026-08-26 17:01:59.315054 | controller | task path: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/operator_build/tasks/build.yml:163
2026-08-26 17:01:59.315059 | controller | fatal: [localhost]: FAILED! =>
2026-08-26 17:01:59.315066 | controller | changed: true
2026-08-26 17:01:59.315072 | controller | msg: non-zero return code
2026-08-26 17:01:59.315077 | controller | rc: 2
2026-08-26 17:01:59.315082 | controller | stderr: ''
2026-08-26 17:01:59.315088 | controller | stderr_lines: []
2026-08-26 17:01:59.315093 | controller | stdout: |
2026-08-26 17:01:59.315098 | controller | ~/src/github.com/openstack-k8s-operators/lightspeed-operator ~/ci-framework-data/artifacts
2026-08-26 17:01:59.315104 | controller | docker build --build-arg GOMAXPROCS= -t 38.129.56.96:5001/openstack-k8s-operators/lightspeed-operator:011ef9691d76dd9e4f494b1c25294ed98df09a39 .
2026-08-26 17:01:59.315110 | controller | bash: line 1: docker: command not found
2026-08-26 17:01:59.315116 | controller | make: *** [Makefile:184: docker-build] Error 127
2026-08-26 17:01:59.315122 | controller | stdout_lines:
2026-08-26 17:01:59.315126 | controller | - ~/src/github.com/openstack-k8s-operators/lightspeed-operator ~/ci-framework-data/artifacts
2026-08-26 17:01:59.315131 | controller | - docker build --build-arg GOMAXPROCS= -t 38.129.56.96:5001/openstack-k8s-operators/lightspeed-operator:011ef9691d76dd9e4f494b1c25294ed98df09a39
2026-08-26 17:01:59.315137 | controller | .
2026-08-26 17:01:59.315142 | controller | - 'bash: line 1: docker: command not found'
2026-08-26 17:01:59.315146 | controller | - 'make: *** [Makefile:184: docker-build] Error 127'
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 50m 09s |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 55m 18s |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 57m 53s |
63b4ae6 to
b30b91e
Compare
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
b30b91e to
fadb45e
Compare
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
Creates lightspeed-operator-deployment-crc job that: - Deploys operator using ci-framework playbook - Uses example LLM endpoint (no real credentials) - Expected to fail on auth verification - Inherits from cifmw-base-crc for CRC cluster provisioning JIRA: OSPRH-33344
creates lightspeed-operator-deployment-crc job that: - deploys operator using ci-framework playbook - expected to fail on auth verification ( uses fake creds for LLM endpoints ) - matches file pattern in openstack-operator (in zuul.d/) Jira: OSPRH-33344
Created ci/playbooks/lightspeed/run.yml to execute deployment hook Added run: directive to lightspeed-operator-deployment-crc job Fixes Zuul CONFIG_ERROR: "job does not specify a run playbook" Related: OSPRH-33344
The previous implementation had incorrect hook variable format and directly included tasks instead of using ci-framework's run_hook role. Changes: - Fixed hook format: Changed from string to proper list structure with name, type, and source fields as expected by run_hook role - Updated run playbook to use run_hook role instead of include_tasks - Added roles directive to make ci-framework roles available - Hook source points to install-openstack-lightspeed.yml in ci-framework/hooks/playbooks/ directory The run_hook role automatically resolves hook sources from ci-framework/hooks/playbooks/ and handles execution, logging, and artifact collection. Fixes: Job failure after 29m execution Related: OSPRH-33344
Added openstack-k8s-operators-content-provide as a job dependency to make sure that operator is built from pr changes before deployment. Without this the job wouold use the upstream catalogue image instead of testing the actual pr content. follows the pattern by openstack-operator https://github.com/openstack-k8s-operators/openstack-operator/blob/aeda4eda1c71ae34dabc5b458e1340ab755f41a4/zuul.d/jobs.yaml#L35 Related: OSPRH-33344
The deployment job depends on OpenStack-k8s-operators-content-provider but it was not configured to run in the pipeline. this cased the error: 'Job depends on openstack-k8s-operators-content-provider which was not run' Added content-provider to the jobs list so it runs before deployment, building the operator from PR code and making it available for testing. Fixes: CONFIG_ERROR - dependency not satisfied Related: OSPRH-33344
The run_hook role requires cifmw.general.ci_script module but can't find it because our custom run playbook (in lightspeed-operator repo) doesn't have access to the collection installed in ci-framework. Error from Zuul build ff62866e450d4e1e850f22ee6eb10dd1: ANSIBLE PARSE ERROR ERROR! couldn't resolve module/action 'cifmw.general.ci_script' The error appears to be in '.../ci-framework/roles/run_hook/tasks/playbook.yml': line 90 Build: https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/ff62866e450d4e1e850f22ee6eb10dd1 Added the symlink setup step that all ci-framework run playbooks use: - name: Make a symlink to local .ansible collection dir ansible.builtin.include_role: name: cifmw_helpers tasks_from: symlink_cifmw_collection.yml This creates a symlink making the cifmw collection plugins available for nested Ansible execution. Pattern from: https://github.com/openstack-k8s-operators/ci-framework/blob/main/ci/playbooks/edpm/run.yml#L18-L23 Module usage in run_hook role: https://github.com/openstack-k8s-operators/ci-framework/blob/main/roles/run_hook/tasks/playbook.yml#L90-L115 Fixes: couldn't resolve module/action 'cifmw.general.ci_script' Related: OSPRH-33344
fadb45e to
7d24a2a
Compare
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 52m 18s |
The cifmw_helpers role's internal conditional logic was preventing symlink creation even when cifmw_helpers_project_dir was set correctly. Replace role call with direct ansible.builtin.file tasks that: 1. Ensure parent directory exists 2. Create symlink forcefully This bypasses the role's broken conditionals and ensures the cifmw collection is always available for install-openstack-lightspeed.yml
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 54m 44s |
|
@malingatembo: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
fix:
1. Symlink: Remove existing cifmw directory before creating symlink
to prevent 'directory is not empty' error
2. Missing catalog image: Add cifmw_openstack_lightspeed_catalog_imag
pointing to content-provider built catalog image
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 24m 01s |
1. Fix cifmw collection symlink (Approach 2)
- ci/playbooks/lightspeed/run.yml: Use pre-built collection
- Symlink to:
ci-framework/ci/playbooks/collections/ansible_collections/cifmw
- Simpler approach with complete cifmw.general structure
2. Enable autohold debugging support
- ci/playbooks/lightspeed/pre-run.yaml: Inject SSH keys from GitHub
- Allows SSH access to failed nodes for debugging
- Fetches keys from: https://github.com/malingatembo.keys
3 participants
Creates lightspeed-operator-deployment-crc job that:
JIRA: OSPRH-33344