Skip to content

[WIP] Test install-trustee-operator step for OCP 4.21 AWS CoCo#80236

Open
tbuskey wants to merge 4 commits into
openshift:mainfrom
tbuskey:install-trustee-260608
Open

[WIP] Test install-trustee-operator step for OCP 4.21 AWS CoCo#80236
tbuskey wants to merge 4 commits into
openshift:mainfrom
tbuskey:install-trustee-260608

Conversation

@tbuskey

@tbuskey tbuskey commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Testing the install-trustee-operator step implementation for Confidential Containers on OCP 4.21 AWS.

Purpose: Rehearsal testing only - validate install-trustee-operator step works correctly

Changes:

  • Add install-trustee-operator step for CoCo tests
  • Make candidate421 aws-ipi-coco rehearsable
  • Adjust timeout to trigger job regeneration

Test Plan:

  • Use /pj-rehearse to test periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-aws-ipi-coco-e2e
  • Validate install-trustee-operator step completes successfully
  • Verify CoCo workload testing succeeds

/hold
/assign @tbuskey

Summary by CodeRabbit

This PR introduces Trustee operator installation automation to the sandboxed-containers-operator CI pipeline. Here's what changes:

New Install-Trustee-Operator Step

A comprehensive new step has been added (sandboxed-containers-operator-install-trustee-operator) that automates Trustee operator provisioning for Confidential Containers (CoCo) workloads. The step:

  • Installs the Trustee operator via OLM (CatalogSource → Subscription → CSV), with full polling to wait for operator readiness
  • Deploys Trustee operands (KBS, Attestation Service) with staged availability checks
  • Derives cluster domain and discovers the Trustee service endpoint (via route, load balancer, or cluster IP)
  • Generates INITDATA by building an initdata.toml with KBS/TLS configuration and image security policy (rego rules), then compresses and base64-encodes it
  • Patches the osc-config ConfigMap with derived trusteeUrl and INITDATA
  • Validates connectivity by deploying a temporary kbs-client pod and testing artifact retrieval from Trustee KBS
  • Handles network-restricted environments using pre-rendered manifests embedded in the script

The step is guarded by TRUSTEE_INSTALL and configurable via environment variables for catalog source, namespace, and KBS client tag discovery.

CI Configuration Updates Across All Candidate Branches

The CI configuration for sandboxed-containers-operator has been updated across all candidate versions (417, 418, 419, 420, 421, and the main candidate) to enable Trustee installation for Azure, ARO, and AWS CoCo test variants:

  • TRUSTEE_INSTALL: Set to "true" to enable the new step
  • TRUSTEE_CATALOG_SOURCE_IMAGE: Points to trustee-test-fbc:1.1.0-1776506656
  • TRUSTEE_CATALOG_SOURCE_NAME: Set to trustee-catalog

For the candidate421 variant specifically, TRUSTEE_NAMESPACE is also added to make it appear in rehearsal notifications, and the job timeout was increased by 1 minute to force job file regeneration.

Pre-Chain Integration

The new step has been added to the sandboxed-containers-operator-pre chain, so Trustee operator installation runs automatically before CoCo workload tests execute.

tbuskey and others added 3 commits June 8, 2026 11:36
This PR adds the install-trustee-operator step to enable automated
Trustee operator installation in sandboxed-containers-operator CoCo tests.

Changes:
- Add install-trustee-operator step to step-registry
- Enable TRUSTEE_INSTALL for all CoCo periodic jobs (aws, azure, aro)
- Configure Trustee catalog source and installation parameters
- Add step to sandboxed-containers-operator-pre chain

The step:
- Installs Trustee operator via OLM (CatalogSource → Subscription → CSV)
- Deploys Trustee operands (KBS, Attestation Service)
- Generates INITDATA for confidential containers
- Verifies connectivity using kbs-client
- Updates osc-config ConfigMap with TRUSTEE_URL and INITDATA
- Works with network restrictions (uses pre-rendered manifests)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add TRUSTEE_NAMESPACE environment variable to aws-ipi-coco test configuration
to make it appear in REHEARSALNOTIFIER affected jobs list for testing
install-trustee-operator step.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Modified timeout from 24h to 24h1m for candidate421 aws-ipi-coco to force
job file regeneration. Environment variable changes alone don't trigger
job updates in variant configs.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@openshift-ci openshift-ci Bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Jun 8, 2026
@coderabbitai

coderabbitai Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Caution

Review failed

An error occurred during the review process. Please try again later.

Walkthrough

This PR introduces comprehensive Trustee operator installation and verification into the sandboxed-containers-operator CI pipeline. A 934-line bash script handles cluster domain discovery, OLM-based operator installation with staged readiness polling, operand deployment, Trustee KBS URL resolution, INITDATA artifact generation, and connectivity verification via kbs-client test pod deployment. The script is wired as a new CI step and integrated into the test pre-chain, and test configurations across five downstream-candidate variants are updated to enable Trustee installation.

Changes

Trustee operator installation in sandboxed-containers CI

Layer / File(s) Summary
Setup, helpers, and cluster discovery
ci-operator/step-registry/sandboxed-containers-operator/install-trustee-operator/sandboxed-containers-operator-install-trustee-operator-commands.sh (lines 1–79)
Script initialization with safety flags, environment defaults, early exit when TRUSTEE_INSTALL is disabled, temporary directory with EXIT trap for diagnostics, retry helper, and cluster domain discovery via ingress domain, console route, or console URL inspection.
Operator installation and readiness polling
ci-operator/step-registry/sandboxed-containers-operator/install-trustee-operator/sandboxed-containers-operator-install-trustee-operator-commands.sh (lines 81–429)
Conditional CatalogSource manifest generation, OLM operator manifests (namespace, image mirror sets, operator group, subscription), operator installation with namespace/catalog placeholder substitution, and multi-stage readiness polling with explicit time-bounded loops for CatalogSource READY, InstallPlan reference, InstallPlan completion, CSV success, and controller deployment availability.
Operand deployment and Trustee configuration artifact generation
ci-operator/step-registry/sandboxed-containers-operator/install-trustee-operator/sandboxed-containers-operator-install-trustee-operator-commands.sh (lines 190–695)
Trustee operand manifests with KBS config, secrets, and route; operand installation and readiness polling; TLS certificate discovery from ingress-related secrets with OpenSSL fallback; Trustee KBS URL/host/port resolution via route, load balancer, or clusterIP; INITDATA TOML assembly with TLS cert, containers-policy secret, rego policy, and KBS/policy URLs, gzipped/base64 encoded; osc-config ConfigMap patching with trusteeUrl and INITDATA.
KBS connectivity verification and test pod deployment
ci-operator/step-registry/sandboxed-containers-operator/install-trustee-operator/sandboxed-containers-operator-install-trustee-operator-commands.sh (lines 696–934)
KBS client pod manifest generation, Trustee operator CSV to kbs-client version mapping, image tag selection with KBS_CLIENT_TAG override, CSV-derived mapping, registry semver auto-discovery via skopeo and jq, and hardcoded fallback; kbs-client test pod deployment with readiness wait, resource fetch execution against Trustee KBS with stdout/stderr capture for diagnostics, attestation log pattern collection from Trustee KBS, test pod cleanup, and failure flagging.
Step registry definition and ownership
ci-operator/step-registry/sandboxed-containers-operator/install-trustee-operator/sandboxed-containers-operator-install-trustee-operator-ref.yaml, sandboxed-containers-operator-install-trustee-operator-ref.metadata.json, OWNERS
Step registry YAML defining compute resource requests, command script reference, environment variables for installation control and connectivity discovery, and comprehensive documentation; metadata JSON with step path and owner lists; OWNERS file with reviewer and approver assignments.
CI test configuration updates across downstream-candidate variants
ci-operator/config/openshift/sandboxed-containers-operator/openshift-sandboxed-containers-operator-devel__downstream-candidate*.yaml
Five CI configuration files updated to add TRUSTEE_CATALOG_SOURCE_IMAGE, TRUSTEE_CATALOG_SOURCE_NAME, and TRUSTEE_INSTALL: "true" environment variables to azure-ipi-coco, aro-ipi-coco, and aws-ipi-coco test jobs; candidate421 additionally sets TRUSTEE_NAMESPACE on aws-ipi-coco, increases job timeout by 1 minute, and adjusts report_template line-break quoting.
Pre-chain step integration
ci-operator/step-registry/sandboxed-containers-operator/pre/sandboxed-containers-operator-pre-chain.yaml
Adds sandboxed-containers-operator-install-trustee-operator step reference to the test pre-chain, positioned before the record-metadata step.

🎯 4 (Complex) | ⏱️ ~60 minutes

Suggested labels

rehearsals-ack

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot requested review from c3d and pmores June 8, 2026 16:08
@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 8, 2026
@tbuskey

tbuskey commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse list

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@tbuskey: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@tbuskey

tbuskey commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-aws-ipi-coco-e2e

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@tbuskey: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@tbuskey: job(s): periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-aws-ipi-coco-e2e either don't exist or were not found to be affected, and cannot be rehearsed

@tbuskey

tbuskey commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse network-access-allowed

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@tbuskey: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@tbuskey: PR author isn't allowed to allow network access rehearsals. This must be done by a different member of the openshift org

@tbuskey

tbuskey commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-aws-ipi-coco

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@tbuskey: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@tbuskey: job(s): periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-aws-ipi-coco either don't exist or were not found to be affected, and cannot be rehearsed

…i-coco rehearsal

To make the job rehearsable without requiring network-access-rehearsals-ok label.
Will revert after testing install-trustee-operator step.
@openshift-ci

openshift-ci Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tbuskey
Once this PR has been reviewed and has the lgtm label, please assign prucek for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 8, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@tbuskey: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate419-aws-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate420-azure-ipi-kata N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate418-aws-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate418-azure-ipi-kata N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-release-azure-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-azure-ipi-kata N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate-aro-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate419-aro-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate418-azure-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate418-aro-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-release-azure-ipi-coco N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate420-aro-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-azure-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate417-azure-ipi-kata N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-release-azure-ipi-kata N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-aro-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate417-aws-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate-azure-ipi-kata N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-release-aro-ipi-coco N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-aws-ipi-coco N/A periodic Ci-operator config changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate420-azure-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate-azure-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate417-aro-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-aws-ipi-peerpods N/A periodic Registry content changed
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-release-aws-ipi-peerpods N/A periodic Registry content changed

A total of 32 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here
The following jobs are not rehearsable without the network-access-rehearsals-ok, and approved labels present on this PR. This is due to the restrict_network_access field being set to false. The network-access-rehearsals-ok label can be added by any openshift org member other than the PR's author by commenting: /pj-rehearse network-access-allowed:

Test name
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate419-aro-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate419-aws-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate419-azure-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate418-azure-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate418-aro-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate418-aws-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-aro-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-azure-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate420-azure-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate420-aro-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate420-aws-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate-azure-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate-aro-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate-aws-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate417-aro-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate417-aws-ipi-coco
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate417-azure-ipi-coco
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@tbuskey

tbuskey commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-aws-ipi-coco

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@tbuskey: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@tbuskey

tbuskey commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-aws-ipi-coco

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@tbuskey: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci

openshift-ci Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

@tbuskey: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-aws-ipi-coco e66325b link unknown /pj-rehearse periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-aws-ipi-coco

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant