Skip to content

USHIFT-7117: Add common script for C2CC scenarios#6827

Open
vanhalenar wants to merge 9 commits into
openshift:mainfrom
vanhalenar:c2cc-common
Open

USHIFT-7117: Add common script for C2CC scenarios#6827
vanhalenar wants to merge 9 commits into
openshift:mainfrom
vanhalenar:c2cc-common

Conversation

@vanhalenar

@vanhalenar vanhalenar commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • Tests

    • Centralized cluster-to-cluster test orchestration into shared helpers; unified create/remove/run flow across scenarios.
    • Added IPsec tunnel setup/verification plus full IPv4 and IPv6 C2CC test support.
    • Automated cross-host readiness checks and centralized collection of secondary cluster kubeconfigs.
  • Refactor

    • Simplified scenario scripts to delegate orchestration to shared C2CC helpers, removing duplicated logic.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 8, 2026
@openshift-ci-robot

openshift-ci-robot commented Jun 8, 2026

Copy link
Copy Markdown

@vanhalenar: This pull request references USHIFT-7117 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 0028f4ce-353f-4c52-8d0f-b63cb18f2150

📥 Commits

Reviewing files that changed from the base of the PR and between 8fdb3be and 1e9bad9.

📒 Files selected for processing (1)
  • test/bin/c2cc_common.sh
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/bin/c2cc_common.sh

Walkthrough

Extracts multi-host C2CC orchestration (readiness, C2CC YAML, firewall, VM lifecycle, test wiring, IPsec) into shared test helper test/bin/c2cc_common.sh and updates EL9/EL10 scenario scripts to source it and delegate VM/config/test tasks.

Changes

C2CC Scenario Test Centralization

Layer / File(s) Summary
Shared constants and host IP helper
test/bin/c2cc_common.sh
Cluster A/B/C CIDR and domain constants added, TEST_RANDOMIZATION exported, and get_host_ip resolves VM IPs.
Greenboot readiness across hosts
test/bin/c2cc_common.sh
wait_for_greenboot_on_hosts polls MicroShift greenboot for host1–host3 and records per-host JUnit results.
Per-host C2CC config writer
test/bin/c2cc_common.sh
configure_c2cc_host writes 50-c2cc.yaml remoteClusters entries, aggregates remote pod/svc CIDRs for firewall trusted sources, reloads firewall, and restarts MicroShift.
Orchestrate C2CC across hosts
test/bin/c2cc_common.sh
configure_c2cc_hosts resolves host IPs, runs pre/post greenboot checks, and configures mappings for host1/host2/host3 using the predefined constants.
VM creation and cleanup
test/bin/c2cc_common.sh
c2cc_create_vms prepares kickstarts (adds IPv6 args when requested), injects post-microshift config for non-default CIDRs on host2/host3, and launches the three VMs; c2cc_remove_vms removes them.
Test orchestration and kubeconfig wiring
test/bin/c2cc_common.sh
c2cc_run_tests sets FOREIGN_CIDR/IP_FAMILY when present, waits for host2/host3 readiness, copies their kubeconfigs into scenario output, and runs run_tests on host1 with cluster CIDR/domain variables and KUBECONFIG_B/C.
IPsec helpers
test/bin/c2cc_common.sh
configure_ipsec_host writes ipsec secrets and tunnel configs; wait_for_ipsec_tunnels polls ipsec trafficstatus; configure_ipsec generates PSK and configures/verifies tunnels for all host pairs.
EL10 IPsec scenario refactor
test/scenarios-bootc/el10/presubmits/el102-src@c2cc-ipsec.sh
Sources c2cc_common.sh, sets TEST_RANDOMIZATION=none, and replaces in-file VM/config/test logic with wrappers that call c2cc_create_vms, c2cc_remove_vms, configure_c2cc_hosts, configure_ipsec, and c2cc_run_tests.
EL10 C2CC IPv4 scenario refactor
test/scenarios-bootc/el10/presubmits/el102-src@c2cc.sh
Sources c2cc_common.sh and delegates VM lifecycle and test execution to shared c2cc_* helpers (passing the IPv4 foreign CIDR).
EL10 C2CC IPv6 scenario refactor
test/scenarios-bootc/el10/presubmits/el102-src@c2cc-ipv6.sh
Sources c2cc_common.sh and delegates VM lifecycle and test execution to shared helpers with IPv6 parameters and foreign CIDR.
EL9 IPsec scenario refactor
test/scenarios-bootc/el9/presubmits/el98-src@c2cc-ipsec.sh
Sources c2cc_common.sh, keeps TEST_RANDOMIZATION=none, and delegates lifecycle, IPsec config, and test runs to shared helpers.
EL9 C2CC IPv6 scenario refactor
test/scenarios-bootc/el9/presubmits/el98-src@c2cc-ipv6.sh
Sources c2cc_common.sh and delegates create/remove/config/run to shared c2cc helpers with IPv6.
EL9 C2CC IPv4 scenario refactor
test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh
Sources c2cc_common.sh and delegates VM/test orchestration to shared helpers for IPv4.

Sequence Diagram(s)

sequenceDiagram
  participant VMManager
  participant Host1
  participant Host2
  participant Host3
  participant MicroShift
  participant IPsec
  VMManager->>Host1: launch VM (host1)
  VMManager->>Host2: launch VM (host2)
  VMManager->>Host3: launch VM (host3)
  Host1->>MicroShift: wait greenboot
  Host2->>MicroShift: wait greenboot
  Host3->>MicroShift: wait greenboot
  Host1->>Host2: apply c2cc YAML + firewall
  Host1->>Host3: apply c2cc YAML + firewall
  Host1->>IPsec: configure PSK + tunnels
  IPsec->>Host1: report tunnel status
Loading

Possibly Related PRs

Suggested Reviewers

  • jerpeter1

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 4.55% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the primary change: extracting common C2CC functionality into a shared script that multiple scenario files now use.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All test names in the PR are static and deterministic. Robot Framework tests use descriptive names without dynamic values like pod names, IPs, timestamps, or UUIDs.
Test Structure And Quality ✅ Passed PR contains only bash shell scripts, not Ginkgo test code. The custom check is inapplicable as it targets Go/Ginkgo tests specifically.
Microshift Test Compatibility ✅ Passed The PR adds only shell scripts and modifies shell script files. No Ginkgo Go test files are added or modified. The custom check applies only to Ginkgo tests, which are absent here.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR adds no Ginkgo e2e tests (It/Describe/Context). It only refactors bash test infrastructure and Robot Framework test scenarios, which are not subject to SNO compatibility checks.
Topology-Aware Scheduling Compatibility ✅ Passed PR adds test/scenario scripts and test utilities only—no deployment manifests, operators, or controllers that could introduce topology-aware scheduling constraints.
Ote Binary Stdout Contract ✅ Passed PR contains only bash shell scripts (test/bin/c2cc_common.sh and scenario scripts), not Go OTE binaries. Check applies only to Go binary code with main/suite setup, not shell script orchestration.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No Ginkgo e2e tests (It(), Describe(), etc.) were added. PR only adds bash shell scripts for test scenario orchestration.
No-Weak-Crypto ✅ Passed No weak crypto detected. IPsec uses AES-256-SHA2-256, PSK generated via openssl rand, libreswan implementation (not custom), no insecure secret comparisons.
Container-Privileges ✅ Passed PR only modifies bash shell scripts for test scenarios; no container/K8s manifests added or modified. Existing C2CC test assets have proper security controls (no privileged settings).
No-Sensitive-Data-In-Logs ✅ Passed No sensitive data (passwords, tokens, API keys, PII, session IDs) is logged. PSK generation is error-silent, and secrets are written with output redirected to /dev/null.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot requested review from jerpeter1 and pmtk June 8, 2026 12:42
@openshift-ci

openshift-ci Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vanhalenar
Once this PR has been reviewed and has the lgtm label, please assign eslutsky for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai coderabbitai Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jun 8, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/bin/c2cc_common.sh`:
- Around line 86-87: The defaulting for network and ip_family in c2cc_create_vms
is using ${3-default} and ${4-ipv4} which do not fall back when the argument is
an empty string; change these to use the shell "use default if unset or empty"
form ${3:-default} and ${4:-ipv4} so that network and ip_family are never empty
when passed to launch_vm (references: function c2cc_create_vms, variables
network and ip_family, and caller launch_vm).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: fc459070-1f6e-4453-ad11-a1ce0d55788f

📥 Commits

Reviewing files that changed from the base of the PR and between ed34e7f and 64c8f8a.

📒 Files selected for processing (5)
  • test/bin/c2cc_common.sh
  • test/scenarios-bootc/el10/presubmits/el102-src@c2cc-ipv6.sh
  • test/scenarios-bootc/el10/presubmits/el102-src@c2cc.sh
  • test/scenarios-bootc/el9/presubmits/el98-src@c2cc-ipv6.sh
  • test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh

Comment thread test/bin/c2cc_common.sh Outdated
@coderabbitai coderabbitai Bot removed the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jun 8, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/bin/c2cc_common.sh`:
- Around line 176-181: The assignments for host2_ip and host3_ip use command
substitution with local -r which masks failures; change to first capture the
output of get_vm_property into a non-readonly local variable (e.g., host2_ip_tmp
/ host3_ip_tmp) and immediately check the command exit status (or test for empty
value) like the pattern used in configure_c2cc_hosts, logging an error and
returning/non-zero on failure, then set the readonly kubeconfig path variables
(kubeconfig_b / kubeconfig_c) only after successful retrieval of
host2_ip/host3_ip so an empty IP cannot propagate.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 26f9f6ba-1ded-4cb5-afab-98b0b232db96

📥 Commits

Reviewing files that changed from the base of the PR and between 70a2234 and 79faeac.

📒 Files selected for processing (7)
  • test/bin/c2cc_common.sh
  • test/scenarios-bootc/el10/presubmits/el102-src@c2cc-ipsec.sh
  • test/scenarios-bootc/el10/presubmits/el102-src@c2cc-ipv6.sh
  • test/scenarios-bootc/el10/presubmits/el102-src@c2cc.sh
  • test/scenarios-bootc/el9/presubmits/el98-src@c2cc-ipsec.sh
  • test/scenarios-bootc/el9/presubmits/el98-src@c2cc-ipv6.sh
  • test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh

Comment thread test/bin/c2cc_common.sh Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/bin/c2cc_common.sh (1)

30-33: ⚡ Quick win

Use get_host_ip for consistent error handling.

get_vm_property failure here leaves host_ip empty and produces a confusing SSH error downstream. The get_host_ip helper (line 24) handles this cleanly and is used in configure_c2cc_hosts.

Proposed fix
     for host in host1 host2 host3; do
-        local host_ip full_host
-        host_ip=$(get_vm_property "${host}" ip)
+        local host_ip
+        host_ip=$(get_host_ip "${host}") || return 1
+        local full_host
         full_host=$(full_vm_name "${host}")
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/bin/c2cc_common.sh` around lines 30 - 33, The loop currently calls
get_vm_property to set host_ip which can return empty and cause confusing SSH
errors; replace the get_vm_property call with the get_host_ip helper to obtain
host_ip (while keeping full_host from full_vm_name) so failures are handled
consistently like in configure_c2cc_hosts; update the loop that sets host_ip and
full_host to use get_host_ip("${host}") instead of get_vm_property and ensure
any early-exit or error message semantics from get_host_ip are preserved.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/bin/c2cc_common.sh`:
- Around line 30-33: The loop currently calls get_vm_property to set host_ip
which can return empty and cause confusing SSH errors; replace the
get_vm_property call with the get_host_ip helper to obtain host_ip (while
keeping full_host from full_vm_name) so failures are handled consistently like
in configure_c2cc_hosts; update the loop that sets host_ip and full_host to use
get_host_ip("${host}") instead of get_vm_property and ensure any early-exit or
error message semantics from get_host_ip are preserved.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 3ca7a0b3-a880-479b-8cea-c95df67b1d39

📥 Commits

Reviewing files that changed from the base of the PR and between 79faeac and 8fdb3be.

📒 Files selected for processing (5)
  • test/bin/c2cc_common.sh
  • test/scenarios-bootc/el10/presubmits/el102-src@c2cc-ipv6.sh
  • test/scenarios-bootc/el10/presubmits/el102-src@c2cc.sh
  • test/scenarios-bootc/el9/presubmits/el98-src@c2cc-ipv6.sh
  • test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh
🚧 Files skipped from review as they are similar to previous changes (3)

--variable "CLUSTER_C_DOMAIN:${CLUSTER_C_DOMAIN}" \
--variable "KUBECONFIG_C:${kubeconfig_c}" \
suites/c2cc-ipsec/
c2cc_run_tests "suites/c2cc-ipsec/"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+ 15:10:42.236037388 microshift/test/bin/c2cc_common.sh:161 	local -r suites_dir=suites/c2cc-ipsec/
/home/ec2-user/microshift/test/bin/c2cc_common.sh: line 162: 2: unbound variable

We need the c2cc_run_tests' $2 and $3 to be optional

@openshift-ci

openshift-ci Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

@vanhalenar: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-tests-periodic 1e9bad9 link true /test e2e-aws-tests-periodic
ci/prow/e2e-aws-tests-bootc-arm-el10 1e9bad9 link true /test e2e-aws-tests-bootc-arm-el10
ci/prow/e2e-aws-tests-bootc-el9 1e9bad9 link true /test e2e-aws-tests-bootc-el9
ci/prow/e2e-aws-tests-bootc-el10 1e9bad9 link true /test e2e-aws-tests-bootc-el10
ci/prow/e2e-aws-tests-bootc-arm-el9 1e9bad9 link true /test e2e-aws-tests-bootc-arm-el9
ci/prow/e2e-aws-tests-bootc-periodic-el10 1e9bad9 link true /test e2e-aws-tests-bootc-periodic-el10

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants