Skip to content

[Kibana Integration Testing] Update workflows to run one after another and add CI status checking#9723

Merged
tkajtoch merged 7 commits into
mainfrom
feat/kibana-integration-testing-chain
Jun 19, 2026
Merged

[Kibana Integration Testing] Update workflows to run one after another and add CI status checking#9723
tkajtoch merged 7 commits into
mainfrom
feat/kibana-integration-testing-chain

Conversation

@tkajtoch

Copy link
Copy Markdown
Member

Summary

This PR updates the Kibana regression integration testing workflows to:

  1. Dispatch update_kibana_dependencies__open_pr.yml at the end of update_kibana_dependencies__prepare_changes.yml
  2. Trigger CI on the created PR by posting a /ci comment
  3. Check CI status after ~2 hours of PR creation
  4. Check CI status after ~4 hours of PR creation if the first check returned zero CI failures and some pending checks

The PR label trigger on EUI PRs and commenting the status of Kibana CI will be added separately.

API Changes

N/A

Screenshots

N/A

Impact Assessment

Note: Most PRs should be tested in Kibana to help gauge their Impact before merging.

  • 🔴 Breaking changes — What will break? How many usages in Kibana/Cloud UI are impacted?
  • 💅 Visual changes — May impact style overrides; could require visual testing. Explain and estimate impact.
  • 🧪 Test impact — May break functional or snapshot tests (e.g., HTML structure, class names, default values).
  • 🔧 Hard to integrate — If changes require substantial updates to Kibana, please stage the changes and link them here.

Impact level: 🟢 None

Release Readiness

  • Documentation: {link to docs page(s)}
  • Figma: {link to Figma or issue}
  • Migration guide: {steps or link, for breaking/visual changes or deprecations}
  • Adoption plan (new features): {link to issue/doc or outline who will integrate this and where}

QA instructions for reviewer

These changes are not possible to be run prior to merging due to the token policy restricting us from running workflows from branches other than main.

Checklist before marking Ready for Review

Reviewer checklist

  • Approved Impact Assessment — Acceptable to merge given the consumer impact.
  • Approved Release Readiness — Docs, Figma, and migration info are sufficient to ship.

@tkajtoch tkajtoch self-assigned this Jun 18, 2026
@tkajtoch tkajtoch requested a review from a team as a code owner June 18, 2026 01:10
@tkajtoch tkajtoch added skip-changelog Use on PRs to skip changelog requirement (Don't delete - used for automation) infrastructure skip-vrt Skips Visual Regression Testing job in CI (clearly state why you're using it in your PR) labels Jun 18, 2026
@weronikaolejniczak weronikaolejniczak self-requested a review June 18, 2026 08:22
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
2 visual difference(s) found - expand to review, then click Approve visual changes to update baselines

euipopover (1 difference)

StoryBeforeAfterDiff
playground desktop

euidatagrid (1 difference)

StoryBeforeAfterDiff
additional controls options desktop

@elasticmachine

Copy link
Copy Markdown
Collaborator

💚 Build Succeeded

History

cc @tkajtoch

@tkajtoch

Copy link
Copy Markdown
Member Author

Test

check_ci_status_1:
name: Check CI status - attempt 1/2
runs-on: ubuntu-slim
environment: delayed-2h

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A good trick! 👍🏻

@weronikaolejniczak weronikaolejniczak left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This cannot be tested now because of the GH ephemeral tokens, so I'm fine merging to main as-is and fixing any issues as they arise. 🟢

Some thoughts after reading through the diff:

  • Should we add concurrency and cancel-in-progress: true so that there can always be only one run in progress? We could group them by the source PR number. I wonder if this would also cancel the probe environments, I believe it should 🤔
  • This is more work but I wonder if we could have a cleanup workflow on: closed, unlabeled. It could cancel the workflow runs and close the Kibana PR. Or another idea is a cron that deletes branches older than N.
  • Could we surface the Buildkite job links in the comment body? "Failing checks" section. This would give an overview of what went wrong without having to go to the Kibana PR. This is already fetched in link: gh pr checks --required --json name,link,bucket, no?
  • After calling gh pr comment "$PR_URL" --body "/ci", this could fail due to auth or race condition or wrong reference or GH outage, whatever, never triggering the CI and the env silently sitting would finally report checks pending, maybe we could somehow surface that error in a comment so that the PR author can go and trigger the CI manually.
  • If open_pr fails we can delete the branch on the eui-kibana fork inline (or as part of the cleanup workflow).
  • As we talked, I believe 1h and 2h is enough but we can start with 2h and 4h, and see how it goes.

@tkajtoch

Copy link
Copy Markdown
Member Author

These are all great points @weronikaolejniczak! I purposefully limited the scope of this to see if we can get it working first, but I agree that we need a cleanup job, concurrency setup and a way to re-test in Kibana when new changes are added to EUI PRs. I suspect we might encounter issues during testing, and I'm ready to fix them all! :D

@tkajtoch tkajtoch merged commit 73a3046 into main Jun 19, 2026
4 checks passed
@tkajtoch tkajtoch deleted the feat/kibana-integration-testing-chain branch June 19, 2026 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infrastructure skip-changelog Use on PRs to skip changelog requirement (Don't delete - used for automation) skip-vrt Skips Visual Regression Testing job in CI (clearly state why you're using it in your PR)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants