Skip to content

Add AI skills to drive end-to-end tests#25354

Open
crazytonyli wants to merge 1 commit intotrunkfrom
explore-e2e-tests
Open

Add AI skills to drive end-to-end tests#25354
crazytonyli wants to merge 1 commit intotrunkfrom
explore-e2e-tests

Conversation

@crazytonyli
Copy link
Contributor

Description

This PR adds two skills:

  1. ios-sim-navigation. This skill is basically a reference of WDA, which Claude Code can use to interact with iOS simulators.
  2. ai-test-runner. This skill describes a workflow for running end-to-end tests on iOS simulators using the ios-sim-navigation skill.

You can use ios-sim-navigation to verify your code changes. For example, you can add this to your prompt:

After completing the code changes, use the general-purpose agent with the ios-sim-navigation skill to verify the changes on the iPhone 17 Simulator.

You can try the ai-test-runner locally using a prompt like this:

Run the test case at Tests/AgentTests/ui-tests/create-blank-page.md on the iPhone 17 simulator. Here are the test site credentials: [url], [username], [application-password].

I have translated some UI tests into test cases in Tests/AgentTests/ui-tests. If you want, you can ask Claude Code to run the full test suite, which takes 40+ minutes. You can also ask Claude Code to run on two simulators simultaneously, which halves the runtime.

@crazytonyli crazytonyli requested review from jkmassel and kean March 9, 2026 08:17
@dangermattic
Copy link
Collaborator

dangermattic commented Mar 9, 2026

1 Warning
⚠️ This PR is larger than 500 lines of changes. Please consider splitting it into smaller PRs for easier and faster reviews.

Generated by 🚫 Danger

@sonarqubecloud
Copy link

sonarqubecloud bot commented Mar 9, 2026

@crazytonyli crazytonyli added the Tooling Build, Release, and Validation Tools label Mar 9, 2026
@crazytonyli crazytonyli added this to the 26.8 milestone Mar 9, 2026
@wpmobilebot
Copy link
Contributor

App Icon📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.
App NameJetpack
ConfigurationRelease-Alpha
Build Number31375
VersionPR #25354
Bundle IDcom.jetpack.alpha
Commit4ce4ab4
Installation URL6pjn4fq5kcv9g
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@wpmobilebot
Copy link
Contributor

App Icon📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.
App NameWordPress
ConfigurationRelease-Alpha
Build Number31375
VersionPR #25354
Bundle IDorg.wordpress.alpha
Commit4ce4ab4
Installation URL68ui02in8rn88
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@wpmobilebot
Copy link
Contributor

🤖 Build Failure Analysis

This build has failures. Claude has analyzed them - check the build annotations for details.

@kean
Copy link
Contributor

kean commented Mar 11, 2026

Hey Tony, is the plan to merge the PR or is it only for evaluation purposes? I've tried running it but quickly realized that it requires https://github.com/facebookarchive/WebDriverAgent, which I don't currently have installed. Would you mind sharing the setup steps required to get these skills to work please?

What was your experience running ai-test-runner? Can you share how the output looks like from your test runs?

@crazytonyli
Copy link
Contributor Author

@kean I think we can merge this PR. I'll ask the apps infra team to see if it's possible to run the tests on CI, which I'll leave them to explore now that a locally runnable pipeline is implemented in this PR.

Have you tried the example prompts in the PR description? Do they work for you? Yes, WebDriveAgent is needed. But the skill should take care of that (cloning, running, etc), you should not need to do anything other than click "yes".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Tooling Build, Release, and Validation Tools

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants