Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .changeset/expo-verify-rename-e2e-tests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
---
---
16 changes: 8 additions & 8 deletions .claude/skills/verify-clerk-expo/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: verify-clerk-expo
description: Drive @clerk/expo and @clerk/expo-native-components in the expo-native test app (native AuthView, UserButton, UserProfileView, custom useSignIn and useSignUp flows, token cache, the Google and biometrics native modules) on an iOS simulator or Android emulator against a real Clerk development instance that the session creates and deletes, and capture video, screenshots, and the app log as evidence. The device runs on this Mac, or on a CI runner when the machine cannot run it. Use it to prove any change to packages/expo or the test app works before calling it done, to reproduce a UI bug, or to run the golden regression specs.
description: Drive @clerk/expo and @clerk/expo-native-components in the expo-native test app (native AuthView, UserButton, UserProfileView, custom useSignIn and useSignUp flows, token cache, the Google and biometrics native modules) on an iOS simulator or Android emulator against a real Clerk development instance that the session creates and deletes, and capture video, screenshots, and the app log as evidence. The device runs on this Mac, or on a CI runner when the machine cannot run it. Use it to prove any change to packages/expo or the test app works before calling it done, to reproduce a UI bug, or to run the e2e tests.
---

# verify-clerk-expo
Expand Down Expand Up @@ -103,14 +103,14 @@ With the remote backend, plain `doctor` starts nothing and pushes nothing, and `
Input reaches the app only through specs. A spec is a TypeScript file that uses the `host` fixture from `specs/fixtures.ts` and e2e's `screen` locators.

```console
$ integration/expo-native/bin/control-clerk-expo run native-auth-view --platform ios # one feature: every spec in specs/golden/native-auth-view/
$ integration/expo-native/bin/control-clerk-expo run custom-flow-sign-in/complete # one golden spec, on iOS
$ integration/expo-native/bin/control-clerk-expo run native-auth-view --platform ios # one feature: every spec in specs/features/native-auth-view/
$ integration/expo-native/bin/control-clerk-expo run custom-flow-sign-in/complete # one spec, on iOS
$ integration/expo-native/bin/control-clerk-expo run specs/explored/<name>.e2e.ts # a spec you wrote
$ integration/expo-native/bin/control-clerk-expo run native-auth-view native-js-sync # several targets in one run, with one video
$ integration/expo-native/bin/control-clerk-expo screen --platform ios # the UI tree that is on screen now; --png adds a screenshot
```

`run --all` runs every golden spec. `run` also takes `--grep <regex>`, `--retries <n>`, `--no-video`, and `--wait <seconds>`. `--retries <n>` runs a failed test again, up to `n` more times. The default is 0, so a run of your own change shows exactly what happened. The wait covers a free lane and another `run` in this worktree that holds the device. A spec limited to one platform with `test(title, { platforms: ['ios'] }, fn)` reports as skipped on the other.
`run --all` runs every committed spec. `run` also takes `--grep <regex>`, `--retries <n>`, `--no-video`, and `--wait <seconds>`. `--retries <n>` runs a failed test again, up to `n` more times. The default is 0, so a run of your own change shows exactly what happened. The wait covers a free lane and another `run` in this worktree that holds the device. A spec limited to one platform with `test(title, { platforms: ['ios'] }, fn)` reports as skipped on the other.

For a JS change on a local device, edit the source and `run` the spec, with no `up` in between. For a change to a native input, the same `run` rebuilds the dev client first. On a remote device, commit and push, then `run`.

Expand Down Expand Up @@ -158,21 +158,21 @@ A spec file that needs other settings than the standard ones has a settings file

### Check your work

Golden specs under `specs/golden/<feature>/` are committed and cover the feature map in `features/`, which starts at `features/README.md`. Prove your own change, on both platforms when the change is not specific to one, and leave the rest of the golden specs to PR CI (`.github/workflows/verify-e2e.yml`), which runs every one of them once the pull request is ready for review. When your change is to behavior a golden spec already covers, that spec is your proof: run it. A spec covers a behavior only when it asserts on it. A spec that passes through the screen you changed without checking what you changed is not your proof. Run another feature's specs yourself only when you changed code that feature shares and want to know before CI does. For new work:
The specs under `specs/features/<feature>/` are committed and cover the feature map in `features/`, which starts at `features/README.md`. Prove your own change, on both platforms when the change is not specific to one, and leave the rest of the committed specs to PR CI (`.github/workflows/verify-e2e.yml`), which runs every one of them once the pull request is ready for review. When your change is to behavior a committed spec already covers, that spec is your proof: run it. A spec covers a behavior only when it asserts on it. A spec that passes through the screen you changed without checking what you changed is not your proof. Run another feature's specs yourself only when you changed code that feature shares and want to know before CI does. For new work:

1. Write a spec under `specs/explored/`, which is gitignored. It imports the fixture as `'../fixtures.ts'`.
2. Run it by path.
3. Run `screen` after the run, pass or fail. It prints the tree that the app is on now, with each node's role, label, and testID.
4. Fix the locators from that output until the spec passes.
5. When the change adds or changes what `@clerk/expo` does, move the spec to `specs/golden/<feature>/`, change its import to `'../../fixtures.ts'`, `git add` it, and update the feature's page and its line in `features/README.md`. A change to wording or appearance alone does not earn a golden spec, because a spec that pins a label fails on the next change of wording: leave that spec in `specs/explored/`. The run directory keeps a copy of every spec the run used.
5. When the change adds or changes what `@clerk/expo` does, move the spec to `specs/features/<feature>/`, change its import to `'../../fixtures.ts'`, `git add` it, and update the feature's page and its line in `features/README.md`. A change to wording or appearance alone does not earn a committed spec, because a spec that pins a label fails on the next change of wording: leave that spec in `specs/explored/`. The run directory keeps a copy of every spec the run used.

A failing spec prints `FAIL`, the first assertion message, and the path of its failure page, and `run` exits 1. The failure page (`.verify/runs/<run-id>/e2e/failures/*.md`) lists every step, the screen tree at the failure, and a screenshot. A run in which no test executed, such as a `--grep` that matches nothing, fails with `NO_SPECS` and exits 3, because it proved nothing. A run in which every selected spec is limited to the other platform is the one exception. It exits 0, prints `skip` for each spec, and its `next` line says `nothing ran: every selected spec was left out by platform, so the run proves nothing to post`. Read that line, because the exit code alone looks like a pass.

With `--retries`, a test that fails and then passes prints `flaky` and the error of its failed attempt, and a `flaky` line under the results counts such tests. `run.json` records the test as `flaky` with its `attempts`, never as passed, and `run` exits 0. The failure page of the failed attempt and the files of every attempt stay in the run directory. A test that fails on every attempt prints `FAIL`, and `run` exits 1.

### Agent steps

e2e has a built-in agent, which a test drives with `agent.act(...)` and `agent.assert(...)`. The CLI configures it only when the machine has a Vercel AI Gateway key. The model is `anthropic/claude-haiku-5.5`, and `openai/gpt-6-luna-fast` is the gateway's backup. Supply the key in `AI_GATEWAY_API_KEY_FILE`, a file that only you can read, or in `AI_GATEWAY_API_KEY`. The `agent` line of `doctor` shows the model and where the key came from. Committed golden specs do not use agent steps.
e2e has a built-in agent, which a test drives with `agent.act(...)` and `agent.assert(...)`. The CLI configures it only when the machine has a Vercel AI Gateway key. The model is `anthropic/claude-haiku-5.5`, and `openai/gpt-6-luna-fast` is the gateway's backup. Supply the key in `AI_GATEWAY_API_KEY_FILE`, a file that only you can read, or in `AI_GATEWAY_API_KEY`. The `agent` line of `doctor` shows the model and where the key came from. Committed specs do not use agent steps.

## Evidence

Expand Down Expand Up @@ -229,7 +229,7 @@ Work in this order, with one `up` for both runs: make the branch and `git push`

On a local device `attach` writes the description itself, and no run needs a push. A remote session builds only the commit it was started on and later commits of the branch, so its before run comes before the push of the change. Its evidence reaches the description about a minute after `down`.

Attach the run of your own change. Run your new or changed spec on its own and attach that run, so the screenshots or the video show only what the change is about. You do not owe a regression run: PR CI (`.github/workflows/verify-e2e.yml`) runs every golden spec on the pull request and reports them there. If you ran other golden specs anyway, cite that run's id in the PR and leave its video in `.verify/runs/`.
Attach the run of your own change. Run your new or changed spec on its own and attach that run, so the screenshots or the video show only what the change is about. You do not owe a regression run: PR CI (`.github/workflows/verify-e2e.yml`) runs every committed spec on the pull request and reports them there. If you ran other committed specs anyway, cite that run's id in the PR and leave its video in `.verify/runs/`.

## Cleanup

Expand Down
4 changes: 2 additions & 2 deletions .claude/skills/verify-clerk-expo/features/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# @clerk/expo verification map

This directory is the maintained source for verifying the user-facing behavior of `@clerk/expo` through the `expo-native` test app in `integration/templates/expo-native`. On a local device the test app runs as a Debug dev client that loads its JS, and `packages/expo` with it, from a local Metro server. On a remote device it runs as a standalone Release build of a pushed commit, with the JS embedded. Read this index before driving the app, then use the matching feature file as the recipe. Every recipe runs through `integration/expo-native/bin/control-clerk-expo` and the golden specs under `specs/golden/<feature>/`.
This directory is the maintained source for verifying the user-facing behavior of `@clerk/expo` through the `expo-native` test app in `integration/templates/expo-native`. On a local device the test app runs as a Debug dev client that loads its JS, and `packages/expo` with it, from a local Metro server. On a remote device it runs as a standalone Release build of a pushed commit, with the JS embedded. Read this index before driving the app, then use the matching feature file as the recipe. Every recipe runs through `integration/expo-native/bin/control-clerk-expo` and the specs under `specs/features/<feature>/`.

## Test users and sign-in

Expand Down Expand Up @@ -39,7 +39,7 @@ The iOS identifiers come from `Sources/ClerkKitUI/Components/Auth/ClerkAccessibi

## Feature entry contract

Each feature file starts with an H1 title and one paragraph describing the user-visible behavior, then exactly four H2 sections in this order: `Sub-features`, `How to get to it (user POV)`, `Driving it with verify`, and `Gotchas`. `Driving it with verify` starts with `Preconditions:` and names the golden specs that prove each sub-feature. A feature file lists only entry points that a spec drives.
Each feature file starts with an H1 title and one paragraph describing the user-visible behavior, then exactly four H2 sections in this order: `Sub-features`, `How to get to it (user POV)`, `Driving it with verify`, and `Gotchas`. `Driving it with verify` starts with `Preconditions:` and names the specs that prove each sub-feature. A feature file lists only entry points that a spec drives.

## Features

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ An app that uses `useSignInWithGoogle` from `@clerk/expo/google` or `useBiometri
Preconditions:

- `up` has built the test app. The build links `@clerk/expo-google-signin` and `@clerk/expo-biometrics` from the workspace. The test app's `app.json` lists the `@clerk/expo-google-signin` config plugin, beside those of `@clerk/expo` and `@clerk/expo-native-components`, and holds three placeholder values under `extra`: a web client ID, an iOS client ID, and the iOS URL scheme that the plugin registers. They belong to no Google project.
- `specs/golden/native-modules/biometric-availability.settings.json` turns biometric sign-in on for the instance (`auth_biometric.used_for_sign_in`), and `run` applies it before the test of that spec file. The Google spec file runs on the standard settings, so a run of both has two settings groups.
- `specs/features/native-modules/biometric-availability.settings.json` turns biometric sign-in on for the instance (`auth_biometric.used_for_sign_in`), and `run` applies it before the test of that spec file. The Google spec file runs on the standard settings, so a run of both has two settings groups.

- **Google sign-in.** Run `integration/expo-native/bin/control-clerk-expo run native-modules/google-sign-in --platform ios`, and again with `--platform android`. The test taps `Native modules` on the home and taps `google-sign-in-button`. On iOS it waits for the system prompt that names `accounts.google.com` and dismisses it with `device.alert('dismiss')`. It then expects `google-result` to say `Google sign-in was cancelled`. On Android it waits for a node whose id begins `com.google.android.gms:id/`, which is a page of Google Play services, and ends there. Screenshots `google-system-prompt` and `google-cancelled`, both on iOS; on Android the video shows Google's page.
- **Biometric availability.** Run `run native-modules/biometric-availability` on each platform. The test opens the same screen, taps `biometric-availability-button`, and expects `biometric-availability-result` to say `biometric availability: biometric_authentication_unavailable` on iOS and `biometric availability: no_local_credential` on Android. Screenshot `biometric-availability`.
Expand Down
2 changes: 1 addition & 1 deletion .claude/skills/verify-clerk-expo/references/instances.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ The CLI checks those 62 leaves and the leaves a settings file declares, and no o

## Declare other settings

A spec file that runs on the standard settings declares nothing. One golden spec of this repository declares settings, `specs/golden/native-modules/biometric-availability.e2e.ts`, which turns biometric sign-in on. A spec file that needs other settings has a settings file beside it, with the same name and `.settings.json` in place of `.e2e.ts`. For `specs/golden/<feature>/<name>.e2e.ts`, that file is `specs/golden/<feature>/<name>.settings.json`:
A spec file that runs on the standard settings declares nothing. One committed spec of this repository declares settings, `specs/features/native-modules/biometric-availability.e2e.ts`, which turns biometric sign-in on. A spec file that needs other settings has a settings file beside it, with the same name and `.settings.json` in place of `.e2e.ts`. For `specs/features/<feature>/<name>.e2e.ts`, that file is `specs/features/<feature>/<name>.settings.json`:

```json
{
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/verify-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ jobs:
run: |
"$CLI" screen --platform ios > /dev/null || echo "::warning::the first launch failed here; the first spec pays for the cold start"

- name: Run the golden specs
- name: Run the end-to-end tests
timeout-minutes: 30
env:
CLERK_PLATFORM_API_KEY: ${{ secrets.MOBILE_VERIFICATION_PLATFORM_API_KEY }}
Expand Down
6 changes: 3 additions & 3 deletions integration/expo-native/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ export CLERK_E2E_DEVICE=<the id of a booted simulator, from the list above>
export CLERK_E2E_APP_PATH=<the path on the last line that the build printed>
export CLERK_PUBLISHABLE_KEY=pk_test_...
export CLERK_SECRET_KEY=sk_test_...
npx e2e run specs/golden/custom-flow-sign-in/complete.e2e.ts
npx e2e run specs/features/custom-flow-sign-in/complete.e2e.ts
```

`node src/fixture.ts ios` runs `pnpm install`, builds the packages the test app uses, and builds the test app in `integration/templates/expo-native` as a standalone app. A standalone app carries its JavaScript, so the run needs no Metro server. `node src/fixture.ts android` builds the APK for an emulator.
Expand Down Expand Up @@ -50,7 +50,7 @@ Every run creates users with addresses like `1412107c1ek3x9+clerk_test@example.c

This test file has a `.settings.json` file beside it, and passes only on an instance with those settings:

- `specs/golden/native-modules/biometric-availability.e2e.ts`
- `specs/features/native-modules/biometric-availability.e2e.ts`

The CLI's `run` puts the instance on the declared settings before the file starts, and e2e alone does not. Run them through the CLI, for example `integration/expo-native/bin/control-clerk-expo run native-modules` from the repo root.

Expand All @@ -65,7 +65,7 @@ The CLI's `run` puts the instance on the declared settings before the file start

## Layout

- `e2e.config.ts` and `specs/` are the e2e project. `specs/golden/` has the tests, `specs/fixtures.ts` has the `host` fixture they use, `specs/app.ts` names the test app, and `specs/support/` is the tests' support code. `specs/support/inputs.ts` is the only file that reads the settings above.
- `e2e.config.ts` and `specs/` are the e2e project. `specs/features/` has the tests, `specs/fixtures.ts` has the `host` fixture they use, `specs/app.ts` names the test app, and `specs/support/` is the tests' support code. `specs/support/inputs.ts` is the only file that reads the settings above.
- `bin/` and `src/` are the CLI. Under the CLI the test process gets the same settings, with one difference: in place of the secret key it gets the address and token of a stand-in that the CLI runs on this machine. The stand-in forwards the three Backend API calls the tests make, for this run's test users only, so the test process never holds the key.
- `test/` and `testing/` are the unit tests of both. `npm test` runs them, with no device, key, or network.
- `src/core/`, `specs/support/`, `specs/fixtures.ts`, and `e2e.config.ts` are the same files, byte for byte, in clerk-ios, clerk-android, and clerk/javascript. `src/core/MANIFEST` lists them, and the CLI's `doctor` fails when they differ from it.
Loading
Loading