diff --git a/.changeset/expo-verify-port-sessions.md b/.changeset/expo-verify-port-sessions.md new file mode 100644 index 00000000000..a845151cc84 --- /dev/null +++ b/.changeset/expo-verify-port-sessions.md @@ -0,0 +1,2 @@ +--- +--- diff --git a/.claude/skills/verify-clerk-expo/SKILL.md b/.claude/skills/verify-clerk-expo/SKILL.md index 8101c80aa14..4b8eeb71991 100644 --- a/.claude/skills/verify-clerk-expo/SKILL.md +++ b/.claude/skills/verify-clerk-expo/SKILL.md @@ -65,7 +65,7 @@ A Mac has four iOS lanes and two Android lanes, shared by every worktree on it. ### Borrow a device on a CI runner -A machine that is not a Mac cannot run the simulator, and a machine with no hardware virtualization cannot run the emulator. There the CLI leases a device on a GitHub Actions runner and drives it through a tunnel. That machine needs Node 24.8.0 or newer on 24, the Platform API key, and access to GitHub, and `doctor` checks each. In the worktree it needs `npm ci --prefix integration/expo-native` and no `pnpm install`, because the runner builds the app. The session builds a pushed commit, never your working tree, so commit and push before `up` or `run`. +A machine that is not a Mac cannot run the simulator, and a machine with no hardware virtualization cannot run the emulator. There the CLI leases a device on a GitHub Actions runner and drives it through a tunnel. That machine needs Node 24.8.0 or newer on 24, the Platform API key, and access to GitHub, and `doctor` checks each. In the worktree it needs `npm ci --prefix integration/expo-native` and no `pnpm install`, because the runner builds the app. The session builds a pushed commit. When the app sources differ from it, `up` and `run` send the difference with it, and the session builds the commit with your changes on top. ```console $ git push @@ -225,7 +225,7 @@ On a machine whose `gh` can attach, `attach` edits the description itself and pr When this machine cannot attach and there is no session to hand the files to, or the hand-off fails, `attach` fails and says which. Then name the run id in the PR and say that the evidence was not attached. A run on a local device has no session, so its fix is a newer `gh`. [Remote devices](references/remote.md) has what the repository needs before a hand-off can be published, and its limits. -`attach` writes only to an open pull request, and fails on one that is closed or merged. It refuses a run that is tainted, that has a failing spec or no passing one, or whose `app.log` names a user that the run did not create. A before run can have failing specs. `attach` refuses a before run that is tainted, that was never sealed, or whose `app.log` names a user that the run did not create. +`attach` writes only to an open pull request, and fails on one that is closed or merged. It refuses a run that is tainted, that ran on a session build with uncommitted changes, that has a failing spec or no passing one, or whose `app.log` names a user that the run did not create. A before run can have failing specs. `attach` refuses a before run that is tainted, that ran on a session build with uncommitted changes, that was never sealed, or whose `app.log` names a user that the run did not create. Work in this order, with one `up` for both runs: make the branch and `git push` it, `up`, `run` the spec without your change when it changes something that exists, make the change, commit, `git push`, `run` the spec again, open the pull request, `attach` with the before run as `--before `, `down`. Open the pull request after the last run, so that it is not left open without its evidence. Commit before the second `run`, so that the line `attach` writes names a commit the pull request holds. In the pull request, name the run id and the specs you ran, and say what you did not run. diff --git a/.claude/skills/verify-clerk-expo/references/freshness.md b/.claude/skills/verify-clerk-expo/references/freshness.md index 86db0ea7689..606ea19bfe3 100644 --- a/.claude/skills/verify-clerk-expo/references/freshness.md +++ b/.claude/skills/verify-clerk-expo/references/freshness.md @@ -60,4 +60,4 @@ A JS edit does not change it, and a new version of a JS-only package does not ch A later standalone build with the same fingerprint runs no `expo prebuild`, `xcodebuild`, or Gradle. It builds the workspace packages, exports the bundle from the working tree with `expo export:embed`, compiles it with the test app's `hermesc`, and puts it in a copy of the kept app. The build fails unless the app then holds exactly the bundle it compiled. It builds natively instead, and prints `not reused:` with the reason, when the kept app holds no Hermes bundle, when it runs another Hermes bytecode version, when an Android bundle has image assets, or when the Android SDK has no build-tools 35 or newer to align the APK again. On Android the app is signed with the generated project's debug keystore when the working tree has one and with a new key otherwise, so a device that already holds the app under another key needs it uninstalled first. -A job of the workflow that had to build natively stores the app as a run artifact named `verify-expo-native--` for seven days, and a later job takes it only from a run of the same branch of this repository. When there is none, the job builds natively. Unset, nothing is kept and every standalone build is a native build. A remote session does not set it. +A job of the workflow that had to build natively stores the app as a run artifact named `verify-expo-native--` for seven days, and a later job takes it only from a run of the same branch of this repository. When there is none, the job builds natively. Unset, nothing is kept and every standalone build is a native build. A remote session sets it. Before its agent starts, the session downloads the native builds of its platform that the newest runs of the workflow on the same branch kept, at most three, each under its fingerprint. `up` then builds natively only when none of them has the fingerprint of the tree it builds. A session stores no native build, because the artifacts of a run are public. It stores no DerivedData or Pods either. It saves the pnpm store on Android and on a Blacksmith runner, and the Gradle cache on a Blacksmith runner. diff --git a/.claude/skills/verify-clerk-expo/references/output.md b/.claude/skills/verify-clerk-expo/references/output.md index 33c71e60b31..6dec2165b19 100644 --- a/.claude/skills/verify-clerk-expo/references/output.md +++ b/.claude/skills/verify-clerk-expo/references/output.md @@ -31,19 +31,19 @@ On an error the object is `{ "ok": false, "error": { "code", "message", "fix", " `retryable` is true for `POOL_FULL`, `DEVICE_BUSY`, `LEASE_LOST`, and `RATE_LIMITED`. For those, the same command can succeed later with no other change. -| Code | Meaning | -| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `USAGE` | The command line, a spec's declaration, or an environment variable is malformed | -| `NOT_READY` | A precondition is missing, such as Metro that does not serve the current code. The fix names it | -| `POOL_FULL` | All four iOS lanes, or both Android lanes, on this machine are in use | -| `DEVICE_BUSY` | Another command in this worktree is driving the device, or a run on the other platform is using this worktree's application | -| `LEASE_LOST` | The leased device or the remote session is gone | -| `BUILD_FAILED` | A step of the test app's build failed, such as `turbo build`, `expo prebuild`, `xcodebuild`, or `gradlew`. The message names the step and has the last 15 lines of its output. With the remote backend, also that the app sources have uncommitted changes, that GitHub does not have HEAD, or that the session could not build the commit | -| `KEYS_MISSING` | No Platform API credential works on this machine, or `AI_GATEWAY_API_KEY_FILE` names a file that is missing, empty, or readable by other users | -| `INSTANCE_MISCONFIGURED` | The Clerk instance does not show the settings a spec needs | -| `NOT_TEST_IDENTITY` | Reserved. No command raises it. A run whose `app.log` names a user that the run did not create fails `attach` with `EVIDENCE_UNSAFE` | -| `NO_SPECS` | The selection matched no spec, or no selected spec ran | -| `E2E_CRASHED` | e2e exited without a readable report. Read `e2e.log` in the run directory | -| `EVIDENCE_UNSAFE` | The run cannot be attached, or a path is not safe to delete | -| `UNSUPPORTED` | This machine cannot do what the command asked, such as `--backend local` for a device it cannot run | -| `RATE_LIMITED` | Clerk's Platform API kept answering 429 | +| Code | Meaning | +| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `USAGE` | The command line, a spec's declaration, or an environment variable is malformed | +| `NOT_READY` | A precondition is missing, such as Metro that does not serve the current code. The fix names it | +| `POOL_FULL` | All four iOS lanes, or both Android lanes, on this machine are in use | +| `DEVICE_BUSY` | Another command in this worktree is driving the device, or a run on the other platform is using this worktree's application | +| `LEASE_LOST` | The leased device or the remote session is gone | +| `BUILD_FAILED` | A step of the test app's build failed, such as `turbo build`, `expo prebuild`, `xcodebuild`, or `gradlew`. The message names the step and has the last 15 lines of its output. With the remote backend, also that GitHub does not have HEAD, that the session could not build the commit, or that the uncommitted changes to the app sources do not apply to it, change a file outside the app sources, or are larger than 16 MiB. A session on `main` refuses uncommitted changes | +| `KEYS_MISSING` | No Platform API credential works on this machine, or `AI_GATEWAY_API_KEY_FILE` names a file that is missing, empty, or readable by other users | +| `INSTANCE_MISCONFIGURED` | The Clerk instance does not show the settings a spec needs | +| `NOT_TEST_IDENTITY` | Reserved. No command raises it. A run whose `app.log` names a user that the run did not create fails `attach` with `EVIDENCE_UNSAFE` | +| `NO_SPECS` | The selection matched no spec, or no selected spec ran | +| `E2E_CRASHED` | e2e exited without a readable report. Read `e2e.log` in the run directory | +| `EVIDENCE_UNSAFE` | The run cannot be attached, or a path is not safe to delete | +| `UNSUPPORTED` | This machine cannot do what the command asked, such as `--backend local` for a device it cannot run | +| `RATE_LIMITED` | Clerk's Platform API kept answering 429 | diff --git a/.claude/skills/verify-clerk-expo/references/remote.md b/.claude/skills/verify-clerk-expo/references/remote.md index c2735b6ad2d..864ef3ac558 100644 --- a/.claude/skills/verify-clerk-expo/references/remote.md +++ b/.claude/skills/verify-clerk-expo/references/remote.md @@ -31,13 +31,21 @@ Without it, `doctor` reports `gh-attach` with its fix, and `attach` hands the ev ## What a session builds -A session builds a commit that GitHub has, never your working tree. It runs `node src/fixture.ts ` on the runner, which is the same module the local build calls. The result is a standalone app: a Release build with `expo-dev-client` left out and the JS embedded, so the device needs no Metro. +A session builds a commit that GitHub has. It runs `node src/fixture.ts ` on the runner, which is the same module the local build calls. The result is a standalone app: a Release build with `expo-dev-client` left out and the JS embedded, so the device needs no Metro. `up` starts the session's run on your branch as GitHub has it. The run builds only the commit it was started on, or a later commit of that branch. When HEAD is not on that branch on GitHub, or is behind it, the run fails at the step `Read the request` and no session starts. Push or pull, then run `up` again. A detached HEAD has no branch, so `doctor` and `up` fail until you switch to a branch that GitHub has. -The build key of a local dev client covers the native inputs only. The build key of a standalone app also covers everything the JS bundle is made from: `.npmrc`, `package.json`, `pnpm-lock.yaml`, `pnpm-workspace.yaml`, `tsconfig.json`, `turbo.json`, `packages/clerk-js`, `packages/expo`, `packages/expo-biometrics`, `packages/expo-google-signin`, `packages/expo-native-components`, `packages/react`, `packages/shared`, and `integration/templates/expo-native`. `up` and `run` fail with `BUILD_FAILED` when one of those paths has uncommitted changes or when GitHub does not have HEAD, and the fix is to commit and push. +The build key of a local dev client covers the native inputs only. The build key of a standalone app also covers everything the JS bundle is made from: `.npmrc`, `package.json`, `pnpm-lock.yaml`, `pnpm-workspace.yaml`, `tsconfig.json`, `turbo.json`, `packages/clerk-js`, `packages/expo`, `packages/expo-biometrics`, `packages/expo-google-signin`, `packages/expo-native-components`, `packages/react`, `packages/shared`, and `integration/templates/expo-native`. `up` and `run` fail with `BUILD_FAILED` when GitHub does not have HEAD, and the fix is to push. These paths are the app sources. -After you push an edit to one of those paths, `run` asks the same session to build the new commit. The session builds it only when GitHub shows it as a later commit of the branch the session was started on. After a rebase or an amend, the pushed commit is not a later commit of that branch, and the build fails with `BUILD_FAILED`. Run `down`, then `up`. The session keeps its generated native project while the native inputs are unchanged, so a JS edit skips `pnpm add`, `expo install`, and `expo prebuild --clean`. Specs run from your working tree, so an edit to a spec needs no commit. A session that already holds the build for your commit is reused as it is. +When the app sources differ from HEAD, `up` and `run` send the difference as one patch: `git diff --binary HEAD` of the app sources, with new files that git does not ignore. The session resets its checkout to the commit, applies the patch, builds, and installs the app on the same device. Edits to a tracked file are sent even when the file matches an ignore rule. When the patch does not apply, the build fails with `BUILD_FAILED` and says so; commit and push the changes, then run again. The patch is at most 16 MiB. The session refuses a patch that changes a file outside the app sources. It also refuses any patch when the session runs on `main`, so work on a branch to send uncommitted changes. + +A patch that changes only JS leaves the native fingerprint as it is, so the session puts a new bundle in the native build it already has and runs no `expo prebuild`, `xcodebuild`, or Gradle. A patch that changes a native input changes the fingerprint, and the session builds natively. [Freshness](freshness.md) lists what the fingerprint covers. + +A run on such a build is not tied to a commit. Its `run.json` names the patch under `remote.builtPatch`, and `attach` refuses it, as the run and as a before run. Commit and push the change, `run` again, and attach that run. + +While the session builds, `up` and `run` print each new line of the build output, after `build |`. + +After you push an edit to one of those paths, `run` asks the same session to build the new commit, with no patch. The session builds it only when GitHub shows it as a later commit of the branch the session was started on. After a rebase or an amend, the pushed commit is not a later commit of that branch, and the build fails with `BUILD_FAILED`. Run `down`, then `up`. While the native fingerprint is unchanged, a JS edit puts a new bundle in the native build the session already has. A session starts from a native build that a `Verify end-to-end tests` run of the branch kept for the same fingerprint, when there is one, and builds natively otherwise; [Freshness](freshness.md) has the rule. Specs run from your working tree, so an edit to a spec needs no commit. A session that already holds the build for your commit is reused as it is. A session keeps the code under `src/core/` from the commit it started on. When `src/core/MANIFEST` in your checkout differs from the one the session started on, `up` and `run` fail with `NOT_READY`. The fix is to commit and push the change, then `down`, then `up`. The session reads `src/fixture.ts` and `src/platform/session-device.ts` again from each commit it builds. The build key leaves out everything under `.claude/`, so a commit that changes only those two files does not make a held session rebuild. After such a commit, run `down`, then `up`. diff --git a/.github/workflows/verify-e2e.yml b/.github/workflows/verify-e2e.yml index 4ff977bdd09..08c4c76187d 100644 --- a/.github/workflows/verify-e2e.yml +++ b/.github/workflows/verify-e2e.yml @@ -160,34 +160,7 @@ jobs: NATIVE_ID: ${{ steps.native.outputs.id }} NATIVE_DIR: ${{ steps.native.outputs.dir }} NATIVE_APP: ${{ steps.native.outputs.app }} - run: | - set -euo pipefail - - fetch() { - local name="verify-expo-native-$NATIVE_ID" artifact unpacked - artifact="$(gh api "repos/$GITHUB_REPOSITORY/actions/artifacts?name=$name&per_page=100" --jq "[.artifacts[] | select(.expired == false and .workflow_run.head_repository_id == .workflow_run.repository_id and .workflow_run.head_branch == \$ENV.BRANCH)] | sort_by(.created_at) | last | .id // empty")" || return 1 - if [ -z "$artifact" ]; then - return 1 - fi - unpacked="$(mktemp -d "$RUNNER_TEMP/native-build.XXXXXX")" || return 1 - gh api "repos/$GITHUB_REPOSITORY/actions/artifacts/$artifact/zip" > "$unpacked.zip" || return 1 - unzip -q "$unpacked.zip" -d "$unpacked" || return 1 - mkdir -p "$NATIVE_DIR" || return 1 - if [ "$PLATFORM" = "ios" ]; then - mv "$unpacked" "$NATIVE_DIR/$NATIVE_APP" || return 1 - else - mv "$unpacked/$NATIVE_APP" "$NATIVE_DIR/$NATIVE_APP" || return 1 - fi - echo "Took the native build from artifact $artifact ($name)." - } - - if fetch; then - echo "found=kept" >> "$GITHUB_OUTPUT" - exit 0 - fi - - echo "No native build of this branch is kept for $NATIVE_ID, so this job builds it." - echo "found=none" >> "$GITHUB_OUTPUT" + run: integration/expo-native/bin/fetch-native-build.sh - name: Wait for the device image wait: device-image diff --git a/.github/workflows/verify-remote.yml b/.github/workflows/verify-remote.yml index 3813d4eeccf..4953b144fb7 100644 --- a/.github/workflows/verify-remote.yml +++ b/.github/workflows/verify-remote.yml @@ -64,10 +64,14 @@ jobs: if: needs.plan.outputs.mode == 'session' runs-on: ${{ needs.plan.outputs.runner }} timeout-minutes: ${{ fromJSON(needs.plan.outputs.timeout) }} + permissions: + contents: read + actions: read env: VERIFY_SESSION_REQUEST: ${{ needs.plan.outputs.request }} PLATFORM: ${{ fromJSON(needs.plan.outputs.request).platform }} DEVICE: ${{ needs.plan.outputs.device }} + VERIFY_NATIVE_CACHE: integration/expo-native/.verify/native # agent-device reaps its daemon, its XCTest runner, and its device lease after five idle minutes by default. AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS: '0' AGENT_DEVICE_IOS_RUNNER_IDLE_STOP_MS: '0' @@ -121,6 +125,17 @@ jobs: echo "id=$(node "$PACKAGE/src/platform/android/session-lane.ts" serial)" >> "$GITHUB_OUTPUT" fi + - name: Fetch the native builds that runs of this branch kept + id: kept + if: needs.plan.outputs.device != '' + timeout-minutes: 5 + env: + GH_TOKEN: ${{ github.token }} + BRANCH: ${{ github.ref_name }} + NATIVE_CACHE: ${{ env.VERIFY_NATIVE_CACHE }} + NATIVE_APP: ${{ env.PLATFORM == 'ios' && 'ClerkExpoNativeBuildFixture.app' || 'app-release.apk' }} + run: integration/expo-native/bin/fetch-native-build.sh + - name: Start the session agent and tunnel id: tunnel timeout-minutes: 20 @@ -128,6 +143,7 @@ jobs: VERIFY_SESSION_DEVICE_ID: ${{ steps.device.outputs.id }} VERIFY_SESSION_DEVICE_NAME: ${{ needs.plan.outputs.device }} VERIFY_SESSION_DEVICE_MODULE: integration/expo-native/src/platform/session-device.ts + VERIFY_SESSION_DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} GITHUB_TOKEN: ${{ github.token }} CLOUDFLARED_VERSION: '2026.9.3' run: | @@ -238,3 +254,4 @@ jobs: if-no-files-found: ignore retention-days: 3 compression-level: 0 + diff --git a/integration/expo-native/bin/fetch-native-build.sh b/integration/expo-native/bin/fetch-native-build.sh new file mode 100755 index 00000000000..a8bd6486cf6 --- /dev/null +++ b/integration/expo-native/bin/fetch-native-build.sh @@ -0,0 +1,50 @@ +#!/usr/bin/env bash +set -euo pipefail + +: "${GITHUB_REPOSITORY:?}" "${GITHUB_OUTPUT:?}" "${RUNNER_TEMP:?}" "${BRANCH:?}" "${PLATFORM:?}" "${NATIVE_APP:?}" + +OURS='.workflow_run.head_repository_id == .workflow_run.repository_id and .workflow_run.head_branch == $ENV.BRANCH' + +take() { + local artifact="$1" dir="$2" unpacked + unpacked="$(mktemp -d "$RUNNER_TEMP/native-build.XXXXXX")" || return 1 + gh api "repos/$GITHUB_REPOSITORY/actions/artifacts/$artifact/zip" > "$unpacked.zip" || return 1 + unzip -q "$unpacked.zip" -d "$unpacked" || return 1 + mkdir -p "$dir" || return 1 + if [ "$PLATFORM" = "ios" ]; then + mv "$unpacked" "$dir/$NATIVE_APP" || return 1 + else + mv "$unpacked/$NATIVE_APP" "$dir/$NATIVE_APP" || return 1 + fi +} + +if [ -z "${NATIVE_ID:-}" ]; then + : "${NATIVE_CACHE:?}" + prefix="verify-expo-native-$PLATFORM-" + runs="$(gh api "repos/$GITHUB_REPOSITORY/actions/workflows/verify-e2e.yml/runs?branch=$BRANCH&per_page=20" --jq '.workflow_runs[] | select(.head_repository.id == .repository.id and .head_branch == $ENV.BRANCH) | .id' || true)" + taken=0 + for run in $runs; do + while read -r artifact name; do + [ -n "$name" ] && [ ! -e "$NATIVE_CACHE/${name#verify-expo-native-}" ] || continue + if take "$artifact" "$NATIVE_CACHE/${name#verify-expo-native-}"; then + echo "Took the native build from artifact $artifact ($name)." + taken=$((taken + 1)) + fi + done < <(gh api "repos/$GITHUB_REPOSITORY/actions/runs/$run/artifacts?per_page=100" --jq ".artifacts[] | select(.expired == false and (.name | startswith(\"$prefix\"))) | \"\(.id) \(.name)\"" || true) + [ "$taken" -lt 3 ] || break + done + echo "taken=$taken" >> "$GITHUB_OUTPUT" + exit 0 +fi + +: "${NATIVE_DIR:?}" +name="verify-expo-native-$NATIVE_ID" +artifact="$(gh api "repos/$GITHUB_REPOSITORY/actions/artifacts?name=$name&per_page=100" --jq "[.artifacts[] | select(.expired == false and $OURS)] | sort_by(.created_at) | last | .id // empty" || true)" +if [ -n "$artifact" ] && take "$artifact" "$NATIVE_DIR"; then + echo "Took the native build from artifact $artifact ($name)." + echo "found=kept" >> "$GITHUB_OUTPUT" + exit 0 +fi + +echo "No native build of this branch is kept for $NATIVE_ID, so this job builds it." +echo "found=none" >> "$GITHUB_OUTPUT" diff --git a/integration/expo-native/specs/support/config.ts b/integration/expo-native/specs/support/config.ts index 3a4f452a675..2ede77495bc 100644 --- a/integration/expo-native/specs/support/config.ts +++ b/integration/expo-native/specs/support/config.ts @@ -1,5 +1,6 @@ import { readFileSync } from 'node:fs'; import { createRequire } from 'node:module'; +import { homedir } from 'node:os'; import { dirname, join } from 'node:path'; import { fileURLToPath } from 'node:url'; import type { E2EConfig } from 'e2e'; @@ -20,11 +21,12 @@ export function engineDevice(device: Device): readonly string[] | DeviceProvider const token = new Secret('session-bearer', readFileSync(device.tokenFile, 'utf8').trim()); return { name: 'borrowed', - acquire: async () => ({ + acquire: async ({ env }) => ({ id: device.id, deviceId: device.id, device: device.id, daemon: token.use('e2e-provider-lease', (authToken) => ({ baseUrl: device.daemonUrl, authToken })), + client: { cwd: env.AGENT_DEVICE_STATE_DIR ?? join(env.HOME ?? homedir(), '.agent-device') }, }), release: async () => {}, }; diff --git a/integration/expo-native/src/core/MANIFEST b/integration/expo-native/src/core/MANIFEST index e0c43057e7a..6b76a823837 100644 --- a/integration/expo-native/src/core/MANIFEST +++ b/integration/expo-native/src/core/MANIFEST @@ -3,7 +3,7 @@ 38e8e99663059810b616b591dfd79bdcc7781f72b7269875294f3702b3edde84 specs/support/agent.ts 6d1ef7700a89e19d0b1214d5a6815ba757d02857c555b8bf108f2e0c9a64aad3 specs/support/busy-runner.ts 153cda5ccf3ad0e38a35e238a52c64633a9b8de9e2086ff060ddb661a2438b1d specs/support/clerk.ts -6c658e254cb6ad8ba9f3d05638941e0809e6709f0abb8096340559b2f81b156a specs/support/config.ts +e313b75e09a06400e050c9405d95d274e885134248837c4230e4c27d9d0d6c04 specs/support/config.ts be45e0eb42f715d603c179c7a86750949ff2ee3cc0da73082179801fe1b8f2ab specs/support/device.ts cec7a42bf8d35e174fd3387cbcc638c01bacda74fc15ea1b14eda8abaa200d34 specs/support/fill.ts 7415ccd89f715d40ca74796a5b235aab02e640c8674d457d3912955ab67146e1 specs/support/inputs.ts @@ -18,10 +18,10 @@ d56c29efc8061adfd65b89f0f02465af3234ecd52289ef6d9f338483afeadeb5 src/core/broke 7c741449c755524bba80f1ce314554293003a34f7f7c30696015ba1c0c3dc1ab src/core/claims.ts 1fc59d13e7197e6c7099c0467744b978d498d1fe409c88254d44e2f99913a7b5 src/core/clerk.ts 8d0fb398aec4187b15b82ef88142b0a548f0ff4a0d26902c3e01ea0659283236 src/core/cli.ts -4aeacea985f5ef80001e5695feec53182abb9695eb07130898568743755fc8af src/core/devices.ts +538811a805dc7aaec864d0e86edd7f60dbe0e23fea257b07ad98dcb0d5c5c430 src/core/devices.ts 3ae461d6aa75a2cefac0392a9ad124aec4105ac25b9320371743e2ea90fe2202 src/core/driver.ts 14e74b156592a645ae579e3a8f900c5bb0a8be487adfacd4ff94ca268ce0dbdf src/core/e2e.ts -c9a29e27eb835d9c3fc9b0bd394ff55b5f1fcc2b2fc0d96da51197e5e7bca003 src/core/evidence.ts +d0780690b31fb3afdb228dcba1c4eda5cef41c6efbc7f982e317851f5ea5d041 src/core/evidence.ts 27bfd4ce59937aa8c97d9fd6849cf52f3da05626fbc9388385f4b5e684577d36 src/core/exec.ts e2075f17f52ded8dea64d8a2473820396ca5ac1a22db59f21a092da28e55911f src/core/github-report.ts 721453f519bba1374dcf11aaae58f7af06e5555be756df6d3beaf52a9c2a2eda src/core/instances/base.json @@ -34,19 +34,19 @@ cf6aa68be73e4abaf3b63d5d84c2e9eee6197412f607ec53d0f05f7b058db420 src/core/insta 098d3eef9d2521054f235f47e8658efb71d482087b927645489fc2240a03f5e8 src/core/launch.mjs a0ee9e4bfd34d4659685108e5e54eb1ec2aff6248236c358e6d3b14b2e2ff6a0 src/core/ledgers.ts afab94a970286c40d6857b03ae4fa39ec59786274beacf6f1e77f6530bf46c54 src/core/manifest.ts -7294bbab3d892f0d9aee1e863f84e32a7f5dc3c467a7b6177be7e4ff2f29fecc src/core/publish.ts -83a0a19bc3cf6ce0c1435f6624204a17060000064cf6ca49b8fa39143d7871b6 src/core/remote/backend.ts +999f5017607d73482bd6e578d347c0fc4e8014e90b39c13827cbd1efba3d9355 src/core/publish.ts +adff12404598fdca77b6f869bdc17cbec27365c094e72e9c70cb48797c67065f src/core/remote/backend.ts 37fc42707be7d8bd5f0313a664a8465287ebb0970c520c0e0617f6ff6c9a614e src/core/remote/github.ts d9f7e8d2f69960aba668dc2e18d6d61481c80967459922ee9fe33265b27c1735 src/core/remote/handoff.ts 2ec436b2c53b34269e0b62ab54e41e4d9a55146cccc46f0f31591db05cdbf5c1 src/core/remote/preflight.ts -bb69fe1f3581942285894c0901d02889c08b4293e4d9db77934fe3cdd2d4a399 src/core/remote/protocol.ts -5a4ee0e7ed20feaeb711b1527083299b432d3fc0b7f428b636678a8a2f57581f src/core/remote/recipe.ts -bbb997d036c320257783f9c576497e4f02cde99a9cc8d164c4dfde40f76e4421 src/core/remote/session-agent.ts +237f675b28f647bfc02a8e0cc9b0eddf9dd8ee0e801506bc29be2ad3c66a0c98 src/core/remote/protocol.ts +913776a45c1bb3dabaf5b9802adcb7f201f7d26104b00fcb972df2979d762734 src/core/remote/recipe.ts +eff1479bb391111364544ba9efa84237e37731ea2c58d8f0504ae2636239cc44 src/core/remote/session-agent.ts 9cb8ac8bdae263362b9e8a3f24592f20cc80fb367417ac0d2ed34fb3f383021b src/core/remote/session.ts d614ee5c98a85d15037b744a67404496273f62985fd7d6a049cacfd5ab335503 src/core/remote/settings.ts e734336ea3b345057b69193790e2efce9a662c67ec5b4a426c811f999b45ccdb src/core/remote/tunnel.ts 685eabf010c7f63d076cfd3da1a681b5cc8024720538c1bf3a99792b563d8b60 src/core/slot.ts a8838aaf1fa9e4a6bc4e350ea674079201da76c5962656039b96db2505591aed src/core/state.ts -b242eff9cb23ed78af02190a7decd47d8299089996648b6f0c2540ca2fe4c0d1 src/core/types.ts -79b822b663c1ac1748cf3c8ec390ef9204378f4c97531f4da9b0bf5736a20d36 src/core/verbs.ts -9be1816773216373548ec9933978b53ace3d3bfdba17b72c1e5ed6d8aef80f62 src/core/workspace.ts +c37fbd8a43613f61e2003637c2b333a5b33c33c44fe6164a37ff9ca65fe68905 src/core/types.ts +3f2e3c6de06f7683882465696dd184e02be2bfb712a18f768e6d14f51c1e52b6 src/core/verbs.ts +31910ffb6bdce6effe85409282c4be5bd206cee15e41ceef04ce2498f6a38a02 src/core/workspace.ts diff --git a/integration/expo-native/src/core/devices.ts b/integration/expo-native/src/core/devices.ts index dca5dbc8bbc..921e75e086f 100644 --- a/integration/expo-native/src/core/devices.ts +++ b/integration/expo-native/src/core/devices.ts @@ -116,16 +116,18 @@ export function readBuiltApp(workspace: Workspace, key: BuildKey): LocalBuild | async function ensureBuild(host: HostAdapter, platform: Platform, workspace: Workspace, backend: DeviceBackend, held: Lease | null, heldIsAlive: boolean, progress: (line: string) => void): Promise<{ app: BuiltApp; view: BuildView }> { const key = await computeBuildKey(host, platform, backend.kind, workspace.worktree); if (backend.sourceCommit !== undefined) { - const builtForThisApp = held !== null && held.backend === 'remote' && held.installedBuild === key ? held.builtSha : null; + const builtForThisApp = held !== null && held.backend === 'remote' && held.installedBuild === key && held.builtPatch === null ? held.builtSha : null; const onSession = heldIsAlive ? builtForThisApp : null; - const sourceSha = - onSession ?? - (await backend.sourceCommit({ worktree: workspace.worktree, inputs: host.buildInputs(platform, backend.kind) }).catch((error: unknown) => { - if (builtForThisApp !== null && error instanceof VerifyFailure && error.code === 'BUILD_FAILED') return builtForThisApp; - throw error; - })); - const app: BuiltApp = { platform, key, appId: host.appId(platform), path: null, source: 'github-actions', sourceSha }; - return { app, view: { platform, key, source: app.source, reused: onSession !== null, seconds: 0 } }; + const source = + onSession === null + ? await backend.sourceCommit({ worktree: workspace.worktree, inputs: host.buildInputs(platform, backend.kind) }).catch((error: unknown) => { + if (builtForThisApp !== null && error instanceof VerifyFailure && error.code === 'BUILD_FAILED') return { sha: builtForThisApp, patch: null }; + throw error; + }) + : { sha: onSession, patch: null }; + const app: BuiltApp = { platform, key, appId: host.appId(platform), path: null, source: 'github-actions', sourceSha: source.sha, patch: source.patch }; + const onTheSession = heldIsAlive && held?.backend === 'remote' && held.installedBuild === key && held.builtSha === source.sha && held.builtPatch === (source.patch?.sha256 ?? null); + return { app, view: { platform, key, source: app.source, reused: onTheSession, seconds: 0 } }; } const existing = readBuiltApp(workspace, key); if (existing !== null) return { app: existing, view: { platform, key, source: existing.source, reused: true, seconds: 0 } }; @@ -178,12 +180,12 @@ export async function ensureLease( const state = held === null ? 'held' : await backend.check(held); const { app, view: build } = await ensureBuild(host, platform, workspace, backend, held, state === 'held', options.progress); - const builtBy = app.source === 'local' ? (build.reused ? 'reused' : `built in ${build.seconds}s`) : `commit ${app.sourceSha.slice(0, 12)} ${build.reused ? 'already on the session' : 'the session builds it'}`; + const builtBy = app.source === 'local' ? (build.reused ? 'reused' : `built in ${build.seconds}s`) : `commit ${app.sourceSha.slice(0, 12)}${app.patch === null ? '' : ` with uncommitted changes ${app.patch.sha256.slice(0, 12)}`} ${build.reused ? 'already on the session' : 'the session builds it'}`; options.progress(`build ${build.key} ${build.source} ${builtBy}`); let lease: Lease | null = held; let renewed = false; - if (lease !== null && state !== 'held') { + if (lease !== null && (state === 'lost' || state === 'expiring')) { options.progress(`${(state === 'expiring' ? 'ending' : 'lost').padEnd(7)} ${backend.describe(lease)} renewing`); await releaseLease(workspace, backend, lease); lease = null; @@ -210,7 +212,8 @@ export async function ensureLease( lease = acquired; } - if (lease.installedBuild !== app.key) { + const sessionHasOtherSource = app.source === 'github-actions' && lease.backend === 'remote' && (lease.builtSha !== app.sourceSha || lease.builtPatch !== (app.patch?.sha256 ?? null)); + if (lease.installedBuild !== app.key || sessionHasOtherSource || state === 'other-app') { const target = lease; options.progress(`install ${app.key} on ${backend.describe(target)}`); const wait = { diff --git a/integration/expo-native/src/core/evidence.ts b/integration/expo-native/src/core/evidence.ts index d67af7491ce..42d2d8cdcb4 100644 --- a/integration/expo-native/src/core/evidence.ts +++ b/integration/expo-native/src/core/evidence.ts @@ -69,6 +69,13 @@ const refuse = (message: string, fix: string): never => { }; export function assertShowable(record: EvidenceRecord, loggedUsers: readonly string[]): Showable { + const patch = record.remote?.builtPatch; + if (typeof patch === 'string') { + refuse( + `run ${record.run} ran on a session build of ${(record.remote?.builtSha ?? record.gitHead).slice(0, 12)} with uncommitted changes (${patch.slice(0, 12)}), so no commit holds the app it shows`, + 'commit and push the changes, {cli} run again, and attach the new run', + ); + } if (record.tainted.length > 0) refuse(`run ${record.run} has secret values in ${record.tainted.join(', ')}`, 'do not attach this run; rerun and attach the new run'); const own = new Set(record.identities.flatMap((i) => (i.userId === null ? [] : [i.userId]))); const foreign = loggedUsers.filter((userId) => !own.has(userId)); diff --git a/integration/expo-native/src/core/publish.ts b/integration/expo-native/src/core/publish.ts index 339d0413af3..8d4183ea55c 100644 --- a/integration/expo-native/src/core/publish.ts +++ b/integration/expo-native/src/core/publish.ts @@ -126,7 +126,8 @@ export function withEvidenceBlock(body: string, platform: Platform, lines: strin } if (starts.length !== 1 || ends.length !== 1) return { ok: false, why: `has \`${start}\` ${starts.length} times and \`${end}\` ${ends.length} times`, fix: ONE_PAIR_OF_MARKERS }; if (ends[0]! < starts[0]!) return { ok: false, why: `has \`${end}\` before \`${start}\``, fix: ONE_PAIR_OF_MARKERS }; - if (!A_BLOCK_STARTS.test(body.slice(starts[0]! + start.length, ends[0]!).trimStart())) return { ok: false, why: `has text between \`${start}\` and \`${end}\` that is not an evidence block`, fix: ONE_PAIR_OF_MARKERS }; + const between = body.slice(starts[0]! + start.length, ends[0]!).trimStart(); + if (between !== '' && !A_BLOCK_STARTS.test(between)) return { ok: false, why: `has text between \`${start}\` and \`${end}\` that is not an evidence block`, fix: ONE_PAIR_OF_MARKERS }; return { ok: true, body: `${body.slice(0, starts[0]!)}${block}${body.slice(ends[0]! + end.length)}`, was: 'replaced' }; } diff --git a/integration/expo-native/src/core/remote/backend.ts b/integration/expo-native/src/core/remote/backend.ts index 6edd24db87c..3e625467c82 100644 --- a/integration/expo-native/src/core/remote/backend.ts +++ b/integration/expo-native/src/core/remote/backend.ts @@ -1,16 +1,18 @@ -import { createWriteStream, existsSync } from 'node:fs'; -import { join } from 'node:path'; +import { execFileSync } from 'node:child_process'; +import { copyFileSync, createWriteStream, existsSync, mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve } from 'node:path'; import { Readable } from 'node:stream'; import { pipeline } from 'node:stream/promises'; import type { ReadableStream } from 'node:stream/web'; import { run, sleep } from '../exec.ts'; -import { VerifyFailure, everyFile, type AcquireRequest, type DeviceBackend, type EvidenceFile, type EvidencePath, type Recording, type RemoteLease } from '../types.ts'; +import { VerifyFailure, everyFile, type AcquireRequest, type DeviceBackend, type EvidenceFile, type EvidencePath, type Recording, type RemoteLease, type TreePatch } from '../types.ts'; import { apiMessage, currentBranch, endSession, liveRuns, openGitHub, startRun, viewRun, waitForStep, waitReporter, type GitHub } from './github.ts'; import { HANDOFF_VERSION, HandoffRefused, describeFile, parseHandoffManifest, pullRequestMismatch, type HandoffManifest } from './handoff.ts'; import { coreVersion } from '../manifest.ts'; import { withEvidenceBlock } from '../publish.ts'; import { remoteDoctorChecks } from './preflight.ts'; -import { BUILD_TIMEOUT_MINUTES, STEP, type SessionHealth } from './protocol.ts'; +import { BUILD_LOG_START, BUILD_TIMEOUT_MINUTES, PATCH_LIMIT_BYTES, STEP, sha256Hex, type SessionHealth } from './protocol.ts'; import { firstHealth, sendEvidence, sessionCall, sessionHealth, tunnelUrl, type SessionRef } from './session.ts'; import { forgetSession, newSessionRequest, saveToken, savedDriverId, type RemoteDeps, type RemoteSettings } from './settings.ts'; @@ -31,6 +33,33 @@ async function pullRequestCommits(hub: GitHub, pr: number): Promise { + try { + return execFileSync('git', [...args], { cwd: worktree, env: { ...process.env, GIT_INDEX_FILE: index }, maxBuffer: PATCH_LIMIT_BYTES, stdio: ['ignore', 'pipe', 'pipe'] }); + } catch (error) { + const failed = error as NodeJS.ErrnoException & { readonly stderr?: Buffer }; + if (failed.code === 'ENOBUFS') throw new VerifyFailure('BUILD_FAILED', `the uncommitted changes to the app make a patch larger than the ${PATCH_LIMIT_BYTES / 1024 / 1024} MiB a session takes`, COMMIT_THE_CHANGES); + throw new VerifyFailure('BUILD_FAILED', `git could not read the uncommitted changes to the app: ${failed.stderr?.toString().trim() || failed.message}`, COMMIT_THE_CHANGES); + } + }; + const own = resolve(worktree, execFileSync('git', ['rev-parse', '--git-path', 'index'], { cwd: worktree, encoding: 'utf8' }).trim()); + if (existsSync(own)) copyFileSync(own, index); + git(['add', '--all', '--', ...inputs]); + const bytes = git(['diff', '--cached', '--binary', '--no-color', '--no-ext-diff', '--no-renames', '--no-relative', '--src-prefix=a/', '--dst-prefix=b/', 'HEAD', '--', ...inputs]); + return bytes.length === 0 ? null : { sha256: sha256Hex(bytes), bytes }; + } finally { + rmSync(scratch, { recursive: true, force: true }); + } +} + +const describeSource = (sha: string, patch: string | null): string => `${sha.slice(0, 12)}${patch === null ? '' : ` with uncommitted changes ${patch.slice(0, 12)}`}`; + async function baseHolds(hub: GitHub, pr: number, base: unknown, commit: string): Promise { if (typeof base !== 'string' || !/^[0-9a-f]{40}$/.test(base)) return false; const compared = await hub.api('GET', `/compare/${commit}...${base}?per_page=1`); @@ -59,29 +88,43 @@ export function remoteBackend(settings: RemoteSettings, deps: RemoteDeps = { env function describeBuild(health: SessionHealth): string { const build = health.build; if (build.state === 'none') return 'no build'; - return `build ${build.sha.slice(0, 12)} ${build.state}${build.state === 'building' ? ` ${build.seconds}s` : ''}`; + return `build ${describeSource(build.sha, build.patch)} ${build.state}${build.state === 'building' ? ` ${build.seconds}s` : ''}`; } - async function waitForBuild(lease: RemoteLease, sha: string, progress: (line: string) => void): Promise> { + async function waitForBuild(lease: RemoteLease, sha: string, patch: string | null, progress: (line: string) => void): Promise> { const deadline = Date.now() + READY_DEADLINE_MS; + const what = describeSource(sha, patch); + const fix = patch === null ? 'fix the build error above, commit, push, then rerun {cli} up' : 'fix the build error above, then rerun {cli} up, which sends the uncommitted changes again'; let last = ''; + let shown = 0; + const showLog = async (): Promise => { + const answer = await sessionCall(lease, `/__sim/build/log?from=${shown}`, { timeoutMs: 15_000 }).catch(() => null); + if (answer?.status !== 200) return; + const start = Number(answer.headers.get(BUILD_LOG_START) ?? shown); + const bytes = Buffer.from(await answer.arrayBuffer()); + const whole = bytes.lastIndexOf(0x0a) + 1; + for (const line of bytes.subarray(0, whole).toString().split('\n')) if (line.trim() !== '') progress(`build | ${line}`); + shown = start + whole; + }; for (;;) { const health = await sessionHealth(lease); if (health === null) { const state = await viewRun(await github(), lease.providerRef); if (state.status === 'completed') throw new VerifyFailure('LEASE_LOST', `the session ended (${state.conclusion ?? 'no conclusion'}) while it was building`, `read ${runUrl(lease.providerRef)} for why, then {cli} up`); } else { - if (health.build.state === 'failed' && health.build.sha === sha) { - throw new VerifyFailure('BUILD_FAILED', `the session could not build ${sha.slice(0, 12)}:\n${health.build.tail}`, 'fix the build error above, commit, push, then rerun {cli} up'); + const ours = health.build.state !== 'none' && health.build.sha === sha && health.build.patch === patch; + if (ours) await showLog(); + if (ours && health.build.state === 'failed') { + throw new VerifyFailure('BUILD_FAILED', `the session could not build ${what}:\n${health.build.tail}`, fix); } - if (health.build.state === 'timed-out' && health.build.sha === sha) { + if (ours && health.build.state === 'timed-out') { throw new VerifyFailure( 'NOT_READY', - `the session stopped the build of ${sha.slice(0, 12)} after ${BUILD_TIMEOUT_MINUTES} minutes on ${lease.runner}, the longest it lets one build run. The build reported no error before that. Its last lines:\n${health.build.tail}`, + `the session stopped the build of ${what} after ${BUILD_TIMEOUT_MINUTES} minutes on ${lease.runner}, the longest it lets one build run. The build reported no error before that. Its last lines:\n${health.build.tail}`, 'rerun {cli} up, and the same session builds again from what the stopped build left; if that build is stopped too, {cli} down, then {cli} up --runner ', ); } - if (health.build.state === 'built' && health.build.sha === sha && health.device?.ready === true && health.daemon) return health.build; + if (ours && health.build.state === 'built' && health.device?.ready === true && health.daemon) return health.build; const line = `wait ${describeBuild(health)}; device ${health.device?.ready === true ? 'ready' : 'booting'}; agent-device ${health.daemon ? 'up' : 'starting'}`; const coarse = line.replace(/ \d+s/, ''); if (coarse !== last) progress(line); @@ -102,21 +145,13 @@ export function remoteBackend(settings: RemoteSettings, deps: RemoteDeps = { env availability: () => ({ usable: true, why: `the device runs on a CI runner (${settings.runner} unless --runner names another), started through ${settings.workflow} on ${settings.repo}` }), async sourceCommit(input) { - const status = await deps.runner('git', ['status', '--porcelain', '--', ...input.inputs], { cwd: input.worktree }); - const files = status.stdout.split('\n').filter((line) => line.length > 3).map((line) => line.slice(3)); - if (files.length > 0) { - throw new VerifyFailure( - 'BUILD_FAILED', - `a remote session builds a pushed commit, and this tree has uncommitted changes to the app: ${files.slice(0, 5).join(', ')}${files.length > 5 ? `, and ${files.length - 5} more` : ''}`, - 'git commit the changes and git push, then rerun', - ); - } const sha = (await deps.runner('git', ['rev-parse', 'HEAD'], { cwd: input.worktree })).stdout.trim(); + const patch = workingPatch(input.worktree, input.inputs); const found = await (await github()).api('GET', `/commits/${sha}`); if (found.status !== 200) { throw new VerifyFailure('BUILD_FAILED', `a remote session builds a pushed commit, and GitHub does not have ${sha.slice(0, 12)} (${found.status})`, 'git push, then rerun'); } - return sha; + return { sha, patch }; }, async acquire(request: AcquireRequest) { @@ -149,6 +184,7 @@ export function remoteBackend(settings: RemoteSettings, deps: RemoteDeps = { env runner: order.runner, expiresAt: health.capAt, builtSha: null, + builtPatch: null, acquiredAt: new Date().toISOString(), installedBuild: null, }; @@ -174,17 +210,23 @@ export function remoteBackend(settings: RemoteSettings, deps: RemoteDeps = { env } if (health.ending !== null) return 'lost'; assertSameCore(health); - return Date.parse(lease.expiresAt) - Date.now() < EXPIRING_MS ? 'expiring' : 'held'; + if (Date.parse(lease.expiresAt) - Date.now() < EXPIRING_MS) return 'expiring'; + const { build } = health; + return lease.builtSha === null || (build.state === 'built' && build.sha === lease.builtSha && build.patch === lease.builtPatch) ? 'held' : 'other-app'; }, async install(lease, app, progress) { if (app.source === 'local') throw new VerifyFailure('BUILD_FAILED', 'a remote session builds a pushed commit and this build names none', '{cli} down, then {cli} up'); - const sha = app.sourceSha; - const started = await sessionCall(lease, `/__sim/build?sha=${sha}`, { method: 'POST' }); - if (started.status !== 202) throw new VerifyFailure('BUILD_FAILED', `the session refused to build ${sha.slice(0, 12)}: ${started.status} ${await started.text()}`, '{cli} down, then {cli} up'); - const built = await waitForBuild(lease, sha, progress); - progress(`build ${app.key} github-actions ${sha.slice(0, 12)} ${built.incremental ? 'rebuilt' : 'built'} in ${built.seconds}s on ${lease.runner}`); - return { ...lease, builtSha: sha }; + const { sourceSha: sha, patch } = app; + const builtPatch = patch?.sha256 ?? null; + const started = await sessionCall(lease, `/__sim/build?sha=${sha}${patch === null ? '' : `&patch=${patch.sha256}`}`, { + method: 'POST', + ...(patch === null ? {} : { json: { patch: patch.bytes.toString('base64') }, timeoutMs: 120_000 }), + }); + if (started.status !== 202) throw new VerifyFailure('BUILD_FAILED', `the session refused to build ${describeSource(sha, builtPatch)}: ${started.status} ${await started.text()}`, '{cli} down, then {cli} up'); + const built = await waitForBuild(lease, sha, builtPatch, progress); + progress(`build ${app.key} github-actions ${describeSource(sha, builtPatch)} ${built.incremental ? 'rebuilt' : 'built'} in ${built.seconds}s on ${lease.runner}`); + return { ...lease, builtSha: sha, builtPatch }; }, async release(lease) { @@ -212,6 +254,7 @@ export function remoteBackend(settings: RemoteSettings, deps: RemoteDeps = { env runner: '', expiresAt: live.createdAt, builtSha: null, + builtPatch: null, acquiredAt: live.createdAt, installedBuild: null, })); diff --git a/integration/expo-native/src/core/remote/protocol.ts b/integration/expo-native/src/core/remote/protocol.ts index 7643081f8e3..8dc8ad5f5b2 100644 --- a/integration/expo-native/src/core/remote/protocol.ts +++ b/integration/expo-native/src/core/remote/protocol.ts @@ -71,7 +71,7 @@ export function parseRequest(text: string): SessionRequest { return request; } -export const sha256Hex = (value: string): string => createHash('sha256').update(value).digest('hex'); +export const sha256Hex = (value: string | Buffer): string => createHash('sha256').update(value).digest('hex'); export function matchesToken(tokenSha256: string, presented: string): boolean { const given = Buffer.from(sha256Hex(presented), 'hex'); @@ -85,15 +85,25 @@ export const STEP = { export const RUN_TITLE = /(?:^|[ /])([a-f0-9]{12})\/([a-z0-9]{4,24})$/; +interface BuildOf { + readonly sha: string; + readonly patch: string | null; + readonly seconds: number; +} + export type BuildState = | { readonly state: 'none' } - | { readonly state: 'building'; readonly sha: string; readonly seconds: number } - | { readonly state: 'built'; readonly sha: string; readonly seconds: number; readonly incremental: boolean } - | { readonly state: 'failed'; readonly sha: string; readonly seconds: number; readonly tail: string } - | { readonly state: 'timed-out'; readonly sha: string; readonly seconds: number; readonly tail: string }; + | (BuildOf & { readonly state: 'building' }) + | (BuildOf & { readonly state: 'built'; readonly incremental: boolean }) + | (BuildOf & { readonly state: 'failed'; readonly tail: string }) + | (BuildOf & { readonly state: 'timed-out'; readonly tail: string }); export const BUILD_TIMEOUT_MINUTES = 45; +export const PATCH_LIMIT_BYTES = 16 * 1024 * 1024; + +export const BUILD_LOG_START = 'x-log-start'; + export type EndReason = 'stop' | 'idle' | 'cap' | 'tunnel-lost' | 'signal'; export interface SessionHealth { @@ -106,6 +116,7 @@ export interface SessionHealth { } export interface SessionDevice { + readonly inputs: readonly string[]; build(work: string): readonly CommandLine[]; readonly record: { start(file: string): CommandLine; @@ -121,11 +132,13 @@ export interface SessionDeviceRef { export type SessionDeviceFactory = (device: SessionDeviceRef) => SessionDevice; export type RecipeRequest = + | { readonly op: 'inputs' } | { readonly op: 'build'; readonly work: string } | { readonly op: 'record'; readonly file: string } | { readonly op: 'logs'; readonly since: string; readonly predicate: string | null }; export interface RecipeAnswers { + readonly inputs: readonly string[]; readonly build: readonly CommandLine[]; readonly record: { readonly start: CommandLine; readonly stop: readonly CommandLine[] | null; readonly collect: readonly CommandLine[] }; readonly logs: CommandLine; diff --git a/integration/expo-native/src/core/remote/recipe.ts b/integration/expo-native/src/core/remote/recipe.ts index ebe510c1c6f..30c33608082 100644 --- a/integration/expo-native/src/core/remote/recipe.ts +++ b/integration/expo-native/src/core/remote/recipe.ts @@ -4,6 +4,8 @@ import type { RecipeAnswers, RecipeRequest, SessionDevice, SessionDeviceFactory, export function answer(device: SessionDevice, request: RecipeRequest): RecipeAnswers[RecipeRequest['op']] { switch (request.op) { + case 'inputs': + return device.inputs; case 'build': return device.build(request.work); case 'record': diff --git a/integration/expo-native/src/core/remote/session-agent.ts b/integration/expo-native/src/core/remote/session-agent.ts index 0132db7c4b9..61893bf3b5b 100644 --- a/integration/expo-native/src/core/remote/session-agent.ts +++ b/integration/expo-native/src/core/remote/session-agent.ts @@ -1,14 +1,18 @@ -import { spawn, type ChildProcess } from 'node:child_process'; +import { execFile, spawn, type ChildProcess } from 'node:child_process'; import { Resolver } from 'node:dns/promises'; -import { appendFileSync, createReadStream, existsSync, mkdirSync, rmSync, statSync, writeFileSync } from 'node:fs'; +import { appendFileSync, createReadStream, existsSync, mkdirSync, rmSync, statSync, writeFileSync, type ReadStream } from 'node:fs'; import http, { type IncomingMessage, type ServerResponse } from 'node:http'; import { join, resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; +import { promisify } from 'node:util'; import { + BUILD_LOG_START, BUILD_TIMEOUT_MINUTES, + PATCH_LIMIT_BYTES, RequestError, matchesToken, parseRequest, + sha256Hex, type BuildState, type EndReason, type RecipeAnswers, @@ -28,6 +32,12 @@ const COMMAND_TIMEOUT_MS = 30_000; const PUBLIC_RESOLVERS = ['1.1.1.1', '8.8.8.8']; const RECIPE = fileURLToPath(new URL('./recipe.ts', import.meta.url)); const PINNED_AGENT_DEVICE = fileURLToPath(new URL('../../../node_modules/.bin/agent-device', import.meta.url)); +const execFileAsync = promisify(execFile); + +function send(stream: ReadStream, res: ServerResponse): void { + stream.on('error', () => res.destroy()); + stream.pipe(res); +} function setOutputs(file: string | undefined, values: Readonly>): void { const text = Object.entries(values).map(([key, value]) => `${key}=${value}\n`).join(''); @@ -39,6 +49,7 @@ type Env = Readonly>; type GitHubGet = (path: string) => Promise; const COMMIT = /^[0-9a-f]{40}$/; +const PATCH_HASH = /^[0-9a-f]{64}$/; function githubOf(env: Env): GitHubGet { const base = `${(env.GITHUB_API_URL ?? 'https://api.github.com').replace(/\/$/, '')}/repos/${env.GITHUB_REPOSITORY ?? ''}`; @@ -175,11 +186,19 @@ function readBody(req: IncomingMessage, limit: number): Promise { }); } +async function changesRefusal(ref: string, named: string, get: GitHubGet): Promise { + if (!ref.startsWith('refs/heads/')) return `this session ${ref === '' ? 'does not know the branch it was started on' : `was started on ${ref}, which is not a branch`}, so it builds no uncommitted changes. Commit and push them to a branch, then run again.`; + const defaultBranch = named !== '' ? named : ((await get('')) as { readonly default_branch?: unknown } | null)?.default_branch; + if (typeof defaultBranch !== 'string' || defaultBranch === '') return 'this session could not learn the default branch of the repository, so it builds no uncommitted changes. Commit and push them to a branch, then run again.'; + return ref === `refs/heads/${defaultBranch}` ? `this session runs on ${defaultBranch}, the default branch, so it builds no uncommitted changes. Work on a branch: commit to a new branch and push it, then run down and up from that branch.` : null; +} + async function serve(env: NodeJS.ProcessEnv): Promise { const request = parseRequest(env.VERIFY_SESSION_REQUEST ?? ''); const dispatched = env.GITHUB_SHA ?? ''; const dispatchedRef = env.GITHUB_REF ?? ''; const github = takeGitHub(env); + const refusesChanges = changesRefusal(dispatchedRef, env.VERIFY_SESSION_DEFAULT_BRANCH ?? '', github); const work = resolve(env.VERIFY_SESSION_WORK ?? 'verify-remote-work'); const port = Number(env.VERIFY_SESSION_PORT ?? 3199); const agentPort = Number(env.VERIFY_SESSION_AGENT_PORT ?? 4310); @@ -211,6 +230,7 @@ async function serve(env: NodeJS.ProcessEnv): Promise { let builtOnce = false; const logFile = (name: 'agent-device-proxy' | 'build' | 'tunnel') => join(work, `${name}.log`); + const patchFile = (patch: string) => join(work, `patch-${patch}.diff`); const scrub = (text: string): string => (token === null ? text : text.split(token).join('')); const log = (name: Parameters[0], chunk: Buffer | string): void => appendFileSync(logFile(name), scrub(chunk.toString())); @@ -280,9 +300,21 @@ async function serve(env: NodeJS.ProcessEnv): Promise { }, Promise.resolve(0)); } - function startBuild(sha: string): void { + async function pathsThePatchWrites(sha: string, patch: string): Promise { + const env = { ...process.env, GIT_INDEX_FILE: join(work, 'patch-index') }; + try { + await execFileAsync('git', ['read-tree', sha], { env }); + await execFileAsync('git', ['apply', '--cached', '--binary', '--whitespace=nowarn', patchFile(patch)], { env }); + const { stdout } = await execFileAsync('git', ['diff', '--cached', '--no-renames', '--name-only', '-z', sha], { env, maxBuffer: 2 * PATCH_LIMIT_BYTES }); + return stdout.split('\0').filter((path) => path !== ''); + } catch { + return null; + } + } + + function startBuild(sha: string, patch: string | null): void { if (device === null) return; - if ((build.state === 'building' || build.state === 'built') && build.sha === sha) return; + if ((build.state === 'building' || build.state === 'built') && build.sha === sha && build.patch === patch) return; buildGeneration += 1; stopGroup(buildChild); const generation = buildGeneration; @@ -297,21 +329,41 @@ async function serve(env: NodeJS.ProcessEnv): Promise { timedOut = true; stopGroup(buildChild); }, BUILD_TIMEOUT_MINUTES * minuteMs); - build = { state: 'building', sha, seconds: 0 }; + build = { state: 'building', sha, patch, seconds: 0 }; rmSync(logFile('build'), { force: true }); - const checkout: CommandLine[] = [ + const cleanCheckout: CommandLine[] = [ { command: 'git', args: ['fetch', '--no-tags', '--depth=1', 'origin', sha] }, { command: 'git', args: ['checkout', '--force', '--detach', sha] }, + { command: 'git', args: ['clean', '-d', '--force'] }, ]; const target = device; + const doesNotApply = (changes: string): string => `the uncommitted changes ${changes.slice(0, 12)} do not apply to ${sha.slice(0, 12)}. Commit and push them, then run again.`; const checkoutThenBuild = async (): Promise => { if (sha !== dispatched && !(await isALaterCommitOfTheBranch(sha, dispatched, dispatchedRef, github))) { tail.push(`this session builds ${dispatched}, the commit it was started on, and later commits of ${dispatchedRef}. GitHub does not show ${sha} as one. Push it to that branch, or run down and then up to start a session on it.`); return 1; } - const checkedOut = await runCommands(checkout, stopped, tail); + const checkedOut = await runCommands(cleanCheckout, stopped, tail); if (checkedOut !== 0) return checkedOut; try { + if (patch !== null) { + const inputs = await recipe(target, { op: 'inputs' }); + const written = await pathsThePatchWrites(sha, patch); + if (written === null) { + tail.push(doesNotApply(patch)); + return 1; + } + const outside = written.find((path) => !inputs.some((input) => path === input || path.startsWith(`${input}/`))); + if (outside !== undefined) { + tail.push(`the uncommitted changes ${patch.slice(0, 12)} change ${outside}, and a session builds changes only to ${inputs.join(', ')}. Commit and push that change, then run again.`); + return 1; + } + const applied = await runCommands([{ command: 'git', args: ['apply', '--binary', '--whitespace=nowarn', patchFile(patch)] }], stopped, tail); + if (applied !== 0) { + tail.push(doesNotApply(patch)); + return applied; + } + } return await runCommands(await recipe(target, { op: 'build', work }), stopped, tail); } catch (error) { tail.push((error as Error).message); @@ -324,9 +376,9 @@ async function serve(env: NodeJS.ProcessEnv): Promise { const seconds = Math.round((Date.now() - began) / 1000); if (code === 0 && !timedOut) { builtOnce = true; - build = { state: 'built', sha, seconds, incremental }; + build = { state: 'built', sha, patch, seconds, incremental }; } else { - build = { state: timedOut ? 'timed-out' : 'failed', sha, seconds, tail: scrub(tail.join('\n')) }; + build = { state: timedOut ? 'timed-out' : 'failed', sha, patch, seconds, tail: scrub(tail.join('\n')) }; } }); } @@ -381,11 +433,41 @@ async function serve(env: NodeJS.ProcessEnv): Promise { return json(res, 200, health()); case 'POST /build': { const sha = url.searchParams.get('sha') ?? ''; + const patch = url.searchParams.get('patch'); if (!/^[0-9a-f]{40}$/.test(sha)) return json(res, 400, { error: 'sha must be a full commit id' }); + if (patch !== null && !PATCH_HASH.test(patch)) return json(res, 400, { error: 'patch must be the sha256 of the patch' }); if (device === null) return json(res, 409, { error: 'this session has no device to build for' }); - startBuild(sha); + if (patch !== null) { + const refusal = await refusesChanges; + if (refusal !== null) return json(res, 409, { error: refusal }); + const body = await readBody(req, Math.ceil(PATCH_LIMIT_BYTES / 3) * 4 + 64); + if (body === null) return json(res, 413, { error: `the patch is larger than ${PATCH_LIMIT_BYTES} bytes` }); + let bytes: Buffer; + try { + bytes = Buffer.from(String((JSON.parse(body) as { readonly patch?: unknown }).patch), 'base64'); + } catch { + return json(res, 400, { error: 'the body must be a JSON object with the patch in base64' }); + } + if (sha256Hex(bytes) !== patch) return json(res, 400, { error: 'the patch that arrived does not have the sha256 it was sent with' }); + appendFileSync(join(work, 'patched'), `${patch}\n`); + writeFileSync(patchFile(patch), bytes); + } + startBuild(sha, patch); return json(res, 202, health()); } + case 'GET /build/log': { + const file = logFile('build'); + const size = existsSync(file) ? statSync(file).size : 0; + const from = Number(url.searchParams.get('from') ?? 0); + const start = Number.isInteger(from) && from >= 0 && from <= size ? from : 0; + res.writeHead(200, { 'content-type': 'text/plain; charset=utf-8', [BUILD_LOG_START]: String(start) }); + if (start === size) { + res.end(); + return; + } + send(createReadStream(file, { start, end: size - 1 }), res); + return; + } case 'POST /record/start': { if (device === null) return json(res, 409, { error: 'this session has no device to record' }); await stopRecording(); @@ -403,7 +485,7 @@ async function serve(env: NodeJS.ProcessEnv): Promise { const file = join(work, 'recording.mp4'); if (!existsSync(file)) return json(res, 404, { error: 'no recording' }); res.writeHead(200, { 'content-type': 'video/mp4', 'content-length': String(statSync(file).size) }); - createReadStream(file).pipe(res); + send(createReadStream(file), res); return; } case 'GET /logs': { @@ -475,7 +557,7 @@ async function serve(env: NodeJS.ProcessEnv): Promise { server.listen(port, '127.0.0.1', () => { console.log(`verify-remote session ${request.session} on :${port}, idle stop ${request.idleMinutes} min, cap ${request.capMinutes} min`); - if (request.sha !== null) startBuild(request.sha); + if (request.sha !== null) startBuild(request.sha, null); }); } diff --git a/integration/expo-native/src/core/types.ts b/integration/expo-native/src/core/types.ts index d51f6822b23..a2b0f83c155 100644 --- a/integration/expo-native/src/core/types.ts +++ b/integration/expo-native/src/core/types.ts @@ -259,10 +259,19 @@ export interface LocalBuild extends BuiltAppBase { readonly source: 'local'; readonly path: ScratchPath; } +export interface TreePatch { + readonly sha256: string; + readonly bytes: Buffer; +} +export interface SessionSource { + readonly sha: string; + readonly patch: TreePatch | null; +} export interface SessionBuild extends BuiltAppBase { readonly source: 'github-actions'; readonly path: null; readonly sourceSha: string; + readonly patch: TreePatch | null; } export type BuiltApp = LocalBuild | SessionBuild; export type BuildSource = BuiltApp['source']; @@ -293,6 +302,7 @@ export interface RemoteLease extends LeaseBase { readonly runner: string; readonly expiresAt: string; readonly builtSha: string | null; + readonly builtPatch: string | null; } export type Lease = LocalLease | RemoteLease; @@ -330,7 +340,7 @@ export interface EvidenceRecord { readonly dirty: boolean; readonly platform: Platform; readonly backend: BackendKind; - readonly remote: { readonly provider: RemoteProvider; readonly runner: string; readonly builtSha: string | null } | null; + readonly remote: { readonly provider: RemoteProvider; readonly runner: string; readonly builtSha: string | null; readonly builtPatch?: string | null } | null; readonly device: string; readonly build: BuildKey; readonly results: readonly SpecResult[]; @@ -412,9 +422,9 @@ export interface DeviceBackend { readonly kind: BackendKind; readonly platform: Platform; availability(): Availability; - sourceCommit?(input: { readonly worktree: string; readonly inputs: readonly string[] }): Promise; + sourceCommit?(input: { readonly worktree: string; readonly inputs: readonly string[] }): Promise; acquire(request: AcquireRequest): Promise; - check(lease: L): Promise<'held' | 'lost' | 'expiring'>; + check(lease: L): Promise<'held' | 'lost' | 'expiring' | 'other-app'>; install(lease: L, app: BuiltApp, progress: (line: string) => void): Promise; release(lease: L): Promise; reapable(owner?: string): Promise; diff --git a/integration/expo-native/src/core/verbs.ts b/integration/expo-native/src/core/verbs.ts index 5d690fe8853..3ee6664b35a 100644 --- a/integration/expo-native/src/core/verbs.ts +++ b/integration/expo-native/src/core/verbs.ts @@ -102,6 +102,24 @@ function readJson(file: string): Record | null { return existsSync(file) ? (JSON.parse(readFileSync(file, 'utf8')) as Record) : null; } +const GRAPHQL_PROBE_MS = 10_000; + +async function ghAttachCheck(runner: Runner): Promise { + const cannotPost = "so `{cli} attach` cannot put a run's video and screenshots in a pull request description from this machine"; + const noAttach = await missingAttach(runner); + if (noAttach !== null) return { id: 'gh-attach', ok: true, state: 'warning', detail: `${noAttach.why}, ${cannotPost}`, fix: noAttach.fix }; + const graphql = await runner('gh', ['api', 'graphql', '-f', 'query={viewer{login}}'], { timeoutMs: GRAPHQL_PROBE_MS }); + if (graphql.code === 0) return check('gh-attach', true, 'gh pr edit supports --attach, and gh can use the GitHub GraphQL API', ''); + const why = graphql.code === 124 ? `no answer in ${GRAPHQL_PROBE_MS / 1000} seconds` : redact(graphql.stderr.trim().split('\n')[0] ?? ''); + return { + id: 'gh-attach', + ok: true, + state: 'warning', + detail: `gh cannot use the GitHub GraphQL API (${why}), which \`gh pr view\` and \`gh pr edit\` need, ${cannotPost}`, + fix: "attach a remote run before {cli} down, so that attach hands its evidence to the session's runner, or check `gh auth status`", + }; +} + export async function doctor(deps: Deps, command: Extract): Promise { const { host, workspace, runner } = deps; const platform = platformOf(host, command.platform); @@ -145,15 +163,10 @@ export async function doctor(deps: Deps, command: Extract c.deviceName).join(', ')} belong to deleted worktrees`, '{cli} down --stale')); @@ -184,7 +197,8 @@ const CLOSE_BEFORE_RECORDING_MS = 30_000; function agentDeviceFor(deps: Deps, lease: Lease, platform: Platform) { const target = agentDeviceOn(deviceOf(lease), platform); const env = { ...withoutClerkKeys(deps.env), AGENT_DEVICE_STATE_DIR: deps.workspace.agentDeviceDir, ...target.env }; - return { selector: target.selector, call: (args: readonly string[], limit: { readonly timeoutMs?: number } = {}) => deps.runner(join(deps.workspace.packageDir, 'node_modules', '.bin', 'agent-device'), args, { env, ...limit }) }; + mkdirSync(deps.workspace.agentDeviceDir, { recursive: true }); + return { selector: target.selector, call: (args: readonly string[], limit: { readonly timeoutMs?: number } = {}) => deps.runner(join(deps.workspace.packageDir, 'node_modules', '.bin', 'agent-device'), args, { env, cwd: deps.workspace.agentDeviceDir, ...limit }) }; } const targetOf = (deps: Deps, outcome: RuntimeOutcome): Target => ({ @@ -502,7 +516,7 @@ export async function runVerb(deps: Deps, command: Extract typeof r[name] !== 'string')) throw bad(); - return { ...(r as object), installedBuild, builtSha: typeof r.builtSha === 'string' ? r.builtSha : null } as Lease; + return { ...(r as object), installedBuild, builtSha: typeof r.builtSha === 'string' ? r.builtSha : null, builtPatch: typeof r.builtPatch === 'string' ? r.builtPatch : null } as Lease; } throw bad(); } diff --git a/integration/expo-native/src/fixture.ts b/integration/expo-native/src/fixture.ts index 1ac603df049..da1d18fb668 100644 --- a/integration/expo-native/src/fixture.ts +++ b/integration/expo-native/src/fixture.ts @@ -9,6 +9,7 @@ import { VerifyFailure, type Platform } from './core/types.ts'; import { embedJavaScript, expoFingerprint, + hashInputFiles, installedNativeInputs, keepNativeBuild, keptNativeBuild, @@ -378,10 +379,10 @@ export async function buildFixture(build: FixtureBuild, site: FixtureSite = THIS return built; } -async function committedNativeKey(platform: Platform): Promise { - const listed = await run('git', ['ls-tree', '-r', 'HEAD', '--', ...nativeInputs(platform)], { cwd: WORKTREE }); - if (listed.code !== 0) throw new VerifyFailure('NOT_READY', `git ls-tree failed: ${listed.stderr.trim()}`, 'retry'); - return createHash('sha256').update(listed.stdout).digest('hex').slice(0, 12); +export async function nativeProjectKey(platform: Platform, worktree: string = WORKTREE): Promise { + const hash = createHash('sha256'); + await hashInputFiles(worktree, nativeInputs(platform), hash); + return hash.digest('hex').slice(0, 12); } export async function buildStandalone( @@ -390,6 +391,7 @@ export async function buildStandalone( say: (line: string) => void, site: FixtureSite = THIS_CHECKOUT, now: () => number = Date.now, + nativeCache: string | null = nativeCacheDir(), ): Promise { const must: typeof mustStep = async (what, command, args, cwd, env) => { const began = now(); @@ -402,7 +404,7 @@ export async function buildStandalone( say('build pnpm install --frozen-lockfile'); await must('pnpm install', 'pnpm', ['install', '--frozen-lockfile'], site.worktree); const built = await buildFixture( - { platform, product: 'standalone', nativeKey, buildPackages: true, progress: say }, + { platform, product: 'standalone', nativeKey, buildPackages: true, nativeCache, progress: say }, { ...site, must }, ); say(`build ${built}`); @@ -427,7 +429,7 @@ if (import.meta.main) { } try { if (first === 'native') await printKeptNativeBuild(platform); - else await buildStandalone(platform, await committedNativeKey(platform), line => console.log(line)); + else await buildStandalone(platform, await nativeProjectKey(platform), line => console.log(line)); } catch (error) { console.error((error as Error).message); process.exit(1); diff --git a/integration/expo-native/src/native-build.ts b/integration/expo-native/src/native-build.ts index 57707874f5e..72c4da06cb5 100644 --- a/integration/expo-native/src/native-build.ts +++ b/integration/expo-native/src/native-build.ts @@ -1,5 +1,5 @@ import { execFileSync } from 'node:child_process'; -import { createHash } from 'node:crypto'; +import { createHash, type Hash } from 'node:crypto'; import { chmodSync, closeSync, @@ -38,28 +38,28 @@ export interface FingerprintInputs { readonly installed: string; } -export async function nativeFingerprint(spec: FingerprintInputs): Promise { - const listed = await run( - 'git', - ['ls-files', '-z', '--cached', '--others', '--exclude-standard', '--', ...spec.inputs], - { - cwd: spec.worktree, - }, - ); +export async function hashInputFiles(worktree: string, inputs: readonly string[], hash: Hash): Promise { + const listed = await run('git', ['ls-files', '-z', '--cached', '--others', '--exclude-standard', '--', ...inputs], { + cwd: worktree, + }); if (listed.code !== 0) throw new VerifyFailure( 'NOT_READY', `git ls-files failed: ${listed.stderr.trim()}`, 'run {cli} from inside a git worktree', ); - const hash = createHash('sha256'); - hash.update(spec.platform).update('\0').update(spec.toolchain).update('\0').update(spec.installed).update('\0'); const tracked = [...new Set(listed.stdout.split('\0').filter(file => file.length > 0))].sort(); for (const file of tracked) { - const path = join(spec.worktree, file); + const path = join(worktree, file); hash.update(file).update('\0'); hash.update(existsSync(path) ? readFileSync(path) : 'deleted').update('\0'); } +} + +export async function nativeFingerprint(spec: FingerprintInputs): Promise { + const hash = createHash('sha256'); + hash.update(spec.platform).update('\0').update(spec.toolchain).update('\0').update(spec.installed).update('\0'); + await hashInputFiles(spec.worktree, spec.inputs, hash); for (const file of spec.files) { hash.update(basename(file)).update('\0'); hash.update(existsSync(file) ? readFileSync(file) : 'missing').update('\0'); diff --git a/integration/expo-native/src/platform/session-device.ts b/integration/expo-native/src/platform/session-device.ts index 129e98b839d..b98d096e98d 100644 --- a/integration/expo-native/src/platform/session-device.ts +++ b/integration/expo-native/src/platform/session-device.ts @@ -2,7 +2,7 @@ import { fileURLToPath } from 'node:url'; import type { CommandLine } from '../core/exec.ts'; import type { SessionDevice, SessionDeviceFactory } from '../core/remote/protocol.ts'; import type { Platform } from '../core/types.ts'; -import { artifact } from '../fixture.ts'; +import { artifact, buildInputs } from '../fixture.ts'; import { collectRecordingArgs, installArgs, @@ -24,6 +24,7 @@ const buildStandalone = (platform: Platform): CommandLine => ({ function simulator(udid: string): SessionDevice { return { + inputs: buildInputs('ios', 'standalone'), build: () => [buildStandalone('ios'), waitForBoot(udid), installApp(udid, artifact('ios', 'standalone'))], record: { start: file => recordVideo(udid, file) }, logs: (since, predicate) => showLogs(udid, since, predicate), @@ -33,6 +34,7 @@ function simulator(udid: string): SessionDevice { function emulator(serial: string): SessionDevice { const adb = (args: readonly string[]): CommandLine => ({ command: 'adb', args: ['-s', serial, ...args] }); return { + inputs: buildInputs('android', 'standalone'), build: work => [buildStandalone('android'), waitForLane(work), adb(installArgs(artifact('android', 'standalone')))], record: { start: () => adb(screenrecordArgs(RECORDING)), diff --git a/integration/expo-native/test/borrowed-device.test.ts b/integration/expo-native/test/borrowed-device.test.ts new file mode 100644 index 00000000000..0378d278c64 --- /dev/null +++ b/integration/expo-native/test/borrowed-device.test.ts @@ -0,0 +1,69 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { mkdtempSync, readdirSync, writeFileSync } from 'node:fs'; +import http from 'node:http'; +import type { AddressInfo } from 'node:net'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it, type TestContext } from 'node:test'; +import type { DeviceProvider, DeviceRequest } from '@e2e-dev/mobile'; +import { createAgentDeviceClient } from 'agent-device'; +import { engineDevice } from '../specs/support/config.ts'; + +const UDID = 'FDF0CD9E-CF9E-42B6-AE3A-116A665F7EF3'; + +async function daemonWithAFallbackScreenshot(): Promise<{ readonly url: string; readonly close: () => void }> { + const server = http.createServer((req, res) => { + let body = ''; + req.on('data', (chunk: Buffer) => (body += chunk.toString())); + req.on('end', () => { + if (req.url?.startsWith('/agent-device/artifacts/') === true) { + res.writeHead(200, { 'content-type': 'image/png' }); + res.end('png'); + return; + } + const id = req.url === '/agent-device/rpc' ? (JSON.parse(body) as { id: string }).id : null; + res.writeHead(200, { 'content-type': 'application/json' }); + res.end(JSON.stringify({ jsonrpc: '2.0', id, result: { ok: true, data: { nodes: [], artifacts: [{ field: 'fallbackScreenshotPath', artifactType: 'screenshot', artifactId: 'fallback', fileName: 'snapshot-fallback.png' }] } } })); + }); + }); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + return { url: `http://127.0.0.1:${(server.address() as AddressInfo).port}/agent-device`, close: () => server.close() }; +} + +describe('a borrowed device', () => { + async function snapshotFrom(t: TestContext, env: Readonly>, dir: string) { + const daemon = await daemonWithAFallbackScreenshot(); + t.after(daemon.close); + const checkout = mkdtempSync(join(dir, 'checkout-')); + const tokenFile = join(dir, 'token'); + writeFileSync(tokenFile, 'b'.repeat(64)); + const before = process.cwd(); + process.chdir(checkout); + t.after(() => process.chdir(before)); + + const provider = engineDevice({ kind: 'borrowed', id: UDID, daemonUrl: daemon.url, tokenFile }) as DeviceProvider; + const lease = await provider.acquire({ env } as unknown as DeviceRequest); + const client = createAgentDeviceClient({ ...lease.client, session: 'verify-ios-test', daemonBaseUrl: lease.daemon!.baseUrl, daemonAuthToken: lease.daemon!.authToken }); + return { snapshot: await client.capture.snapshot(), checkout }; + } + + it('puts a fallback screenshot of a remote snapshot in the agent-device state directory, not in the checkout', async (t) => { + const dir = mkdtempSync(join(tmpdir(), 'verify-borrowed-')); + const stateDir = join(dir, 'state'); + const { snapshot, checkout } = await snapshotFrom(t, { AGENT_DEVICE_STATE_DIR: stateDir }, dir); + + assert.equal(snapshot.fallbackScreenshotPath, join(stateDir, 'snapshot-fallback.png')); + assert.deepEqual(readdirSync(stateDir), ['snapshot-fallback.png']); + assert.deepEqual(readdirSync(checkout), []); + }); + + it('puts it in agent-device\'s default state directory when a person runs e2e without one', async (t) => { + const dir = mkdtempSync(join(tmpdir(), 'verify-borrowed-')); + const home = join(dir, 'home'); + const { snapshot, checkout } = await snapshotFrom(t, { HOME: home }, dir); + + assert.equal(snapshot.fallbackScreenshotPath, join(home, '.agent-device', 'snapshot-fallback.png')); + assert.deepEqual(readdirSync(checkout), []); + }); +}); diff --git a/integration/expo-native/test/device-command.test.ts b/integration/expo-native/test/device-command.test.ts index 6898d6bd852..16fc0c259aa 100644 --- a/integration/expo-native/test/device-command.test.ts +++ b/integration/expo-native/test/device-command.test.ts @@ -19,7 +19,7 @@ afterEach(() => void (globalThis.fetch = realFetch)); function remoteLease(platform: 'ios' | 'android'): RemoteLease { const tokenFile = join(mkdtempSync(join(tmpdir(), 'verify-command-')), 'token'); writeFileSync(tokenFile, 'd'.repeat(64)); - return { backend: 'remote', provider: 'github-actions', platform, session: `${platform}abc123`, providerRef: '7', baseUrl: 'https://quick-fox.trycloudflare.com', tokenFile, deviceId: 'emulator-5554', deviceName: 'Pixel', runner: 'linux', expiresAt: new Date(Date.now() + 3_600_000).toISOString(), builtSha: null, acquiredAt: '', installedBuild: null }; + return { backend: 'remote', provider: 'github-actions', platform, session: `${platform}abc123`, providerRef: '7', baseUrl: 'https://quick-fox.trycloudflare.com', tokenFile, deviceId: 'emulator-5554', deviceName: 'Pixel', runner: 'linux', expiresAt: new Date(Date.now() + 3_600_000).toISOString(), builtSha: null, builtPatch: null, acquiredAt: '', installedBuild: null }; } function tunnel(reply: (path: string) => { status: number; body: unknown }): { readonly calls: { path: string; method: string; authorization: string | null; body: unknown }[] } { @@ -100,6 +100,34 @@ describe('a session started from another core', () => { }); }); +describe('a held session whose build is not the one the lease names', () => { + const sha = 'a'.repeat(40); + const patch = 'c'.repeat(64); + const backend = () => + remoteBackend( + { platform: 'ios', repo: 'clerk/clerk-ios', workflow: 'verify-remote.yml', sessionsDir: mkdtempSync(join(tmpdir(), 'verify-check-')), runner: 'xcode-27', plumbingRunner: 'ubuntu-latest', device: 'iPhone Air', idleMinutes: 15, capMinutes: 60, requirement: '' }, + { env: {}, runner: run, github: async () => assert.fail('a session that answers is not looked up at GitHub') }, + ); + const checked = (build: SessionHealth['build'], lease: Partial) => { + tunnel(() => ({ status: 200, body: { core: coreVersion(), device: { id: 'UDID', name: 'iPhone Air', ready: true }, daemon: true, build, capAt: '', ending: null } satisfies SessionHealth })); + return backend().check({ ...remoteLease('ios'), ...lease }); + }; + + it('is held only while the session last built, and finished, the commit and the changes the lease names', async () => { + const onCommit = { builtSha: sha, builtPatch: null }; + const answers: readonly (readonly [string, SessionHealth['build'], Partial, string])[] = [ + ['the build the lease names', { state: 'built', sha, patch: null, seconds: 9, incremental: true }, onCommit, 'held'], + ['the changes of an install that was cut short, built', { state: 'built', sha, patch, seconds: 9, incremental: true }, onCommit, 'other-app'], + ['the changes of an install that was cut short, still building', { state: 'building', sha, patch, seconds: 9 }, onCommit, 'other-app'], + ['the commit the lease names, failed after the changes were built', { state: 'failed', sha, patch: null, seconds: 9, tail: 'error' }, onCommit, 'other-app'], + ['another commit', { state: 'built', sha: 'b'.repeat(40), patch: null, seconds: 9, incremental: true }, onCommit, 'other-app'], + ['the commit, when the lease names changes on it', { state: 'built', sha, patch: null, seconds: 9, incremental: true }, { builtSha: sha, builtPatch: patch }, 'other-app'], + ['anything, when the lease names no build yet', { state: 'built', sha, patch, seconds: 9, incremental: false }, {}, 'held'], + ]; + for (const [what, build, lease, state] of answers) assert.equal(await checked(build, lease), state, what); + }); +}); + describe('acquiring a session whose agent has another core', () => { it('asks that session to stop through its tunnel and keeps no lease or token', async () => { const sessionsDir = mkdtempSync(join(tmpdir(), 'verify-acquire-')); diff --git a/integration/expo-native/test/evidence.test.ts b/integration/expo-native/test/evidence.test.ts index 0080b214127..aeeae314492 100644 --- a/integration/expo-native/test/evidence.test.ts +++ b/integration/expo-native/test/evidence.test.ts @@ -224,6 +224,13 @@ describe('the evidence block of a pull request description', () => { assert.deepEqual(withEvidenceBlock(notes, 'ios', block), { ok: false, why: `has text between \`${start}\` and \`${end}\` that is not an evidence block`, fix: 'leave one pair of those markers in the description, or none' }); }); + it('writes the block in place of an empty pair of markers, so a description may say where the block goes', () => { + const { start, end } = evidenceMarkers('ios'); + assert.deepEqual(withEvidenceBlock(`## Verification\n\n${start}\n\n${end}\n\n## Notes\n`, 'ios', block), { ok: true, body: `## Verification\n\n${block}\n\n## Notes\n`, was: 'replaced' }); + assert.deepEqual(withEvidenceBlock(`intro\r\n${start}\r\n \t\r\n${end}\r\noutro`, 'ios', block), { ok: true, body: `intro\r\n${block.replaceAll('\n', '\r\n')}\r\noutro`, was: 'replaced' }); + assert.deepEqual(withEvidenceBlock(`${start}\n${end}`, 'ios', block), { ok: true, body: block, was: 'replaced' }); + }); + it('takes no marker from inside a code fence, so a description may show a whole example block', () => { const example = evidenceBlock({ ...summary, run: 'r20261007-010101-aaaa' as RunId }, null, media); const later = evidenceBlock({ ...summary, run: 'r20261009-020202-bbbb' as RunId }, null, media); @@ -367,6 +374,18 @@ describe('attach', () => { assert.equal(calls.length, 0); }); + it('refuses a run on a session build with uncommitted changes, as the run or as its before, calls no gh, and says to commit, push and run again', async () => { + const calls: string[] = []; + const { deps, sealed } = checkout({ runner: async (command) => (calls.push(command), { code: 0, stdout: '', stderr: '' }) }); + const onSession = (builtPatch: string | null): Partial => ({ backend: 'remote', remote: { provider: 'github-actions', runner: 'xcode-27', builtSha: AFTER_COMMIT, builtPatch } }); + const patched = sealed({ called: 'after', change: onSession('c'.repeat(64)) }); + const committed = sealed({ called: 'after', change: onSession(null) }); + const refusal = { code: 'EVIDENCE_UNSAFE', message: `run ${patched.run} ran on a session build of ${AFTER_COMMIT.slice(0, 12)} with uncommitted changes (cccccccccccc), so no commit holds the app it shows`, fix: 'commit and push the changes, {cli} run again, and attach the new run' }; + await assert.rejects(attach(deps, { verb: 'attach', run: patched.run, pr: 9, media: still, before: null }), refusal); + await assert.rejects(attach(deps, { verb: 'attach', run: committed.run, pr: 9, media: still, before: patched.run }), refusal); + assert.deepEqual(calls, []); + }); + it('shows a still change to something that exists as before and after screenshots, side by side at a fixed width, and no video', async () => { const gh = fakeGh(); const { deps, sealed } = checkout(gh); diff --git a/integration/expo-native/test/handoff.test.ts b/integration/expo-native/test/handoff.test.ts index 7891727337b..81920fef8be 100644 --- a/integration/expo-native/test/handoff.test.ts +++ b/integration/expo-native/test/handoff.test.ts @@ -332,7 +332,7 @@ describe('the receiver in the session job', () => { }); }); -const LEASE: RemoteLease = { backend: 'remote', provider: 'github-actions', platform: 'ios', session: 'ios0000aa', providerRef: '37731397352', baseUrl: 'https://session.trycloudflare.com', tokenFile: '', deviceId: 'UDID', deviceName: 'iPhone Air', runner: 'xcode-27', expiresAt: '2026-10-08T06:00:00Z', builtSha: COMMIT, acquiredAt: '2026-10-08T05:00:00.000Z', installedBuild: null }; +const LEASE: RemoteLease = { backend: 'remote', provider: 'github-actions', platform: 'ios', session: 'ios0000aa', providerRef: '37731397352', baseUrl: 'https://session.trycloudflare.com', tokenFile: '', deviceId: 'UDID', deviceName: 'iPhone Air', runner: 'xcode-27', expiresAt: '2026-10-08T06:00:00Z', builtSha: COMMIT, builtPatch: null, acquiredAt: '2026-10-08T05:00:00.000Z', installedBuild: null }; function remoteRun(options: { readonly backend?: EvidenceRecord['backend']; readonly lease?: Lease | null; readonly startedAt?: string; readonly handOff?: DeviceBackend['handOffEvidence'] | null; readonly gh?: (args: readonly string[]) => { readonly code: number; readonly stdout: string; readonly stderr: string } } = {}) { const packageDir = mkdtempSync(join(tmpdir(), 'verify-handoff-verb-')); @@ -353,7 +353,7 @@ function remoteRun(options: { readonly backend?: EvidenceRecord['backend']; read dirty: false, platform: 'ios', backend, - remote: backend === 'remote' ? { provider: 'github-actions', runner: 'xcode-27', builtSha: COMMIT } : null, + remote: backend === 'remote' ? { provider: 'github-actions', runner: 'xcode-27', builtSha: COMMIT, builtPatch: null } : null, device: 'iPhone Air on xcode-27', build: 'ios-000000000000' as BuildKey, results: [{ spec: { kind: 'explored', path: 'specs/explored/a.e2e.ts', feature: null }, title: 'a', platform: 'ios', status: 'passed', seconds: 1, attempts: 1, error: null, skipReason: null, skippedBy: null, tags: [], failurePage: null, failureScreen: null, failureScreenshot: null }], diff --git a/integration/expo-native/test/lease-flow.test.ts b/integration/expo-native/test/lease-flow.test.ts index 12a0b5c9d6f..4948f121e62 100644 --- a/integration/expo-native/test/lease-flow.test.ts +++ b/integration/expo-native/test/lease-flow.test.ts @@ -115,6 +115,41 @@ describe('lease flow', () => { }); }); + it('doctor warns when gh can attach but cannot use the GitHub GraphQL API, as in a Claude Code cloud session, and names the hand-off', async () => { + const { deps } = setup(0); + const calls: { readonly args: readonly string[]; readonly timeoutMs: number | undefined }[] = []; + const gh = async (graphql: { readonly code: number; readonly stderr: string }) => + ( + await doctor( + { + ...deps, + runner: async (command, args, options) => { + if (command !== 'gh') return { code: 0, stdout: '', stderr: '' }; + calls.push({ args, timeoutMs: options?.timeoutMs }); + return args[0] === 'api' ? { ...graphql, stdout: '' } : { code: 0, stdout: ' -a, --attach file Attach a file', stderr: '' }; + }, + }, + { verb: 'doctor', live: false }, + ) + ).checks.find((c) => c.id === 'gh-attach')!; + const fix = "attach a remote run before {cli} down, so that attach hands its evidence to the session's runner, or check `gh auth status`"; + const cannotPost = "so `{cli} attach` cannot put a run's video and screenshots in a pull request description from this machine"; + + assert.deepEqual(await gh({ code: 1, stderr: 'HTTP 403: Forbidden (https://api.github.com/graphql)\n' }), { + id: 'gh-attach', + ok: true, + state: 'warning', + detail: `gh cannot use the GitHub GraphQL API (HTTP 403: Forbidden (https://api.github.com/graphql)), which \`gh pr view\` and \`gh pr edit\` need, ${cannotPost}`, + fix, + }); + assert.deepEqual(calls, [ + { args: ['pr', 'edit', '--help'], timeoutMs: undefined }, + { args: ['api', 'graphql', '-f', 'query={viewer{login}}'], timeoutMs: 10_000 }, + ]); + assert.equal((await gh({ code: 124, stderr: '' })).detail, `gh cannot use the GitHub GraphQL API (no answer in 10 seconds), which \`gh pr view\` and \`gh pr edit\` need, ${cannotPost}`); + assert.deepEqual(await gh({ code: 0, stderr: '' }), { id: 'gh-attach', ok: true, detail: 'gh pr edit supports --attach, and gh can use the GitHub GraphQL API' }); + }); + it('refuses --runner with the local backend as a usage error, before it builds, leases, or checks anything', async () => { const { deps, events } = setup(0); const usage = (error: VerifyFailure) => error.code === 'USAGE' && error.message === '--runner names a CI runner and the local backend has none' && error.fix === 'drop --runner, or pass --backend remote'; diff --git a/integration/expo-native/test/native-build.test.ts b/integration/expo-native/test/native-build.test.ts index 699fab2e765..483decade9f 100644 --- a/integration/expo-native/test/native-build.test.ts +++ b/integration/expo-native/test/native-build.test.ts @@ -12,6 +12,7 @@ import { buildStandalone, locateNativeBuild, nativeCacheDir, + nativeProjectKey, type FixtureSite, } from '../src/fixture.ts'; import { VerifyFailure } from '../src/core/types.ts'; @@ -368,6 +369,18 @@ describe('the standalone build that a person or a remote session starts', () => rmSync(at.site.worktree, { recursive: true, force: true }); }); + it('takes the native build kept for these native inputs and embeds only the new JS when a cache is named', async () => { + const at = checkout(); + const say = () => undefined; + await buildStandalone('ios', 'key', say, at.site, Date.now, at.cache); + at.write('packages/expo/src/index.ts', 'js two'); + at.steps.length = 0; + const app = await buildStandalone('ios', 'key', say, at.site, Date.now, at.cache); + assert.deepEqual(at.steps, ['pnpm install', 'turbo build', ...EMBED]); + assert.equal(bundleOf(app), 'embedded js two'); + rmSync(at.site.worktree, { recursive: true, force: true }); + }); + it('says how long a step ran before it failed', async () => { const at = checkout(); const said: string[] = []; @@ -390,6 +403,32 @@ describe('the standalone build that a person or a remote session starts', () => }); }); +describe('the native project a remote session generates', () => { + it('is generated again when uncommitted changes touch a native input, and not for a JS-only change', async () => { + const at = checkout(); + const worktree = at.site.worktree; + execFileSync('git', ['-C', worktree, 'add', '-A']); + execFileSync('git', ['-C', worktree, '-c', 'user.name=t', '-c', 'user.email=t@example.com', 'commit', '-q', '-m', 'base']); + const committed = await nativeProjectKey('ios', worktree); + await build(at, null, committed); + at.write('packages/expo/src/index.ts', 'js two'); + assert.equal(await nativeProjectKey('ios', worktree), committed); + at.steps.length = 0; + await build(at, null, committed); + assert.deepEqual(at.steps, ['xcodebuild']); + at.write('packages/expo-native-components/ios/ClerkExpo.swift', 'native two'); + const edited = await nativeProjectKey('ios', worktree); + assert.notEqual(edited, committed); + at.write('packages/expo-native-components/ios/Added.swift', 'native three'); + const added = await nativeProjectKey('ios', worktree); + assert.notEqual(added, edited); + at.steps.length = 0; + await build(at, null, added); + assert.deepEqual(at.steps, NATIVE_BUILD); + rmSync(worktree, { recursive: true, force: true }); + }); +}); + describe('the build key of a standalone app', () => { it('covers the JS of every workspace package the test app links, which a dev client leaves to Metro', () => { const linked = [ diff --git a/integration/expo-native/test/native-store.test.ts b/integration/expo-native/test/native-store.test.ts new file mode 100644 index 00000000000..58ca9655772 --- /dev/null +++ b/integration/expo-native/test/native-store.test.ts @@ -0,0 +1,263 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { execFileSync, spawnSync } from 'node:child_process'; +import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { basename, join } from 'node:path'; +import { after, describe, it } from 'node:test'; +import { artifact } from '../src/fixture.ts'; + +const WORKTREE = join(import.meta.dirname, '..', '..', '..'); +const FETCH = join(import.meta.dirname, '..', 'bin', 'fetch-native-build.sh'); +const workflow = (name: string) => readFileSync(join(WORKTREE, '.github', 'workflows', name), 'utf8'); +const stepsOf = (text: string) => text.split(/\n(?= {6}- )/); + +const scratch: string[] = []; +after(() => { + for (const dir of scratch) rmSync(dir, { recursive: true, force: true }); +}); +const tempDir = (prefix: string) => { + const dir = mkdtempSync(join(tmpdir(), prefix)); + scratch.push(dir); + return dir; +}; + +type Platform = 'ios' | 'android'; +const APP: Readonly> = { ios: 'ClerkExpoNativeBuildFixture.app', android: 'app-release.apk' }; + +interface Artifact { + readonly id: number; + readonly name: string; + readonly branch?: string; + readonly fork?: boolean; + readonly expired?: boolean; + readonly created?: string; +} + +interface Run { + readonly id: number; + readonly branch?: string; + readonly fork?: boolean; + readonly artifacts: readonly Artifact[]; +} + +function fetched( + platform: Platform, + env: Readonly>, + store: { readonly artifacts?: readonly Artifact[]; readonly runs?: readonly Run[] }, +) { + const dir = tempDir('verify-expo-fetch-'); + const bin = join(dir, 'bin'); + mkdirSync(bin); + const ofRun = (entry: Artifact) => ({ + id: entry.id, + name: entry.name, + expired: entry.expired ?? false, + created_at: entry.created ?? '2026-10-14T10:00:00Z', + workflow_run: { head_branch: entry.branch ?? 'mike/change', repository_id: 1, head_repository_id: entry.fork ? 2 : 1 }, + }); + writeFileSync(join(dir, 'artifacts.json'), JSON.stringify({ artifacts: (store.artifacts ?? []).map(ofRun) })); + writeFileSync( + join(dir, 'runs.json'), + JSON.stringify({ + workflow_runs: (store.runs ?? []).map(run => ({ + id: run.id, + head_branch: run.branch ?? 'mike/change', + repository: { id: 1 }, + head_repository: { id: run.fork ? 2 : 1 }, + })), + }), + ); + for (const run of store.runs ?? []) + writeFileSync(join(dir, `run-${run.id}.json`), JSON.stringify({ artifacts: run.artifacts.map(ofRun) })); + for (const entry of [...(store.artifacts ?? []), ...(store.runs ?? []).flatMap(run => run.artifacts)]) { + const zipped = join(dir, `zipped-${entry.id}`); + mkdirSync(zipped, { recursive: true }); + if (platform === 'ios') writeFileSync(join(zipped, 'main.jsbundle'), `bundle of ${entry.id}`); + else writeFileSync(join(zipped, APP.android), `apk of ${entry.id}`); + execFileSync('zip', ['-q', '-r', join(dir, `artifact-${entry.id}.zip`), '.'], { cwd: zipped }); + } + writeFileSync( + join(bin, 'gh'), + `#!/bin/bash +url="$2" +echo "$url" >> "${join(dir, 'gh.log')}" +case "$url" in + repos/clerk/javascript/actions/artifacts/*/zip) id="\${url%/zip}"; cat "${dir}/artifact-\${id##*/}.zip" ;; + repos/clerk/javascript/actions/runs/*/artifacts*) id="\${url%/artifacts*}"; jq -r "$4" "${dir}/run-\${id##*/}.json" ;; + repos/clerk/javascript/actions/workflows/verify-e2e.yml/runs*) jq -r "$4" "${join(dir, 'runs.json')}" ;; + repos/clerk/javascript/actions/artifacts\\?*) jq -r "$4" "${join(dir, 'artifacts.json')}" ;; + *) exit 1 ;; +esac +`, + { mode: 0o755 }, + ); + const output = join(dir, 'output'); + writeFileSync(output, ''); + const ran = spawnSync('bash', [FETCH], { + cwd: dir, + env: { + PATH: `${bin}:${process.env.PATH}`, + GITHUB_REPOSITORY: 'clerk/javascript', + GITHUB_OUTPUT: output, + RUNNER_TEMP: dir, + BRANCH: 'mike/change', + PLATFORM: platform, + NATIVE_APP: APP[platform], + ...env, + }, + encoding: 'utf8', + timeout: 20_000, + }); + const read = (path: string) => + !existsSync(path) + ? null + : platform === 'ios' + ? readFileSync(join(path, 'main.jsbundle'), 'utf8') + : readFileSync(path, 'utf8'); + const cache = join(dir, 'cache'); + return { + status: ran.status, + stdout: ran.stdout, + output: readFileSync(output, 'utf8'), + kept: read(join(dir, 'native', 'kept', APP[platform])), + cached: Object.fromEntries( + (existsSync(cache) ? readdirSync(cache) : []).map(id => [id, read(join(cache, id, APP[platform]))]), + ), + gh: existsSync(join(dir, 'gh.log')) ? readFileSync(join(dir, 'gh.log'), 'utf8').trim().split('\n') : [], + }; +} + +const exact = (platform: Platform) => ({ NATIVE_ID: `${platform}-0123456789abcdef`, NATIVE_DIR: 'native/kept' }); + +describe('the device suite takes the native build an earlier run kept for its fingerprint', () => { + it('takes the newest artifact of that name from a run of the same branch of this repository', () => { + const name = 'verify-expo-native-ios-0123456789abcdef'; + const result = fetched('ios', exact('ios'), { + artifacts: [ + { id: 11, name, created: '2026-10-14T10:00:00Z' }, + { id: 12, name, created: '2026-10-14T11:00:00Z' }, + { id: 13, name, created: '2026-10-14T09:00:00Z' }, + ], + }); + assert.equal(result.stdout, `Took the native build from artifact 12 (${name}).\n`); + assert.equal(result.output, 'found=kept\n'); + assert.equal(result.kept, 'bundle of 12'); + }); + + it('takes the APK out of the archive on Android', () => { + const result = fetched('android', exact('android'), { + artifacts: [{ id: 21, name: 'verify-expo-native-android-0123456789abcdef' }], + }); + assert.equal(result.output, 'found=kept\n'); + assert.equal(result.kept, 'apk of 21'); + }); + + it('builds when the only artifacts are of another branch, of a fork, or expired', () => { + const name = 'verify-expo-native-ios-0123456789abcdef'; + const result = fetched('ios', exact('ios'), { + artifacts: [ + { id: 31, name, branch: 'main' }, + { id: 32, name, fork: true }, + { id: 33, name, expired: true }, + ], + }); + assert.equal(result.status, 0); + assert.equal(result.output, 'found=none\n'); + assert.equal(result.kept, null); + assert.equal(result.gh.length, 1); + assert.equal( + result.stdout, + 'No native build of this branch is kept for ios-0123456789abcdef, so this job builds it.\n', + ); + }); +}); + +describe('a session takes the native builds that runs of its branch kept, before it knows its fingerprint', () => { + const cache = { NATIVE_CACHE: 'cache' }; + + it('puts each kept build of its platform under its fingerprint, newest run first, and at most three', () => { + const result = fetched('ios', cache, { + runs: [ + { + id: 5, + artifacts: [ + { id: 51, name: 'verify-expo-native-ios-aaaa' }, + { id: 52, name: 'verify-expo-native-android-aaaa' }, + { id: 53, name: 'verify-e2e-ios' }, + ], + }, + { + id: 4, + artifacts: [ + { id: 41, name: 'verify-expo-native-ios-aaaa' }, + { id: 42, name: 'verify-expo-native-ios-bbbb' }, + ], + }, + { + id: 3, + artifacts: [ + { id: 31, name: 'verify-expo-native-ios-cccc' }, + { id: 32, name: 'verify-expo-native-ios-dddd', expired: true }, + ], + }, + { id: 2, artifacts: [{ id: 21, name: 'verify-expo-native-ios-eeee' }] }, + ], + }); + assert.equal(result.status, 0); + assert.deepEqual(result.cached, { + 'ios-aaaa': 'bundle of 51', + 'ios-bbbb': 'bundle of 42', + 'ios-cccc': 'bundle of 31', + }); + assert.equal(result.output, 'taken=3\n'); + assert.ok(!result.gh.some(url => url.includes('/runs/2/')), 'stops after three'); + }); + + it('takes nothing from a run of another branch or of a fork', () => { + const result = fetched('android', cache, { + runs: [ + { id: 7, branch: 'main', artifacts: [{ id: 71, name: 'verify-expo-native-android-aaaa' }] }, + { id: 6, fork: true, artifacts: [{ id: 61, name: 'verify-expo-native-android-bbbb' }] }, + ], + }); + assert.deepEqual(result.cached, {}); + assert.equal(result.output, 'taken=0\n'); + assert.deepEqual(result.gh, [ + 'repos/clerk/javascript/actions/workflows/verify-e2e.yml/runs?branch=mike/change&per_page=20', + ]); + }); + + it('starts with nothing when GitHub does not answer', () => { + const result = fetched('ios', { ...cache, GITHUB_REPOSITORY: 'nowhere/at-all' }, { runs: [] }); + assert.equal(result.status, 0); + assert.equal(result.output, 'taken=0\n'); + }); +}); + +describe('the workflows and the native build store', () => { + const session = workflow('verify-remote.yml'); + const steps = stepsOf(session); + const at = (needle: string) => steps.findIndex(step => step.includes(needle)); + const fetch = steps[at('- name: Fetch the native builds that runs of this branch kept')] ?? ''; + + it('runs the same script in the device suite and in a session', () => { + const e2e = stepsOf(workflow('verify-e2e.yml')).find(step => step.includes('id: kept')) ?? ''; + assert.ok(e2e.includes('run: integration/expo-native/bin/fetch-native-build.sh')); + assert.ok(e2e.includes('NATIVE_ID: ${{ steps.native.outputs.id }}')); + assert.ok(fetch.includes('run: integration/expo-native/bin/fetch-native-build.sh')); + }); + + it('fetches before the session agent starts, into the cache the session build reads, under the app name the build looks for', () => { + assert.ok(at('id: kept') > 0 && at('id: kept') < at('- name: Start the session agent and tunnel')); + assert.ok(session.includes('\n VERIFY_NATIVE_CACHE: integration/expo-native/.verify/native\n')); + assert.ok(fetch.includes('NATIVE_CACHE: ${{ env.VERIFY_NATIVE_CACHE }}')); + assert.ok(fetch.includes('BRANCH: ${{ github.ref_name }}')); + const app = (platform: Platform) => basename(artifact(platform, 'standalone')); + assert.ok(fetch.includes(`NATIVE_APP: \${{ env.PLATFORM == 'ios' && '${app('ios')}' || '${app('android')}' }}`)); + assert.match( + session.slice(session.indexOf('\n session:\n')), + /^\n {2}session:\n(?: {4}.*\n)*? {4}permissions:\n {6}contents: read\n {6}actions: read\n/, + ); + }); +}); diff --git a/integration/expo-native/test/remote-backend.test.ts b/integration/expo-native/test/remote-backend.test.ts index 5088274e203..ca442835564 100644 --- a/integration/expo-native/test/remote-backend.test.ts +++ b/integration/expo-native/test/remote-backend.test.ts @@ -1,15 +1,15 @@ import '../testing/git-env.ts'; import assert from 'node:assert/strict'; import { execFileSync } from 'node:child_process'; -import { mkdirSync, mkdtempSync, writeFileSync } from 'node:fs'; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { describe, it } from 'node:test'; import { run } from '../src/core/exec.ts'; -import { remoteBackend } from '../src/core/remote/backend.ts'; +import { remoteBackend, workingPatch } from '../src/core/remote/backend.ts'; import type { GitHub } from '../src/core/remote/github.ts'; import { driverId, newSessionRequest, type RemoteSettings } from '../src/core/remote/settings.ts'; -import { BUILD_TIMEOUT_MINUTES, type SessionHealth } from '../src/core/remote/protocol.ts'; +import { BUILD_LOG_START, BUILD_TIMEOUT_MINUTES, PATCH_LIMIT_BYTES, sha256Hex, type SessionHealth } from '../src/core/remote/protocol.ts'; import { VerifyFailure, type BuildKey, type BuiltApp, type RemoteLease } from '../src/core/types.ts'; function repo(): { dir: string; git: (...args: string[]) => string } { @@ -45,27 +45,64 @@ function github(routes: (method: string, path: string) => { status: number; json describe('the remote backend owns the build', () => { const input = (dir: string) => ({ worktree: dir, inputs: ['App.swift'] }); - it('names the pushed HEAD as the commit to build', async () => { + it('names the pushed HEAD as the commit to build, with nothing on top of a clean tree', async () => { const { dir, git } = repo(); const { hub } = github(() => ({ status: 200, json: {} })); - assert.equal(await remoteBackend(settings(dir), { env: {}, runner: run, github: async () => hub }).sourceCommit!(input(dir)), git('rev-parse', 'HEAD')); + assert.deepEqual(await remoteBackend(settings(dir), { env: {}, runner: run, github: async () => hub }).sourceCommit!(input(dir)), { sha: git('rev-parse', 'HEAD'), patch: null }); }); - it('refuses a tree whose app sources are edited, and says to commit and push', async () => { - const { dir } = repo(); + it('sends edits to the app sources as a patch on the pushed HEAD, with the sha256 of the whole patch', async () => { + const { dir, git } = repo(); writeFileSync(join(dir, 'App.swift'), 'edited'); const { hub } = github(() => ({ status: 200, json: {} })); - await assert.rejects( - remoteBackend(settings(dir), { env: {}, runner: run, github: async () => hub }).sourceCommit!(input(dir)), - (error: VerifyFailure) => error.code === 'BUILD_FAILED' && error.message.includes('App.swift') && error.fix.includes('git push'), - ); + const source = await remoteBackend(settings(dir), { env: {}, runner: run, github: async () => hub }).sourceCommit!(input(dir)); + assert.equal(source.sha, git('rev-parse', 'HEAD')); + assert.match(source.patch?.bytes.toString() ?? '', /^-app\n\\ No newline at end of file\n\+edited$/m); + assert.equal(source.patch?.sha256, sha256Hex(source.patch!.bytes)); }); - it('ignores edits outside the app sources', async () => { + it('sends nothing for edits outside the app sources', async () => { const { dir } = repo(); writeFileSync(join(dir, 'README.md'), 'edited docs'); const { hub } = github(() => ({ status: 200, json: {} })); - assert.ok(await remoteBackend(settings(dir), { env: {}, runner: run, github: async () => hub }).sourceCommit!(input(dir))); + assert.equal((await remoteBackend(settings(dir), { env: {}, runner: run, github: async () => hub }).sourceCommit!(input(dir))).patch, null); + }); + + it('refuses uncommitted changes larger than a session takes, however large, with the fix to commit them', async () => { + const { dir, git } = repo(); + writeFileSync(join(dir, 'App.swift'), 'edited'); + const bin = mkdtempSync(join(tmpdir(), 'verify-remote-bin-')); + const realGit = execFileSync('which', ['git'], { encoding: 'utf8' }).trim(); + writeFileSync(join(bin, 'git'), `#!/bin/sh\nif [ "$1" = diff ]; then exec head -c "$PATCH_BYTES" /dev/zero; fi\nexec "${realGit}" "$@"\n`, { mode: 0o755 }); + const { hub } = github(() => ({ status: 200, json: {} })); + const path = process.env.PATH; + process.env.PATH = `${bin}:${path}`; + try { + process.env.PATCH_BYTES = String(PATCH_LIMIT_BYTES); + const fits = await remoteBackend(settings(dir), { env: {}, runner: run, github: async () => hub }).sourceCommit!(input(dir)); + assert.deepEqual([fits.sha, fits.patch?.bytes.length], [git('rev-parse', 'HEAD'), PATCH_LIMIT_BYTES]); + for (const bytes of [PATCH_LIMIT_BYTES + 1, 4 * PATCH_LIMIT_BYTES + 1]) { + process.env.PATCH_BYTES = String(bytes); + await assert.rejects(remoteBackend(settings(dir), { env: {}, runner: run, github: async () => hub }).sourceCommit!(input(dir)), { + code: 'BUILD_FAILED', + message: 'the uncommitted changes to the app make a patch larger than the 16 MiB a session takes', + fix: 'git commit the changes and git push, then rerun', + }); + } + } finally { + process.env.PATH = path; + delete process.env.PATCH_BYTES; + } + }); + + it('refuses with the fix to commit the changes when git cannot read them', async () => { + const { dir } = repo(); + const { hub } = github(() => ({ status: 200, json: {} })); + await assert.rejects(remoteBackend(settings(dir), { env: {}, runner: run, github: async () => hub }).sourceCommit!({ worktree: dir, inputs: ['Missing.swift'] }), { + code: 'BUILD_FAILED', + message: "git could not read the uncommitted changes to the app: fatal: pathspec 'Missing.swift' did not match any files", + fix: 'git commit the changes and git push, then rerun', + }); }); it('refuses a commit GitHub does not have', async () => { @@ -78,39 +115,93 @@ describe('the remote backend owns the build', () => { }); }); +describe('the patch of a working tree', () => { + const PNG = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0x00, 0x00, 0x00, 0x0d, 0xff, 0x00]); + + function edited(): { dir: string; git: (...args: string[]) => string } { + const { dir, git } = repo(); + mkdirSync(join(dir, 'Sources', 'Generated'), { recursive: true }); + writeFileSync(join(dir, '.gitignore'), 'Sources/Generated/\n'); + writeFileSync(join(dir, 'Sources', 'Kept.swift'), 'let kept = "one"\n'); + writeFileSync(join(dir, 'Sources', 'Gone.swift'), 'let gone = 1\n'); + writeFileSync(join(dir, 'Sources', 'icon.png'), PNG); + git('add', '-A'); + git('commit', '-q', '-m', 'sources'); + writeFileSync(join(dir, 'Sources', 'Kept.swift'), 'let kept = "two"\n'); + git('add', 'Sources/Kept.swift'); + writeFileSync(join(dir, 'Sources', 'Kept.swift'), 'let kept = "three"\n'); + rmSync(join(dir, 'Sources', 'Gone.swift')); + writeFileSync(join(dir, 'Sources', 'New.swift'), 'let made = true\n'); + writeFileSync(join(dir, 'Sources', 'icon.png'), Buffer.concat([PNG, Buffer.from([0x00, 0x01, 0x02])])); + writeFileSync(join(dir, 'Sources', 'Generated', 'Secrets.swift'), 'let key = "ignored"\n'); + writeFileSync(join(dir, 'README.md'), 'edited docs'); + return { dir, git }; + } + + it('turns a tracked change, a new file, a deleted file and a binary file into one patch that a clean checkout of the commit takes', () => { + const { dir, git } = edited(); + const patch = workingPatch(dir, ['Sources']); + assert.ok(patch !== null); + const clean = mkdtempSync(join(tmpdir(), 'verify-remote-clean-')); + execFileSync('git', ['clone', '-q', dir, clean]); + writeFileSync(join(clean, '..', `${patch.sha256}.diff`), patch.bytes); + execFileSync('git', ['apply', '--binary', join(clean, '..', `${patch.sha256}.diff`)], { cwd: clean }); + assert.equal(readFileSync(join(clean, 'Sources', 'Kept.swift'), 'utf8'), 'let kept = "three"\n', 'the working file, not the staged one'); + assert.equal(existsSync(join(clean, 'Sources', 'Gone.swift')), false); + assert.equal(readFileSync(join(clean, 'Sources', 'New.swift'), 'utf8'), 'let made = true\n'); + assert.deepEqual(readFileSync(join(clean, 'Sources', 'icon.png')), Buffer.concat([PNG, Buffer.from([0x00, 0x01, 0x02])])); + assert.equal(existsSync(join(clean, 'Sources', 'Generated')), false, 'a file in an ignored folder is left out'); + assert.equal(readFileSync(join(clean, 'README.md'), 'utf8'), 'docs', 'a file outside the app sources is left out'); + assert.equal(git('diff', '--cached', '--name-only'), 'Sources/Kept.swift', 'the index of the checkout is as it was'); + }); + + it('is nothing for a tree that matches its commit', () => { + const { dir } = repo(); + writeFileSync(join(dir, 'README.md'), 'edited docs'); + assert.equal(workingPatch(dir, ['App.swift']), null); + }); +}); + describe('the remote backend waits for the session to build', () => { const SHA = 'a'.repeat(40); const realFetch = globalThis.fetch; - async function installOnASessionWhoseBuildIs(build: SessionHealth['build']): Promise<{ failure: VerifyFailure; asked: string[] }> { + async function installOnASessionWhoseBuildIs(build: SessionHealth['build'] | readonly SessionHealth['build'][], log: readonly string[] = ['']): Promise<{ failure: VerifyFailure; asked: string[]; progress: string[] }> { const { dir } = repo(); const session = join(dir, '.verify', 'remote', 'ios0000cc'); mkdirSync(session, { recursive: true }); writeFileSync(join(session, 'token'), 'f'.repeat(64)); - const lease: RemoteLease = { backend: 'remote', provider: 'github-actions', platform: 'ios', session: 'ios0000cc', providerRef: '7', baseUrl: 'https://quick-fox.trycloudflare.com', tokenFile: join(session, 'token'), deviceId: 'UDID-1', deviceName: 'iPhone Air', runner: 'free-mac', expiresAt: '2026-10-13T18:00:00.000Z', builtSha: null, acquiredAt: '2026-10-13T17:00:00.000Z', installedBuild: null }; - const app: BuiltApp = { platform: 'ios', key: 'ios-k1' as BuildKey, appId: 'com.example.app', path: null, source: 'github-actions', sourceSha: SHA }; - const health: SessionHealth = { core: 'unread', device: { id: 'UDID-1', name: 'iPhone Air', ready: true }, daemon: true, build, capAt: '2026-10-13T18:00:00.000Z', ending: null }; + const lease: RemoteLease = { backend: 'remote', provider: 'github-actions', platform: 'ios', session: 'ios0000cc', providerRef: '7', baseUrl: 'https://quick-fox.trycloudflare.com', tokenFile: join(session, 'token'), deviceId: 'UDID-1', deviceName: 'iPhone Air', runner: 'free-mac', expiresAt: '2026-10-13T18:00:00.000Z', builtSha: null, builtPatch: null, acquiredAt: '2026-10-13T17:00:00.000Z', installedBuild: null }; + const app: BuiltApp = { platform: 'ios', key: 'ios-k1' as BuildKey, appId: 'com.example.app', path: null, source: 'github-actions', sourceSha: SHA, patch: null }; + const builds = Array.isArray(build) ? build : [build]; + const health = (polled: number): SessionHealth => ({ core: 'unread', device: { id: 'UDID-1', name: 'iPhone Air', ready: true }, daemon: true, build: builds[Math.min(polled, builds.length - 1)]!, capAt: '2026-10-13T18:00:00.000Z', ending: null }); const asked: string[] = []; + let polled = 0; globalThis.fetch = (async (input: string | URL | Request, init?: RequestInit) => { const url = new URL(String(input)); asked.push(`${init?.method ?? 'GET'} ${url.pathname}${url.search}`); - return url.pathname === '/__sim/build' ? Response.json(health, { status: 202 }) : Response.json(health); + if (url.pathname === '/__sim/build') return Response.json(health(0), { status: 202 }); + if (url.pathname === '/__sim/health') return Response.json(health(polled++)); + const written = Buffer.from(log[Math.min(polled - 1, log.length - 1)]!); + const from = Number(url.searchParams.get('from')); + return new Response(written.subarray(from), { headers: { [BUILD_LOG_START]: String(from) } }); }) as typeof fetch; try { const { hub } = github(() => ({ status: 200, json: {} })); - const failure = await remoteBackend(settings(dir), { env: {}, runner: run, github: async () => hub }).install(lease, app, () => undefined).then( + const progress: string[] = []; + const failure = await remoteBackend(settings(dir), { env: {}, runner: run, github: async () => hub }).install(lease, app, (line) => progress.push(line)).then( () => assert.fail('the install did not fail'), (error: VerifyFailure) => error, ); - return { failure, asked }; + return { failure, asked, progress }; } finally { globalThis.fetch = realFetch; } } it('says a build that the session stopped at its limit ran out of time, and to run up again', { timeout: 20_000 }, async () => { - const { failure, asked } = await installOnASessionWhoseBuildIs({ state: 'timed-out', sha: SHA, seconds: BUILD_TIMEOUT_MINUTES * 60, tail: 'build pnpm install took 412s\nbuild xcodebuild Release (JS embedded)' }); - assert.deepEqual(asked, [`POST /__sim/build?sha=${SHA}`, 'GET /__sim/health']); + const { failure, asked } = await installOnASessionWhoseBuildIs({ state: 'timed-out', sha: SHA, patch: null, seconds: BUILD_TIMEOUT_MINUTES * 60, tail: 'build pnpm install took 412s\nbuild xcodebuild Release (JS embedded)' }); + assert.deepEqual(asked, [`POST /__sim/build?sha=${SHA}`, 'GET /__sim/health', 'GET /__sim/build/log?from=0']); assert.equal(failure.code, 'NOT_READY'); assert.equal( failure.message, @@ -120,9 +211,25 @@ describe('the remote backend waits for the session to build', () => { }); it('says a build that exited with an error failed, and to fix the error', async () => { - const { failure } = await installOnASessionWhoseBuildIs({ state: 'failed', sha: SHA, seconds: 61, tail: 'error: no such module' }); + const { failure } = await installOnASessionWhoseBuildIs({ state: 'failed', sha: SHA, patch: null, seconds: 61, tail: 'error: no such module' }); assert.deepEqual([failure.code, failure.message, failure.fix], ['BUILD_FAILED', 'the session could not build aaaaaaaaaaaa:\nerror: no such module', 'fix the build error above, commit, push, then rerun {cli} up']); }); + + it('shows each line of the build output once, as the session writes it, and holds back a line that is not finished', { timeout: 20_000 }, async () => { + const building = { state: 'building', sha: SHA, patch: null, seconds: 5 } as const; + const failed = { state: 'failed', sha: SHA, patch: null, seconds: 9, tail: 'error: no such module' } as const; + const { asked, progress } = await installOnASessionWhoseBuildIs([building, failed], ['$ xcodebuild build\ncompil', '$ xcodebuild build\ncompiling ClerkKit\nerror: no such module\n']); + assert.deepEqual(asked.filter((line) => line.includes('/log')), ['GET /__sim/build/log?from=0', 'GET /__sim/build/log?from=19']); + assert.deepEqual(progress.filter((line) => line.startsWith('build |')), ['build | $ xcodebuild build', 'build | compiling ClerkKit', 'build | error: no such module']); + }); + + it('waits out a build other than the one it asked for, and shows none of its output', { timeout: 20_000 }, async () => { + const another = { state: 'failed', sha: SHA, patch: 'c'.repeat(64), seconds: 9, tail: 'error: of another build' } as const; + const ours = { state: 'failed', sha: SHA, patch: null, seconds: 9, tail: 'error: no such module' } as const; + const { asked, failure } = await installOnASessionWhoseBuildIs([another, ours], ['output of another build\n', 'error: no such module\n']); + assert.deepEqual(asked, [`POST /__sim/build?sha=${SHA}`, 'GET /__sim/health', 'GET /__sim/health', 'GET /__sim/build/log?from=0']); + assert.equal(failure.message, 'the session could not build aaaaaaaaaaaa:\nerror: no such module'); + }); }); describe('session settings', () => { diff --git a/integration/expo-native/test/remote-contract.test.ts b/integration/expo-native/test/remote-contract.test.ts index 5a7ef194705..46547e465ab 100644 --- a/integration/expo-native/test/remote-contract.test.ts +++ b/integration/expo-native/test/remote-contract.test.ts @@ -51,6 +51,11 @@ describe('the session workflow and the driver agree', () => { assert.match(agentStep, /^ {10}GITHUB_TOKEN: \$\{\{ github\.token \}\}$/m); }); + it('tells the session agent the default branch of the repository, on which it builds no uncommitted changes', () => { + const agentStep = workflow.split(/\n(?= {6}- )/).find((step) => step.includes('session-agent.ts" serve'))!; + assert.match(agentStep, /^ {10}VERIFY_SESSION_DEFAULT_BRANCH: \$\{\{ github\.event\.repository\.default_branch \}\}$/m); + }); + it('gives the agent the files and variables it waits on', () => { for (const needle of ['VERIFY_SESSION_REQUEST', 'VERIFY_SESSION_WORK', 'VERIFY_SESSION_DEVICE_ID', 'VERIFY_SESSION_DEVICE_MODULE', 'VERIFY_SESSION_CLOUDFLARED', '$WORK/tunnel', 'device-ready', '$VERIFY_SESSION_WORK/ended', 'persist-credentials: false']) { assert.ok(workflow.includes(needle), needle); @@ -119,11 +124,12 @@ describe('a borrowed device as the tests are told of it', () => { assert.equal(JSON.stringify(config).includes('f'.repeat(64)), false, 'the token is not in the composed config'); const provider = engineDevice(target.device); assert.ok(!Array.isArray(provider) && 'acquire' in provider); - assert.deepEqual(await provider.acquire({} as unknown as Parameters[0]), { + assert.deepEqual(await provider.acquire({ env: { AGENT_DEVICE_STATE_DIR: '/state' } } as unknown as Parameters[0]), { id: 'AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE', deviceId: 'AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE', device: 'AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE', daemon: { baseUrl: 'https://x.trycloudflare.com/agent-device', authToken: 'f'.repeat(64) }, + client: { cwd: '/state' }, }); assert.ok(usedSecretValues().includes('f'.repeat(64))); }); diff --git a/integration/expo-native/test/remote-host.test.ts b/integration/expo-native/test/remote-host.test.ts index a50e2b8417b..a2d3b08d7ca 100644 --- a/integration/expo-native/test/remote-host.test.ts +++ b/integration/expo-native/test/remote-host.test.ts @@ -315,6 +315,15 @@ describe("a remote session's device recipe", () => { ]); }); + it('lets a patch change exactly the paths the CLI sends from the tree', () => { + for (const platform of ['ios', 'android'] as const) + assert.deepEqual( + answer(sessionDevice({ id: 'device-1', platform }), { op: 'inputs' }), + host.buildInputs(platform, 'remote'), + platform, + ); + }); + it('reads the same logcat tags as the local lane', () => { const logs = answer(emulator, { op: 'logs', @@ -338,6 +347,10 @@ describe('the session workflow serves both platforms', () => { ); }); + it('tells the session agent the default branch, so it refuses uncommitted changes in a session on it', () => { + assert.ok(workflow.includes('VERIFY_SESSION_DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}')); + }); + it('installs the image the lane AVD names and waits on the boot files this code writes', () => { const image = systemImage({ os: 'linux', arch: 'x64', home: empty, env: {}, kvm: join(empty, 'kvm') }); assert.ok(workflow.includes(`IMAGE="${image}"`), image); diff --git a/integration/expo-native/test/remote-lease-flow.test.ts b/integration/expo-native/test/remote-lease-flow.test.ts index fb9794873d6..5ee18feea63 100644 --- a/integration/expo-native/test/remote-lease-flow.test.ts +++ b/integration/expo-native/test/remote-lease-flow.test.ts @@ -8,7 +8,7 @@ import { heldInstances } from '../testing/fake-instances.ts'; import { describe, it } from 'node:test'; import { doctor, down, up, type Deps } from '../src/core/verbs.ts'; import { openWorkspace } from '../src/core/workspace.ts'; -import { VerifyFailure, type BuiltApp, type DeviceBackend, type HostAdapter, type RemoteLease } from '../src/core/types.ts'; +import { VerifyFailure, type BuiltApp, type DeviceBackend, type HostAdapter, type RemoteLease, type SessionSource } from '../src/core/types.ts'; const shaOf = (app: BuiltApp): string | null => (app.source === 'local' ? null : app.sourceSha); @@ -20,10 +20,11 @@ function setup() { const progress: string[] = []; const finished: string[] = []; let sha = 'a'.repeat(40); - let health: 'held' | 'lost' = 'held'; + let health: 'held' | 'lost' | 'other-app' = 'held'; let pushed = true; let stuck = false; - const lease = (session: string): RemoteLease => ({ backend: 'remote', provider: 'github-actions', platform: 'ios', session, providerRef: '1', baseUrl: 'https://x.trycloudflare.com', tokenFile: join(dir, 'token'), deviceId: 'UDID', deviceName: 'iPhone Air', runner: 'mac-runner', expiresAt: '2026-10-05T01:00:00Z', builtSha: null, acquiredAt: '', installedBuild: null }); + let patch: string | null = null; + const lease = (session: string): RemoteLease => ({ backend: 'remote', provider: 'github-actions', platform: 'ios', session, providerRef: '1', baseUrl: 'https://x.trycloudflare.com', tokenFile: join(dir, 'token'), deviceId: 'UDID', deviceName: 'iPhone Air', runner: 'mac-runner', expiresAt: '2026-10-05T01:00:00Z', builtSha: null, builtPatch: null, acquiredAt: '', installedBuild: null }); let sessions = 0; const inputsAskedFor: string[] = []; const remote = { @@ -31,9 +32,9 @@ function setup() { platform: 'ios', requirement: 'a pushed branch', availability: () => ({ usable: true, why: 'a device on a runner' }), - async sourceCommit(): Promise { + async sourceCommit(): Promise { if (!pushed) throw new VerifyFailure('BUILD_FAILED', 'GitHub does not have HEAD', 'git push, then rerun'); - return sha; + return { sha, patch: patch === null ? null : { sha256: patch, bytes: Buffer.from(patch) } }; }, reapable: async () => [], check: async () => health, @@ -43,8 +44,10 @@ function setup() { return { ...lease(`s${sessions}`), ...(request.runner === undefined ? {} : { runner: request.runner }) }; }, async install(held: RemoteLease, app: BuiltApp): Promise { - events.push(`install ${held.session} sha=${shaOf(app)?.slice(0, 1)}`); - return { ...held, builtSha: shaOf(app) }; + const sent = app.source === 'local' ? null : app.patch?.sha256 ?? null; + events.push(`install ${held.session} sha=${shaOf(app)?.slice(0, 1)}${sent === null ? '' : ` patch=${sent}`}`); + if (health === 'other-app') health = 'held'; + return { ...held, builtSha: shaOf(app), builtPatch: sent }; }, async release(held: RemoteLease) { events.push(`release ${held.session}`); @@ -74,11 +77,12 @@ function setup() { progress: (line: string) => void progress.push(line), instances: heldInstances({ finish: async (ledger) => (finished.push(ledger.worktree), []) }), }; - const edit = (text: string, newSha: string) => { + const edit = (text: string, newSha: string, uncommitted: string | null = null) => { writeFileSync(join(dir, 'app.swift'), text); sha = newSha.repeat(40); + patch = uncommitted; }; - return { deps, events, progress, edit, commit: (newSha: string) => (sha = newSha.repeat(40)), lose: () => (health = 'lost'), unpush: () => (pushed = false), stick: () => (stuck = true), finished, inputsAskedFor }; + return { deps, events, progress, edit, commit: (newSha: string) => (sha = newSha.repeat(40)), lose: () => (health = 'lost'), cutShort: () => (health = 'other-app'), unpush: () => (pushed = false), stick: () => (stuck = true), finished, inputsAskedFor }; } describe('lease flow with a backend that builds the app', () => { @@ -106,6 +110,41 @@ describe('lease flow with a backend that builds the app', () => { assert.equal((deps.workspace.readLease('ios') as RemoteLease).builtSha, 'b'.repeat(40)); }); + it('asks the session for the commit once the uncommitted changes it built are committed, though the app sources are the same', async () => { + const { deps, events, edit } = setup(); + await up(deps, { verb: 'up', waitSeconds: 0 }); + edit('two', 'a', 'p1'); + await up(deps, { verb: 'up', waitSeconds: 0 }); + edit('two', 'b'); + await up(deps, { verb: 'up', waitSeconds: 0 }); + await up(deps, { verb: 'up', waitSeconds: 0 }); + assert.deepEqual(events.slice(2), ['install s1 sha=a patch=p1', 'install s1 sha=b']); + const held = deps.workspace.readLease('ios') as RemoteLease; + assert.deepEqual([held.builtSha, held.builtPatch], ['b'.repeat(40), null]); + }); + + it('installs on the held session what the tree needs when the session holds a build the lease does not name, as after an install that was cut short', async () => { + const { deps, events, progress, cutShort } = setup(); + await up(deps, { verb: 'up', waitSeconds: 0 }); + cutShort(); + const result = await up(deps, { verb: 'up', waitSeconds: 0 }); + assert.deepEqual(events.slice(2), ['install s1 sha=a']); + assert.equal(result.leases[0]!.renewed, false); + assert.ok(progress.includes(`build ${result.builds[0]!.key} github-actions commit ${'a'.repeat(12)} the session builds it`), progress.join('\n')); + const held = deps.workspace.readLease('ios') as RemoteLease; + assert.deepEqual([held.session, held.builtSha, held.builtPatch], ['s1', 'a'.repeat(40), null]); + }); + + it('says that the build with uncommitted changes is already on the session when it installs nothing', async () => { + const { deps, events, progress, edit } = setup(); + await up(deps, { verb: 'up', waitSeconds: 0 }); + edit('two', 'a', 'p1'); + await up(deps, { verb: 'up', waitSeconds: 0 }); + const result = await up(deps, { verb: 'up', waitSeconds: 0 }); + assert.deepEqual(events.slice(2), ['install s1 sha=a patch=p1']); + assert.equal(progress.filter((line) => line.startsWith('build ')).at(-1), `build ${result.builds[0]!.key} github-actions commit ${'a'.repeat(12)} with uncommitted changes p1 already on the session`); + }); + it('leases a new session when the held one has stopped itself', async () => { const { deps, events, progress, lose } = setup(); await up(deps, { verb: 'up', waitSeconds: 0 }); diff --git a/integration/expo-native/test/run-groups.test.ts b/integration/expo-native/test/run-groups.test.ts index d23e7ebfbf8..f5b8c9e6795 100644 --- a/integration/expo-native/test/run-groups.test.ts +++ b/integration/expo-native/test/run-groups.test.ts @@ -11,9 +11,9 @@ import type { Runner } from '../src/core/exec.ts'; import { createInstances } from '../src/core/instances/instances.ts'; import { SettingsRefused, settingsFileOf } from '../src/core/instances/settings.ts'; import { summarize } from '../src/core/publish.ts'; -import { runVerb, type Deps } from '../src/core/verbs.ts'; +import { attach, runVerb, type Deps } from '../src/core/verbs.ts'; import { openWorkspace } from '../src/core/workspace.ts'; -import { VerifyFailure, type BuildKey, type Command, type DeviceBackend, type EvidencePath, type HostAdapter, type InstanceSettings, type LocalLease, type ScratchPath } from '../src/core/types.ts'; +import { VerifyFailure, type BuildKey, type BuiltApp, type Command, type DeviceBackend, type EvidencePath, type HostAdapter, type InstanceSettings, type LocalLease, type RemoteLease, type ScratchPath } from '../src/core/types.ts'; import { PLATFORM_KEY, fakeClerk, type FakeClerkOptions } from '../testing/fake-clerk.ts'; import { LEFT_OUT, linesLeftOut, watchDriver } from '../src/core/driver.ts'; import { readAgent } from '../specs/support/agent.ts'; @@ -146,7 +146,9 @@ interface FakeE2EPlan { const SPEC_SOURCE = "import { test } from '../../fixtures.ts';\n\ntest('x', async ({ host }) => {\n await host.seedUser();\n});\n"; const settingsText = (declared: InstanceSettings): string => `${JSON.stringify(declared, null, 2)}\n`; -function world(options: { readonly specs?: Readonly>; readonly plan?: FakeE2EPlan; readonly env?: Record; readonly clerk?: FakeClerkOptions; readonly logs?: () => Promise } = {}) { +const SESSION_COMMIT = 'a'.repeat(40); + +function world(options: { readonly specs?: Readonly>; readonly plan?: FakeE2EPlan; readonly env?: Record; readonly clerk?: FakeClerkOptions; readonly logs?: () => Promise; readonly session?: { readonly patch: string | null } } = {}) { const dir = mkdtempSync(join(tmpdir(), 'verify-run-')); execFileSync('git', ['init', '-q'], { cwd: dir }); writeFileSync(join(dir, 'app.swift'), 'app'); @@ -186,10 +188,24 @@ function world(options: { readonly specs?: Readonly ''), describe: (lease: LocalLease) => lease.deviceName, } as unknown as DeviceBackend; + const patch = options.session?.patch ?? null; + const session = { + kind: 'remote', + platform: 'ios', + availability: () => ({ usable: true, why: 'test' }), + reapable: async () => [], + check: async () => 'held', + sourceCommit: async () => ({ sha: SESSION_COMMIT, patch: patch === null ? null : { sha256: patch, bytes: Buffer.from('uncommitted') } }), + acquire: async (): Promise => ({ backend: 'remote', provider: 'github-actions', platform: 'ios', session: 'ios0000aa', providerRef: '1', baseUrl: 'https://quick-fox.trycloudflare.com', tokenFile: join(dir, 'token'), deviceId: 'UDID-1', deviceName: 'iPhone Air', runner: 'xcode-27', expiresAt: '2099-01-01T00:00:00Z', builtSha: null, builtPatch: null, acquiredAt: '', installedBuild: null }), + install: async (lease: RemoteLease, app: BuiltApp) => ({ ...lease, builtSha: app.source === 'local' ? null : app.sourceSha, builtPatch: app.source === 'local' ? null : (app.patch?.sha256 ?? null) }), + release: async () => undefined, + logs: async () => '', + describe: (lease: RemoteLease) => lease.deviceName, + } as unknown as DeviceBackend; const host = { repo: 'clerk-ios', platforms: ['ios'], - backends: [backend], + backends: [options.session === undefined ? backend : session], appId: () => 'com.clerk.E2EHost', buildInputs: () => ['app.swift'], async build(platform: 'ios', key: BuildKey, into: ScratchPath) { @@ -300,6 +316,14 @@ describe('a run whose spec files declare different settings', () => { assert.deepEqual(r.videoLines(), ['video the app could not be closed before the recording, so the video may open on the screen it was showing: DEVICE_IN_USE']); }); + it('runs agent-device in its state folder, so a file it writes to its working folder stays out of the checkout', async () => { + const r = recorded({ launches: true }); + const folders: (string | undefined)[] = []; + const watching: Runner = async (command, args, options) => (command.endsWith('agent-device') && folders.push(options?.cwd), r.deps.runner(command, args, options)); + await runVerb({ ...r.deps, runner: watching }, { ...RUN_ALL, video: true }); + assert.deepEqual(folders, [r.w.workspace.agentDeviceDir]); + }); + it('gives the close before the recording 30 seconds, and records all the same when it runs out', async () => { const r = recorded({ launches: true }); const limits: (number | undefined)[] = []; @@ -779,3 +803,28 @@ describe('the driver logs in the evidence of a run', () => { assert.deepEqual(record.tainted, []); }); }); + +describe('a run on a session that built uncommitted changes', () => { + const PATCH = 'c'.repeat(64); + + it('names the changes the session built in the run.json it writes, and names none for a session that built the commit', async () => { + const remoteOf = async (patch: string | null) => { + const { dir } = await runVerb(world({ specs: { [AUTH_START]: null }, session: { patch } }).deps(), RUN_ALL); + return (JSON.parse(readFileSync(join(dir, 'run.json'), 'utf8')) as { remote: unknown }).remote; + }; + assert.deepEqual(await remoteOf(PATCH), { provider: 'github-actions', runner: 'xcode-27', builtSha: SESSION_COMMIT, builtPatch: PATCH }); + assert.deepEqual(await remoteOf(null), { provider: 'github-actions', runner: 'xcode-27', builtSha: SESSION_COMMIT, builtPatch: null }); + }); + + it('is refused by attach from what run.json says, before attach runs anything', async () => { + const w = world({ specs: { [AUTH_START]: null }, session: { patch: PATCH } }); + const { record } = await runVerb(w.deps(), RUN_ALL); + const ran: string[] = []; + const deps = { ...w.deps(), runner: async (command: string) => (ran.push(command), { code: 0, stdout: '', stderr: '' }) }; + await assert.rejects(attach(deps, { verb: 'attach', run: record.run, pr: 9, media: { screenshots: ['home'], video: false }, before: null }), { + code: 'EVIDENCE_UNSAFE', + message: `run ${record.run} ran on a session build of ${SESSION_COMMIT.slice(0, 12)} with uncommitted changes (${PATCH.slice(0, 12)}), so no commit holds the app it shows`, + }); + assert.deepEqual(ran, []); + }); +}); diff --git a/integration/expo-native/test/session-agent.test.ts b/integration/expo-native/test/session-agent.test.ts index 36d9881de5f..9c925b8f279 100644 --- a/integration/expo-native/test/session-agent.test.ts +++ b/integration/expo-native/test/session-agent.test.ts @@ -2,7 +2,7 @@ import '../testing/git-env.ts'; import assert from 'node:assert/strict'; import { execFileSync, spawn, type ChildProcess } from 'node:child_process'; import { createHash } from 'node:crypto'; -import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, writeFileSync } from 'node:fs'; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; import http from 'node:http'; import net from 'node:net'; import { tmpdir } from 'node:os'; @@ -10,9 +10,11 @@ import { join } from 'node:path'; import { after, describe, it } from 'node:test'; import { coreVersion } from '../src/core/manifest.ts'; import { HANDOFF_LIMITS, parseHandoffManifest, type HandoffManifest } from '../src/core/remote/handoff.ts'; +import { workingPatch } from '../src/core/remote/backend.ts'; import { sendEvidence } from '../src/core/remote/session.ts'; import { plan } from '../src/core/remote/session-agent.ts'; -import { sha256Hex, type SessionHealth, type SessionRequest } from '../src/core/remote/protocol.ts'; +import { BUILD_LOG_START, sha256Hex, type SessionHealth, type SessionRequest } from '../src/core/remote/protocol.ts'; +import type { TreePatch } from '../src/core/types.ts'; const here = import.meta.dirname; const token = 'b'.repeat(64); @@ -89,7 +91,9 @@ async function gitHubWith(repo: { readonly sha: string; comesBefore(earlier: str const path = (req.url ?? '').replace(/^\/repos\/clerk\/sample/, ''); const [, base, head] = /^\/compare\/([0-9a-f]{40})\.\.\.([0-9a-f]{40})$/.exec(path) ?? []; const found = - path === '/git/ref/heads/feature/change' + path === '' + ? { default_branch: 'main' } + : path === '/git/ref/heads/feature/change' ? { object: { sha: branchHead } } : base !== undefined && head !== undefined ? { status: base === head ? 'identical' : comesBefore(base, head) ? 'ahead' : comesBefore(head, base) ? 'behind' : 'diverged' } @@ -103,7 +107,7 @@ async function gitHubWith(repo: { readonly sha: string; comesBefore(earlier: str authorizations, push: (sha: string) => (branchHead = sha), stop: () => new Promise((stopped) => server.close(stopped)), - env: { GITHUB_API_URL: `http://127.0.0.1:${port}`, GITHUB_REPOSITORY: 'clerk/sample', GITHUB_SHA: repo.sha, GITHUB_REF: 'refs/heads/feature/change', GITHUB_TOKEN }, + env: { GITHUB_API_URL: `http://127.0.0.1:${port}`, GITHUB_REPOSITORY: 'clerk/sample', GITHUB_SHA: repo.sha, GITHUB_REF: 'refs/heads/feature/change', GITHUB_TOKEN, VERIFY_SESSION_DEFAULT_BRANCH: 'main' }, }; } @@ -297,6 +301,168 @@ describe('session agent', () => { await agent.exited; }); + it('builds the changes it is sent on a clean checkout of the commit, never on changes it was sent before, and refuses changes that do not apply', async () => { + const repo = repoWithCommit(); + const github = await gitHubWith(repo); + const agent = await startAgent({ ...base, sha: repo.sha }, { cwd: repo.dir, env: github.env }); + const settled = (patch: string | null) => until(async () => { const b = (await agent.health()).build; return b.state !== 'building' && b.state !== 'none' && b.patch === patch ? b : null; }, `the build with ${patch ?? 'no changes'}`); + const send = (patch: TreePatch | null, sha256 = patch?.sha256) => + agent.call(`/__sim/build?sha=${repo.sha}${patch === null ? '' : `&patch=${sha256}`}`, { method: 'POST', ...(patch === null ? {} : { body: JSON.stringify({ patch: patch.bytes.toString('base64') }) }) }); + const checkout = (file: string) => (existsSync(join(repo.dir, file)) ? readFileSync(join(repo.dir, file), 'utf8') : null); + assert.equal((await settled(null)).state, 'built'); + + const client = mkdtempSync(join(tmpdir(), 'verify-agent-client-')); + execFileSync('git', ['clone', '-q', repo.dir, client]); + writeFileSync(join(client, 'app.txt'), 'edited'); + writeFileSync(join(client, 'extra.txt'), 'added'); + const first = workingPatch(client, ['.'])!; + assert.equal(existsSync(join(agent.work, 'patched')), false, 'a build of the commit leaves no mark'); + assert.equal((await send(first)).status, 202); + assert.equal((await settled(first.sha256)).state, 'built'); + assert.deepEqual([readFileSync(join(agent.work, 'built'), 'utf8'), checkout('extra.txt')], ['edited', 'added']); + assert.equal(readFileSync(join(agent.work, 'patched'), 'utf8'), `${first.sha256}\n`, 'the mark that keeps the job from storing what it built'); + + rmSync(join(client, 'extra.txt')); + writeFileSync(join(client, 'app.txt'), 'edited again'); + const second = workingPatch(client, ['.'])!; + assert.equal((await send(second, 'd'.repeat(64))).status, 400, 'a patch that does not hash to what was sent with it'); + assert.equal((await send(second)).status, 202); + const rebuilt = await settled(second.sha256); + assert.equal(rebuilt.state === 'built' && rebuilt.incremental, true); + assert.deepEqual([readFileSync(join(agent.work, 'built'), 'utf8'), checkout('extra.txt')], ['edited again', null], 'the file the first changes added is gone'); + + execFileSync('git', ['-c', 'user.name=t', '-c', 'user.email=t@example.com', 'commit', '-q', '-am', 'not pushed'], { cwd: client }); + writeFileSync(join(client, 'app.txt'), 'edited on a commit the session does not have'); + const elsewhere = workingPatch(client, ['.'])!; + await send(elsewhere); + const refused = await settled(elsewhere.sha256); + assert.equal(refused.state, 'failed'); + assert.match(refused.state === 'failed' ? refused.tail : '', new RegExp(`the uncommitted changes ${elsewhere.sha256.slice(0, 12)} do not apply to ${repo.sha.slice(0, 12)}\\. Commit and push them, then run again\\.$`)); + assert.deepEqual([repo.git('status', '--porcelain'), checkout('app.txt')], ['', 'one'], 'the checkout is the commit, with nothing half applied'); + + await send(null); + assert.equal((await settled(null)).state, 'built'); + assert.equal(readFileSync(join(agent.work, 'built'), 'utf8'), 'one'); + assert.ok(existsSync(join(agent.work, 'patched')), 'a later build of the commit does not take the mark away'); + await agent.call('/__sim/stop', { method: 'POST' }); + await agent.exited; + }); + + it('marks the job as soon as it accepts changes, before any of the work that can fail', async () => { + const repo = repoWithCommit(); + const github = await gitHubWith(repo); + const agent = await startAgent({ ...base, sha: repo.sha }, { cwd: repo.dir, env: github.env }); + await until(async () => ((await agent.health()).build.state === 'built' ? true : null), 'the build of the commit'); + const client = mkdtempSync(join(tmpdir(), 'verify-agent-client-')); + execFileSync('git', ['clone', '-q', repo.dir, client]); + writeFileSync(join(client, 'app.txt'), 'edited'); + execFileSync('git', ['-c', 'user.name=t', '-c', 'user.email=t@example.com', 'commit', '-q', '-am', 'not pushed'], { cwd: client }); + writeFileSync(join(client, 'app.txt'), 'edited on a commit the session does not have'); + const changes = workingPatch(client, ['.'])!; + const sent = await agent.call(`/__sim/build?sha=${repo.sha}&patch=${changes.sha256}`, { method: 'POST', body: JSON.stringify({ patch: changes.bytes.toString('base64') }) }); + assert.equal(sent.status, 202); + assert.equal(readFileSync(join(agent.work, 'patched'), 'utf8'), `${changes.sha256}\n`, 'marked when the agent answered, before the checkout'); + const failed = await until(async () => { const b = (await agent.health()).build; return b.state === 'failed' && b.patch === changes.sha256 ? b : null; }, 'the changes to fail'); + assert.match(failed.state === 'failed' ? failed.tail : '', /do not apply/); + assert.equal(readFileSync(join(agent.work, 'patched'), 'utf8'), `${changes.sha256}\n`, 'changes that did not apply keep the mark'); + await agent.call('/__sim/stop', { method: 'POST' }); + await agent.exited; + }); + + it('builds no uncommitted changes in a session on the default branch, or in one that does not know its branch or the default branch', async () => { + const repo = repoWithCommit(); + const github = await gitHubWith(repo); + const client = mkdtempSync(join(tmpdir(), 'verify-agent-client-')); + execFileSync('git', ['clone', '-q', repo.dir, client]); + writeFileSync(join(client, 'app.txt'), 'edited'); + const changes = workingPatch(client, ['.'])!; + const onMain = 'this session runs on main, the default branch, so it builds no uncommitted changes. Work on a branch: commit to a new branch and push it, then run down and up from that branch.'; + const refusals = [ + [{ GITHUB_REF: 'refs/heads/main' }, onMain], + [{ GITHUB_REF: 'refs/heads/main', VERIFY_SESSION_DEFAULT_BRANCH: '' }, onMain], + [{ VERIFY_SESSION_DEFAULT_BRANCH: '', GITHUB_REPOSITORY: 'clerk/unknown' }, 'this session could not learn the default branch of the repository, so it builds no uncommitted changes. Commit and push them to a branch, then run again.'], + [{ GITHUB_REF: '' }, 'this session does not know the branch it was started on, so it builds no uncommitted changes. Commit and push them to a branch, then run again.'], + [{ GITHUB_REF: 'refs/tags/v1' }, 'this session was started on refs/tags/v1, which is not a branch, so it builds no uncommitted changes. Commit and push them to a branch, then run again.'], + ] as const; + for (const [env, error] of refusals) { + const agent = await startAgent({ ...base, sha: repo.sha }, { cwd: repo.dir, env: { ...github.env, ...env } }); + await until(async () => ((await agent.health()).build.state === 'built' ? true : null), 'the build of the commit'); + const sent = await agent.call(`/__sim/build?sha=${repo.sha}&patch=${changes.sha256}`, { method: 'POST', body: JSON.stringify({ patch: changes.bytes.toString('base64') }) }); + assert.deepEqual([sent.status, await sent.json()], [409, { error }]); + const build = (await agent.health()).build; + assert.deepEqual([build.state, build.state === 'built' && build.patch, readdirSync(agent.work).filter((name) => name.startsWith('patch-'))], ['built', null, []], 'the changes were not kept or built'); + assert.equal((await agent.call(`/__sim/build?sha=${repo.sha}`, { method: 'POST' })).status, 202, 'the commit itself still builds'); + await agent.call('/__sim/stop', { method: 'POST' }); + await agent.exited; + } + }); + + it('refuses changes to anything but the app sources, by what git would write, before it applies any of them', async () => { + const repo = repoWithCommit(); + writeFileSync(join(repo.dir, 'outside.txt'), 'kept'); + const started = repo.commit('two'); + const github = await gitHubWith({ ...repo, sha: started }); + const agent = await startAgent({ ...base, sha: started }, { cwd: repo.dir, env: github.env }); + const settled = (patch: string | null) => until(async () => { const b = (await agent.health()).build; return b.state !== 'building' && b.state !== 'none' && b.patch === patch ? b : null; }, `the build with ${patch ?? 'no changes'}`); + const send = (patch: TreePatch) => agent.call(`/__sim/build?sha=${started}&patch=${patch.sha256}`, { method: 'POST', body: JSON.stringify({ patch: patch.bytes.toString('base64') }) }); + assert.equal((await settled(null)).state, 'built'); + + const client = mkdtempSync(join(tmpdir(), 'verify-agent-client-')); + execFileSync('git', ['clone', '-q', repo.dir, client]); + writeFileSync(join(client, 'app.txt'), 'edited'); + mkdirSync(join(client, 'e2e-tests')); + writeFileSync(join(client, 'e2e-tests', 'recipe.ts'), 'a new file outside the app sources'); + const added = workingPatch(client, ['.'])!; + rmSync(join(client, 'e2e-tests'), { recursive: true }); + execFileSync('git', ['checkout', '-q', '--', 'app.txt'], { cwd: client }); + execFileSync('git', ['mv', 'outside.txt', 'extra.txt'], { cwd: client }); + const renamedBytes = execFileSync('git', ['diff', '--cached', '--binary', '-M', 'HEAD'], { cwd: client }); + const renamed: TreePatch = { sha256: createHash('sha256').update(renamedBytes).digest('hex'), bytes: renamedBytes }; + assert.match(renamedBytes.toString(), /^rename from outside\.txt$/m, 'git writes the rename source without the patch naming it as a file'); + + for (const [patch, path] of [[added, 'e2e-tests/recipe.ts'], [renamed, 'outside.txt']] as const) { + assert.equal((await send(patch)).status, 202); + const refused = await settled(patch.sha256); + assert.equal(refused.state, 'failed', path); + assert.equal(refused.state === 'failed' ? refused.tail.split('\n').at(-1) : '', `the uncommitted changes ${patch.sha256.slice(0, 12)} change ${path}, and a session builds changes only to app.txt, extra.txt, Sources. Commit and push that change, then run again.`); + assert.deepEqual([repo.git('status', '--porcelain', '--ignored'), readFileSync(join(repo.dir, 'app.txt'), 'utf8'), readFileSync(join(agent.work, 'built'), 'utf8')], ['', 'two', 'two'], `nothing of the changes to ${path} was applied or built`); + } + await agent.call('/__sim/stop', { method: 'POST' }); + await agent.exited; + }); + + it('returns the build log from the byte it is asked for, and all of it when that byte is past the end', async () => { + const repo = repoWithCommit(); + const github = await gitHubWith(repo); + const agent = await startAgent({ ...base, sha: repo.sha }, { cwd: repo.dir, env: github.env }); + await until(async () => ((await agent.health()).build.state === 'built' ? true : null), 'the build'); + const log = readFileSync(join(agent.work, 'build.log')); + assert.match(log.toString(), /^\n\$ git fetch --no-tags --depth=1 origin /); + const read = async (from: number) => { + const answer = await agent.call(`/__sim/build/log?from=${from}`); + return [answer.status, answer.headers.get(BUILD_LOG_START), Buffer.from(await answer.arrayBuffer()).toString()]; + }; + assert.deepEqual(await read(0), [200, '0', log.toString()]); + assert.deepEqual(await read(10), [200, '10', log.subarray(10).toString()]); + assert.deepEqual(await read(log.length), [200, String(log.length), '']); + assert.deepEqual(await read(log.length + 5), [200, '0', log.toString()], 'a log that was started again since the last read'); + await agent.call('/__sim/stop', { method: 'POST' }); + await agent.exited; + }); + + it('keeps serving when the build log cannot be read', async () => { + const repo = repoWithCommit(); + const github = await gitHubWith(repo); + const agent = await startAgent({ ...base, sha: repo.sha }, { cwd: repo.dir, env: github.env }); + await until(async () => ((await agent.health()).build.state === 'built' ? true : null), 'the build'); + rmSync(join(agent.work, 'build.log')); + mkdirSync(join(agent.work, 'build.log')); + await agent.call('/__sim/build/log?from=0').then((answer) => answer.arrayBuffer()).catch(() => null); + assert.equal((await agent.call('/__sim/health')).status, 200, 'the agent still answers after a failed read of the log'); + await agent.call('/__sim/stop', { method: 'POST' }); + await agent.exited; + }); + it('builds only the commit it was started on and later commits of that branch, whoever asks with the session token', async () => { const repo = repoWithCommit(); const earlier = repo.sha; diff --git a/integration/expo-native/testing/fake-session-device.ts b/integration/expo-native/testing/fake-session-device.ts index eedb0df7e41..ad0ba23e06b 100644 --- a/integration/expo-native/testing/fake-session-device.ts +++ b/integration/expo-native/testing/fake-session-device.ts @@ -5,6 +5,7 @@ const node = process.execPath; const write = (file: string, text: string) => ({ command: node, args: ['-e', `require('fs').writeFileSync(${JSON.stringify(file)}, ${JSON.stringify(text)})`] }); const fake: SessionDeviceFactory = ({ id: deviceId, platform }) => ({ + inputs: ['app.txt', 'extra.txt', 'Sources'], build: (work) => [{ command: node, args: ['-e', `require('fs').writeFileSync(${JSON.stringify(join(work, 'built'))}, require('fs').readFileSync('app.txt', 'utf8'))`] }], record: process.env.FAKE_DEVICE_RECORDER === 'on-device' diff --git a/scripts/verify-attach.mjs b/scripts/verify-attach.mjs index 5763f64f2cf..ba389dd0a39 100644 --- a/scripts/verify-attach.mjs +++ b/scripts/verify-attach.mjs @@ -556,7 +556,7 @@ function withEvidenceBlock(body, platform, lines) { return { ok: false, why: 'has the end marker before the start marker' }; } const old = body.slice(starts[0] + start.length, ends[0]); - if (!A_BLOCK_STARTS.test(old.trimStart())) { + if (old.trim() !== '' && !A_BLOCK_STARTS.test(old.trimStart())) { return { ok: false, why: 'has text between the markers that is not an evidence block' }; } return { diff --git a/scripts/verify-attach.test.mjs b/scripts/verify-attach.test.mjs index a734c4ca961..6f8262ba85b 100644 --- a/scripts/verify-attach.test.mjs +++ b/scripts/verify-attach.test.mjs @@ -909,6 +909,21 @@ describe('the evidence block of a pull request description', () => { }); }); + it('writes the block in place of an empty pair of markers, so a description may say where the block goes', () => { + const { start, end } = evidenceMarkers('ios'); + assert.deepEqual(withEvidenceBlock(`## Verification\n\n${start}\n\n${end}\n\n## Notes\n`, 'ios', block), { + ok: true, + body: `## Verification\n\n${block}\n\n## Notes\n`, + was: 'replaced', + old: '\n\n', + }); + assert.equal( + withEvidenceBlock(`intro\r\n${start}\r\n \t\r\n${end}\r\noutro`, 'ios', block).body, + `intro\r\n${block.replaceAll('\n', '\r\n')}\r\noutro`, + ); + assert.equal(withEvidenceBlock(`${start}\n${end}`, 'ios', block).body, block); + }); + it('takes no marker from inside a code fence, so a description may show a whole example block', () => { const example = blockOf(manifestFor(MEDIA, { run: 'r20261007-010101-aaaa' })); for (const [open, close] of [ @@ -2049,6 +2064,17 @@ describe('the step that edits the description', () => { assert.equal(fake.state.writtenBody, null); }); + it('publishes into an empty pair of markers and keeps the text around them', async () => { + const env = await fetched(); + const fake = fakeGh({ + ...PULL, + body: '## Verification\n\n\n\n\n\n## Notes\n', + }); + assert.equal(publish(env, fake).outcome, 'published'); + assert.equal(fake.state.writtenBody, `## Verification\n\n${PLACED}\n\n## Notes\n`); + assert.equal(fake.state.laidOutBody, `## Verification\n\n${laidOut(SAID, ...AFTER_ONLY)}\n\n## Notes\n`); + }); + it('refuses files that changed after the first step wrote them', async () => { const env = await fetched(); fs.rmSync(path.join(env.WORK, 'files', 'profile.png'));