test(expo): post the evidence that shows the change, before and after

.changeset/expo-verify-evidence.md

@@ -0,0 +1,2 @@
+---
+---

.claude/skills/verify-clerk-expo/features/README.md

@@ -34,7 +34,7 @@ The iOS identifiers come from `Sources/ClerkKitUI/Components/Auth/ClerkAccessibi
 ## Proof and skip reporting
 
 - A proof is a passing `run` whose run directory holds `video.mp4`, `screenshots/`, `app.log`, and `e2e/report.json`. The video and the screenshots are the proof on both platforms.
-- Name the run id, the platform, and the specs in the PR. Attach the run with `attach <run-id> --pr <n>`.
+- Name the run id, the platform, and the specs in the PR. Attach the run with `attach <run-id> --pr <n>` and `--screenshot <label>` for a still change or `--video` for behavior or a flow. Add `--before <run-id>` when the change is to something that exists. `SKILL.md` has the choice under Evidence.
 - A spec limited to one platform reports as skipped on the other. Say which platform, and local or remote device, each proof ran on.
 
 ## Feature entry contract

.claude/skills/verify-clerk-expo/references/remote.md

@@ -76,9 +76,9 @@ Each checkout has a random id in `.verify/remote/owner`, and its sessions carry
 
 ## Evidence from a machine that cannot attach
 
-`attach` always tries `gh pr edit --attach` itself first, on every kind of machine, and hands off only when that cannot work. The hand-off sends the evidence of a remote run to the session's runner. It sends the video and the screenshots from the sealed run directory, with a manifest that names the run, the pull request, the device, the commit, and each file's size and SHA-256. The files travel over the session's tunnel, with the session's token, in pieces of 1 MiB. The session job keeps them in one directory and uploads that directory as the artifact `verify-evidence` when the job ends. The artifact holds nothing else of the session.
+`attach` always tries `gh pr edit --attach` itself first, on every kind of machine, and hands off only when that cannot work. The hand-off sends the evidence of a remote run to the session's runner. It sends the files that the flags of `attach` chose from the sealed run directory, with a manifest that names the run, the pull request, the device, the commit, and each file's size and SHA-256. With `--before`, the manifest also names the before run, its commit, and its counts, and the hand-off carries one file of the before run for each file of the run. Those files come from the before run's directory in this checkout, whichever session made that run. When that directory is gone, as it is in a new sandbox, `attach` fails with `USAGE` before it sends anything, and its fix is to attach without `--before` and name the before run in the PR. The files travel over the session's tunnel, with the session's token, in pieces of 1 MiB. The session job keeps them in one directory and uploads that directory as the artifact `verify-evidence` when the job ends. The artifact holds nothing else of the session.
 
-`.github/workflows/verify-attach.yml` then runs, from the default branch. It downloads the artifact, checks every file against the manifest and against its own limits, and puts the block of the run's platform in the description with `gh pr edit --attach`. It never runs anything from the branch or from the artifact. The workflow did not run the tests, so the line it writes says that the session reported the result, links the session's run in the Actions tab, and names the account that started that run, as GitHub records it.
+`.github/workflows/verify-attach.yml` then runs, from the default branch. It downloads the artifact, checks every file against the manifest and against its own limits, and puts the block of the run's platform in the description with `gh pr edit --attach`. It holds a file of the before run to the same checks as a file of the run. It writes the description twice, as `attach` does: the first write uploads the files as a list, and the second lays them out in the tables that `SKILL.md` describes under Evidence. For the second write it reads the addresses of the uploads from the block, and takes only addresses of GitHub uploads. If the pull request or the block changed between the two writes, it leaves the list and says so in its run. A rerun of that job uploads the files again and lays them out. The tables take about twice the characters of the list, so in a description near GitHub's limit of 65,536 characters the second write can fail after the first one succeeded. The job then fails, and the list stays until the description is shorter and the job runs again. It never runs anything from the branch or from the artifact. The workflow did not run the tests, so the line it writes says that the session reported the result, links the session's run in the Actions tab, and names the account that started that run, as GitHub records it. With a before run, the line says that the session reports that run, and it links the commit of that run so that a reader can open it.
 
 The repository needs this set up once:
 
@@ -98,15 +98,20 @@ The workflow publishes only when all of these hold. Otherwise it prints the reas
 - Its branch is the branch the session was started on.
 - The commit the run was made at is one of the pull request's commits.
 - The commit that branch had when `up` started the session is one of them too. GitHub lists the first 250 commits of a pull request, and the workflow reads no further. When a longer pull request has either commit past the first 250, the workflow says so and publishes nothing. Squash or rebase the branch to 250 commits or fewer, or attach from a machine whose `gh` can attach.
+- With `--before`, the commit of the before run is the commit of the run or a commit before it in its history. The workflow asks GitHub to compare the two commits.
 - The hand-off has at least one file.
 - The description has the two evidence comments of the platform once each, or neither.
 - A description without the comments does not end inside a code fence that is never closed.
 
 `attach` checks the same rules before it sends anything, and fails with the reason.
 
+A before run is the build without the change, so its commit is often the one the branch started from, which is no commit of the pull request. The workflow therefore does not look for that commit among the commits of the pull request. It asks GitHub whether that commit is the commit of the run or a commit before it in its history, and it publishes nothing when the answer is no. After a rebase that rewrote the commit of the before run, `run` the spec again on a commit that the branch still has, and pass that run as `--before`. The workflow cannot check that the files of the before run came from a build of that commit, which is why the line says that the session reports the before run.
+
+The manifest of a hand-off has a version, and the workflow publishes only the version of the script on the default branch. A branch whose verify code is older than the default branch's hands off a manifest that the workflow refuses. The refusal names both versions in the run of `verify-attach`, and nothing on the pull request shows it. Merge the default branch into the branch before you `attach`, then `down`, `up`, `run`, and `attach`. A session takes only the version of its own code. When the checkout changed version after `up`, `attach` fails, names both versions, and says to `down` and `up`.
+
 A push after the run does not lose the evidence. The line in the block names the commit the run was made at, and when the pull request has moved on it ends with "The pull request has newer commits." To show the newer commit, `run` again and `attach` again in the same session, which replaces the block. The session builds the new commit, and no new session is needed. After a rewrite of the branch's history that removes either commit, `down` and `up` before the next `run`.
 
-A hand-off holds one video of up to 100 MiB and screenshots of up to 10 MiB each, with at most 50 files and 150 MiB in all. A session keeps one hand-off, and a later `attach` in the same session replaces it. `down` gives a session that holds evidence up to five minutes to upload it before it cancels the run.
+A hand-off holds one video of up to 100 MiB and screenshots of up to 10 MiB each, with at most 50 files and 150 MiB in all. A before run adds one more video and as many screenshots as the run has, and its files count toward the 50 files and the 150 MiB. A session keeps one hand-off, and a later `attach` in the same session replaces it. `down` gives a session that holds evidence up to five minutes to upload it before it cancels the run.
 
 The runs of `verify-attach` for one branch take turns, and each one publishes.
 

integration/expo-native/specs/fixtures.ts

@@ -5,7 +5,7 @@ import { test as base } from '@e2e-dev/mobile';
 import { expect, type Locator } from 'e2e';
 import { app as testApp } from './app.ts';
 import { busyWaits, onceTheRunnerIsFree, type BusyWait } from './support/busy-runner.ts';
-import { testUsers } from './support/clerk.ts';
+import { aboutToLaunch, testUsers } from './support/clerk.ts';
 import { ASSERTION_TIMEOUT_MS, PACKAGE_DIR } from './support/config.ts';
 import { agentDevice, deviceCommand } from './support/device.ts';
 import { fillField } from './support/fill.ts';
@@ -74,6 +74,7 @@ export const test = base.extend<{ host: HostFixture }>({
       {
         installApp: (path) => device.installApp(path),
         signInTicket: users.signInTicket,
+        aboutToLaunch: () => aboutToLaunch(clerk.access),
         openApp: (id, options) => (options === undefined ? device.openApp(id) : device.openApp(id, { relaunch: true, launchArguments: [...options.launchArguments] })),
         adb: (args) => adb(target, args),
       },

integration/expo-native/specs/support/clerk.ts

@@ -16,6 +16,8 @@ export const BACKEND_CALLS = {
 
 export type BackendCall = keyof typeof BACKEND_CALLS;
 
+export const ABOUT_TO_LAUNCH = { method: 'POST', path: '/about_to_launch' } as const;
+
 export const TICKET_SECONDS = 120;
 
 // Node gives each address of a host 250 ms to connect before it tries the next, and fails the request when
@@ -90,6 +92,17 @@ function endpoint(access: ClerkAccess): { readonly baseUrl: string; readonly bea
   return { baseUrl: access.url, bearer: new Secret('stand-in-token', readFileSync(access.tokenFile, 'utf8')) };
 }
 
+export async function aboutToLaunch(access: ClerkAccess, fetchImpl: typeof fetch = fetch): Promise<void> {
+  if (access.kind !== 'stand-in') return;
+  const { baseUrl, bearer } = endpoint(access);
+  const response = await bearer.use('bapi-authorization', (plain) =>
+    fetchImpl(`${baseUrl}${ABOUT_TO_LAUNCH.path}`, { method: ABOUT_TO_LAUNCH.method, headers: { Authorization: `Bearer ${plain}` }, signal: AbortSignal.timeout(60_000) }),
+  );
+  if (response.ok) return;
+  const said = errorsOf({ status: response.status, json: await response.json().catch(() => null) }).map((error) => error.long_message ?? error.message ?? error.code);
+  throw new Error(`the verify CLI was not ready for the app to launch: ${said.join('; ') || `it answered ${response.status}`}`);
+}
+
 export function testUsers(clerk: ClerkInstance, run: RunId, fetchImpl: typeof fetch = fetch): TestUsers {
   runPassword(run);
   const handedOut = new Set<TestPhone>();

integration/expo-native/specs/support/launch.ts

@@ -115,6 +115,7 @@ export interface LaunchedApp {
 export interface LaunchDriver extends AppStartDriver {
   installApp(path: string): Promise<unknown>;
   signInTicket(user: SeededUser): Promise<SecretLike>;
+  aboutToLaunch(): Promise<void>;
 }
 
 export function appLauncher(app: LaunchedApp, driver: LaunchDriver, installedBuilds: Set<string>, newId: () => string): (options: LaunchOptions) => Promise<LaunchId> {
@@ -140,7 +141,9 @@ export function appLauncher(app: LaunchedApp, driver: LaunchDriver, installedBui
         ...(options.debugLogs === true ? { verifyLogLevel: 'debug' as const } : {}),
         ...(ticket === undefined ? {} : { verifySignInTicket: ticket }),
       });
-      return appStart(app.platform, app.id, app.entry, launchArguments);
+      const start = appStart(app.platform, app.id, app.entry, launchArguments);
+      await driver.aboutToLaunch();
+      return start;
     };
     await performAppStart(nextStart, app.id, driver);
     if (latest.launchId === undefined) throw new Error('the app start never minted a launch id');

integration/expo-native/src/core/MANIFEST

@@ -1,27 +1,27 @@
 0ec17d3e3a153410d40bd6cc57ddb259745c77403fa681c6670a3aa6456aa826  e2e.config.ts
-f1ac21eeca82c1f04cbb42f0b77d51c798c71842a5a0b97a4660c3265ab601b6  specs/fixtures.ts
+e563089e9dcaf8a1edda603b7816d313ca634db078289829e21f5db4d7f1ce3f  specs/fixtures.ts
 38e8e99663059810b616b591dfd79bdcc7781f72b7269875294f3702b3edde84  specs/support/agent.ts
 6d1ef7700a89e19d0b1214d5a6815ba757d02857c555b8bf108f2e0c9a64aad3  specs/support/busy-runner.ts
-b272dcd59155f0beff1eb1bc5ce7e3e2283d9e0e16e0639ef7eb6d8edbde9b3f  specs/support/clerk.ts
+153cda5ccf3ad0e38a35e238a52c64633a9b8de9e2086ff060ddb661a2438b1d  specs/support/clerk.ts
 6c658e254cb6ad8ba9f3d05638941e0809e6709f0abb8096340559b2f81b156a  specs/support/config.ts
 be45e0eb42f715d603c179c7a86750949ff2ee3cc0da73082179801fe1b8f2ab  specs/support/device.ts
 cec7a42bf8d35e174fd3387cbcc638c01bacda74fc15ea1b14eda8abaa200d34  specs/support/fill.ts
 7415ccd89f715d40ca74796a5b235aab02e640c8674d457d3912955ab67146e1  specs/support/inputs.ts
-1772127f0a42d10003f16e5a59197180f4731837e0a00d2d1717f0b0f1bdfe8b  specs/support/launch.ts
+27a69309a568e90dc06b8d04abfb6efe7dadd16f4374a19f61d0ebf0a99a87fc  specs/support/launch.ts
 2065bc90a33659fd8f5a6e54e8b0fa57a50edc415063d4ac7e0dab744972e1b7  specs/support/secret.ts
 7446b59792b957f2ace72e6c2eceb305bde25378d1f879e2884706c3d7f9f3a4  specs/support/tapping.ts
 df955a42a2bd4ca1780f56837ebb9c832c68e97f74a35764d42a32e44b28286b  specs/support/types.ts
 7bdb018b337d923e96ada265abfebafd75c6fd07188ab313117effdc893e32d7  specs/support/typing.ts
 10bcf31d87a05655fc263ed810a0e10c0b63a77265720d7f22f3720052f67c38  specs/support/waiting.ts
 792ec6414c7e95446346abe2db850f6e4cc865a7c83a21892166fae969654277  src/core/agent.ts
-1b144b8f7362be2019f578bbdf94b7e6fe3196440de5c79d4f790fa8394374e3  src/core/broker.ts
+d56c29efc8061adfd65b89f0f02465af3234ecd52289ef6d9f338483afeadeb5  src/core/broker.ts
 7c741449c755524bba80f1ce314554293003a34f7f7c30696015ba1c0c3dc1ab  src/core/claims.ts
 1fc59d13e7197e6c7099c0467744b978d498d1fe409c88254d44e2f99913a7b5  src/core/clerk.ts
-51f330273bb083139ff2f5f7ee3a7b616ba614c83cd050c188d5f224631050d6  src/core/cli.ts
+4b8f5d87af69b85cd0bd5e729de238ccdb39f42e54d2e38af9ca43c2f172a433  src/core/cli.ts
 4aeacea985f5ef80001e5695feec53182abb9695eb07130898568743755fc8af  src/core/devices.ts
 3ae461d6aa75a2cefac0392a9ad124aec4105ac25b9320371743e2ea90fe2202  src/core/driver.ts
 b59071aaefeec02dea89bd25615f8ada0da0a51e805a2523011b8b591e7435d3  src/core/e2e.ts
-8c0ae6c44155105ce15011176bc630c59d6b4a2d2aa050f3fad4aa9674386ad8  src/core/evidence.ts
+c9a29e27eb835d9c3fc9b0bd394ff55b5f1fcc2b2fc0d96da51197e5e7bca003  src/core/evidence.ts
 27bfd4ce59937aa8c97d9fd6849cf52f3da05626fbc9388385f4b5e684577d36  src/core/exec.ts
 e2075f17f52ded8dea64d8a2473820396ca5ac1a22db59f21a092da28e55911f  src/core/github-report.ts
 721453f519bba1374dcf11aaae58f7af06e5555be756df6d3beaf52a9c2a2eda  src/core/instances/base.json
@@ -34,19 +34,19 @@ cf6aa68be73e4abaf3b63d5d84c2e9eee6197412f607ec53d0f05f7b058db420  src/core/insta
 098d3eef9d2521054f235f47e8658efb71d482087b927645489fc2240a03f5e8  src/core/launch.mjs
 a0ee9e4bfd34d4659685108e5e54eb1ec2aff6248236c358e6d3b14b2e2ff6a0  src/core/ledgers.ts
 afab94a970286c40d6857b03ae4fa39ec59786274beacf6f1e77f6530bf46c54  src/core/manifest.ts
-c8f71088aa30904b8c69196f6df9bae968a5de16d2007d7e5e569feef3ff8246  src/core/publish.ts
-0f016cc412897b66038f56ce4c15b5b633bf8ca2c29eda05dbe28bf17fb29fd2  src/core/remote/backend.ts
+7294bbab3d892f0d9aee1e863f84e32a7f5dc3c467a7b6177be7e4ff2f29fecc  src/core/publish.ts
+76de0f98013b8a5d011dee8cf51cce63fcb3d7e68acfee51945fa5cecf2e7763  src/core/remote/backend.ts
 37fc42707be7d8bd5f0313a664a8465287ebb0970c520c0e0617f6ff6c9a614e  src/core/remote/github.ts
-2ed83b089a07969efe4369e027b8840f4c4b7b8fe055058fb22f44cf289618d6  src/core/remote/handoff.ts
+cbd837689f2e0eff48205709a8c99508a0bcd0bea76f2647a93faa8866a76ae0  src/core/remote/handoff.ts
 2ec436b2c53b34269e0b62ab54e41e4d9a55146cccc46f0f31591db05cdbf5c1  src/core/remote/preflight.ts
 bb69fe1f3581942285894c0901d02889c08b4293e4d9db77934fe3cdd2d4a399  src/core/remote/protocol.ts
 5a4ee0e7ed20feaeb711b1527083299b432d3fc0b7f428b636678a8a2f57581f  src/core/remote/recipe.ts
 bbb997d036c320257783f9c576497e4f02cde99a9cc8d164c4dfde40f76e4421  src/core/remote/session-agent.ts
-12df6d34398f8d1dce753839be00a0dc50f2c6de00638e4e4a43f8570912b403  src/core/remote/session.ts
+9cb8ac8bdae263362b9e8a3f24592f20cc80fb367417ac0d2ed34fb3f383021b  src/core/remote/session.ts
 d614ee5c98a85d15037b744a67404496273f62985fd7d6a049cacfd5ab335503  src/core/remote/settings.ts
 e734336ea3b345057b69193790e2efce9a662c67ec5b4a426c811f999b45ccdb  src/core/remote/tunnel.ts
 685eabf010c7f63d076cfd3da1a681b5cc8024720538c1bf3a99792b563d8b60  src/core/slot.ts
 a8838aaf1fa9e4a6bc4e350ea674079201da76c5962656039b96db2505591aed  src/core/state.ts
-5d9bd1c8120b1ae77a1c42c40248f3b73cb1fce4c61e8c78ae21778a8ec3b39d  src/core/types.ts
-e0be9be1adfbfeec510d86a97e75d4b18b032712b6653daf2674458051ae6254  src/core/verbs.ts
+c9900be2f1097ed0395018531bed554064440f57962afac1daa794040b1b2c28  src/core/types.ts
+79b822b663c1ac1748cf3c8ec390ef9204378f4c97531f4da9b0bf5736a20d36  src/core/verbs.ts
 9be1816773216373548ec9933978b53ace3d3bfdba17b72c1e5ed6d8aef80f62  src/core/workspace.ts

integration/expo-native/src/core/broker.ts

@@ -2,7 +2,7 @@ import { randomBytes, timingSafeEqual } from 'node:crypto';
 import { writeFileSync } from 'node:fs';
 import { createServer, type IncomingMessage, type ServerResponse } from 'node:http';
 import { join } from 'node:path';
-import { BACKEND_API_URL, BACKEND_CALLS, TICKET_SECONDS, parseTestEmail, parseTestPhone, runEmailPrefix, typedPassword, type BackendCall } from '../../specs/support/clerk.ts';
+import { ABOUT_TO_LAUNCH, BACKEND_API_URL, BACKEND_CALLS, TICKET_SECONDS, parseTestEmail, parseTestPhone, runEmailPrefix, typedPassword, type BackendCall } from '../../specs/support/clerk.ts';
 import { protect } from '../../specs/support/secret.ts';
 import { ownKeyRefused, userLimitReached } from './clerk.ts';
 import type { InstanceKeys } from './keys.ts';
@@ -16,6 +16,7 @@ const BODY_LIMIT_BYTES = 64 * 1024;
 export interface BrokerDeps {
   readonly keys: () => InstanceKeys;
   readonly fetch: typeof fetch;
+  readonly beforeLaunch: () => Promise<void>;
 }
 
 export interface Broker {
@@ -139,8 +140,18 @@ export async function startBroker(run: RunId, workspace: Workspace, scratch: Scr
     return { status: response.status, body: await response.text() };
   }
 
+  async function readyToLaunch(): Promise<Answer> {
+    try {
+      await deps.beforeLaunch();
+      return { status: 200, body: '{}' };
+    } catch (error) {
+      return error instanceof VerifyFailure ? failure(503, 'verify_not_ready_to_launch', error) : clerkError(503, 'verify_not_ready_to_launch', (error as Error).message ?? String(error));
+    }
+  }
+
   async function answer(request: IncomingMessage): Promise<Answer> {
     const url = new URL(request.url ?? '/', 'http://127.0.0.1');
+    if (request.method === ABOUT_TO_LAUNCH.method && url.pathname === `${STAND_IN_PATH}${ABOUT_TO_LAUNCH.path}`) return readyToLaunch();
     const guard = guardFor(request.method, url.pathname);
     if (guard === undefined) return clerkError(404, 'verify_stand_in_unknown_call', `the verify CLI's stand-in for the Backend API has no ${request.method} ${url.pathname}; the tests make ${Object.values(BACKEND_CALLS).map((call) => `${call.method} ${call.path}`).join(', ')}`);
     let text = '';