Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
9fe9532
perf(android): measure target readiness and reuse helper app control
thymikee Oct 19, 2026
7620d4f
fix(android): retain lifecycle fences and verified launch status
thymikee Oct 19, 2026
dad1e5a
fix(android): preserve launch output across helper transport
thymikee Oct 19, 2026
deedd1c
fix(android): keep reconciled targets in their application
thymikee Oct 19, 2026
7d5e436
fix(android): preserve launcher intent semantics
thymikee Oct 19, 2026
8bd3471
refactor(android): isolate launch and observation reconciliation
thymikee Oct 19, 2026
950e61f
test(android): scroll checkout replay to its submitted summary
thymikee Oct 19, 2026
dac9687
docs(evidence): record reliable Android readiness gains
thymikee Oct 19, 2026
3fa8b4a
docs(evidence): record Android startup safety experiments
thymikee Oct 19, 2026
6d5aed3
docs(evidence): clarify checksummed archive file count
thymikee Oct 19, 2026
9f542eb
fix(android): confirm moved empty fields by captured identity
thymikee Oct 19, 2026
85bcad7
fix(android): require complete fresh captures for empty-target recovery
thymikee Oct 19, 2026
7efc384
fix(android): anchor clear verification to unique pre-action identity
thymikee Oct 19, 2026
6a90ce7
docs(evidence): record edited-field experiments and guarded clear ver…
thymikee Oct 19, 2026
d3c22da
refactor(android): separate empty-target corroboration from point sca…
thymikee Oct 19, 2026
01a9562
docs(evidence): pin post-review verifier validation
thymikee Oct 19, 2026
ffb7ed1
test(android): cover helper app admission and launch protocol
thymikee Oct 19, 2026
b8a692d
fix(android): normalize package before helper launch admission
thymikee Oct 19, 2026
745530d
test(android): cover moved-clear proof through native fill
thymikee Oct 19, 2026
8879798
test(android): isolate moved-clear helper installation probe
thymikee Oct 19, 2026
c2fddc1
fix(android): keep helper admission within native package grammar
thymikee Oct 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions android/snapshot-helper/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,3 +162,7 @@ The release manifest is a stable provider contract for the current helper protoc
resolve the APK from `apkUrl`, verify `sha256`, install it with `adb install -r <apk>`, and run
`instrumentationRunner`. Manifests up to 0.21.3 also carried an `installArgs` array; it only ever
spelled `install -r` (plus `-t` while the helper was `testOnly`), so consumers can ignore it.

Capture metadata reports `cacheInvalidated` when the capture cleared the platform node cache through `UiAutomation.clearCache()`. A complete capture with that evidence need not change routes after a control press. Incomplete captures and captures without that evidence retain freshness recovery.

Warm application open commands use the same automation session. Close retains the existing foreground and process-absence checks. A failed application command is never resent.
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,13 @@ static void prepare(UiAutomation automation, long quietMs, long timeoutMs)
static Result capture(
UiAutomation automation, int maxDepth, int maxNodes, long stabilizationTimeoutMs)
throws InterruptedException, AccessibilityCaptureStabilizer.IncompleteCaptureException {
clearAccessibilityCache(automation);
return AccessibilityCaptureStabilizer.capture(
() -> captureOnce(automation, maxDepth, maxNodes), stabilizationTimeoutMs);
}

@SuppressWarnings("deprecation")
private static Result captureOnce(UiAutomation automation, int maxDepth, int maxNodes) {
boolean cacheInvalidated = clearAccessibilityCache(automation);
AccessibilityTreeXml.Stats stats = new AccessibilityTreeXml.Stats();
StringBuilder xml = new StringBuilder();
xml.append("<?xml version='1.0' encoding='UTF-8' standalone='yes' ?>");
Expand Down Expand Up @@ -87,27 +87,28 @@ private static Result captureOnce(UiAutomation automation, int maxDepth, int max
windowCount,
stats.nodeCount,
stats.truncated,
missingRootWindowTypes);
missingRootWindowTypes,
cacheInvalidated);
}

static void clearAccessibilityCache(UiAutomation automation) {
static boolean clearAccessibilityCache(UiAutomation automation) {
// Navigation 3 and any in-place content swap that keeps the same window/Activity render every
// destination inside one AndroidComposeView. A persistent helper session reuses a single
// UiAutomation connection across captures, and its per-connection accessibility node cache is
// not always invalidated by such a swap. Drop the cache before each traversal.
//
// Declared residue (agent-device #1832): the reset mechanism differs by API level — the public
// clearCache() on 34+, a setServiceInfo() re-apply below — and the two are not known to be
// equivalent. This is an acquisition-freshness difference the payload does not disclose.
// equivalent. Only the public cache reset establishes cacheInvalidated evidence.
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.UPSIDE_DOWN_CAKE) {
try {
automation.clearCache();
return;
return automation.clearCache();
} catch (RuntimeException ignored) {
// Fall back to the setServiceInfo() flush below if the platform rejects the public API.
}
}
clearAccessibilityCacheViaServiceInfo(automation);
return false;
}

private static void clearAccessibilityCacheViaServiceInfo(UiAutomation automation) {
Expand Down Expand Up @@ -269,6 +270,7 @@ static final class Result implements AccessibilityCaptureStabilizer.Capture {
final int windowCount;
final int nodeCount;
final boolean truncated;
final boolean cacheInvalidated;
// AccessibilityWindowInfo types of listed windows whose root read null or threw, so the host
// can tell a window it could not read (an input method, say) from one that was not on screen.
final List<Integer> missingRootWindowTypes;
Expand All @@ -281,14 +283,16 @@ static final class Result implements AccessibilityCaptureStabilizer.Capture {
int windowCount,
int nodeCount,
boolean truncated,
List<Integer> missingRootWindowTypes) {
List<Integer> missingRootWindowTypes,
boolean cacheInvalidated) {
this.xml = xml;
this.rootPresent = rootPresent;
this.foregroundWindowRootsPresent = foregroundWindowRootsPresent;
this.captureMode = captureMode;
this.windowCount = windowCount;
this.nodeCount = nodeCount;
this.truncated = truncated;
this.cacheInvalidated = cacheInvalidated;
this.missingRootWindowTypes = missingRootWindowTypes;
}

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
package com.callstack.agentdevice.snapshothelper;

import android.app.UiAutomation;
import android.os.Bundle;
import android.util.Base64;
import android.os.ParcelFileDescriptor;
import java.io.ByteArrayOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
import org.json.JSONObject;

final class ApplicationCommandHandler {
private ApplicationCommandHandler() {}

static void execute(UiAutomation automation, JSONObject input, Bundle result) throws Exception {
String packageName = input.getString("packageName");
if (!packageName.matches("[A-Za-z][A-Za-z0-9_]*(\\.[A-Za-z][A-Za-z0-9_]*)+")) {
throw new IllegalArgumentException("Invalid application package");
}
switch (input.getString("kind")) {
case "open":
open(automation, packageName, result);
return;
default:
throw new IllegalArgumentException("Unknown application operation");
}
}

private static void open(UiAutomation automation, String packageName, Bundle result) throws Exception {
String resolved = shell(automation, "cmd package resolve-activity --brief -a android.intent.action.MAIN -c android.intent.category.LAUNCHER " + packageName).trim();
String[] lines = resolved.split("\\n");
String component = lines[lines.length - 1].trim();
if (!component.matches(java.util.regex.Pattern.quote(packageName) + "/[A-Za-z_.$][A-Za-z0-9_.$]*")) {
result.putString("componentUnavailable", "true");
return;
}
String output = shell(automation, "am start -W -a android.intent.action.MAIN -c android.intent.category.DEFAULT -c android.intent.category.LAUNCHER -n '" + component + "'");
result.putString("launchOutputBase64", Base64.encodeToString(output.getBytes(StandardCharsets.UTF_8), Base64.NO_WRAP));
}

private static String shell(UiAutomation automation, String command) throws Exception {
ParcelFileDescriptor[] pipes = automation.executeShellCommandRw("sh");
if (pipes == null || pipes.length != 2) throw new IllegalStateException("Application shell command was unavailable");
try (InputStream stream = new ParcelFileDescriptor.AutoCloseInputStream(pipes[0]);
OutputStream input = new ParcelFileDescriptor.AutoCloseOutputStream(pipes[1]);
ByteArrayOutputStream bytes = new ByteArrayOutputStream()) {
input.write(("exec 2>&1\n" + command + "\n").getBytes(StandardCharsets.UTF_8));
input.close();
byte[] buffer = new byte[4096];
int count;
while ((count = stream.read(buffer)) != -1) {
if (bytes.size() + count > 65536) throw new IllegalStateException("Application command output exceeded its limit");
bytes.write(buffer, 0, count);
}
return bytes.toString(StandardCharsets.UTF_8.name());
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ static void writeSessionResponse(
Bundle result = SessionResponseWriter.sessionResponseBundle(requestId);
InputDispatch dispatch = new InputDispatch();
try {
if ("screenshot".equals(action)) {
if ("app".equals(action)) {
ApplicationCommandHandler.execute(automation, new JSONObject(new String(Base64.decode(payload, Base64.DEFAULT), StandardCharsets.UTF_8)), result);
} else if ("screenshot".equals(action)) {
screenshot(result, automation);
} else if ("focus".equals(action)) {
focus(result, automation);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ private static void putCaptureMetadata(
DisplayExtent.Extent display,
long elapsedMs) {
result.putString("rootPresent", Boolean.toString(capture.rootPresent));
result.putString("cacheInvalidated", Boolean.toString(capture.cacheInvalidated));
result.putString("captureMode", capture.captureMode);
result.putString("windowCount", Integer.toString(capture.windowCount));
result.putString(
Expand Down Expand Up @@ -180,7 +181,8 @@ private void runSnapshotSession(
putBaseMetadata(
ready, waitForIdleTimeoutMs, waitForIdleQuietMs, timeoutMs, maxDepth, maxNodes);
ready.putString("sessionReady", "true");
ready.putString("sessionCommands", "input,screenshot,focus,snapshot-immediate");
ready.putString("sessionCommands", "input,screenshot,focus,snapshot-immediate"
+ (android.os.Build.VERSION.SDK_INT >= 31 ? ",app" : ""));
ready.putString("sessionPort", Integer.toString(sessionPort));
sendStatus(2, ready);

Expand All @@ -202,7 +204,7 @@ private void runSnapshotSession(
SessionResponseWriter.writeSessionOk(socket.getOutputStream(), requestId);
return;
}
if ("input".equals(action) || "screenshot".equals(action) || "focus".equals(action)) {
if ("input".equals(action) || "screenshot".equals(action) || "focus".equals(action) || "app".equals(action)) {
AutomationCommandHandler.writeSessionResponse(
socket.getOutputStream(), requestId, action,
parts.length > 2 ? parts[2] : "", getConnectedUiAutomationUnchecked());
Expand Down
1 change: 1 addition & 0 deletions docs/adr/0011-interaction-guarantee-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ Completed process history is in git history.
- The Apple runner does not resend a mutating command whose first send may have run.
- Multiple mutating selector matches collapse only if they form one ancestor–descendant chain and every member resolves to the same actionable node. Otherwise, the mutation fails with `AMBIGUOUS_MATCH`. Maestro's explicit expected-point/non-hittable compatibility path stays separate.
- Keep synthesized iOS gesture policy outside the element dispatch-path matrix.
- On Android, the press/click moving-target guard samples fresh geometry until the target remains on one surface at one position for 100 ms, within a 1,500 ms transition window. A target that cannot establish that evidence is refused. Other platforms retain their existing guard schedule.

## Context

Expand Down
2 changes: 2 additions & 0 deletions docs/adr/0014-session-ref-frame-lifetime.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,8 @@ A ref-targeting mutation is admitted only when all of these conditions hold:

The target is resolved against the authorized frame, never against a newer operational observation by positional coincidence.
Android freshness captures may verify current identity or provide comparison evidence.
On Android, an identifier that names exactly one same-local-identity node in both the authorized frame and the observation can carry fresh geometry across a positional reindex when both nodes name the same application package.
This does not change the frame namespace or authorize a different identity.
But they must not redefine which node an already-admitted `@eN` authorizes.

Read-only ref consumers remain fail-open with a structured staleness warning only when the retained frame contains the evidence necessary to resolve that ref.
Expand Down
Loading
Loading