diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index 41ff35d..25089a5 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -49,8 +49,12 @@ jobs: - run: cargo fmt --all -- --check - run: cargo test --workspace --locked - run: npm test -- --run + - run: node --test resources/frida/tests/scope-engine.test.cjs - run: node --test scripts/release.test.mjs - run: python -m unittest discover -s resources/device/tests + - name: Test owned Android fixture host controls + if: runner.os == 'Linux' + run: python -m unittest discover -s examples/android-engine-fixture/tests - run: npm audit --audit-level=high - run: npm run package - name: Exercise real desktop engine and stdio MCP diff --git a/README.md b/README.md index 7375133..d94fb12 100644 --- a/README.md +++ b/README.md @@ -97,7 +97,7 @@ npm run desktop Discover connected devices and installed apps over ADB. Use the **device-wide proxy** path, or instrument a **selected process with Frida** and explicit hook profiles. Read-only diagnostics explain authorization, root, ABI, runtime/server versions, mappings and recovery state before you connect. -Frida is an optional, separate runtime installation. Pinning and native networking support depend on the target; selecting an app in proxy mode does not prove per-app attribution. **Stop** restores recorded Scope-owned state and preserves external changes. [Setup, supported paths and recovery →](docs/ANDROID.md) · [Diagnostic reference →](docs/DIAGNOSTICS.md) +Frida is an optional, separate runtime installation. The experimental Scope-owned engine supports selected Java URL, SSLContext and OkHttp3 seams; native TLS and custom pinners are unsupported. **Stop ends the instrumented app process** before restoring recorded Scope-owned state and preserving external changes. System proxy mode requires manually configured app trust for HTTPS and never installs a system CA. [Setup, supported paths and recovery →](docs/ANDROID.md) · [Diagnostic reference →](docs/DIAGNOSTICS.md) ## An agent can read the evidence. You set the scope. @@ -124,7 +124,7 @@ The same engine powers the native inspector and MCP. Real HTTP/TLS fixtures test | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | macOS arm64 | Local UI/package evidence is recorded by version; 1.3.0 interactive validation is recorded separately by scenario. Ad-hoc signed; not Apple-notarized. | | Windows x64 / Linux x64 | CI builds, packages and headless MCP/capture smoke pass. Interactive UI and device validation remain open. Intel macOS is unverified. | -| Android / Frida | Earlier smoke covered one rooted Android 16 arm64 emulator. The 1.0 diagnostics changes use fixtures; arbitrary apps and TLS stacks are unverified. | +| Android / Frida | Earlier smoke covered the previous hook backend on one emulator. The current Scope-owned engine has local simulated tests only; real Android/Frida/OkHttp TLS and Stop acceptance are NOT RUN. | | Capture | Bodies are buffered up to **16 MiB each**. Streaming, WebSocket frames, trailers and HTTP3 are not implemented. | | Archives & analysis | Records currently load into memory. Large-session streaming/export is planned. Observed schemas are not complete API contracts. | | Dependencies | A Linux glib unsoundness advisory remains tracked in [#11](https://github.com/dalkedev/scope/issues/11). See the full [dependency record](docs/DEPENDENCIES.md). | @@ -145,6 +145,6 @@ Every change goes through maintainer review and required platform checks. [Owner Scope logo and brand assets

-An independent project by **[dalkedev](https://github.com/dalkedev)**. Scope includes attributed HTTP Toolkit interception scripts and Frida integration; their original licenses and notices remain intact. HTTP Toolkit does not need to be installed. +An independent project by **[dalkedev](https://github.com/dalkedev)**. Scope uses its original Java interception engine with Frida and an unchanged, attributed Frida Java bridge. The bridge distribution provenance and third-party licenses remain in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md). **AGPL-3.0-or-later** · [License](LICENSE) · [Third-party notices](THIRD_PARTY_NOTICES.md) · [Brand assets](docs/BRAND.md) diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 8874a2c..ba7f37d 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -1,5 +1,3 @@ -Scope includes scripts from the user's SculptTechProject/bypass-scripts-adb repository. The notices below preserve that distribution's provenance. Scope's Rust device manager and Python worker are new integration code; they do not depend on an installed HTTP Toolkit instance. The original JavaScript files and CA template are vendored unchanged. Scope substitutes its own public CA and chosen proxy port only in runtime copies. The runtime CA template also uses a Scope-specific temporary directory and certificate filename. - # Third-party notices The project is distributed under **AGPL-3.0-or-later**; see [LICENSE](LICENSE). @@ -33,24 +31,9 @@ hashes. Both directories must be retained in corresponding source archives. GLib's original license and authorship remain applicable; linked native GNOME libraries have their own license terms described in GLib's `COPYRIGHT`. -## HTTP Toolkit interception scripts - -Files: `resources/frida/config-template.js` and `resources/frida/android/*.js`. - -- Authors: Tim Perry and HTTP Toolkit contributors; `android-disable-root-detection.js` also credits Riyad Mondal. -- License: **AGPL-3.0-or-later**. The original notices remain in each source file; the full license is included in [LICENSE](LICENSE). -- Upstream project: [frida-interception-and-unpinning](https://github.com/httptoolkit/frida-interception-and-unpinning). -- Exact distribution source: [HTTP Toolkit Server v1.26.1, overrides/frida](https://github.com/httptoolkit/httptoolkit-server/tree/1abef0adc96d07bd64377e85e847025a55bac2ac/overrides/frida), commit `1abef0adc96d07bd64377e85e847025a55bac2ac`. -- Local changes: `config.js` was renamed to `config-template.js`. The JavaScript content is otherwise unchanged from that distribution. Runtime certificate and proxy substitutions are written to the ignored `.runtime/` directory, rather than these source files. - -## HTTP Toolkit system CA installation template - -File: `resources/device/install-system-ca.template.sh`. +## Original Scope engine and historical migration -- Authors: Tim Perry and HTTP Toolkit contributors. -- License: **AGPL-3.0-or-later**; see [LICENSE](LICENSE). -- Source: the `injectSystemCertificate` shell template in [HTTP Toolkit Server v1.26.1, adb-commands.ts](https://github.com/httptoolkit/httptoolkit-server/blob/1abef0adc96d07bd64377e85e847025a55bac2ac/src/interceptors/android/adb-commands.ts). -- Modified on **2026-09-30**: extracted the embedded shell script into a standalone template, added provenance and license comments, and retained `${certificatePath}` and `${injectionScriptPath}` as substitutions supplied by the launcher. Any later changes should keep this notice up to date. +`resources/frida/scope-engine.js` and the Python worker are original Scope integration code under the project AGPL-3.0-or-later license. The current runtime no longer includes the six HTTP Toolkit hook/config files or its extracted system-CA installation template. The project previously distributed them from the owner's `SculptTechProject/bypass-scripts-adb` repository, originating in HTTP Toolkit Server v1.26.1 (`1abef0adc96d07bd64377e85e847025a55bac2ac`). Their historical copyright, provenance and AGPL notices remain in public Git history. No history rewrite or project license change is part of this migration. This is not a legally certified clean-room claim. ## Frida Java bridge @@ -83,21 +66,9 @@ The helper license texts were copied verbatim from the published `@frida/buffer@ The GNU Library GPL text was obtained from [GNU](https://www.gnu.org/licenses/old-licenses/lgpl-2.0.txt). The wxWindows license and exception were obtained from [Frida Gum 17.9.1 COPYING](https://github.com/frida/frida-gum/blob/17.9.1/COPYING), preserving that license section verbatim. -The extracted CA installation template has SHA-256 `201fa259470c9fa4e1b70dd3a020e7807f762bf838346ae2f8297f62586eb67b` after the modifications listed above. +## Retained JavaScript integrity -## Vendored JavaScript integrity - -These SHA-256 values identify the unmodified JavaScript copied from HTTP Toolkit Server v1.26.1. The renamed configuration template has the same bytes as upstream `config.js`. - -| File | SHA-256 | -| --- | --- | -| `resources/frida/android/android-certificate-unpinning-fallback.js` | `5190e449ae590e896f362a5751267199c2bef5417f69635715705def00462cd7` | -| `resources/frida/android/android-certificate-unpinning.js` | `21cfb61cc492c557fc55e2b504a2e5e8f679b3515bff824acb12b14975381d9b` | -| `resources/frida/android/android-disable-root-detection.js` | `25e91287ab38111a188eb5786fa47586a252c711f0e401b9dc6a7c727600fda8` | -| `resources/frida/android/android-proxy-override.js` | `aed9941c65bd9d23789f9e54e11d37cd95ab83c5818f0c39a36e4581b05ad688` | -| `resources/frida/android/android-system-certificate-injection.js` | `f751f783f30fc9b296bf7f05692f27faabb31a37dafd46c31cd756bab98f8641` | -| `resources/frida/config-template.js` | `5afad65ed908006eb1af99aa6da3a73359036bdaa870fe32bf654695fa95d02a` | -| `resources/frida/frida-java-bridge.js` | `88f0ca1fb34d2bd7b589f3d4c44aa0077780473530922887a53311ec55d9c214` | +The unchanged retained bridge distributed by HTTP Toolkit Server v1.26.1 has SHA-256 `88f0ca1fb34d2bd7b589f3d4c44aa0077780473530922887a53311ec55d9c214`. The exact bridge distribution source and corresponding source are listed above; retention does not make it original Scope code. ## Optional packaged Frida runtime @@ -126,8 +97,7 @@ The main Frida licenses incorporate GNU Library GPL 2 or later, with an explicit exception permitting binary object code distribution under the distributor's own terms. The [GNU Library GPL text](resources/device/licenses/LGPL-2.0.txt) is included alongside those exceptions. Scope does not modify Frida, CPython or -PyInstaller. The JavaScript interception scripts remain subject to the separate -AGPL notices above. +PyInstaller. The original Scope engine remains under the project AGPL license. The older Python 3.9 developer environment optionally installs `typing_extensions` 4.16.0; its [license](resources/device/licenses/typing_extensions-4.16.0-LICENSE.txt) @@ -139,3 +109,7 @@ are recorded in the [runtime notice index](resources/device/licenses/README.md). Keep that directory with the integration sources. Any future redistribution of the optional native runtime requires a complete artifact-specific dependency and corresponding-source review. The public packaging checks currently reject it. + +## Owned Android acceptance APK (test only) + +The original [lab fixture](examples/android-engine-fixture/README.md) downloads checksum-locked OkHttp 3.14.9 and Okio 1.17.2 only into an explicit external build directory. Its APK includes [dependency notices](examples/android-engine-fixture/THIRD_PARTY_NOTICES.txt), Apache-2.0 text and any embedded JAR notices, including the Public Suffix List MPL-2.0 notice. These JARs and APKs are excluded from source control and Scope production resources. diff --git a/crates/scope-device/src/diagnostics.rs b/crates/scope-device/src/diagnostics.rs index 46b122e..7cc5854 100644 --- a/crates/scope-device/src/diagnostics.rs +++ b/crates/scope-device/src/diagnostics.rs @@ -1075,6 +1075,8 @@ mod tests { server_pid: Some(481), ca_hash: None, root_shell: true, + target_spawn_pending: false, + target_pid: None, } } fn elf(path: &Path, machine: u16, class: u8, endian: u8) { diff --git a/crates/scope-device/src/lib.rs b/crates/scope-device/src/lib.rs index a0d9c29..421e89b 100644 --- a/crates/scope-device/src/lib.rs +++ b/crates/scope-device/src/lib.rs @@ -1,5 +1,4 @@ use anyhow::{bail, Context, Result}; -use md5::{Digest, Md5}; use serde::{Deserialize, Serialize}; use std::{ collections::HashMap, @@ -75,11 +74,15 @@ struct Recovery { server_pid: Option, ca_hash: Option, root_shell: bool, + #[serde(default)] + target_spawn_pending: bool, + #[serde(default)] + target_pid: Option, } struct Active { recovery: Recovery, - worker: Option, + worker: Option, } struct Inner { runtime: PathBuf, @@ -88,6 +91,7 @@ struct Inner { operation: AsyncMutex<()>, active: AsyncMutex>, worker_generation: AtomicU64, + recovery_io: Mutex<()>, } #[derive(Clone)] pub struct DeviceManager { @@ -118,6 +122,7 @@ impl DeviceManager { operation: AsyncMutex::new(()), active: AsyncMutex::new(None), worker_generation: AtomicU64::new(0), + recovery_io: Mutex::new(()), }), } } @@ -126,12 +131,22 @@ impl DeviceManager { self.inner.status.lock().unwrap().clone() } - fn log(&self, text: impl Into) { + fn log(&self, text: impl AsRef) { let mut status = self.inner.status.lock().unwrap(); - status.logs.push(text.into()); - if status.logs.len() > 200 { + if status.logs.len() == 200 { status.logs.remove(0); } + status.logs.push(bounded_diagnostic(text.as_ref())); + } + + fn log_worker(&self, generation: u64, text: impl AsRef) { + let mut status = self.inner.status.lock().unwrap(); + if self.inner.worker_generation.load(Ordering::SeqCst) == generation { + if status.logs.len() == 200 { + status.logs.remove(0); + } + status.logs.push(bounded_diagnostic(text.as_ref())); + } } async fn command(&self, serial: &str, args: &[&str]) -> Result { @@ -261,12 +276,7 @@ impl DeviceManager { if !["proxy", "frida"].contains(&mode.as_str()) { bail!("Mode must be proxy or frida"); } - for value in &profile { - if !["proxy", "certificate", "unpinning", "root", "system-ca"].contains(&value.as_str()) - { - bail!("Unknown instrumentation profile: {value}"); - } - } + let profile = normalize_profiles(profile)?; if proxy_port == 0 { bail!("Proxy port must be greater than zero"); } @@ -350,6 +360,8 @@ impl DeviceManager { server_pid: None, ca_hash: None, root_shell, + target_spawn_pending: false, + target_pid: None, }; *self.inner.status.lock().unwrap() = DeviceStatus { running: false, @@ -367,33 +379,18 @@ impl DeviceManager { self.log(format!("ADB reverse: device 127.0.0.1:{proxy_port} → Scope localhost:{proxy_port}")); let worker = if mode == "proxy" { self.log("Proxy mode is DEVICE-WIDE. The selected package is launched, but Android may route other applications through this proxy."); - if rooted { - let hash = certificate_hash(&ca_cert)?; - let existing = self.root_command(&state, &format!("cat /apex/com.android.conscrypt/cacerts/{hash}.0 2>/dev/null || cat /system/etc/security/cacerts/{hash}.0 2>/dev/null || true")).await?; - if !existing.trim().is_empty() && existing.trim() != fs::read_to_string(&ca_cert)?.trim() { - bail!("An existing Android CA uses the same subject hash. Scope will not overwrite it"); - } - if existing.trim().is_empty() { - // Record the exact certificate before attempting injection so partial setup is recoverable. - state.ca_hash = Some(hash); - self.save_recovery(&state)?; - self.install_ca(&state, &ca_cert).await?; - } else { - self.log("Scope CA was already present; it will be preserved on Stop"); - } - } else { - self.log("No root: HTTP capture is available. HTTPS requires the app to trust Scope's CA manually; certificate pinning remains active."); - } + self.log("HTTPS requires manually configured application trust. System proxy mode does not install a CA or bypass pinning."); self.command(&state.serial, &["shell", "settings", "put", "global", "http_proxy", &format!("127.0.0.1:{proxy_port}")]).await?; self.launch(&state).await?; - if state.ca_hash.is_some() { self.verify_ca(&state, &ca_cert).await?; } (None, None) } else { self.log("Frida mode instruments the selected application process only; Java HTTP stacks are supported. Native TLS, child processes and QUIC are not guaranteed."); let (abi, version) = frida_preflight.as_ref().context("Missing Frida preflight")?; let address = self.prepare_frida_server(&mut state, abi, version).await?; self.save_recovery(&state)?; - let (worker, live) = self.spawn_worker(&state, &ca_cert, &profile, address, generation).await?; + let worker = self.spawn_worker(&mut state, &ca_cert, &profile, address, generation).await?; + let live = worker.live.clone(); + state = worker.observation.lock().unwrap().recovery.clone(); (Some(worker), Some(live)) }; Ok::<_, anyhow::Error>(worker) @@ -419,7 +416,11 @@ impl DeviceManager { } Err(error) => { self.log(format!("Setup failed: {error:#}")); - let cleanup = self.cleanup(&state).await; + let cleanup = if state.target_spawn_pending { + Err(anyhow::anyhow!("Target process cleanup is unconfirmed; recovery retained. Press Stop to reconcile the recorded PID, or manually reconcile an unknown spawn")) + } else { + self.cleanup(&state).await + }; let mut status = self.inner.status.lock().unwrap(); status.running = false; status.message = match cleanup { @@ -477,64 +478,6 @@ impl DeviceManager { Ok(()) } - async fn install_ca(&self, state: &Recovery, cert: &std::path::Path) -> Result<()> { - let hash = state.ca_hash.as_ref().context("Missing CA hash")?; - let remote_cert = format!("/data/local/tmp/scope-{hash}.0"); - let remote_script = "/data/local/tmp/scope-install-ca.sh"; - let script = include_str!("../../../resources/device/install-system-ca.template.sh") - .replace("htk-ca-copy", "scope-ca-copy") - .replace("${certificatePath}", &remote_cert) - .replace("${injectionScriptPath}", remote_script); - // The template moves the input basename into cacerts: it must be .0. - let script = script.replace( - &format!("mv {remote_cert} /system/etc/security/cacerts/"), - &format!("cp {remote_cert} /system/etc/security/cacerts/{hash}.0"), - ); - let path = self.inner.runtime.join("install-system-ca.sh"); - fs::write(&path, script)?; - self.command( - &state.serial, - &["push", &cert.to_string_lossy(), &remote_cert], - ) - .await?; - self.command( - &state.serial, - &["push", &path.to_string_lossy(), remote_script], - ) - .await?; - let result = self - .root_command(state, &format!("sh {remote_script}")) - .await?; - if !result.contains("System cert successfully injected") { - bail!("System CA injection was not confirmed"); - } - self.log(format!( - "Temporary Scope CA {hash}.0 injected into Android trust stores; removed on Stop" - )); - Ok(()) - } - - async fn verify_ca(&self, state: &Recovery, cert: &PathBuf) -> Result<()> { - let pid = self - .command(&state.serial, &["shell", "pidof", &state.package]) - .await?; - let pid = pid - .split_whitespace() - .next() - .context("Launched app PID was not found")?; - if !pid.bytes().all(|c| c.is_ascii_digit()) { - bail!("Invalid application PID"); - } - let hash = state.ca_hash.as_ref().context("Missing CA hash")?; - let command = format!("nsenter --mount=/proc/{pid}/ns/mnt -- cat /apex/com.android.conscrypt/cacerts/{hash}.0 || nsenter --mount=/proc/{pid}/ns/mnt -- cat /system/etc/security/cacerts/{hash}.0"); - let installed = self.root_command(state, &command).await?; - if installed.trim() != fs::read_to_string(cert)?.trim() { - bail!("Scope CA was not visible in the selected app's mount namespace"); - } - self.log("Scope CA verified in the selected application's mount namespace"); - Ok(()) - } - async fn prepare_frida_server( &self, state: &mut Recovery, @@ -633,12 +576,12 @@ impl DeviceManager { async fn spawn_worker( &self, - state: &Recovery, + state: &mut Recovery, cert: &PathBuf, profile: &[String], address: Option, generation: u64, - ) -> Result<(Child, Arc)> { + ) -> Result { let mut command = worker_command(&self.inner.runtime); command .args([ @@ -655,7 +598,7 @@ impl DeviceManager { if let Some(address) = address { command.args(["--address", &address]); } - for profile in profile.iter().filter(|v| v.as_str() != "system-ca") { + for profile in profile { command.args(["--profile", profile]); } command @@ -665,55 +608,110 @@ impl DeviceManager { .kill_on_drop(true); #[cfg(windows)] command.creation_flags(0x08000000); - let mut child = command.spawn().context("Cannot start Frida worker; set SCOPE_PYTHON to a Python executable with frida installed")?; + // Persist uncertainty before a worker can attempt a remote spawn, including a host crash. + state.target_spawn_pending = true; + self.save_recovery(state)?; + let mut child = match command.spawn() { + Ok(child) => child, + Err(error) => { + state.target_spawn_pending = false; + self.save_recovery(state)?; + return Err(error).context("Cannot start Frida worker; set SCOPE_PYTHON to a Python executable with frida installed"); + } + }; let stdout = child.stdout.take().context("Missing Frida worker stdout")?; let stderr = child.stderr.take().context("Missing Frida worker stderr")?; let (sender, receiver) = oneshot::channel(); let live = Arc::new(AtomicBool::new(true)); - let worker_live = live.clone(); - tokio::spawn(monitor_worker_output( + let observation = Arc::new(Mutex::new(WorkerObservation::new( + state.clone(), + profile.to_vec(), + ))); + let monitor = tokio::spawn(monitor_worker_output( self.clone(), stdout, generation, - worker_live, + live.clone(), sender, + observation.clone(), )); - let manager = self.clone(); - tokio::spawn(async move { - let mut lines = BufReader::new(stderr).lines(); - while let Ok(Some(line)) = lines.next_line().await { - manager.log(line); - } - }); + tokio::spawn(monitor_worker_stderr(self.clone(), stderr, generation)); + let mut worker = Worker { + child, + live, + observation, + monitor, + }; match tokio::time::timeout(Duration::from_secs(40), receiver).await { - Ok(Ok(Ok(()))) => Ok((child, live)), + Ok(Ok(Ok(()))) => Ok(worker), result => { - drop(child.stdin.take()); - let _ = child.kill().await; - match result { - Ok(Ok(Err(message))) => bail!(message), - _ => bail!("Frida worker did not confirm hook readiness within 40 seconds"), + let teardown = self.finish_worker(&mut worker).await; + *state = worker.observation.lock().unwrap().recovery.clone(); + self.invalidate_worker(); + if teardown.is_ok() { + state.target_spawn_pending = false; + self.save_recovery(state)?; } + let cause = match result { + Ok(Ok(Err(message))) => message, + _ => "Frida worker did not confirm hook readiness within 40 seconds".into(), + }; + bail!( + "{cause}; teardown: {}", + teardown + .err() + .map(|e| e.to_string()) + .unwrap_or_else(|| "confirmed".into()) + ); } } } + // Never holds operation/recovery locks while awaiting stdout. The monitor must drain the final receipt. + async fn finish_worker(&self, worker: &mut Worker) -> Result<()> { + if let Some(mut stdin) = worker.child.stdin.take() { + let _ = stdin.write_all(b"stop\n").await; + } + let exit = match tokio::time::timeout(Duration::from_secs(12), worker.child.wait()).await { + Ok(result) => result.context("Cannot reap Frida worker"), + Err(_) => { + let _ = worker.child.kill().await; + Err(anyhow::anyhow!("Frida worker exceeded 12-second cleanup grace; target termination is unconfirmed")) + } + }; + let drained = tokio::time::timeout(Duration::from_secs(2), &mut worker.monitor).await; + if !matches!(drained, Ok(Ok(()))) { + worker.monitor.abort(); + bail!("Frida worker output did not drain; target termination is unconfirmed"); + } + let exit = exit?; + if !exit.success() { + bail!("Frida worker failed ({exit}); recovery retained"); + } + let observed = worker.observation.lock().unwrap(); + observed.confirm_stopped()?; + Ok(()) + } + + fn invalidate_worker(&self) { + let _io = self.inner.recovery_io.lock().unwrap(); + self.inner.worker_generation.fetch_add(1, Ordering::SeqCst); + } + pub async fn stop(&self) -> Result { let _operation = self.inner.operation.lock().await; - // Ignore late stdout/EOF notifications from a worker being intentionally stopped. - self.inner.worker_generation.fetch_add(1, Ordering::SeqCst); let active = self.inner.active.lock().await.take(); let state = if let Some(mut active) = active { - if let Some(mut child) = active.worker.take() { - if let Some(mut stdin) = child.stdin.take() { - let _ = stdin.write_all(b"stop\n").await; - } - if tokio::time::timeout(Duration::from_secs(5), child.wait()) - .await - .is_err() - { - let _ = child.kill().await; + if let Some(mut worker) = active.worker.take() { + let result = self.finish_worker(&mut worker).await; + active.recovery = worker.observation.lock().unwrap().recovery.clone(); + self.invalidate_worker(); + if let Err(error) = result { + self.cleanup_failed(&active.recovery, &error); + return Err(error); } + active.recovery.target_spawn_pending = false; + self.save_recovery(&active.recovery)?; } Some(active.recovery) } else if self.recovery_path().is_file() { @@ -725,24 +723,70 @@ impl DeviceManager { }; if let Some(state) = state { if let Err(error) = self.cleanup(&state).await { - let mut status = self.inner.status.lock().unwrap(); - status.running = false; - status.message = format!( - "Cleanup incomplete. Reconnect {} and press Stop again: {error:#}", - state.serial - ); + self.cleanup_failed(&state, &error); return Err(error); } } let mut status = self.inner.status.lock().unwrap(); status.running = false; - status.message = "Device settings restored; saved captures remain available".into(); + status.message = "Device settings restored; instrumented process ended when applicable; saved captures remain available".into(); Ok(status.clone()) } + fn cleanup_failed(&self, state: &Recovery, error: &anyhow::Error) { + let mut status = self.inner.status.lock().unwrap(); + status.running = false; + status.message = format!( + "Cleanup incomplete. Reconnect {} and press Stop again: {error:#}", + state.serial + ); + } + + async fn reconcile_target(&self, state: &Recovery) -> Result<()> { + if !state.target_spawn_pending { + return Ok(()); + } + let pid = state.target_pid.context("Spawn was attempted without a recorded PID. Manually reconcile the app/process on the device; Scope cannot safely infer or kill it. Recovery retained")?; + if pid == 0 { + bail!("Invalid recorded target PID; recovery retained"); + } + // Require a readable root procfs view. A package pid lookup or worker exit cannot establish absence. + let probe = format!( + r#"if [ "$(id -u)" != 0 ] || [ ! -r /proc/1/stat ] || [ ! -r /proc/self/stat ]; then + echo UNKNOWN +elif [ -d /proc/{pid} ]; then + echo PRESENT +elif mounts=$(cat /proc/mounts 2>/dev/null) && entries=$(ls /proc 2>/dev/null); then + case "$mounts" in + *hidepid=*) echo UNKNOWN;; + *) found=0; init=0 + for entry in $entries; do + [ "$entry" = "{pid}" ] && found=1 + [ "$entry" = 1 ] && init=1 + done + if [ "$found" = 1 ]; then echo PRESENT + elif [ "$init" = 1 ] && [ ! -e /proc/{pid} ]; then echo ABSENT + else echo UNKNOWN; fi;; + esac +else echo UNKNOWN; fi"# + ); + let result = self + .root_command(state, &probe) + .await + .context("Cannot inspect recorded target process; recovery retained")?; + if result.trim() != "ABSENT" { + bail!("Recorded target PID {pid} is present or its state is inaccessible. Scope will not kill a saved or reused PID; manually reconcile it and retry Stop. Recovery retained"); + } + self.log(format!( + "Recorded target PID {pid} is absent in the observed root procfs view" + )); + Ok(()) + } + async fn cleanup(&self, state: &Recovery) -> Result<()> { // Validate every persisted field before the first cleanup command. diagnostics::validate_recovery(state)?; + self.reconcile_target(state).await?; let mut errors = vec![]; if let Some(old) = &state.previous_proxy { let current = self @@ -874,6 +918,7 @@ impl DeviceManager { if !errors.is_empty() { bail!("{}", errors.join("; ")); } + let _io = self.inner.recovery_io.lock().unwrap(); if self.recovery_path().exists() { fs::remove_file(self.recovery_path())?; } @@ -908,6 +953,11 @@ impl DeviceManager { ) } fn save_recovery(&self, state: &Recovery) -> Result<()> { + let _io = self.inner.recovery_io.lock().unwrap(); + self.write_recovery(state) + } + + fn write_recovery(&self, state: &Recovery) -> Result<()> { fs::create_dir_all(&self.inner.runtime)?; let temporary = self.inner.runtime.join("device-recovery.tmp"); fs::write(&temporary, serde_json::to_vec_pretty(state)?)?; @@ -916,7 +966,7 @@ impl DeviceManager { } fn materialize(&self) -> Result<()> { - fs::create_dir_all(self.inner.runtime.join("frida/android"))?; + fs::create_dir_all(self.inner.runtime.join("frida"))?; for (path, content) in ASSETS { fs::write(self.inner.runtime.join(path), content)?; } @@ -924,57 +974,372 @@ impl DeviceManager { } } +const ENGINE: &str = "scope-java-v1"; + +const WORKER_LINE_LIMIT: usize = 65536; +const DIAGNOSTIC_LIMIT: usize = 4096; + +fn bounded_diagnostic(text: &str) -> String { + const MARKER: &str = " [truncated]"; + if text.len() <= DIAGNOSTIC_LIMIT { + return text.to_owned(); + } + let mut end = DIAGNOSTIC_LIMIT - MARKER.len(); + while !text.is_char_boundary(end) { + end -= 1; + } + let mut retained = String::with_capacity(DIAGNOSTIC_LIMIT); + retained.push_str(&text[..end]); + retained.push_str(MARKER); + retained +} + +enum WorkerLine { + Text(String), + Oversized, + InvalidUtf8, +} + +// Keep one bounded line plus the reader's fixed-size input buffer. Report overflow +// immediately, then discard through LF without retaining the oversized remainder. +struct WorkerLines { + reader: BufReader, + bytes: Vec, + discarding: bool, +} +impl WorkerLines { + fn new(reader: R) -> Self { + Self { + reader: BufReader::with_capacity(8192, reader), + // One extra byte permits a CR immediately before LF at the exact limit. + bytes: Vec::with_capacity(WORKER_LINE_LIMIT + 1), + discarding: false, + } + } + + fn finish_line(&mut self) -> WorkerLine { + if self.bytes.last() == Some(&b'\r') { + self.bytes.pop(); + } + let line = match std::str::from_utf8(&self.bytes) { + Ok(text) => WorkerLine::Text(text.to_owned()), + Err(_) => WorkerLine::InvalidUtf8, + }; + self.bytes.clear(); + line + } + + async fn next_line(&mut self) -> std::io::Result> { + loop { + let available = self.reader.fill_buf().await?; + if available.is_empty() { + return if self.discarding || self.bytes.is_empty() { + Ok(None) + } else { + Ok(Some(self.finish_line())) + }; + } + let newline = available.iter().position(|byte| *byte == b'\n'); + let count = newline.unwrap_or(available.len()); + let consumed = count + usize::from(newline.is_some()); + if self.discarding { + self.reader.consume(consumed); + self.discarding = newline.is_none(); + continue; + } + let remaining = WORKER_LINE_LIMIT + 1 - self.bytes.len(); + self.bytes + .extend_from_slice(&available[..count.min(remaining)]); + let oversized = count > remaining + || (self.bytes.len() > WORKER_LINE_LIMIT && self.bytes.last() != Some(&b'\r')); + self.reader.consume(consumed); + if oversized { + self.bytes.clear(); + self.discarding = newline.is_none(); + return Ok(Some(WorkerLine::Oversized)); + } + if newline.is_some() { + return Ok(Some(self.finish_line())); + } + } + } +} + +async fn monitor_worker_stderr( + manager: DeviceManager, + stderr: R, + generation: u64, +) { + let mut lines = WorkerLines::new(stderr); + loop { + match lines.next_line().await { + Ok(Some(WorkerLine::Text(line))) => manager.log_worker(generation, line), + Ok(Some(WorkerLine::Oversized)) => manager.log_worker( + generation, + "Frida worker stderr line exceeded 65536 bytes; discarded", + ), + Ok(Some(WorkerLine::InvalidUtf8)) => { + manager.log_worker(generation, "Invalid UTF-8 in Frida worker stderr") + } + Ok(None) => break, + Err(error) => { + manager.log_worker( + generation, + format!("Cannot read Frida worker stderr: {error}"), + ); + break; + } + } + } +} + +fn normalize_profiles(profiles: Vec) -> Result> { + let profiles = if profiles.is_empty() { + vec!["proxy".into(), "certificate".into(), "unpinning".into()] + } else { + profiles + }; + let mut selected = Vec::new(); + for profile in profiles { + if !["proxy", "certificate", "unpinning"].contains(&profile.as_str()) { + bail!("Unsupported instrumentation profile: {profile}"); + } + if !selected.contains(&profile) { + selected.push(profile); + } + } + if selected.iter().any(|p| p == "unpinning") && !selected.iter().any(|p| p == "certificate") { + bail!("unpinning profile requires certificate"); + } + // Match the worker/engine canonical order while retaining exactly the explicit selection. + Ok(["proxy", "certificate", "unpinning"] + .into_iter() + .filter(|profile| selected.iter().any(|selected| selected == profile)) + .map(String::from) + .collect()) +} + +#[derive(Deserialize)] +#[serde(tag = "kind", rename_all = "camelCase")] +enum WorkerEvent { + Spawned { + engine: String, + pid: u32, + }, + Ready { + engine: String, + pid: u32, + profiles: Vec, + capabilities: HashMap>, + }, + Stopped { + engine: String, + pid: Option, + #[serde(rename = "spawnAttempted")] + spawn_attempted: bool, + #[serde(rename = "cleanupComplete")] + cleanup_complete: bool, + #[serde(rename = "processTerminated")] + process_terminated: bool, + }, + Error { + message: String, + }, + Detached { + message: String, + }, + Log { + message: serde_json::Value, + }, +} + +struct Worker { + child: Child, + live: Arc, + observation: Arc>, + monitor: tokio::task::JoinHandle<()>, +} +struct WorkerObservation { + recovery: Recovery, + profiles: Vec, + spawned: bool, + terminal: bool, + failure: Option, + stopped: bool, +} +impl WorkerObservation { + fn new(recovery: Recovery, profiles: Vec) -> Self { + Self { + recovery, + profiles, + spawned: false, + terminal: false, + failure: None, + stopped: false, + } + } + fn confirm_stopped(&self) -> Result<()> { + if let Some(error) = &self.failure { + bail!("{error}; recovery retained"); + } + if !self.stopped { + bail!("Missing matching target-termination acknowledgment; recovery retained"); + } + Ok(()) + } + fn ready( + &self, + engine: &str, + pid: u32, + profiles: &[String], + capabilities: &HashMap>, + ) -> bool { + !self.terminal + && self.spawned + && engine == ENGINE + && self.recovery.target_pid == Some(pid) + && profiles == self.profiles + && capabilities.len() == profiles.len() + && profiles.iter().all(|profile| { + let required = match profile.as_str() { + "proxy" => "java-url", + "certificate" => "sslcontext-scope-ca", + "unpinning" => "okhttp3-certificate-pinner", + _ => return false, + }; + capabilities + .get(profile) + .is_some_and(|installed| installed.iter().any(|item| item == required)) + }) + } +} + async fn monitor_worker_output( manager: DeviceManager, stdout: R, generation: u64, live: Arc, sender: oneshot::Sender>, + observation: Arc>, ) { let mut sender = Some(sender); - let mut lines = BufReader::new(stdout).lines(); - while let Ok(Some(line)) = lines.next_line().await { - if let Ok(value) = serde_json::from_str::(&line) { - match value.get("kind").and_then(|v| v.as_str()).unwrap_or("log") { - "ready" => { - if let Some(sender) = sender.take() { - let _ = sender.send(Ok(())); - } - } - "error" => { - let message = value - .get("message") - .and_then(|v| v.as_str()) - .unwrap_or("Frida worker failed") - .to_string(); - if let Some(sender) = sender.take() { - let _ = sender.send(Err(message.clone())); + let mut lines = WorkerLines::new(stdout); + loop { + let line = lines.next_line().await; + if matches!(line, Ok(None)) { + break; + } + let io_failed = line.is_err(); + let _io = manager.inner.recovery_io.lock().unwrap(); + if manager.inner.worker_generation.load(Ordering::SeqCst) != generation { + if io_failed { + break; + } + continue; + } + let mut observed = observation.lock().unwrap(); + let event = match line { + Ok(Some(WorkerLine::Text(line))) => serde_json::from_str::(&line) + .map_err(|_| "Invalid Frida worker protocol receipt".to_owned()), + Ok(Some(WorkerLine::Oversized)) => { + Err("Frida worker stdout line exceeded 65536 bytes; recovery retained".into()) + } + Ok(Some(WorkerLine::InvalidUtf8)) => Err("Invalid UTF-8 in Frida worker stdout".into()), + Err(error) => Err(format!("Cannot read Frida worker stdout: {error}")), + Ok(None) => unreachable!(), + }; + let mut error = None; + match event { + Ok(WorkerEvent::Spawned { engine, pid }) + if engine == ENGINE && pid > 0 && !observed.spawned && !observed.terminal => + { + observed.recovery.target_pid = Some(pid); + observed.spawned = true; + if manager.recovery_path().is_file() { + if let Err(failure) = manager.write_recovery(&observed.recovery) { + error = Some(format!("Cannot persist spawned target PID: {failure}")); } - manager.log(message); - live.store(false, Ordering::SeqCst); - manager.worker_ended(generation, "Frida worker reported an error; press Stop to restore the recorded device state"); + } else { + error = + Some("Recovery record disappeared before target PID persistence".into()); } - "stopped" | "detached" => { + } + Ok(WorkerEvent::Ready { + engine, + pid, + profiles, + capabilities, + }) => { + if observed.ready(&engine, pid, &profiles, &capabilities) { if let Some(sender) = sender.take() { - let detail = value - .get("message") - .and_then(|value| value.as_str()) - .unwrap_or("device session ended"); - let _ = sender.send(Err(format!("Frida detached before hook readiness: {detail}. Check Android compatibility and the selected app's process"))); + let _ = sender.send(Ok(())); } - live.store(false, Ordering::SeqCst); - manager.worker_ended( - generation, - "Frida worker detached; press Stop to restore the recorded device state", + } else { + error = Some( + "Invalid engine readiness/capability receipt or terminal session".into(), ); } - _ => manager.log( - value - .get("message") - .map(|v| v.as_str().unwrap_or("Frida diagnostic")) - .unwrap_or("Frida diagnostic"), - ), } + Ok(WorkerEvent::Stopped { + engine, + pid, + spawn_attempted, + cleanup_complete, + process_terminated, + }) => { + let valid = engine == ENGINE + && cleanup_complete + && if spawn_attempted { + observed.spawned + && pid == observed.recovery.target_pid + && process_terminated + } else { + !observed.spawned && pid.is_none() && !process_terminated + }; + if valid { + observed.stopped = true; + } else { + error = Some("Invalid target-termination acknowledgment".into()); + } + observed.terminal = true; + if let Some(sender) = sender.take() { + let _ = sender.send(Err("Frida stopped before hook readiness".into())); + } + } + Ok(WorkerEvent::Error { message }) => error = Some(message), + Ok(WorkerEvent::Detached { message }) => { + observed.terminal = true; + if let Some(sender) = sender.take() { + let _ = sender.send(Err(format!( + "Frida detached before hook readiness: {message}" + ))); + } + } + Ok(WorkerEvent::Log { message }) => { + manager.log(message.as_str().unwrap_or("Frida diagnostic")) + } + Err(message) => error = Some(message), + _ => error = Some("Invalid Frida worker protocol receipt".into()), + } + if let Some(message) = error { + observed.terminal = true; + observed + .failure + .get_or_insert_with(|| bounded_diagnostic(&message)); + if let Some(sender) = sender.take() { + let _ = sender.send(Err(message.clone())); + } + manager.log(message); + } + if observed.terminal { + live.store(false, Ordering::SeqCst); + manager.worker_ended( + generation, + "Frida worker ended; press Stop to reconcile the recorded device state", + ); + } + if io_failed { + break; } } if let Some(sender) = sender.take() { @@ -1093,55 +1458,682 @@ fn frida_arch(abi: &str) -> Option<&'static str> { } } -fn certificate_hash(path: &PathBuf) -> Result { - let pem = pem::parse(fs::read(path)?).context("Cannot parse Scope public CA")?; - let (_, cert) = x509_parser::parse_x509_certificate(pem.contents()) - .map_err(|e| anyhow::anyhow!("Cannot parse Scope CA: {e}"))?; - let hash = Md5::digest(cert.subject().as_raw()); - Ok(format!( - "{:08x}", - u32::from_le_bytes(hash[..4].try_into().unwrap()) - )) -} - const ASSETS: &[(&str, &str)] = &[ ( "frida_worker.py", include_str!("../../../resources/device/frida_worker.py"), ), ( - "frida/config-template.js", - include_str!("../../../resources/frida/config-template.js"), + "frida/scope-engine.js", + include_str!("../../../resources/frida/scope-engine.js"), ), ( "frida/frida-java-bridge.js", include_str!("../../../resources/frida/frida-java-bridge.js"), ), - ( - "frida/android/android-proxy-override.js", - include_str!("../../../resources/frida/android/android-proxy-override.js"), - ), - ( - "frida/android/android-system-certificate-injection.js", - include_str!("../../../resources/frida/android/android-system-certificate-injection.js"), - ), - ( - "frida/android/android-certificate-unpinning.js", - include_str!("../../../resources/frida/android/android-certificate-unpinning.js"), - ), - ( - "frida/android/android-certificate-unpinning-fallback.js", - include_str!("../../../resources/frida/android/android-certificate-unpinning-fallback.js"), - ), - ( - "frida/android/android-disable-root-detection.js", - include_str!("../../../resources/frida/android/android-disable-root-detection.js"), - ), ]; #[cfg(test)] mod tests { use super::*; + fn test_observation() -> Arc> { + Arc::new(Mutex::new(WorkerObservation::new( + Recovery { + serial: "fixture-01".into(), + package: "com.scope.demo".into(), + mode: "frida".into(), + port: 17890, + previous_proxy: None, + previous_reverse: None, + forward_port: None, + server_pid: None, + ca_hash: None, + root_shell: true, + target_spawn_pending: true, + target_pid: None, + }, + vec!["proxy".into()], + ))) + } + #[test] + fn retained_diagnostics_are_byte_bounded_on_utf8_boundaries() { + let directory = tempfile::tempdir().unwrap(); + let manager = DeviceManager::new(directory.path().into()); + manager.log("a".repeat(4096)); + assert_eq!(manager.status().logs[0], "a".repeat(4096)); + manager.log("b".repeat(4097)); + assert_eq!(manager.status().logs[1].len(), 4096); + assert!(manager.status().logs[1].ends_with(" [truncated]")); + for _ in 0..205 { + manager.log("😀".repeat(4096)); + } + let status = manager.status(); + assert_eq!(status.logs.len(), 200); + assert!(status.logs.iter().all(|line| line.len() <= 4096)); + assert!(status.logs.iter().all(|line| line.ends_with("[truncated]"))); + assert!(status.logs.iter().map(String::len).sum::() <= 819200); + assert!(manager + .inner + .status + .lock() + .unwrap() + .logs + .iter() + .all(|line| line.capacity() <= 4096)); + manager.log_worker(0, "é".repeat(4096)); + assert!(manager.status().logs.last().unwrap().len() <= 4096); + let mut oversized_allocation = String::with_capacity(1024 * 1024); + oversized_allocation.push_str("short"); + manager.log(&oversized_allocation); + manager.log_worker(0, oversized_allocation); + let retained = manager.inner.status.lock().unwrap(); + assert_eq!(retained.logs.last().unwrap(), "short"); + assert!(retained.logs.iter().all(|line| line.capacity() <= 4096)); + assert!(retained.logs.iter().map(String::capacity).sum::() <= 819200); + } + + async fn monitor_fixture( + input: Vec, + fragment: usize, + ) -> ( + DeviceManager, + Arc>, + std::result::Result<(), String>, + tempfile::TempDir, + ) { + let directory = tempfile::tempdir().unwrap(); + let manager = DeviceManager::new(directory.path().into()); + let observation = test_observation(); + manager + .save_recovery(&observation.lock().unwrap().recovery) + .unwrap(); + let (mut writer, reader) = tokio::io::duplex(4096); + let (sender, receiver) = oneshot::channel(); + let task = tokio::spawn(monitor_worker_output( + manager.clone(), + reader, + 0, + Arc::new(AtomicBool::new(true)), + sender, + observation.clone(), + )); + for bytes in input.chunks(fragment) { + writer.write_all(bytes).await.unwrap(); + } + drop(writer); + tokio::time::timeout(Duration::from_secs(2), task) + .await + .unwrap() + .unwrap(); + let readiness = receiver.await.unwrap(); + (manager, observation, readiness, directory) + } + + const SPAWN_RECEIPT: &str = r#"{"kind":"spawned","engine":"scope-java-v1","pid":42}"#; + const READY_RECEIPT: &str = r#"{"kind":"ready","engine":"scope-java-v1","pid":42,"profiles":["proxy"],"capabilities":{"proxy":["java-url"]}}"#; + const STOP_RECEIPT: &str = r#"{"kind":"stopped","engine":"scope-java-v1","pid":42,"spawnAttempted":true,"cleanupComplete":true,"processTerminated":true}"#; + + fn padded_receipt(receipt: &str, size: usize) -> String { + let prefix = format!("{},\"padding\":\"", receipt.strip_suffix('}').unwrap()); + format!("{prefix}{}\"}}", "x".repeat(size - prefix.len() - 2)) + } + + #[tokio::test] + async fn oversized_stdout_is_terminal_even_before_newline() { + let directory = tempfile::tempdir().unwrap(); + let manager = DeviceManager::new(directory.path().into()); + let observation = test_observation(); + manager + .save_recovery(&observation.lock().unwrap().recovery) + .unwrap(); + let (mut writer, reader) = tokio::io::duplex(4096); + let (sender, receiver) = oneshot::channel(); + let task = tokio::spawn(monitor_worker_output( + manager.clone(), + reader, + 0, + Arc::new(AtomicBool::new(true)), + sender, + observation.clone(), + )); + writer.write_all(&vec![b'x'; 65537]).await.unwrap(); + let ready = tokio::time::timeout(Duration::from_millis(300), receiver).await; + task.abort(); + assert!( + matches!(ready, Ok(Ok(Err(_)))), + "oversize must fail without waiting for LF/EOF: {ready:?}" + ); + assert!(manager.recovery_path().is_file()); + assert!(observation.lock().unwrap().confirm_stopped().is_err()); + } + + #[tokio::test] + async fn oversized_stdout_cannot_become_ready_or_confirm_cleanup() { + for oversized in [ + serde_json::json!({"kind":"log", "message":"x".repeat(1024 * 1024)}).to_string(), + padded_receipt(READY_RECEIPT, 65537), + ] { + let input = format!("{SPAWN_RECEIPT}\n{oversized}\n{READY_RECEIPT}\n{STOP_RECEIPT}"); + let (manager, observation, readiness, _directory) = + monitor_fixture(input.into_bytes(), 997).await; + assert!( + readiness.is_err(), + "oversize may not be parsed as a valid receipt" + ); + let observed = observation.lock().unwrap(); + assert!( + observed.stopped, + "bounded draining must resume at the next line" + ); + assert!(observed.confirm_stopped().is_err()); + assert!(manager.recovery_path().is_file()); + let saved: Recovery = + serde_json::from_slice(&fs::read(manager.recovery_path()).unwrap()).unwrap(); + assert!(saved.target_spawn_pending); + assert_eq!(saved.target_pid, Some(42)); + assert!(manager.status().logs.iter().all(|line| line.len() <= 4096)); + } + } + + #[tokio::test] + async fn invalid_utf8_after_stopped_retains_failure() { + let mut input = format!("{SPAWN_RECEIPT}\n{READY_RECEIPT}\n{STOP_RECEIPT}\n").into_bytes(); + input.extend_from_slice(b"\xff\n"); + let (_, observation, readiness, _directory) = monitor_fixture(input, 17).await; + assert!(readiness.is_ok()); + assert!(observation.lock().unwrap().confirm_stopped().is_err()); + } + + #[tokio::test] + async fn exact_frame_limit_preserves_fragmented_crlf_and_final_receipts() { + let ready = padded_receipt(READY_RECEIPT, 65536); + let input = format!("{SPAWN_RECEIPT}\r\n{ready}\r\n{{\"kind\":\"log\",\"message\":\"é😀\"}}\r\n{STOP_RECEIPT}"); + let (manager, observation, readiness, _directory) = + monitor_fixture(input.into_bytes(), 7).await; + assert!(readiness.is_ok()); + observation.lock().unwrap().confirm_stopped().unwrap(); + assert_eq!(manager.status().logs, ["é😀"]); + } + + #[tokio::test] + async fn final_stop_receipt_limit_applies_with_lf_crlf_and_eof() { + for ending in ["\n", "\r\n", ""] { + for (size, accepted) in [(65536, true), (65537, false)] { + let stopped = padded_receipt(STOP_RECEIPT, size); + let input = format!("{SPAWN_RECEIPT}\n{READY_RECEIPT}\n{stopped}{ending}"); + let (_, observation, readiness, _directory) = + monitor_fixture(input.into_bytes(), 1023).await; + assert!(readiness.is_ok()); + assert_eq!( + observation.lock().unwrap().confirm_stopped().is_ok(), + accepted, + "size={size}, ending={ending:?}" + ); + } + } + } + + #[tokio::test] + async fn stderr_discards_one_oversized_line_and_resumes_utf8_diagnostics() { + let directory = tempfile::tempdir().unwrap(); + let manager = DeviceManager::new(directory.path().into()); + let (mut writer, reader) = tokio::io::duplex(4096); + let task = tokio::spawn(monitor_worker_stderr(manager.clone(), reader, 0)); + let mut input = vec![b'x'; 1024 * 1024]; + input.extend_from_slice(b"\n\xff\n"); + input.extend_from_slice("é😀\r\nlast diagnostic".as_bytes()); + for bytes in input.chunks(997) { + writer.write_all(bytes).await.unwrap(); + } + drop(writer); + tokio::time::timeout(Duration::from_secs(2), task) + .await + .unwrap() + .unwrap(); + let logs = manager.status().logs; + assert_eq!( + logs.len(), + 4, + "overflow must produce one diagnostic, not one per fragment" + ); + assert!(logs[0].contains("exceeded 65536")); + assert!(logs[1].contains("Invalid UTF-8")); + assert_eq!(&logs[2..], ["é😀", "last diagnostic"]); + assert!(logs.iter().all(|line| line.len() <= 4096)); + manager.inner.worker_generation.store(1, Ordering::SeqCst); + monitor_worker_stderr(manager.clone(), b"stale\n".as_slice(), 0).await; + assert_eq!(manager.status().logs, logs); + } + + #[tokio::test] + async fn stderr_no_newline_overflow_is_reported_before_eof() { + let directory = tempfile::tempdir().unwrap(); + let manager = DeviceManager::new(directory.path().into()); + let (mut writer, reader) = tokio::io::duplex(4096); + let task = tokio::spawn(monitor_worker_stderr(manager.clone(), reader, 0)); + writer.write_all(&vec![b'x'; 65537]).await.unwrap(); + tokio::time::timeout(Duration::from_secs(2), async { + while manager.status().logs.is_empty() { + tokio::task::yield_now().await; + } + }) + .await + .unwrap(); + assert_eq!(manager.status().logs.len(), 1); + writer.write_all(&vec![b'x'; 1024 * 1024]).await.unwrap(); + drop(writer); + tokio::time::timeout(Duration::from_secs(2), task) + .await + .unwrap() + .unwrap(); + assert_eq!(manager.status().logs.len(), 1); + } + + #[tokio::test] + async fn bounded_reader_keeps_fixed_storage_when_discarding_without_newline() { + let (mut writer, reader) = tokio::io::duplex(4096); + let writing = tokio::spawn(async move { + for _ in 0..256 { + writer.write_all(&[b'x'; 8192]).await.unwrap(); + } + }); + let mut lines = WorkerLines::new(reader); + assert!(matches!( + lines.next_line().await.unwrap(), + Some(WorkerLine::Oversized) + )); + assert!(lines.bytes.len() <= 65537); + assert!(lines.bytes.capacity() <= 65537); + assert!(lines.next_line().await.unwrap().is_none()); + writing.await.unwrap(); + assert!(lines.bytes.capacity() <= 65537); + } + + struct ReadThenError { + bytes: Vec, + offset: usize, + } + impl tokio::io::AsyncRead for ReadThenError { + fn poll_read( + self: std::pin::Pin<&mut Self>, + _context: &mut std::task::Context<'_>, + buffer: &mut tokio::io::ReadBuf<'_>, + ) -> std::task::Poll> { + let reader = self.get_mut(); + if reader.offset == reader.bytes.len() { + return std::task::Poll::Ready(Err(std::io::Error::other("fixture read failure"))); + } + let end = (reader.offset + buffer.remaining()).min(reader.bytes.len()); + buffer.put_slice(&reader.bytes[reader.offset..end]); + reader.offset = end; + std::task::Poll::Ready(Ok(())) + } + } + + #[tokio::test] + async fn stdout_io_failure_after_stopped_cannot_confirm_cleanup() { + let directory = tempfile::tempdir().unwrap(); + let manager = DeviceManager::new(directory.path().into()); + let observation = test_observation(); + manager + .save_recovery(&observation.lock().unwrap().recovery) + .unwrap(); + let input = format!("{SPAWN_RECEIPT}\n{READY_RECEIPT}\n{STOP_RECEIPT}\n"); + let (sender, receiver) = oneshot::channel(); + monitor_worker_output( + manager.clone(), + ReadThenError { + bytes: input.into_bytes(), + offset: 0, + }, + 0, + Arc::new(AtomicBool::new(true)), + sender, + observation.clone(), + ) + .await; + assert!(receiver.await.unwrap().is_ok()); + assert!(observation.lock().unwrap().confirm_stopped().is_err()); + assert!(manager.recovery_path().is_file()); + assert!(manager + .status() + .logs + .iter() + .any(|line| line.contains("fixture read failure"))); + } + + #[test] + fn old_recovery_defaults_do_not_invent_target_ownership() { + let state: Recovery = serde_json::from_str(r#"{"serial":"fixture-01","package":"com.scope.demo","mode":"proxy","port":17890,"previousProxy":null,"previousReverse":null,"forwardPort":null,"serverPid":null,"caHash":"1234abcd","rootShell":true}"#).unwrap(); + assert!(!state.target_spawn_pending); + assert_eq!(state.target_pid, None); + assert_eq!(state.ca_hash.as_deref(), Some("1234abcd")); + } + + #[test] + fn explicit_profiles_remain_exact_and_defaults_are_compatible() { + assert_eq!( + normalize_profiles(vec![]).unwrap(), + ["proxy", "certificate", "unpinning"] + ); + assert_eq!( + normalize_profiles(vec!["proxy".into(), "proxy".into()]).unwrap(), + ["proxy"] + ); + assert_eq!( + normalize_profiles(vec!["certificate".into()]).unwrap(), + ["certificate"] + ); + assert_eq!( + normalize_profiles(vec!["certificate".into(), "proxy".into()]).unwrap(), + ["proxy", "certificate"] + ); + } + + #[cfg(unix)] + #[tokio::test] + async fn worker_stop_drains_and_requires_matching_successful_receipts() { + // Run actual child pipes and the production monitor/teardown, including final-line races. + let owned = r#"{"kind":"stopped","engine":"scope-java-v1","pid":42,"spawnAttempted":true,"cleanupComplete":true,"processTerminated":true}"#; + let safe = r#"{"kind":"stopped","engine":"scope-java-v1","pid":null,"spawnAttempted":false,"cleanupComplete":true,"processTerminated":false}"#; + let spawned = r#"{"kind":"spawned","engine":"scope-java-v1","pid":42}"#; + let wrong = owned.replace("42", "43"); + let error = r#"{"kind":"error","message":"cleanup uncertain"}"#; + for (spawn, receipt, exit, success) in [ + (spawned, owned.to_owned(), 0, true), + ("", safe.to_owned(), 0, true), + (spawned, "".into(), 0, false), + (spawned, wrong, 0, false), + (spawned, safe.to_owned(), 0, false), + (spawned, owned.to_owned(), 1, false), + (spawned, format!("{error}\n{owned}"), 0, false), + ] { + let directory = tempfile::tempdir().unwrap(); + let manager = DeviceManager::new(directory.path().into()); + let observation = test_observation(); + manager + .save_recovery(&observation.lock().unwrap().recovery) + .unwrap(); + let before_stop = if spawn.is_empty() { + String::new() + } else { + format!("printf '%s\\n' {}; ", adb::quote(spawn)) + }; + let script = format!( + "{before_stop}read reply; printf '%s\\n' {}; exit {exit}", + adb::quote(&receipt) + ); + let mut child = Command::new("sh") + .arg("-c") + .arg(script) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .spawn() + .unwrap(); + let stdout = child.stdout.take().unwrap(); + let live = Arc::new(AtomicBool::new(true)); + let (sender, _receiver) = oneshot::channel(); + let monitor = tokio::spawn(monitor_worker_output( + manager.clone(), + stdout, + 0, + live.clone(), + sender, + observation.clone(), + )); + let mut worker = Worker { + child, + live, + observation, + monitor, + }; + let result = manager.finish_worker(&mut worker).await; + assert_eq!(result.is_ok(), success, "{receipt}: {result:?}"); + let saved: Recovery = + serde_json::from_slice(&fs::read(manager.recovery_path()).unwrap()).unwrap(); + assert!( + saved.target_spawn_pending, + "receipt alone must not clear recovery before exit/drain validation" + ); + if !spawn.is_empty() { + assert_eq!(saved.target_pid, Some(42)); + } + } + } + + #[cfg(unix)] + #[tokio::test] + async fn full_worker_cleanup_budget_is_allowed_before_killing() { + let directory = tempfile::tempdir().unwrap(); + let manager = DeviceManager::new(directory.path().into()); + let observation = test_observation(); + manager + .save_recovery(&observation.lock().unwrap().recovery) + .unwrap(); + let mut child = Command::new("sh").args(["-c", "read reply; sleep 9; printf '%s\\n' '{\"kind\":\"stopped\",\"engine\":\"scope-java-v1\",\"pid\":null,\"spawnAttempted\":false,\"cleanupComplete\":true,\"processTerminated\":false}'"]).stdin(Stdio::piped()).stdout(Stdio::piped()).spawn().unwrap(); + let stdout = child.stdout.take().unwrap(); + let live = Arc::new(AtomicBool::new(true)); + let (sender, _receiver) = oneshot::channel(); + let monitor = tokio::spawn(monitor_worker_output( + manager.clone(), + stdout, + 0, + live.clone(), + sender, + observation.clone(), + )); + let mut worker = Worker { + child, + live, + observation, + monitor, + }; + manager.finish_worker(&mut worker).await.unwrap(); + } + + #[cfg(unix)] + #[tokio::test] + async fn recovery_only_accepts_observed_known_pid_absence_and_preserves_legacy_ca_cleanup() { + use std::os::unix::fs::PermissionsExt; + for (pid, proc_state, succeeds) in [ + (Some(42), "ABSENT", true), + (Some(42), "PRESENT", false), + (Some(42), "UNKNOWN", false), + (None, "ABSENT", false), + ] { + let directory = tempfile::tempdir().unwrap(); + let log = directory.path().join("commands.log"); + let adb_path = directory.path().join("fake-adb"); + fs::write(&adb_path, format!("#!/bin/sh\nprintf '%s\\n' \"$*\" >> {}\ncase \"$*\" in\n*'/proc/42'*) echo {proc_state};;\n*) exit 0;;\nesac\n", adb::quote(&log.to_string_lossy()))).unwrap(); + fs::set_permissions(&adb_path, fs::Permissions::from_mode(0o755)).unwrap(); + let mut manager = DeviceManager::new(directory.path().into()); + Arc::get_mut(&mut manager.inner).unwrap().adb = adb_path; + let mut state = test_observation().lock().unwrap().recovery.clone(); + state.target_pid = pid; + state.ca_hash = Some("1234abcd".into()); + manager.save_recovery(&state).unwrap(); + assert_eq!(manager.stop().await.is_ok(), succeeds); + assert_eq!(manager.recovery_path().exists(), !succeeds); + let commands = fs::read_to_string(&log).unwrap_or_default(); + assert!(!commands.contains("kill 42")); + assert!( + !commands.contains("pidof com.scope.demo"), + "package PID lookup cannot prove target termination" + ); + if succeeds { + assert!(commands.contains("rm -f /system/etc/security/cacerts/1234abcd.0")); + } else { + assert!(!commands.contains("rm -f")); + } + } + } + + #[cfg(unix)] + #[tokio::test] + async fn inaccessible_pid_listed_in_procfs_is_not_treated_as_absent() { + use std::os::unix::fs::PermissionsExt; + let directory = tempfile::tempdir().unwrap(); + for (name, source) in [ + ("id", "#!/bin/sh\necho 0\n".to_owned()), + ("ls", "#!/bin/sh\nprintf '1\\n4294967295\\n'\n".to_owned()), + ( + "adb", + format!( + "#!/bin/sh\nPATH={}:$PATH\nexport PATH\nexec sh -c \"$4\"\n", + adb::quote(&directory.path().to_string_lossy()) + ), + ), + ] { + let path = directory.path().join(name); + fs::write(&path, source).unwrap(); + fs::set_permissions(path, fs::Permissions::from_mode(0o755)).unwrap(); + } + let mut manager = DeviceManager::new(directory.path().into()); + Arc::get_mut(&mut manager.inner).unwrap().adb = directory.path().join("adb"); + let mut state = test_observation().lock().unwrap().recovery.clone(); + state.target_pid = Some(u32::MAX); + assert!(manager.reconcile_target(&state).await.is_err(), "a visible directory entry is evidence of presence even if stat cannot observe the target"); + } + + #[tokio::test] + async fn terminal_error_cannot_be_revived_by_ready_or_late_spawn() { + let directory = tempfile::tempdir().unwrap(); + let manager = DeviceManager::new(directory.path().into()); + let observation = test_observation(); + manager + .save_recovery(&observation.lock().unwrap().recovery) + .unwrap(); + let (mut writer, reader) = tokio::io::duplex(4096); + let (sender, receiver) = oneshot::channel(); + let live = Arc::new(AtomicBool::new(true)); + let task = tokio::spawn(monitor_worker_output( + manager.clone(), + reader, + 0, + live.clone(), + sender, + observation.clone(), + )); + writer.write_all(b"{\"kind\":\"error\",\"message\":\"failed\"}\n{\"kind\":\"spawned\",\"engine\":\"scope-java-v1\",\"pid\":42}\n{\"kind\":\"ready\",\"engine\":\"scope-java-v1\",\"pid\":42,\"profiles\":[\"proxy\"],\"capabilities\":{\"proxy\":[\"java-url\"]}}\n").await.unwrap(); + drop(writer); + assert!(receiver.await.unwrap().is_err()); + task.await.unwrap(); + assert!(!live.load(Ordering::SeqCst)); + assert!(!observation.lock().unwrap().spawned); + let saved: Recovery = + serde_json::from_slice(&fs::read(manager.recovery_path()).unwrap()).unwrap(); + assert!(saved.target_spawn_pending); + assert_eq!(saved.target_pid, None); + } + + #[tokio::test] + async fn unsupported_profiles_fail_before_proxy_or_device_probes() { + for profiles in [ + vec!["root"], + vec!["system-ca"], + vec!["unknown"], + vec!["unpinning"], + ] { + let directory = tempfile::tempdir().unwrap(); + let manager = DeviceManager::new(directory.path().into()); + let error = manager + .start( + "fixture-01".into(), + "com.scope.demo".into(), + "frida".into(), + 1, + directory.path().join("missing-ca"), + profiles.into_iter().map(String::from).collect(), + ) + .await + .unwrap_err() + .to_string(); + assert!( + error.contains("profile") || error.contains("requires certificate"), + "{error}" + ); + assert!(!manager.recovery_path().exists()); + } + } + + #[test] + fn materialized_worker_can_construct_only_the_owned_engine() { + let directory = tempfile::tempdir().unwrap(); + let manager = DeviceManager::new(directory.path().into()); + manager.materialize().unwrap(); + assert!(directory.path().join("frida/scope-engine.js").is_file()); + assert!(directory + .path() + .join("frida/frida-java-bridge.js") + .is_file()); + assert!(!directory.path().join("frida/config-template.js").exists()); + assert!(!directory.path().join("frida/android").exists()); + } + + #[tokio::test] + async fn invalid_readiness_never_confirms_startup() { + for receipt in [ + r#"{"kind":"ready"}"#, + r#"{"kind":"ready","engine":"other","pid":42,"profiles":["proxy"],"capabilities":{"proxy":["java-url"]}}"#, + r#"{"kind":"ready","engine":"scope-java-v1","pid":43,"profiles":["proxy"],"capabilities":{"proxy":["java-url"]}}"#, + r#"{"kind":"ready","engine":"scope-java-v1","pid":42,"profiles":["certificate"],"capabilities":{"certificate":["sslcontext-scope-ca"]}}"#, + r#"{"kind":"ready","engine":"scope-java-v1","pid":42,"profiles":["proxy"],"capabilities":{"proxy":[]}}"#, + ] { + let directory = tempfile::tempdir().unwrap(); + let manager = DeviceManager::new(directory.path().into()); + let observation = test_observation(); + manager + .save_recovery(&observation.lock().unwrap().recovery) + .unwrap(); + let (mut writer, reader) = tokio::io::duplex(4096); + let (sender, receiver) = oneshot::channel(); + let task = tokio::spawn(monitor_worker_output( + manager, + reader, + 0, + Arc::new(AtomicBool::new(true)), + sender, + observation, + )); + writer.write_all(format!("{{\"kind\":\"spawned\",\"engine\":\"scope-java-v1\",\"pid\":42}}\n{receipt}\n").as_bytes()).await.unwrap(); + drop(writer); + assert!(receiver.await.unwrap().is_err(), "{receipt}"); + task.await.unwrap(); + } + } + + #[tokio::test] + async fn late_generation_cannot_replace_or_recreate_recovery() { + for saved in [true, false] { + let directory = tempfile::tempdir().unwrap(); + let manager = DeviceManager::new(directory.path().into()); + manager.inner.worker_generation.store(2, Ordering::SeqCst); + if saved { + manager + .save_recovery(&test_observation().lock().unwrap().recovery) + .unwrap(); + } + let before = fs::read(manager.recovery_path()).ok(); + let (mut writer, reader) = tokio::io::duplex(4096); + let (sender, _receiver) = oneshot::channel(); + let task = tokio::spawn(monitor_worker_output( + manager.clone(), + reader, + 1, + Arc::new(AtomicBool::new(true)), + sender, + test_observation(), + )); + writer + .write_all(b"{\"kind\":\"spawned\",\"engine\":\"scope-java-v1\",\"pid\":42}\n") + .await + .unwrap(); + drop(writer); + task.await.unwrap(); + assert_eq!(fs::read(manager.recovery_path()).ok(), before); + } + } + #[cfg(unix)] #[tokio::test] async fn cleanup_checks_live_mapping_ownership_before_mutations() { @@ -1175,6 +2167,8 @@ mod tests { server_pid: None, ca_hash: None, root_shell: false, + target_spawn_pending: false, + target_pid: None, }) .unwrap(); manager.stop().await.unwrap(); @@ -1226,8 +2220,12 @@ mod tests { 1, live.clone(), sender, + test_observation(), )); - writer.write_all(b"{\"kind\":\"ready\"}\n").await.unwrap(); + manager + .save_recovery(&test_observation().lock().unwrap().recovery) + .unwrap(); + writer.write_all(b"{\"kind\":\"spawned\",\"engine\":\"scope-java-v1\",\"pid\":42}\n{\"kind\":\"ready\",\"engine\":\"scope-java-v1\",\"pid\":42,\"profiles\":[\"proxy\"],\"capabilities\":{\"proxy\":[\"java-url\"]}}\n").await.unwrap(); assert!(receiver.await.unwrap().is_ok()); drop(writer); task.await.unwrap(); @@ -1235,6 +2233,35 @@ mod tests { assert!(!manager.status().running); assert!(manager.status().message.contains("press Stop")); } + #[test] + fn stderr_waiting_for_status_lock_cannot_append_to_a_new_generation() { + let directory = tempfile::tempdir().unwrap(); + let manager = DeviceManager::new(directory.path().into()); + let mut status = manager.inner.status.lock().unwrap(); + let (started, waiting) = std::sync::mpsc::channel(); + let old_worker = manager.clone(); + let writer = std::thread::spawn(move || { + started.send(()).unwrap(); + old_worker.log_worker(0, "old worker stderr"); + }); + waiting.recv().unwrap(); + // Hold the actual status lock while the old worker reaches its append boundary. + std::thread::sleep(Duration::from_millis(20)); + manager.inner.worker_generation.store(1, Ordering::SeqCst); + *status = DeviceStatus { + message: "new session".into(), + ..Default::default() + }; + drop(status); + writer.join().unwrap(); + assert!( + manager.status().logs.is_empty(), + "stale stderr must not enter the replacement status" + ); + manager.log_worker(1, "current worker stderr"); + assert_eq!(manager.status().logs, ["current worker stderr"]); + } + #[tokio::test] async fn late_worker_eof_cannot_overwrite_a_new_session_status() { let directory = tempfile::tempdir().unwrap(); @@ -1249,6 +2276,7 @@ mod tests { 1, Arc::new(AtomicBool::new(true)), sender, + test_observation(), )); drop(writer); task.await.unwrap(); @@ -1266,6 +2294,7 @@ mod tests { 0, Arc::new(AtomicBool::new(true)), sender, + test_observation(), )); writer .write_all(b"{\"kind\":\"detached\",\"message\":\"process-terminated\"}\n") @@ -1310,6 +2339,8 @@ mod tests { server_pid: None, ca_hash: None, root_shell: false, + target_spawn_pending: false, + target_pid: None, }; let mut manager = DeviceManager::new(directory.path().to_owned()); Arc::get_mut(&mut manager.inner).unwrap().adb = directory.path().join("missing-adb"); @@ -1335,6 +2366,8 @@ mod tests { server_pid: None, ca_hash: None, root_shell: false, + target_spawn_pending: false, + target_pid: None, }; manager.save_recovery(&state).unwrap(); let error = manager.stop().await.unwrap_err().to_string(); diff --git a/docs/ANDROID.md b/docs/ANDROID.md index 3d9acc7..300e827 100644 --- a/docs/ANDROID.md +++ b/docs/ANDROID.md @@ -1,14 +1,14 @@ # Android interception -Scope's Android adapter uses its own localhost proxy and public CA. HTTP Toolkit does not need to be installed or running. +Scope's Android adapter uses its own localhost proxy, public CA and experimental Java interception engine on Frida. The retained Java bridge has separate third-party provenance. [Engine coverage and evidence](ANDROID_ENGINE.md). ## What is implemented - Real ADB device enumeration, including offline/unauthorized states, model and a read-only root check. - Installed packages and running state. If an existing Frida server is available, app display labels come from Frida; otherwise the exact package ID is shown as the name. -- **System proxy:** Android's global HTTP proxy, a matching ADB reverse mapping and, on rooted devices, temporary system CA injection. The selected application is restarted. This mode is **device-wide**, so traffic from other apps may appear. -- **Frida:** a separate Python worker spawns and attaches to the selected application. Bundled JavaScript profiles configure the app proxy, inject Scope's CA, disable supported certificate pinning and optionally hide selected root indicators. -- Stop restores the previous Android proxy and reverse mapping, detaches hooks, removes Scope's temporary CA and stops only a server process spawned by Scope. +- **System proxy:** Android's global HTTP proxy and a matching ADB reverse mapping. The selected application is restarted. This mode is **device-wide**, so traffic from other apps may appear. HTTPS requires manually configured application trust; Scope does not install a system CA. +- **Frida:** a separate Python worker spawns and attaches to the selected application. Original Scope Java hooks configure supported proxy seams, add trust for chains validated against Scope's CA, and conditionally bypass supported OkHttp3 pins for those chains. Hostname verification remains active. +- **Frida Stop ends the exact instrumented app process**, unloads/detaches and restores recorded owned mappings/server state. System proxy Stop restores recorded proxy/reverse state. Older recovery records retain cleanup of their Scope-owned CA. - A saved recovery file allows Stop to retry after a disconnected device or interrupted session. Starting another session is refused until that cleanup succeeds. - The 1.0 **Diagnostics** workspace and `device_diagnostics` MCP tool provide read-only authorization/root/ABI/runtime/mapping/recovery preflight with actionable fixes. See [diagnostic boundaries and troubleshooting](DIAGNOSTICS.md). @@ -104,14 +104,15 @@ The locally tested optional worker used CPython 3.12.13 from python-build-standa ### Hook profiles -| ID | Behavior | -| ------------- | ----------------------------------------------------------------------------- | -| `proxy` | Redirect supported Java HTTP proxy configuration to Scope. | -| `certificate` | Inject Scope's public CA into supported Java trust managers. | -| `unpinning` | Disable supported Java certificate pinning checks, including a fallback hook. | -| `root` | Hide selected root indicators from the instrumented app. | +| ID | Supported behavior | +| --- | --- | +| `proxy` | Subsequent Java URL HTTP(S) connections and available OkHttp3 builders use the Scope listener. | +| `certificate` | Subsequent SSLContext initialization wraps original X509 trust managers; separate Java PKIX validation admits only valid chains to the supplied Scope CA. | +| `unpinning` | Ordinary Java OkHttp3 CertificatePinner List and Certificate[] overloads may accept a Scope-validated chain after an original rejection; acceptance target 3.14.9. Requires `certificate`. Lazy Kotlin `check$okhttp` configurations fail before readiness. | -An empty profile array selects `proxy`, `certificate`, `unpinning`. Explicit profiles select exactly those hooks. Root hiding is opt-in. The UI and worker report hook startup diagnostics; readiness means the Java callbacks ran, not that every possible network stack is covered. +An omitted/empty API or CLI profile array selects `proxy`, `certificate`, `unpinning` for compatibility. Explicit selections remain exact. The UI disables Start for an empty selection; deselecting certificate also deselects/disables unpinning. `root`, `system-ca` and unknown profiles fail before device probes. No root hiding is provided. + +Readiness requires the exact engine, owned PID, requested profiles and installed capability receipts. An unavailable supported OkHttp pinner causes default-all startup to fail; choose explicit proxy/certificate profiles for the supported available seams. A callback alone cannot establish readiness or network coverage. ### Compatibility boundaries @@ -119,19 +120,19 @@ The source repository recorded Android 16 ART/GC crashes with Frida 17.9.1. This The bundled profiles target **Java** networking. Native TLS implementations, QUIC/HTTP3, child processes, anti-instrumentation and app-specific pinning may need additional work. Do not infer complete capture from an empty request list. -The version pin is the imported repository's integration baseline, not a claim that it supports every current Android release. Python/server upgrades should be tested together with the Java bridge bundle. +The Frida version pin is the existing integration baseline, not a claim that it supports every current Android release. Python/server upgrades should be tested together with the Java bridge bundle. ## Cleanup and recovery -Scope writes `device-recovery.json` under its runtime directory before changing the device. It contains the selected serial/package, previous proxy/reverse state, temporary CA hash and owned forwarding/server identifiers. +Scope writes `device-recovery.json` under its runtime directory before changing the device. It contains the selected serial/package, previous proxy/reverse state, legacy temporary CA hash, owned forwarding/server identifiers and pending spawned-process cleanup/PID. The spawned PID is persisted before readiness; an attempted spawn without a returned PID remains explicitly uncertain. Press Stop while the device is connected. If cleanup fails, the file remains and the log explains that reconnection is needed. Reconnect the **same device**, then press Stop again. Do not delete the recovery file as a substitute for restoring device settings. Stop verifies that current proxy and ADB mappings still match Scope's recorded values before restoring them. External changes are preserved and logged. Invalid saved fields are rejected before cleanup commands; failed inspection leaves recovery available for a retry. -Rooted CA injection uses temporary mount overlays, following the upstream Android 14+ namespace handling. Stop removes Scope's certificate from those shared stores and the selected app. The remaining mount overlays disappear on reboot. The adapter does not unmount shared trust stores underneath other apps. +New sessions never inject system CAs. For older recovery records only, Stop still removes the recorded Scope-owned certificate from shared stores and the selected app. Existing temporary mount overlays disappear on reboot; Scope does not unmount shared trust stores underneath other apps. -Frida detaches hooks on Stop and leaves the selected app running. Restart that app to return its full runtime state to a clean baseline. A server binary downloaded/copied for Scope remains available for the next session; its process is stopped. +Frida Stop terminates the owned app process because created client/TLS objects can outlive method unhooking. The worker has 12 seconds for bounded cleanup, followed by stdout drain. A successful worker exit alone cannot prove termination: Rust requires a matching successful acknowledgment. Failed/timed-out workers and missing acknowledgments retain recovery. After a restart, only observed absence of the recorded PID in a readable root procfs view allows automatic reconciliation. A present or inaccessible PID remains an error and is never killed from saved state. Unknown attempted-spawn completion requires manual process/device reconciliation; Scope cannot safely discover it by package name. Preserve the recovery file until that reconciliation and owned setting/mapping restoration are complete. A server binary downloaded/copied for Scope remains for the next session; only its owned process is stopped. ## Local Android demo @@ -166,10 +167,12 @@ Set `SCOPE_DEVICE_RUNTIME` to an absolute directory to keep the smoke-test recov ## Source and licenses -The JavaScript, bridge and original CA template are preserved with their original copyright notices. See [THIRD_PARTY_NOTICES.md](../THIRD_PARTY_NOTICES.md) and the included license files. Runtime config substitutes Scope's own CA and selected proxy port; an installed HTTP Toolkit service or certificate is never read. +The runtime uses original `scope-engine.js` plus the unchanged Frida Java bridge. Six imported HTTP Toolkit hook/config files and the CA installation template were removed; source history and retained license/provenance notices remain. Runtime data is JSON-serialized. This migration changes neither the project AGPL license nor the existing Rust proxy/CA/upstream verification. See [THIRD_PARTY_NOTICES.md](../THIRD_PARTY_NOTICES.md). ## Recorded local smoke test +**Historical backend receipt:** the following 2026-11-02 results used the removed hook backend and CA-injection path. They do not validate the current Scope-owned engine or its process-ending Stop contract. Current real Android/Frida/OkHttp TLS and Stop acceptance: **NOT RUN**. + On 2026-11-02, the macOS build was checked against an existing rooted Android 16 arm64 emulator using only the included `com.scope.demo` application: - The APK built offline, signed, installed and launched successfully. diff --git a/docs/ANDROID_ENGINE.md b/docs/ANDROID_ENGINE.md new file mode 100644 index 0000000..fe584e2 --- /dev/null +++ b/docs/ANDROID_ENGINE.md @@ -0,0 +1,50 @@ +# Scope-owned Android Java engine + +The experimental `scope-java-v1` engine uses documented Frida/Java APIs and the unchanged third-party Frida Java bridge. It replaces the imported HTTP Toolkit runtime hooks. The project remains AGPL-3.0-or-later; [notices and bridge provenance](../THIRD_PARTY_NOTICES.md) remain applicable. No legally certified clean-room claim is made. + +## Coverage + +| Profile | Supported seam | Limits | +| --- | --- | --- | +| `proxy` | Java URL openConnection overloads and available OkHttp3 Builder.build | Existing clients, other Java clients and native connections are not covered. | +| `certificate` | Subsequent SSLContext.init, original basic/extended X509 managers | Existing/default contexts are not covered. Original socket/SSLEngine delegation and hostname verification remain active. | +| `unpinning` | Ordinary Java OkHttp3 CertificatePinner List and Certificate[] signatures; acceptance target 3.14.9 | Requires certificate profile; missing/unknown pinner capabilities fail startup. Custom/obfuscated/native pinners are unsupported. | + +A separate empty KeyStore contains only the supplied Scope CA, and a real Java TrustManagerFactory performs chain validation. CA constraints, certificate validity, server-leaf checks and original Java exception types bound acceptance. A valid Scope chain can override an original certificate/pin rejection only at the supported seams; unrelated failures and non-Scope chain decisions remain unchanged. No trust-all manager, hostname bypass or expiry bypass is installed. Native TLS, Cronet, QUIC, Flutter, child processes and root hiding are unsupported. + +System proxy mode remains device-wide and requires manually configured application trust for HTTPS. Neither mode injects a system CA. Legacy recovery records retain removal of their recorded owned CA. + +## Lifecycle + +The worker serializes runtime data as JSON, validates profiles and exactly one complete public X.509 CA before importing Frida or contacting a device and requires every requested capability before ready. Rust persists attempted-spawn uncertainty before launching the worker and the exact spawned PID before accepting ready. Terminal errors/detach/EOF cannot revive readiness; a prior worker generation cannot write a newer session's state. + +Stop ends the exact process spawned by the active worker, then unloads/detaches. This is necessary because created proxy/TLS objects can outlive unhooking. Rust allows 12 seconds for the worker's bounded nine-second cleanup plus scheduling overhead, then drains stdout. A matching successful cleanup/termination receipt and successful worker exit are required before clearing pending cleanup. A receipt proving Stop occurred before any attempted spawn is also accepted without claiming termination. + +A failed/timed-out worker or missing acknowledgment retains recovery. After restart, only observed absence of a saved PID in a readable root procfs view can establish that it ended. Present/inaccessible PID state and unknown attempted-spawn completion require manual reconciliation; saved target PIDs are never killed. Existing mapping ownership checks preserve external proxy/reverse/forward changes. + +## Bounded host diagnostics + +Worker stdout and stderr are incrementally framed at at most 65,536 bytes per line, excluding LF and its optional preceding CR. The reader retains a fixed 65,537-byte frame buffer plus an 8,192-byte input buffer per pipe. It discards oversized remainders through the next LF without growing a buffer. Final non-newline receipts and valid fragmented UTF-8 remain supported. + +Oversized, invalid UTF-8 or failed stdout reads are terminal protocol failures; later readiness/termination messages cannot clear that failure or establish successful cleanup. JSON receipts are never truncated and parsed. Stderr is diagnostic only: oversized/invalid lines produce bounded diagnostics and draining resumes for subsequent lines, with the existing generation isolation. + +Retained log entries contain at most 4,096 UTF-8 bytes including any truncation marker, with at most 200 entries (819,200 stored text bytes). Entries are copied into bounded fresh strings so a shortened entry cannot retain its former large allocation. These are diagnostic framing/storage limits, not a bound on all application memory or a measured Android throughput improvement. Normal Python worker messages remain compatible with the framing limit even when JSON escaping expands 4,000 Unicode characters. + +## Evidence and acceptance + +| Gate | Current evidence | +| --- | --- | +| Java policy and simulated Frida adapter | Executable Node suite covers original validation/pinning controls, capability rollback and supported API wiring. | +| Worker lifecycle and source construction | Executable Python suite covers profiles, JSON config, readiness, cancellation and owned-process cleanup. | +| Rust application lifecycle | Local child pipes and fixture ADB cover typed receipts, output drain, recovery, profile preflight and legacy cleanup. | +| UI | Selection/dependency regressions and production frontend build are local checks. | +| Real Android/Frida/OkHttp TLS and hostname controls | **NOT RUN** | +| Actual app termination/restoration and API/ABI compatibility | **NOT RUN** | + +Simulated tests do not prove Android execution, cryptography or compatibility. Historical emulator observations used the previous backend. The source candidate must remain draft until real-device acceptance is separately recorded. No native UI/device evidence is inferred from a headless build. + +## Owned acceptance fixture + +The [original lab APK and manual matrix](../examples/android-engine-fixture/README.md) target real OkHttp 3.14.9 with a deliberately mismatching pin, fresh platform contexts and ephemeral loopback-only valid/foreign/wrong-host/expired TLS endpoints. Build/signature and host certificate-fixture checks are separate from Android execution. All Android/Frida/TLS/Stop gates remain **NOT RUN**. + +Recognized Kotlin lazy `check$okhttp` configurations are rejected transactionally before unpinning readiness, including classes that also expose ordinary public overloads. Their original-manager chain cleaner can discard the raw chain before the pinner sees it. No lazy support or chain-cleaner override is claimed. Ordinary Java overloads preserve the raw chain and typed original/non-Scope error controls. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 8444f4a..ac85801 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -27,8 +27,8 @@ flowchart LR | `crates/scope-mcp` | stdio MCP protocol and twelve tool schemas; calls the desktop's local bridge | | `src-tauri` | Desktop lifecycle, native dialogs, command dispatch, MCP access policy and local demo server | | `src` | React inspector, sessions, device setup, analysis, comparison and MCP settings | -| `resources/device` | Frida worker, setup helpers and certificate injection template | -| `resources/frida` | Attributed upstream JavaScript hook profiles and Java bridge | +| `resources/device` | Frida worker, runtime setup helpers and retained third-party licenses | +| `resources/frida` | Original Scope Java engine and attributed unchanged Frida Java bridge | | `examples/android-demo` | Small Java/Android localhost test target | | `scripts/smoke.mjs` | Real-process MCP/capture/persistence/redaction smoke test | @@ -60,6 +60,8 @@ is a local operational journal, not a tamper-proof compliance audit system. System proxy mode is device-wide. Frida mode spawns/attaches to the selected package through an external worker and uses supported Java hooks. The worker runs outside the Tauri process to isolate instrumentation failures. It uses a standalone executable when bundled, or a Python fallback. Host and device Frida versions must match. +Frida Stop terminates the exact process spawned by the worker, because created proxy/TLS objects can survive unhooking. Typed capability readiness follows durable spawned-PID recording. A successful exit and matching cleanup receipt are required; worker failure or missing acknowledgment retains target recovery. After restart, only observed absence of a recorded PID in a readable root procfs view can reconcile it automatically; unknown spawn completion requires manual reconciliation. Saved target PIDs are never killed. System proxy mode does not install certificates; legacy owned-CA removal remains supported. + A failed cleanup retains its recovery record and blocks a conflicting new start. Reconnect the same device and retry Stop. Cleanup removes Scope-owned resources and restores recorded settings; it does not stop unrelated Frida servers. [Android details](ANDROID.md) ## MCP boundary @@ -90,6 +92,6 @@ The application settings expose the actual data directory. `--data-dir` or `SCOP ## Verification and design choices -Tests exercise actual local HTTP and CA-signed HTTPS, rather than only proxy parsers. The release smoke launches the actual desktop engine and MCP executable, tests access/redaction, and reopens the same SQLite archive. Native UI and one rooted Android emulator were checked separately. [Recorded results and limits](VERIFICATION.md) +Tests exercise actual local HTTP and CA-signed HTTPS, rather than only proxy parsers. The release smoke launches the actual desktop engine and MCP executable, tests access/redaction, and reopens the same SQLite archive. Native UI and one rooted Android emulator were checked separately for earlier versions. These historical receipts do not validate the current Scope-owned engine; its real-device acceptance remains NOT RUN. [Recorded results and limits](VERIFICATION.md) Scope 1.0.0 does not capture WebSocket frames, HTTP trailers, QUIC/HTTP3 or arbitrary TCP/UDP. Windows/Linux runtime parity and physical Android support need evidence from those environments. New networking-stack support should include a minimal reproducible client/server fixture and a cleanup test. diff --git a/docs/VERIFICATION.md b/docs/VERIFICATION.md index 64d74cf..81f5a8d 100644 --- a/docs/VERIFICATION.md +++ b/docs/VERIFICATION.md @@ -1,5 +1,29 @@ # Recorded verification +## Android worker diagnostic memory follow-up — 2026-11-07 + +The follow-up to the initial reviewed engine head `4edeb09` bounds worker stdout/stderr framing before JSON parsing and retained device logs. See [exact limits and lifecycle behavior](ANDROID_ENGINE.md#bounded-host-diagnostics). + +A reproducible synthetic profile sends 205 diagnostics of 16,384 bytes and retains the latest 200. Stored UTF-8 payload fell from **3,276,800 to 819,200 bytes (75% less)**. The bounded candidate's actual retained String capacities also total 819,200 bytes. A malformed/custom 1 MiB JSON diagnostic is now a terminal stdout protocol failure, rather than a stored large entry. These are host diagnostic-storage measurements, not total RSS, normal-traffic latency, JNI or Android throughput claims. The fixed framing buffers add a known bounded allocation per pipe; no general low-traffic memory improvement is inferred. + +- **PASS:** observed RED→GREEN production regressions for oversized unterminated stdout, over-limit readiness/cleanup receipts, UTF-8 storage boundaries/capacity and invalid UTF-8 after a valid Stop. Previously that read error silently ended the monitor; it now preserves failure/recovery. +- **PASS:** 45 device Rust tests and 94 headless core/device/MCP tests; 16 Java engine adapter tests; 23 Python device tests. Added guards revalidate the same peer/CA after expiry changes and verify exact CA input-size boundaries and reread changed contents. +- **PASS:** ten independent production-path acceptance probes cover fragmented UTF-8, exact/over-limit LF/CRLF/EOF, no-newline overflow, sticky failure despite later receipts, I/O error after Stop, bounded stderr resumption/generation isolation and actual child-pipe draining/cleanup. These copied-source probes are separate from the committed regression suite and are not Android evidence. +- **PASS:** independent scoped source review, formatting and whitespace checks. One unchanged fake-ADB fixture produced `Text file busy`; an isolated check and final full suite passed. No timeout or fixture behavior was weakened; the failure is preserved in the implementation receipt. + +A separate class-wrapper-cache candidate was rejected: the retained Frida bridge already caches successful wrappers, warmed class-factory misses were zero before/after, and bounded Node timing samples were mixed. Fewer Java.use dispatches do not establish a native or Android speedup; no such product change was made. + +The emulator prerequisite investigation progressed beyond the earlier crash site using ordinary private directory preparation, but ADB still aborts because its standard user-storage directory is read-only in this executor. No host security setting, standard home directory or system trust store was changed. Actual Android/Frida/TLS/pinning/hostname/Stop acceptance remains **NOT RUN** and the PR stays draft. Exact final hosted check receipts are recorded in the same PR after publication. + +## Scope-owned Android engine boundary — 2026-11-07 + +The current source replaces the imported hook backend with Scope-owned Java hooks and ends the owned app process on Frida Stop. Local executable Node/Python policy/adapter and Rust lifecycle tests use simulated Java/Frida or local child processes; they do not execute Android TLS. Real Android/Frida/OkHttp TLS, hostname controls, API/ABI compatibility and Stop restoration acceptance are **NOT RUN**. Earlier emulator receipts below describe the previous backend and remain historical. See [current engine coverage](ANDROID_ENGINE.md). + +Local source gates passed on Linux: 84 locked headless Rust tests (core/device/MCP), 56 UI tests plus TypeScript/Vite build, 26 Node engine/license/resource tests, 22 device-worker Python tests, 17 integrity-script tests and 10 fixture-helper tests. The final worker preflight regression was observed failing on malformed DER, relabelled synthetic key DER, trailing DER and a non-CA leaf, then passing with complete single-CA parsing before Frida import. The affected Python and embedded-worker Rust suites were rerun after that fix; other checked component source remained unchanged. + +The owned fixture APK compiled, signed and passed apksigner v2/v3 verification with the real checksum-locked OkHttp 3.14.9/Okio 1.17.2 test dependencies. Host TLS controls accepted the valid fixture CA and rejected foreign, wrong-host and expired peers. These are host/build results, **not Android execution**. One bounded software-emulator prerequisite boot failed with SIGSEGV before Android startup; cause is unestablished. Independent component and whole-branch source reviews resolved all reported findings with covering regressions and scoped re-reviews. The first hosted Windows run exposed CRLF serialization differences after the bounded binary read. Explicit LF/CRLF/CR byte-input regressions reproduced that failure, and universal-newline normalization restored the prior text-read behavior without changing CA validation. Hosted exact-head check receipts belong in the draft PR and are separate from these local results. + + ## Endpoint manual editing candidate — 2026-11-06 Scope 1.3.0 adds a `pause` Traffic rule with request, response or both-stage diff --git a/docs/superpowers/plans/2026-11-07-owned-android-engine.md b/docs/superpowers/plans/2026-11-07-owned-android-engine.md new file mode 100644 index 0000000..6c083cb --- /dev/null +++ b/docs/superpowers/plans/2026-11-07-owned-android-engine.md @@ -0,0 +1,73 @@ +# Scope-owned Android engine Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace HTTP Toolkit runtime hooks with an original CA-scoped Android interception engine in one independently reviewed PR. +**Architecture:** Original JavaScript hook/policy code runs in the selected app through the existing Python/Frida worker. Rust retains proxy/recovery management and materializes only the new assets. Imported system trust injection and unsupported root profiles are removed explicitly. +**Tech Stack:** JavaScript, Python, Rust, existing Frida Java bridge; pinned existing dependencies. +**Spec:** docs/superpowers/specs/2026-11-07-owned-android-engine-design.md + +## Global Constraints + +- Preserve existing Rust proxy/TLS/archive behavior and source history; no license change. +- New runtime implementations must be original, not copied or mechanically translated upstream hooks. +- Supported profiles are `proxy`, `certificate`, `unpinning`; an empty selection means all three. `unpinning` requires `certificate`. +- Unsupported `root`, `system-ca` or unknown profiles fail before any device probes or state mutation. +- No trust-all manager, hostname-verifier bypass, certificate-expiry bypass or global system trust-store mutation is allowed. +- Only validated Scope-CA chains may bypass supported OkHttp pin checks; all other chains keep original checks/exceptions. +- Readiness requires installed requested capabilities. Stop terminates only the exact worker-spawned process; failures remain visible. +- Keep legacy CA removal recovery; document proxy/root/native coverage reductions. +- Public artifacts contain no private context, paths, captures, credentials or machine identifiers. +- Never claim mock tests are Android results. Real unavailable gates are NOT RUN. No merge/release/tags/admin/engine changes. + +## Review Focus + +- Unrelated/expired/malformed chains must not gain trust or bypass pins. +- Partial hook installation and missing overloads must roll back, never report ready. +- Explicit profiles including stale root/system-ca values must fail before device writes. +- Worker cancellation/startup error must not leave an owned instrumented process active or kill unrelated PIDs. +- Old recovery records must still clean only Scope-owned CA/mappings after injection is removed. + +### Task 1: Original hook engine and worker contract + +**Files:** Create `resources/frida/scope-engine.js`, `resources/frida/tests/scope-engine.test.cjs`, `resources/device/tests/test_frida_worker.py`; modify `resources/device/frida_worker.py`. +**Interfaces:** Worker `build_script(root, ca, port, profiles) -> str` remains available. `normalize_profiles(profiles) -> list[str]` is shared by the CLI/builder. New JS exports a testable engine factory/policy boundary and runs through documented Frida `Java` APIs. JSON runtime config uses public CA/loopback proxy/normalized profiles. Readiness payload identifies engine `scope-java-v1`, selected profiles and installed capabilities; Python validates it. Existing Rust JSONL `ready/error/stopped/detached` kinds remain compatible. + +- [x] Write executable JS and Python failing tests for CA-scoped pin decisions, original rejection propagation, original trust delegation, explicit/default/unsupported profiles, missing classes and transactional rollback, hostile config serialization, readiness validation and owned-PID cleanup. +- [x] Observe RED with expected missing implementation or wrong existing behavior; record exact commands/output in report. +- [x] Implement original engine (small focused helpers in one runtime file), worker JSON config construction, capability-aware ready gate and finite owned-process cleanup; preserve CLI listing/version and bounded diagnostics. Use the actual Frida API, not a production test stub. Any unimplemented requested capability is an error. +- [x] Run focused tests plus complete Python device suite; self-review. Do not claim fixture-based adapter tests are device verification. +- [x] Commit only Task1 files and report exact RED/GREEN and remaining limitations. + +### Task 2: Application integration, migration and release verification + +**Files:** Modify `crates/scope-device/src/lib.rs`, `src/App.tsx`, `README.md`, `docs/ANDROID.md`, `THIRD_PARTY_NOTICES.md`, `.github/workflows/checks.yml`, affected existing tests; remove six imported HTK runtime JS/config files and `resources/device/install-system-ca.template.sh`; retain Frida bridge/licenses. Add `docs/ANDROID_ENGINE.md` with explicit coverage/evidence. +**Interfaces:** Consume Task1 `scope-engine.js`, `normalize_profiles` and JSONL engine/capability payload. Rust persists worker-spawned target PID/pending-cleanup before readiness where observable and requires explicit successful termination acknowledgment (or conservative proof that the recorded process no longer exists) before clearing recovery; worker timeout/nonzero/missing acknowledgment must remain an error. New recovery fields are backward-compatible serde defaults. Startup failure allows bounded worker teardown before kill. Rust rejects unsupported profiles before ADB/root probes; ASSETS materializes only `frida/scope-engine.js`, existing Frida bridge and worker. Remove automatic system CA installation, preserve cleanup of old recovery ca_hash. UI removes unsupported root option and explains Stop ends the instrumented process. + +- [x] Add regression tests for early unsupported profile rejection, asset construction and legacy owned-CA recovery behavior where affected; observe RED. +- [x] Wire original engine and remove imported runtime source/injection path. Preserve unrelated Rust proxy and recovery semantics. +- [x] Update docs/notices accurately including retained bridge distribution provenance, current original engine, explicit reduced coverage and device NOT RUN gates; no licensing cleanup beyond truthful component inventory. +- [x] Add the executable JS engine test command to the existing three-platform checks using existing pinned actions. Keep existing checks/security unchanged. +- [x] Run focused Rust device tests, all JS/Python suites, UI tests/build when available, formatting and source/privacy/provenance checks; record unavailable desktop/device gates explicitly. +- [x] Commit and report component evidence for independent spec/quality review; record real-device gates as NOT RUN. + +### Task 3: Honest OkHttp boundary and reproducible Android acceptance fixture + +**Files:** Modify `resources/frida/scope-engine.js`, its executable tests and `docs/ANDROID_ENGINE.md`; create `examples/android-engine-fixture/AndroidManifest.xml`, `src/com/scope/enginefixture/MainActivity.java`, `build.py`, `serve.py`, `README.md`, `.gitignore` and small host-side tests for the helpers. Update notices/plan/spec where needed. A deferred stderr-generation log race may be fixed in `crates/scope-device/src/lib.rs` with a narrow regression. +**Interfaces:** Fixture APK package `com.scope.enginefixture`; explicit local test URLs only (`http[s]://127.0.0.1:/...`), `platform` and real `okhttp` request paths, bounded one-request-at-a-time execution and machine-readable synthetic-only Logcat receipts. Build takes explicit SDK/JDK/output paths, verifies locked real OkHttp3.14.9 + Okio1.17.2 artifacts by SHA-256, preserves their license notices in the APK, and performs actual APK signing/verification. No libraries, APKs, keystores or TLS keys are committed or included in production resources. + +- [x] Add a failing regression showing recognized Kotlin lazy OkHttp cannot be advertised as supported: original-manager cleaning can discard the Scope peer chain before the lazy pinner sees it. Reject configurations containing `check$okhttp` before unpinning readiness, with transactional rollback. Remove the misleading lazy-support claim/code; preserve typed non-Scope/original-error controls for ordinary Java overloads. Do not add an unmeasured chain-cleaner override. +- [x] Add an original owned HTTPS test app using fresh SSLContext.init and real pinned OkHttp3.14.9; a deliberately mismatching pin must remain configured in the fixture. Preserve hostname verification, certificate trust and original request errors. A bounded local HTTP reachability control is separate from actual certificate/pin rejection. +- [x] Build helpers acquire only the two locked small test JARs from exact Maven Central URLs; SDK/JDK are supplied, never installed. Pin OkHttp SHA-256 `2570fab55515cbf881d7a4ceef49fc515490bc027057e666776a2832465aeca0`, Okio SHA-256 `f80ce42d2ffac47ad4c47e1d6f980d604d247ceb1a886705cf4581ab0c9fe2b8`. Reject corrupt downloads before executing build tools, bound download size/time, keep all generated output outside tracked source. +- [x] Provide a loopback-only synthetic TLS fixture generator/server with ephemeral Scope/foreign CAs and valid/wrong-host/expired leaf cases, finite execution/cleanup, no global trust-store change or broad device actions. Private test keys stay in a temporary build/runtime directory; no request bodies, tokens or real account data in reports. +- [x] Document exact no-hooks / certificate-only / certificate+unpinning / Stop+restart manual acceptance matrix and commands for an explicitly selected owned lab app/device. Do not automatically select/install/start a real device. Actual runtime gates remain NOT RUN after the failed software-emulator prerequisite; a compiled APK is not a TLS PASS. +- [x] Observe focused host-helper/engine RED→GREEN, compile a real signed APK with the task-local supplied SDK/JDK, record exact tool/library versions and SHA-256 artifact receipt outside source. Run affected JS/Python/Rust checks if changed, then commit only source. Independent task review covers this component; whole-branch review and draft publication remain separate gates below. + +## Final review and publication gates + +- [x] Obtain independent whole-branch source and requirements review, including cross-component lifecycle, compatibility and provenance. +- [x] Resolve blocking review findings with covering tests and scoped re-review. +- [ ] Record final local and hosted PASS / FAIL / NOT RUN for the exact published head. +- [ ] Create exactly one draft PR after local source review and verify its URL, head, body and state by readback. + +Actual Android/Frida/TLS/unpinning/Stop gates remain NOT RUN after the failed software-emulator prerequisite. The PR stays draft unless these acceptance gates are separately evidenced; successful host fixtures or APK signatures do not replace them. No merge, release or tag operation is part of this plan. diff --git a/docs/superpowers/plans/2026-11-07-owned-engine-bounded-diagnostics.md b/docs/superpowers/plans/2026-11-07-owned-engine-bounded-diagnostics.md new file mode 100644 index 0000000..12087d4 --- /dev/null +++ b/docs/superpowers/plans/2026-11-07-owned-engine-bounded-diagnostics.md @@ -0,0 +1,36 @@ +# Bounded Android worker diagnostics follow-up + +> **For agentic workers:** Use subagent-driven implementation, TDD and independent review for this focused existing-engine optimization. + +**Goal:** Bound host memory consumed by worker stdout/stderr and retained device diagnostics without changing TLS, interception or recovery guarantees. + +**Architecture:** Retain the current Scope Java/Frida engine and JSONL protocol. Bound framing before JSON parsing and bound stored diagnostic bytes. No trust decisions, expiry checks or path validation are cached. + +**Spec:** Existing `docs/superpowers/specs/2026-11-07-owned-android-engine-design.md`; current worker messages are limited to4000 Python characters. Their ASCII JSON escaping can require48000 bytes, so the protocol frame ceiling is65536 bytes excluding LF/optional CR. + +## Evidence and choices + +A synthetic malformed/custom worker retained a1MiB log entry;200 entries of16KiB retained3276800 bytes. Standard Python emit already limits normal messages. This is defensive host-memory control, not an observed Android throughput result or demonstrated TLS bypass. + +A separate fixed class-wrapper-cache candidate reduced repeated Java.use dispatch but the retained bridge already caches wrappers. Warm class-factory misses were zero before and after; Node timings were mixed. Do not apply it or claim Android/JNI speedups. + +## Constraints and review focus + +- Maximum stdout/stderr line:65536 bytes, enforced during incremental reads before JSON parsing; never allocate proportional to an oversized line. +- Stored diagnostic entry: at most4096 UTF8 bytes including any truncation marker; maximum200 entries. Preserve valid UTF8 boundaries. +- Oversized/malformed stdout is a terminal protocol failure and retains recovery. Never truncate and parse readiness or termination receipts. Continue bounded draining/discard through line boundaries so pipes and cleanup do not deadlock. +- Stderr is diagnostic only: report/truncate oversize safely, discard its remainder and resume subsequent lines; retain generation isolation. +- Preserve LF/CRLF and final non-newline receipt behavior, exact matching Stop acknowledgments, output-drain grace and existing pending-PID recovery. I/O/UTF8 errors must not create a successful cleanup acknowledgment. +- Preserve full TLS/CA/pin/hostname/expiry/original-error behavior, support scope, native settings, dependency locks, source history and draft status. No merge, release, dispatch or host-security change. + +## Implementation and validation + +- [x] Add production-path regressions demonstrating oversized log retention and unbounded framing; observe RED. +- [x] Implement a shared bounded incremental reader and byte-limited retained diagnostics in `crates/scope-device/src/lib.rs`; keep changes focused. +- [x] Verify exact/over-limit frames, fragmented and no-newline input, UTF8/CRLF, recovery after oversize, terminal oversized stdout followed by apparently valid ready/stopped, and stale stderr generation. Prove deliberate controls fail without bounds. +- [x] Run affected Rust suites and independent adversarial tests; compare deterministic retained bytes, not incompatible wall times. +- [x] Obtain independent scoped review and resolve findings; no actionable findings remain. + +Publication and exact-head hosted check receipts are tracked in the same draft PR after ref reconciliation. No merge or ready transition is implied. + +Real Android/Frida acceptance remains a separate prerequisite gate. Local host tests never count as Android execution. diff --git a/docs/superpowers/specs/2026-11-07-owned-android-engine-design.md b/docs/superpowers/specs/2026-11-07-owned-android-engine-design.md new file mode 100644 index 0000000..9eb3d76 --- /dev/null +++ b/docs/superpowers/specs/2026-11-07-owned-android-engine-design.md @@ -0,0 +1,43 @@ +# Scope-owned Android interception engine + +## Intent and scope + +Replace HTTP Toolkit-origin interception/unpinning scripts with original Scope code using documented Android, Java and Frida APIs. Retain Frida and its independently attributed Java bridge as third-party instrumentation infrastructure. This is a source migration, not a new TLS implementation or a license change. Retain public source history and accurate third-party notices. + +The existing Scope Rust HTTP(S) proxy, CA, archive and upstream certificate verification stay unchanged. New runtime code must not be copied or mechanically translated from the removed upstream hooks. No claim of legally certified clean-room development is made. + +## Approach + +A small per-process Java engine is the default and only built-in hook backend. Keeping the imported hook backend indefinitely would leave the ownership problem unresolved. Reimplementing Frida or every native networking stack would expand this work beyond a reviewable change. Scope-owned Java hooks on Frida provide a bounded, testable boundary. + +Supported profiles are `proxy`, `certificate`, `unpinning`; an empty selection means all three. `unpinning` requires `certificate`. Unsupported `root`, `system-ca` or unknown profiles fail before any device probes or state mutation. No native TLS, QUIC, Flutter, Cronet, custom obfuscated pinners, child process or root-hiding coverage is claimed. + +`proxy` directs supported Java HTTP clients to the ADB-reversed loopback Scope listener. `certificate` adds trust only for chains validated against the supplied Scope CA while preserving original validation for other chains. No trust-all manager, hostname-verifier bypass, certificate-expiry bypass or global system trust-store mutation is allowed. `unpinning` may skip supported OkHttp pin checks only after the peer chain validates against the configured Scope CA; non-Scope chains must execute original pin checks and preserve their result/exception. + +## Runtime and lifecycle contract + +The Python worker validates finite profile, PEM and port inputs before device work. Source construction uses JSON serialization for runtime data, not source-text replacement. The engine reports per-capability installation results. Readiness requires successful installation of every requested profile, not merely entry into Java.perform. Unsupported classes/overloads or hook errors produce actionable bounded diagnostics and no success claim. Installation failure rolls back installed method implementations. + +Process-created proxy/trust objects can outlive method unhooking. Therefore Stop terminates the exact process spawned by this worker, then unloads/detaches; it does not promise that unloading alone restores the running app. It must not kill an unrelated process. Cleanup errors remain visible, and no clean-stop receipt is emitted after failed cleanup. The worker emits a spawned-PID event before attach and an explicit termination acknowledgment only after cleanup. Rust preserves pending target-process cleanup in recovery records; it must not convert a killed/failed worker into a successful Stop. A saved owned PID can be checked for absence before clearing an interrupted session; ambiguous state is an actionable error, never a reason to kill an arbitrary or reused PID. Existing Rust recovery of owned ADB/server state and legacy CA removal remains available after restart. + +## Compatibility changes + +Remove the imported system-CA injection template and automatic rooted proxy-mode trust-store injection. System proxy mode remains device-wide; HTTPS requires manually configured application trust. Do not remove legacy CA cleanup for recovery records created by earlier versions. Remove root-hiding selection from the UI; stale API/CLI selections are rejected before mutations. Explain these reductions prominently instead of claiming full parity with the imported scripts. + +## Validation and delivery + +Use executable JavaScript policy/adapter tests, worker builder/lifecycle tests and existing Rust/UI/device/release suites. Include controls proving unrelated or invalid chains still fail, original pin exceptions survive, unsupported capabilities never become ready, partial startup rolls back, and Stop ends only the owned PID. Simulated Frida/Java adapter tests are explicitly not Android execution evidence. + +Real Android/Frida/OkHttp TLS, app-stop restoration and API/ABI compatibility are separate acceptance gates. Record exact PASS / FAIL / NOT RUN without inventing coverage. An independently reviewed PR may remain draft if these device gates cannot be executed. No merge, release, tags, engine/service changes or live external account actions. + +## Primary API references + +- https://frida.re/docs/javascript-api/#java +- https://developer.android.com/privacy-and-security/security-config +- https://github.com/square/okhttp + +## Pinned OkHttp compatibility clarification + +Primary-source inspection of OkHttp4.12 shows that the client's original trust manager can reject Scope CA during lazy Handshake chain cleaning; Handshake then exposes an empty peer chain to `check$okhttp`. The pinner hook alone cannot safely recover the lost raw chain. Reject this recognized lazy configuration before claiming unpinning readiness. Retain only the ordinary Java overload path targeted by pinned OkHttp3.14.9 until a coherent cleaner/trust-manager integration has actual runtime validation. This deliberately reduces compatibility instead of treating an empty chain as trusted. + +Ship an original, buildable lab APK and bounded loopback TLS fixtures so missing Android acceptance can be reproduced when an owned device is available. Real pinned OkHttp3.14.9 and Okio1.17.2 are test-only dependencies, checksum locked; generated APKs and test keys never enter production resources or source control. APK compilation/signature verification is a separate gate from executing Android TLS/unpinning/Stop tests. diff --git a/examples/android-engine-fixture/.gitignore b/examples/android-engine-fixture/.gitignore new file mode 100644 index 0000000..b2ca15d --- /dev/null +++ b/examples/android-engine-fixture/.gitignore @@ -0,0 +1,8 @@ +__pycache__/ +*.pyc +*.jar +*.apk +*.p12 +*.pem +*.key +build/ diff --git a/examples/android-engine-fixture/AndroidManifest.xml b/examples/android-engine-fixture/AndroidManifest.xml new file mode 100644 index 0000000..cd570f2 --- /dev/null +++ b/examples/android-engine-fixture/AndroidManifest.xml @@ -0,0 +1,9 @@ + + + + + + + + + diff --git a/examples/android-engine-fixture/README.md b/examples/android-engine-fixture/README.md new file mode 100644 index 0000000..a56b8bc --- /dev/null +++ b/examples/android-engine-fixture/README.md @@ -0,0 +1,115 @@ +# Owned Android engine acceptance fixture + +This original lab APK exercises a fresh platform `SSLContext.init` and real OkHttp **3.14.9**, with a deliberately wrong SHA-256 pin for `127.0.0.1`. It keeps the original hostname verifier, certificate checks and request errors. The explicit `okhttp-scope` control uses an isolated original manager containing only an optional public fixture CA; that same manager and its socket factory reach OkHttp, so the deliberate wrong pin is tested after successful chain cleaning. `platform` and default `okhttp` retain original system trust. Only `http[s]://127.0.0.1:PORT/` inputs are accepted; credentials, queries, fragments and redirects are forbidden. One action makes one GET, with one request in flight, four-second socket timeouts and an eight-second OkHttp call deadline. No response bodies are logged. HTTP reachability is separate from TLS/pin acceptance. + +**Android/Frida/TLS/unpinning/Stop acceptance: NOT RUN.** A software-emulator prerequisite exited with SIGSEGV before usable Android execution. Compiling/signing this APK and passing host Python/OpenSSL TLS controls do not establish any Android engine result. The APK is test-only and is never packaged with Scope. + +## Build with existing tools + +Supply Android SDK platform `android-35`, build-tools `36.0.0` and JDK 17. The helper installs no SDK, JDK, Gradle or Maven. Its only network acquisition is these two exact Maven Central JARs, checked before any build tool executes: + +| Artifact | SHA-256 | +| --- | --- | +| `com.squareup.okhttp3:okhttp:3.14.9` | `2570fab55515cbf881d7a4ceef49fc515490bc027057e666776a2832465aeca0` | +| `com.squareup.okio:okio:1.17.2` | `f80ce42d2ffac47ad4c47e1d6f980d604d247ceb1a886705cf4581ab0c9fe2b8` | + +Downloads reject redirects, exceed neither 2 MiB per artifact nor a bounded network operation, and use a 45-second transfer deadline with a 15-second socket timeout. Cached JARs are rechecked. Generated output must be outside the repository. Use a dedicated empty directory. The helper creates an ownership marker and refuses nonempty unowned directories before changing permissions or replacing build subdirectories; marked directories support repeat builds. + +```sh +python3 examples/android-engine-fixture/build.py \ + --sdk /path/to/existing/android-sdk --jdk /path/to/existing/jdk-17 \ + --output /tmp/scope-owned-apk +# Repeat without network after the locked JARs have been acquired: +python3 examples/android-engine-fixture/build.py \ + --sdk /path/to/existing/android-sdk --jdk /path/to/existing/jdk-17 \ + --output /tmp/scope-owned-apk --offline +python3 -m unittest discover -s examples/android-engine-fixture/tests -v +``` + +For the genuine pin-mismatch control, start the TLS fixture below first, then pass its public CA via `--scope-ca /path/from/server/scope-ca.pem`. Only a single actual X.509 CA certificate of at most 64 KiB is accepted. Parsing must return exactly one CA whose complete DER equals the decoded input; malformed DER, relabelled private keys, server leaves and trailing data are rejected. It is embedded as an APK asset, never installed in a system trust store. Rebuild after regenerating fixture CAs. + +The helper runs `javac`, `d8`, `aapt`, `zipalign`, `keytool`, and `apksigner sign` followed by actual `apksigner verify --verbose --print-certs`. The output contains the signed APK and a JSON receipt with its SHA-256, exact libraries, tool versions and signature verification. Signing uses a disposable, seven-day lab key and the public password `android`; never reuse that key for production. Delete the dedicated output directory when finished. The APK contains the Apache license and dependency notices, including OkHttp's embedded Public Suffix List notice; see [THIRD_PARTY_NOTICES.txt](THIRD_PARTY_NOTICES.txt). + +## Bounded direct TLS fixture + +The server uses the installed `openssl` command and Python standard library. It generates unrelated ephemeral Scope/foreign CAs, an IP-valid Scope leaf, a foreign leaf, a Scope leaf for the wrong IP and an expired Scope leaf. Listeners bind **only 127.0.0.1**. Private keys are mode 0600 inside a temporary mode-0700 directory and are deleted at exit. A server instance lasts at most 600 seconds, has two-second connection timeouts and accepts a bounded number of connections per endpoint. It does not modify any trust store or device. + +```sh +python3 examples/android-engine-fixture/serve.py --seconds 600 --max-connections 100 +``` + +The single JSON startup receipt supplies the temporary public `scope_ca` path and URLs: Scope `18443`, foreign `18444`, wrong-host `18445`, expired `18446`, and HTTP `18080`. Keep that process running during the manual test. Restarting it creates new CAs; restart the worker with the new public CA. Host helper tests use real Python SSL verification for all four certificate cases. These are host certificate-fixture checks, not Android or Scope engine proof. + +## Manual acceptance on an explicitly selected owned lab device + +Use only a device and application you own and have selected for this test. The commands below are instructions, not an automatic device driver. They assume an already prepared compatible Frida server and the pinned Python Frida dependencies described in [Android setup](../../docs/ANDROID.md). Do not install a CA in the device system store. Do not use a real account or traffic. + +Start the server, copy its public CA path, and build with that same certificate before installing: + +```sh +CA='/temporary/path/from/server/receipt/scope-ca.pem' +python3 examples/android-engine-fixture/build.py \ + --sdk /path/to/existing/android-sdk --jdk /path/to/existing/jdk-17 \ + --output /tmp/scope-owned-apk --scope-ca "$CA" --offline +``` + +Set the explicit serial and artifact paths, inspect the selected device's existing reverse mappings, then add only unused test mappings. If a port already belongs to another mapping, choose unused fixture ports instead. Record which entries you added; remove only those entries at cleanup. + +```sh +SERIAL='YOUR_OWNED_LAB_SERIAL' +APK='/tmp/scope-owned-apk/scope-engine-fixture.apk' +CA='/temporary/path/from/server/receipt/scope-ca.pem' +adb -s "$SERIAL" get-state +adb -s "$SERIAL" reverse --list +adb -s "$SERIAL" install -r "$APK" +adb -s "$SERIAL" reverse tcp:18080 tcp:18080 +adb -s "$SERIAL" reverse tcp:18443 tcp:18443 +adb -s "$SERIAL" reverse tcp:18444 tcp:18444 +adb -s "$SERIAL" reverse tcp:18445 tcp:18445 +adb -s "$SERIAL" reverse tcp:18446 tcp:18446 +``` + +For the baseline, start the owned app directly. The `run` intent explicitly triggers one request; omit it to open the manual UI. Repeat with `client=okhttp`, `client=okhttp-scope` and each URL. The `okhttp-scope` mode requires the public-CA build above. Inspect JSON Logcat receipts; a `response` with status 200 is success, an `error` records the original exception class/message, `busy` requires waiting for the existing request to end. A connection refusal or timeout is an inconclusive reachability failure, never a TLS-negative PASS. + +```sh +adb -s "$SERIAL" shell am start -n com.scope.enginefixture/.MainActivity \ + --ez run true --es client platform --es url http://127.0.0.1:18080/ +adb -s "$SERIAL" shell am start -n com.scope.enginefixture/.MainActivity \ + --ez run true --es client platform --es url https://127.0.0.1:18443/ +adb -s "$SERIAL" logcat -d -t 200 ScopeEngineFixture:I '*:S' +``` + +For certificate-only, run this worker in a separate interactive terminal, keeping stdin open. Wait for its exact `scope-java-v1` ready receipt before triggering requests through `am start`. Do not use `am start -S`, which would kill the worker-owned process. Type `stop` into the worker terminal and require its successful termination acknowledgment before proceeding to another phase. + +```sh +python3 resources/device/frida_worker.py --serial "$SERIAL" \ + --package com.scope.enginefixture --ca "$CA" --scripts resources/frida \ + --profile certificate +# For the next phase, after stopping the preceding worker: +python3 resources/device/frida_worker.py --serial "$SERIAL" \ + --package com.scope.enginefixture --ca "$CA" --scripts resources/frida \ + --profile certificate --profile unpinning +``` + +These standalone commands deliberately select direct certificate/unpinning hooks. An omitted profile selects proxy as well and would route TLS through a different CA; that is a different experiment. This recipe does not exercise Rust/UI readiness, recovery or proxy capture. + +| Phase / client | HTTP control | Valid Scope TLS | Foreign CA | Wrong host | Expired leaf | Actual result | +| --- | --- | --- | --- | --- | --- | --- | +| No hooks / platform | 200 | trust rejection | trust rejection | rejection | rejection | NOT RUN | +| No hooks / default OkHttp | 200 | trust rejection | trust rejection | rejection | rejection | NOT RUN | +| No hooks / `okhttp-scope` | 200 | specific pin mismatch | trust rejection | hostname rejection | expiry rejection | NOT RUN | +| Certificate only / platform | 200 | 200 | trust rejection | hostname rejection | expiry rejection | NOT RUN | +| Certificate only / default OkHttp | 200 | rejection at original cleaner or pin enforcement | trust rejection | hostname rejection | expiry rejection | NOT RUN | +| Certificate only / `okhttp-scope` | 200 | specific pin mismatch | trust rejection | hostname rejection | expiry rejection | NOT RUN | +| Certificate + unpinning / platform | 200 | 200 | trust rejection | hostname rejection | expiry rejection | NOT RUN | +| Certificate + unpinning / both OkHttp modes | 200 | 200 despite configured wrong pin | trust rejection | hostname rejection | expiry rejection | NOT RUN | +| Stop + fresh restart / platform and default OkHttp | 200 | original trust rejection restored | trust rejection | rejection | rejection | NOT RUN | +| Stop + fresh restart / `okhttp-scope` | 200 | specific pin mismatch restored | trust rejection | hostname rejection | expiry rejection | NOT RUN | + +Record actual exception types/causes for every failure. For `okhttp-scope`, require `SSLPeerUnverifiedException` with `Certificate pinning failure!` for the valid-Scope no-hooks/certificate-only/Stop controls; a generic trust failure is insufficient. The default OkHttp chain cleaner retains its original manager and can reject Scope CA before the wrong pin is evaluated in certificate-only mode; that rejection alone does not prove a measured pin mismatch. Ordinary 3.14.9 pinner hooks retain the raw chain for isolated Scope validation. Kotlin lazy `check$okhttp` configurations are rejected before readiness because their cleaner can discard that chain. No chain-cleaner override is installed. + +After Stop, require the worker's matching successful termination acknowledgment, check `adb -s "$SERIAL" shell pidof com.scope.enginefixture` reports absence, and launch the fresh baseline again. Missing acknowledgment, a surviving PID or unknown termination is a failed cleanup gate; do not kill arbitrary PIDs. Exercise Scope UI Stop/application restart separately to validate the full Rust/UI lifecycle and saved recovery; those gates remain NOT RUN. + +Proxy capture is another separate manual phase: run actual Scope on its selected listener and use only the synthetic HTTP URL with an explicitly selected proxy profile. Confirm capture in Scope, then Stop and confirm owned mapping/process cleanup. It supplies no HTTPS/pinning evidence. Do not combine that proxy experiment's CA with the direct fixture matrix. + +Finally remove only your recorded test reverse mappings (`adb -s "$SERIAL" reverse --remove tcp:PORT` per owned entry), stop the temporary server and delete the disposable build directory. No wildcard mapping deletion, global trust-store change or automatic device selection is required. diff --git a/examples/android-engine-fixture/THIRD_PARTY_NOTICES.txt b/examples/android-engine-fixture/THIRD_PARTY_NOTICES.txt new file mode 100644 index 0000000..f43dec3 --- /dev/null +++ b/examples/android-engine-fixture/THIRD_PARTY_NOTICES.txt @@ -0,0 +1,19 @@ +Test-only dependencies included in the owned Scope Engine Fixture APK: + +OkHttp 3.14.9 +Copyright Square, Inc. and contributors. +https://github.com/square/okhttp/tree/parent-3.14.9 +Licensed under the Apache License, Version 2.0. + +Okio 1.17.2 +Copyright Square, Inc. and contributors. +https://github.com/square/okio/tree/okio-parent-1.17.2 +Licensed under the Apache License, Version 2.0. + +The complete Apache License, Version 2.0 is included as Apache-2.0.txt. +Any notices embedded in the exact locked JARs are also retained in this APK. +Scope fixture source is covered by the repository AGPL-3.0-or-later license. + +Exact upstream license sources: +https://github.com/square/okhttp/blob/parent-3.14.9/LICENSE.txt +https://github.com/square/okio/blob/okio-parent-1.17.2/LICENSE.txt diff --git a/examples/android-engine-fixture/build.py b/examples/android-engine-fixture/build.py new file mode 100644 index 0000000..9dcf599 --- /dev/null +++ b/examples/android-engine-fixture/build.py @@ -0,0 +1,185 @@ +#!/usr/bin/env python3 +"""Build only the owned lab APK; requires an existing SDK and JDK, no installers.""" +import argparse +import hashlib +import json +import os +import re +import ssl +from pathlib import Path +import shutil +import subprocess +import time +import urllib.request +import zipfile + +ROOT = Path(__file__).resolve().parent +REPO = ROOT.parents[1] +DEPENDENCIES = ( + ('okhttp-3.14.9.jar', 'https://repo.maven.apache.org/maven2/com/squareup/okhttp3/okhttp/3.14.9/okhttp-3.14.9.jar', '2570fab55515cbf881d7a4ceef49fc515490bc027057e666776a2832465aeca0'), + ('okio-1.17.2.jar', 'https://repo.maven.apache.org/maven2/com/squareup/okio/okio/1.17.2/okio-1.17.2.jar', 'f80ce42d2ffac47ad4c47e1d6f980d604d247ceb1a886705cf4581ab0c9fe2b8'), +) + + +def validate_output(path): + path = Path(path).resolve() + if path == REPO or REPO in path.parents or path in REPO.parents: + raise ValueError('Generated output must be outside the source repository') + return path + + +def prepare_output(path): + output = validate_output(path) + marker = output / '.scope-engine-fixture-build' + identity = 'scope-engine-fixture-v1\n' + if output.exists(): + if not output.is_dir(): + raise ValueError('Output must be a dedicated build directory') + owned = marker.is_file() and not marker.is_symlink() and marker.stat().st_size == len(identity) and marker.read_text() == identity + if any(output.iterdir()) and not owned: + raise ValueError('Refusing nonempty unowned output directory') + else: + output.mkdir(parents=True, mode=0o700) + # Ownership is established before changing permissions or removing children. + os.chmod(output, 0o700) + if not marker.exists(): + marker.write_text(identity) + return output + + +def verify_jar(path, expected): + path = Path(path) + if path.stat().st_size > 2 * 1024 * 1024 or hashlib.sha256(path.read_bytes()).hexdigest() != expected: + raise ValueError('SHA-256 mismatch: ' + path.name) + + +def read_public_ca(path): + path = Path(path) + if path.stat().st_size > 65536: + raise ValueError('Expected one public PEM certificate, at most 64 KiB') + pem = path.read_text(encoding='ascii') + if not re.fullmatch(r'-----BEGIN CERTIFICATE-----\s+[A-Za-z0-9+/=\s]+-----END CERTIFICATE-----\s*', pem, flags=re.ASCII): + raise ValueError('Expected one public PEM certificate without private keys') + try: + der = ssl.PEM_cert_to_DER_cert(pem) + # PEM decoding alone accepts arbitrary ASN.1, including relabelled keys. + # An empty context parses actual X.509 CAs without trusting system roots. + context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + context.load_verify_locations(cadata=pem) + anchors = context.get_ca_certs(binary_form=True) + if len(anchors) != 1 or anchors[0] != der: + raise ValueError('Expected exactly one complete public CA certificate') + except (ValueError, ssl.SSLError) as error: + raise ValueError('Expected one valid complete public PEM CA certificate') from None + return pem + + +class NoRedirect(urllib.request.HTTPRedirectHandler): + def redirect_request(self, req, fp, code, msg, headers, newurl): + raise ValueError('Dependency redirects are not permitted') + + +def acquire(output, offline): + jars = [] + for name, url, digest in DEPENDENCIES: + target = output / name + if not target.exists(): + if offline: + raise ValueError('Missing locked dependency in offline mode: ' + name) + temporary = target.with_suffix('.part') + deadline = time.monotonic() + 45 + try: + opener = urllib.request.build_opener(NoRedirect) + with opener.open(url, timeout=15) as response, temporary.open('xb') as stream: + total = 0 + while True: + data = response.read1(65536) + if not data: + break + total += len(data) + if total > 2 * 1024 * 1024 or time.monotonic() > deadline: + raise ValueError('Dependency download exceeded bounds') + stream.write(data) + verify_jar(temporary, digest) + temporary.replace(target) + finally: + temporary.unlink(missing_ok=True) + verify_jar(target, digest) # Cached artifacts are checked on every build. + jars.append(target) + return jars + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--sdk', type=Path, required=True) + parser.add_argument('--jdk', type=Path, required=True) + parser.add_argument('--output', type=Path, required=True) + parser.add_argument('--offline', action='store_true') + parser.add_argument('--scope-ca', type=Path, help='Optional public fixture CA for the explicit okhttp-scope pin control') + args = parser.parse_args() + public_ca = read_public_ca(args.scope_ca) if args.scope_ca else None + output = prepare_output(args.output) + jars = acquire(output, args.offline) # No build tool runs until both hashes pass. + sdk, jdk = args.sdk.resolve(), args.jdk.resolve() + tools = sdk / 'build-tools/36.0.0' + android = sdk / 'platforms/android-35/android.jar' + for path in [android, tools / 'aapt', tools / 'd8', tools / 'zipalign', tools / 'apksigner', jdk / 'bin/javac', jdk / 'bin/keytool']: + if not path.is_file(): + raise ValueError('Required supplied tool missing: ' + str(path)) + env = os.environ.copy() + env['JAVA_HOME'] = str(jdk) + env['PATH'] = str(jdk / 'bin') + os.pathsep + env.get('PATH', '') + + def run(*command): + result = subprocess.run([str(x) for x in command], env=env, cwd=output, capture_output=True, text=True, timeout=90, check=True) + return (result.stdout + result.stderr).strip() + + classes, dex, assets = [output / name for name in ['classes', 'dex', 'assets']] + for directory in [classes, dex, assets]: + shutil.rmtree(directory, ignore_errors=True) + directory.mkdir() + if public_ca: + (assets / 'scope-ca.pem').write_text(public_ca) + notices = assets / 'third-party' + notices.mkdir() + shutil.copyfile(ROOT / 'THIRD_PARTY_NOTICES.txt', notices / 'NOTICE.txt') + shutil.copyfile(REPO / 'LICENSE', assets / 'Scope-AGPL-3.0.txt') + shutil.copyfile(REPO / 'resources/licenses/rust-standard-library/licenses/Apache-2.0.txt', notices / 'Apache-2.0.txt') + for jar in jars: + with zipfile.ZipFile(jar) as archive: + for index, entry in enumerate(archive.infolist()): + if any(word in entry.filename.lower() for word in ['license', 'notice']): + # Keep any embedded notices without trusting archive paths. + (notices / (jar.stem + '-' + str(index) + '.txt')).write_bytes(archive.read(entry)) + classpath = os.pathsep.join(str(path) for path in [android, *jars]) + run(jdk / 'bin/javac', '-source', '8', '-target', '8', '-Xlint:-options', '-cp', classpath, '-d', classes, *sorted((ROOT / 'src').rglob('*.java'))) + classjar = output / 'fixture-classes.jar' + with zipfile.ZipFile(classjar, 'w') as archive: + for path in classes.rglob('*.class'): + archive.write(path, path.relative_to(classes)) + run(tools / 'd8', '--min-api', '26', '--lib', android, '--output', dex, classjar, *jars) + unsigned = output / 'unsigned.apk' + run(tools / 'aapt', 'package', '-f', '-M', ROOT / 'AndroidManifest.xml', '-I', android, '-A', assets, '-F', unsigned) + with zipfile.ZipFile(unsigned, 'a', compression=zipfile.ZIP_DEFLATED) as archive: + archive.write(dex / 'classes.dex', 'classes.dex') + aligned, signed, keystore = [output / name for name in ['aligned.apk', 'scope-engine-fixture.apk', 'fixture.p12']] + run(tools / 'zipalign', '-f', '4', unsigned, aligned) + if not keystore.exists(): + # Public disposable lab password; this key has no production identity. + run(jdk / 'bin/keytool', '-genkeypair', '-keystore', keystore, '-storetype', 'PKCS12', '-storepass', 'android', '-keypass', 'android', '-alias', 'fixture', '-keyalg', 'RSA', '-keysize', '2048', '-validity', '7', '-dname', 'CN=Scope disposable fixture') + os.chmod(keystore, 0o600) + run(tools / 'apksigner', 'sign', '--ks', keystore, '--ks-key-alias', 'fixture', '--ks-pass', 'pass:android', '--key-pass', 'pass:android', '--out', signed, aligned) + verification = run(tools / 'apksigner', 'verify', '--verbose', '--print-certs', signed) + receipt = { + 'artifact': signed.name, 'sha256': hashlib.sha256(signed.read_bytes()).hexdigest(), + 'dependencies': {name: digest for name, _url, digest in DEPENDENCIES}, + 'javac': run(jdk / 'bin/javac', '-version'), 'build_tools': '36.0.0', 'platform': 'android-35', + 'signature_verification': verification, 'android_execution': 'NOT RUN', + 'fixture_ca_sha256': hashlib.sha256(public_ca.encode('ascii')).hexdigest() if public_ca else None, + } + (output / 'build-receipt.json').write_text(json.dumps(receipt, indent=2) + '\n') + print(json.dumps(receipt, indent=2)) + + +if __name__ == '__main__': + main() diff --git a/examples/android-engine-fixture/serve.py b/examples/android-engine-fixture/serve.py new file mode 100644 index 0000000..a420e85 --- /dev/null +++ b/examples/android-engine-fixture/serve.py @@ -0,0 +1,142 @@ +#!/usr/bin/env python3 +"""Finite synthetic loopback HTTP/TLS lab. Keys live only in a temporary directory.""" +import argparse +from contextlib import contextmanager +from datetime import datetime, timedelta, timezone +import http.server +import json +import os +from pathlib import Path +import ssl +import subprocess +import tempfile +import threading +import time + +CASES = ('scope', 'foreign', 'wrong-host', 'expired') + + +def generate_fixtures(root): + root = Path(root).resolve() + root.mkdir(parents=True, exist_ok=True, mode=0o700) + os.chmod(root, 0o700) + + def openssl(*args): + subprocess.run(['openssl', *map(str, args)], cwd=root, check=True, timeout=20, capture_output=True) + + for ca in ['scope', 'foreign']: + openssl('req', '-x509', '-newkey', 'rsa:2048', '-nodes', '-days', '2', '-subj', '/CN=Scope synthetic ' + ca + ' CA', '-addext', 'basicConstraints=critical,CA:TRUE', '-addext', 'keyUsage=critical,keyCertSign,cRLSign', '-keyout', ca + '-ca.key', '-out', ca + '-ca.pem') + for case in CASES: + ca = 'foreign' if case == 'foreign' else 'scope' + host = 'IP:127.0.0.2' if case == 'wrong-host' else 'IP:127.0.0.1' + ext = root / (case + '.ext') + ext.write_text('basicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth\nsubjectAltName=' + host + '\n') + openssl('req', '-new', '-newkey', 'rsa:2048', '-nodes', '-subj', '/CN=Scope synthetic ' + case, '-keyout', case + '.key', '-out', case + '.csr') + if case != 'expired': + openssl('x509', '-req', '-in', case + '.csr', '-CA', ca + '-ca.pem', '-CAkey', ca + '-ca.key', '-CAcreateserial', '-days', '1', '-extfile', ext, '-out', case + '.pem') + else: + (root / 'index').touch() + (root / 'serial').write_text('1000\n') + (root / 'ca.cnf').write_text('[ca]\ndefault_ca=lab\n[lab]\ndatabase=index\nserial=serial\nnew_certs_dir=.\ncertificate=scope-ca.pem\nprivate_key=scope-ca.key\ndefault_md=sha256\npolicy=policy\n[policy]\ncommonName=supplied\n') + now = datetime.now(timezone.utc) + openssl('ca', '-batch', '-config', 'ca.cnf', '-in', case + '.csr', '-out', case + '.pem', '-extfile', ext, + '-startdate', (now - timedelta(days=3)).strftime('%Y%m%d%H%M%SZ'), '-enddate', (now - timedelta(days=1)).strftime('%Y%m%d%H%M%SZ')) + for path in root.glob('*.key'): + os.chmod(path, 0o600) + + +class Handler(http.server.BaseHTTPRequestHandler): + def do_GET(self): + body = b'scope-engine-fixture\n' + self.send_response(200) + self.send_header('Content-Type', 'text/plain') + self.send_header('Content-Length', str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, _format, *args): + pass # Never record request URLs, headers, bodies or real account data. + + +class LabServer(http.server.HTTPServer): + def get_request(self): + connection, address = super().get_request() + self.connection_count += 1 + connection.settimeout(2) + try: + if self.tls_context: + connection = self.tls_context.wrap_socket(connection, server_side=True) + return connection, address + except BaseException: + connection.close() + raise + + def handle_error(self, request, client_address): + pass # Rejected handshakes are expected negative controls. + + +@contextmanager +def running_servers(root, base_port, http_port, seconds, max_connections): + if not (0 < seconds <= 600 and 1 <= max_connections <= 200): + raise ValueError('Use 0 < seconds <= 600 and 1 <= max-connections <= 200') + if not (0 <= base_port <= 65532 and 0 <= http_port <= 65535): + raise ValueError('Invalid fixture ports') + servers, threads, urls = [], [], {} + stop = threading.Event() + deadline = time.monotonic() + seconds + + def run(server): + try: + while not stop.is_set() and time.monotonic() < deadline and server.connection_count < max_connections: + server.handle_request() + finally: + server.server_close() + + try: + for index, case in enumerate((*CASES, 'http')): + port = http_port if case == 'http' else (base_port + index if base_port else 0) + server = LabServer(('127.0.0.1', port), Handler) + servers.append(server) + server.timeout = 0.2 + server.connection_count = 0 + server.tls_context = None + if case != 'http': + context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + context.load_cert_chain(Path(root) / (case + '.pem'), Path(root) / (case + '.key')) + server.tls_context = context + urls[case] = ('http' if case == 'http' else 'https') + '://127.0.0.1:' + str(server.server_port) + '/' + thread = threading.Thread(target=run, args=(server,), daemon=True) + thread.start() + threads.append(thread) + yield urls + finally: + stop.set() + for thread in threads: + thread.join(timeout=5) + for server in servers: + server.server_close() + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--base-port', type=int, default=18443) + parser.add_argument('--http-port', type=int, default=18080) + parser.add_argument('--seconds', type=float, default=120) + parser.add_argument('--max-connections', type=int, default=100) + parser.add_argument('--parent', type=Path, help='Existing private temporary parent directory') + args = parser.parse_args() + with tempfile.TemporaryDirectory(prefix='scope-engine-tls-', dir=args.parent) as temporary: + root = Path(temporary) + generate_fixtures(root) + with running_servers(root, args.base_port, args.http_port, args.seconds, args.max_connections) as urls: + print(json.dumps({'fixture': 'scope-engine-v1', 'urls': urls, 'scope_ca': str(root / 'scope-ca.pem'), 'seconds': args.seconds, 'android_execution': 'NOT RUN'}), flush=True) + try: + deadline = time.monotonic() + args.seconds + while time.monotonic() < deadline: + time.sleep(min(0.2, deadline - time.monotonic())) + except KeyboardInterrupt: + pass + + +if __name__ == '__main__': + main() diff --git a/examples/android-engine-fixture/src/com/scope/enginefixture/MainActivity.java b/examples/android-engine-fixture/src/com/scope/enginefixture/MainActivity.java new file mode 100644 index 0000000..21b095a --- /dev/null +++ b/examples/android-engine-fixture/src/com/scope/enginefixture/MainActivity.java @@ -0,0 +1,182 @@ +package com.scope.enginefixture; + +import android.app.Activity; +import android.content.Intent; +import android.os.Bundle; +import android.util.Log; +import android.view.View; +import android.widget.Button; +import android.widget.EditText; +import android.widget.LinearLayout; +import android.widget.TextView; +import java.io.InputStream; +import java.net.HttpURLConnection; +import java.net.URI; +import java.net.URL; +import java.security.KeyStore; +import java.security.cert.CertificateFactory; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import javax.net.ssl.HttpsURLConnection; +import javax.net.ssl.SSLContext; +import javax.net.ssl.TrustManager; +import javax.net.ssl.TrustManagerFactory; +import javax.net.ssl.X509TrustManager; +import okhttp3.CertificatePinner; +import okhttp3.OkHttpClient; +import okhttp3.Request; +import okhttp3.Response; +import org.json.JSONObject; + +/** Owned synthetic-only lab client. Each explicit action makes one finite request. */ +public final class MainActivity extends Activity { + private static final AtomicBoolean busy = new AtomicBoolean(false); + private static final AtomicInteger sequence = new AtomicInteger(); + private EditText input; + private TextView result; + + @Override public void onCreate(Bundle state) { + super.onCreate(state); + LinearLayout layout = new LinearLayout(this); + layout.setOrientation(LinearLayout.VERTICAL); + input = new EditText(this); + input.setSingleLine(true); + input.setText("https://127.0.0.1:18443/"); + layout.addView(input); + for (final String client : new String[] {"platform", "okhttp", "okhttp-scope"}) { + Button button = new Button(this); + button.setText("One request: " + client); + button.setOnClickListener(new View.OnClickListener() { + @Override public void onClick(View view) { request(client, input.getText().toString()); } + }); + layout.addView(button); + } + result = new TextView(this); + result.setText("Owned loopback fixture only. No requests run until explicitly started."); + layout.addView(result); + setContentView(layout); + consumeIntent(getIntent()); + } + + @Override public void onNewIntent(Intent intent) { + super.onNewIntent(intent); + setIntent(intent); + consumeIntent(intent); + } + + private void consumeIntent(Intent intent) { + if (intent != null && intent.getBooleanExtra("run", false)) { + request(intent.getStringExtra("client"), intent.getStringExtra("url")); + } + } + + private static URL localUrl(String text) throws Exception { + if (text == null || text.length() > 256) throw new IllegalArgumentException("Invalid fixture URL"); + URI uri = new URI(text); + if (!("http".equals(uri.getScheme()) || "https".equals(uri.getScheme())) + || !"127.0.0.1".equals(uri.getHost()) || uri.getPort() < 1 || uri.getPort() > 65535 + || uri.getRawUserInfo() != null || uri.getRawQuery() != null || uri.getRawFragment() != null + || !(uri.getRawPath().isEmpty() || "/".equals(uri.getRawPath()))) { + throw new IllegalArgumentException("Use http[s]://127.0.0.1:PORT/ without credentials or query"); + } + return uri.toURL(); + } + + private void request(final String client, String text) { + final URL url; + try { + if (!("platform".equals(client) || "okhttp".equals(client) || "okhttp-scope".equals(client))) throw new IllegalArgumentException("Unknown fixture client"); + url = localUrl(text); + } catch (Exception invalid) { + // URI parser messages can echo rejected input; never log those values. + receipt(0, client, "invalid-input", 0, new IllegalArgumentException("Invalid local fixture input")); + return; + } + if (!busy.compareAndSet(false, true)) { + receipt(0, client, "busy", 0, null); + return; + } + final int id = sequence.incrementAndGet(); + new Thread(new Runnable() { + @Override public void run() { + try { + int status = "platform".equals(client) ? platform(url) : okhttp(url, "okhttp-scope".equals(client)); + receipt(id, client, "response", status, null); + } catch (Exception error) { + // Preserve the request's error; never retry or convert rejection to success. + receipt(id, client, "error", 0, error); + } finally { busy.set(false); } + } + }, "scope-fixture-request").start(); + } + + private static int platform(URL url) throws Exception { + HttpURLConnection connection = (HttpURLConnection) url.openConnection(); + connection.setConnectTimeout(4000); + connection.setReadTimeout(4000); + connection.setInstanceFollowRedirects(false); + connection.setRequestMethod("GET"); + if (connection instanceof HttpsURLConnection) { + SSLContext context = SSLContext.getInstance("TLS"); + context.init(null, null, null); // Fresh default manager; Scope hooks this exact seam. + ((HttpsURLConnection) connection).setSSLSocketFactory(context.getSocketFactory()); + } + try { return connection.getResponseCode(); } + finally { connection.disconnect(); } + } + + private int okhttp(URL url, boolean fixtureTrust) throws Exception { + TrustManagerFactory factory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm()); + KeyStore anchors = null; + if (fixtureTrust) { + anchors = KeyStore.getInstance(KeyStore.getDefaultType()); + anchors.load(null, null); + try (InputStream pem = getAssets().open("scope-ca.pem")) { + anchors.setCertificateEntry("fixture-scope", CertificateFactory.getInstance("X.509").generateCertificate(pem)); + } + } + factory.init(anchors); + X509TrustManager manager = null; + for (TrustManager candidate : factory.getTrustManagers()) { + if (candidate instanceof X509TrustManager) { manager = (X509TrustManager) candidate; break; } + } + if (manager == null) throw new IllegalStateException("No default X509 manager"); + SSLContext context = SSLContext.getInstance("TLS"); + context.init(null, new TrustManager[] {manager}, null); + CertificatePinner pinner = new CertificatePinner.Builder() + .add("127.0.0.1", "sha256/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=").build(); + OkHttpClient client = new OkHttpClient.Builder() + .sslSocketFactory(context.getSocketFactory(), manager).certificatePinner(pinner) + .followRedirects(false).followSslRedirects(false).retryOnConnectionFailure(false) + .connectTimeout(4, TimeUnit.SECONDS).readTimeout(4, TimeUnit.SECONDS) + .writeTimeout(4, TimeUnit.SECONDS).callTimeout(8, TimeUnit.SECONDS).build(); + try (Response response = client.newCall(new Request.Builder().url(url).get().build()).execute()) { + return response.code(); + } finally { + client.connectionPool().evictAll(); + client.dispatcher().executorService().shutdownNow(); + } + } + + private void receipt(int id, String client, String outcome, int status, Exception error) { + try { + JSONObject json = new JSONObject(); + json.put("fixture", "scope-engine-v1"); + json.put("pid", android.os.Process.myPid()); + json.put("request", id); + json.put("client", "okhttp-scope".equals(client) ? "okhttp-scope" : ("okhttp".equals(client) ? "okhttp" : "platform")); + json.put("outcome", outcome); + json.put("status", status); + if (error != null) { + json.put("errorClass", error.getClass().getName()); + String message = error.getMessage(); + json.put("error", message == null ? "" : message.substring(0, Math.min(message.length(), 512))); + if (error.getCause() != null) json.put("causeClass", error.getCause().getClass().getName()); + } + final String line = json.toString(); + Log.i("ScopeEngineFixture", line); + runOnUiThread(new Runnable() { @Override public void run() { result.setText(line); } }); + } catch (Exception encoding) { Log.e("ScopeEngineFixture", "receipt encoding failed"); } + } +} diff --git a/examples/android-engine-fixture/tests/test_helpers.py b/examples/android-engine-fixture/tests/test_helpers.py new file mode 100644 index 0000000..d519446 --- /dev/null +++ b/examples/android-engine-fixture/tests/test_helpers.py @@ -0,0 +1,154 @@ +"""Host helper checks. These do not execute Android or Scope hooks.""" +import importlib.util +from pathlib import Path +import ssl +import tempfile +import unittest +import urllib.request + +ROOT = Path(__file__).resolve().parents[1] + + +def load(name): + spec = importlib.util.spec_from_file_location(name, ROOT / (name + '.py')) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +class BuildTests(unittest.TestCase): + def test_corrupt_cached_jar_is_rejected_before_any_build(self): + build = load('build') + with tempfile.TemporaryDirectory() as temp: + jar = Path(temp) / 'bad.jar' + jar.write_bytes(b'corrupt') + with self.assertRaisesRegex(ValueError, 'SHA-256'): + build.verify_jar(jar, '0' * 64) + + def test_corrupt_build_exits_before_needing_sdk_or_jdk(self): + import subprocess + import sys + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + # This intentionally owned build directory reaches dependency validation. + (root / '.scope-engine-fixture-build').write_text('scope-engine-fixture-v1\n') + (root / 'okhttp-3.14.9.jar').write_bytes(b'corrupt') + completed = subprocess.run([sys.executable, str(ROOT / 'build.py'), '--sdk', str(root / 'missing-sdk'), + '--jdk', str(root / 'missing-jdk'), '--output', str(root), '--offline'], + capture_output=True, text=True, timeout=5) + self.assertNotEqual(completed.returncode, 0) + self.assertIn('SHA-256 mismatch', completed.stderr) + self.assertNotIn('Required supplied tool missing', completed.stderr) + + def test_output_cannot_write_generated_keys_into_source(self): + build = load('build') + with self.assertRaisesRegex(ValueError, 'outside'): + build.validate_output(ROOT / 'build') + with tempfile.TemporaryDirectory() as temp: + self.assertEqual(build.validate_output(Path(temp)), Path(temp).resolve()) + + def test_public_ca_input_rejects_private_key_and_oversized_input(self): + build = load('build') + self.assertTrue(callable(getattr(build, 'read_public_ca', None)), 'public CA input validation is implemented') + with tempfile.TemporaryDirectory() as temp: + path = Path(temp) / 'input.pem' + for content in [b'-----BEGIN PRIVATE KEY-----\nsecret\n-----END PRIVATE KEY-----', b'X' * 65537]: + path.write_bytes(content) + with self.assertRaisesRegex(ValueError, 'public PEM'): + build.read_public_ca(path) + + def test_public_ca_requires_real_complete_ca_der(self): + import subprocess + build, serve = load('build'), load('serve') + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + serve.generate_fixtures(root) + genuine = (root / 'scope-ca.pem').read_text() + self.assertEqual(build.read_public_ca(root / 'scope-ca.pem'), genuine) + genuine_der = ssl.PEM_cert_to_DER_cert(genuine) + private_der = subprocess.run(['openssl', 'genpkey', '-algorithm', 'RSA', '-pkeyopt', 'rsa_keygen_bits:1024', '-outform', 'DER'], + capture_output=True, check=True, timeout=10).stdout + for case, der in [('malformed', b'ABC'), ('relabelled-private-key', private_der), ('trailing-bytes', genuine_der + b'ABC')]: + with self.subTest(case=case): + path = root / 'invalid.pem' + path.write_text(ssl.DER_cert_to_PEM_cert(der)) + with self.assertRaisesRegex(ValueError, 'public PEM'): + build.read_public_ca(path) + with self.assertRaisesRegex(ValueError, 'public PEM'): + build.read_public_ca(root / 'scope.pem') # A genuine server leaf is not a CA. + + def test_owned_output_can_be_prepared_again_without_losing_files(self): + build = load('build') + with tempfile.TemporaryDirectory() as temp: + root = build.prepare_output(Path(temp) / 'new-build') + (root / 'keep.txt').write_text('owned output') + self.assertEqual(build.prepare_output(root), root) + self.assertEqual((root / 'keep.txt').read_text(), 'owned output') + + def test_unowned_output_is_preserved_before_permissions_or_cleanup(self): + import os + import stat + import subprocess + import sys + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + for name in ['classes', 'dex', 'assets']: + (root / name).mkdir() + (root / name / 'keep.txt').write_text('unrelated') + os.chmod(root, 0o755) + completed = subprocess.run([sys.executable, str(ROOT / 'build.py'), '--sdk', str(root / 'missing-sdk'), + '--jdk', str(root / 'missing-jdk'), '--output', str(root), '--offline'], + capture_output=True, text=True, timeout=5) + self.assertNotEqual(completed.returncode, 0) + self.assertIn('unowned', completed.stderr) + self.assertEqual(stat.S_IMODE(root.stat().st_mode), 0o755) + self.assertFalse((root / '.scope-engine-fixture-build').exists()) + for name in ['classes', 'dex', 'assets']: + self.assertEqual((root / name / 'keep.txt').read_text(), 'unrelated') + + +class FixtureTests(unittest.TestCase): + def test_real_tls_scope_foreign_wrong_host_and_expired_controls(self): + serve = load('serve') + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + serve.generate_fixtures(root) + context = ssl.create_default_context(cafile=str(root / 'scope-ca.pem')) + with serve.running_servers(root, 0, 0, seconds=15, max_connections=12) as urls: + with urllib.request.urlopen(urls['http'], timeout=3) as response: + self.assertEqual(response.read(), b'scope-engine-fixture\n') + with urllib.request.urlopen(urls['scope'], context=context, timeout=3) as response: + self.assertEqual(response.status, 200) + for case in ['foreign', 'wrong-host', 'expired']: + with self.subTest(case=case): + with self.assertRaises(urllib.error.URLError) as caught: + urllib.request.urlopen(urls[case], context=context, timeout=3) + self.assertIsInstance(caught.exception.reason, ssl.SSLCertVerificationError) + expected = {'foreign': 'unable to get local issuer', 'wrong-host': 'IP address mismatch', 'expired': 'expired'}[case] + self.assertIn(expected, str(caught.exception.reason)) + + def test_connection_limit_counts_requests_rather_than_idle_polling(self): + serve = load('serve') + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + serve.generate_fixtures(root) + with serve.running_servers(root, 0, 0, seconds=3, max_connections=1) as urls: + import time + time.sleep(0.5) + with urllib.request.urlopen(urls['http'], timeout=1) as response: + self.assertEqual(response.status, 200) + + def test_servers_have_a_finite_deadline_and_release_ports(self): + serve = load('serve') + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + serve.generate_fixtures(root) + with serve.running_servers(root, 0, 0, seconds=0.3, max_connections=1) as urls: + import time + time.sleep(0.7) + with self.assertRaises(urllib.error.URLError): + urllib.request.urlopen(urls['http'], timeout=1) + + +if __name__ == '__main__': + unittest.main() diff --git a/resources/device/frida_worker.py b/resources/device/frida_worker.py index 9e3d5fe..92428ce 100644 --- a/resources/device/frida_worker.py +++ b/resources/device/frida_worker.py @@ -1,53 +1,307 @@ #!/usr/bin/env python3 -"""Scope's isolated Frida worker. Stdout is JSON Lines; stdin EOF requests cleanup. +"""Scope's isolated Frida worker. Stdout is JSON Lines; stdin EOF requests Stop. -The worker controls exactly one selected application. Hook source stays JavaScript. -Python/Frida is deliberately outside the desktop process so an ART crash cannot -take the session archive or UI with it. No HTTP Toolkit installation is used. +The worker owns one spawned application process. Stop ends that process because +unloading methods cannot revoke already-created proxy or TLS objects. """ import argparse +from collections import deque +from contextlib import nullcontext import json from pathlib import Path import re import signal +import ssl import sys import threading +import time -PROFILES = { - "proxy": ["android/android-proxy-override.js"], - "certificate": ["android/android-system-certificate-injection.js"], - "unpinning": ["android/android-certificate-unpinning.js", "android/android-certificate-unpinning-fallback.js"], - "root": ["android/android-disable-root-detection.js"], -} +PROFILES = ("proxy", "certificate", "unpinning") +CAPABILITIES = {"proxy": "java-url", "certificate": "sslcontext-scope-ca", "unpinning": "okhttp3-certificate-pinner"} def emit(kind, **fields): - # Keep Frida diagnostics local and avoid logging tokens inside query strings. if "message" in fields: fields["message"] = re.sub(r"(https?://[^\s?]+)\?[^\s]+", r"\1?[query omitted]", str(fields["message"]))[:4000] print(json.dumps({"kind": kind, **fields}), flush=True) +def normalize_profiles(profiles): + if not isinstance(profiles, (list, tuple)) or any(not isinstance(p, str) or p not in PROFILES for p in profiles): + raise ValueError("Unsupported hook profile; supported: proxy, certificate, unpinning") + result = [p for p in PROFILES if p in profiles] if profiles else list(PROFILES) + if "unpinning" in result and "certificate" not in result: + raise ValueError("unpinning profile requires certificate") + return result + + def build_script(root, ca, port, profiles): - cert = ca.read_text().strip() - if not re.fullmatch(r"-----BEGIN CERTIFICATE-----\s+[A-Za-z0-9+/=\s]+-----END CERTIFICATE-----", cert): - raise ValueError("Scope CA must be a public PEM certificate") - unknown = set(profiles) - set(PROFILES) - if unknown: - raise ValueError("Unknown hook profile: " + ", ".join(sorted(unknown))) - config = (root / "config-template.js").read_text() - config, count = re.subn(r"const CERT_PEM = `.*?`;", lambda _: "const CERT_PEM = `" + cert + "`;", config, count=1, flags=re.S) - if count != 1: - raise ValueError("Unexpected Frida configuration template") - config, count = re.subn(r"const PROXY_PORT = \d+;", "const PROXY_PORT = %d;" % port, config, count=1) - if count != 1: - raise ValueError("Missing proxy port configuration") - scripts = [(root / "frida-java-bridge.js").read_text(), config] - for profile in profiles: - for name in PROFILES[profile]: - scripts.append((root / name).read_text()) - scripts.append("Java.perform(() => send({kind: 'scope-hooks-ready'}));\n") - return "\n".join(scripts) + profiles = normalize_profiles(profiles) + if type(port) is not int or not 1 <= port <= 65535: + raise ValueError("A valid proxy port is required") + try: + # Bound the read itself, including files that grow after opening. + with ca.open("rb") as stream: + data = stream.read(65537) + if len(data) > 65536: + raise ValueError("oversized CA") + # Preserve universal-newline behavior of the former text-mode read. + cert = data.decode("ascii").replace("\r\n", "\n").replace("\r", "\n").strip() + if not re.fullmatch(r"-----BEGIN CERTIFICATE-----\s+[A-Za-z0-9+/=\s]+-----END CERTIFICATE-----", cert, flags=re.ASCII): + raise ValueError("invalid PEM") + der = ssl.PEM_cert_to_DER_cert(cert) + # Fresh context with no system roots: enumerate only this public CA. + # PEM decoding alone also accepts relabelled keys and trailing DER. + context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + context.load_verify_locations(cadata=cert) + anchors = context.get_ca_certs(binary_form=True) + if len(anchors) != 1 or anchors[0] != der: + raise ValueError("incomplete or non-CA certificate") + except (OSError, UnicodeError, ValueError, ssl.SSLError): + raise ValueError("Scope CA must be one valid complete public PEM CA certificate, at most 64 KiB") from None + config = {"caPem": cert, "proxyHost": "127.0.0.1", "proxyPort": port, "profiles": profiles} + return "\n".join([ + (root / "frida-java-bridge.js").read_text(encoding="utf-8"), + (root / "scope-engine.js").read_text(encoding="utf-8"), + "globalThis.ScopeJavaEngine.start(" + json.dumps(config, ensure_ascii=True) + ");\n", + ]) + + +def validate_ready(payload, profiles): + selected = normalize_profiles(profiles) + if not isinstance(payload, dict) or payload.get("kind") != "scope-hooks-ready" or payload.get("engine") != "scope-java-v1" or payload.get("profiles") != selected: + raise ValueError("Invalid Scope engine readiness identity or profiles") + capabilities = payload.get("capabilities") + if not isinstance(capabilities, dict) or set(capabilities) != set(selected): + raise ValueError("Missing requested profile capabilities") + for profile in selected: + installed = capabilities[profile] + if not isinstance(installed, list) or any(not isinstance(c, str) for c in installed) or CAPABILITIES[profile] not in installed: + raise ValueError("Missing installed capability for profile " + profile) + return payload + + +def bounded_call(label, operation, timeout): + done = threading.Event() + outcome = [] + + def run(): + try: + outcome.append((True, operation())) + except BaseException as error: + outcome.append((False, error)) + finally: + done.set() + + threading.Thread(target=run, daemon=True).start() + if not done.wait(timeout): + raise RuntimeError(label + " timed out; completion is unknown") + success, value = outcome[0] + if not success: + raise RuntimeError(label + ": " + str(value)) from value + return value + + +class ProcessOwnership: + """Current local evidence about one spawn, not an atomic remote PID handle.""" + + def __init__(self): + self.exited = threading.Event() + self.replaced = threading.Event() + self.lock = threading.Lock() + self.pending_kill = None + self.kill_dispatched = False + self.kill_uncertain = False + + def observe(self, reason): + pending = None + with self.lock: + if reason == "process-terminated": + self.exited.set() + elif reason == "process-replaced": + self.replaced.set() + if self.kill_dispatched: + self.kill_uncertain = True + pending = self.pending_kill + if pending is not None: + pending.cancel() + + def kill_current(self, operation, cancellable): + try: + with cancellable if cancellable is not None else nullcontext(): + with self.lock: + if self.replaced.is_set(): + raise RuntimeError("Process ownership lost: original image was replaced; refusing kill") + if self.exited.is_set(): + return + self.pending_kill = cancellable + self.kill_dispatched = True + # Check inside the destructive operation's worker thread, rather + # than relying on the caller's earlier ownership snapshot. + if self.replaced.is_set(): + raise RuntimeError("Process ownership lost before kill dispatch; refusing kill") + if self.exited.is_set(): + return + operation() + finally: + with self.lock: + self.pending_kill = None + self.kill_dispatched = False + uncertain = self.kill_uncertain + if uncertain: + raise RuntimeError("Kill completion is unknown after in-flight process replacement; cancellation cannot prove remote outcome") + + +def cleanup_owned(device, pid, script, session, timeout=3, process_not_found=(), ownership=None, cancellable_factory=None): + errors = [] + ownership = ownership if ownership is not None else ProcessOwnership() + + def kill_pid(): + try: + device.kill(pid) + except process_not_found: + ownership.observe("process-terminated") + + def kill(): + cancellable = cancellable_factory() if cancellable_factory else None + ownership.kill_current(kill_pid, cancellable) + + def unload(): + if script.is_destroyed is not True: + try: + script.unload() + except Exception: + if script.is_destroyed is not True: + raise + + def detach(): + if session.is_detached is not True: + try: + session.detach() + except Exception: + if session.is_detached is not True: + raise + + operations = [] + if pid is not None: + operations.append(("kill owned application", kill)) + if script is not None: + operations.append(("unload hooks", unload)) + if session is not None: + operations.append(("detach session", detach)) + for label, operation in operations: + try: + bounded_call(label, operation, timeout) + except Exception as error: + errors.append(str(error)) + if errors: + raise RuntimeError("Cleanup failed: " + "; ".join(errors)) + + +def run_session(device, package, source, profiles, version, stopped, ready_timeout=30, cleanup_timeout=3, process_not_found=(), cancellable_factory=None): + profiles = normalize_profiles(profiles) + ready = threading.Event() + ownership = ProcessOwnership() + failure = deque(maxlen=2) + receipt = [] + replaced = ownership.replaced + session = script = pid = None + + def receive(message, _data): + payload = message.get("payload") + if message.get("type") == "send" and isinstance(payload, dict): + if payload.get("kind") == "scope-hooks-ready": + if failure or stopped.is_set(): + return + try: + receipt[:] = [validate_ready(payload, profiles)] + ready.set() + except ValueError as error: + failure.append(str(error)) + stopped.set() + return + if payload.get("kind") == "scope-hooks-error": + failure.append(str(payload.get("message", "Hook installation failed"))[:2000]) + stopped.set() + return + if message.get("type") == "error": + failure.append(str(message.get("stack") or message.get("description") or "Hook error")[:2000]) + stopped.set() + emit("log", message=message.get("stack") or payload or message.get("description") or message) + + def detached(reason, crash=None): + ownership.observe(reason) + if reason == "process-replaced": + failure.append("Owned process image was replaced; termination cannot be confirmed safely") + emit("detached", message=reason, crash=bool(crash)) + stopped.set() + + primary_error = None + startup_complete = threading.Event() + deadline = time.monotonic() + ready_timeout + cancellable = cancellable_factory() if cancellable_factory else None + + def cancel_startup(): + while not startup_complete.wait(0.025): + if stopped.is_set(): + cancellable.cancel() + return + if time.monotonic() >= deadline: + failure.append("Startup deadline expired; pending operation completion may be unknown") + stopped.set() + cancellable.cancel() + return + + if cancellable is not None: + threading.Thread(target=cancel_startup, daemon=True).start() + attempted_spawn = False + try: + if stopped.is_set(): + return + with cancellable if cancellable is not None else nullcontext(): + attempted_spawn = True + pid = device.spawn([package]) + emit("spawned", engine="scope-java-v1", pid=pid) + session = device.attach(pid) + session.on("detached", detached) + script = session.create_script(source) + script.on("message", receive) + script.load() + if not stopped.is_set(): + device.resume(pid) + startup_complete.set() + while not ready.is_set() and not stopped.is_set() and time.monotonic() < deadline: + ready.wait(min(0.05, max(0, deadline - time.monotonic()))) + if failure: + raise RuntimeError("Hook startup failed: " + "; ".join(failure)) + if stopped.is_set(): + if failure: + raise RuntimeError("Hook startup failed: " + "; ".join(failure)) + return + if not ready.is_set(): + raise RuntimeError("Java hooks did not become ready within the startup deadline") + emit("ready", pid=pid, version=version, engine=receipt[0]["engine"], profiles=profiles, capabilities=receipt[0]["capabilities"]) + stopped.wait() + if failure: + raise RuntimeError("Hook runtime failed: " + "; ".join(failure)) + except BaseException as error: + primary_error = RuntimeError("Spawn did not return an owned PID; completion is unknown: " + str(error)) if attempted_spawn and pid is None else error + finally: + startup_complete.set() + try: + cleanup_owned(device, pid, script, session, cleanup_timeout, process_not_found, ownership, cancellable_factory) + except Exception as error: + if primary_error: + raise RuntimeError(str(primary_error) + "; " + str(error)) from primary_error + raise + if replaced.is_set(): + raise RuntimeError("Owned process image was replaced; cleanup outcome is uncertain") from primary_error + if not attempted_spawn: + emit("stopped", engine="scope-java-v1", pid=None, spawnAttempted=False, cleanupComplete=True, processTerminated=False) + elif pid is not None: + emit("stopped", engine="scope-java-v1", pid=pid, spawnAttempted=True, cleanupComplete=True, processTerminated=True, message="Owned application process ended; hooks unloaded and session detached") + if primary_error: + raise primary_error def main(): @@ -62,6 +316,12 @@ def main(): parser.add_argument("--profile", action="append", default=[]) parser.add_argument("--list-apps", action="store_true") args = parser.parse_args() + profiles = normalize_profiles(args.profile) # Reject stale profiles before any device operation. + source = None + if not args.version and not args.list_apps: + if not args.package or not args.ca or not args.scripts: + raise ValueError("Package, public CA and scripts are required") + source = build_script(args.scripts, args.ca, args.port, profiles) try: import frida except ImportError as error: @@ -76,29 +336,7 @@ def main(): if args.list_apps: emit("apps", apps=[{"package": app.identifier, "name": app.name, "running": bool(app.pid)} for app in device.enumerate_applications()]) return - if not args.package or not args.ca or not args.scripts or not 1 <= args.port <= 65535: - raise ValueError("Package, public CA, scripts and a valid port are required") - source = build_script(args.scripts, args.ca, args.port, args.profile or ["proxy", "certificate", "unpinning"]) stopped = threading.Event() - ready = threading.Event() - failure = [] - session = None - script = None - pid = None - resumed = False - - def receive(message, _data): - payload = message.get("payload") - if message.get("type") == "send" and isinstance(payload, dict) and payload.get("kind") == "scope-hooks-ready": - ready.set() - return - if message.get("type") == "error": - failure.append(message.get("stack") or message.get("description") or "Hook error") - emit("log", message=message.get("stack") or payload or message.get("description") or message) - - def detached(reason, crash=None): - emit("detached", message=reason, crash=bool(crash)) - stopped.set() def wait_for_stop(): for line in sys.stdin: @@ -109,39 +347,8 @@ def wait_for_stop(): signal.signal(signal.SIGINT, lambda *_: stopped.set()) signal.signal(signal.SIGTERM, lambda *_: stopped.set()) threading.Thread(target=wait_for_stop, daemon=True).start() - emit("log", message="Experimental Java hooks: Android 16 ART/GC crashes were observed upstream. Native TLS/QUIC stacks are not covered.") - try: - pid = device.spawn([args.package]) - session = device.attach(pid) - session.on("detached", detached) - script = session.create_script(source) - script.on("message", receive) - script.load() - device.resume(pid) - resumed = True - if not ready.wait(30): - raise RuntimeError("Java hooks did not become ready within 30 seconds. " + "; ".join(failure[-2:])) - if failure: - raise RuntimeError("A hook failed during startup: " + "; ".join(failure[-2:])) - emit("ready", pid=pid, version=frida.__version__, profiles=args.profile or ["proxy", "certificate", "unpinning"]) - stopped.wait() - finally: - if script: - try: - script.unload() - except Exception: - pass - if session: - try: - session.detach() - except Exception: - pass - if pid and not resumed: - try: - device.kill(pid) - except Exception: - pass - emit("stopped", message="Frida hooks detached; the selected application was left running") + emit("log", message="Experimental Scope Java hooks; Stop ends the spawned app. Native TLS/QUIC and custom pinners are not covered.") + run_session(device, args.package, source, profiles, frida.__version__, stopped, process_not_found=(frida.ProcessNotFoundError,), cancellable_factory=frida.Cancellable) if __name__ == "__main__": diff --git a/resources/device/install-system-ca.template.sh b/resources/device/install-system-ca.template.sh deleted file mode 100644 index f3f017a..0000000 --- a/resources/device/install-system-ca.template.sh +++ /dev/null @@ -1,93 +0,0 @@ -# Derived from HTTP Toolkit Server 1.26.1 injectSystemCertificate. -# Source: https://github.com/httptoolkit/httptoolkit-server -# SPDX-License-Identifier: AGPL-3.0-or-later -# Copyright Tim Perry and HTTP Toolkit contributors. -# Modified 2026-09-30: extracted as a standalone template with certificate/script path placeholders. -set -e # Fail on error - -echo "\n---\nInjecting certificate:" - -# Create a separate temp directory, to hold the current certificates -# Without this, when we add the mount we can't read the current certs anymore. -mkdir -p /data/local/tmp/htk-ca-copy -chmod 700 /data/local/tmp/htk-ca-copy -rm -rf /data/local/tmp/htk-ca-copy/* - -# Copy out the existing certificates -if [ -d "/apex/com.android.conscrypt/cacerts" ]; then - cp /apex/com.android.conscrypt/cacerts/* /data/local/tmp/htk-ca-copy/ -else - cp /system/etc/security/cacerts/* /data/local/tmp/htk-ca-copy/ -fi - -# Create the in-memory mount on top of the system certs folder -mount -t tmpfs tmpfs /system/etc/security/cacerts - -# Copy the existing certs back into the tmpfs mount, so we keep trusting them -mv /data/local/tmp/htk-ca-copy/* /system/etc/security/cacerts/ - -# Copy our new cert in, so we trust that too -mv ${certificatePath} /system/etc/security/cacerts/ - -# Update the perms & selinux context labels, so everything is as readable as before -chown root:root /system/etc/security/cacerts/* -chmod 644 /system/etc/security/cacerts/* - -chcon u:object_r:system_file:s0 /system/etc/security/cacerts/ -chcon u:object_r:system_file:s0 /system/etc/security/cacerts/* - -echo 'System cacerts setup completed' - -# Deal with the APEX overrides in Android 14+, which need injecting into each namespace: -if [ -d "/apex/com.android.conscrypt/cacerts" ]; then - echo 'Injecting certificates into APEX cacerts' - - # When the APEX manages cacerts, we need to mount them at that path too. We can't do - # this globally as APEX mounts are namespaced per process, so we need to inject a - # bind mount for this directory into every mount namespace. - - # First we mount for the shell itself, for completeness and so we can see this - # when we check for correct installation on later runs - mount --bind /system/etc/security/cacerts /apex/com.android.conscrypt/cacerts - - # First we get the Zygote process(es), which launch each app - ZYGOTE_PID=$(pidof zygote || true) - ZYGOTE64_PID=$(pidof zygote64 || true) - Z_PIDS="$ZYGOTE_PID $ZYGOTE64_PID" - # N.b. some devices appear to have both, some have >1 of each (!) - - # Apps inherit the Zygote's mounts at startup, so we inject here to ensure all newly - # started apps will see these certs straight away: - for Z_PID in $Z_PIDS; do - if [ -n "$Z_PID" ]; then - nsenter --mount=/proc/$Z_PID/ns/mnt -- \ - /bin/mount --bind /system/etc/security/cacerts /apex/com.android.conscrypt/cacerts - fi - done - - echo 'Zygote APEX certificates remounted' - - # Then we inject the mount into all already running apps, so they see these certs immediately. - - # Get the PID of every process whose parent is one of the Zygotes: - APP_PIDS=$( - echo $Z_PIDS | \ - xargs -n1 ps -o 'PID' -P | \ - grep -v PID - ) - - # Inject into the mount namespace of each of those apps: - for PID in $APP_PIDS; do - nsenter --mount=/proc/$PID/ns/mnt -- \ - /bin/mount --bind /system/etc/security/cacerts /apex/com.android.conscrypt/cacerts & - done - wait # Launched in parallel - wait for completion here - - echo "APEX certificates remounted for $(echo $APP_PIDS | wc -w) apps" -fi - -# Delete the temp cert directory & this script itself -rm -r /data/local/tmp/htk-ca-copy -rm ${injectionScriptPath} - -echo "System cert successfully injected\n---\n" diff --git a/resources/device/tests/fixtures/ca.pem b/resources/device/tests/fixtures/ca.pem new file mode 100644 index 0000000..681b794 --- /dev/null +++ b/resources/device/tests/fixtures/ca.pem @@ -0,0 +1,20 @@ +-----BEGIN CERTIFICATE----- +MIIDOTCCAiGgAwIBAgIUaOJhl+csv98J08Q2GXcK4Ul4D14wDQYJKoZIhvcNAQEL +BQAwJDEiMCAGA1UEAwwZU2NvcGUgd29ya2VyIHN5bnRoZXRpYyBDQTAeFw0yNjEx +MDcwOTA1NDlaFw0zNjExMDQwOTA1NDlaMCQxIjAgBgNVBAMMGVNjb3BlIHdvcmtl +ciBzeW50aGV0aWMgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCt +/+dFyhmZCvh1SjSu/AZUOM9ABb1J2UWwGGA4t+zrAhfFGJTdIaGNHHJmm/5T9YCr +ShQLJ6pyZdk3UwP+ht5GTcGNiAsDFpm/Yez3/zjeiNgOenw2mBh506B0e1tshka9 +DVywree0OTrBMBv3QeKydr4C/zWrexdjVonfqzgFDkwYFhP9sIJwSLq81d11w14/ +QZ9wBeTT8PMgBjqPMO+x4Oa886CTyBCh5OCc9raLLRJ1VmGsRyAQwBp6Xm6CVglN +jc+4D/UMoap5pfZJxH/i9nGEmt4968zQxhlEK79wccA0/kdnc2IKT5YC/H2noLGB +sDn+qqBkGAyQE9jOGCZ3AgMBAAGjYzBhMB0GA1UdDgQWBBRF2O5Gu/bqfoiPbKWZ +0ckPvp+6ejAfBgNVHSMEGDAWgBRF2O5Gu/bqfoiPbKWZ0ckPvp+6ejAPBgNVHRMB +Af8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAQEAGNFh +nVmdWa2r/cmO36XkvJUVxDA8/aVOm6h7f1+d15d8w1QWizCULwtGxIa0LB4rohT4 +sIdZ54n83EO+lYxOxwwJkqaZA1S9e19ADDgSkXGyZgco9dnQjrea8sgA/PLEFpSR +CZi/bBtrdgPr227Spf5efXMBsj5YRszSGEqrxXd+Vp96boknShB/tU7OnbsBFLHi +YADUxZOEiE2wS0S+Yk/w7+CRPVvhQ+mdbdo6xlo9aUk+deYb4tfnWBmSAbjcRQ9C +2K/IEmz9UUFuGFwWm54irJcGA2HAUutydPMaQHcLiRoXAXwf/jL0K5Q2HHT0fA6L +VmO+lH26XSTiESlEtw== +-----END CERTIFICATE----- diff --git a/resources/device/tests/fixtures/leaf.pem b/resources/device/tests/fixtures/leaf.pem new file mode 100644 index 0000000..1f8fea0 --- /dev/null +++ b/resources/device/tests/fixtures/leaf.pem @@ -0,0 +1,20 @@ +-----BEGIN CERTIFICATE----- +MIIDODCCAiCgAwIBAgIULiero5ao+Y9qqMd7VGb0k3LfOhIwDQYJKoZIhvcNAQEL +BQAwJDEiMCAGA1UEAwwZU2NvcGUgd29ya2VyIHN5bnRoZXRpYyBDQTAeFw0yNjEx +MDcwOTA1NDlaFw0zNjExMDQwOTA1NDlaMCYxJDAiBgNVBAMMG1Njb3BlIHdvcmtl +ciBzeW50aGV0aWMgbGVhZjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB +AIv5APPbr48vcWKQJyTAycWc31Bf7z/gNdrk7N2dN8fMbcvHC3/3uvntSki+xp+r +YcyMQLFhotTonyOBZilkOfELjfG0tuuyUB5ySN9H2KDWCHGhkG7UqtC+Fh3CPCd2 +b731CeZrewWU8fnh9wqXa8mxrQ+Wv/z/6svwf4boEhUCLtIOFvdeO+apDOY3iAdH +kKRzCHBbdf7Yhf+fc/PU6ZJ7EF70LIxw/0cvEZ3NwvuGNwWJDjaKaE2tvZSaQ3/n +EmKGpoTn/r3RN+oKD/XAB/4EcoQZCcjCCM0fZnYiWiNz6Pjto8iCOVoTf9mZwy1e +U6FOljMf1RXtBzAFw7w4RnkCAwEAAaNgMF4wDAYDVR0TAQH/BAIwADAOBgNVHQ8B +Af8EBAMCBaAwHQYDVR0OBBYEFGG6kWQvVXLFJ9vT8LRJkD86APhZMB8GA1UdIwQY +MBaAFEXY7ka79up+iI9spZnRyQ++n7p6MA0GCSqGSIb3DQEBCwUAA4IBAQBAK1H/ +F/szZlb7//MBCfHg8oYfAY4AeWQwXA+Xb4ut0uMlHtkS9MD3TbV2ViuB30aATJPS +r7m6pqa7NtNz+7qtnS10M6Ck5fmI8sdMnOkMvScxX4ukcy7cuWOY++1cnw8tzN/w +0Sv2KQE0WGkuOdx+Flrwj2CzkSNb+JNsnS04LmvgV4Wzjvt/JT3ajdzJ4viG8xyO +fRAH/r5T0/wXsm+/BbOt7s+UlEA39wMhpyYVGoANQwtHgt0KR/io9enuLtPpdj4D +k09ZHnAiE8TzBm8pSSkdsoEiCe7ZEJreY7JqEgyIc56MGG2skMtoMpfANoBySiYr +W9BWPf3N8OSFYbVn +-----END CERTIFICATE----- diff --git a/resources/device/tests/test_frida_worker.py b/resources/device/tests/test_frida_worker.py new file mode 100644 index 0000000..60a1dc7 --- /dev/null +++ b/resources/device/tests/test_frida_worker.py @@ -0,0 +1,348 @@ +"""Worker contract tests; fake Frida calls are not Android coverage.""" +import builtins +import importlib.util +import json +from pathlib import Path +import ssl +import subprocess +import tempfile +import time +import unittest +from unittest import mock + +SOURCE = Path(__file__).resolve().parents[1] / "frida_worker.py" +SPEC = importlib.util.spec_from_file_location("scope_frida_worker", SOURCE) +WORKER = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(WORKER) +FIXTURES = Path(__file__).resolve().parent / "fixtures" +# Public synthetic certificates only; their disposable signing keys are not retained. +PEM = (FIXTURES / "ca.pem").read_text(encoding="ascii").strip() + + +class WorkerTests(unittest.TestCase): + def test_profiles_are_shared_normalized_and_unsupported_fail(self): + self.assertTrue(hasattr(WORKER, "normalize_profiles"), "worker provides normalized profile contract") + self.assertEqual(WORKER.normalize_profiles([]), ["proxy", "certificate", "unpinning"]) + self.assertEqual(WORKER.normalize_profiles(["certificate", "proxy", "certificate"]), ["proxy", "certificate"]) + for profiles in [["root"], ["system-ca"], ["unpinning"], ["unknown"], "proxy", [None]]: + with self.assertRaises(ValueError): + WORKER.normalize_profiles(profiles) + + def test_builder_serializes_data_and_loads_only_owned_engine_and_bridge(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + ca = root / "ca.pem" + ca.write_text(PEM) + (root / "frida-java-bridge.js").write_text("// bridge\n") + (root / "scope-engine.js").write_text("globalThis.ScopeJavaEngine = {start(c) { globalThis.observed = c; }};\n") + source = WORKER.build_script(root, ca, 17890, ["proxy"]) + result = subprocess.run(["node", "-e", source + "\nconsole.log(JSON.stringify(globalThis.observed))"], capture_output=True, text=True, check=True) + self.assertEqual(json.loads(result.stdout), {"caPem": PEM, "proxyHost": "127.0.0.1", "proxyPort": 17890, "profiles": ["proxy"]}) + ca.write_text(PEM + "`;globalThis.pwned=true;//") + with self.assertRaises(ValueError): + WORKER.build_script(root, ca, 17890, ["proxy"]) + ca.write_text(PEM) + for port in [0, 65536, True, "17890;evil()"]: + with self.assertRaises(ValueError): + WORKER.build_script(root, ca, port, ["proxy"]) + + def test_builder_serializes_public_ca_with_canonical_lf_for_all_newlines(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + ca = root / "ca.pem" + (root / "frida-java-bridge.js").write_text("// bridge\n") + (root / "scope-engine.js").write_text("globalThis.ScopeJavaEngine = {start(c) { globalThis.observed = c; }};\n") + for newline in [b"\n", b"\r\n", b"\r"]: + with self.subTest(newline=newline): + # Explicit bytes reproduce every platform independently of text I/O. + ca.write_bytes(PEM.encode("ascii").replace(b"\n", newline)) + try: + source = WORKER.build_script(root, ca, 17890, ["proxy"]) + except ValueError as error: + self.fail("Valid public CA newline form was rejected: " + str(error)) + result = subprocess.run(["node", "-e", source + "\nconsole.log(JSON.stringify(globalThis.observed))"], capture_output=True, text=True, check=True) + self.assertEqual(json.loads(result.stdout)["caPem"], PEM) + + def test_builder_requires_one_complete_public_ca(self): + # Missing DER/CA validation would serialize these inputs to the device. + der = ssl.PEM_cert_to_DER_cert(PEM) + # Complete PKCS#1 RSAPrivateKey ASN.1 with tiny known integers (n=15). + # This unusable synthetic key needs no crypto tool or secret material. + key_der = b"\x30\x1b" + b"".join( + b"\x02\x01" + bytes([value]) for value in (0, 15, 3, 3, 3, 5, 1, 3, 2) + ) + cases = { + "malformed-der": ssl.DER_cert_to_PEM_cert(b"abc"), + "relabelled-key": ssl.DER_cert_to_PEM_cert(key_der), + "trailing-der": ssl.DER_cert_to_PEM_cert(der + b"abc"), + "leaf": (FIXTURES / "leaf.pem").read_text(encoding="ascii"), + "multiple-certificates": PEM + "\n" + PEM, + "private-label": "-----BEGIN PRIVATE KEY-----\nYWJj\n-----END PRIVATE KEY-----", + "oversized": " " * 65537, + "non-ascii": "\N{SNOWMAN}", + } + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + ca = root / "input.pem" + (root / "frida-java-bridge.js").write_text("// bridge") + (root / "scope-engine.js").write_text("// engine") + for name, content in cases.items(): + with self.subTest(case=name): + ca.write_text(content, encoding="utf-8") + with self.assertRaisesRegex(ValueError, "Scope CA"): + WORKER.build_script(root, ca, 17890, ["proxy"]) + + def test_main_rejects_invalid_ca_before_frida_import_or_device_work(self): + # An import guard catches regression before any real Frida/device action. + original_import = builtins.__import__ + def guarded_import(name, *args, **kwargs): + if name == "frida": + self.fail("Frida import reached before public CA validation") + return original_import(name, *args, **kwargs) + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + ca = root / "input.pem" + ca.write_text("-----BEGIN CERTIFICATE-----\nYWJj\n-----END CERTIFICATE-----") + (root / "frida-java-bridge.js").write_text("// bridge") + (root / "scope-engine.js").write_text("// engine") + argv = ["frida_worker", "--serial", "synthetic", "--package", "app.test", + "--ca", str(ca), "--scripts", str(root), "--profile", "proxy"] + with mock.patch.object(WORKER.sys, "argv", argv), mock.patch("builtins.__import__", side_effect=guarded_import): + with self.assertRaisesRegex(ValueError, "Scope CA"): + WORKER.main() + + def test_exact_ca_size_boundary_and_new_contents_are_revalidated(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "frida-java-bridge.js").write_text("// bridge") + (root / "scope-engine.js").write_text("// engine") + ca = root / "ca.pem" + public = PEM.encode("ascii") + ca.write_bytes(public + b" " * (65536 - len(public))) + self.assertIn("ScopeJavaEngine.start", WORKER.build_script(root, ca, 17890, ["proxy"])) + ca.write_bytes(ca.read_bytes() + b" ") + with self.assertRaisesRegex(ValueError, "64 KiB"): + WORKER.build_script(root, ca, 17890, ["proxy"]) + ca.write_bytes(b"not a certificate") + with self.assertRaisesRegex(ValueError, "Scope CA"): + WORKER.build_script(root, ca, 17890, ["proxy"]) + + def test_ready_gate_requires_engine_exact_profiles_and_real_capabilities(self): + self.assertTrue(hasattr(WORKER, "validate_ready"), "worker validates installed capabilities") + good = {"kind": "scope-hooks-ready", "engine": "scope-java-v1", "profiles": ["proxy"], "capabilities": {"proxy": ["java-url"]}} + self.assertEqual(WORKER.validate_ready(good, ["proxy"]), good) + for changes in [{"engine": "other"}, {"profiles": []}, {"capabilities": {}}, {"capabilities": {"proxy": []}}, {"capabilities": {"proxy": ["not-implemented"]}}]: + with self.assertRaises(ValueError): + WORKER.validate_ready({**good, **changes}, ["proxy"]) + + def test_owned_cleanup_kills_exact_pid_before_unload_and_detach(self): + self.assertTrue(hasattr(WORKER, "cleanup_owned"), "worker terminates its instrumented PID") + calls = [] + device = mock.Mock() + device.kill.side_effect = lambda pid: calls.append(("kill", pid)) + script = mock.Mock() + script.unload.side_effect = lambda: calls.append(("unload",)) + session = mock.Mock() + session.detach.side_effect = lambda: calls.append(("detach",)) + WORKER.cleanup_owned(device, 427, script, session, timeout=0.2) + self.assertEqual(calls, [("kill", 427), ("unload",), ("detach",)]) + calls.clear() + WORKER.cleanup_owned(device, None, None, None, timeout=0.2) + self.assertEqual(calls, []) + + def test_cleanup_errors_and_timeouts_are_visible_and_bounded(self): + self.assertTrue(hasattr(WORKER, "cleanup_owned")) + device, script, session = mock.Mock(), mock.Mock(), mock.Mock() + device.kill.side_effect = RuntimeError("permission denied") + with self.assertRaisesRegex(RuntimeError, "kill.*permission denied"): + WORKER.cleanup_owned(device, 427, script, session, timeout=0.05) + script.unload.assert_called_once() + session.detach.assert_called_once() + device.kill.side_effect = lambda _pid: time.sleep(0.3) + start = time.monotonic() + with self.assertRaisesRegex(RuntimeError, "kill.*timed out"): + WORKER.cleanup_owned(device, 427, None, None, timeout=0.02) + self.assertLess(time.monotonic() - start, 0.2) + + def test_cli_rejects_unsupported_profile_before_import_or_device_contact(self): + with mock.patch.object(WORKER.sys, "argv", ["frida_worker", "--profile", "root"]): + with self.assertRaisesRegex(ValueError, "profile"): + WORKER.main() + +class SessionTests(unittest.TestCase): + def fixture(self, behavior): + calls = [] + device, session, script = mock.Mock(), mock.Mock(), mock.Mock() + device.spawn.side_effect = lambda argv: calls.append(("spawn", argv)) or 427 + device.attach.return_value = session + session.create_script.return_value = script + callback = [] + script.on.side_effect = lambda _name, handler: callback.append(handler) + script.load.side_effect = lambda: behavior(callback[0]) + device.kill.side_effect = lambda pid: calls.append(("kill", pid)) + script.unload.side_effect = lambda: calls.append(("unload",)) + session.detach.side_effect = lambda: calls.append(("detach",)) + return device, session, script, calls + + def test_stop_before_ready_cleans_owned_suspended_process_without_ready(self): + stopped = WORKER.threading.Event() + device, session, script, calls = self.fixture(lambda _cb: stopped.set()) + output = [] + with mock.patch.object(WORKER, "emit", side_effect=lambda kind, **fields: output.append({"kind": kind, **fields})): + start = time.monotonic() + WORKER.run_session(device, "app.test", "source", ["proxy"], "17.9.1", stopped, ready_timeout=30, cleanup_timeout=0.1) + self.assertLess(time.monotonic() - start, 0.5) + self.assertEqual(calls, [("spawn", ["app.test"]), ("kill", 427), ("unload",), ("detach",)]) + device.resume.assert_not_called() + self.assertFalse(any(row["kind"] == "ready" for row in output)) + self.assertEqual(output[0], {"kind": "spawned", "engine": "scope-java-v1", "pid": 427}) + self.assertEqual(output[-1]["processTerminated"], True) + self.assertEqual(output[-1]["pid"], 427) + + def test_late_ready_cannot_revive_hook_error_and_startup_failure_kills_pid(self): + stopped = WORKER.threading.Event() + def behavior(callback): + callback({"type": "send", "payload": {"kind": "scope-hooks-error", "message": "no pinner"}}, None) + callback({"type": "send", "payload": {"kind": "scope-hooks-ready", "engine": "scope-java-v1", "profiles": ["proxy"], "capabilities": {"proxy": ["java-url"]}}}, None) + device, session, script, calls = self.fixture(behavior) + output = [] + with mock.patch.object(WORKER, "emit", side_effect=lambda kind, **fields: output.append({"kind": kind, **fields})): + with self.assertRaisesRegex(RuntimeError, "no pinner"): + WORKER.run_session(device, "app.test", "source", ["proxy"], "17.9.1", stopped, cleanup_timeout=0.1) + self.assertIn(("kill", 427), calls) + self.assertFalse(any(row["kind"] == "ready" for row in output)) + + def test_failed_cleanup_never_emits_clean_stopped(self): + stopped = WORKER.threading.Event() + device, session, script, calls = self.fixture(lambda _cb: stopped.set()) + device.kill.side_effect = RuntimeError("cannot kill") + output = [] + with mock.patch.object(WORKER, "emit", side_effect=lambda kind, **fields: output.append({"kind": kind, **fields})): + with self.assertRaisesRegex(RuntimeError, "cannot kill"): + WORKER.run_session(device, "app.test", "source", ["proxy"], "17.9.1", stopped, cleanup_timeout=0.1) + self.assertFalse(any(row["kind"] == "stopped" for row in output)) + +class DestroyedCleanupTests(unittest.TestCase): + def test_already_destroyed_script_and_detached_session_need_no_second_cleanup(self): + device, script, session = mock.Mock(), mock.Mock(), mock.Mock() + script.is_destroyed = True + session.is_detached = True + script.unload.side_effect = RuntimeError("script has been destroyed") + session.detach.side_effect = RuntimeError("already detached") + WORKER.cleanup_owned(device, 427, script, session, timeout=0.1) + device.kill.assert_called_once_with(427) + script.unload.assert_not_called() + session.detach.assert_not_called() + +class StartupCancellationTests(unittest.TestCase): + fixture = SessionTests.fixture + def test_stop_cancels_inflight_script_load_then_kills_only_owned_pid(self): + stopped = WORKER.threading.Event() + cancelled = WORKER.threading.Event() + class Cancellable: + def __enter__(self): + return self + def __exit__(self, *_args): + pass + def cancel(self): + cancelled.set() + device, session, script, calls = self.fixture(lambda _cb: None) + def blocked_load(): + stopped.set() + if not cancelled.wait(0.5): + raise RuntimeError("startup was not cancelled") + raise RuntimeError("operation cancelled") + script.load.side_effect = blocked_load + with mock.patch.object(WORKER, "emit"): + with self.assertRaisesRegex(RuntimeError, "operation cancelled"): + WORKER.run_session(device, "app.test", "source", ["proxy"], "17.9.1", stopped, cleanup_timeout=0.1, cancellable_factory=Cancellable) + self.assertTrue(cancelled.is_set()) + self.assertIn(("kill", 427), calls) + +class ReviewedLifecycleTests(unittest.TestCase): + fixture = SessionTests.fixture + + def test_pre_spawn_stop_has_explicit_safe_no_spawn_receipt(self): + stopped = WORKER.threading.Event() + stopped.set() + device = mock.Mock() + output = [] + with mock.patch.object(WORKER, "emit", side_effect=lambda kind, **fields: output.append({"kind": kind, **fields})): + WORKER.run_session(device, "app.test", "source", ["proxy"], "17.9.1", stopped) + device.spawn.assert_not_called() + self.assertEqual(output, [{"kind": "stopped", "engine": "scope-java-v1", "pid": None, "spawnAttempted": False, "cleanupComplete": True, "processTerminated": False}]) + + def test_replacement_before_kill_dispatch_never_kills_new_image(self): + stopped = WORKER.threading.Event() + device, session, script, calls = self.fixture(lambda _cb: stopped.set()) + detached = [] + session.on.side_effect = lambda _name, handler: detached.append(handler) + bounded = WORKER.bounded_call + def interleave(label, operation, timeout): + if label == "kill owned application": + detached[0]("process-replaced") + return bounded(label, operation, timeout) + output = [] + with mock.patch.object(WORKER, "bounded_call", side_effect=interleave), mock.patch.object(WORKER, "emit", side_effect=lambda kind, **fields: output.append({"kind": kind, **fields})): + with self.assertRaisesRegex(RuntimeError, "replaced|ownership"): + WORKER.run_session(device, "app.test", "source", ["proxy"], "17.9.1", stopped, cleanup_timeout=0.1) + device.kill.assert_not_called() + self.assertFalse(any(row["kind"] == "stopped" for row in output)) + + def test_replacement_cancels_dispatched_kill_and_retains_uncertainty(self): + stopped = WORKER.threading.Event() + cancelled = WORKER.threading.Event() + class Cancellable: + def __enter__(self): return self + def __exit__(self, *_args): pass + def cancel(self): cancelled.set() + device, session, script, calls = self.fixture(lambda _cb: stopped.set()) + detached = [] + session.on.side_effect = lambda _name, handler: detached.append(handler) + def kill(_pid): + detached[0]("process-replaced") + if not cancelled.wait(0.2): + raise RuntimeError("pending kill was not cancelled") + device.kill.side_effect = kill + with mock.patch.object(WORKER, "emit") as output: + with self.assertRaisesRegex(RuntimeError, "unknown|uncertain"): + WORKER.run_session(device, "app.test", "source", ["proxy"], "17.9.1", stopped, cleanup_timeout=0.3, cancellable_factory=Cancellable) + self.assertTrue(cancelled.is_set()) + self.assertFalse(any(call.args[0] == "stopped" for call in output.call_args_list)) + + +class OwnershipEvidenceTests(unittest.TestCase): + fixture = SessionTests.fixture + + def test_attempted_spawn_without_returned_pid_never_claims_no_spawn_cleanup(self): + stopped = WORKER.threading.Event() + device = mock.Mock() + device.spawn.side_effect = RuntimeError("connection lost during spawn") + with mock.patch.object(WORKER, "emit") as output: + with self.assertRaisesRegex(RuntimeError, "completion is unknown"): + WORKER.run_session(device, "app.test", "source", ["proxy"], "17.9.1", stopped) + device.kill.assert_not_called() + self.assertFalse(any(call.args[0] == "stopped" for call in output.call_args_list)) + + def test_observed_exit_before_kill_dispatch_skips_pid_and_acknowledges_termination(self): + stopped = WORKER.threading.Event() + device, session, script, calls = self.fixture(lambda _cb: stopped.set()) + detached = [] + session.on.side_effect = lambda _name, handler: detached.append(handler) + bounded = WORKER.bounded_call + def interleave(label, operation, timeout): + if label == "kill owned application": + detached[0]("process-terminated") + return bounded(label, operation, timeout) + with mock.patch.object(WORKER, "bounded_call", side_effect=interleave), mock.patch.object(WORKER, "emit") as output: + WORKER.run_session(device, "app.test", "source", ["proxy"], "17.9.1", stopped, cleanup_timeout=0.1) + device.kill.assert_not_called() + receipts = [call.kwargs for call in output.call_args_list if call.args[0] == "stopped"] + self.assertEqual(len(receipts), 1) + self.assertIs(receipts[0]["processTerminated"], True) + self.assertIs(receipts[0]["spawnAttempted"], True) + self.assertIs(receipts[0]["cleanupComplete"], True) + + +if __name__ == "__main__": + unittest.main() diff --git a/resources/frida/android/android-certificate-unpinning-fallback.js b/resources/frida/android/android-certificate-unpinning-fallback.js deleted file mode 100644 index 2a2641f..0000000 --- a/resources/frida/android/android-certificate-unpinning-fallback.js +++ /dev/null @@ -1,267 +0,0 @@ -/************************************************************************************************** - * - * Once we've set up the configuration and certificate, and then disabled all the pinning - * techniques we're aware of, we add one last touch: a fallback hook, designed to spot and handle - * unknown unknowns. - * - * This can also be useful for heavily obfuscated apps, where 3rd party libraries are obfuscated - * sufficiently that our hooks no longer recognize the methods we care about. - * - * To handle this, we watch for methods that throw known built-in TLS errors (these are *very* - * widely used, and always recognizable as they're defined natively), and then subsequently patch - * them for all future calls. Whenever a method throws this, we attempt to recognize it from - * signatures alone, and automatically hook it. - * - * These are very much a fallback! They might not work! They almost certainly won't work on the - * first request, so applications will see at least one failure. Even when they fail though, they - * will at least log the method that's failing, so this works well as a starting point for manual - * reverse engineering. If this does fail and cause problems, you may want to skip this script - * and use only the known-good patches provided elsewhere. - * - * Source available at https://github.com/httptoolkit/frida-interception-and-unpinning/ - * SPDX-License-Identifier: AGPL-3.0-or-later - * SPDX-FileCopyrightText: Tim Perry - * - *************************************************************************************************/ - -// Capture the full fields or methods from a Frida class reference via JVM reflection: -const getFields = (cls) => getFridaValues(cls, cls.class.getDeclaredFields()); -const getMethods = (cls) => getFridaValues(cls, cls.class.getDeclaredMethods()); - -// Take a Frida class + JVM reflection result, and turn it into a clear list -// of names -> Frida values (field or method references) -const getFridaValues = (cls, values) => values.map((value) => - [value.getName(), cls[value.getName()]] -); - -Java.perform(function () { - try { - const X509TrustManager = Java.use("javax.net.ssl.X509TrustManager"); - const defaultTrustManager = getCustomX509TrustManager(); // Defined in the unpinning script - const certBytes = Java.use("java.lang.String").$new(CERT_PEM).getBytes(); - const trustedCACert = buildX509CertificateFromBytes(certBytes); // Ditto - - const isX509TrustManager = (cls, methodName) => - methodName === 'checkServerTrusted' && - X509TrustManager.class.isAssignableFrom(cls.class); - - // There are two standard methods that X509TM implementations might override. We confirm we're - // matching the methods we expect by double-checking against the argument types: - const BASE_METHOD_ARGUMENTS = [ - '[Ljava.security.cert.X509Certificate;', - 'java.lang.String' - ]; - const EXTENDED_METHOD_ARGUMENTS = [ - '[Ljava.security.cert.X509Certificate;', - 'java.lang.String', - 'java.lang.String' - ]; - - const isOkHttpCheckMethod = (errorMessage, method) => - errorMessage.startsWith("Certificate pinning failure!" + "\n Peer certificate chain:") && - method.argumentTypes.length === 2 && - method.argumentTypes[0].className === 'java.lang.String'; - - const isAppmattusOkHttpInterceptMethod = (errorMessage, method) => { - if (errorMessage !== 'Certificate transparency failed') return; - - // Takes a single OkHttp chain argument: - if (method.argumentTypes.length !== 1) return; - - // The method must take an Interceptor.Chain, for which we need to - // call chain.proceed(chain.request()) to return a Response type. - // To do that, we effectively pattern match our way through all the - // related types to work out what's what: - - const chainType = Java.use(method.argumentTypes[0].className); - const responseTypeName = method.returnType.className; - - const matchedChain = matchOkHttpChain(chainType, responseTypeName); - return !!matchedChain; - }; - - const isMetaPinningMethod = (errorMessage, method) => - method.argumentTypes.length === 1 && - method.argumentTypes[0].className === 'java.util.List' && - method.returnType.className === 'void' && - errorMessage.includes('pinning error'); - - const matchOkHttpChain = (cls, expectedReturnTypeName) => { - // Find the chain.proceed() method: - const methods = getMethods(cls); - const matchingMethods = methods.filter(([_, method]) => - method.returnType.className === expectedReturnTypeName - ); - if (matchingMethods.length !== 1) return; - - const [proceedMethodName, proceedMethod] = matchingMethods[0]; - if (proceedMethod.argumentTypes.length !== 1) return; - - const argumentTypeName = proceedMethod.argumentTypes[0].className; - - // Find the chain.request private field (.request() getter can be - // optimized out, so we read the field directly): - const fields = getFields(cls); - const matchingFields = fields.filter(([_, field]) => - field.fieldReturnType?.className === argumentTypeName - ); - if (matchingFields.length !== 1) return; - - const [requestFieldName] = matchingFields[0]; - - return { - proceedMethodName, - requestFieldName - }; - }; - - const buildUnhandledErrorPatcher = (errorClassName, originalConstructor) => { - return function (errorArg) { - try { - console.log('\n !!! --- Unexpected TLS failure --- !!!'); - - // This may be a message, or an cause, or plausibly maybe other types? But - // stringifying gives something consistently message-shaped, so that'll do. - const errorMessage = errorArg?.toString() ?? ''; - - // Parse the stack trace to work out who threw this error: - const stackTrace = Java.use('java.lang.Thread').currentThread().getStackTrace(); - const exceptionStackIndex = stackTrace.findIndex(stack => - stack.getClassName() === errorClassName - ); - const callingFunctionStack = stackTrace[exceptionStackIndex + 1]; - - const className = callingFunctionStack.getClassName(); - const methodName = callingFunctionStack.getMethodName(); - - const errorTypeName = errorClassName.split('.').slice(-1)[0]; - console.log(` ${errorTypeName}: ${errorMessage}`); - console.log(` Thrown by ${className}->${methodName}`); - - const callingClass = Java.use(className); - const callingMethod = callingClass[methodName]; - - callingMethod.overloads.forEach((failingMethod) => { - if (failingMethod.implementation) { - console.warn(' Already patched - but still failing!') - return; // Already patched by Frida - skip it - } - - // Try to spot known methods (despite obfuscation) and disable them: - if (isOkHttpCheckMethod(errorMessage, failingMethod)) { - // See okhttp3.CertificatePinner patches in unpinning script: - failingMethod.implementation = () => { - if (DEBUG_MODE) console.log(` => Fallback OkHttp patch`); - }; - console.log(` [+] ${className}->${methodName} (fallback OkHttp patch)`); - } else if (isAppmattusOkHttpInterceptMethod(errorMessage, failingMethod)) { - // See Appmattus CertificateTransparencyInterceptor patch in unpinning script: - const chainType = Java.use(failingMethod.argumentTypes[0].className); - const responseTypeName = failingMethod.returnType.className; - const okHttpChain = matchOkHttpChain(chainType, responseTypeName); - failingMethod.implementation = (chain) => { - if (DEBUG_MODE) console.log(` => Fallback Appmattus+OkHttp patch`); - const proceed = chain[okHttpChain.proceedMethodName].bind(chain); - const request = chain[okHttpChain.requestFieldName].value; - return proceed(request); - }; - console.log(` [+] ${className}->${methodName} (fallback Appmattus+OkHttp patch)`); - } else if (isX509TrustManager(callingClass, methodName)) { - const argumentTypes = failingMethod.argumentTypes.map(t => t.className); - const returnType = failingMethod.returnType.className; - - if ( - argumentTypes.length === 2 && - argumentTypes.every((t, i) => t === BASE_METHOD_ARGUMENTS[i]) && - returnType === 'void' - ) { - // For the base method, just check against the default: - failingMethod.implementation = (certs, authType) => { - if (DEBUG_MODE) console.log(` => Fallback X509TrustManager patch of ${ - className - } base method`); - - const defaultTrustManager = getCustomX509TrustManager(); // Defined in the unpinning script - defaultTrustManager.checkServerTrusted(certs, authType); - }; - console.log(` [+] ${className}->${methodName} (fallback X509TrustManager base patch)`); - } else if ( - argumentTypes.length === 3 && - argumentTypes.every((t, i) => t === EXTENDED_METHOD_ARGUMENTS[i]) && - returnType === 'java.util.List' - ) { - // For the extended method, we just ignore the hostname, and if the certs are good - // (i.e they're ours), then we say the whole chain is good to go: - failingMethod.implementation = function (certs, authType, _hostname) { - if (DEBUG_MODE) console.log(` => Fallback X509TrustManager patch of ${ - className - } extended method`); - - try { - defaultTrustManager.checkServerTrusted(certs, authType); - } catch (e) { - console.error('Default TM threw:', e); - } - return Java.use('java.util.Arrays').asList(certs); - }; - console.log(` [+] ${className}->${methodName} (fallback X509TrustManager ext patch)`); - } else { - console.warn(` [ ] Skipping unrecognized checkServerTrusted signature in class ${ - callingClass.class.getName() - }`); - } - } else if (isMetaPinningMethod(errorMessage, failingMethod)) { - failingMethod.implementation = function (certs) { - if (DEBUG_MODE) console.log(` => Fallback patch for meta proxygen pinning`); - for (const cert of certs.toArray()) { - if (cert.equals(trustedCACert)) { - return; // Our own cert - all good - } - } - - if (DEBUG_MODE) { - console.warn(' Meta unpinning fallback found only untrusted certificates'); - } - // Fall back to normal logic, in case of passthrough or similar - return failingMethod.call(this, certs); - } - - console.log(` [+] ${className}->${methodName} (Meta proxygen pinning fallback patch)`); - } else { - console.error(' [ ] Unrecognized TLS error - this must be patched manually'); - return; - // Later we could try to cover other cases here - automatically recognizing other - // OkHttp interceptors for example, or potentially other approaches, but we need - // to do so carefully to avoid disabling TLS checks entirely. - } - }); - } catch (e) { - console.log(' [ ] Failed to automatically patch failure'); - console.warn(e); - } - - return originalConstructor.call(this, ...arguments); - } - }; - - // These are the exceptions we watch for and attempt to auto-patch out after they're thrown: - [ - 'javax.net.ssl.SSLPeerUnverifiedException', - 'java.security.cert.CertificateException' - ].forEach((errorClassName) => { - const ErrorClass = Java.use(errorClassName); - ErrorClass.$init.overloads.forEach((overload) => { - overload.implementation = buildUnhandledErrorPatcher( - errorClassName, - overload - ); - }); - }) - - console.log('== Unpinning fallback auto-patcher installed =='); - } catch (err) { - console.error(err); - console.error(' !!! --- Unpinning fallback auto-patcher installation failed --- !!!'); - } - -}); \ No newline at end of file diff --git a/resources/frida/android/android-certificate-unpinning.js b/resources/frida/android/android-certificate-unpinning.js deleted file mode 100644 index 4df1c91..0000000 --- a/resources/frida/android/android-certificate-unpinning.js +++ /dev/null @@ -1,539 +0,0 @@ -/************************************************************************************************** - * - * This script defines a large set of targeted certificate unpinning hooks: matching specific - * methods in certain classes, and transforming their behaviour to ensure that restrictions to - * TLS trust are disabled. - * - * This does not disable TLS protections completely - each hook is designed to disable only - * *additional* restrictions, and to explicitly trust the certificate provided as CERT_PEM in the - * config.js configuration file, preserving normal TLS protections wherever possible, even while - * allowing for controlled MitM of local traffic. - * - * The file consists of a few general-purpose methods, then a data structure declaratively - * defining the classes & methods to match, and how to transform them, and then logic at the end - * which uses this data structure, applying the transformation for each found match to the - * target process. - * - * For more details on what was matched, and log output when each hooked method is actually used, - * enable DEBUG_MODE in config.js, and watch the Frida output after running this script. - * - * Source available at https://github.com/httptoolkit/frida-interception-and-unpinning/ - * SPDX-License-Identifier: AGPL-3.0-or-later - * SPDX-FileCopyrightText: Tim Perry - * - *************************************************************************************************/ - -function buildX509CertificateFromBytes(certBytes) { - const ByteArrayInputStream = Java.use('java.io.ByteArrayInputStream'); - const CertFactory = Java.use('java.security.cert.CertificateFactory'); - const certFactory = CertFactory.getInstance("X.509"); - return certFactory.generateCertificate(ByteArrayInputStream.$new(certBytes)); -} - -function getCustomTrustManagerFactory() { - // This is the one X509Certificate that we want to trust. No need to trust others (we should capture - // _all_ TLS traffic) and risky to trust _everything_ (risks interception between device & proxy, or - // worse: some traffic being unintercepted & sent as HTTPS with TLS effectively disabled over the - // real web - potentially exposing auth keys, private data and all sorts). - const certBytes = Java.use("java.lang.String").$new(CERT_PEM).getBytes(); - const trustedCACert = buildX509CertificateFromBytes(certBytes); - - // Build a custom TrustManagerFactory with a KeyStore that trusts only this certificate: - - const KeyStore = Java.use("java.security.KeyStore"); - const keyStore = KeyStore.getInstance(KeyStore.getDefaultType()); - keyStore.load(null); - keyStore.setCertificateEntry("ca", trustedCACert); - - const TrustManagerFactory = Java.use("javax.net.ssl.TrustManagerFactory"); - const customTrustManagerFactory = TrustManagerFactory.getInstance( - TrustManagerFactory.getDefaultAlgorithm() - ); - customTrustManagerFactory.init(keyStore); - - return customTrustManagerFactory; -} - -function getCustomX509TrustManager() { - const customTrustManagerFactory = getCustomTrustManagerFactory(); - const trustManagers = customTrustManagerFactory.getTrustManagers(); - - const X509TrustManager = Java.use('javax.net.ssl.X509TrustManager'); - - const x509TrustManager = trustManagers.find((trustManager) => { - return trustManager.class.isAssignableFrom(X509TrustManager.class); - }); - - // We have to cast it explicitly before Frida will allow us to use the X509 methods: - return Java.cast(x509TrustManager, X509TrustManager); -} - -// Some standard hook replacements for various cases: -const NO_OP = () => {}; -const RETURN_TRUE = () => true; -const CHECK_OUR_TRUST_MANAGER_ONLY = () => { - const trustManager = getCustomX509TrustManager(); - return (certs, authType) => { - trustManager.checkServerTrusted(certs, authType); - }; -}; - -const PINNING_FIXES = { - // --- Native HttpsURLConnection - - 'javax.net.ssl.HttpsURLConnection': [ - { - methodName: 'setDefaultHostnameVerifier', - replacement: () => NO_OP - }, - { - methodName: 'setSSLSocketFactory', - replacement: () => NO_OP - }, - { - methodName: 'setHostnameVerifier', - replacement: () => NO_OP - }, - ], - - // --- Native SSLContext - - 'javax.net.ssl.SSLContext': [ - { - methodName: 'init', - overload: ['[Ljavax.net.ssl.KeyManager;', '[Ljavax.net.ssl.TrustManager;', 'java.security.SecureRandom'], - replacement: (targetMethod) => { - const customTrustManagerFactory = getCustomTrustManagerFactory(); - - // When constructor is called, replace the trust managers argument: - return function (keyManager, _providedTrustManagers, secureRandom) { - return targetMethod.call(this, - keyManager, - customTrustManagerFactory.getTrustManagers(), // Override their trust managers - secureRandom - ); - } - } - } - ], - - // --- Native Conscrypt CertPinManager - - 'com.android.org.conscrypt.CertPinManager': [ - { - methodName: 'isChainValid', - replacement: () => RETURN_TRUE - }, - { - methodName: 'checkChainPinning', - replacement: () => NO_OP - } - ], - - // --- Native pinning configuration loading (used for configuration by many libraries) - - 'android.security.net.config.NetworkSecurityConfig': [ - { - methodName: '$init', - overload: '*', - replacement: (targetMethod) => { - const PinSet = Java.use('android.security.net.config.PinSet'); - const EMPTY_PINSET = PinSet.EMPTY_PINSET.value; - return function () { - // Always ignore the 2nd 'pins' PinSet argument entirely: - arguments[2] = EMPTY_PINSET; - targetMethod.call(this, ...arguments); - } - } - } - ], - - // --- Native HostnameVerification override (n.b. Android contains its own vendored OkHttp v2!) - - 'com.android.okhttp.internal.tls.OkHostnameVerifier': [ - { - methodName: 'verify', - overload: [ - 'java.lang.String', - 'javax.net.ssl.SSLSession' - ], - replacement: (targetMethod) => { - // Our trust manager - this trusts *only* our extra CA - const trustManager = getCustomX509TrustManager(); - - return function (hostname, sslSession) { - try { - const certs = sslSession.getPeerCertificates(); - - // https://stackoverflow.com/a/70469741/68051 - const authType = "RSA"; - - // This throws if the certificate isn't trusted (i.e. if it's - // not signed by our extra CA specifically): - trustManager.checkServerTrusted(certs, authType); - - // If the cert is from our CA, great! Skip hostname checks entirely. - return true; - } catch (e) {} // Ignore errors and fallback to default behaviour - - // We fallback to ensure that connections with other CAs (e.g. direct - // connections allowed past the proxy) validate as normal. - return targetMethod.call(this, ...arguments); - } - } - } - ], - - 'com.android.okhttp.Address': [ - { - methodName: '$init', - overload: [ - 'java.lang.String', - 'int', - 'com.android.okhttp.Dns', - 'javax.net.SocketFactory', - 'javax.net.ssl.SSLSocketFactory', - 'javax.net.ssl.HostnameVerifier', - 'com.android.okhttp.CertificatePinner', - 'com.android.okhttp.Authenticator', - 'java.net.Proxy', - 'java.util.List', - 'java.util.List', - 'java.net.ProxySelector' - ], - replacement: (targetMethod) => { - const defaultHostnameVerifier = Java.use("com.android.okhttp.internal.tls.OkHostnameVerifier") - .INSTANCE.value; - const defaultCertPinner = Java.use("com.android.okhttp.CertificatePinner") - .DEFAULT.value; - - return function () { - // Override arguments, to swap any custom check params (widely used - // to add stricter rules to TLS verification) with the defaults instead: - arguments[5] = defaultHostnameVerifier; - arguments[6] = defaultCertPinner; - - targetMethod.call(this, ...arguments); - } - } - }, - // Almost identical patch, but for Nougat and older. In these versions, the DNS argument - // isn't passed here, so the arguments to patch changes slightly: - { - methodName: '$init', - overload: [ - 'java.lang.String', - 'int', - // No DNS param - 'javax.net.SocketFactory', - 'javax.net.ssl.SSLSocketFactory', - 'javax.net.ssl.HostnameVerifier', - 'com.android.okhttp.CertificatePinner', - 'com.android.okhttp.Authenticator', - 'java.net.Proxy', - 'java.util.List', - 'java.util.List', - 'java.net.ProxySelector' - ], - replacement: (targetMethod) => { - const defaultHostnameVerifier = Java.use("com.android.okhttp.internal.tls.OkHostnameVerifier") - .INSTANCE.value; - const defaultCertPinner = Java.use("com.android.okhttp.CertificatePinner") - .DEFAULT.value; - - return function () { - // Override arguments, to swap any custom check params (widely used - // to add stricter rules to TLS verification) with the defaults instead: - arguments[4] = defaultHostnameVerifier; - arguments[5] = defaultCertPinner; - - targetMethod.call(this, ...arguments); - } - } - } - ], - - // --- OkHttp v3 - - 'okhttp3.CertificatePinner': [ - { - methodName: 'check', - overload: ['java.lang.String', 'java.util.List'], - replacement: () => NO_OP - }, - { - methodName: 'check', - overload: ['java.lang.String', 'java.security.cert.Certificate'], - replacement: () => NO_OP - }, - { - methodName: 'check', - overload: ['java.lang.String', '[Ljava.security.cert.Certificate;'], - replacement: () => NO_OP - }, - { - methodName: 'check$okhttp', - replacement: () => NO_OP - }, - ], - - // --- SquareUp OkHttp (< v3) - - 'com.squareup.okhttp.CertificatePinner': [ - { - methodName: 'check', - overload: ['java.lang.String', 'java.security.cert.Certificate'], - replacement: () => NO_OP - }, - { - methodName: 'check', - overload: ['java.lang.String', 'java.util.List'], - replacement: () => NO_OP - } - ], - - // --- Trustkit (https://github.com/datatheorem/TrustKit-Android/) - - 'com.datatheorem.android.trustkit.pinning.PinningTrustManager': [ - { - methodName: 'checkServerTrusted', - replacement: CHECK_OUR_TRUST_MANAGER_ONLY - } - ], - - // --- Appcelerator (https://github.com/tidev/appcelerator.https) - - 'appcelerator.https.PinningTrustManager': [ - { - methodName: 'checkServerTrusted', - replacement: CHECK_OUR_TRUST_MANAGER_ONLY - } - ], - - // --- PhoneGap sslCertificateChecker (https://github.com/EddyVerbruggen/SSLCertificateChecker-PhoneGap-Plugin) - - 'nl.xservices.plugins.sslCertificateChecker': [ - { - methodName: 'execute', - overload: ['java.lang.String', 'org.json.JSONArray', 'org.apache.cordova.CallbackContext'], - replacement: () => (_action, _args, context) => { - context.success("CONNECTION_SECURE"); - return true; - } - // This trusts _all_ certs, but that's fine - this is used for checks of independent test - // connections, rather than being a primary mechanism to secure the app's TLS connections. - } - ], - - // --- IBM WorkLight - - 'com.worklight.wlclient.api.WLClient': [ - { - methodName: 'pinTrustedCertificatePublicKey', - getMethod: (WLClientCls) => WLClientCls.getInstance().pinTrustedCertificatePublicKey, - overload: '*' - } - ], - - 'com.worklight.wlclient.certificatepinning.HostNameVerifierWithCertificatePinning': [ - { - methodName: 'verify', - overload: '*', - replacement: () => NO_OP - } - // This covers at least 4 commonly used WorkLight patches. Oddly, most sets of hooks seem - // to return true for 1/4 cases, which must be wrong (overloads must all have the same - // return type) but also it's very hard to find any modern (since 2017) references to this - // class anywhere including WorkLight docs, so it may no longer be relevant anyway. - ], - - 'com.worklight.androidgap.plugin.WLCertificatePinningPlugin': [ - { - methodName: 'execute', - overload: '*', - replacement: () => RETURN_TRUE - } - ], - - // --- CWAC-Netsecurity (unofficial back-port pinner for Android<4.2) CertPinManager - - 'com.commonsware.cwac.netsecurity.conscrypt.CertPinManager': [ - { - methodName: 'isChainValid', - overload: '*', - replacement: () => RETURN_TRUE - } - ], - - // --- Netty - - 'io.netty.handler.ssl.util.FingerprintTrustManagerFactory': [ - { - methodName: 'checkTrusted', - replacement: () => NO_OP - } - ], - - // --- Cordova / PhoneGap Advanced HTTP Plugin (https://github.com/silkimen/cordova-plugin-advanced-http) - - // Modern version: - 'com.silkimen.cordovahttp.CordovaServerTrust': [ - { - methodName: '$init', - replacement: (targetMethod) => function () { - // Ignore any attempts to set trust to 'pinned'. Default settings will trust - // our cert because of the separate system-certificate injection step. - if (arguments[0] === 'pinned') { - arguments[0] = 'default'; - } - - return targetMethod.call(this, ...arguments); - } - } - ], - - // --- Appmattus Cert Transparency (https://github.com/appmattus/certificatetransparency/) - - 'com.appmattus.certificatetransparency.internal.verifier.CertificateTransparencyHostnameVerifier': [ - { - methodName: 'verify', - replacement: () => RETURN_TRUE - // This is not called unless the cert passes basic trust checks, so it's safe to blindly accept. - } - ], - - 'com.appmattus.certificatetransparency.internal.verifier.CertificateTransparencyInterceptor': [ - { - methodName: 'intercept', - replacement: () => (a) => a.proceed(a.request()) - // This is not called unless the cert passes basic trust checks, so it's safe to blindly accept. - } - ], - - 'com.appmattus.certificatetransparency.internal.verifier.CertificateTransparencyTrustManager': [ - { - methodName: 'checkServerTrusted', - overload: ['[Ljava.security.cert.X509Certificate;', 'java.lang.String'], - replacement: CHECK_OUR_TRUST_MANAGER_ONLY, - methodName: 'checkServerTrusted', - overload: ['[Ljava.security.cert.X509Certificate;', 'java.lang.String', 'java.lang.String'], - replacement: () => { - const trustManager = getCustomX509TrustManager(); - return (certs, authType, _hostname) => { - // We ignore the hostname - if the certs are good (i.e they're ours), then the - // whole chain is good to go. - trustManager.checkServerTrusted(certs, authType); - return Java.use('java.util.Arrays').asList(certs); - }; - } - } - ] - -}; - -const getJavaClassIfExists = (clsName) => { - try { - return Java.use(clsName); - } catch { - return undefined; - } -} - -Java.perform(function () { - if (DEBUG_MODE) console.log('\n === Disabling all recognized unpinning libraries ==='); - - const classesToPatch = Object.keys(PINNING_FIXES); - - classesToPatch.forEach((targetClassName) => { - const TargetClass = getJavaClassIfExists(targetClassName); - if (!TargetClass) { - // We skip patches for any classes that don't seem to be present. This is common - // as not all libraries we handle are necessarily used. - if (DEBUG_MODE) console.log(`[ ] ${targetClassName} *`); - return; - } - - const patches = PINNING_FIXES[targetClassName]; - - let patchApplied = false; - - patches.forEach(({ methodName, getMethod, overload, replacement }) => { - const namedTargetMethod = getMethod - ? getMethod(TargetClass) - : TargetClass[methodName]; - - const methodDescription = `${methodName}${ - overload === '*' - ? '(*)' - : overload - ? '(' + overload.map((argType) => { - // Simplify arg names to just the class name for simpler logs: - const argClassName = argType.split('.').slice(-1)[0]; - if (argType.startsWith('[L')) return `${argClassName}[]`; - else return argClassName; - }).join(', ') + ')' - // No overload: - : '' - }` - - let targetMethodImplementations = []; - try { - if (namedTargetMethod) { - if (!overload) { - // No overload specified - targetMethodImplementations = [namedTargetMethod]; - } else if (overload === '*') { - // Targetting _all_ overloads - targetMethodImplementations = namedTargetMethod.overloads; - } else { - // Or targetting a specific overload: - targetMethodImplementations = [namedTargetMethod.overload(...overload)]; - } - } - } catch (e) { - // Overload not present - } - - - // We skip patches for any methods that don't seem to be present. This is rarer, but does - // happen due to methods that only appear in certain library versions or whose signatures - // have changed over time. - if (targetMethodImplementations.length === 0) { - if (DEBUG_MODE) console.log(`[ ] ${targetClassName} ${methodDescription}`); - return; - } - - targetMethodImplementations.forEach((targetMethod, i) => { - const patchName = `${targetClassName} ${methodDescription}${ - targetMethodImplementations.length > 1 ? ` (${i})` : '' - }`; - - try { - const newImplementation = replacement(targetMethod); - if (DEBUG_MODE) { - // Log each hooked method as it's called: - targetMethod.implementation = function () { - console.log(` => ${patchName}`); - return newImplementation.apply(this, arguments); - } - } else { - targetMethod.implementation = newImplementation; - } - - if (DEBUG_MODE) console.log(`[+] ${patchName}`); - patchApplied = true; - } catch (e) { - // In theory, errors like this should never happen - it means the patch is broken - // (e.g. some dynamic patch building fails completely) - console.error(`[!] ERROR: ${patchName} failed: ${e}`); - } - }) - }); - - if (!patchApplied) { - console.warn(`[!] Matched class ${targetClassName} but could not patch any methods`); - } - }); - - console.log('== Certificate unpinning completed =='); -}); \ No newline at end of file diff --git a/resources/frida/android/android-disable-root-detection.js b/resources/frida/android/android-disable-root-detection.js deleted file mode 100644 index aa5a204..0000000 --- a/resources/frida/android/android-disable-root-detection.js +++ /dev/null @@ -1,400 +0,0 @@ -/************************************************************************************************** - * - * This script defines a large set of root detection bypasses for Android. Hooks included here - * block detection of many known root indicators, including file paths, package names, commands, - * notably binaries, and system properties. - * - * Enable DEBUG_MODE to see debug output for each bypassed check. - * - * Source available at https://github.com/httptoolkit/frida-interception-and-unpinning/ - * SPDX-License-Identifier: AGPL-3.0-or-later - * SPDX-FileCopyrightText: Tim Perry - * SPDX-FileCopyrightText: Riyad Mondal - * - *************************************************************************************************/ - -(() => { - let loggedRootDetectionWarning = false; - function logFirstRootDetection() { - if (!loggedRootDetectionWarning) { - console.log(" => Blocked possible root detection checks. Enable DEBUG_MODE for more details."); - loggedRootDetectionWarning = true; - } - } - - const LIB_C = Process.findModuleByName("libc.so"); - - const BUILD_FINGERPRINT_REGEX = /^([\w.-]+\/[\w.-]+\/[\w.-]+):([\w.]+\/[\w.-]+\/[\w.-]+):(\w+\/[\w,.-]+)$/; - - const CONFIG = { - secureProps: { - "ro.secure": "1", - "ro.debuggable": "0", - "ro.build.type": "user", - "ro.build.tags": "release-keys" - } - }; - - const ROOT_INDICATORS = { - paths: new Set([ - "/data/local/bin/su", - "/data/local/su", - "/data/local/xbin/su", - "/dev/com.koushikdutta.superuser.daemon/", - "/sbin/su", - "/su/bin/su", - "/system/bin/su", - "/system/xbin/su", - "/system/sbin/su", - "/vendor/bin/su", - "/data/adb/su/bin/su", - "/system/bin/failsafe/su", - "/system/bin/.ext/.su", - "/system/bin/.ext/su", - "/system/bin/failsafe/su", - "/system/sd/xbin/su", - "/system/usr/we-need-root/su", - "/cache/su", - "/data/su", - "/dev/su", - "/data/adb/magisk", - "/sbin/.magisk", - "/cache/.disable_magisk", - "/dev/.magisk.unblock", - "/cache/magisk.log", - "/data/adb/magisk.img", - "/data/adb/magisk.db", - "/data/adb/magisk_simple", - "/init.magisk.rc", - "/system/app/Superuser.apk", - "/system/etc/init.d/99SuperSUDaemon", - "/system/xbin/daemonsu", - "/system/xbin/ku.sud", - "/data/adb/ksu", - "/data/adb/ksud", - "/system/xbin/busybox", - "/system/app/Kinguser.apk" - ]), - - packages: new Set([ - "com.noshufou.android.su", - "com.noshufou.android.su.elite", - "eu.chainfire.supersu", - "com.koushikdutta.superuser", - "com.thirdparty.superuser", - "com.yellowes.su", - "com.koushikdutta.rommanager", - "com.koushikdutta.rommanager.license", - "com.dimonvideo.luckypatcher", - "com.chelpus.lackypatch", - "com.ramdroid.appquarantine", - "com.ramdroid.appquarantinepro", - "com.topjohnwu.magisk", - "me.weishu.kernelsu" - ]), - - commands: new Set([ - "su", - "which su", - "whereis su", - "locate su", - "find / -name su", - "mount", - "magisk", - "/system/bin/su", - "/system/xbin/su", - "/sbin/su", - "/su/bin/su" - ]), - - binaries: new Set([ - "su", - "busybox", - "magisk", - "supersu", - "ksud", - "daemonsu" - ]) - }; - - function bypassNativeFileCheck() { - const fopen = LIB_C.findExportByName("fopen"); - if (fopen) { - Interceptor.attach(fopen, { - onEnter(args) { - this.path = args[0].readUtf8String(); - }, - onLeave(retval) { - if (retval.toInt32() !== 0) { - const path = this.path.toLowerCase(); - if (ROOT_INDICATORS.paths.has(this.path) || path.includes("magisk") || path.includes("/su") || path.endsWith("/su")) { - if (DEBUG_MODE) { - console.log(`Blocked possible root-detection: fopen ${this.path}`); - } else logFirstRootDetection(); - retval.replace(ptr(0x0)); - } - } - } - }); - } - - const access = LIB_C.findExportByName("access"); - if (access) { - Interceptor.attach(access, { - onEnter(args) { - this.path = args[0].readUtf8String(); - }, - onLeave(retval) { - if (retval.toInt32() === 0) { - const path = this.path.toLowerCase(); - if (ROOT_INDICATORS.paths.has(this.path) || path.includes("magisk") || path.includes("/su") || path.endsWith("/su")) { - if (DEBUG_MODE) { - console.debug(`Blocked possible root detection: access ${this.path}`); - } else logFirstRootDetection(); - retval.replace(ptr(-1)); - } - } - } - }); - } - - const stat = LIB_C.findExportByName("stat"); - if (stat) { - Interceptor.attach(stat, { - onEnter(args) { - this.path = args[0].readUtf8String(); - }, - onLeave(retval) { - const path = this.path.toLowerCase(); - if (ROOT_INDICATORS.paths.has(this.path) || path.includes("magisk") || path.includes("/su") || path.endsWith("/su")) { - if (DEBUG_MODE) { - console.debug(`Blocked possible root detection: stat ${this.path}`); - } else logFirstRootDetection(); - retval.replace(ptr(-1)); - } - } - }); - } - - const lstat = LIB_C.findExportByName("lstat"); - if (lstat) { - Interceptor.attach(lstat, { - onEnter(args) { - this.path = args[0].readUtf8String(); - }, - onLeave(retval) { - const path = this.path.toLowerCase(); - if (ROOT_INDICATORS.paths.has(this.path) || path.includes("magisk") || path.includes("/su") || path.endsWith("/su")) { - if (DEBUG_MODE) { - console.debug(`Blocked possible root detection: lstat ${this.path}`); - } else logFirstRootDetection(); - retval.replace(ptr(-1)); - } - } - }); - } - } - - function bypassJavaFileCheck() { - function isRootIndicatorFile(file) { - const path = file.getAbsolutePath(); - const filename = file.getName(); - return ROOT_INDICATORS.paths.has(path) || - path.includes("magisk") || - filename === "su"; - } - - const UnixFileSystem = Java.use("java.io.UnixFileSystem"); - UnixFileSystem.checkAccess.implementation = function(file, access) { - if (isRootIndicatorFile(file)) { - if (DEBUG_MODE) { - console.debug(`Blocked possible root detection: filesystem access check for ${file.getAbsolutePath()}`); - } else logFirstRootDetection(); - return false; - } - return this.checkAccess(file, access); - }; - - const File = Java.use("java.io.File"); - File.exists.implementation = function() { - if (isRootIndicatorFile(this)) { - if (DEBUG_MODE) { - console.debug(`Blocked possible root detection: file exists check for ${this.getAbsolutePath()}`); - } else logFirstRootDetection(); - return false; - } - return this.exists(); - }; - - File.length.implementation = function() { - if (isRootIndicatorFile(this)) { - if (DEBUG_MODE) { - console.debug(`Blocked possible root detection: file length check for ${this.getAbsolutePath()}`); - } else logFirstRootDetection(); - return 0; - } - return this.length(); - }; - - const FileInputStream = Java.use("java.io.FileInputStream"); - FileInputStream.$init.overload('java.io.File').implementation = function(file) { - if (isRootIndicatorFile(file)) { - if (DEBUG_MODE) { - console.debug(`Blocked possible root detection: file stream for ${file.getAbsolutePath()}`); - } else logFirstRootDetection(); - throw Java.use("java.io.FileNotFoundException").$new(file.getAbsolutePath()); - } - return this.$init(file); - }; - } - - function setProp() { - const Build = Java.use("android.os.Build"); - - // We do a little work to make the minimum changes required to hide in the BUILD fingerprint, - // but otherwise keep matching the real device wherever possible. - const realFingerprint = Build.FINGERPRINT.value; - - const fingerprintMatch = BUILD_FINGERPRINT_REGEX.exec(realFingerprint); - let fixedFingerprint; - if (fingerprintMatch) { - let [, device, versions, tags] = BUILD_FINGERPRINT_REGEX.exec(realFingerprint); - tags = 'user/release-keys'; // Should always be the case in production builds - if (device.includes('generic') || device.includes('sdk') || device.includes('lineage')) { - device = 'google/raven/raven'; - } - - fixedFingerprint = `${device}:${versions}:${tags}`; - } else { - console.warn(`Unexpected BUILD fingerprint format: ${realFingerprint}`); - // This should never happen in theory (the format is standard), but just in case, - // we use this fallback fingerprint: - fixedFingerprint = "google/crosshatch/crosshatch:10/QQ3A.200805.001/6578210:user/release-keys"; - } - - const fields = { - "TAGS": "release-keys", - "TYPE": "user", - "FINGERPRINT": fixedFingerprint - }; - - Object.entries(fields).forEach(([field, value]) => { - const fieldObj = Build.class.getDeclaredField(field); - fieldObj.setAccessible(true); - fieldObj.set(null, value); - }); - - const system_property_get = LIB_C.findExportByName("__system_property_get"); - if (system_property_get) { - Interceptor.attach(system_property_get, { - onEnter(args) { - this.key = args[0].readCString(); - this.ret = args[1]; - }, - onLeave(retval) { - const secureValue = CONFIG.secureProps[this.key]; - if (secureValue !== undefined) { - if (DEBUG_MODE) { - console.debug(`Blocked possible root detection: system_property_get ${this.key}`); - } else logFirstRootDetection(); - const valuePtr = Memory.allocUtf8String(secureValue); - Memory.copy(this.ret, valuePtr, secureValue.length + 1); - } - } - }); - } - - const Runtime = Java.use('java.lang.Runtime'); - Runtime.exec.overload('java.lang.String').implementation = function(cmd) { - if (cmd.startsWith("getprop ")) { - const prop = cmd.split(" ")[1]; - if (CONFIG.secureProps[prop]) { - if (DEBUG_MODE) { - console.debug(`Blocked possible root detection: getprop ${prop}`); - } else logFirstRootDetection(); - return null; - } - } - return this.exec(cmd); - }; - } - - function bypassRootPackageCheck() { - const ApplicationPackageManager = Java.use("android.app.ApplicationPackageManager"); - - ApplicationPackageManager.getPackageInfo.overload('java.lang.String', 'int').implementation = function(str, i) { - if (ROOT_INDICATORS.packages.has(str)) { - if (DEBUG_MODE) { - console.debug(`Blocked possible root detection: package info for ${str}`); - } else logFirstRootDetection(); - str = "invalid.example.nonexistent.package"; - } - return this.getPackageInfo(str, i); - }; - - ApplicationPackageManager.getInstalledPackages.overload('int').implementation = function(flags) { - const packages = this.getInstalledPackages(flags); - const packageList = packages.toArray(); - const filteredPackages = packageList.filter(pkg => !ROOT_INDICATORS.packages.has(pkg.packageName?.value)); - return Java.use("java.util.ArrayList").$new(Java.use("java.util.Arrays").asList(filteredPackages)); - }; - } - - function bypassShellCommands() { - const ProcessBuilder = Java.use('java.lang.ProcessBuilder'); - ProcessBuilder.command.overload('java.util.List').implementation = function(commands) { - const cmdArray = commands.toArray(); - if (cmdArray.length > 0) { - const cmd = cmdArray[0].toString(); - if (ROOT_INDICATORS.commands.has(cmd) || (cmdArray.length > 1 && ROOT_INDICATORS.binaries.has(cmdArray[1].toString()))) { - if (DEBUG_MODE) { - console.debug(`Blocked possible root detection: ProcessBuilder with ${cmdArray.join(' ')}`); - } else logFirstRootDetection(); - return this.command(Java.use("java.util.Arrays").asList([""])); - } - } - return this.command(commands); - }; - - const Runtime = Java.use('java.lang.Runtime'); - Runtime.exec.overload('[Ljava.lang.String;').implementation = function(cmdArray) { - if (cmdArray.length > 0) { - const cmd = cmdArray[0]; - if (ROOT_INDICATORS.commands.has(cmd) || (cmdArray.length > 1 && ROOT_INDICATORS.binaries.has(cmdArray[1]))) { - if (DEBUG_MODE) { - console.debug(`Blocked possible root detection: Runtime.exec for ${cmdArray.join(' ')}`); - } else logFirstRootDetection(); - return this.exec([""]); - } - } - return this.exec(cmdArray); - }; - - const ProcessImpl = Java.use("java.lang.ProcessImpl"); - ProcessImpl.start.implementation = function(cmdArray, env, dir, redirects, redirectErrorStream) { - if (cmdArray.length > 0) { - const cmd = cmdArray[0].toString(); - const arg = cmdArray.length > 1 ? cmdArray[1].toString() : ""; - - if (ROOT_INDICATORS.commands.has(cmd) || ROOT_INDICATORS.binaries.has(arg)) { - if (DEBUG_MODE) { - console.debug(`Blocked possible root detection: ProcessImpl.start for ${cmdArray.join(' ')}`); - } else logFirstRootDetection(); - return ProcessImpl.start.call(this, [Java.use("java.lang.String").$new("")], env, dir, redirects, redirectErrorStream); - } - } - return ProcessImpl.start.call(this, cmdArray, env, dir, redirects, redirectErrorStream); - }; - } - - try { - bypassNativeFileCheck(); - bypassJavaFileCheck(); - setProp(); - bypassRootPackageCheck(); - bypassShellCommands(); - console.log("== Disabled Android root detection =="); - } catch (error) { - console.error("\n !!! Error setting up root detection bypass !!!", error); - } -})(); diff --git a/resources/frida/android/android-proxy-override.js b/resources/frida/android/android-proxy-override.js deleted file mode 100644 index d4248e6..0000000 --- a/resources/frida/android/android-proxy-override.js +++ /dev/null @@ -1,98 +0,0 @@ -/************************************************************************************************** - * - * The first step in intercepting HTTP & HTTPS traffic is to set the default proxy settings, - * telling the app that all requests should be sent via our HTTP proxy. - * - * In this script, we set that up via a few different mechanisms, which cumulatively should - * ensure that all connections are sent via the proxy, even if they attempt to use their - * own custom proxy configurations to avoid this. - * - * Despite that, this still only covers well behaved apps - it's still possible for apps - * to send network traffic directly if they're determined to do so, or if they're built - * with a framework that does not do this by default (Flutter is notably in this category). - * To handle those less tidy cases, we manually capture traffic to recognized target ports - * in the native connect() hook script. - * - * Source available at https://github.com/httptoolkit/frida-interception-and-unpinning/ - * SPDX-License-Identifier: AGPL-3.0-or-later - * SPDX-FileCopyrightText: Tim Perry - * - *************************************************************************************************/ - -Java.perform(() => { - // Set default JVM system properties for the proxy address. Notably these are used - // to initialize WebView configuration. - Java.use('java.lang.System').setProperty('http.proxyHost', PROXY_HOST); - Java.use('java.lang.System').setProperty('http.proxyPort', PROXY_PORT.toString()); - Java.use('java.lang.System').setProperty('https.proxyHost', PROXY_HOST); - Java.use('java.lang.System').setProperty('https.proxyPort', PROXY_PORT.toString()); - - Java.use('java.lang.System').clearProperty('http.nonProxyHosts'); - Java.use('java.lang.System').clearProperty('https.nonProxyHosts'); - - // Some Android internals attempt to reset these settings to match the device configuration. - // We block that directly here: - const controlledSystemProperties = [ - 'http.proxyHost', - 'http.proxyPort', - 'https.proxyHost', - 'https.proxyPort', - 'http.nonProxyHosts', - 'https.nonProxyHosts' - ]; - Java.use('java.lang.System').clearProperty.implementation = function (property) { - if (controlledSystemProperties.includes(property)) { - if (DEBUG_MODE) console.log(`Ignoring attempt to clear ${property} system property`); - return this.getProperty(property); - } - return this.clearProperty(...arguments); - } - Java.use('java.lang.System').setProperty.implementation = function (property) { - if (controlledSystemProperties.includes(property)) { - if (DEBUG_MODE) console.log(`Ignoring attempt to override ${property} system property`); - return this.getProperty(property); - } - return this.setProperty(...arguments); - } - - // Configure the app's proxy directly, via the app connectivity manager service: - const ConnectivityManager = Java.use('android.net.ConnectivityManager'); - const ProxyInfo = Java.use('android.net.ProxyInfo'); - ConnectivityManager.getDefaultProxy.implementation = () => ProxyInfo.$new(PROXY_HOST, PROXY_PORT, ''); - // (Not clear if this works 100% - implying there are ConnectivityManager subclasses handling this) - - console.log(`== Proxy system configuration overridden to ${PROXY_HOST}:${PROXY_PORT} ==`); - - // Configure the proxy indirectly, by overriding the return value for all ProxySelectors everywhere: - const Collections = Java.use('java.util.Collections'); - const ProxyType = Java.use('java.net.Proxy$Type'); - const InetSocketAddress = Java.use('java.net.InetSocketAddress'); - const ProxyCls = Java.use('java.net.Proxy'); // 'Proxy' is reserved in JS - - const targetProxy = ProxyCls.$new( - ProxyType.HTTP.value, - InetSocketAddress.$new(PROXY_HOST, PROXY_PORT) - ); - const getTargetProxyList = () => Collections.singletonList(targetProxy); - - const ProxySelector = Java.use('java.net.ProxySelector'); - - // Find every implementation of ProxySelector by quickly scanning method signatures, and - // then checking whether each match actually implements java.net.ProxySelector: - const proxySelectorClasses = Java.enumerateMethods('*!select(java.net.URI): java.util.List/s') - .flatMap((matchingLoader) => matchingLoader.classes - .map((classData) => Java.use(classData.name)) - .filter((Cls) => ProxySelector.class.isAssignableFrom(Cls.class)) - ); - - // Replace the 'select' of every implementation, so they all send traffic to us: - proxySelectorClasses.forEach(ProxySelectorCls => { - if (DEBUG_MODE) { - console.log('Rewriting', ProxySelectorCls.toString()); - } - ProxySelectorCls.select.implementation = () => getTargetProxyList() - }); - - console.log(`== Proxy configuration overridden to ${PROXY_HOST}:${PROXY_PORT} ==`); -}); - diff --git a/resources/frida/android/android-system-certificate-injection.js b/resources/frida/android/android-system-certificate-injection.js deleted file mode 100644 index 5b30422..0000000 --- a/resources/frida/android/android-system-certificate-injection.js +++ /dev/null @@ -1,88 +0,0 @@ -/************************************************************************************************** - * - * Once we have captured traffic (once it's being sent to our proxy port) the next step is - * to ensure any clients using TLS (HTTPS) trust our CA certificate, to allow us to intercept - * encrypted connections successfully. - * - * This script does so by attaching to the internals of Conscrypt (the Android SDK's standard - * TLS implementation) and pre-adding our certificate to the 'already trusted' cache, so that - * future connections trust it implicitly. This ensures that all normal uses of Android APIs - * for HTTPS & TLS will allow interception. - * - * This does not handle all standalone certificate pinning techniques - where the application - * actively rejects certificates that are trusted by default on the system. That's dealt with - * in the separate certificate unpinning script. - * - * Source available at https://github.com/httptoolkit/frida-interception-and-unpinning/ - * SPDX-License-Identifier: AGPL-3.0-or-later - * SPDX-FileCopyrightText: Tim Perry - * - *************************************************************************************************/ - -Java.perform(() => { - // First, we build a JVM representation of our certificate: - const String = Java.use("java.lang.String"); - const ByteArrayInputStream = Java.use('java.io.ByteArrayInputStream'); - const CertFactory = Java.use('java.security.cert.CertificateFactory'); - - let cert; - try { - const certFactory = CertFactory.getInstance("X.509"); - const certBytes = String.$new(CERT_PEM).getBytes(); - cert = certFactory.generateCertificate(ByteArrayInputStream.$new(certBytes)); - } catch (e) { - console.error('Could not parse provided certificate PEM!'); - console.error(e); - Java.use('java.lang.System').exit(1); - } - - // Then we hook TrustedCertificateIndex. This is used for caching known trusted certs within Conscrypt - - // by prepopulating all instances, we ensure that all TrustManagerImpls (and potentially other - // things) automatically trust our certificate specifically (without disabling validation entirely). - // This should apply to Android v7+ - previous versions used SSLContext & X509TrustManager. - [ - 'com.android.org.conscrypt.TrustedCertificateIndex', - 'org.conscrypt.TrustedCertificateIndex', // Might be used (com.android is synthetic) - unclear - 'org.apache.harmony.xnet.provider.jsse.TrustedCertificateIndex' // Used in Apache Harmony version of Conscrypt - ].forEach((TrustedCertificateIndexClassname, i) => { - let TrustedCertificateIndex; - try { - TrustedCertificateIndex = Java.use(TrustedCertificateIndexClassname); - } catch (e) { - if (i === 0) { - throw new Error(`${TrustedCertificateIndexClassname} not found - could not inject system certificate`); - } else { - // Other classnames are optional fallbacks - if (DEBUG_MODE) { - console.log(`[ ] Skipped cert injection for ${TrustedCertificateIndexClassname} (not present)`); - } - return; - } - } - - TrustedCertificateIndex.$init.overloads.forEach((overload) => { - overload.implementation = function () { - this.$init(...arguments); - // Index our cert as already trusted, right from the start: - this.index(cert); - } - }); - - TrustedCertificateIndex.reset.overloads.forEach((overload) => { - overload.implementation = function () { - const result = this.reset(...arguments); - // Index our cert in here again, since the reset removes it: - this.index(cert); - return result; - }; - }); - - if (DEBUG_MODE) console.log(`[+] Injected cert into ${TrustedCertificateIndexClassname}`); - }); - - // This effectively adds us to the system certs, and also defeats quite a bit of basic certificate - // pinning too! It auto-trusts us in any implementation that uses TrustManagerImpl (Conscrypt) as - // the underlying cert checking component. - - console.log('== System certificate trust injected =='); -}); \ No newline at end of file diff --git a/resources/frida/config-template.js b/resources/frida/config-template.js deleted file mode 100644 index 42326bf..0000000 --- a/resources/frida/config-template.js +++ /dev/null @@ -1,255 +0,0 @@ -/************************************************************************************************** - * - * This file defines various config parameters, used later within the other scripts. - * - * In all cases, you'll want to set CERT_PEM and likely PROXY_HOST and PROXY_PORT. - * - * Source available at https://github.com/httptoolkit/frida-interception-and-unpinning/ - * SPDX-License-Identifier: AGPL-3.0-or-later - * SPDX-FileCopyrightText: Tim Perry - * - *************************************************************************************************/ - -// Put your CA certificate data here in PEM format: -const CERT_PEM = `-----BEGIN CERTIFICATE----- - -[!! Put your CA certificate data here, in PEM format !!] - ------END CERTIFICATE-----`; - -// Put your intercepting proxy's address here: -const PROXY_HOST = '127.0.0.1'; -const PROXY_PORT = 8000; - -// If you like, set to to true to enable extra logging: -const DEBUG_MODE = false; - -// If you find issues with non-HTTP traffic being captured (due to the -// native connect hook script) you can add ports here to exempt traffic -// on that port from being redirected. Note that this will only affect -// traffic captured by the raw connection hook - for apps using the -// system HTTP proxy settings, traffic on these ports will still be -// sent via the proxy and intercepted despite this setting. -const IGNORED_NON_HTTP_PORTS = []; - -// As HTTP/3 is often not well supported by MitM proxies, by default it -// is blocked entirely, so all outgoing UDP connections to port 443 -// will fail. If this is set to false, they will instead be left unintercepted. -const BLOCK_HTTP3 = true; - -// Set this to true if your proxy supports SOCKS5 connections. -// This makes it possible for native-connect-hook to redirect -// non-HTTP traffic through your proxy (to view it raw, and -// avoid breaking non-HTTP traffic en route). -const PROXY_SUPPORTS_SOCKS5 = false; - - -// ---------------------------------------------------------------------------- -// You don't need to modify any of the below, it just checks and applies some -// of the configuration that you've entered above. -// ---------------------------------------------------------------------------- - - -if (DEBUG_MODE) { - // Add logging just for clean output & to separate reloads: - console.log('\n*** Starting scripts ***'); - if (globalThis.Java?.available) { - Java.perform(() => { - setTimeout(() => console.log('*** Scripts completed ***\n'), 5); - // (We assume that nothing else will take more than 5ms, but app startup - // probably will, so this should separate script & runtime logs) - }); - } else { - setTimeout(() => console.log('*** Scripts completed ***\n'), 5); - // (We assume that nothing else will take more than 5ms, but app startup - // probably will, so this should separate script & runtime logs) - } -} else { - console.log(''); // Add just a single newline, for minimal clarity -} - -// Check the certificate (without literally including the instruction phrasing -// here, as that can be confusing for some users): -if (CERT_PEM.match(/\[!!.* CA certificate data .* !!\]/)) { - throw new Error('No certificate was provided' + - '\n\n' + - 'You need to set CERT_PEM in the Frida config script ' + - 'to the contents of your CA certificate.' - ); -} - - - -// ---------------------------------------------------------------------------- -// Don't modify any of the below unless you know what you're doing! -// This section defines various utilities & calculates some constants which may -// be used by later scripts elsewhere in this project. -// ---------------------------------------------------------------------------- - - - -// As web atob & Node.js Buffer aren't available, we need to reimplement base64 decoding -// in pure JS. This is a quick rough implementation without much error handling etc! - -// Base64 character set (plus padding character =) and lookup: -const BASE64_CHARS = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/='; -const BASE64_LOOKUP = new Uint8Array(123); -for (let i = 0; i < BASE64_CHARS.length; i++) { - BASE64_LOOKUP[BASE64_CHARS.charCodeAt(i)] = i; -} - - -/** - * Take a base64 string, and return the raw bytes - * @param {string} input - * @returns Uint8Array - */ -function decodeBase64(input) { - // Calculate the length of the output buffer based on padding: - let outputLength = Math.floor((input.length * 3) / 4); - if (input[input.length - 1] === '=') outputLength--; - if (input[input.length - 2] === '=') outputLength--; - - const output = new Uint8Array(outputLength); - let outputPos = 0; - - // Process each 4-character block: - for (let i = 0; i < input.length; i += 4) { - const a = BASE64_LOOKUP[input.charCodeAt(i)]; - const b = BASE64_LOOKUP[input.charCodeAt(i + 1)]; - const c = BASE64_LOOKUP[input.charCodeAt(i + 2)]; - const d = BASE64_LOOKUP[input.charCodeAt(i + 3)]; - - // Assemble into 3 bytes: - const chunk = (a << 18) | (b << 12) | (c << 6) | d; - - // Add each byte to the output buffer, unless it's padding: - output[outputPos++] = (chunk >> 16) & 0xff; - if (input.charCodeAt(i + 2) !== 61) output[outputPos++] = (chunk >> 8) & 0xff; - if (input.charCodeAt(i + 3) !== 61) output[outputPos++] = chunk & 0xff; - } - - return output; -} - -/** - * Take a single-certificate PEM string, and return the raw DER bytes - * @param {string} input - * @returns Uint8Array - */ -function pemToDer(input) { - const pemLines = input.split('\n'); - if ( - pemLines[0] !== '-----BEGIN CERTIFICATE-----' || - pemLines[pemLines.length- 1] !== '-----END CERTIFICATE-----' - ) { - throw new Error( - 'Your certificate should be in PEM format, starting & ending ' + - 'with a BEGIN CERTIFICATE & END CERTIFICATE header/footer' - ); - } - - const base64Data = pemLines.slice(1, -1).map(l => l.trim()).join(''); - if ([...base64Data].some(c => !BASE64_CHARS.includes(c))) { - throw new Error( - 'Your certificate should be in PEM format, containing only ' + - 'base64 data between a BEGIN & END CERTIFICATE header/footer' - ); - } - - return decodeBase64(base64Data); -} - -const CERT_DER = pemToDer(CERT_PEM); - -// Right now this API is a bit funky - the callback will be called with a Frida Module instance -// if the module is properly detected, but may be called with just { name, path, base, size } -// in some cases (e.g. shared libraries loaded from inside an APK on Android). Works OK right now, -// as it's not widely used but needs improvement in future if we extend this. -function waitForModule(moduleName, callback) { - if (Array.isArray(moduleName)) { - moduleName.forEach(module => waitForModule(module, callback)); - } - - try { - const module = Process.getModuleByName(moduleName) - module.ensureInitialized(); - callback(module); - return; - } catch (e) { - try { - const module = Module.load(moduleName); - callback(module); - return; - } catch (e) {} - } - - MODULE_LOAD_CALLBACKS[moduleName] = callback; -} - -const getModuleName = (nameOrPath) => { - const endOfPath = nameOrPath.lastIndexOf('/'); - return nameOrPath.slice(endOfPath + 1); -}; - -const MODULE_LOAD_CALLBACKS = {}; -new ApiResolver('module').enumerateMatches('exports:linker*!*dlopen*').forEach((dlopen) => { - Interceptor.attach(dlopen.address, { - onEnter(args) { - const moduleArg = args[0].readCString(); - if (moduleArg) { - this.path = moduleArg; - this.moduleName = getModuleName(moduleArg); - } - }, - onLeave(retval) { - if (!this.path || !retval || retval.isNull()) return; - if (!MODULE_LOAD_CALLBACKS[this.moduleName]) return; - - let module = Process.findModuleByName(this.moduleName) - ?? Process.findModuleByAddress(retval); - if (!module) { - // Some modules are loaded in ways that mean Frida can't detect them, and - // can't look them up by name (notably when loading libraries from inside an - // APK on Android). To handle this, we can use dlsym to look up an example - // symbol and find the underlying module details directly, where possible. - module = getAnonymousModule(this.moduleName, this.path, retval); - if (!module) return; - } - - Object.keys(MODULE_LOAD_CALLBACKS).forEach((key) => { - if (this.moduleName === key) { - if (module) { - MODULE_LOAD_CALLBACKS[key](module); - delete MODULE_LOAD_CALLBACKS[key]; - } - } - }); - } - }); -}); - -const getAnonymousModule = (name, path, handle) => { - const dlsymAddr = Module.findGlobalExportByName('dlsym'); - if (!dlsymAddr) { - console.error(`[!] Cannot find dlsym, cannot get anonymous module info for ${name}`); - return; - } - - const dlsym = new NativeFunction(dlsymAddr, 'pointer', ['pointer', 'pointer']); - - // Handle here is the return value from dlopen - but in this scenario, it's just an - // opaque handle into to 'soinfo' data that other methods can use to get the - // real pointer to parts of the module, like so: - const onLoadPointer = dlsym(handle, Memory.allocUtf8String('JNI_OnLoad')); - - // Once we have an actual pointer, we can get the range that holds it: - const range = Process.getRangeByAddress(onLoadPointer); - - return { - base: range.base, - size: range.size, - name, - path, - } -}; \ No newline at end of file diff --git a/resources/frida/scope-engine.js b/resources/frida/scope-engine.js new file mode 100644 index 0000000..202d28e --- /dev/null +++ b/resources/frida/scope-engine.js @@ -0,0 +1,288 @@ +/* Scope-owned Java interception engine. Uses the documented Frida Java API. + * Coverage is deliberately limited to java.net.URL, SSLContext and OkHttp3. + * Method rollback cannot revoke already-created clients: the worker must end + * its spawned process on Stop or failure. No hostname verifier is replaced. + */ +(function (root) { + 'use strict'; + const PROFILES = ['proxy', 'certificate', 'unpinning']; + const REQUIRED = { proxy: 'java-url', certificate: 'sslcontext-scope-ca', unpinning: 'okhttp3-certificate-pinner' }; + + function normalizeProfiles(profiles) { + if (!Array.isArray(profiles) || profiles.some(p => !PROFILES.includes(p))) { + throw new Error('Unsupported hook profile; supported: proxy, certificate, unpinning'); + } + const selected = profiles.length ? PROFILES.filter(p => profiles.includes(p)) : PROFILES.slice(); + if (selected.includes('unpinning') && !selected.includes('certificate')) { + throw new Error('unpinning profile requires certificate'); + } + return selected; + } + + function createPolicy(validateScopeChain, rejectionTypes = {}) { + function isScope(chain, context) { + try { validateScopeChain(chain, context); return true; } catch (_) { return false; } + } + return { + checkPins(chain, original) { + try { return original(); } catch (rejection) { + if (!rejectionTypes.isPinRejection || !rejectionTypes.isPinRejection(rejection) || !isScope(chain)) throw rejection; + } + }, + checkTrust(chain, original, context) { + try { return original(); } catch (rejection) { + if (!rejectionTypes.isCertificateRejection || !rejectionTypes.isCertificateRejection(rejection) || !isScope(chain, context)) throw rejection; + } + } + }; + } + + function createEngine(config, adapter) { + const profiles = normalizeProfiles(config.profiles); + const undo = []; + const transaction = { + hook(method, implementation) { + const previous = method.implementation; + // Register undo first: a failing setter can have partially installed. + undo.push(() => { method.implementation = previous; }); + method.implementation = implementation; + } + }; + function rollback() { + const errors = []; + while (undo.length) { + try { undo.pop()(); } catch (error) { errors.push(String(error)); } + } + if (errors.length) throw new Error('Hook rollback failed: ' + errors.join('; ').slice(0, 1500)); + } + let installed = false; + return { + rollback, + install() { + if (installed) throw new Error('Engine already installed'); + const capabilities = {}; + try { + for (const profile of profiles) { + const name = 'install' + profile[0].toUpperCase() + profile.slice(1); + const result = adapter[name](transaction); + if (!Array.isArray(result) || !result.includes(REQUIRED[profile])) { + throw new Error('Missing installed capability for profile ' + profile); + } + capabilities[profile] = result; + } + installed = true; + return { kind: 'scope-hooks-ready', engine: 'scope-java-v1', profiles, capabilities }; + } catch (error) { + try { rollback(); } catch (cleanupError) { + throw new Error(String(error) + '; ' + String(cleanupError)); + } + throw error; + } + } + }; + } + + function createJavaAdapter(Java, config) { + let trust; + function use(name) { + try { return Java.use(name); } catch (error) { + throw new Error('Required Java class unavailable: ' + name + ': ' + String(error).slice(0, 300)); + } + } + function overload(klass, method, signature) { + if (!klass[method]) throw new Error('Required method unavailable: ' + method); + try { return klass[method].overload(...signature); } catch (error) { + throw new Error('Required overload unavailable: ' + method + '(' + signature.join(', ') + '): ' + String(error).slice(0, 300)); + } + } + function trustState() { + if (trust) return trust; + const X509 = use('javax.net.ssl.X509TrustManager'); + const Certificate = use('java.security.cert.X509Certificate'); + const Factory = use('javax.net.ssl.TrustManagerFactory'); + const Store = use('java.security.KeyStore'); + const bytes = use('java.lang.String').$new(config.caPem).getBytes('UTF-8'); + const stream = use('java.io.ByteArrayInputStream').$new(bytes); + const ca = Java.cast(use('java.security.cert.CertificateFactory').getInstance('X.509').generateCertificate(stream), Certificate); + ca.checkValidity(); + if (ca.getBasicConstraints() < 0) throw new Error('Scope certificate must be a CA'); + const store = Store.getInstance(Store.getDefaultType()); + store.load(null, null); // Empty, never the Android/system trust store. + store.setCertificateEntry('scope', ca); + const factory = Factory.getInstance(Factory.getDefaultAlgorithm()); + factory.init(store); + const managers = factory.getTrustManagers(); + let manager; + for (let i = 0; i < managers.length; i++) { + if (X509.class.isInstance(managers[i])) { manager = Java.cast(managers[i], X509); break; } + } + if (!manager) throw new Error('No X509 trust manager for isolated Scope CA'); + let Extended; + try { Extended = Java.use('javax.net.ssl.X509ExtendedTrustManager'); } catch (_) { /* Android before API 24. */ } + trust = { manager, ca, Factory, X509, Certificate, Extended }; + return trust; + } + function certificates(chain) { + if (!chain) throw new Error('Peer chain is empty'); + const values = typeof chain.toArray === 'function' ? chain.toArray() : chain; + if (!values.length) throw new Error('Peer chain is empty'); + const { Certificate } = trustState(); + const result = []; + for (let i = 0; i < values.length; i++) { + if (!Certificate.class.isInstance(values[i])) throw new Error('Peer chain contains a non-X509 certificate'); + result.push(Java.cast(values[i], Certificate)); + } + return result; + } + function validateScopeChain(chain, context) { + const state = trustState(); + const values = certificates(chain); + if (values[0].getBasicConstraints() >= 0) throw new Error('Peer chain must start with a server leaf certificate'); + // Check every supplied certificate, including trust anchors. Trust manager + // performs path construction, signature, usage and algorithm validation. + state.ca.checkValidity(); + values.forEach(cert => cert.checkValidity()); + const authType = context ? context.authType : String(values[0].getPublicKey().getAlgorithm()); + const peer = Java.array('java.security.cert.X509Certificate', values); + if (context && context.type) { + if (!state.Extended || !state.Extended.class.isInstance(state.manager)) throw new Error('Isolated Scope manager cannot preserve extended TLS context'); + const manager = Java.cast(state.manager, state.Extended); + manager.checkServerTrusted.overload('[Ljava.security.cert.X509Certificate;', 'java.lang.String', context.type).call(manager, peer, authType, context.transport); + } else { + state.manager.checkServerTrusted(peer, authType); + } + } + function isJavaException(error, className) { + if (!error || !error.$h) return false; + try { + // The retained bridge attaches the original JNI Throwable handle to + // Java invocation errors. Inspect that object, never message text. + return use(className).class.isInstance(Java.cast(error.$h, use('java.lang.Throwable'))); + } catch (_) { return false; } + } + const policy = createPolicy(validateScopeChain, { + isCertificateRejection: error => isJavaException(error, 'java.security.cert.CertificateException'), + isPinRejection: error => isJavaException(error, 'javax.net.ssl.SSLPeerUnverifiedException') + }); + + function installCertificate(transaction) { + const state = trustState(); + const init = overload(use('javax.net.ssl.SSLContext'), 'init', ['[Ljavax.net.ssl.KeyManager;', '[Ljavax.net.ssl.TrustManager;', 'java.security.SecureRandom']); + function wrapperClass(extended) { + const methods = { + getAcceptedIssuers: { + returnType: '[Ljava.security.cert.X509Certificate;', argumentTypes: [], + implementation() { + const original = this.delegate.value.getAcceptedIssuers(); + return Java.array('java.security.cert.X509Certificate', Array.from(original).concat([state.ca])); + } + } + }; + for (const name of ['checkClientTrusted', 'checkServerTrusted']) { + const implementations = []; + for (const type of extended ? [null, 'java.net.Socket', 'javax.net.ssl.SSLEngine'] : [null]) { + implementations.push({ + returnType: 'void', + argumentTypes: ['[Ljava.security.cert.X509Certificate;', 'java.lang.String'].concat(type ? [type] : []), + implementation(chain, authType, transport) { + const delegate = Java.cast(this.delegate.value, extended ? state.Extended : state.X509); + const original = () => type ? delegate[name].overload('[Ljava.security.cert.X509Certificate;', 'java.lang.String', type).call(delegate, chain, authType, transport) : delegate[name](chain, authType); + if (name === 'checkClientTrusted') return original(); + return policy.checkTrust(chain, original, { authType, type, transport }); + } + }); + } + methods[name] = implementations; + } + return Java.registerClass({ + name: extended ? 'dev.scope.runtime.ScopeExtendedTrustManager' : 'dev.scope.runtime.ScopeTrustManager', + ...(extended ? { superClass: state.Extended } : { implements: [state.X509] }), + fields: { delegate: 'javax.net.ssl.X509TrustManager' }, + methods + }); + } + const BasicWrapper = wrapperClass(false); + const ExtendedWrapper = state.Extended && state.Extended.class.isInstance(state.manager) ? wrapperClass(true) : null; + transaction.hook(init, function (keys, managers, random) { + if (managers === null) { + const defaults = state.Factory.getInstance(state.Factory.getDefaultAlgorithm()); + // Select the KeyStore overload explicitly; null otherwise ambiguous. + defaults.init.overload('java.security.KeyStore').call(defaults, null); + managers = defaults.getTrustManagers(); + } + let count = 0; + const wrapped = Array.from(managers, manager => { + if (!state.X509.class.isInstance(manager)) return manager; + const extended = state.Extended && state.Extended.class.isInstance(manager); + if (extended && !ExtendedWrapper) throw new Error('Cannot preserve X509ExtendedTrustManager context with this Scope manager'); + const wrapper = (extended ? ExtendedWrapper : BasicWrapper).$new(); + wrapper.delegate.value = Java.cast(manager, state.X509); + count++; + return wrapper; + }); + if (!count) throw new Error('SSLContext has no X509 trust manager to extend'); + return init.call(this, keys, Java.array('javax.net.ssl.TrustManager', wrapped), random); + }); + return ['sslcontext-scope-ca']; + } + + function installProxy(transaction) { + const URL = use('java.net.URL'); + const Proxy = use('java.net.Proxy'); + const address = use('java.net.InetSocketAddress').$new(config.proxyHost, config.proxyPort); + const proxy = Proxy.$new(use('java.net.Proxy$Type').HTTP.value, address); + const plain = overload(URL, 'openConnection', []); + const explicit = overload(URL, 'openConnection', ['java.net.Proxy']); + function http(url) { return ['http', 'https'].includes(String(url.getProtocol())); } + transaction.hook(plain, function () { return http(this) ? explicit.call(this, proxy) : plain.call(this); }); + transaction.hook(explicit, function (selected) { return explicit.call(this, http(this) ? proxy : selected); }); + const capabilities = ['java-url']; + let Builder; + try { Builder = Java.use('okhttp3.OkHttpClient$Builder'); } catch (_) { return capabilities; } + const build = overload(Builder, 'build', []); + const setProxy = overload(Builder, 'proxy', ['java.net.Proxy']); + transaction.hook(build, function () { setProxy.call(this, proxy); return build.call(this); }); + capabilities.push('okhttp3-proxy'); + return capabilities; + } + + function installUnpinning(transaction) { + trustState(); + const Pinner = use('okhttp3.CertificatePinner'); + // Normal Kotlin connections may lose the raw chain during original-manager + // cleaning before check$okhttp sees it. Public overloads on the same class + // cannot establish coverage. Refuse this configuration transactionally. + if (Pinner['check$okhttp']) throw new Error('Unsupported lazy OkHttp configuration: check$okhttp; use the ordinary Java OkHttp 3.14.9 acceptance target'); + let count = 0; + if (Pinner.check) { + for (const method of Pinner.check.overloads) { + const args = method.argumentTypes.map(type => type.className); + const ordinary = args.length === 2 && args[0] === 'java.lang.String' && ['java.util.List', '[Ljava.security.cert.Certificate;'].includes(args[1]); + if (!ordinary) throw new Error('Unsupported OkHttp pinner overload: check(' + args.join(', ') + ')'); + transaction.hook(method, function (host, peer) { + return policy.checkPins(peer, () => method.call(this, host, peer)); + }); + count++; + } + } + if (!count) throw new Error('No supported okhttp3.CertificatePinner overloads'); + return ['okhttp3-certificate-pinner']; + } + return { validateScopeChain, installProxy, installCertificate, installUnpinning }; + } + + function start(config) { + try { + normalizeProfiles(config.profiles); + if (config.proxyHost !== '127.0.0.1' || !Number.isInteger(config.proxyPort) || config.proxyPort < 1 || config.proxyPort > 65535) throw new Error('Invalid loopback proxy configuration'); + if (!root.Java || !root.Java.available) throw new Error('Frida Java runtime is unavailable'); + root.Java.perform(() => { + try { root.send(createEngine(config, createJavaAdapter(root.Java, config)).install()); } + catch (error) { root.send({ kind: 'scope-hooks-error', engine: 'scope-java-v1', message: String(error).slice(0, 2000) }); } + }); + } catch (error) { root.send({ kind: 'scope-hooks-error', engine: 'scope-java-v1', message: String(error).slice(0, 2000) }); } + } + const api = { normalizeProfiles, createPolicy, createEngine, createJavaAdapter, start }; + if (typeof module !== 'undefined' && module.exports) module.exports = api; + else root.ScopeJavaEngine = api; +})(globalThis); diff --git a/resources/frida/tests/scope-engine.test.cjs b/resources/frida/tests/scope-engine.test.cjs new file mode 100644 index 0000000..a795b87 --- /dev/null +++ b/resources/frida/tests/scope-engine.test.cjs @@ -0,0 +1,315 @@ +'use strict'; +// Java/Frida doubles exercise Scope policy and API wiring, not Android execution. +const test = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const vm = require('node:vm'); +let engine; +try { + const context = { module: { exports: {} } }; + vm.runInNewContext(fs.readFileSync(require('node:path').join(__dirname, '../scope-engine.js'), 'utf8'), context); + engine = context.module.exports; +} catch (error) { if (error.code !== 'ENOENT') throw error; } +const api = () => { assert.ok(engine, 'Scope-owned Java engine is implemented'); return engine; }; +const plain = value => JSON.parse(JSON.stringify(value)); +function javaError(javaType, message) { const error = new Error(message); error.$h = { javaType }; return error; } + +test('profiles default, deduplicate and reject unsupported or missing dependencies', () => { + const { normalizeProfiles } = api(); + assert.deepEqual(plain(normalizeProfiles([])), ['proxy', 'certificate', 'unpinning']); + assert.deepEqual(plain(normalizeProfiles(['certificate', 'proxy', 'certificate'])), ['proxy', 'certificate']); + for (const value of [['root'], ['system-ca'], ['other'], ['unpinning'], 'proxy', [null]]) { + assert.throws(() => normalizeProfiles(value), /profile|certificate/i); + } +}); + +test('pins skip only a chain accepted by the isolated CA validator; preserve original rejection identity', () => { + const { createPolicy } = api(); + const valid = {}, unrelated = {}, expired = {}, malformed = {}; + const rejection = new Error('original pin mismatch'); + const checked = []; + const policy = createPolicy(chain => { checked.push(chain); if (chain !== valid) throw new Error('PKIX rejection'); }, { isPinRejection: error => error === rejection }); + let originalCalls = 0; + const original = () => { originalCalls++; throw rejection; }; + assert.equal(policy.checkPins(valid, original), undefined); + for (const chain of [unrelated, expired, malformed, null]) { + assert.throws(() => policy.checkPins(chain, original), error => error === rejection); + } + assert.equal(originalCalls, 5); + assert.equal(checked.length, 5); +}); + +test('trust delegates original decisions and preserves original errors for invalid Scope chains', () => { + const { createPolicy } = api(); + const valid = {}, rejection = new Error('original trust rejection'); + const policy = createPolicy(chain => { if (chain !== valid) throw new Error('invalid'); }, { isCertificateRejection: error => error === rejection }); + assert.equal(policy.checkTrust({}, () => 'original acceptance'), 'original acceptance'); + assert.equal(policy.checkTrust(valid, () => { throw rejection; }), undefined); + assert.throws(() => policy.checkTrust({}, () => { throw rejection; }), error => error === rejection); +}); + +test('missing requested capabilities fail transactionally and restore earlier hooks', () => { + const { createEngine } = api(); + const method = { implementation: 'original' }; + const adapter = { + installProxy(transaction) { transaction.hook(method, () => 'proxy'); return ['java-url']; }, + installCertificate() { throw new Error('SSLContext.init overload missing'); }, + }; + assert.throws(() => createEngine({ profiles: ['proxy', 'certificate'] }, adapter).install(), /SSLContext.init/); + assert.equal(method.implementation, 'original'); + assert.throws(() => createEngine({ profiles: ['proxy'] }, { installProxy() { return []; } }).install(), /capabilit/i); +}); + +test('readiness reports exactly installed requested capabilities and rollback is explicit', () => { + const { createEngine } = api(); + const method = { implementation: null }; + const instance = createEngine({ profiles: ['proxy'] }, { + installProxy(transaction) { transaction.hook(method, () => 'proxy'); return ['java-url']; }, + }); + assert.deepEqual(plain(instance.install()), { kind: 'scope-hooks-ready', engine: 'scope-java-v1', profiles: ['proxy'], capabilities: { proxy: ['java-url'] } }); + instance.rollback(); + assert.equal(method.implementation, null); +}); + +function javaFixture() { + const calls = []; + const ca = { checkValidity() { calls.push('ca-valid'); }, getBasicConstraints() { return 1; } }; + const scopeTrust = { checkServerTrusted(chain, auth) { calls.push(['validate', chain, auth]); if (chain.some(c => c.unrelated)) throw new Error('bad signature'); } }; + const x509 = { class: { isInstance(value) { return value === scopeTrust; } } }; + const store = { load(stream, pass) { calls.push(['empty-store', stream, pass]); }, setCertificateEntry(name, cert) { calls.push(['anchor', name, cert]); } }; + const classes = { + 'java.lang.Throwable': {}, + 'java.security.cert.CertificateException': { class: { isInstance: value => value.javaType === 'certificate' } }, + 'javax.net.ssl.SSLPeerUnverifiedException': { class: { isInstance: value => value.javaType === 'pin' } }, + 'java.lang.String': { $new(value) { return { getBytes() { return value; } }; } }, + 'java.io.ByteArrayInputStream': { $new() { return {}; } }, + 'java.security.cert.CertificateFactory': { getInstance() { return { generateCertificate() { return ca; } }; } }, + 'java.security.KeyStore': { getDefaultType() { return 'BKS'; }, getInstance() { return store; } }, + 'javax.net.ssl.TrustManagerFactory': { getDefaultAlgorithm() { return 'PKIX'; }, getInstance() { return { init(value) { calls.push(['tm-init', value]); }, getTrustManagers() { return [scopeTrust]; } }; } }, + 'javax.net.ssl.X509TrustManager': x509, + 'java.security.cert.X509Certificate': { class: { isInstance(value) { return Boolean(value && value.checkValidity); } } }, + }; + const Java = { use(name) { if (!classes[name]) throw new Error('Missing class ' + name); return classes[name]; }, cast(value) { return value; }, array(type, value) { return value; } }; + return { Java, calls, ca, scopeTrust, store, classes }; +} +function leaf(options = {}) { return { checkValidity() { if (options.expired) throw new Error('expired'); }, getBasicConstraints() { return -1; }, getPublicKey() { return { getAlgorithm() { return 'RSA'; } }; }, ...options }; } + +test('Java validator uses empty keystore with only Scope CA and validates peer signatures and validity', () => { + const { createJavaAdapter } = api(); + const fixture = javaFixture(); + const adapter = createJavaAdapter(fixture.Java, { caPem: 'public certificate', proxyHost: '127.0.0.1', proxyPort: 17890 }); + const good = leaf(); + adapter.validateScopeChain([good]); + assert.deepEqual(fixture.calls.find(call => Array.isArray(call) && call[0] === 'empty-store'), ['empty-store', null, null]); + assert.deepEqual(fixture.calls.filter(call => Array.isArray(call) && call[0] === 'anchor'), [['anchor', 'scope', fixture.ca]]); + assert.deepEqual(plain(fixture.calls.find(call => Array.isArray(call) && call[0] === 'validate')), ['validate', [{}], 'RSA']); + assert.throws(() => adapter.validateScopeChain([leaf({ unrelated: true }), fixture.ca]), /bad signature/); + assert.throws(() => adapter.validateScopeChain([leaf({ expired: true })]), /expired/); + assert.throws(() => adapter.validateScopeChain([fixture.ca]), /leaf/); + assert.throws(() => adapter.validateScopeChain([]), /empty|chain/i); + assert.throws(() => adapter.validateScopeChain([{}]), /certificate/i); +}); + +test('real adapter fails closed on missing Java classes and missing OkHttp pinner', () => { + const { createJavaAdapter, createEngine } = api(); + const fixture = javaFixture(); + const adapter = createJavaAdapter(fixture.Java, { caPem: 'public certificate', proxyHost: '127.0.0.1', proxyPort: 17890 }); + assert.throws(() => createEngine({ profiles: ['proxy'] }, adapter).install(), /java.net|Missing class/); + assert.throws(() => adapter.installUnpinning({ hook() {} }), /okhttp3.CertificatePinner/); +}); + +function method(signature, original = () => undefined) { + return { argumentTypes: signature.map(className => ({ className })), implementation: null, call(receiver, ...args) { return original.apply(receiver, args); } }; +} +function dispatcher(...methods) { + return { overloads: methods, overload(...signature) { + const found = methods.find(m => JSON.stringify(m.argumentTypes.map(t => t.className)) === JSON.stringify(signature)); + if (!found) throw new Error('overload missing'); + return found; + } }; +} +function tlsFixture(extended = false) { + const f = javaFixture(); + const calls = []; + const rejection = javaError('certificate', 'original rejected'); + const original = { extended, getAcceptedIssuers() { return []; }, checkClientTrusted(...args) { calls.push(['client', ...args]); }, checkServerTrusted(...args) { calls.push(['server', ...args]); throw rejection; } }; + for (const name of ['checkClientTrusted', 'checkServerTrusted']) { + const dynamic = original[name]; + Object.assign(dynamic, dispatcher(...[null, 'java.net.Socket', 'javax.net.ssl.SSLEngine'].map(type => method(['[Ljava.security.cert.X509Certificate;', 'java.lang.String'].concat(type ? [type] : []), dynamic)))); + } + f.classes['javax.net.ssl.X509TrustManager'].class.isInstance = value => value === f.scopeTrust || value === original; + f.classes['javax.net.ssl.X509ExtendedTrustManager'] = { class: { isInstance: value => Boolean(value.extended) } }; + f.scopeTrust.extended = extended; + let wrapped; + const init = method(['[Ljavax.net.ssl.KeyManager;', '[Ljavax.net.ssl.TrustManager;', 'java.security.SecureRandom'], function (_keys, managers) { wrapped = managers[0]; }); + f.classes['javax.net.ssl.SSLContext'] = { init: dispatcher(init) }; + let definition; + f.Java.registerClass = spec => { + definition = spec; + return { $new() { + const object = { delegate: { value: null } }; + for (const [name, entry] of Object.entries(spec.methods)) { + object[name] = Array.isArray(entry) ? (...args) => entry.find(m => m.argumentTypes.length === args.length && (args.length !== 3 || m.argumentTypes[2] === args[2].type)).implementation.apply(object, args) : (typeof entry === 'function' ? entry : entry.implementation).bind(object); + } + return object; + } }; + }; + return { ...f, init, original, calls, rejection, get wrapped() { return wrapped; }, get definition() { return definition; } }; +} + +test('SSLContext adds Scope trust while preserving client checks, original errors and issuer inputs', () => { + const { createJavaAdapter, createEngine } = api(); + const f = tlsFixture(); + createEngine({ profiles: ['certificate'] }, createJavaAdapter(f.Java, { caPem: 'public certificate' })).install(); + f.init.implementation(null, [f.original], null); + const valid = [leaf()]; + assert.equal(f.wrapped.checkServerTrusted(valid, 'RSA'), undefined); + assert.throws(() => f.wrapped.checkServerTrusted([leaf({ unrelated: true })], 'RSA'), e => e === f.rejection); + f.wrapped.checkClientTrusted(valid, 'RSA'); + assert.equal(f.calls.at(-1)[0], 'client'); + assert.equal(f.calls.at(-1)[1], valid); + assert.equal(f.wrapped.getAcceptedIssuers()[0], f.ca); +}); + +test('extended trust preserves socket and engine context and scope context rejection', () => { + const { createJavaAdapter, createEngine } = api(); + const f = tlsFixture(true); + const contexts = []; + const basicValidation = f.scopeTrust.checkServerTrusted; + f.scopeTrust.checkServerTrusted = dispatcher( + method(['[Ljava.security.cert.X509Certificate;', 'java.lang.String'], basicValidation), + method(['[Ljava.security.cert.X509Certificate;', 'java.lang.String', 'java.net.Socket'], (_chain, _auth, context) => { contexts.push(context); throw new Error('hostname mismatch'); }), + method(['[Ljava.security.cert.X509Certificate;', 'java.lang.String', 'javax.net.ssl.SSLEngine'], (_chain, _auth, context) => { contexts.push(context); }) + ); + createEngine({ profiles: ['certificate'] }, createJavaAdapter(f.Java, { caPem: 'public certificate' })).install(); + f.init.implementation(null, [f.original], null); + assert.equal(f.definition.superClass, f.classes['javax.net.ssl.X509ExtendedTrustManager']); + const socket = { type: 'java.net.Socket' }, tlsEngine = { type: 'javax.net.ssl.SSLEngine' }; + assert.throws(() => f.wrapped.checkServerTrusted([leaf()], 'RSA', socket), e => e === f.rejection); + assert.equal(f.wrapped.checkServerTrusted([leaf()], 'RSA', tlsEngine), undefined); + assert.equal(contexts[0], socket); + assert.equal(contexts[1], tlsEngine); + assert.equal(f.calls[0][3], socket); + assert.equal(f.calls[1][3], tlsEngine); +}); + +test('URL proxy hooks redirect HTTP only and rollback restores original implementations', () => { + const { createJavaAdapter, createEngine } = api(); + const f = javaFixture(); + const scopeProxy = {}; + const plainMethod = method([], () => 'original non-http'); + const explicit = method(['java.net.Proxy'], selected => selected); + f.classes['java.net.URL'] = { openConnection: dispatcher(plainMethod, explicit) }; + f.classes['java.net.Proxy'] = { $new(_type, address) { assert.equal(address.host, '127.0.0.1'); assert.equal(address.port, 17890); return scopeProxy; } }; + f.classes['java.net.Proxy$Type'] = { HTTP: { value: 'HTTP' } }; + f.classes['java.net.InetSocketAddress'] = { $new(host, port) { return { host, port }; } }; + const instance = createEngine({ profiles: ['proxy'] }, createJavaAdapter(f.Java, { proxyHost: '127.0.0.1', proxyPort: 17890 })); + instance.install(); + assert.equal(plainMethod.implementation.call({ getProtocol: () => 'https' }), scopeProxy); + assert.equal(plainMethod.implementation.call({ getProtocol: () => 'file' }), 'original non-http'); + const custom = {}; + assert.equal(explicit.implementation.call({ getProtocol: () => 'file' }, custom), custom); + instance.rollback(); + assert.equal(plainMethod.implementation, null); + assert.equal(explicit.implementation, null); +}); + +test('OkHttp actual hook preserves unrelated pin exception and rejects unsupported overload transactionally', () => { + const { createJavaAdapter, createEngine } = api(); + const f = javaFixture(); + const rejection = javaError('pin', 'pin mismatch'); + const check = method(['java.lang.String', 'java.util.List'], () => { throw rejection; }); + f.classes['okhttp3.CertificatePinner'] = { check: dispatcher(check) }; + const adapter = createJavaAdapter(f.Java, { caPem: 'public certificate' }); + // Certificate profile orchestration is tested separately; install the actual pin adapter here. + const instance = createEngine({ profiles: ['certificate', 'unpinning'] }, { installCertificate: () => ['sslcontext-scope-ca'], installUnpinning: adapter.installUnpinning }); + instance.install(); + assert.equal(check.implementation('example.test', [leaf()]), undefined); + assert.throws(() => check.implementation('example.test', [leaf({ unrelated: true }), f.ca]), e => e === rejection); + instance.rollback(); + f.classes['okhttp3.CertificatePinner'].check.overloads.push(method(['java.lang.String', 'unknown.Chain'])); + assert.throws(() => instance.install(), /already installed/); + const second = createEngine({ profiles: ['certificate', 'unpinning'] }, { installCertificate: () => ['sslcontext-scope-ca'], installUnpinning: adapter.installUnpinning }); + assert.throws(() => second.install(), /Unsupported OkHttp/); + assert.equal(check.implementation, null); +}); + +test('recognized lazy OkHttp refuses readiness and rolls back certificate and ordinary hooks', () => { + const { createJavaAdapter, createEngine } = api(); + const f = tlsFixture(); + const ordinary = method(['java.lang.String', 'java.util.List']); + const lazy = method(['java.lang.String', 'kotlin.jvm.functions.Function0']); + f.classes['kotlin.jvm.functions.Function0'] = {}; + f.classes['okhttp3.CertificatePinner'] = { check: dispatcher(ordinary), 'check$okhttp': dispatcher(lazy) }; + const adapter = createJavaAdapter(f.Java, { caPem: 'public certificate' }); + const engine = createEngine({ profiles: ['certificate', 'unpinning'] }, adapter); + // A lazy Handshake supplier may already have discarded the raw Scope chain. + // Public overloads on that class do not prove normal connection coverage. + assert.throws(() => engine.install(), /Unsupported lazy OkHttp/); + assert.equal(f.init.implementation, null); + assert.equal(ordinary.implementation, null); + assert.equal(lazy.implementation, null); +}); + +test('extended accepted issuers registration explicitly implements the X509 array descriptor', () => { + const { createJavaAdapter, createEngine } = api(); + const f = tlsFixture(true); + createEngine({ profiles: ['certificate'] }, createJavaAdapter(f.Java, { caPem: 'public certificate' })).install(); + const descriptor = f.definition.methods.getAcceptedIssuers; + assert.equal(descriptor.returnType, '[Ljava.security.cert.X509Certificate;'); + assert.deepEqual(plain(descriptor.argumentTypes), []); +}); + +test('valid Scope chain cannot suppress unrelated trust or ordinary pinner errors', () => { + const { createJavaAdapter, createEngine } = api(); + const unrelated = new Error('internal programming failure'); + const f = tlsFixture(); + f.original.checkServerTrusted = () => { throw unrelated; }; + createEngine({ profiles: ['certificate'] }, createJavaAdapter(f.Java, { caPem: 'public certificate' })).install(); + f.init.implementation(null, [f.original], null); + assert.throws(() => f.wrapped.checkServerTrusted([leaf()], 'RSA'), e => e === unrelated); + for (const chainType of ['java.util.List', '[Ljava.security.cert.Certificate;']) { + const p = javaFixture(); + const check = method(['java.lang.String', chainType], () => { throw unrelated; }); + p.classes['okhttp3.CertificatePinner'] = { check: dispatcher(check) }; + const adapter = createJavaAdapter(p.Java, { caPem: 'public certificate' }); + createEngine({ profiles: ['certificate', 'unpinning'] }, { installCertificate: () => ['sslcontext-scope-ca'], installUnpinning: adapter.installUnpinning }).install(); + assert.throws(() => check.implementation('test', [leaf()]), e => e === unrelated); + } +}); + +test('nullable extended delegation selects exact original Socket and SSLEngine overloads', () => { + const { createJavaAdapter, createEngine } = api(); + const f = tlsFixture(true); + const selected = []; + for (const name of ['checkClientTrusted', 'checkServerTrusted']) { + const legacy = f.original[name]; + const dynamic = (...args) => legacy(...args); // Baseline dynamic dispatcher cannot distinguish null. + Object.assign(dynamic, dispatcher(...['java.net.Socket', 'javax.net.ssl.SSLEngine'].map(type => method(['[Ljava.security.cert.X509Certificate;', 'java.lang.String', type], (_chain, _auth, transport) => { assert.equal(transport, null); selected.push([name, type]); })))); + f.original[name] = dynamic; + } + createEngine({ profiles: ['certificate'] }, createJavaAdapter(f.Java, { caPem: 'public certificate' })).install(); + f.init.implementation(null, [f.original], null); + for (const name of ['checkClientTrusted', 'checkServerTrusted']) { + for (const type of ['java.net.Socket', 'javax.net.ssl.SSLEngine']) { + f.definition.methods[name].find(entry => entry.argumentTypes[2] === type).implementation.call(f.wrapped, [leaf({ unrelated: true })], 'RSA', null); + } + } + assert.deepEqual(selected, [['checkClientTrusted', 'java.net.Socket'], ['checkClientTrusted', 'javax.net.ssl.SSLEngine'], ['checkServerTrusted', 'java.net.Socket'], ['checkServerTrusted', 'javax.net.ssl.SSLEngine']]); +}); + + +test('the same peer and CA are revalidated after an earlier accepted fallback', () => { + const f = tlsFixture(); + api().createEngine({ profiles: ['certificate'] }, api().createJavaAdapter(f.Java, { caPem: 'public certificate' })).install(); + f.init.implementation(null, [f.original], null); + let expired = false; + const chain = [leaf({ checkValidity() { if (expired) throw Error('now expired'); } })]; + f.wrapped.checkServerTrusted(chain, 'RSA'); + expired = true; + assert.throws(() => f.wrapped.checkServerTrusted(chain, 'RSA'), error => error === f.rejection); + expired = false; + f.ca.checkValidity = () => { throw Error('CA now expired'); }; + assert.throws(() => f.wrapped.checkServerTrusted(chain, 'RSA'), error => error === f.rejection); + assert.equal(f.calls.filter(call => call[0] === 'server').length, 3); +}); diff --git a/src/App.breakpoints.test.tsx b/src/App.breakpoints.test.tsx index 49b5219..44cd558 100644 --- a/src/App.breakpoints.test.tsx +++ b/src/App.breakpoints.test.tsx @@ -139,6 +139,26 @@ describe('Proxy listener settings', () => { fireEvent.click(screen.getByRole('button', { name: 'Start interception' })); }); } + it('requires an explicit Frida selection and certificate for unpinning', async () => { + androidAvailable(); + await mount(); + await act(async () => { + fireEvent.click(screen.getByRole('button', { name: /Connect a device Android/ })); + }); + await act(async () => { + fireEvent.click(screen.getByRole('button', { name: /Demo app com.example.demo/ })); + }); + fireEvent.click(screen.getByRole('button', { name: /Frida Optional instrumentation/ })); + fireEvent.click(screen.getByRole('checkbox', { name: 'Trust the Scope CA' })); + expect(screen.getByRole('checkbox', { name: 'TLS certificate unpinning' })).not.toBeChecked(); + expect(screen.getByRole('checkbox', { name: 'TLS certificate unpinning' })).toBeDisabled(); + fireEvent.click(screen.getByRole('checkbox', { name: 'Proxy configuration' })); + expect(screen.getByRole('button', { name: 'Start interception' })).toBeDisabled(); + expect(mocks.start_device).not.toHaveBeenCalled(); + expect( + screen.queryByRole('checkbox', { name: 'Root detection hooks' }), + ).not.toBeInTheDocument(); + }); it('uses the configured port when demo traffic starts an idle listener', async () => { idleListener(); await mount(); diff --git a/src/App.tsx b/src/App.tsx index f8295f0..f6819a2 100644 --- a/src/App.tsx +++ b/src/App.tsx @@ -1651,6 +1651,7 @@ function DevicePanel({ ); const start = () => void perform('device', async () => { + if (mode === 'frida' && profiles.length === 0) return; let p = proxy; if (!p.running) { const port = validatedListenerPort(portDraft); @@ -1783,14 +1784,17 @@ function DevicePanel({

Device-wide capture. Android's proxy setting applies to the whole device. Other apps may appear in this session; choosing an app launches it and - does not isolate its traffic. + does not isolate its traffic. HTTPS requires manually configured app trust; no + system CA is installed.

) : (

Select the bundled hook profiles for an app you own or are authorized to test. - Frida support depends on Android and application versions. + Experimental Java URL/OkHttp hooks only. Stop ends the instrumented app process. + Unpinning requires the Scope CA profile; native TLS and custom pinners are + unsupported.

@@ -1803,15 +1807,19 @@ function DevicePanel({ ['certificate', 'Trust the Scope CA'], ['proxy', 'Proxy configuration'], ['unpinning', 'TLS certificate unpinning'], - ['root', 'Root detection hooks'], ].map(([id, label]) => (