Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .agents/skills/headless-computer-use/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ controls the desktop, native applications, OS chrome, microphone, or system audi
selection, Docker access, visible-browser mode, or MCP configuration matters.

Run `headless capabilities` once before relying on optional features. Network
emulation and mocking are Linux Chromium capabilities; macOS WebKit reports them
as unsupported.
emulation, request mocking, and file upload are Linux Chromium capabilities;
macOS WebKit reports them as unsupported.

## Follow the mandatory interaction loop

Expand Down
9 changes: 7 additions & 2 deletions .agents/skills/headless-computer-use/references/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ headless --session NAME click REF
headless --session NAME click --role ROLE --name NAME
headless --session NAME fill REF "TEXT"
headless --session NAME fill REF -- "--json stays literal"
headless --session NAME upload REF --artifact FILE
headless --session NAME upload --role textbox --name NAME --artifact FILE
headless --session NAME press KEY
headless --session NAME scroll up|down|top|bottom --amount PIXELS
headless --session NAME back
Expand All @@ -44,7 +46,9 @@ large pages, request `outline`, select a returned `@rN` region, then use
bound the result; check `omitted` before assuming it describes the whole page.
Use `click --role ... --name ...` for unique accessible controls. Use an `@eN`
ref from the latest inspection when role/name is ambiguous. Inspect again after
navigation or a large rerender.
navigation or a large rerender. File inputs advertise `upload` for an existing
private artifact-store basename. Upload never accepts or imports a filesystem
path. Ask before uploading, as in [safety.md](safety.md).

Pass fill text as one quoted shell argument so whitespace is preserved. Put
`--` before a value that contains a literal global flag such as `--json` or
Expand Down Expand Up @@ -78,7 +82,8 @@ headless artifacts list
```

Single artifact output names are basenames ending in `.png`, `.jpg`, `.jpeg`,
`.pdf`, `.mp4`, `.mov`, `.webm`, `.gif`, or `.json`. Screenshot series output
`.gif`, `.webp`, `.txt`, `.csv`, `.pdf`, `.mp4`, `.mov`, `.webm`, or `.json`.
Screenshot series output
uses a safe prefix and creates numbered PNG/JPG artifacts. Headless refuses
paths and overwrites. Built-in recording captures browser pixels only.

Expand Down
7 changes: 7 additions & 0 deletions .agents/skills/headless-computer-use/references/safety.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,13 @@ publishes content, makes a purchase, changes permissions, accepts legal terms,
uploads a file, or otherwise creates a meaningful external effect not already
explicitly authorized by the user.

To attach a file after that confirmation, use
`headless upload --role textbox --name NAME --artifact FILE` (or `upload @eN
--artifact FILE`) with an existing private artifact-store basename. No
agent-facing command imports a local filesystem path. Downloads remain denied.
File bytes never appear on the protocol socket. macOS WebKit returns
`UNSUPPORTED_CAPABILITY` until a native attach path exists.

Routine mutations inside an explicitly requested disposable/local E2E test are
in scope. Do not transfer that authorization to a production site.

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,8 @@ structural region references such as `@r4`, then inspect only that region with
`contextStats` make pruning explicit. `--context full --text` remains the
explicit broad-page escape hatch.

Each control's `actions` list contains only protocol verbs that can run (`click`
or `fill`); unsupported controls never advertise nonexistent commands. Full
Each control's `actions` list contains only protocol verbs that can run (`click`,
`fill`, or `upload`); unsupported controls never advertise nonexistent commands. Full
inspection retains roles, names, rendered media metadata, safety markers,
bounds, and element references such as `@e1`.
`capture-info` returns the browser surface, page state, action trace, and
Expand Down
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ These are host-enforced contracts. Anything that defeats one is in scope:
| **Navigation** | HTTP/HTTPS only. Optional `headless start --allow` host allowlist. `file:`, `javascript:`, `data:`, credential-bearing URLs, and external application schemes must be refused at every layer. |
| **Downloads** | Page-initiated downloads are denied. Executables, installers, scripts, libraries, and disk images are blocked by extension. |
| **Control plane** | A `0600` Unix socket inside a `0700` per-user directory, with a peer-UID check. There is no TCP listener and no Chromium debug port. Any remote reachability is a vulnerability. |
| **Artifacts** | Bare validated names, `O_EXCL` creation at `0600` inside a `0700` root, never overwritten. Path traversal or reading outside the store is a vulnerability. |
| **Artifacts** | Bare validated names, `O_EXCL` creation at `0600` inside a `0700` root, never overwritten. Path traversal or reading outside the store is a vulnerability. Upload accepts only an existing store basename; agent-facing surfaces cannot ingest local files. |
| **Secrets** | Cookie and storage _values_ require both `--values` and `HEADLESS_ALLOW_SENSITIVE_DIAGNOSTICS=1`. Authorization, cookie, token, and secret headers, plus URL credentials, are always redacted. Flow recordings never contain typed values. |
| **Untrusted content** | Everything derived from a page is marked `untrustedContent` and is never executed as a command. A page that induces the host to act on its own text is a vulnerability. |
| **Sandbox** | The Linux host refuses to run as root and never passes `--no-sandbox`. Snap Chromium is rejected before launch. |
Expand Down
5 changes: 4 additions & 1 deletion apps/headless/Host/AgentBridge.swift
Original file line number Diff line number Diff line change
Expand Up @@ -465,7 +465,10 @@ extension BrowserWindowController {
// WKWebView can expose its initial about:blank document before
// document-start scripts run. Install once in that document, then
// subsequent calls use the cached isolated-world runtime.
return try evaluate(agentRuntimeJavaScript + "\n" + agentEvaluationBody(body))
return try evaluate(
"globalThis.__headlessFileUpload = false;\n" + agentRuntimeJavaScript
+ "\n" + agentEvaluationBody(body)
)
}
}
}
Expand Down
93 changes: 91 additions & 2 deletions apps/headless/LinuxHost/BrowserProcess.swift
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ import Darwin
import Glibc
#endif

private let linuxAgentRuntimeJavaScript = "globalThis.__headlessFileUpload = true;\n" + agentRuntimeJavaScript

private final class ChromiumChildProcess {
let processIdentifier: Int32
private let lock = NSLock()
Expand Down Expand Up @@ -629,9 +631,10 @@ final class LinuxBrowserSession: @unchecked Sendable {
self.connection = connection
_ = try command("Page.enable")
_ = try command("Runtime.enable")
_ = try command("DOM.enable")
_ = try command("Log.enable")
_ = try command("Page.addScriptToEvaluateOnNewDocument", parameters: [
"source": agentRuntimeJavaScript,
"source": linuxAgentRuntimeJavaScript,
"worldName": "HeadlessAgent",
"runImmediately": true,
])
Expand Down Expand Up @@ -750,6 +753,24 @@ final class LinuxBrowserSession: @unchecked Sendable {
])
}

func upload(parameters: [String: JSONValue], artifactURL: URL) throws -> JSONValue {
let args = try browserTargetArguments(parameters)
let objectId = try evaluateNode(
"return globalThis.__headlessAgent.fileInput(args);",
input: ["args": args]
)
defer { _ = try? command("Runtime.releaseObject", parameters: ["objectId": objectId]) }
guard case .object(var metadata) = try fileInputMetadata(objectId: objectId) else {
throw CDPError.invalidResponse("file input metadata")
}
_ = try command("DOM.setFileInputFiles", parameters: [
"objectId": objectId,
"files": [artifactURL.path],
])
metadata["artifact"] = .string(artifactURL.lastPathComponent)
return .object(metadata)
}

func fill(parameters: [String: JSONValue]) throws -> JSONValue {
guard let value = parameters["value"]?.stringValue else { throw CDPError.commandFailed("missing value") }
let target = try trustedInputTarget(parameters: parameters, action: "fill")
Expand Down Expand Up @@ -1205,6 +1226,74 @@ final class LinuxBrowserSession: @unchecked Sendable {
]
}

/// Runtime.evaluate with returnByValue false so a DOM node keeps its
/// objectId for `DOM.setFileInputFiles`. The existing `evaluate` helper
/// always returns JSON and cannot yield a node handle.
private func evaluateNode(_ body: String, input: [String: Any] = [:]) throws -> String {
let inputData = try JSONSerialization.data(withJSONObject: input, options: [.sortedKeys])
guard let inputJSON = String(data: inputData, encoding: .utf8) else {
throw CDPError.invalidResponse("input encoding")
}
let expression = """
(() => {
const __input = \(inputJSON);
const args = __input.args;
\(body)
})()
"""
func evaluateParameters() throws -> [String: Any] {
[
"expression": expression,
"returnByValue": false,
"userGesture": true,
"contextId": try isolatedExecutionContextID(),
]
}
let response: [String: Any]
do {
response = try command("Runtime.evaluate", parameters: try evaluateParameters())
} catch let error as CDPError where isTransientNavigationContext(error) {
clearIsolatedContext()
response = try command("Runtime.evaluate", parameters: try evaluateParameters())
}
if let exception = response["exceptionDetails"] as? [String: Any] {
throw hostError(fromCDPException: exception)
}
guard let result = response["result"] as? [String: Any],
result["subtype"] as? String == "node",
let objectId = result["objectId"] as? String, !objectId.isEmpty else {
throw CDPError.invalidResponse("file input objectId")
}
return objectId
}

private func fileInputMetadata(objectId: String) throws -> JSONValue {
let response = try command("Runtime.callFunctionOn", parameters: [
"objectId": objectId,
"functionDeclaration": "function() { return globalThis.__headlessAgent.fileInputMetadata(this); }",
"returnByValue": true,
])
if let exception = response["exceptionDetails"] as? [String: Any] {
throw hostError(fromCDPException: exception)
}
guard let result = response["result"] as? [String: Any], let value = result["value"] else {
throw CDPError.invalidResponse("file input metadata")
}
return try JSONValue.foundationValue(value)
}

private func hostError(fromCDPException exception: [String: Any]) -> HostError {
let description = ((exception["exception"] as? [String: Any])?["description"] as? String)
?? (exception["text"] as? String)
?? "Browser operation failed"
let firstLine = description.split(whereSeparator: \.isNewline).first.map(String.init) ?? description
let trimmed = firstLine.hasPrefix("Error: ") ? String(firstLine.dropFirst(7)) : firstLine
let codeText = trimmed.split(separator: ":", maxSplits: 1, omittingEmptySubsequences: false)
.first.map(String.init) ?? ""
let code = HostErrorCode(rawValue: codeText) ?? .operationFailed
return HostError(code: code, message: String(decoding: trimmed.utf8.prefix(4_096), as: UTF8.self))
}

private func evaluate(_ body: String, input: [String: Any] = [:], timeout: TimeInterval = 10) throws -> JSONValue {
let timeoutMilliseconds = Int32(min(125_000, max(1, ceil(timeout * 1_000))))
let inputData = try JSONSerialization.data(withJSONObject: input, options: [.sortedKeys])
Expand Down Expand Up @@ -1291,7 +1380,7 @@ final class LinuxBrowserSession: @unchecked Sendable {
let installedValue = (installed["result"] as? [String: Any])?["value"] as? Bool ?? false
if !installedValue {
_ = try command("Runtime.evaluate", parameters: [
"expression": agentRuntimeJavaScript,
"expression": linuxAgentRuntimeJavaScript,
"returnByValue": true,
"contextId": identifier,
])
Expand Down
3 changes: 3 additions & 0 deletions apps/headless/LinuxHost/main.swift
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,9 @@ final class ChromiumBrowserEngineSession: BrowserEngineSession {
func hostFill(parameters: [String: JSONValue]) throws -> JSONValue {
try browserSession.fill(parameters: parameters)
}
func hostUpload(parameters: [String: JSONValue], artifactURL: URL) throws -> JSONValue {
try browserSession.upload(parameters: parameters, artifactURL: artifactURL)
}
func hostPress(parameters: [String: JSONValue]) throws -> JSONValue {
try browserSession.press(parameters: parameters)
}
Expand Down
22 changes: 21 additions & 1 deletion apps/headless/Sources/HeadlessProtocol/Artifacts.swift
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@ public final class ArtifactStore: @unchecked Sendable {
private static let listedExtensions: Set<String> =
ScreenshotFormat.artifactExtensions
.union(RecordingFormat.artifactExtensions)
.union(["json"])
.union(uploadArtifactExtensions)

private func metadata(for url: URL) throws -> JSONValue {
let attributes = try FileManager.default.attributesOfItem(atPath: url.path)
Expand All @@ -222,6 +222,26 @@ public final class ArtifactStore: @unchecked Sendable {
])
}

/// Resolves an already-stored upload artifact to its on-disk URL. Callers
/// receive a path inside this store only, never an agent-supplied path.
public func urlForExistingArtifact(
name: String, allowedExtensions: Set<String> = uploadArtifactExtensions
) throws -> URL {
lock.lock(); defer { lock.unlock() }
do { try validateArtifactName(name, expectedExtensions: allowedExtensions) }
catch { throw ArtifactError.invalidName(name) }
let url = rootURL.appendingPathComponent(name, isDirectory: false)
guard url.deletingLastPathComponent().standardizedFileURL == rootURL.standardizedFileURL else {
throw ArtifactError.invalidName(name)
}
var info = stat()
guard lstat(url.path, &info) == 0 else { throw ArtifactError.missing(name) }
guard (info.st_mode & S_IFMT) == S_IFREG else {
throw ArtifactError.writeFailed("Artifact is not a permitted regular file")
}
return url
}

/// Reads only a regular artifact owned by this store. Callers never receive
/// a path supplied by the agent, preventing an artifact command from
/// becoming an arbitrary-file read primitive.
Expand Down
18 changes: 18 additions & 0 deletions apps/headless/Sources/HeadlessProtocol/CLI.swift
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,8 @@ public struct CLIParser {
return try parseInspect(arguments, session: session, jsonOutput: jsonOutput)
case "click":
return try parseTargeted(.click, arguments: arguments, session: session, jsonOutput: jsonOutput)
case "upload":
return try parseUpload(arguments, session: session, jsonOutput: jsonOutput)
case "fill":
guard arguments.count == 2 else { throw CLIParseError.missingArgument("TARGET TEXT") }
return remote(.fill, session: session, parameters: [
Expand Down Expand Up @@ -367,6 +369,21 @@ public struct CLIParser {
return number
}

private func parseUpload(
_ arguments: [String], session: String?, jsonOutput: Bool
) throws -> CLIInvocation {
var args = arguments
let artifact = try removeOption("--artifact", from: &args)
guard let artifact else { throw CLIParseError.missingArgument("--artifact") }
try validateArtifactName(artifact, expectedExtensions: uploadArtifactExtensions)
let invocation = try parseTargeted(
.upload, arguments: args, session: session, jsonOutput: jsonOutput
)
var parameters = invocation.request?.parameters ?? [:]
parameters["artifact"] = .string(artifact)
return remote(.upload, session: session, parameters: parameters, jsonOutput: jsonOutput)
}

private func parseTargeted(
_ command: CommandName,
arguments: [String],
Expand Down Expand Up @@ -808,6 +825,7 @@ Commands:
[--within @rN] [--limit N] [--budget TOKENS] [--depth N] [--text]
click REF | click --role ROLE [--name NAME]
fill REF TEXT | fill REF -- TEXT_WITH_LITERAL_FLAGS | press KEY
upload REF --artifact FILE | upload --role ROLE [--name NAME] --artifact FILE
scroll [up|down|top|bottom] [--amount PX]
back | reload
wait [--settled] [--url PATTERN] [--text TEXT] [--timeout MS]
Expand Down
Loading