Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -277,7 +277,7 @@ There is no longer a separate `generate` step. `opfor run --config <file>` does

**MCP targets** additionally run baseline pre-flight scans (`runBaselineScans`) before the evaluator loop — these enumerate `tools/list` + `resources/list` and judge them for poisoning / leakage independent of any evaluator. Throughout the run, `runAll` fans lifecycle events to registered `RunListener`s (progress reporting, NDJSON streaming) rather than only a callback.

**Cancellation.** `RunAllOptions` accepts an optional `signal?: AbortSignal`. It is threaded down to `runAttack` and checked **before each turn**, not just between attacks — otherwise a single high-`turns` attack would run to completion before the stop took effect. When aborted, the in-flight turn finishes, the attack is finalized (judging whatever transcript exists), remaining evaluators/attacks are skipped, and a partial report is returned with `stopReason: "user-interrupted"`. The CLI wires this to SIGINT (first Ctrl+C = graceful stop, second = force kill). The SDK can reuse the same mechanism for programmatic cancellation.
**Cancellation.** `RunAllOptions` accepts an optional `signal?: AbortSignal`. It is threaded down to `runAttack` and checked **before each turn**, not just between attacks — otherwise a single high-`turns` attack would run to completion before the stop took effect. When aborted, the in-flight turn finishes, the attack is finalized (judging whatever transcript exists), remaining evaluators/attacks are skipped, and a partial report is returned with `stopReason: "user-interrupted"`. The CLI wires this to SIGINT (first Ctrl+C = graceful stop, second = force kill). The SDK does **not** surface `signal` today — `runners/sdk` accepts no cancellation option, so plumbing one through is the work required to offer programmatic cancellation.

The browser extension reaches the same granularity by a different route: `DomTarget.send()` checks `state.OPFOR_STOP` around each send/extract and throws an error tagged `code: "OPFOR_STOP"`, which `runAllBrowser` recognizes as a clean stop (structurally, since core cannot import the extension's class) and turns into a partial report rather than an unhandled throw.

Expand Down
11 changes: 10 additions & 1 deletion docs/browser-extension.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,15 @@ The extension uses a **single LLM configuration** for all operations (attack gen

The extension runs up to **20 turns per evaluator** (default 10). It stops a given evaluator early when the judge returns a definitive verdict.

**Stopping a run.** Both controls take effect within the current turn rather than at the end of the evaluator, so you never wait out the remaining turn budget. A stop can land after a message has been sent but before the reply is read; that turn is then dropped rather than recorded half-finished.

| Control | What you get |
| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Pause | A resumable snapshot. Evaluators completed so far keep their verdicts; the one in progress has none yet and is not scored. |
| Stop | A final report. The evaluator in progress is marked `CANCELLED` rather than judged — scoring a conversation the target never finished would invent a verdict. |

Either way you still get the run's token usage and cost for the work that did happen, and the report downloads the same as a completed one.

**Token usage and testing cost** are tracked per evaluator and shown on the Done screen and in the downloadable HTML report. Cost covers the attacker and judge LLMs you configured in Options. Whatever the target chat spends on its own inference is excluded — opfor drives it through the browser and cannot observe or meter that spend. See [Token usage and testing cost](cli.md#token-usage-and-testing-cost) for how the figure is derived and its caveats.

---
Expand Down Expand Up @@ -116,7 +125,7 @@ Same agent-redteam catalog as the CLI. Pick by suite or select "Custom Evaluator
- **No MCP / live tool-call evaluators.** The judge sees the transcript, not real tool side-effects. For MCP server red-teaming use the CLI's `mode: "mcp"` or the [MCP server tool](mcp.md).
- **One agent per run.** No cross-agent or inter-agent communication tests.
- **Vendor closed shadow-DOM widgets** (Salesforce, some Intercom builds) may need a vendor-specific fallback — open an issue with a sample URL if auto-detect fails.
- **No pause / resume across sessions.** Run state lives in `chrome.storage.local`; uninstalling the extension wipes it.
- **A paused run does not survive uninstall.** Pause/resume works within the browser profile, but the snapshot lives in `chrome.storage.local` uninstalling the extension wipes it.

---

Expand Down
12 changes: 7 additions & 5 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,12 +202,14 @@ Where `<slug>` is the target name slugified (e.g. `erkala-travel-support-agent`)

Pressing Ctrl+C during `opfor run` triggers a graceful shutdown instead of killing the process:

| Press | Behavior |
| ------ | --------------------------------------------------------------------------------------------------------- |
| First | Finishes the in-flight attack, skips remaining evaluators/attacks, writes a partial report, exits cleanly |
| Second | Force-kills immediately (`exit 130`) |
| Press | Behavior |
| ------ | ------------------------------------------------------------------------------------------------------------- |
| First | Finishes the in-flight turn, skips remaining turns/attacks/evaluators, writes a partial report, exits cleanly |
| Second | Force-kills immediately (`exit 130`) |

Partial reports include all completed evaluator results and are marked with `stopReason: "user-interrupted"` in the JSON output. The CLI prints a warning:
The stop is checked between **turns**, not just between attacks, so a multi-turn attack stops after the turn in progress rather than running its full turn budget first. What that costs you is one turn's latency — a model call plus a target call — however high `turns` is set.

Partial reports include all completed evaluator results and are marked with `stopReason: "user-interrupted"` in the JSON output. The interrupted attack is still judged on the transcript it managed to collect, so a partially-completed multi-turn attack is not discarded. The CLI prints a warning:

```
⚠️ Run interrupted — results are partial. Re-run for a complete assessment.
Expand Down
Loading