From 0096871aab24d9d134aecfd63c44f4023c09075a Mon Sep 17 00:00:00 2001 From: Arun Sunny Date: Thu, 6 Aug 2026 16:34:32 +0530 Subject: [PATCH] docs: correct cancellation and stop behaviour across the docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A sweep of every first-party doc against the behaviour changed in this PR turned up three stale or missing pieces: - docs/cli.md promised Ctrl+C "finishes the in-flight attack". It now finishes the in-flight turn, which is the point of the change — the old wording described the bug as if it were the contract. - docs/browser-extension.md never documented Stop or Pause at all, though both are prominent controls and this PR changed what a stopped run leaves behind. Added what each produces, and corrected the neighbouring limitation that read as though pause did not exist. - AGENTS.md said the SDK "can reuse the same mechanism" for cancellation, which reads as though it already does; runners/sdk accepts no signal. Verified against the code rather than recollection: the extension checks for a stop between sending a message and reading the reply, so it can abandon a sent turn — the docs now say so instead of claiming turns always complete. Co-Authored-By: Claude Opus 5 --- AGENTS.md | 2 +- docs/browser-extension.md | 11 ++++++++++- docs/cli.md | 12 +++++++----- 3 files changed, 18 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 92639ba..bc287f9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -277,7 +277,7 @@ There is no longer a separate `generate` step. `opfor run --config ` does **MCP targets** additionally run baseline pre-flight scans (`runBaselineScans`) before the evaluator loop — these enumerate `tools/list` + `resources/list` and judge them for poisoning / leakage independent of any evaluator. Throughout the run, `runAll` fans lifecycle events to registered `RunListener`s (progress reporting, NDJSON streaming) rather than only a callback. -**Cancellation.** `RunAllOptions` accepts an optional `signal?: AbortSignal`. It is threaded down to `runAttack` and checked **before each turn**, not just between attacks — otherwise a single high-`turns` attack would run to completion before the stop took effect. When aborted, the in-flight turn finishes, the attack is finalized (judging whatever transcript exists), remaining evaluators/attacks are skipped, and a partial report is returned with `stopReason: "user-interrupted"`. The CLI wires this to SIGINT (first Ctrl+C = graceful stop, second = force kill). The SDK can reuse the same mechanism for programmatic cancellation. +**Cancellation.** `RunAllOptions` accepts an optional `signal?: AbortSignal`. It is threaded down to `runAttack` and checked **before each turn**, not just between attacks — otherwise a single high-`turns` attack would run to completion before the stop took effect. When aborted, the in-flight turn finishes, the attack is finalized (judging whatever transcript exists), remaining evaluators/attacks are skipped, and a partial report is returned with `stopReason: "user-interrupted"`. The CLI wires this to SIGINT (first Ctrl+C = graceful stop, second = force kill). The SDK does **not** surface `signal` today — `runners/sdk` accepts no cancellation option, so plumbing one through is the work required to offer programmatic cancellation. The browser extension reaches the same granularity by a different route: `DomTarget.send()` checks `state.OPFOR_STOP` around each send/extract and throws an error tagged `code: "OPFOR_STOP"`, which `runAllBrowser` recognizes as a clean stop (structurally, since core cannot import the extension's class) and turns into a partial report rather than an unhandled throw. diff --git a/docs/browser-extension.md b/docs/browser-extension.md index ad4d857..00bea39 100644 --- a/docs/browser-extension.md +++ b/docs/browser-extension.md @@ -88,6 +88,15 @@ The extension uses a **single LLM configuration** for all operations (attack gen The extension runs up to **20 turns per evaluator** (default 10). It stops a given evaluator early when the judge returns a definitive verdict. +**Stopping a run.** Both controls take effect within the current turn rather than at the end of the evaluator, so you never wait out the remaining turn budget. A stop can land after a message has been sent but before the reply is read; that turn is then dropped rather than recorded half-finished. + +| Control | What you get | +| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Pause | A resumable snapshot. Evaluators completed so far keep their verdicts; the one in progress has none yet and is not scored. | +| Stop | A final report. The evaluator in progress is marked `CANCELLED` rather than judged — scoring a conversation the target never finished would invent a verdict. | + +Either way you still get the run's token usage and cost for the work that did happen, and the report downloads the same as a completed one. + **Token usage and testing cost** are tracked per evaluator and shown on the Done screen and in the downloadable HTML report. Cost covers the attacker and judge LLMs you configured in Options. Whatever the target chat spends on its own inference is excluded — opfor drives it through the browser and cannot observe or meter that spend. See [Token usage and testing cost](cli.md#token-usage-and-testing-cost) for how the figure is derived and its caveats. --- @@ -116,7 +125,7 @@ Same agent-redteam catalog as the CLI. Pick by suite or select "Custom Evaluator - **No MCP / live tool-call evaluators.** The judge sees the transcript, not real tool side-effects. For MCP server red-teaming use the CLI's `mode: "mcp"` or the [MCP server tool](mcp.md). - **One agent per run.** No cross-agent or inter-agent communication tests. - **Vendor closed shadow-DOM widgets** (Salesforce, some Intercom builds) may need a vendor-specific fallback — open an issue with a sample URL if auto-detect fails. -- **No pause / resume across sessions.** Run state lives in `chrome.storage.local`; uninstalling the extension wipes it. +- **A paused run does not survive uninstall.** Pause/resume works within the browser profile, but the snapshot lives in `chrome.storage.local` — uninstalling the extension wipes it. --- diff --git a/docs/cli.md b/docs/cli.md index e78cb65..4ec5a24 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -202,12 +202,14 @@ Where `` is the target name slugified (e.g. `erkala-travel-support-agent`) Pressing Ctrl+C during `opfor run` triggers a graceful shutdown instead of killing the process: -| Press | Behavior | -| ------ | --------------------------------------------------------------------------------------------------------- | -| First | Finishes the in-flight attack, skips remaining evaluators/attacks, writes a partial report, exits cleanly | -| Second | Force-kills immediately (`exit 130`) | +| Press | Behavior | +| ------ | ------------------------------------------------------------------------------------------------------------- | +| First | Finishes the in-flight turn, skips remaining turns/attacks/evaluators, writes a partial report, exits cleanly | +| Second | Force-kills immediately (`exit 130`) | -Partial reports include all completed evaluator results and are marked with `stopReason: "user-interrupted"` in the JSON output. The CLI prints a warning: +The stop is checked between **turns**, not just between attacks, so a multi-turn attack stops after the turn in progress rather than running its full turn budget first. What that costs you is one turn's latency — a model call plus a target call — however high `turns` is set. + +Partial reports include all completed evaluator results and are marked with `stopReason: "user-interrupted"` in the JSON output. The interrupted attack is still judged on the transcript it managed to collect, so a partially-completed multi-turn attack is not discarded. The CLI prints a warning: ``` ⚠️ Run interrupted — results are partial. Re-run for a complete assessment.