fix: cancellation turn-granularity, extension report gaps, and cache-aware cost pricing - #236
Conversation
The CLI's AbortSignal was only checked between attacks/evaluators, so a long multi-turn attack (e.g. turns: 100) couldn't be interrupted until every turn finished. The extension had a parallel gap: its own stop signal wasn't recognized by runAllBrowser, so a cancelled run returned no report at all — dropping token/cost data and leaving a fake blank turn in the HTML report. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
WalkthroughThe change adds cancellation to attack execution, preserves partial stopped-run data, tracks cache-tier token usage, calculates tiered costs, and displays cost and final transcript details in the extension report. ChangesCancellation and cost reporting
Estimated code review effort: 3 (Moderate) | ~30 minutes Sequence Diagram(s)sequenceDiagram
participant Extension as Extension orchestrator
participant Browser as runAllBrowser
participant Evaluator as evaluatorLoop
participant Runner as runAttack
participant Tracker as TokenTracker
participant Popup as Extension popup
Extension->>Browser: request evaluator run
Browser->>Evaluator: run evaluators with AbortSignal
Evaluator->>Runner: start MCP or agent attack
Runner-->>Evaluator: stop before next turn and finalize
Evaluator->>Tracker: record token usage
Browser-->>Extension: run_stopped with partial results
Extension->>Popup: provide token usage and evaluator results
Popup->>Popup: calculate and display estimated cost
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
runners/extension/popup.js (1)
1423-1443: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winKeep
lastlimited to turns with an assistant response.
turnsForReport()outputs a turn when either the user or assistant entry exists. If an interrupted evaluator leaves an in-flight user message inraw.transcript,lastcan come from a turn whoseresponseis empty, andturnscan include that incomplete entry. Filter turns with a non-empty assistant response before selectinglastand before returningturns.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@runners/extension/popup.js` around lines 1423 - 1443, Update the turn handling in turnsForReport to retain only turns with a non-empty assistant response, then select last from that filtered collection and return it as turns. Preserve the existing fallback to empty prompt and response when no completed turns remain.
🧹 Nitpick comments (1)
runners/extension/popup.js (1)
14-15: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueImport the cost helpers directly from their source module.
estimateRunCostandformatUsdare defined incore/src/pricing/estimateCost.ts, so import them from that file instead of re-exporting them throughcore/src/browser.tsand then bundling from./dist/core.bundle.js.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@runners/extension/popup.js` around lines 14 - 15, Update the imports for estimateRunCost and formatUsd in the popup module to reference their defining source module directly, core/src/pricing/estimateCost.ts, instead of importing through core/src/browser.ts or ./dist/core.bundle.js. Keep both helper usages unchanged.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@core/src/execute/runAllBrowser.ts`:
- Around line 217-222: Update the extension orchestrator’s event handling to add
a run_stopped branch alongside the existing attack_start and attack_done
branches. Forward the stopped event by calling broadcastProgress with its
reason/state so the popup displays the partial run reason emitted by
runAllBrowser.
---
Outside diff comments:
In `@runners/extension/popup.js`:
- Around line 1423-1443: Update the turn handling in turnsForReport to retain
only turns with a non-empty assistant response, then select last from that
filtered collection and return it as turns. Preserve the existing fallback to
empty prompt and response when no completed turns remain.
---
Nitpick comments:
In `@runners/extension/popup.js`:
- Around line 14-15: Update the imports for estimateRunCost and formatUsd in the
popup module to reference their defining source module directly,
core/src/pricing/estimateCost.ts, instead of importing through
core/src/browser.ts or ./dist/core.bundle.js. Keep both helper usages unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e5ddf7a0-b139-4175-a656-5bff67a48eb7
📒 Files selected for processing (9)
core/src/execute/attackRunner.tscore/src/execute/evaluatorLoop.tscore/src/execute/mcpAttackDriver.tscore/src/execute/runAgentLoop.tscore/src/execute/runAllBrowser.tscore/tests/attackRunner.test.tsrunners/extension/orchestrator.jsrunners/extension/popup.htmlrunners/extension/popup.js
| if ((err as { code?: string })?.code === "OPFOR_STOP") { | ||
| stopReason = err instanceof Error ? err.message : "Run stopped by user."; | ||
| notify({ type: "run_stopped", reason: stopReason }); | ||
| pushPartialResult(stopReason); | ||
| break evaluatorLoop; | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
ast-grep outline runners/extension/orchestrator.js --items all
rg -n -C 6 'run_stopped|onProgress|broadcastProgress|setRunStatus|finalizeUserInterruption' \
runners/extension/orchestrator.jsRepository: KeyValueSoftwareSystems/agent-opfor
Length of output: 12727
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## runAllBrowser relevant section"
sed -n '180,235p' core/src/execute/runAllBrowser.ts
echo
echo "## extension run_stopped occurrences"
rg -n -C 8 'run_stopped|OPFOR_STOP|type === "attack_done"|emit\(|create.*Run|runAll\(' .
echo
echo "## executeAdaptiveRedTeamRun message send around stop"
sed -n '840,905p' runners/extension/orchestrator.jsRepository: KeyValueSoftwareSystems/agent-opfor
Length of output: 50394
Forward run_stopped from core to the extension popup.
runAllBrowser emits run_stopped for user-cancel/error paths, but runners/extension/orchestrator.js only calls broadcastProgress() for attack_start and attack_done. Add a matching branch and broadcast the stopped state so the extension UI can show the partial run reason.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@core/src/execute/runAllBrowser.ts` around lines 217 - 222, Update the
extension orchestrator’s event handling to add a run_stopped branch alongside
the existing attack_start and attack_done branches. Forward the stopped event by
calling broadcastProgress with its reason/state so the popup displays the
partial run reason emitted by runAllBrowser.
Cost estimation charged every input token at the full input rate, but `inputTokens` is the inclusive total — it already contains the cached tokens providers bill far more cheaply. The token counter was also discarding the cache split the AI SDK already reports. Input is now divided across its tiers and each priced at its own rate. Validated against LiteLLM billing for a 24-request run: previously reported $0.070749, actually billed $0.057939, now reports $0.057939 — every request agreeing to 12 decimal places. Reported fields are unchanged; only the cost figure moves. A run whose provider reports no cache split prices exactly as before, and a tier with no published rate falls back to the full input rate rather than to free. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
arunSunnyKVS
changed the title
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@core/src/execute/tokenTracker.ts`:
- Around line 99-124: Update the token usage transform around the
inputTokenDetails schema and its noCache calculation to always derive noCache
from inputTokens minus cacheRead and cacheWrite, ignoring any provider-supplied
noCacheTokens override. Validate and reject inputs where cacheRead plus
cacheWrite exceeds inputTokens, and add regression tests covering both the
override case and the over-limit case.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: b2540b77-5e97-4f72-bf8d-999e9b831573
📒 Files selected for processing (5)
core/src/execute/tokenTracker.tscore/src/pricing/estimateCost.tscore/src/pricing/types.tscore/tests/pricing.test.tscore/tests/tokenTracker.test.ts
The transform read `noCacheTokens` verbatim, so a provider reporting inputTokens 100 with noCache 100 and cacheRead 50 produced a 150-token split against a 100-token call — and estimateRunCost billed all 150. The invariant was asserted in the docs and tests but never enforced. Derive the fresh count from inputTokens instead. For a well-formed provider the two agree (the AI SDK builds inputTokens as the sum), so this costs nothing and makes the invariant hold by construction. A split claiming more cached tokens than there was input can't be divided at all, so it is dropped and the call prices at the full input rate. Addresses CodeRabbit review on #236. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An attack interrupted before its first turn completed has no turns and an empty detail card, but the renderer emitted a transcript anyway — headed "1 turn" with two blank bubbles. That reads as an exchange that happened and came back empty, rather than as nothing having run. Skip the transcript and its toggle when there is no content on either the turns list or the detail card. The judge's error message still renders, so the reason the attack produced nothing is not lost with it. Docs: the cache-aware pricing landed earlier in this PR made the "multi-turn runs are over-estimated" caveat wrong in README.md and docs/cli.md; replaced with what actually happens now and the cases that still over-estimate. AGENTS.md gains the turn-granular cancellation contract and the inclusive-split invariant behind the cost maths. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@README.md`:
- Around line 127-129: Update the README pricing explanation to distinguish
cached-input reads from cache writes: describe cache reads as potentially much
cheaper than fresh input, while noting that cache writes may cost more,
including provider-specific rates. Preserve the existing explanation that
repeated prefixes are billed according to reported provider cache rates.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 964875dc-7abd-46a4-a2f1-25ce8d4492bc
📒 Files selected for processing (5)
AGENTS.mdREADME.mdcore/src/report/render.tscore/tests/render.test.tsdocs/cli.md
| Prices come from a snapshot of LiteLLM's public price map that ships with the package, so runs work offline and a report re-rendered later produces the same figure. Cached input is billed at the provider's cache rate — multi-turn attacks re-send the conversation each turn, and that repeated prefix is often ~100× cheaper than fresh text, so the figure tracks the real bill rather than a worst case. Caveats worth knowing: | ||
|
|
||
| - **Multi-turn runs read high.** Providers discount repeated context, and a multi-turn attack re-sends the conversation each turn — opfor prices every input token at full rate, so the real bill is usually lower. | ||
| - **Caching is only credited when it's reported.** A provider that doesn't break out cached tokens, or a model with no published cache rate, is charged at the full input rate — an over-estimate, chosen over quietly under-reporting. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Anthropic cache write multipliers 1.25x 2x pricing
💡 Result:
Anthropic's prompt caching pricing is structured around a "write premium" paid when content is first stored, followed by a significant discount on subsequent reads [1][2]. These multipliers are applied relative to the model's base input token price [1][3]. The pricing multipliers for prompt caching are as follows: * 5-Minute Cache Write: 1.25x the base input price [1][2]. This tier keeps the cache valid for 5 minutes [1][3]. * 1-Hour Cache Write: 2x the base input price [1][2]. This tier keeps the cache valid for 1 hour [1][3]. * Cache Read (Hit): 0.1x the base input price [1][2]. This provides a 90% discount on standard input costs for any content retrieved from the cache [4][2]. Because cache reads cost only 10% of the standard input price, prompt caching generally becomes cost-effective after a small number of hits [1][4]. For the 5-minute cache (1.25x write), the investment is typically recovered after just one cache read [1][2]. For the 1-hour cache (2x write), it typically takes two cache reads to break even [1][2]. These multipliers stack with other pricing modifiers, such as the Batch API discount and data residency adjustments [3][5].
Citations:
- 1: https://platform.claude.com/docs/en/about-claude/pricing
- 2: https://ssimplifi.com/blog/anthropic-prompt-caching-explained
- 3: https://platform.claude.com/docs/en/build-with-claude/prompt-caching
- 4: https://tokenmix.ai/blog/claude-api-cache-pricing
- 5: https://platform.claude.com/docs/en/about-claude/pricing?hsLang=en
🌐 Web query:
Anthropic pricing cache read 90% discount cache write multiplier
💡 Result:
Anthropic’s prompt caching pricing is based on a system of multipliers applied to the base input token rate [1][2][3]. Cache reads are charged at 0.1x (10%) of the base input price, effectively providing a 90% discount on those tokens compared to standard, uncached input [1][4][3]. Cache writes involve a premium because the content must be stored [1][2][5]. There are two tiers for cache writes, each with a different multiplier relative to the base input rate: 5-Minute Cache Write: 1.25x the base input price [1][6][3]. This tier is valid for 5 minutes and is generally more cost-effective for bursty or high-frequency interactions [3][5]. 1-Hour Cache Write: 2x the base input price [1][2][3]. This tier is valid for 1 hour and is intended for longer-running workflows where the cached content will be reused multiple times [4][3][5]. Because cache hits cost only 10% of the standard input rate, the upfront write premium is typically recovered quickly: the 5-minute cache write generally pays for itself after a single cache read, while the 1-hour cache write typically breaks even after two cache reads [1][2][4][3].
Citations:
- 1: https://platform.claude.com/docs/en/about-claude/pricing.md
- 2: https://flatkey.ai/blog/claude-api-pricing-explained
- 3: https://www.romainlespinasse.dev/posts/choosing-prompt-cache-tier/
- 4: https://cadence.withremote.ai/blog/prompt-caching-anthropic
- 5: https://tokenmix.ai/blog/claude-api-cache-pricing
- 6: https://brandonwie.dev/posts/anthropic-prompt-cache-ttl
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'README relevant section:\n'
sed -n '120,136p' README.md | cat -n
printf '\nCache/tier terms and LiteLLM price map references:\n'
rg -n "cache|LiteLLM|cache-write|cache_read|cache_write|cache reads|cache writes|pricing|price map" README.mdRepository: KeyValueSoftwareSystems/agent-opfor
Length of output: 2672
Distinguish cache reads from cache writes.
Line 127 says cached input is “often ~100× cheaper than fresh text,” but that describes cache reads. Cache writes can be pricier than fresh input, including Anthropic’s 5-minute cache-write rate at 1.25× base input pricing and the 1-hour cache-write rate at 2× base input pricing. Update this to describe cache-read and cache-write rates separately.
Proposed documentation change
- Cached input is billed at the provider's cache rate — multi-turn attacks re-send the conversation each turn, and that repeated prefix is often ~100× cheaper than fresh text, so the figure tracks the real bill rather than a worst case.
+ Cache-read and cache-write input are billed at their separate published rates. In multi-turn attacks, a repeated prefix can be much cheaper when it is reported as a cache read. Cache writes can cost more than fresh input.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Prices come from a snapshot of LiteLLM's public price map that ships with the package, so runs work offline and a report re-rendered later produces the same figure. Cached input is billed at the provider's cache rate — multi-turn attacks re-send the conversation each turn, and that repeated prefix is often ~100× cheaper than fresh text, so the figure tracks the real bill rather than a worst case. Caveats worth knowing: | |
| - **Multi-turn runs read high.** Providers discount repeated context, and a multi-turn attack re-sends the conversation each turn — opfor prices every input token at full rate, so the real bill is usually lower. | |
| - **Caching is only credited when it's reported.** A provider that doesn't break out cached tokens, or a model with no published cache rate, is charged at the full input rate — an over-estimate, chosen over quietly under-reporting. | |
| Prices come from a snapshot of LiteLLM's public price map that ships with the package, so runs work offline and a report re-rendered later produces the same figure. Cache-read and cache-write input are billed at their separate published rates. In multi-turn attacks, a repeated prefix can be much cheaper when it is reported as a cache read. Cache writes can cost more than fresh input. Caveats worth knowing: | |
| - **Caching is only credited when it's reported.** A provider that doesn't break out cached tokens, or a model with no published cache rate, is charged at the full input rate — an over-estimate, chosen over quietly under-reporting. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` around lines 127 - 129, Update the README pricing explanation to
distinguish cached-input reads from cache writes: describe cache reads as
potentially much cheaper than fresh input, while noting that cache writes may
cost more, including provider-specific rates. Preserve the existing explanation
that repeated prefixes are billed according to reported provider cache rates.
2 participants
Problem
Four bugs, surfaced while testing
opfor runwith Ctrl+C, the extension's Stop button, and comparing reported cost against actual billing.Cancellation (1) —
AbortSignalcancellation was only checked between attacks/evaluators, never between turns within one multi-turn attack. A long-running attack (e.g.turns: 100) was atomic — Ctrl+C wouldn't stop it until every turn finished, contradicting the "finishes in-flight attack" promise in the CLI's own interrupt message.Empty transcripts (2) — an attack interrupted before its first turn completed has no turns and an empty detail card, but the renderer emitted a transcript anyway: headed "1 turn", containing two blank bubbles. That reads as an exchange that happened and came back empty, rather than as nothing having run.
Extension reporting (3) — cancelled runs showed no token usage or cost at all.
DomTarget.send()throws anErrortaggedcode: "OPFOR_STOP"on user cancel, whichrunAllBrowserdidn't recognise as a clean stop condition, so it fell through to an unhandledthrowand returned no report — silently dropping the run's token totals. Separately,costwas never plumbed through the extension even on successful runs.Cost accuracy (4) — cost estimation charged every input token at the full input rate. But
inputTokensis the inclusive total: it already contains the cached tokens that providers bill far more cheaply. The token counter was also discarding the cache split the AI SDK already reports. On a real 24-request run this overstated cost by 22%.Solution
AbortSignalone level deeper, intorunAttack()'s per-turn loop, so it's checked before every turn and breaks out the same way the existing early-stop-on-target-error path does — still callingfinalize()so a partial transcript is judged and reported.OPFOR_STOP-tagged-error branch torunAllBrowser's catch block (recognised structurally via.code, since core can't import the extension's error class) so a user-cancelled run returns a proper partial report. ThreadedtokenUsageByModelthroughorchestrator.js, aggregated it inpopup.jsvia the already-bundledestimateRunCost, and wired the result into both the downloaded report and a new Cost stat card.inputTokenDetails(so DeepSeek'sprompt_tokens_details.cached_tokensand Anthropic'scache_read_input_tokensboth work without provider-specific code).Validation against real billing
The cost change was verified against LiteLLM's own spend logs for a 24-request run:
Every one of the 24 requests agrees to 12 decimal places. The old formula reported $0.070749 for the same run — a 22.1% overstatement.
Changes
Cancellation
core/src/execute/attackRunner.ts—runAttack()accepts an optionalsignal, checked at the top of each turncore/src/execute/runAgentLoop.ts,core/src/execute/mcpAttackDriver.ts— forwardsignalthrough torunAttackcore/src/execute/evaluatorLoop.ts— pass the already-availablesignalinto both attack call sitesReporting
core/src/report/render.ts— omit the transcript when there is no exchange to showcore/src/execute/runAllBrowser.ts— recogniseOPFOR_STOP-tagged errors as a clean stop condition, same asTargetStopErrorrunners/extension/popup.js— derivedetailfrom the last completed turn (aligning with core'sbuildReport.tscontract); aggregatetokenUsageByModeland derivecost; populate the new Cost statrunners/extension/popup.html— add astatCostcard next to the existingstatTokenscardrunners/extension/orchestrator.js— threadtokenUsageByModelthrough all three result paths (success, error, stopped)Cost pricing
core/src/execute/tokenTracker.ts— capture the cache split instead of discarding it; derive the fresh count frominputTokensso the tiers always sum to itcore/src/pricing/estimateCost.ts— price each input tier at its own ratecore/src/pricing/types.ts— the "Not applied yet" doc comments, now appliedDocs —
README.mdanddocs/cli.mdcarried a "multi-turn runs are over-estimated" caveat that this PR makes false; replaced with current behaviour and the cases that still over-estimate.AGENTS.mdgains the turn-granular cancellation contract and the inclusive-split invariant behind the cost maths.Tests —
core/tests/attackRunner.test.ts,core/tests/pricing.test.ts,core/tests/tokenTracker.test.ts, and a newcore/tests/render.test.ts(28 new).Backward compatibility
cacheRead/cacheWritefall to 0,noCacheabsorbs the total, and the formula collapses to the old one).0is honoured (??not||) — DeepSeek genuinely publishescw: 0, and there's a test pinning this.inputTokensis not trusted: the fresh count is derived, and a split claiming more cached tokens than there was input is dropped entirely.Review feedback
CodeRabbit raised four findings; one was valid and is fixed in
c481c02:noCacheTokens, soinputTokens: 100+noCacheTokens: 100+cacheRead: 50produced a 150-token split against a 100-token call. The invariant was asserted in the docs and tests but never enforced. Two regression tests added.run_stoppedto the extension popup (Minor) — skipped. The popup already learns the stop reason via a different channel: user-cancel goes throughfinalizeUserInterruption(explicit pause/cancel intent) and error paths persist a partial result carryingstopReason. Pre-existing gap, not introduced here.DomTarget.send()pushes the user and assistant transcript entries atomically after extraction, or throws before pushing either, so no odd trailing user message is produced. The suggestion's second half would also drop the in-flight prompt, which is the most informative part of a cancelled run's transcript.estimateCost.tsdirectly (Trivial) — not applicable.popup.jsis extension code and cannot import fromcore/src/; every import there comes from./dist/core.bundle.js, the esbuild bundle ofcore/src/browser.ts.Issue
N/A
How to test
npm run buildturnMode: "multi"and a highturnscount (e.g. 20+), Ctrl+C mid-attack, confirm it stops within one turn's latency instead of waiting for all turns.npm test— 304 tests pass (28 new).npm run typecheck/npm run lint— clean.Screenshots
N/A
Summary by CodeRabbit
New Features
Bug Fixes