fix(hunt): stabilize autonomous mode reliability - #228
Open
jithin23-kv wants to merge 1 commit into
Open
Conversation
…tection, truncation) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
WalkthroughChangesThe PR updates autonomous dispatch configuration, tool-search guidance, budget correction, truncation reporting, dispatch prompt rendering, transient HTTP retries, and credential-based hunt verification defaults. Autonomous execution and reporting
Transient HTTP retries
Hunt verification configuration
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Agent
participant ClaudeAgentSDK
participant ToolSearch
participant RunLog
Agent->>ClaudeAgentSDK: invoke dispatch tool
ClaudeAgentSDK-->>Agent: empty tool result
Agent->>ToolSearch: resolve deferred schema
Agent->>ClaudeAgentSDK: retry tool call
ClaudeAgentSDK-->>RunLog: record dispatch prompt
RunLog-->>Agent: render dispatch details and truncation reason
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
core/src/autonomous/report/mapRunLog.ts (1)
170-179: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winValidate Agent/Task input before building dispatch decisions.
entry.inputis a tool-input payload, not something typed by this report module. Cast it here, and a non-stringdescriptioncan enterReportDecision.rationale; the rendering contract expects string fields. Parse it with Zod and use parsed values or safe defaults before creatingReportDecision.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/src/autonomous/report/mapRunLog.ts` around lines 170 - 179, Validate Agent/Task entry.input with Zod before constructing the dispatch decision in the transcript loop, rather than relying on the current type cast. Use the parsed string description and prompt values, with safe defaults for invalid or absent fields, so ReportDecision.rationale and dispatchPrompt remain string-compatible.Source: Coding guidelines
🧹 Nitpick comments (1)
core/src/targets/httpClient.ts (1)
248-309: 🩺 Stability & Availability | 🔵 Trivial | 🏗️ Heavy liftMock
fetchto cover the bounded retry paths.
opfor huntrelies onhttpSendretrying transient5xx, timeout/network failures, and returning after a bounded number of attempts, while handling429and non-retriable4xxdistinctly. Add unit coverage in a TypeScript test that mocksfetchfor the 5xx retry-then-succeed, exhausted-retries error, network/timeout retry,429passthrough, and4xximmediate return cases.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/src/targets/httpClient.ts` around lines 248 - 309, Add TypeScript unit tests for httpSend that mock fetch and cover 5xx retry-then-success, exhausted transient retries, network/timeout failures, 429 passthrough, and immediate non-retriable 4xx responses. Assert bounded fetch attempts and the returned response/error fields, using the existing retry and timeout configuration symbols without changing httpSend behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@runners/cli/src/commands/hunt.ts`:
- Around line 424-425: Update the --ui setup path’s initialConfig construction
to pass opts.verify, preserving both explicit true and false values before
SetupPage’s automatic credential-based defaulting runs. Ensure setup mode honors
--verify without credentials and --no-verify despite available credentials, and
add coverage for both flags in the setup-mode tests.
- Around line 189-193: Prevent simultaneous use of the --verify and --no-verify
options in the hunt command: track their presence independently or detect both
flags before constructing huntOptions, then reject the invocation with a clear
error. Ensure huntOptions.verify is derived only after this validation so option
order cannot determine the verification behavior.
In `@runners/cli/ui/src/components/SetupPage.tsx`:
- Around line 144-151: Validate the `/api/brain-auth` payload with a Zod schema
before `brainAuth.method` is used by the verification `useEffect`. Parse the
JSON response through that schema, retain only successfully validated data, and
update state only when the validated response contains a method; preserve the
existing manual-choice guard.
---
Outside diff comments:
In `@core/src/autonomous/report/mapRunLog.ts`:
- Around line 170-179: Validate Agent/Task entry.input with Zod before
constructing the dispatch decision in the transcript loop, rather than relying
on the current type cast. Use the parsed string description and prompt values,
with safe defaults for invalid or absent fields, so ReportDecision.rationale and
dispatchPrompt remain string-compatible.
---
Nitpick comments:
In `@core/src/targets/httpClient.ts`:
- Around line 248-309: Add TypeScript unit tests for httpSend that mock fetch
and cover 5xx retry-then-success, exhausted transient retries, network/timeout
failures, 429 passthrough, and immediate non-retriable 4xx responses. Assert
bounded fetch attempts and the returned response/error fields, using the
existing retry and timeout configuration symbols without changing httpSend
behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 3747cace-4573-423e-b7d9-83e666d3457d
📒 Files selected for processing (14)
core/src/autonomous/lib/budget.tscore/src/autonomous/orchestrator/run.tscore/src/autonomous/prompts/commander.tscore/src/autonomous/prompts/defenses.tscore/src/autonomous/prompts/operator.tscore/src/autonomous/prompts/scout.tscore/src/autonomous/report/html.tscore/src/autonomous/report/mapRunLog.tscore/src/autonomous/report/types.tscore/src/targets/httpClient.tscore/tests/autonomousBudgetGuard.test.tsdocs/hunt.mdrunners/cli/src/commands/hunt.tsrunners/cli/ui/src/components/SetupPage.tsx
Comment on lines
+189
to
+193
| .option( | ||
| "--verify", | ||
| "Force-enable the independent second-model verifier (self_check); on by default when a Claude credential is available" | ||
| ) | ||
| .option("--no-verify", "Disable the verifier even if a Claude credential is available") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
npm ls commander
node --input-type=module <<'NODE'
const { Command } = await import("commander");
for (const args of [
["--verify", "--no-verify"],
["--no-verify", "--verify"],
]) {
const command = new Command();
command.option("--verify").option("--no-verify");
command.parse(["node", "hunt", ...args]);
console.log(`${args.join(" ")} => ${String(command.opts().verify)}`);
}
NODERepository: KeyValueSoftwareSystems/agent-opfor
Length of output: 848
🏁 Script executed:
#!/bin/bash
set -euo pipefail
fd -a 'hunt\.ts$' . | sed 's#^\./##'
file=$(fd 'hunt\.ts$' runners/cli/src/commands | head -n 1 || true)
if [ -n "${file:-}" ]; then
wc -l "$file"
sed -n '160,210p' "$file" | cat -n
sed -n '400,440p' "$file" | cat -n
fi
python3 - <<'PY'
from pathlib import Path
p = next(Path('runners/cli/src/commands').rglob('hunt.ts'))
text = p.read_text()
print("hunt.ts_path:", p)
print("--verify occurrences:", text.count('--verify'))
print("--no-verify occurrences:", text.count('--no-verify'))
print('uses "verify" field:', '.verify' in text or 'verify:' in text)
PYRepository: KeyValueSoftwareSystems/agent-opfor
Length of output: 5346
🏁 Script executed:
#!/bin/bash
set -euo pipefail
node --input-type=module <<'NODE'
const fs = require('node:fs');
const packageJson = JSON.parse(fs.readFileSync('./package.json', 'utf8'));
const lock = fs.existsSync('./package-lock.json') ? JSON.parse(fs.readFileSync('./package-lock.json', 'utf8')) : null;
console.log('package manager:', packageJson.engines?.node || 'node');
console.log('runner cli script:', packageJson.scripts?.start || packageJson.scripts?.dev || 'none');
console.log('lock commander versions:', lock && lock.packages ? Object.fromEntries(
Object.entries(lock.packages)
.filter(([k,v]) => (/node_modules\/commander$/.test(k) || /@keyvaluesystems\/agent-opfor-cli/.test(k)))
.map(([k,v]) => [k.replace('node_modules/', ''), v.version])
) : null);
NODERepository: KeyValueSoftwareSystems/agent-opfor
Length of output: 776
🏁 Script executed:
#!/bin/bash
set -euo pipefail
node --input-type=module <<'NODE'
import fs from 'node:fs';
const packageJson = JSON.parse(fs.readFileSync('./package.json', 'utf8'));
console.log('package engine:', packageJson.engines?.node || 'node');
console.log('cli scripts:', JSON.stringify(packageJson.scripts, null, 2));
NODE
node --input-type=module <<'NODE'
const { Command } = await import("commander");
for (const args of [
["--verify", "--no-verify"],
["--no-verify", "--verify"],
["--verify", "--no-verify", "--verify"],
["--no-verify", "--verify", "--no-verify"],
]) {
const command = new Command();
command.option("--verify").option("--no-verify");
command.parse(["node", "hunt", ...args]);
console.log(`${args.join(" ")} => ${JSON.stringify(command.opts())}`);
}
NODERepository: KeyValueSoftwareSystems/agent-opfor
Length of output: 1897
Do not allow both verification flags.
--verify and --no-verify map to the same Commander verify option, so the last one wins. Make the order affect huntOptions.verify and makes verification enable/disable inconsistently. Track the flags separately or reject both options if they are present together before creating huntOptions.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@runners/cli/src/commands/hunt.ts` around lines 189 - 193, Prevent
simultaneous use of the --verify and --no-verify options in the hunt command:
track their presence independently or detect both flags before constructing
huntOptions, then reject the invocation with a clear error. Ensure
huntOptions.verify is derived only after this validation so option order cannot
determine the verification behavior.
Comment on lines
+424
to
+425
| // On by default when a credential is available; `--no-verify` forces it off. | ||
| verify: opts.verify === false ? false : opts.verify === true || verifierAuthAvailable, |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Forward explicit verification flags through setup mode.
The --ui setup branch returns before huntOptions is built. Its initialConfig at Lines 231-245 does not include opts.verify. Therefore, --no-verify can be lost and the effect in runners/cli/ui/src/components/SetupPage.tsx, Lines 144-151, can enable verification when credentials are detected. --verify also cannot force the checkbox on when no credential is detected.
Pass verify: opts.verify through initialConfig. Mark a supplied value as explicit before the automatic default effect runs. Add tests for both flags in setup mode.
Proposed fix
const initialConfig = {
endpoint: opts.endpoint,
model: opts.targetModel,
targetName: opts.name,
objective: opts.objective,
apiKeyEnv: opts.targetKeyEnv,
commanderModel: opts.model,
operatorModel: opts.operatorModel,
scoutModel: opts.scoutModel,
maxOperators: opts.maxOperators,
maxTurns: opts.maxTurns,
maxThreadTurns: opts.maxThreadTurns,
budgetUsd: opts.budgetUsd,
+ verify: opts.verify,
};🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@runners/cli/src/commands/hunt.ts` around lines 424 - 425, Update the --ui
setup path’s initialConfig construction to pass opts.verify, preserving both
explicit true and false values before SetupPage’s automatic credential-based
defaulting runs. Ensure setup mode honors --verify without credentials and
--no-verify despite available credentials, and add coverage for both flags in
the setup-mode tests.
Comment on lines
+144
to
+151
| // Mirrors the CLI's --verify default (on when a credential is available). Reuses the | ||
| // brainAuth fetch above instead of a second, narrower check; never overrides a manual choice. | ||
| useEffect(() => { | ||
| if (brainAuth.method && !verifyTouchedRef.current) { | ||
| setConfig((prev) => (verifyTouchedRef.current ? prev : { ...prev, verify: true })); | ||
| } | ||
| }, [brainAuth.method]); | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files:"
git ls-files | rg 'SetupPage\.tsx$|brain-auth|BrainAuth|setup' || true
echo
echo "Target file outline/size:"
wc -l runners/cli/ui/src/components/SetupPage.tsx
ast-grep outline runners/cli/ui/src/components/SetupPage.tsx --view compact || true
echo
echo "Relevant SetupPage lines 120-170:"
sed -n '120,170p' runners/cli/ui/src/components/SetupPage.tsx | nl -v120
echo
echo "Search BrainAuth/type definitions and zod imports:"
rg -n "type BrainAuth|interface BrainAuth|BrainAuth|brain-auth|\(/api/brain-auth\)|zod|Zod" runners -g '*.ts' -g '*.tsx'Repository: KeyValueSoftwareSystems/agent-opfor
Length of output: 1057
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "SetupPage lines 120-170:"
sed -n '120,170p' runners/cli/ui/src/components/SetupPage.tsx | awk '{print NR+119 ": " $0}'
echo
echo "BrainAuth and brain-auth symbols:"
git ls-files '*.ts' '*.tsx' '*.js' '*.jsx' '*.mjs' | xargs rg -n "interface BrainAuth|type BrainAuth|BrainAuth|brain-auth|zod|z\\.|Zod" || trueRepository: KeyValueSoftwareSystems/agent-opfor
Length of output: 34712
Validate the /api/brain-auth response before trusting method.
res.json() is parsed directly as BrainAuth, then brainAuth.method can enable verification. Use a Zod schema for the API response and update state only after parsing succeeds.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@runners/cli/ui/src/components/SetupPage.tsx` around lines 144 - 151, Validate
the `/api/brain-auth` payload with a Zod schema before `brainAuth.method` is
used by the verification `useEffect`. Parse the JSON response through that
schema, retain only successfully validated data, and update state only when the
validated response contains a method; preserve the existing manual-choice guard.
Source: Coding guidelines
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Running
opfor huntend-to-end against a real target surfaced several independent reliability gaps in the autonomous engine:BudgetGuard.recordCost()only corrected the cost estimate upward. An inflated token-based estimate could never self-correct downward, soisOverBudget()could falsely truncate a run nowhere near its real spend.tools: []on the commander's SDK options (tried previously to shrink its tool list) also silently dropped the built-in Task/Agent dispatch tool, leaving the commander unable to spawn operators.httpSend()failed immediately on any 5xx / network / timeout error, with no retry for what are often transient blips.--verifydefault and the setup UI's auto-detect only checkedANTHROPIC_API_KEY, missing OAuth token, gateway, and Claude subscription auth — so subscription-authenticated runs silently ran without the second-model verifier.truncatedwhen there was zero activity at all. A run with real attack activity but no confirmedsubmit_report(e.g. a tool-transport outage mid-run) was misreported as a clean, complete run and skipped the better forced-synthesis path.Solution
recordCost()now corrects the estimate in both directions.DISPATCH_TOOLSallowlist instead oftools: [].TOOL_SEARCH_FALLBACKprompt note (commander/operator/scout) telling the agent to resolve viaToolSearchand retry before concluding an outage.httpSend()retries 5xx/network/timeout failures up to 2x with backoff, logging a warning about possible duplicate side effects on stateful targets.resolveBrainAuth()helper (covers all 4 valid credential paths) instead of a narrow env var check.submit_report, regardless of activity level.Changes
core/src/autonomous/lib/budget.ts— bidirectional cost correctioncore/src/autonomous/orchestrator/run.ts— dispatch-tool fix, truncation-detection fixcore/src/autonomous/report/mapRunLog.ts,report/types.ts,report/html.ts— dispatch prompt in decision log, truncation reason in fallback narrativecore/src/autonomous/prompts/{commander,operator,scout,defenses}.ts—TOOL_SEARCH_FALLBACKnotecore/src/targets/httpClient.ts— transient retry with backoff + warning loggingrunners/cli/src/commands/hunt.ts,runners/cli/ui/src/components/SetupPage.tsx— verify default viaresolveBrainAuth()core/tests/autonomousBudgetGuard.test.ts— new regression testdocs/hunt.md— Retries + Verification sectionsIssue
N/A
How to test
core/tests/autonomousBudgetGuard.test.tscovers the budget-correction fix directly.truncated/verifier status, no regressions in typecheck/lint/test/build.Screenshots
N/A — backend/CLI reliability fixes, no UI changes beyond a hint-text update in the setup form.
Summary by CodeRabbit
New Features
opfor huntwhen supported credentials are available, with options to enable or disable it.Improvements
Bug Fixes