diff --git a/.cargo/config.toml b/.cargo/config.toml new file mode 100644 index 000000000..603fb7e0e --- /dev/null +++ b/.cargo/config.toml @@ -0,0 +1,9 @@ +# Use mold for linking on x86_64 Linux (much faster than default ld). +# The cross-rs image's pre-build in Cross.toml installs mold and drops a +# `ld` symlink at /opt/mold-shim/ld → mold. We pass `-B/opt/mold-shim` so +# gcc picks that up as its linker without needing gcc ≥12 (which is when +# `-fuse-ld=mold` support landed; the cross image ships gcc 9). Host macOS +# builds are unaffected — the config is target-scoped, and macOS builds +# target aarch64/x86_64-apple-darwin. Devs can override in ~/.cargo/config.toml. +[target.x86_64-unknown-linux-gnu] +rustflags = ["-C", "link-arg=-B/opt/mold-shim"] diff --git a/.claude/agents/ares-operator.md b/.claude/agents/ares-operator.md index 77741b9d9..a23c8f721 100644 --- a/.claude/agents/ares-operator.md +++ b/.claude/agents/ares-operator.md @@ -1,12 +1,22 @@ --- name: ares-operator -description: Operates the Ares distributed red/blue team system. Use when asked to deploy code, run operations, monitor progress, debug stuck operations, check loot, generate reports, or manage infrastructure across K8s and EC2. +description: Operates the Ares distributed red/blue team system. Use for multi-step Ares workflows — launching/monitoring/debugging operations, deploying code, injecting state, generating reports. DO NOT use for one-shot kubectl/task commands the parent can run inline (e.g., `kubectl rollout restart`, `kubectl get pods`, `task ec2:status`); dispatching a subagent for these adds latency without value. Spawn this agent only when the work needs ≥3 dependent commands or domain knowledge of Ares-specific flags. tools: Bash, Read, Grep, Glob model: opus --- You operate a distributed multi-agent penetration testing system called Ares. The system runs on remote infrastructure (K8s cluster or EC2 instance) — you drive it from the local machine via `ares-cli` or Taskfile commands. +## Scope: when NOT to use this agent + +The parent should handle these inline, not delegate to you: + +- Single kubectl commands (`get pods`, `rollout restart`, `logs`, `describe`). +- Single task commands the user already named (`task rust:build`, `task ec2:status`). +- One-shot reads of status/loot/queue that don't require follow-up reasoning. + +Delegation is only worth the overhead when the work is multi-step, requires Ares-specific flags the parent doesn't know, or involves interpreting state across commands. + ## Architecture ``` @@ -130,6 +140,133 @@ ares-cli --k8s ares-red ops kill --all # Kill all running ops ares-cli --k8s ares-red ops cleanup --max-age-hours 24 # Delete old checkpoints ``` +## Red Team Operations (Proxmox) + +A third deployment target for the GOAD Ludus range: a single attack-box VM +(`attacker-1`, VMID 200) on the `proxmox` SSH alias that runs ares in +standalone mode (`ARES_TOOL_DISPATCH=local`, no worker StatefulSets, local +Redis/NATS). Reachable only through the proxmox jump host (DHCP-assigned +IP on `vmbr1001` VLAN 10). All operator commands live under the `proxmox:` +task namespace and resolve the current attacker IP automatically each run. + +### Submit + dispatcher healthcheck + +```bash +task proxmox:submit # uses DEFAULT_IPS/DOMAIN/MODEL +task proxmox:submit IPS=10.1.10.10,10.1.10.11 DOMAIN=... +``` + +`proxmox:submit` waits up to ~15s after the CLI returns and confirms the +dispatcher actually wrote `Starting operation: ` to `/var/log/ares/dispatch.log` +before exiting (PR #58). If the SUCCESS line doesn't print, the wrapper +warns to run `task proxmox:deploy:restart` — the dispatcher silently +wedging is a known symptom of stale orchestrator state and the submit +healthcheck is the first place it surfaces. + +### Watch progress every minute (with wedge detection) + +`Monitor` against a polling script is the right pattern; emit one line per +minute showing the deltas an operator would scan for. When tokens flatline +for ≥2 ticks while `status=running`, that's the same orchestrator wedge +PR #66 partially addressed — fall through to `task proxmox:logs` to +identify which subsystem stalled. + +```bash +# Inline shell to feed into Monitor (persistent, ~1h timeout): +prev_tokens=""; frozen_ticks=0; while true; do + out=$(task proxmox:runtime 2>&1) + op=$(echo "$out" | grep -oE 'op-[0-9]{8}-[0-9]{6}' | head -1) + op_status=$(echo "$out" | grep -oE 'Status:[[:space:]]+\S+' | awk '{print $2}') + runtime=$(echo "$out" | grep -oE 'Runtime:.*' | sed 's/.*Runtime:[[:space:]]*//' | head -1) + creds=$(echo "$out" | grep -oE 'Credentials: [0-9]+' | awk '{print $2}') + hashes=$(echo "$out" | grep -oE 'Hashes: [0-9]+' | awk '{print $2}') + vulns=$(echo "$out" | grep -oE '[0-9]+ discovered, [0-9]+ exploited') + domains=$(echo "$out" | grep -oE 'Domains \([0-9]+/[0-9]+ compromised' | grep -oE '[0-9]+/[0-9]+') + tokens=$(echo "$out" | grep -oE 'Tokens: [0-9,]+' | tr -d ',' | awk '{print $2}') + cost=$(echo "$out" | grep -oE 'Cost:[[:space:]]+\$[0-9.]+' | grep -oE '\$[0-9.]+') + ts=$(date -u +%H:%M:%SZ); flag="" + if [ -n "$prev_tokens" ] && [ "$tokens" = "$prev_tokens" ] && [ "$op_status" = "running" ]; then + frozen_ticks=$((frozen_ticks + 1)) + flag=" ⚠️ TOKENS FROZEN ${frozen_ticks}m" + else + frozen_ticks=0 + fi + echo "$ts $op rt=$runtime doms=$domains c=$creds h=$hashes v=$vulns tokens=$tokens $cost status=$op_status$flag" + if [ "$frozen_ticks" -ge 2 ]; then + echo "=== wedge dig: last 30 WARN/ERROR lines from dispatch ===" + task proxmox:logs LINES=200 FILTER='WARN|ERROR|FATAL|Stale task|stale eviction' 2>&1 | tail -30 + echo "=== orchestrator outbound HTTPS connection count ===" + task proxmox:exec CMD='ORCH=$(pgrep -f "ares orchestrator" | head -1); echo orch_pid=$ORCH; sudo ss -tnp 2>/dev/null | grep "pid=$ORCH" | grep -v 127.0.0.1 | wc -l' 2>&1 | tail -5 + echo "=== end wedge dig ===" + frozen_ticks=0 + fi + prev_tokens=$tokens + if [ "$op_status" = "completed" ] || [ "$op_status" = "stopped" ]; then + echo "$ts Op finished ($op_status) — stopping monitor"; break + fi + sleep 60 +done +``` + +Note: `status` is read-only in zsh — use `op_status`. Tasks run via +the `Bash` tool inherit a zsh environment. + +### Debugging a stuck op via `task proxmox:logs` + +`proxmox:logs LINES= FILTER=` tails the orchestrator dispatch +log over SSH and strips ANSI for clean grepping. Useful filters when the +1-min monitor flags a freeze: + +```bash +# What was the last thing that actually completed? +task proxmox:logs LINES=500 FILTER='Task completed via LLM' + +# Are auto-planner tasks being deferred while no LLM call runs? +# (worker-slot leak symptom — pre-PR-66 binaries; verify with the HTTPS conn count) +task proxmox:logs LINES=300 FILTER='Task deferred|throttler' + +# Trust-follow / cross-forest forge progress +task proxmox:logs LINES=300 FILTER='Cross-forest forge|raise_child|Cleared stale trust_follow|forge_inter_realm' + +# Cracker (remote crackd) +task proxmox:logs LINES=200 FILTER='crackd|Cracked password|crack_with_hashcat' + +# Domain admin / golden ticket events +task proxmox:logs LINES=500 FILTER='discovery.domain_admin|tool.generate_golden_ticket|Forest trust escalation' + +# Anything explicitly fatal +task proxmox:logs LINES=500 FILTER='FATAL|panic|Traceback|RUST_BACKTRACE|thread .* panicked' +``` + +Cross-reference with the orchestrator's outbound HTTPS connection count +(via `proxmox:exec` + `ss -tnp` filtered to the orch PID): zero open OpenAI +connections while `status=running` is the canonical wedge signature. + +### Known wedge patterns + first-pass remedies + +| Symptom | Likely cause | Fix | +| --- | --- | --- | +| Tokens frozen, 0 OpenAI conns, `llm_count>0`, only `Task deferred` lines | Worker-slot leak (pre-#66) | `task proxmox:deploy:restart` | +| Op submits but `Starting operation:` never logged | Dispatcher wedge | `task proxmox:deploy:restart` then re-submit | +| `crackd backend error: failed to GET /jobs/{id}` repeatedly | Idle-keepalive race vs uvicorn (pre-#64 client; bump server `--timeout-keep-alive` if pre-deploy) | Rebuild from main; verify `pool_idle_timeout` in `ares-tools/src/cracker/remote.rs::http_client` | +| `Child-to-parent forge dispatched` count is 0 but child trust hash + DCs are in state | `auto_trust_follow` dedup leak (pre-#64) | Rebuild from main; the staleness sweep clears stuck `trust_follow:*` marks every 30s tick | +| Cross-forest forge dispatched but no target krbtgt | Not a bug. SID filtering on the receiving DC strips the injected claim regardless of RID — the forge has never taken a second forest | Expect the foreign forest to fall to a native escalation instead (ADCS ESC13 above all, then MSSQL linked servers, AS-REP roasting, foreign security principals). Chase why no forest-native credential was acquired, not why the forge failed | +| Op marks `completed` at N/M domains with N essos.local +Trust: sevenkingdoms.local <──parent/child──> north.sevenkingdoms.local +``` + +Stock GOAD subnet is `192.168.56.0/24`. **DreadGOAD AWS deployments use per-environment VPC CIDRs** (`dev=10.0.0.0/16`, `staging=10.1.0.0/16`, `prod=10.2.0.0/16`, `test=10.8.0.0/16`); resolve actual IPs from the active environment's inventory, not the stock IPs in the docs. + +## High-Value Accounts (memorize these) + +These are the bootstrap credentials — the ones an operator most often needs to recall: + +| Account | Domain | Password | Why it matters | +|---|---|---|---| +| `samwell.tarly` | north | `Heartsbane` | Plaintext in description; MSSQL impersonate `sa` on castelblack | +| `hodor` | north | `hodor` | username==password (spray hit) | +| `brandon.stark` | north | `iseedeadpeople` | AS-REP roastable; MSSQL impersonate `jon.snow` on castelblack | +| `jon.snow` | north | `iknownothing` | Kerberoastable; **MSSQL sysadmin on castelblack** (linked-server pivot) | +| `robb.stark` | north | `sexywolfy` | Local admin on winterfell; rockyou-crackable NetNTLMv2 via Responder (scheduled task every 1m) | +| `eddard.stark` | north | `FightP3aceAndHonor!` | Domain Admin (north); NTLM-relay target via 5m scheduled task on kingslanding | +| `arya.stark` | north | `Needle` | MSSQL impersonate `dbo` on castelblack | +| `sansa.stark` | north | `345ertdfg` | SPN HTTP/eyrie (Kerberoast); unconstrained delegation | +| `jeor.mormont` | north | `_L0ngCl@w_` | Local admin on castelblack | +| `sql_svc` | north/essos | `YouWillNotKerboroast1ngMeeeeee` | MSSQLSvc SPN on both castelblack and braavos | +| `khal.drogo` | essos | `horse` | Local admin on braavos; **MSSQL sysadmin on braavos**; GenericAll on viserys/ESC4 template | +| `jorah.mormont` | essos | `H0nnor!` | LAPS reader; MSSQL impersonate `sa` on braavos | +| `missandei` | essos | `fr3edom` | GenericAll on `khal.drogo` | +| `daenerys.targaryen` | essos | `BurnThemAll!` | Domain Admin (essos); cross-forest member of `AcrossTheNarrowSea` and `DragonsFriends` | +| `lord.varys` | sevenkingdoms | `_W1sper_$` | GenericAll on `Domain Admins` (sevenkingdoms) | +| `tyron.lannister` | sevenkingdoms | `Alc00L&S3x` | Cross-forest member of essos `DragonsFriends` (LAPS reader) | + +MSSQL `sa` passwords: `Sup1_sa_P@ssw0rd!` (castelblack), `sa_P@ssw0rd!Ess0s` (braavos). + +## Canonical Killchains + +When the operator describes a state, map it to one of these chains and tell them the *next* step: + +### 1. Cold-start → Domain Admin (north) + +``` +Responder (1m wait) → robb.stark NetNTLMv2 → hashcat (rockyou) → robb.stark:sexywolfy + → local admin on winterfell → secretsdump → eddard.stark NT hash → DCSync north +``` + +Or, in parallel: + +``` +GetNPUsers → brandon.stark AS-REP → crack → iseedeadpeople + → MSSQL impersonate jon.snow on castelblack → xp_cmdshell as sql_svc → SeImpersonate → SweetPotato → SYSTEM +``` + +### 2. Cold-start → Domain Admin (sevenkingdoms) + +NTLM-relay: kingslanding runs scheduled task as `eddard.stark` (DA) every 5m → relay to unsigned SMB (winterfell, castelblack, braavos): + +``` +Responder + ntlmrelayx -t winterfell --smb2support → wait ≤5m → eddard.stark relayed → secretsdump +``` + +### 3. ACL killchain (sevenkingdoms — the "tywin chain") + +``` +tywin → ForceChangePassword → jaime → GenericWrite → joffrey → WriteDacl → tyron + → AddSelf → Small Council → AddMember → DragonStone → WriteOwner → KingsGuard + → GenericAll → stannis → GenericAll → kingslanding$ (DC01) → RBCD → DA +``` + +Shortcut edge: `lord.varys --GenericAll--> Domain Admins` (single-step DA if you have varys). `AcrossTheNarrowSea --GenericAll--> kingslanding$` (one-step DC compromise from cross-forest essos members). + +### 4. ACL killchain (essos) + +``` +missandei --GenericAll--> khal.drogo --GenericAll--> viserys.targaryen +khal.drogo --GenericAll--> ESC4 cert template → modify → ESC1 → DA cert → certipy auth +DragonsFriends --GenericWrite--> braavos$ (SRV03) → RBCD +``` + +### 5. MSSQL pivot (north → essos via linked server) + +``` +jon.snow on castelblack → linked → sa on braavos (password sa_P@ssw0rd!Ess0s) + → xp_cmdshell on braavos → SeImpersonate → SYSTEM → DCSync essos? not yet — need DA +``` + +And in reverse: + +``` +khal.drogo → sysadmin on braavos → linked → sa on castelblack (Sup1_sa_P@ssw0rd!) +``` + +### 6. Child → Parent (north → sevenkingdoms) + +- **Golden ticket + ExtraSid:** DCSync north → forge ticket with `extra-sid=-519` (Enterprise Admins) → DCSync sevenkingdoms. +- **Trust ticket:** extract trust key (`secretsdump` for the trust account) → forge inter-realm TGT for `krbtgt/sevenkingdoms.local`. +- **raiseChild.py** — single command, does both. + +### 7. Forest hop (sevenkingdoms ↔ essos) + +- Bidirectional trust + cross-forest group memberships: + - `tyron.lannister` ∈ essos `DragonsFriends` (LAPS reader on essos) + - `daenerys.targaryen` ∈ sevenkingdoms `AcrossTheNarrowSea` (GenericAll on kingslanding$) +- Compromise tyron → read essos LAPS → local admin on braavos → DCSync essos. +- Compromise daenerys → AcrossTheNarrowSea → DA on sevenkingdoms. + +### 8. ADCS paths + +- **ESC1** templates exist (vulnerable enrollee-supplies-subject) — `certipy find -vulnerable` first. +- **ESC4:** `khal.drogo` has GenericAll on a template → modify → ESC1. +- **ESC8:** ADCS web enrollment on braavos → coerce DC (PetitPotam) → relay to `/certsrv/certfnsh.asp --adcs` → DC certificate → DA. +- **ESC6/7/9/10/11/13/14/15:** see comprehensive doc; meereen runs ADCS *custom templates* role specifically for these. + +## How to Answer Questions + +1. **Always anchor in the docs.** When asked "what's $user's password?" or "what does $user unlock?", read `domains-and-users.md` directly — passwords change in variants, and approximations get the operator stuck. +2. **Trace the full path.** When the operator gives you a state ("I have `samwell.tarly`"), output: (a) what they can do *now*, (b) the highest-value pivot, (c) the next step's exact command. +3. **Give exact tool invocations** with the right domain, DC IP placeholder, and impacket caveats. Prefer impacket/certipy/cme/ntlmrelayx commands the operator can paste. +4. **Resolve IPs lazily.** Don't hardcode `192.168.56.x` — ask which env (`dev`/`staging`/`prod`/`test`), or have the operator pull from inventory. The DreadGOAD CIDRs differ per env. +5. **Surface the impacket Kerberos gotchas** when relevant — they are documented in `/Users/l/dreadnode/ares/.claude/CLAUDE.md` and bite *every* cross-realm chain: + - Cross-realm referral broken (#315): forge inter-realm TGT, present to target DC directly + - `-just-dc-user` accepts only one account — chain `secretsdump` calls with `;` + - Target string domain prefix must match TGT realm + - No ccache persistence across `run_tool` calls — chain `ticketer && secretsdump` in one bash +6. **Variant awareness.** If the operator mentions `variant: true`, do not trust the stock GOAD names — read `ad/GOAD-variant-1/data/config.json` (or wherever `variant_target` points). The structure is graph-isomorphic; the *names* are randomized. +7. **When something doesn't work, suspect the lab first.** GOAD vulns are provisioned by Ansible roles (`roles/vulns/*`). If `responder` isn't catching robb.stark, the `responder` vuln role may have failed to provision the scheduled task — check `dreadgoad validate --quick` and the `roles/vulns/responder` task list. +8. **Be precise.** Cite exact file/line when the operator wants verification: e.g., `domains-and-users.md:108-117` for the north users table. + +## What This Agent Will *Not* Do + +- Will not invent credentials/SPNs/templates not present in the docs. If a name isn't in `domains-and-users.md` or the variant config, say so. +- Will not advise on real-world targets. This is a lab operator's assistant — every fact here applies only to the GOAD lab. +- Will not run code or modify the ares codebase. Read-only research and operational advice. + +## Important Repo Convention (when touching ares code) + +The ares repo's CLAUDE.md mandates that **GOAD names are banned in repo code, tests, comments, and templates** — they leak into LLM tool calls and create phantom entries in dreadgoad's scoreboard. Allowed in: `.taskfiles/*.yaml`, root `Taskfile.yaml`, `docs/goad-checklist.md`, `config/ares.yaml`. Use `contoso.local` / `fabrikam.local` / `192.168.58.x` / role-based hostnames (`dc01`/`dc02`/`sql01`/etc.) for *test fixtures and code*. This agent is allowed to discuss GOAD names freely (it is operational advice, not committed code) — but if asked to *write code*, switch to the contoso/fabrikam conventions. diff --git a/.claude/skills/ares-debug/SKILL.md b/.claude/skills/ares-debug/SKILL.md new file mode 100644 index 000000000..c1f127e76 --- /dev/null +++ b/.claude/skills/ares-debug/SKILL.md @@ -0,0 +1,469 @@ +--- +name: ares-debug +description: Diagnose a stuck, slow, or broken Ares operation by triangulating across three data sources — SSM (live ares logs + Redis on EC2), Grafana Loki (historical logs via mcp__grafana__query_loki_logs), and OTEL traces in Tempo. Use when an operation is hung/wedged, a worker keeps crashing, the orchestrator stops making progress, or a task fails with no obvious local clue. Default deployment is EC2 (`kali-ares`); K8s notes included for completeness. +--- + +# Debugging Ares + +You are debugging a running or recent Ares operation. Pick the cheapest source first; only escalate if it doesn't answer the question. + +## Read this before you do anything + +**Do not declare an op healthy from process liveness, NATS/Redis ping, or token-rate alone.** A wedged Ares op happily presents as `status=running`, workers `active`, Redis green, cache hit ≥80%, and tokens climbing — while making zero external progress for hours. This has happened. Don't repeat it. + +**The only valid "healthy" verdict requires a comparison:** + +1. Compare *this op's* objective state now vs. 60s ago — `has_domain_admin`, domain compromise count, hosts owned, creds, hashes, vulns exploited. If none changed, that's churn, not progress. +2. Compare this op to recent ops' baseline. Pull `ares ops list` and look at how long prior ops took to hit DA / 2nd domain. **If this op is more than ~2× slower to a milestone the last 3 ops hit, treat it as wedged regardless of token rate.** + +Token churn is the signature of the LLM re-evaluating the same frozen state every tick; high cache-hit rate (>80%) on a slow op is a *symptom of the wedge*, not evidence of health. + +**Worker per-role log mtimes are not a signal.** In steady state the orchestrator centralizes everything via NATS into `/var/log/ares/orchestrator.log`; per-role files (`recon.log`, `cracker.log`, etc.) stay near-empty. Don't read into stale mtimes. + +## Before you propose a code fix + +Ares timeline events (`evt-exploit-fail-*` in `ares:op:*:timeline`) and "Assistance needed" strings the LLM emits ("the tool schema does not accept X", "current toolset lacks Y", "tool requires password but only hash available") are the failing LLM agent's confabulated explanation of its own failure — **not a bug report**. The agent does not know its own tool schemas or the orchestrator's dispatch layer, and it will invent plausible-sounding gaps that don't exist. + +Before recommending a fix from one: + +1. Open the tool wrapper in `ares-tools/src/**/*.rs` — does the tool actually accept the arg the LLM said was missing? +2. Open the LLM-facing schema in `ares-llm/src/tool_registry/**/*.rs` — does it declare the field? +3. Open the automation dispatcher in `ares-cli/src/orchestrator/automation/*.rs` — does it inject the credential/state from Redis into the payload? + +If all three already do the thing, the LLM was confabulating. The real failure is elsewhere — the tool ran and hit a Kerberos error, dispatch timed out, worker didn't have the credential in state, etc. Grep the orchestrator log for the actual dispatch record + tool stdout/stderr; those are ground truth. Timeline events are not. + +## Tight-loop / wedge signatures (grep the orchestrator tail for these first) + +Run Step 0, then **before drawing any conclusion** grep the tail of `orchestrator.log` for each pattern below. If any hit, that's almost certainly your wedge: + +| Pattern (regex) | Means | +|---|---| +| `clearing dedup for retry` | Wrapper-level retry loop; same task being re-dispatched every tick | +| `Dispatching ... ` repeated ≥3× | Automation hot loop with no backoff | +| `KDC_ERR_TGT_REVOKED\|KDC_ERR_S_PRINCIPAL_UNKNOWN\|KDC_ERR_PREAUTH_FAILED\|TGT has been revoked` | Kerberos error that will not self-heal; orchestrator may be retrying anyway | +| `tool exited with code Some\(0\)` followed by stderr content | Zero-exit-with-error: wrapper treats stderr-on-zero-exit as transient and re-tries | +| Same `task_id` shape (e.g. `trust_raise_child_`) repeated with distinct hex per tick | Dedup key churning instead of blacklisting | +| `Processing real-time discoveries count=1` ticking every 5s with no other state change | Orchestrator stuck in discovery-replay loop | +| `Waiting for blue team to finish\.\.\. active_investigations=[0-9]+` ticking every 10s | **Not a wedge — red is DONE.** Op is holding open until blue investigations drain. Check `red_completed_at` / `red_completion_reason` in meta (see Step 0). | +| `Loki request error \(retryable\)` / `Retrying Loki query after transient failure` flooding the tail | Blue team's external Loki (`$LOKI_URL`) is flapping; blue investigations grind to a crawl and starve out post-red op close. Not a red bug. | +| `Tool binary not found \(spawn failed\) — removing from available tools` firing across many recon tools (nmap_scan, enumerate_users, enumerate_shares, smb_signing_check, username_as_password) in the first seconds of the op | Tool-pruning cascade — a prior spawn failure poisoned the worker's per-process `unavailable_tools` HashSet. Only a genuine worker-process restart clears it — **not** `task ec2:restart`, which never touches `ares@` units (see Step 8). Fix: `task ec2:exec EC2_NAME=kali-ares CMD='systemctl restart "ares@*.service"'`. Full mechanism + confirmation queries in Step 3.5. | + +If you don't see these but the op is slow vs. baseline, escalate to Loki / Tempo for cross-tick LLM latency or tool-call stalls. + +## What goes where + +| Source | Latency | Coverage | How to query | +|-------------------|----------|-------------------------------------------------|----------------------------------------------------------| +| `task ec2:status` | seconds | Worker process state, Redis ping | Bash | +| `task ec2:runtime`| seconds | Per-op token/cost/domain banner | Bash | +| Loki (Grafana) | seconds | Historical `/var/log/ares/*.log` + syslog/auth | `mcp__grafana__query_loki_logs` (datasourceUid `loki`) | +| Tempo (Grafana) | seconds | OTEL traces of LLM calls + tool dispatch | `mcp__grafana__*` Tempo proxy tools | +| SSM `task ec2:exec` | ~5-15s | Anything on the host (redis-cli, journalctl) | Bash, never `tail -f` | +| `task ec2:logs` | streaming| Live tail of one role's log | **DO NOT use in Claude** — it's an interactive SSM session | + +**Rule:** never run `task ec2:logs` from an agent — it opens an interactive SSM session that won't terminate. Always use Loki (preferred) or `task ec2:exec EC2_NAME=kali-ares CMD='tail -n 200 /var/log/ares/.log'`. + +**AWS auth:** use whatever ambient AWS profile has SSM access to the box — do not hard-code one. Ownership of the `kali-ares` instance has moved between profiles/accounts multiple times; a stale prefix (`AWS_PROFILE=personal AWS_REGION=us-east-1`) will produce `No running instance found matching: kali-ares` even when the box is up. Verify resolution with `task ec2:ops EC2_NAME=kali-ares` first; if it fails, try flipping between `lab` and `personal` and between `us-east-1` and `us-west-2`. The command examples below run against the ambient profile — set it explicitly only if the ambient one doesn't resolve the box. + +## Step 0 — mandatory baseline triage (run all in parallel, on every invocation) + +Do not skip any of these. Do not respond to the user with a verdict until you've inspected each output. The point of this step is to make it impossible to declare "healthy" without the evidence. + +```bash +# 0a. Current op id + status +task ec2:ops EC2_NAME=kali-ares LATEST=true + +# 0b. Current op objective state + tokens +task ec2:runtime EC2_NAME=kali-ares LATEST=true + +# 0c. Process / Redis / NATS health +task ec2:status EC2_NAME=kali-ares + +# 0d. The single most important probe — orchestrator tail. Grep it for the wedge signatures listed above. +task ec2:exec EC2_NAME=kali-ares \ + CMD='tail -n 300 /var/log/ares/orchestrator.log' + +# 0e. Historical baseline — last several ops, to compare runtime-to-milestone +ares --ec2 kali-ares ops list | head -20 + +# 0f. Failed tasks for the current op +ares --ec2 kali-ares ops tasks --latest --status failed | head -80 +``` + +Pull `op-YYYYMMDD-HHMMSS` from 0a/0b and use that as `$OP` below. After collecting: + +1. **Is red already done?** Before anything else, check `red_completed_at`, `red_completion_reason`, and `red_blocked_on_blue` in the op's meta. If `red_completed_at` is set, red is NOT wedged — it ended (either by success, `"all forests dominated (post-exploitation complete)"`, or by hitting `"max runtime exceeded"`). The op status will still show `running` because the operation as a whole is holding open for blue investigations to drain; that's the "Waiting for blue team to finish" pattern in the wedge table. Don't misdiagnose an ended-red as wedged. One command: + + ```bash + task ec2:exec EC2_NAME=kali-ares CMD="sudo redis-cli hmget ares:op:$OP:meta red_completed_at red_completion_reason red_blocked_on_blue has_domain_admin has_golden_ticket" + ``` + +2. Grep the 0d output for each pattern in the "Tight-loop / wedge signatures" table. **If any hits ≥3 times, you have your root cause; jump to reporting.** +3. Compare 0b's `Domains compromised` and `Vulns exploited` against the runtime banner of recent ops in 0e. If the prior 3 ops compromised more domains in less time at this point, the current op is regressed regardless of how healthy 0a/0c look. +4. Read 0f — the failure mode of the first 5-10 failed tasks usually points at the role/tool that's flailing. + +Only proceed past Step 0 to deeper probes (Loki, Tempo, SSM journals) if none of the above lands a verdict. + +**Two footguns in the Step 0 commands themselves — read before you file a "Redis broken" bug:** + +- `ares --ec2 kali-ares ops list` (0e/0f) connects to **local** Redis on the machine you're running from, not to the box's Redis over SSM. From an agent host with no `redis-server` and no `ec2:redis:forward` running, it will exit with `Failed to connect to Redis: Connection refused`. That's not "the box is broken" — it's the CLI wanting a live connection. When you see it, fall back to `task ec2:exec EC2_NAME=kali-ares CMD='sudo redis-cli ...'` for anything you'd have asked the CLI for. +- **There is no `ares:op::creds` key.** It is `:credentials`. Any command built on `:creds` returns an empty/zero result that reads exactly like "no credentials found" — the most expensive false negative in this document's history. Verified against `ares-core/src/state/keys.rs` and the writer verbs in `ares-core/src/state/reader.rs`: + + | Key | Writer verb | TYPE | Count with | Dump with | + |---|---|---|---|---| + | `:meta` | `hset` | HASH | `HLEN` | `HGETALL` / `HMGET` | + | `:credentials` | `hset_nx` | HASH | `HLEN` | `HGETALL` | + | `:hashes` | `hset` | HASH | `HLEN` | `HGETALL` | + | `:vulns` | `hset_nx` | HASH | `HLEN` | `HGETALL` | + | `:completed_tasks` | `hset` | HASH | `HLEN` | `HGETALL` | + | `:hosts` | `rpush` | LIST | `LLEN` | `LRANGE k 0 -1` | + | `:users` | `rpush` | LIST | `LLEN` | `LRANGE k 0 -1` | + | `:timeline` | `rpush` | LIST | `LLEN` | `LRANGE k -50 -1` | + + Wrong verb → `WRONGTYPE`, which is loud. Wrong *key name* → `0`, which is silent. When in doubt: `redis-cli type `. + +## Step 1 — fast triage (Loki, last hour) + +Loki has every ares log line shipped from the EC2 box. Datasource UID is `loki`. Logs are JSON; the actual line is in the `message` field, with labels `app="ares"`, `deployment="alpha-operator-range-kali-ares"`, `job=.log`. + +Run these in parallel: + +``` +mcp__grafana__query_loki_logs + datasourceUid: "loki" + logql: '{app="ares", deployment="alpha-operator-range-kali-ares"} |~ "(?i)error|fatal|panic|traceback|RUST_BACKTRACE"' + limit: 30 +``` + +``` +mcp__grafana__query_loki_logs + datasourceUid: "loki" + logql: '{app="ares", deployment="alpha-operator-range-kali-ares", job="orchestrator.log"} |~ "WARN|ERROR"' + limit: 30 +``` + +Narrow by role when you know the suspect: change `job="orchestrator.log"` to one of +`recon.log`, `credential_access.log`, `cracker.log`, `acl.log`, `privesc.log`, `lateral.log`, `coercion.log`. + +Narrow by op id (substring match on the log line): + +``` +logql: '{app="ares", deployment="alpha-operator-range-kali-ares"} |= "op-20260630-201500"' +``` + +Use `query_loki_stats` first when you're guessing the selector — it tells you whether the stream has any entries before you waste a `query_loki_logs` call. + +## Step 2 — failed tasks (operation-level) + +```bash +task red:multi:tasks:list LATEST=true STATUS=failed # K8s +ares --ec2 kali-ares ops tasks --latest --status failed # EC2 +``` + +Failed tasks include the worker's error message and the role that failed. Cross-reference against Loki by role + timestamp. + +## Step 2.5 — attribute a specific tool call to the worker that ran it + +Use this when the question is "did tool X actually run for task Y, on which worker, and did it succeed?" The canonical case is verifying cross-role routing (e.g. `credential_access`-originated `password_spray` / `username_as_password` / `laps_dump` calls must land on a `recon` worker because netexec lives there — see `RECON_ROUTED_TOOLS` in `orchestrator/tool_dispatcher/mod.rs`). + +**Ground truth is the OTel span line each worker emits at INFO level when it starts a tool:** + +``` +Executing tool tool= call_id=_ task_id=_ +``` + +The span attributes on that same line are what you actually want: + +- `agent.role` = the worker that executed the tool. Cross-routing fired if this differs from the role prefix of `task_id`. +- `attack_operation_id` / `op.id` = the op — scope every grep to this to avoid conflating past ops. +- The follow-up line for the same `call_id` carries `Tool execution failed tool= err=` on failure. + +**The three canonical failure strings** and where they come from — memorize these because they distinguish "binary missing" from "tool ran and errored": + +| String | Source | Meaning | +|---|---|---| +| `failed to spawn '' — is it installed?` | `ares-tools/src/executor.rs:219` | ENOENT: the binary isn't on this worker's `$PATH` | +| `failed to spawn impacket-ntlmrelayx (is it installed?)` | `ares-tools/src/coercion.rs:586` | Same, special-cased (no single quote — do not narrow greps to require one) | +| `Tool '' is not installed on this worker.` | `worker/tool_executor.rs::unavailable_tool_response` | Cached unavailability — a prior call ENOENT'd and future calls return this without re-spawning | + +Everything else in `err=...` means the binary ran and the tool logic failed (timeout, KDC error, no creds, etc.). + +**Query patterns.** Prefer Loki when the label narrow is easy; SSM `grep -a` when you need cross-file correlation on the box. + +``` +# Loki: every executor span for tool X in this op +mcp__grafana__query_loki_logs + datasourceUid: "loki" + logql: '{app="ares", deployment="alpha-operator-range-kali-ares"} |= "Executing tool" |= "tool=" |= ""' + limit: 50 +``` + +```bash +# SSM: same thing, plus the failure line for the same call_id +task ec2:exec EC2_NAME=kali-ares \ + CMD='sudo grep -a "" /var/log/ares/recon.log /var/log/ares/credential_access.log | grep -a "tool=" | head -20' + +# End-to-end trace of one call_id across every worker log +task ec2:exec EC2_NAME=kali-ares \ + CMD='sudo grep -a "" /var/log/ares/*.log' + +# Sanity: is the binary the caller expects actually on the box right now? +task ec2:exec EC2_NAME=kali-ares \ + CMD='which netexec; ls -la /usr/local/bin/netexec /usr/bin/netexec 2>/dev/null; netexec --version 2>&1 | head -3' +``` + +**Gotchas (do not skip):** + +1. `task ec2:exec` runs `CMD` through go-task's template engine. `{{ ... }}`, backticks, and some quoting silently fail with `"CMD required"` — that means the template ate the arg, not that `CMD` was empty. Workarounds: bind `Q="…"` locally and pass `CMD="$Q"`; keep single quotes on the outside; avoid `{{`. If you see `"CMD required"`, simplify quoting before assuming the file is empty. +2. Per-role log files are **ANSI-color-coded** on disk. `grep 'tool.name="X"'` returns 0 hits even when the tool ran because the bytes are `tool.name[0m[2m=[0m"X"`. Anchor on invariant plain-text substrings: `Executing tool`, `tool= call_id=`, `err=failed to spawn`, `attack_operation_id=""`. `grep -a` (force text mode) is required — the escapes make grep treat these files as binary and go silent otherwise. +3. Per-role log files stay near-empty in steady state (see the intro's "worker per-role log mtimes are not a signal") — but executor OTel spans DO land there. `recon.log` and `credential_access.log` are the right files for tool-attribution greps even though they look sparse. +4. `ingest.log` is a firehose (multi-GB); do not grep it without a `--max-count` or a very narrow anchor. + +**Case study — was cross-routing broken on op-20260716-181136?** credential_access called `username_as_password`, the runner pruned it after "spawn failed". The trace resolved it in three greps: + +``` +agent.role=recon +task_id=credential_access_de9f5fa0be53 +err=failed to spawn 'netexec' — is it installed? +``` + +`agent.role=recon` proved routing fired (a recon worker picked up a credential_access-originated call — cross-routing correct). The `err=` matched `executor.rs:219` verbatim, pinning the root cause on netexec missing from the box at that moment. Fix was ansible provisioning drift, not code. **Without the span attributes there was no way to distinguish "routing bug" from "environment drift" — every hypothesis based on just the runner's `WARN` line would have been wrong.** + +## Step 3 — wedge detection (objective state frozen) + +**The canonical wedge is NOT "tokens flatlined" — tokens almost always keep climbing during a wedge because the LLM re-evaluates the same frozen state every tick.** The canonical wedge is "objective state frozen while tokens climb." Probe state, not tokens: + +```bash +# Snapshot 1 — verb matches TYPE per the table in Step 0. `credentials` NOT `creds`. +task ec2:exec EC2_NAME=kali-ares \ + CMD='redis-cli hmget "ares:op:'"$OP"':meta" has_domain_admin has_golden_ticket target_ips initialized red_completed_at red_blocked_on_blue; echo ---; for k in credentials hashes vulns completed_tasks; do printf "%s=" "$k"; redis-cli hlen "ares:op:'"$OP"':$k"; done; for k in hosts users timeline; do printf "%s=" "$k"; redis-cli llen "ares:op:'"$OP"':$k"; done' +# wait 60s +# Snapshot 2 — same command. Diff the two. Identical = wedge. +``` + +Cross-check against tokens: pull `ec2:runtime` at both snapshots. **Tokens climbing + state identical = textbook wedge.** Tokens climbing + state changing = healthy. Tokens flatlined + state identical = orchestrator hung (rarer). + +If wedged, two further probes pinpoint where: + +```bash +# Outbound HTTPS from orchestrator — zero connections = LLM API stall +task ec2:exec EC2_NAME=kali-ares CMD='ORCH=$(pgrep -f "ares orchestrator" | head -1); echo "orch_pid=$ORCH"; sudo ss -tnp 2>/dev/null | grep "pid=$ORCH" | grep -v 127.0.0.1 | wc -l' +``` + +``` +# Loki search for retry/throttle/dedup markers in the last 30 minutes +mcp__grafana__query_loki_logs + datasourceUid: "loki" + logql: '{app="ares", deployment="alpha-operator-range-kali-ares", job="orchestrator.log"} |~ "clearing dedup for retry|KDC_ERR_|Task deferred|throttler|stale|wedge"' + limit: 80 +``` + +Remedy depends on root cause: + +- Hot retry loop on a tool (`clearing dedup for retry`) → fix the dedup/blacklist logic in the relevant `automation/auto_*.rs`; in the meantime `task ec2:stop-op ... LATEST=true` to stop the burn. +- LLM API stall → check the model provider's status, then restart the orchestrator with `task ec2:restart EC2_NAME=kali-ares` (that is stop+start of `ares-orchestrator.service` and infra only — it preserves Redis but leaves workers untouched; add `task ec2:exec EC2_NAME=kali-ares CMD='systemctl restart "ares@*.service"'` if the workers are the stalled party). +- State frozen but no signature → escalate to Tempo (Step 7) to find the slow span. + +## Step 3.5 — tool-pruning cascade (recon suddenly does nothing) + +Distinct failure class from "wedge" and "crash." Signature: the LLM issues a normal task, workers stay `active`, but every recon/credential-access tool the LLM tries is immediately marked `Tool binary not found (spawn failed) — removing from available tools` and the LLM burns through its 24-tool list in seconds without any external effect. The op then presents as slow-vs-baseline with 0 creds / 0 hashes / 0 hosts. + +**Grep the LLM runner side for the pattern:** + +```bash +task ec2:exec EC2_NAME=kali-ares CMD="sudo grep -aE 'Tool binary not found \(spawn failed\)' /var/log/ares/orchestrator.log | grep -a '$OP' | grep -oE 'tool=[a-z_]+' | sort | uniq -c | sort -rn" +``` + +If a bunch of nxc/netexec-backed tools (`nmap_scan`, `enumerate_users`, `enumerate_shares`, `smb_signing_check`, `check_rdp_reachability`, `check_winrm_reachability`, `username_as_password`, `smb_sweep`) all show up, that's the cascade. + +**Mechanism** (three separate files): + +1. `ares-tools/src/executor.rs:219` — real spawn failure emits `failed to spawn '' — is it installed?`. +2. `ares-cli/src/worker/tool_executor.rs:332-333` (`is_tool_unavailable_error`) — classifies that string as "unavailable" and inserts the tool name into a **per-process `unavailable_tools: HashSet`**. Every subsequent call to that tool on that worker skips the spawn entirely and returns the cached `"Tool 'X' is not installed on this worker. Do not call this tool again — it failed to spawn previously."` response (`tool_executor.rs:318-328`). +3. `ares-llm/src/agent_loop/runner.rs:60` — `dispatch_one` flattens the worker's `error` field into `output` (`"Error: {err}\n\nPartial output:\n{output}"`), then `runner.rs:532` detects `output.contains("failed to spawn")` and yanks the tool from the LLM's active list for the rest of this task, plus injects a `[SYSTEM]` message telling the LLM to stop trying. + +The trap: **one transient spawn failure poisons the tool for the worker's lifetime** — no TTL, no re-probe. Runs whose spawn genuinely failed (a mid-deploy race, an apt lock, an ephemeral cgroup hiccup) leave dead tool entries that persist across every subsequent op the same worker handles. + +**Deploys restart workers only if their units are already `active`** (`.taskfiles/ec2/Taskfile.yaml:255-257`), so `task ec2:deploy` usually clears the poison — but silently skips any worker whose unit is inactive, printing `no ares@ worker units active — skipping restart`. When that happens the worker keeps its poisoned `unavailable_tools` set and `/proc//exe` points at the pre-deploy inode with `(deleted)` on it (see the Step 8 deploy note). + +**Confirmation & fix:** + +```bash +# Check worker uptime — anything > a few hours across multiple ops is suspicious +task ec2:exec EC2_NAME=kali-ares CMD='systemctl show ares@recon.service -p ActiveEnterTimestamp,MainPID; ps -o pid,etime,cmd -C ares | head -10' + +# Verify the binaries actually work from the shell (rules out "genuinely uninstalled") +task ec2:exec EC2_NAME=kali-ares CMD='which netexec nxc nmap; nxc --version 2>&1 | head -1; nmap --version 2>&1 | head -1' + +# If binaries work but pruning still fires → bounce the WORKER units (keeps Redis). +# `task ec2:restart` will NOT do this — it never touches ares@ units. +task ec2:exec EC2_NAME=kali-ares CMD='systemctl restart "ares@*.service"; systemctl is-active "ares@*.service" | sort | uniq -c' +``` + +If the pruning cascade repeats on the very next op with **fresh** workers, the spawn failure is reproducible — probe from inside the worker's cgroup for AppArmor denials, broken Python venvs (nxc/netexec is a pipx shim; `python3 -c 'from nxc.netexec import main'` is a direct test), or `system-ares.slice` restrictions. + +Note: `sprayhound`-backed tools (`password_spray`, `asrep_roast`) use a different binary and are unaffected — seeing those still `Executing tool` in the recon.log while nxc-backed tools are pruned is the signature that isolates this to the netexec side. + +## Step 4 — worker crash loop + +A specific role keeps respawning. Check systemd journal via SSM: + +```bash +task ec2:exec EC2_NAME=kali-ares CMD='systemctl status ares@recon --no-pager | head -30' +task ec2:exec EC2_NAME=kali-ares CMD='journalctl -u ares@recon -n 100 --no-pager' +``` + +(Substitute `recon` with the failing role: `credential_access`, `cracker`, `acl`, `privesc`, `lateral`, `coercion`.) + +If OOM-killed, check the cgroup: + +```bash +task ec2:exec EC2_NAME=kali-ares CMD='dmesg -T | grep -iE "killed process|oom" | tail -20' +``` + +The system-ares.slice caps memory at 12G global, ~2G per worker (see `.taskfiles/ec2/scripts/setup.sh:160`). Worker OOM = a tool process (netexec, hashcat, etc.) blew up inside the worker's cgroup. + +## Step 5 — Redis state introspection + +```bash +task ec2:exec EC2_NAME=kali-ares CMD='redis-cli ping' +task ec2:exec EC2_NAME=kali-ares CMD='redis-cli info keyspace' +task ec2:exec EC2_NAME=kali-ares CMD='redis-cli keys "ares:operation:*" | head -20' +task ec2:exec EC2_NAME=kali-ares CMD='redis-cli get ares:operation:active' +task ec2:exec EC2_NAME=kali-ares CMD='redis-cli hgetall "ares:op:'"$OP"':meta"' +``` + +For loot or shared state, prefer the typed CLI over raw Redis: + +```bash +task ec2:loot EC2_NAME=kali-ares LATEST=true # users, creds, hashes, hosts +task ec2:loot EC2_NAME=kali-ares LATEST=true DIFF=true # only what changed since last call +``` + +To run blue-team queries or arbitrary `ares` commands against EC2 Redis locally, port-forward: + +```bash +task ec2:redis:forward EC2_NAME=kali-ares # blocks in foreground — DO NOT run from an agent +``` + +If you need local access from an agent, use `ec2:exec` with `redis-cli` instead. + +## Step 6 — NATS broker + +NATS is the task/RPC broker. If workers are alive but no tasks dispatch: + +```bash +task ec2:exec EC2_NAME=kali-ares CMD='curl -s http://127.0.0.1:8222/varz | jq ".connections, .in_msgs, .out_msgs, .slow_consumers"' +task ec2:exec EC2_NAME=kali-ares CMD='curl -s http://127.0.0.1:8222/connz | jq ".num_connections, [.connections[].name]"' +task ec2:exec EC2_NAME=kali-ares CMD='systemctl status nats-server --no-pager | head -15' +``` + +## Step 7 — OTEL traces (LLM + tool call timing) + +OTEL traces ship to Tempo with `service.name=ares-orchestrator|ares--agent` and `deployment.environment=staging`, `attack.team=red`. Use the Grafana Tempo proxy tools — search by `service.name` and op id (op id is set as a span attribute by the orchestrator). + +Useful when: + +- You want to see the LLM call latency that's stalling a tick +- A specific tool call is silent in logs but you want to confirm it ran +- You need to attribute time spent across roles for a long-running op + +If Tempo search returns nothing, the orchestrator may not be exporting — verify with: + +```bash +task ec2:exec EC2_NAME=kali-ares CMD='grep OTEL_EXPORTER /etc/ares/env' +``` + +## Step 8 — verify deploy state (binary mismatch) + +A common false positive: the local CLI and the EC2 binary diverge. + +```bash +ares --version # local +task ec2:exec EC2_NAME=kali-ares CMD='/usr/local/bin/ares --version' # remote +task ec2:exec EC2_NAME=kali-ares CMD='stat -c "%y %s" /usr/local/bin/ares' # mtime + size +``` + +If you just landed code, re-deploy before continuing to debug. Canonical "upload updated code, then run a fresh op against dreadgoad" one-liner (Apple-Silicon-safe — `DOCKER_DEFAULT_PLATFORM` forces an x86 build, the S3 bucket is the alpha-operator-range artifact store): + +```bash +DOCKER_DEFAULT_PLATFORM=linux/amd64 task -y ec2:deploy EC2_NAME=kali-ares S3_BUCKET=dread-infra-alpha-operator-range-prod-us-east-1 \ + && task -y red:ec2:multi TARGET=dreadgoad EC2_NAME=kali-ares +``` + +(Both halves rely on the ambient AWS profile resolving `kali-ares` — see the "AWS auth" note above. Drop the `&&` and run just the first half for a deploy-only.) + +**`task ec2:restart` does NOT bounce the workers.** It is `stop` + `start` (`.taskfiles/ec2/Taskfile.yaml`): `stop` stops `ares-orchestrator.service` and `pkill -f "ares orchestrator"`; `start` brings up redis-server, nats-server and postgresql. **Neither touches a single `ares@.service` unit.** Any advice that says otherwise — including older revisions of this file — has you running a command that cannot fix the problem it is prescribed for. + +**`task ec2:deploy` already restarts the workers, with one catch.** After installing the binary it runs (`.taskfiles/ec2/Taskfile.yaml:255-257`, and again at `:452-454`): + +```sh +UNITS=$(systemctl list-units --type=service --state=active --no-legend "ares@*.service" | awk '{print $1}' | sort -u) +if [ -z "$UNITS" ]; then echo "no ares@ worker units active — skipping restart"; else systemctl restart $UNITS; fi +``` + +The catch is `--state=active`: a worker that crashed, or one whose unit is loaded-but-inactive, is invisible to that query and silently keeps its stale binary. **`no ares@ worker units active — skipping restart` in deploy output means nothing was restarted.** Read for that line; don't assume the restart happened. + +To force every worker unit regardless of state: + +```bash +task ec2:exec EC2_NAME=kali-ares CMD='systemctl restart "ares@*.service"; systemctl is-active "ares@*.service" | sort | uniq -c' +``` + +Faster deploy-only when you don't need to publish to S3 (builds natively on EC2): + +```bash +task ec2:deploy EC2_NAME=kali-ares BUILD_TOOL=remote +``` + +## Step 9 — kill, clear, retry (last resort) + +Don't do this until you've captured logs and runtime — these are destructive. + +```bash +task ec2:stop-op EC2_NAME=kali-ares LATEST=true # graceful stop of one op +task ec2:stop EC2_NAME=kali-ares # stop all workers (keeps Redis) +task ec2:restart EC2_NAME=kali-ares # stop+start orchestrator + infra ONLY (keeps Redis; does NOT touch ares@ workers) +task ec2:exec EC2_NAME=kali-ares CMD='systemctl restart "ares@*.service"' # the actual worker bounce +``` + +To actually wipe state, use the CLI cleanup command instead of FLUSHALL: + +```bash +ares --ec2 kali-ares ops cleanup --max-age-hours 0 +``` + +## K8s deployment notes + +Same triage flow, different transport: + +| EC2 command | K8s equivalent | +|--------------------------------------------|-------------------------------------------------| +| `task ec2:status` | `task remote:status` | +| `task ec2:exec CMD='...'` | `kubectl exec -n attack-simulation -- ...`| +| `task ec2:logs ROLE=orchestrator` | `task remote:logs ROLE=orchestrator` | +| `task ec2:redis:forward` | `kubectl port-forward -n attack-simulation svc/redis 6379:6379` | +| Loki query (same Grafana) | Filter on `namespace="attack-simulation"` instead of `deployment="alpha-operator-range-kali-ares"` | + +## Reference: Loki labels seen on grafana.techvomit.xyz + +- `app`: `ares` covers everything ares writes +- `deployment`: `alpha-operator-range-kali-ares` for the EC2 box +- `environment`: `prod` or `local` +- `job`: `orchestrator.log`, `recon.log`, `credential_access.log`, `cracker.log`, `acl.log`, `privesc.log`, `lateral.log`, `coercion.log`, `syslog`, `auth.log`, `user-data`, `ansible` +- `service_name`: `ares`, `ares-orchestrator`, `ares--agent`, also blue: `ares-blue-orchestrator`, `ares-blue-triage`, etc. +- `host`: `kali` + +If a label value is missing from this list, run `mcp__grafana__list_loki_label_values` to discover what's actually shipping. + +## Reporting + +When you finish debugging, return a short report: + +- **Op id**, current `status`, runtime, token total, **objective state** (domains compromised, hosts owned, creds, hashes). +- **Baseline comparison** in one line: how this op's progress curve compares to the last 2-3 ops at the same runtime. Skip only if no prior op exists. +- **Verdict**: `healthy / wedged / crashed / slow-vs-baseline / unknown`. **Never** say "healthy" without citing two state snapshots 60s apart that show state advancing, or fresh log lines showing tool calls succeeding in the last minute. +- **Root cause** in one sentence, with the SSM/Loki/CLI evidence that pins it (quote the log line; cite the failed-task `task_type` and `role`). +- **Next action** — restart, redeploy, inject state, file a bug — and the exact command(s) to run. + +Do not narrate every probe. The user wants the answer and the command to fix it, not the journey. But do not skip probes either: if you find yourself drafting a "healthy" verdict without having grepped the orchestrator tail for the wedge signatures and pulled `ares ops list` for baseline, stop and go back to Step 0. diff --git a/.claude/skills/ares/SKILL.md b/.claude/skills/ares/SKILL.md new file mode 100644 index 000000000..bf3d53fa3 --- /dev/null +++ b/.claude/skills/ares/SKILL.md @@ -0,0 +1,136 @@ +--- +name: ares +description: Operating, debugging, deploying and reasoning about the Ares autonomous red/blue AD attack platform in this repo. Use for launching or stopping red ops (task red:ec2:multi, ec2:launch, red:multi) and blue investigations, reading loot/reports/scorecards, deploying or gating a binary on EC2 kali-ares / k8s attack-simulation / the proxmox attacker VM, inspecting Redis op state and key types, LogQL/Loki, Tempo/OTEL and Grafana-MCP queries against ares, config/ares.yaml or ARES_* env questions, per-role model assignment, benchmark/replay/diversity-sweep/eval work, the tool catalog and tool-failure strings, CI gates and pre-commit hooks, and the banned-lab-token test-data rule. Also use whenever a claim about ares needs verifying before it is stated — the reference set here carries file:line citations and the mistakes this assistant has repeatedly made on this repo. +--- + +# Ares + +Router + non-negotiables. The detail lives in `references/`; every claim there is cited to `file:line` at HEAD. + +## Before you touch anything + +Sourced from `references/hard-won-lessons.md` — 30 rules mined from 209 sessions where the operator had to correct this assistant. **Read that file first.** Each rule below names the check that satisfies it. + +- **Test data is a closed set.** Only `contoso.local` / `fabrikam.local`, `192.168.58.x`, `dc01`/`dc02`/`sql01`/`web01`/`ws01`/`ca01`, `alice`/`bob`/`carol`/`admin`/`svc_*`, `P@ssw0rd!`. Gate with `scripts/goad-token-sweep.sh` — never by eyeballing, and **never** by widening the Write hook's bypass `case` list. The three enforcers, their divergences and their coverage holes are tabulated once, in `references/tools-and-gates.md#the-banned-token-sweep`; do not restate them elsewhere. + - **This skill directory is unswept by both automated layers.** `scripts/goad-token-sweep.sh:36` exempts all of `.claude/`, and its whole-tree mode enumerates `git ls-files` (`:41-47`) — `.claude/skills/ares/**` is untracked at HEAD (`git ls-files .claude` lists only the three agents and the `ares-debug` / `attack-path-diversity-sweep` skills). Only the PreToolUse Write/Edit hook covers it, so a file arriving here by `cp`/`mv`/`rsync` is never scanned. Passing the paths to the script explicitly does **not** help — its exempt filter runs on `"$@"` too (`:52`), so it exits 0 vacuously. Borrow the regex: + + ```bash + bash -c 'eval "$(sed -n "23,29p" scripts/goad-token-sweep.sh)"; grep -rHniE "$banned" .claude/skills/ares/' + ``` + + - That hook is itself **operator-local and untracked** — `.gitignore:31-35` ignores `.claude/*` and un-ignores only `agents/` and `skills/**`. It is wired at `.claude/settings.json:10` as a bare command path, which requires the exec bit; at HEAD the file is mode 644. Verify before relying on it: `test -x .claude/hooks/check-banned-strings.sh`. +- **Pin the EC2 box to an explicit instance id + region before the first remote command, and pin it twice.** `AWS_REGION` alone selects staging (`us-west-1`, profile `lab`) vs prod (`us-east-1`); `EC2_NAME=kali-ares` is a `*kali-ares*` glob that matches in both. Default to staging; touch prod only when the user says "prod" in that message. `task ec2:launch` runs `redis-cli FLUSHDB` on whatever it resolves. + + ```bash + AWS_PROFILE=lab AWS_REGION=us-west-1 task ec2:resolve EC2_NAME=kali-ares # prints id+IP+Name for EVERY match + ``` + + SSM-backed tasks honour `EC2_INSTANCE_ID` (`run-ssm.sh:69-73`); CLI-backed ones (`ec2:runtime/loot/ops/watch/kill/stop-op/teardown`, `blue:*`) do not — pin those as `EC2_NAME=i-…` and pass `AWS_PROFILE=`/`AWS_REGION=` explicitly, because clap hard-defaults `lab`/`us-west-1` and ignores your exports. `ec2:report` is SSM-backed despite looking like the others (`.taskfiles/ec2/Taskfile.yaml:878-900`). +- **Never attribute an op result to your change until a literal NEW with that change is present in the deployed binary.** `task ec2:exec EC2_NAME= CMD="grep -ac -- '' /usr/local/bin/ares"` must be ≥ 1. **Outer double quotes, inner single** — the inverted form dies with `CMD required` / exit 201 whenever the literal contains a space (`.taskfiles/ec2/Taskfile.yaml:1477-1479`; empirically verified in `references/operations.md`), and gate literals are normally log sentences. Pick the literal from a `contains("…")`, `format!`/`bail!`/`panic!` fragment, or `.arg("…")` — **never** `starts_with`/`ends_with`/`==`, which the `dev-deploy` profile folds out. A failed gate means your change did not ship; there is no other reading. +- **Progress is a state diff, not liveness.** Two Redis snapshots 60 s apart that show objective state advancing. Process up, workers `active`, Redis ping, ≥80% cache hit and climbing tokens are all compatible with a wedged op — token churn plus a high cache-hit rate is the *signature* of the wedge, not evidence of health. +- **Never run an interactive or blocking command from an agent.** Banned: `task ec2:logs` (interactive SSM session, never terminates), `task ec2:redis:forward` / `ec2:nats:forward` (foreground, and each `xargs kill`s whatever holds its local port), `task ec2:watch`, `task ec2:launch` (`WAIT` defaults `true`), `task red:multi` (`FOLLOW` defaults `true`), `red:multi:watch` without `ONCE=true`, `task remote:logs` (`FOLLOW` defaults `true`), `task run WAIT=true|CAPTURE=true`, `ec2:loot`/`red:multi:loot` with `DIFF=true` (promoted to a 10 s watch loop), `task blue:multi:operation-status WATCH=…` (no timeout), `task blue:reports:clean` (`read -p`, hangs under `task -y`), `ares ops delete` without `--force`. Use `ec2:logs:fetch`, `ec2:exec CMD='tail -n 200 …'`, `ec2:ops:ids`, `ec2:runtime` instead. +- **Never read an exit code through a pipe.** The Bash tool's zsh inherits `pipefail`, so a pipeline can invent or hide a failure: `cmd >/tmp/out 2>&1; echo "REAL_EXIT=$?"; rg -n 'pattern' /tmp/out`. A trailing `rg` in a compound call sets the call's exit code. +- **Base64-wrap any remote command containing a double quote, `$( )`, or a space-in-arg.** `{{.CMD}}` is spliced textually into `run_ssm_cmd`; `$( )` evaluates **locally**. `B64=$(printf '%s' '