Skip to content

17. Troubleshooting

Jump to bottom
BaddKharma edited this page Sep 2, 2026 · 11 revisions

Troubleshooting

Reference for components misbehaving after deployment. Each section names a specific failure mode with symptoms, root cause, and fix. If you don't find your issue here, check /var/log/user-data.log on the affected instance; it captures the full provisioning output.

Connectivity Checks

C2 Server Isolation

All three C2 servers (Mythic, Sliver, Adaptix) have no public IPs. All C2 traffic must flow through the redirector. From the redirector:

ping -c 3 mythic
ping -c 3 sliver
ping -c 3 adaptix

Verify Agent Callbacks

C2 Where to look
Mythic Active Callbacks page in the web UI
Sliver sessions command in the Sliver console
Adaptix Active sessions in the Adaptix client

Review Redirector Logs

sudo tail -20 /var/log/apache2/redirector-ssl-access.log
sudo tail -20 /var/log/apache2/redirector-access.log

URI prefixes identify which C2 is receiving traffic:

/cdn/media/stream/      = Mythic
/cloud/storage/objects/ = Sliver
/edge/cache/assets/     = Adaptix

Internal Hosts Can't Reach Target After Guacamole Rebuild

Symptoms: The redirector reaches the tunneled target cleanly (ping and TCP probe both succeed from the redirector, tun0 is up, OpenVPN shows Initialization Sequence Completed), but every internal host behind Guacamole (Windows operator, Kali, C2 servers) times out to the same target. It looks like a routing problem but is not.

Cause: A Guacamole rebuild generates fresh WireGuard keys and pushes a new server config to the redirector, but guacamole_setup.sh brings up the redirector side with systemctl start wg-quick@wg0 instead of restart. Because wg0 is already running from the first deploy, start is a no-op, so the redirector keeps the old keys. The new guac keys and the stale redirector keys don't match, the WireGuard handshake fails, and the guac-to-redirector hop goes down while the redirector's own OpenVPN path to the target stays up.

Confirm on the redirector:

sudo wg show

If the Guacamole peer shows no recent handshake and transfer near zero, that's it.

Fix:

sudo systemctl restart wg-quick@wg0
sudo wg show

Give it a few seconds for guac's keepalive to produce a fresh handshake, then re-test from the Windows operator (Test-NetConnection <TARGET_IP> -Port 445).

Prevention: Do not rebuild Guacamole mid-session. Deploy and validate the lab before it's needed and leave guac alone. Source fix (start to restart in guacamole_setup.sh) is backlogged.

Component Health Checks

Mythic

cd /opt/Mythic
sudo ./mythic-cli status

Expected: eight core containers + apollo + http, all running. The localhost-binding warnings are expected and harmless.

Get admin password:

sudo cat /opt/Mythic/.env | grep MYTHIC_ADMIN_PASSWORD

Web UI from the Windows workstation: https://mythic:7443. Log in as mythic_admin with the password from above.

Guacamole

docker ps

Expected: three containers all up: guacamole/guacamole, postgres:15, guacamole/guacd.

Redirector

sudo /home/admin/test_redirector.sh

Checks Apache status, VirtualHost config, connectivity to all three C2 backends, and header/decoy page behavior.

Check redirect.rules is loaded:

grep -c 'RewriteCond' /etc/apache2/redirect.rules

Test security layers manually (curl's default User-Agent matches scanner patterns and is blocked by redirect.rules — the block below spoofs a browser UA and is the correct test method):

UA="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
HEADER_VALUE="<token from deployment_info.txt>"
TARGET="yourdomain.tld"          # or <REDIR_PUBLIC_IP> for IP-only/Tunneled Access

# Should return decoy page (no header)
curl -sk -A "$UA" https://$TARGET/

# Should return decoy page (wrong header)
curl -sk -A "$UA" -H "X-Request-ID: wrong-value" https://$TARGET/cdn/media/stream/test

# Should proxy to Mythic (connection refused if no listener, expected)
curl -sk -A "$UA" -H "X-Request-ID: $HEADER_VALUE" https://$TARGET/cdn/media/stream/test

Sliver

which sliver-server

If missing, see Sliver > Troubleshooting.

Adaptix

sudo systemctl status adaptix

If not running: sudo systemctl start adaptix. If the binary is missing, the build did not finish; run ~/build_adaptix_server.sh and see Adaptix > Troubleshooting.

SSH Connections via Guacamole

Each Guacamole SSH connection should connect without a password prompt and land at the correct hostname:

Mythic (SSH)   > admin@mythic:~$
Guacamole (SSH)   > admin@guac:~$
Redirector (SSH)  > admin@redirector:~$
Sliver (SSH)   > admin@sliver:~$
Adaptix (SSH)  > admin@adaptix:~$
Kali (SSH)      > admin@kali:~$

SSH access model: All lab hosts are accessed via the Guacamole portal (SSH connections) or MobaXterm pre-configured sessions on the Windows operator workstation. The redirector accepts only ports 80 and 443 publicly. Guacamole has a break-glass direct SSH rule for localPub_ip (key-based, advanced users only) if the portal itself is unavailable.

Warning

Do not destroy individual components with targeted terraform destroy -target on the redirector alone. (terraform taint was removed in Terraform 1.0; redStack requires ≥ 1.0, so it is unavailable. The modern equivalent is terraform apply -replace=<resource>, which carries the same state risk.) Partial destroys can corrupt Terraform state and leave orphaned AWS resources (dangling ENIs, stale security group rules, broken VPC peering). If the redirector is in a broken state that cannot be recovered by restarting Apache or re-running setup scripts, the supported recovery path is a full terraform destroy && terraform apply.

Note

AWS and Azure cloud IP blocks are commented out by default in redirect.rules because this lab runs in AWS. If you deploy outside cloud environments you can re-enable them by editing /etc/apache2/redirect.rules and uncommenting the relevant blocks, then sudo systemctl reload apache2.

Agent Won't Callback

Symptoms: Agent executes but no callback in Mythic / Sliver / Adaptix.

Checklist:

  • Listener is running on the C2 server
  • Callback Host and Port match the redirector's domain/IP and port 443
  • Agent sends the correct X-Request-ID header with the auto-generated token
  • Agent URI uses the correct prefix (/cdn/media/stream/, /cloud/storage/objects/, or /edge/cache/assets/)
  • Redirector Apache is running with all VirtualHosts enabled
  • Redirector can reach the C2 server's private IP (ping mythic)
  • Agent user-agent isn't blocked by redirect.rules (no scanner/AV strings)

Debug on the redirector:

sudo apache2ctl -S
systemctl status apache2

# Logs (differentiate by URI prefix)
sudo tail -100 /var/log/apache2/redirector-ssl-access.log
sudo tail -100 /var/log/apache2/redirector-ssl-error.log

# Pre-installed end-to-end test
sudo /home/admin/test_redirector.sh

Terraform Errors

Invalid value for input variable on localPub_ip 

Error: Invalid value for input variable

  on terraform.tfvars line 11:
  11: localPub_ip = ["203.0.113.5/32", "198.51.100.42/32"]

The given value is not suitable for var.localPub_ip declared at
variables.tf:15,1-23: string required, but have tuple.

Cause: your clone predates the change that made localPub_ip a list(string). The tfvars supplies a bracketed list, but variables.tf in that checkout still declares a single string, so Terraform rejects the value before it reaches any resource. terraform init succeeding does not clear this; the type check happens at plan time.

Fix: update the repo, then re-plan.

cd redStack
git pull
cd terraform
terraform plan

Keep the list form in terraform.tfvars; it is correct on current main. If you cannot pull yet, the one-line workaround is to collapse the value back to a single string (localPub_ip = "203.0.113.5/32"), which allowlists only that one address.

OptInRequired on first apply

Cause: AWS Marketplace EULA for the Kali Linux AMI hasn't been accepted on this account.

Fix: Visit https://aws.amazon.com/marketplace/pp/prodview-fznsw3f7mq7to, click Continue to Subscribe, then Accept Terms. Re-run terraform apply; no state cleanup needed. See Prerequisites > Step 5.

InvalidKeyPair.NotFound

# List available keys
aws ec2 describe-key-pairs --query 'KeyPairs[].KeyName'
# Update terraform.tfvars with correct name

VPCLimitExceeded

AWS accounts have a default limit of five VPCs per region. redStack creates two (team server + redirector), so you need at least two free slots.

If at the limit, either delete unused VPCs or switch to the existing default VPC: in terraform.tfvars, set use_default_vpc = true.

Marketplace AMI: only Kali requires acceptance

The Kali Linux AMI is the only AWS Marketplace dependency in redStack. All other AMIs (Redirector, Mythic, Sliver, Adaptix, Guacamole, Windows) are standard Amazon-published AMIs with no Marketplace gate. If you hit OptInRequired for any non-Kali host, check that terraform.tfvars is not pointing at a custom AMI ID.

← Previous: Cost Management | Next: redStack on GitHub →

"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it."

Brian Kernighan, The Elements of Programming Style (1978)

Clone this wiki locally