diff --git a/.claude/skills/deep-code-review/references/tool-hub.md b/.claude/skills/deep-code-review/references/tool-hub.md index f6a6aca61..7cdeeb212 100644 --- a/.claude/skills/deep-code-review/references/tool-hub.md +++ b/.claude/skills/deep-code-review/references/tool-hub.md @@ -26,6 +26,7 @@ The agent sandbox denies installs and most network, so the owner provisions scan - osv-scanner v2.7.0 stores the DB as `DIR/osv-scalibr/ECOSYSTEM/all.zip`; its offline-mode docs say `osv-scanner/`. Both count as warm. A lockfile whose ecosystem has no DB stays not-run; osv-scanner's `no offline version` message names the fix, `--print DIR` (a DIR always re-downloads: another ecosystem's zip already counts as warm). - Trivy's DB download fails when `~/.docker/config.json` names a `credsStore` helper that is not installed (seen on one host, not in the docs); the owner line sets `DOCKER_CONFIG` to an empty dir for it. - `pipx` and `uv tool` install into `~/.local/bin`, which a sandboxed PATH may lack: `review_checks.sh` appends it (`DCR_USER_BIN` overrides, empty disables). `DCR_CACHE_ROOT` overrides the user cache dir. `scrub_env.sh` pins both, so CI tests do not depend on what the host has installed. +- Pins in `provision_tools.sh` (the one place the versions live): pins checked against PyPI, RubyGems, npm, NuGet, Packagist and Hex on 2026-11-18 (every pipx, gem, npm, NuGet, Composer and Hex version exists); `skillspector` is the commit SHA of the `v2.13.6` tag (checked with `git ls-remote --tags` on 2026-11-18). Homebrew rows are unpinned; names are unverified, notably `scorecard`, `gosec`, `nuclei`, `codeql`. `brew info tflint` found no core formula, so `tflint` stays a `terraform-linters/tap` row (tap path unverified). `codeql` installs via `brew install --cask` on macOS only (unverified); on other OSes it prints BLOCKED. `--print` installs each package alone and prints `FAILED ` on failure. - Every run ends with `review_checks: ran X / wired Y scanners; not run: ...; partial: ...; fixable: ...` and records it in `/review-checks-last.json`. Wired = had work this run (shellcheck, ruff/pyflakes and `--history` gitleaks too, never fixable); `partial` = ran with a narrower scope (trivy `secret-only`, trufflehog with files it could not scan, semgrep with errors or `bundled-only`: its `p/default` pack did not load, which a snapshot fixes). `perun_doctor.py` shows a WARN `review scanners` row while a wired tool had a fixable cause (osv-scanner's command names this repo as DIR) and INFO for a partial scope or an unfixable cause. ## CLI tools (run by `review_checks.sh`) @@ -82,6 +83,7 @@ The agent sandbox denies installs and most network, so the owner provisions scan | `mcp-scanner` | Not routed (Cisco mcp-scanner, https://github.com/cisco-ai-defense/mcp-scanner, Apache-2.0, tag 4.8.7, fetched 2026-11-14): `--analyzers yara` needs no key and `static --tools FILE` reads a pre-generated tools list, but the README does not define the `--format raw` output shape, and `--config-path` / known-configs scans connect to the listed MCP servers (not offline). Output shape and offline behavior of a config scan are `unverified`, so no row | not run | not routed | not routed | | `snyk-agent-scan` | MCP and skill configs | not run | needs a `SNYK_TOKEN` and calls the Agent Scan API (no offline mode documented; it can also start MCP servers): never run by the hub (checked 2026-11-15, Apache-2.0, 0.6.11) | route-only | | `opengrep` | SAST fallback: stands in for `semgrep` when `semgrep` is absent and `opengrep` is installed; same rules dir (repo config, else the bundled rules), never the registry pack; rows keep tool `semgrep` and the closing line says `opengrep` ran | `opengrep scan --config DIR --json --quiet -- FILES` (env `SEMGREP_SEND_METRICS=off`) | unverified (its README shows `scan` and `-f` only; `--config` and `--json` are inherited from semgrep, not confirmed); a crash is `not run`, never clean | route-only, local | +| `opengrep` install source | `provision_tools.sh` does not install it: no Homebrew formula, and the upstream installer is an unpinned script on `main` (no checksummed pin verified) | install it yourself from a release you verified; the route only uses it when present | blocked: no pinned, checksummed install route | route-only, local | | `scorecard` | OpenSSF Scorecard for the `origin` GitHub repo; checks scoring 0-4 become rows (rule `scorecard-`, file `.`); -1 (inconclusive) is ignored; a report with no checks, or only -1 checks, is `not run` (`not-checked` row), never a clean score. Opt-in: `REVIEW_SCORECARD=1` plus `GITHUB_TOKEN`; no token is `not run` | `scorecard --repo=github.com/OWNER/REPO --format=json` (token passed as `GITHUB_AUTH_TOKEN`) | verified (`scorecard --help`, 2026-11-12) | route-only, opt-in, network | | `nuclei` | Conservative, not passive: sends real requests to one running app. Opt-in `REVIEW_DAST_URL`, exact localhost origin only unless `REVIEW_DAST_ALLOW_REMOTE=1`. Allowlist: info and low http templates tagged `tech` or `exposure`, minus `default-login`, `xss`, `ssrf`, `oast`, `fuzz`, `fuzzing`, `takeover`, `cve`, `intrusive`, `dos`, `bruteforce`, `rce`, `sqli`. A failed `curl` probe of the target, or zero templates listed by `nuclei -tl`, is `NOT CHECKED` (an empty scan never reads as clean) | `nuclei -u URL -jsonl -silent -nc -ni -duc -rl 10 -s info,low -type http -tags tech,exposure -etags ...` | flags verified (`nuclei -h`, 2026-11-12); the tag choice is a conservative judgment, not a nuclei-certified safe set | route-only, opt-in, network | | `zap-baseline.py` | ZAP baseline (a spider that sends real requests, plus passive rules; no active attack scan) against the same `REVIEW_DAST_URL`; uses the local script, else `docker run --pull=never --user "$(id -u)" ghcr.io/zaproxy/zaproxy:stable` only when docker is present and the image is already pulled (a missing image is `not run`, never auto-pulled) | `zap-baseline.py -t URL -J FILE -m 1`; exit 1 and 2 (fail, warn) are findings, 3 and others are `not run` | unverified (not run here; ZAP may follow redirects, so a redirect off the target is still requested and its scope is only what `-t` names; docker `--network host` does not reach the host's localhost on Docker Desktop) | route-only, opt-in, network | diff --git a/.claude/skills/deep-code-review/scripts/provision_tools.sh b/.claude/skills/deep-code-review/scripts/provision_tools.sh index ebaf52fb7..a84fa514b 100755 --- a/.claude/skills/deep-code-review/scripts/provision_tools.sh +++ b/.claude/skills/deep-code-review/scripts/provision_tools.sh @@ -3,8 +3,8 @@ # the tools and their offline caches are provisioned ONCE by the owner, outside the sandbox. # Default (dry run): list the scanner binaries not on PATH and the cold caches (trivy DB, osv-scanner offline DB, local semgrep p/default # snapshot). Installs nothing, writes nothing, exit 0. -# --print: emit ONE copy-paste line (type it yourself outside the sandbox, with the `!` prefix) that installs what is missing (brew, pipx, gem; -# pipx pins are the PyPI versions checked 2026-10-29, brew formulae cannot be pinned) and warms the cold caches; it ends with this script's +# --print: emit ONE copy-paste line (type it yourself outside the sandbox, with the `!` prefix) that installs what is missing (brew, brew --cask on macOS only, pipx, gem; +# pins are unverified, see tool-hub.md; brew formulae cannot be pinned) and warms the cold caches; it ends with this script's # dry run so the result is visible. Nothing missing or cold: a `# nothing to provision` comment. DIR (default: the current dir) is the # project whose root lockfiles osv-scanner downloads offline databases for (one per ecosystem found there). A DIR given on the command line # always re-downloads, so `--print OTHER_DIR` adds the ecosystems OTHER_DIR needs. @@ -43,42 +43,83 @@ esac dir=$(cd "$dir" && pwd) exec 3>&1; [ "$pr" = 0 ] || exec 1>&2 # --print: stdout carries only the one owner line, the listing goes to stderr self=$(cd "$(dirname "$0")" && pwd)/$(basename "$0") -# binary installer spec +# binary installer spec: BIN how SPEC. how = brew|cask|pipx|gem (SPEC is the package), cmd (SPEC is the whole install command: cargo, go, npm, +# dotnet, composer, mix, uv) or blocked (SPEC is the reason no install line is printed). Every review_checks.sh --list-tools name is covered here or listed with a reason in scripts/test-provision-covers-list-tools.sh. tools='gitleaks brew gitleaks trufflehog brew trufflehog -semgrep pipx semgrep==1.182.1 +semgrep pipx semgrep==1.186.0 trivy brew trivy osv-scanner brew osv-scanner actionlint brew actionlint -zizmor pipx zizmor==1.31.1 +zizmor pipx zizmor==1.32.0 bandit pipx bandit==1.9.4 scorecard brew scorecard gosec brew gosec grype brew grype checkov pipx checkov==3.4.5 hadolint brew hadolint -ruff pipx ruff +ruff pipx ruff==0.17.5 syft brew syft kube-linter brew kube-linter kube-score brew kube-score -tflint brew tflint +tflint brew terraform-linters/tap/tflint staticcheck brew staticcheck -brakeman gem brakeman +brakeman gem brakeman -v 8.1.0 kics brew kics -licensee gem licensee +licensee brew licensee cosign brew cosign -opengrep brew opengrep -nuclei brew nuclei' +nuclei brew nuclei +ansible-lint pipx ansible-lint==26.11.0 +cfn-lint pipx cfn-lint==1.60.1 +detect-secrets pipx detect-secrets==1.5.0 +flawfinder brew flawfinder +guarddog pipx guarddog==3.2.0 +mobsfscan pipx mobsfscan==1.2.0 +njsscan pipx njsscan==1.0.2 +pyflakes pipx pyflakes==4.1.0 +slither brew slither-analyzer +codeql cask codeql +cppcheck brew cppcheck +jq brew jq +kubescape brew kubescape +pmd brew pmd +shellcheck brew shellcheck +tfsec brew tfsec +node brew node +go brew go +bundler-audit gem bundler-audit -v 0.9.3 +rubocop gem rubocop -v 1.92.3 +cargo-audit brew cargo-audit +cargo-deny brew cargo-deny +govulncheck brew govulncheck +retire cmd npm install -g retire@6.1.1 +solhint cmd npm install -g solhint@6.2.5 +tsc cmd npm install -g typescript@7.0.2 +devskim cmd dotnet tool install --global Microsoft.CST.DevSkim.CLI --version 1.0.103 +psalm cmd composer global require vimeo/psalm:7.10.0 +sobelow cmd mix escript.install --force hex sobelow 0.19.0 +opengrep blocked no Homebrew formula and the upstream installer is an unpinned script on main with no checksummed release pin verified: install it yourself from a release you verified +uv brew uv +skillspector cmd uv tool install git+https://github.com/NVIDIA/skillspector.git@e2674da339b0bddc6e565d059c0577c6d9cd25da' lb=${DCR_USER_BIN-${HOME:-/nonexistent}/.local/bin} # pipx and `uv tool` install here; a sandboxed PATH may lack it -miss=0 cold=0 brew=() pipx=() gem=() fix=() +# dotnet/composer/mix installs land in their own dirs; probe them too (same dirs review_checks.sh adds to PATH; empty DCR_USER_BIN disables all) +dirs=(); [ -z "$lb" ] || for d in "$lb" "${HOME:-/nonexistent}/.dotnet/tools" "${HOME:-/nonexistent}/.composer/vendor/bin" "${HOME:-/nonexistent}/.config/composer/vendor/bin" "${HOME:-/nonexistent}/.mix/escripts"; do [ -d "$d" ] && dirs+=("$d"); done +have() { local d; for d in ${dirs[@]+"${dirs[@]}"}; do [ -x "$d/$1" ] && { echo "$d"; return 0; }; done; return 1; } +miss=0 cold=0 brew=() cask=() pipx=() gem=() oth=() othb=() fix=() while read -r bin how spec; do if command -v "$bin" >/dev/null; then echo "ok $bin"; continue; fi - if [ -n "$lb" ] && [ -x "$lb/$bin" ]; then echo "ok $bin ($lb/$bin, not on PATH; review_checks.sh adds $lb)"; continue; fi - echo "MISSING $bin ($how install $spec)"; miss=$((miss + 1)) - case $how in brew) brew+=("$spec") ;; gem) gem+=("$spec") ;; *) pipx+=("$spec") ;; esac + if d=$(have "$bin"); then echo "ok $bin ($d/$bin, not on PATH; review_checks.sh adds $d)"; continue; fi + # a cmd entry needs its own toolchain (first word of the command); without it the line would fail, so name the blocker instead + if [ "$how" = blocked ]; then echo "BLOCKED $bin ($spec)"; miss=$((miss + 1)); continue; fi + if [ "$how" = cask ] && [ "$(uname)" != Darwin ]; then echo "BLOCKED $bin (cask install is macOS-only: install it yourself)"; miss=$((miss + 1)); continue; fi + tc=${spec%% *}; [ "$tc" != npm ] || tc=node # npm ships with Homebrew's node + if [ "$how" = cmd ] && ! command -v "$tc" >/dev/null && ! have "$tc" >/dev/null && [[ " ${brew[*]-} " != *" $tc "* ]]; then echo "BLOCKED $bin (needs $tc: install it first, then re-run)"; miss=$((miss + 1)); continue; fi + if [ "$how" = cmd ]; then echo "MISSING $bin (run: $spec)"; else echo "MISSING $bin ($how install $spec)"; fi; miss=$((miss + 1)) + case $how in brew) brew+=("$spec") ;; cask) cask+=("$spec") ;; gem) gem+=("$spec") ;; cmd) oth+=("$spec"); othb+=("$bin") ;; *) pipx+=("$spec") ;; esac done <<<"$tools" +[ ${#gem[@]} -eq 0 ] || brew=(ruby ${brew[@]+"${brew[@]}"}) [ ${#pipx[@]} -eq 0 ] || command -v pipx >/dev/null || brew=(pipx ${brew[@]+"${brew[@]}"}) -[ ${#brew[@]} -eq 0 ] || command -v brew >/dev/null || echo "note: brew not found; install Homebrew (https://brew.sh) or use each tool's own installer" +[ ${#brew[@]} -eq 0 ] && [ ${#cask[@]} -eq 0 ] || command -v brew >/dev/null || echo "note: brew not found; install Homebrew (https://brew.sh) or use each tool's own installer${gem[0]+"; without it the gem rows fall back to the system ruby, which is too old and may fail"}" q() { printf '%q' "$1"; } cache() { # NAME WARM-FN PATH COMMAND [FORCE] if [ -z "${5:-}" ] && "$2"; then echo "warm $1 ($3)"; else echo "COLD $1 ($3)"; cold=$((cold + 1)); fix+=("$4"); fi @@ -91,9 +132,13 @@ cache "semgrep p/default snapshot" sg_warm "$sgf" "mkdir -p $(q "$(dirname "$sgf echo "provision_tools: $miss missing, $cold cold; nothing was installed or downloaded (--print emits the one line to run yourself)" [ "$pr" = 1 ] || exit 0 cmd=() -[ ${#brew[@]} -eq 0 ] || cmd+=("brew install ${brew[*]}") -for p in ${pipx[@]+"${pipx[@]}"}; do cmd+=("pipx install $p"); done -for p in ${gem[@]+"${gem[@]}"}; do cmd+=("gem install $p"); done +# every package installs on its own, so one unknown name or failed install prints FAILED and does not stop the rest; groups join with ; +[ ${#brew[@]} -eq 0 ] || cmd+=("for f in ${brew[*]}; do brew install \"\$f\" || echo \"FAILED \$f\"; done") +[ ${#cask[@]} -eq 0 ] || cmd+=("if [ \"\$(uname)\" = Darwin ]; then for f in ${cask[*]}; do brew install --cask \"\$f\" || echo \"FAILED \$f\"; done; fi") # casks are macOS-only +[ ${#pipx[@]} -eq 0 ] || cmd+=("for f in ${pipx[*]}; do pipx install \"\$f\" || echo \"FAILED \$f\"; done") +# system ruby is too old for these gems: use Homebrew's when brew exists, installing into $DCR_USER_BIN (default ~/.local/bin, on review_checks.sh's PATH) +for p in ${gem[@]+"${gem[@]}"}; do cmd+=("\$(command -v brew >/dev/null && echo \$(brew --prefix ruby)/bin/)gem install $p${lb:+ -n $(q "$lb")} || echo FAILED ${p%% *}"); done +for i in ${!oth[@]}; do cmd+=("${oth[i]} || echo FAILED ${othb[i]}"); done [ ${#fix[@]} -eq 0 ] || cmd+=("${fix[@]}") [ ${#cmd[@]} -gt 0 ] || { echo "# nothing to provision" >&3; exit 0; } cmd+=("bash $(q "$self")") # re-check at the end so the owner sees what is still missing diff --git a/.claude/skills/deep-code-review/scripts/review_checks.sh b/.claude/skills/deep-code-review/scripts/review_checks.sh index 1240d072e..2075b5459 100755 --- a/.claude/skills/deep-code-review/scripts/review_checks.sh +++ b/.claude/skills/deep-code-review/scripts/review_checks.sh @@ -53,6 +53,10 @@ set -euo pipefail # pipx and `uv tool` install into ~/.local/bin, which a sandboxed shell's PATH may lack: append it so provisioned scanners are found (DCR_USER_BIN overrides; empty disables) lb=${DCR_USER_BIN-${HOME:-/nonexistent}/.local/bin} case ":$PATH:" in *":$lb:"*) ;; *) [ -z "$lb" ] || [ ! -d "$lb" ] || PATH=$PATH:$lb ;; esac +# dotnet/composer/mix tool installs (provision_tools.sh) land in their own dirs; add each that exists (empty DCR_USER_BIN disables) +for td in .dotnet/tools .composer/vendor/bin .config/composer/vendor/bin .mix/escripts; do + td=${HOME:-/nonexistent}/$td; [ -z "$lb" ] || [ ! -d "$td" ] || case ":$PATH:" in *":$td:"*) ;; *) PATH=$PATH:$td ;; esac +done repo=. base=origin/main tests=0 fuzz=0 hist=0 hyg=0 tmo=60 ran="" nr="" failed="" missing="" gap="" dg="" w1="" nice19=0 lease=ok leased=0 full=0 wr=0 hmax= cov="" wired="" wiredx="" while [ $# -gt 0 ]; do diff --git a/changelog.d/provision-cask-blocked.md b/changelog.d/provision-cask-blocked.md new file mode 100644 index 000000000..eff859632 --- /dev/null +++ b/changelog.d/provision-cask-blocked.md @@ -0,0 +1,2 @@ +### Fixed +- scripts/test-provision-cask-blocked.sh now runs in CI: registered as scripts/ci-tests.d/provision-cask-blocked.sh (it was never executed). diff --git a/changelog.d/provision-covers-list-tools.md b/changelog.d/provision-covers-list-tools.md new file mode 100644 index 000000000..91df95965 --- /dev/null +++ b/changelog.d/provision-covers-list-tools.md @@ -0,0 +1,7 @@ +### Added +- provision_tools.sh now covers all 55 external tools that `review_checks.sh --list-tools` prints (56 rows in its table; slither and scorecard added). pipx, gem, npm, NuGet, Packagist and Hex installs carry version pins, and skillspector is pinned to a commit SHA; pins unverified (not re-fetched), see tool-hub.md. `uv` is a brew row so the skillspector line works in one owner run. A tool whose toolchain (dotnet, composer, mix) is absent is reported as `BLOCKED (needs )` instead of printing an install line that would fail. opengrep has no Homebrew formula and its upstream installer is an unpinned script, so the dry run prints `BLOCKED opengrep ()` and no install line. +- The gem install line honors `DCR_USER_BIN` instead of a hardcoded `~/.local/bin`. +- The installed-tool probe in provision_tools.sh, and the PATH set up by review_checks.sh, also include `~/.dotnet/tools`, the composer global `vendor/bin` and `~/.mix/escripts` when they exist (an empty `DCR_USER_BIN` still disables all extra dirs). +- scripts/test-provision-covers-list-tools.sh fails when an external `--list-tools` name has no provision row and is not on an explicit no-install list with a reason; scripts/test-provision-toolchain-dirs.sh covers the extra tool dirs, the gem install dir and the opengrep blocker line. +- `--print` installs every brew formula, pipx package, gem and npm/other row on its own and prints `FAILED ` for a failure, so one bad name does not stop the rest; casks (codeql) install through a macOS-only `brew install --cask` call. The gem line falls back to plain `gem` when brew is absent, sobelow installs with `--force`, and tool-hub.md marks every pin and Homebrew name `unverified` (tflint has no core formula). +size-budget-raise: .claude/skills/deep-code-review/references/tool-hub.md 63930→64982 main gained tool-hub rows while this PR added its pin provenance note and opengrep blocker row; lazy reference, off every must-load set diff --git a/scripts/ci-tests.d/provision-cask-blocked.sh b/scripts/ci-tests.d/provision-cask-blocked.sh new file mode 100644 index 000000000..2d0894591 --- /dev/null +++ b/scripts/ci-tests.d/provision-cask-blocked.sh @@ -0,0 +1,2 @@ +# Registers the cask-blocked provisioning test: a missing cask tool on a non-Darwin host prints BLOCKED and adds no cask install line. +bash scripts/test-provision-cask-blocked.sh diff --git a/scripts/ci-tests.d/provision-covers-list-tools.sh b/scripts/ci-tests.d/provision-covers-list-tools.sh new file mode 100644 index 000000000..9b2414345 --- /dev/null +++ b/scripts/ci-tests.d/provision-covers-list-tools.sh @@ -0,0 +1,2 @@ +# Registers the check that provision_tools.sh covers every external tool review_checks.sh --list-tools prints. +bash scripts/test-provision-covers-list-tools.sh diff --git a/scripts/ci-tests.d/provision-failed-names.sh b/scripts/ci-tests.d/provision-failed-names.sh new file mode 100644 index 000000000..fcc006355 --- /dev/null +++ b/scripts/ci-tests.d/provision-failed-names.sh @@ -0,0 +1,2 @@ +# Registers the check that a failed install prints FAILED and the no-brew gem note. +bash scripts/test-provision-failed-names.sh diff --git a/scripts/ci-tests.d/provision-toolchain-dirs.sh b/scripts/ci-tests.d/provision-toolchain-dirs.sh new file mode 100644 index 000000000..4c51998f3 --- /dev/null +++ b/scripts/ci-tests.d/provision-toolchain-dirs.sh @@ -0,0 +1,2 @@ +# Registers the check that provision/review_checks probe dotnet, composer and mix tool dirs. +bash scripts/test-provision-toolchain-dirs.sh diff --git a/scripts/size-budgets.tsv b/scripts/size-budgets.tsv index db57d9153..c3a56c1d6 100644 --- a/scripts/size-budgets.tsv +++ b/scripts/size-budgets.tsv @@ -234,7 +234,7 @@ README.md 16118 .claude/skills/deep-code-review/references/small-diff-read-budget.md 526 .claude/skills/deep-code-review/references/app-pentest.md 8135 .claude/skills/deep-code-review/references/supply-chain-check.md 3646 -.claude/skills/deep-code-review/references/tool-hub.md 63930 +.claude/skills/deep-code-review/references/tool-hub.md 64982 .claude/skills/deep-code-review/references/desktop-native-appsec.md 4724 .claude/skills/deep-code-review/references/dynamic-probe.md 2095 .claude/skills/agentic-delivery/references/flow-guards.md 3171 diff --git a/scripts/test-provision-cask-blocked.sh b/scripts/test-provision-cask-blocked.sh new file mode 100644 index 000000000..75102282c --- /dev/null +++ b/scripts/test-provision-cask-blocked.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# provision_tools.sh: on a non-Darwin host a missing cask tool (codeql) prints BLOCKED and adds no cask install to the --print line. +# Hermetic: stub uname=Linux, throwaway HOME/cache; nothing is installed. +set -euo pipefail +P=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/.claude/skills/deep-code-review/scripts/provision_tools.sh +t=$(mktemp -d "${TMPDIR:-/tmp}/dcr-test-cask.XXXXXX"); trap 'rm -rf "$t"' EXIT +mkdir "$t/bin" "$t/home"; fail=0 +printf '#!/bin/sh\necho Linux\n' >"$t/bin/uname"; chmod +x "$t/bin/uname" +E=(env -i HOME="$t/home" PATH="$t/bin:/usr/bin:/bin" DCR_CACHE_ROOT="$t/cache" DCR_USER_BIN="$t/home/.local/bin" TMPDIR="$t") +out=$("${E[@]}" bash "$P" 2>&1 || true) +line=$("${E[@]}" bash "$P" --print 2>/dev/null) +grep -q '^BLOCKED codeql (cask install is macOS-only' <<<"$out" && echo "ok - codeql BLOCKED off macOS" || { echo "FAIL - no BLOCKED codeql"; fail=1; } +grep -q -- '--cask' <<<"$line" && { echo "FAIL - cask install line printed"; fail=1; } || echo "ok - no cask install line" +exit $fail diff --git a/scripts/test-provision-covers-list-tools.sh b/scripts/test-provision-covers-list-tools.sh new file mode 100644 index 000000000..52a57e533 --- /dev/null +++ b/scripts/test-provision-covers-list-tools.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Every tool name `review_checks.sh --list-tools` prints is BUILT-IN (Perun code, listed below), has a provision_tools.sh entry for its binary, +# or is in NOINSTALL with a reason. A new name defaults to EXTERNAL, so it fails here until it is provisioned or justified. +set -euo pipefail +ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +S=$ROOT/.claude/skills/deep-code-review/scripts +BUILTIN=" agent-injection attempt-limit bash-n committed-data cost-limit csrf-cookie dep-currency dep-risk header-config insecure-defaults invisible-chars literals mcp-config ownership pii-data pipefail-sigpipe privileged-route py_compile route-auth sibling-sweep tenant-scope " +# name:binary where the --list-tools name is not the binary the script runs +ALIAS=" gitleaks-history:gitleaks ruff-security:ruff node-check:node go-vet:go fuzz:go " +# name:reason, no global install path +NOINSTALL=" eslint-security:project-local npm devDependency (eslint plus eslint-plugin-security from the scanned repo's own node_modules), never installed globally snyk-agent-scan:route-only, needs SNYK_TOKEN and calls a remote API, never run by the hub (tool-hub.md) " +bins=$(PATH=/usr/bin:/bin DCR_USER_BIN= bash "$S/provision_tools.sh" 2>&1 | awk '$1=="ok"||$1=="MISSING"||$1=="BLOCKED"{print $2}') +fail=0; n=0 +all=$(bash "$S/review_checks.sh" --list-tools) +# every ALIAS and NOINSTALL key must be a real --list-tools name, so a renamed tool cannot leave a dead exemption behind +for k in $(for a in $ALIAS; do echo "${a%%:*}"; done) eslint-security snyk-agent-scan; do + grep -qxF "$k" <<<"$all" || { echo "FAIL - exemption key $k is not in --list-tools"; fail=1; } +done +for t in $all; do + case "$BUILTIN" in *" $t "*) continue ;; esac + case "$NOINSTALL" in *" $t:"*) continue ;; esac + b=$t; for a in $ALIAS; do [ "${a%%:*}" = "$t" ] && b=${a#*:}; done + n=$((n + 1)) + grep -qxF "$b" <<<"$bins" || { echo "FAIL - $t (binary $b) has no provision_tools.sh entry"; fail=1; } +done +[ "$n" -gt 40 ] || { echo "FAIL - only $n external names found; --list-tools parse broke"; fail=1; } +[ "$fail" = 0 ] && echo "ok - $n external --list-tools names covered by provision_tools.sh" +exit "$fail" diff --git a/scripts/test-provision-failed-names.sh b/scripts/test-provision-failed-names.sh new file mode 100644 index 000000000..ed6ad3a25 --- /dev/null +++ b/scripts/test-provision-failed-names.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# provision_tools.sh --print: a failing install prints `FAILED ` and does not stop the rest of the line; with brew absent the dry run +# says gems fall back to the system ruby. Hermetic: stub node/npm/gem/uv/curl on PATH, throwaway HOME and cache root; nothing is installed. +set -euo pipefail +P=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/.claude/skills/deep-code-review/scripts/provision_tools.sh +t=$(mktemp -d "${TMPDIR:-/tmp}/dcr-test-failed.XXXXXX"); trap 'rm -rf "$t"' EXIT +mkdir "$t/bin" "$t/home"; fail=0 +for x in node trivy osv-scanner curl true; do printf '#!/bin/sh\nexit 0\n' >"$t/bin/$x"; chmod +x "$t/bin/$x"; done +for x in npm gem uv pipx; do printf '#!/bin/sh\nexit 1\n' >"$t/bin/$x"; chmod +x "$t/bin/$x"; done +E=(env -i HOME="$t/home" PATH="$t/bin:/usr/bin:/bin" DCR_CACHE_ROOT="$t/cache" DCR_USER_BIN="$t/home/.local/bin" TMPDIR="$t") +line=$("${E[@]}" bash "$P" --print 2>"$t/err") +if grep -q '^note: brew not found.*system ruby.*may fail' "$t/err"; then echo "ok - no brew: note says gems fall back to system ruby"; else echo "FAIL - no gem/brew note"; fail=1; fi +out=$("${E[@]}" bash -c "$line" 2>&1 || true) +for n in retire brakeman skillspector; do + if grep -qx "FAILED $n" <<<"$out" || grep -q "^FAILED $n\$" <<<"$out"; then echo "ok - FAILED $n printed"; else echo "FAIL - no 'FAILED $n'"; echo "$out" | tail -8; fail=1; fi +done +grep -q 'FAILED solhint' <<<"$out" && echo "ok - a failing row does not stop the next (solhint)" || { echo "FAIL - line stopped early"; fail=1; } +exit $fail diff --git a/scripts/test-provision-toolchain-dirs.sh b/scripts/test-provision-toolchain-dirs.sh new file mode 100644 index 000000000..f7bbe72e6 --- /dev/null +++ b/scripts/test-provision-toolchain-dirs.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# provision_tools.sh and review_checks.sh must both see tools that dotnet/composer/mix install outside ~/.local/bin. +set -euo pipefail +P=.claude/skills/deep-code-review/scripts/provision_tools.sh +R=.claude/skills/deep-code-review/scripts/review_checks.sh +t=$(mktemp -d "${TMPDIR:-/tmp}/dcr-test-tcd.XXXXXX"); trap 'rm -rf "$t"' EXIT +mkdir -p "$t/home/.local/bin" "$t/home/.dotnet/tools" "$t/home/.config/composer/vendor/bin" "$t/home/.mix/escripts" +for x in .dotnet/tools/devskim .config/composer/vendor/bin/psalm .mix/escripts/sobelow; do printf '#!/bin/sh\n' >"$t/home/$x"; chmod +x "$t/home/$x"; done +fail=0 +out=$(HOME=$t/home PATH=/usr/bin:/bin bash "$P") +for n in devskim psalm sobelow; do + if echo "$out" | grep -q "^ok *$n "; then echo "ok - provision sees $n"; else echo "FAIL - provision misses $n"; fail=1; fi +done +out=$(HOME=$t/home DCR_USER_BIN= PATH=/usr/bin:/bin bash "$P") +if echo "$out" | grep -qE "^ok *(devskim|psalm|sobelow)"; then echo "FAIL - empty DCR_USER_BIN must disable extra dirs"; fail=1; else echo "ok - empty DCR_USER_BIN disables extra dirs"; fi +# run review_checks.sh's own PATH prelude (lb= .. first done) and read the resulting PATH +sed -n '/^lb=/,/^done$/p' "$R" >"$t/prelude.sh" +p=$(HOME=$t/home PATH=/usr/bin:/bin bash -c 'set -u; . "$1"; echo "$PATH"' _ "$t/prelude.sh") +case $p in *"$t/home/.dotnet/tools"*"$t/home/.config/composer/vendor/bin"*"$t/home/.mix/escripts"*) echo "ok - review_checks adds toolchain dirs to PATH" ;; *) echo "FAIL - PATH=$p"; fail=1 ;; esac +# --print: gems install into DCR_USER_BIN, and the opengrep blocker is listed (stderr) +line=$(HOME=$t/home DCR_USER_BIN=$t/home/.local/bin PATH=/usr/bin:/bin bash "$P" --print 2>"$t/err") +case $line in *"gem install brakeman -v 8.1.0 -n $t/home/.local/bin"*) echo "ok - gem installs into DCR_USER_BIN" ;; *) echo "FAIL - gem line: $line"; fail=1 ;; esac +if grep -q '^BLOCKED opengrep (' "$t/err"; then echo "ok - opengrep blocker is printed"; else echo "FAIL - no opengrep BLOCKED line"; fail=1; fi +# npm rows are gated on node (npm ships with it), not on an npm binary +mkdir "$t/nodebin"; printf '#!/bin/sh\n' >"$t/nodebin/node"; chmod +x "$t/nodebin/node" +out=$(HOME=$t/home PATH=$t/nodebin:/usr/bin:/bin bash "$P" 2>&1) +if grep -q '^MISSING retire ' <<<"$out"; then echo "ok - npm rows follow node"; else echo "FAIL - retire still BLOCKED with node present"; fail=1; fi +exit $fail diff --git a/scripts/test-provision-tools.sh b/scripts/test-provision-tools.sh index 5a5910d90..c70249d14 100644 --- a/scripts/test-provision-tools.sh +++ b/scripts/test-provision-tools.sh @@ -12,12 +12,13 @@ T=$(mktemp -d "${TMPDIR:-/tmp}/dcr-test-prov.XXXXXX") && [ -n "$T" ] && [ -d "$T trap '[ -n "$T" ] && rm -rf "$T"' EXIT; fail=0 ok() { echo "ok - $1"; }; bad() { echo "FAIL - $1"; fail=1; } BASH_BIN=$(type -P bash) -ALL="gitleaks trufflehog semgrep trivy osv-scanner actionlint zizmor bandit scorecard gosec grype checkov hadolint ruff syft kube-linter kube-score tflint staticcheck brakeman kics licensee cosign opengrep nuclei" +ALL="gitleaks trufflehog semgrep trivy osv-scanner actionlint zizmor bandit scorecard gosec grype checkov hadolint ruff syft kube-linter kube-score tflint staticcheck brakeman kics licensee cosign opengrep nuclei ansible-lint cfn-lint detect-secrets flawfinder guarddog mobsfscan njsscan pyflakes slither codeql cppcheck jq kubescape pmd shellcheck tfsec node go bundler-audit rubocop cargo-audit cargo-deny govulncheck retire solhint tsc devskim psalm sobelow skillspector uv" mkdir "$T/farm" "$T/ub" "$T/tmp" "$T/proj" for c in awk bash basename cat chmod cp cut date dirname env find git grep gtimeout head jq ls mkdir mktemp mv nice paste perl python3 readlink rm sed sleep sort tail tee timeout tr uname uniq wc xargs; do p=$(type -P "$c") || continue printf '#!/bin/sh\nexec %s "$@"\n' "$p" >"$T/farm/$c"; chmod +x "$T/farm/$c" done +printf '#!/bin/sh\necho Darwin\n' >"$T/farm/uname" # the cask line below is the macOS one, whatever the host stubs() { local d t; d=$(mktemp -d "$T/st.XXXXXX"); for t in "$@"; do printf '#!/bin/sh\nexit 0\n' >"$d/$t"; chmod +x "$d/$t"; done; echo "$d"; } # run [VAR=val ...] CMD...: PA (a dir of stubs) comes first on PATH, then the wrappers; caches and ~/.local/bin are throwaway dirs. run() { env -i PATH="${PA:+$PA:}$T/farm" HOME="$T/home" TMPDIR="$T/tmp" DCR_CACHE_ROOT="$T/cache" DCR_USER_BIN="$T/ub" "$@"; } @@ -29,15 +30,15 @@ cd "$T/proj"; PHYS=$(pwd -P); q() { printf '%q' "$1"; } # 1. dry run: lists every missing tool and cold cache, exits 0, writes nothing PA=; rc=0; o=$(PT 2>&1) || rc=$? -[ "$rc" = 0 ] && [ "$(grep -c "^MISSING " <<<"$o")" = 25 ] && grep -q 'COLD trivy DB' <<<"$o" && grep -q 'COLD osv-scanner offline DB' <<<"$o" \ - && grep -q 'COLD semgrep p/default snapshot' <<<"$o" && grep -q '25 missing, 3 cold; nothing was installed' <<<"$o" && ok "dry run lists 25 missing, 3 cold" || { bad "dry run"; echo "$o"; } +[ "$rc" = 0 ] && grep -q '^MISSING gitleaks' <<<"$o" && grep -q '^MISSING codeql' <<<"$o" && grep -q 'COLD trivy DB' <<<"$o" && grep -q 'COLD osv-scanner offline DB' <<<"$o" \ + && grep -q 'COLD semgrep p/default snapshot' <<<"$o" && grep -qE '[0-9]+ missing, 3 cold; nothing was installed' <<<"$o" && ok "dry run lists named missing tools and 3 cold caches" || { bad "dry run"; echo "$o"; } [ ! -e "$T/cache" ] && [ ! -e "$T/home" ] && ok "dry run writes nothing" || bad "dry run wrote" # 2. --print: exactly one stdout line with the whole fix; the listing goes to stderr PA=; o=$(PT --print 2>"$T/e") [ "$(wc -l <<<"$o")" -eq 1 ] && ! grep -q 'MISSING' <<<"$o" && grep -q 'MISSING gitleaks' "$T/e" && ok "--print: one stdout line, listing on stderr" || { bad "--print shape"; echo "$o"; } -for want in "brew install pipx gitleaks trufflehog trivy osv-scanner actionlint scorecard gosec grype hadolint syft kube-linter kube-score tflint staticcheck kics cosign opengrep nuclei" \ - "pipx install semgrep==1.182.1; pipx install zizmor==1.31.1; pipx install bandit==1.9.4; pipx install checkov==3.4.5; pipx install ruff; gem install brakeman; gem install licensee" \ +for want in "for f in pipx ruby gitleaks trufflehog trivy osv-scanner actionlint scorecard gosec grype hadolint syft kube-linter kube-score terraform-linters/tap/tflint staticcheck kics licensee cosign nuclei" \ + "for f in semgrep==1.186.0 zizmor==1.32.0 bandit==1.9.4 checkov==3.4.5 ruff==0.17.5 ansible-lint==26.11.0" \ 'mkdir -p '"$(q "$T/cache/dcr-tools/docker-config-empty")"' && DOCKER_CONFIG='"$(q "$T/cache/dcr-tools/docker-config-empty")"' trivy --cache-dir '"$(q "$T/cache/trivy")"' fs --download-db-only' \ "OSV_SCANNER_LOCAL_DB_CACHE_DIRECTORY=$(q "$T/cache") osv-scanner --offline-vulnerabilities --download-offline-databases $(q "$PHYS")" \ "curl -fsSL https://semgrep.dev/c/p/default -o $(q "$T/cache/dcr-tools/semgrep/p-default.yml.part") && mv " ; do @@ -48,7 +49,7 @@ grep -qE '; bash [^ ]*/provision_tools\.sh$' <<<"$o" && ok "--print ends with a # 3. present tools and warm caches are left out; pipx present means no `brew install pipx` PA=$(stubs gitleaks trufflehog trivy semgrep pipx); warm_trivy o=$(PT --print 2>/dev/null) -grep -qF "brew install osv-scanner actionlint scorecard gosec grype hadolint syft kube-linter kube-score tflint staticcheck kics cosign opengrep nuclei; pipx install zizmor==1.31.1" <<<"$o" && ! grep -q 'semgrep==\|download-db-only' <<<"$o" && ok "--print omits present tools and the warm trivy DB" || { bad "partial"; echo "$o"; } +grep -qF "for f in ruby osv-scanner actionlint scorecard gosec grype hadolint syft kube-linter kube-score terraform-linters/tap/tflint staticcheck kics licensee cosign nuclei flawfinder slither-analyzer cppcheck kubescape pmd shellcheck tfsec node go cargo-audit cargo-deny govulncheck uv; do brew install \"\$f\" || echo \"FAILED \$f\"; done; if [ \"\$(uname)\" = Darwin ]; then for f in codeql; do brew install --cask \"\$f\" || echo \"FAILED \$f\"; done; fi; for f in zizmor==1.32.0" <<<"$o" && ! grep -q 'semgrep==\|download-db-only' <<<"$o" && ok "--print omits present tools and the warm trivy DB" || { bad "partial"; echo "$o"; } : >"$T/cache/trivy/db/trivy.db"; o=$(PT --print 2>/dev/null); grep -q 'download-db-only' <<<"$o" && ok "an empty trivy.db is cold" || bad "empty db warm" # 4. a tool only in ~/.local/bin (pipx, uv tool) counts as installed; DCR_USER_BIN= (empty) turns that off @@ -195,7 +196,7 @@ st '.not_run == ["trivy","osv-scanner"] and .fixable == ["trivy","osv-scanner"]' cmd=$(sed -n 's/.*(run: \(.*\))$/\1/p' "$T/err" | tail -n1); line=$(run DCR_CACHE_ROOT="$C2" "$BASH_BIN" -c "$cmd" 2>/dev/null) run DCR_CACHE_ROOT="$C2" "$BASH_BIN" -c "$line" >"$T/prov.out" 2>&1 || true cold=; for n in trivy osv semgrep; do run DCR_CACHE_ROOT="$C2" "$BASH_BIN" "$S/provision_tools.sh" --cache "$n" >/dev/null || cold+=" $n"; done -[ -z "$cold" ] && grep -qx 'pipx install zizmor==1.31.1' "$PL" && ok "the printed owner line installs and warms every cache" || { bad "owner line left cold:$cold"; cat "$T/prov.out" "$PL"; } +[ -z "$cold" ] && grep -qx 'pipx install zizmor==1.32.0' "$PL" && ok "the printed owner line installs and warms every cache" || { bad "owner line left cold:$cold"; cat "$T/prov.out" "$PL"; } rc_run DCR_CACHE_ROOT="$C2" TV_NEEDDB=1 # trivy must find the DB the owner line warmed: no loop of NOT RUN after a provision e "review_checks: ran 5 / wired 5 scanners" && ! e "fixable:" && st '.not_run == [] and .fixable == [] and .partial == []' && ok "after the owner line: ran 5 / wired 5, nothing fixable" || { bad "e2e re-run"; cat "$T/err"; } PA=$SCN diff --git a/scripts/test_audit_skills.py b/scripts/test_audit_skills.py index 19653ed05..19fd91226 100644 --- a/scripts/test_audit_skills.py +++ b/scripts/test_audit_skills.py @@ -206,7 +206,7 @@ def run(*args): "deep-code-review/references/security-agent-skills.md": "the skill-security checklist that describes these attacks", "agentic-delivery/scripts/subagent_brief_lint.py": "advisory hook; its docstring and stamp writer name the per-user cache dir where it records briefs it already flagged", "agentic-ceo/scripts/context_audit.py": "docstring listing the instruction files the read-only audit measures; the script never writes", - "deep-code-review/references/tool-hub.md": "hub doc naming the ZAP baseline container image host; documents the route, which never pulls the image silently", + "deep-code-review/references/tool-hub.md": "hub doc naming the ZAP baseline container image host (the route never pulls it silently)", "deep-code-review/scripts/provision_tools.sh": "prints, never runs, the owner line that downloads the semgrep p/default snapshot from semgrep.dev; the owner pastes it outside the sandbox", }