Repository navigation
Add audit_sweep.py: recurring org or repo audit that reports only new findings - #2429
Merged
Merged
Conversation
… findings audit_sweep.py keeps the last scanned sha and a stable fingerprint of every finding per repo. An unchanged HEAD is skipped, a descendant HEAD is scanned with review_checks.sh over the new range only, and a missing or non-ancestor sha (history rewrite), a scanner change or a last --full scan 7 or more days old forces a --full rescan (tool versions, advisory DBs and rule snapshots change outside the tool stamp). It prints only unseen fingerprints as a hash-only TSV (never finding text or a secret), and a failed scan gives an ERROR row and exit 1 without advancing the state. Each repo's rows are flushed to stdout before its state is saved, so a kill between the two re-reports a row instead of losing it. Coverage gaps are classified instead of silently blocking: a tool that is not installed is listed in a COVERAGE row without holding the sha, and installing it later triggers one rescan; config gaps (ENVGAP), including layout and provisioning rows a rescan cannot close (no go.mod, no psalm.xml, no go or cargo toolchain, no local kubescape, cargo-audit or retire data), and rows every --full scan emits by design (not applicable, noise paths) do not hold the sha either; transient failures such as a full heavy lease still do. The scanner runs without GH_TOKEN, GITHUB_TOKEN, the --token-env variable, REVIEW_TEST_CMD or REVIEW_FUZZ, so a swept repo never gets a token and never runs its own code. To make that hold, review_checks.sh now gates every tool that executes project code behind REVIEW_TEST_CMD, in a diff review and a --full audit alike: checkov with a .checkov.yaml or .checkov.yml in the repo root, tflint with a .tflint.hcl, .tflint.json or .tflint.d beside the file, eslint-security, psalm, and ansible-lint with a project config (repo-local tsc was already gated). Their NOT CHECKED rows are ENVGAP rows in the sweep. review_checks.sh also words the gitleaks and trufflehog fallback row as "not installed" or "heavy lease full" depending on whether the binary is on PATH, so the sweep can tell a missing tool from a transient skip. The tool stamp now also hashes templates/gitleaks.toml. The hub cds to the scanned repo's root, and python3 with -, -c or -m puts the cwd first on sys.path, so a committed json.py or pyflakes.py ran there. Every stdlib-only python3 call in review_checks.sh now uses -I, and pyflakes runs as python3 -P -m pyflakes, which drops the cwd but keeps site-packages; Python older than 3.11 rejects -P, so pyflakes then reads as not installed. scripts/test-review-checks-python-isolation.sh plants those modules at the repo root and asserts they never run. tool-hub.md gains a "Recurring audit sweep" section and documents the project-code gates; the single size-budget-raise marker is in changelog.d/audit-sweep.md. Closes #2282 The real-scanner bench (scripts/bench_tool_only.py) now drops DCR_CACHE_ROOT from the hub's env, so semgrep reads its host p/default snapshot instead of the empty cache dir scrub_env.sh points at. Review fixes: the real-hub smoke sets REVIEW_TEST_CMD=true for its hub call, because eslint-security runs the project's own config and binary and now needs that opt-in. audit_sweep.py skips an unborn (empty) repo instead of an ERROR row and exit 1 on every run, scrubs REVIEW_SCORECARD from the scanner env so a scorecard row cannot pin the sha, and keys state and fingerprints on org/name (absolute path for --local) so two orgs or two same-named repos cannot clobber each other; old bare-name state is still read. Each fix has a regression test that fails on the previous audit_sweep.py. Local run of every scripts/ci-tests.d test: 416/420 pass. The 4 failures (g1-api10-v2, semgrep-taint, semgrep-defaults, workflow-persist-credentials) fail the same way on origin/main f380162 in this environment: no PyYAML, semgrep 1.183.0 vs the pinned 1.182.1, and semgrep.dev unreachable. After the review fixes, audit-sweep.sh and the 10 ci-tests.d entries that read the changed files were re-run: all pass. scripts/test-provision-toolchain-dirs.sh now unsets DCR_USER_BIN. scrub_env.sh exports it empty, which disables the toolchain dirs, so the local --affected run failed 4 cases that CI (no scrubber) passes; it now passes 8/8 both scrubbed and unscrubbed. Tests: 416/420 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… pyflakes on Python < 3.11 - A blocking not-checked row (a scanner timeout, crash or unread file) now gets its own GAP row per tool, distinct from the benign COVERAGE count, and the sweep exits 1 when any repo ends blocked. A --full scan passes --timeout 600 so a whole-repo semgrep run is not cut off at the hub's 60 s diff default. - The core scanners the hub skips silently when missing (shellcheck, semgrep, trivy, osv-scanner, ruff/pyflakes) are named in one COVERAGE "not installed" row, reusing the installed() check behind the tool stamp. - New test: a scanner that prints [] and then exits 3 is an ERROR with exit 1. - review_checks.sh drops the cwd entry from sys.path before importing pyflakes instead of using -P, so pyflakes still runs on Python before 3.11 while a repo's pyflakes.py still cannot shadow the linter. Tests: 165/165 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
.claude/skills/deep-code-review/scripts/audit_sweep.py, a recurring org or repo audit that reports only findings it has not seen before. Closes #2282.What it does
--fullrescan on a history rewrite, a scanner change, or a last full scan 7 or more days old.--local), so two orgs sharing a state file cannot clobber each other. Old bare-name state is still read.REVIEW_SCORECARDis scrubbed from the scanner env, so a scorecard row cannot pin the sha.Safety changes in
review_checks.shREVIEW_TEST_CMD: checkov and tflint with a repo config, eslint-security, psalm, and ansible-lint.REVIEW_TEST_CMD=truefor its hub call, so it still runs eslint-security.-I. pyflakes is imported after the cwd entry is dropped fromsys.path(not-P, which needs Python 3.11), so a plantedjson.pyorpyflakes.pyno longer runs and pyflakes still works on older Python.Second review fixes (ab141fd)
A second independent review found 0 🔴 and 3 🟡. All three are fixed, plus one 🔵:
not-checkedrow (a scanner timeout, crash or unread file) now gets its ownGAProw per tool, separate from the benignCOVERAGEcount. The sweep exits 1 when any repo ends blocked, so a blocked repo is never reported clean. A--fullscan passes--timeout 600, so a whole-repo semgrep run is not cut off at the 60 s diff default.COVERAGE"not installed" row.[]and then exits 3 is anERROR, and the sweep exits 1.The other 🔵 notes are logged for a follow-up.
Tests
Tests: 416/420 local
scripts/ci-tests.dentries. The 4 failures (g1-api10-v2, semgrep-taint, semgrep-defaults, workflow-persist-credentials) fail the same way onmainin this environment: no PyYAML, semgrep 1.183.0 vs the pinned 1.182.1, and semgrep.dev unreachable. CI runs them with the pinned stack.Fix commit: Tests: 165/165. CI-parity pre-push on ab141fd:
scripts/run-ci-tests.sh --affectedran 120 suites with 0 failures (121 on ff24a93).Two existing tests no longer depend on what is installed on the host:
test-semgrep-pack-fallback.shruns on a stubs-only PATH, so a scanner installed on the host cannot change its counts.test-review-checks-security-folds.shcounts only the hub's own semgrep call, because an installed njsscan adds lines to the semgrep stub log.The review fixes have one regression test each, and each test fails on the previous
audit_sweep.py.scripts/bench_tool_only.pydropsDCR_CACHE_ROOTfrom the hub env, so the real-scanner bench reads the host semgrepp/defaultsnapshot instead of the empty cache dir.🤖 Generated with Claude Code