Skip to content

Add audit_sweep.py: recurring org or repo audit that reports only new findings - #2429

Merged
remigiusz-antczak merged 3 commits into
mainfrom
sweep-v9
Nov 19, 2026
Merged

remigiusz-antczak merged 3 commits into
mainfrom
sweep-v9

Conversation

@remigiusz-antczak

@remigiusz-antczak remigiusz-antczak commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Adds .claude/skills/deep-code-review/scripts/audit_sweep.py, a recurring org or repo audit that reports only findings it has not seen before. Closes #2282.

What it does

  • Keeps the last scanned sha and a stable fingerprint of every finding, per repo.
  • Skips an unchanged HEAD and scans a descendant HEAD over the new range only.
  • Forces a --full rescan on a history rewrite, a scanner change, or a last full scan 7 or more days old.
  • Prints unseen fingerprints only, as a hash-only TSV. It never prints finding text or a secret.
  • A failed scan prints an ERROR row, exits 1, and does not advance the state.
  • Coverage gaps (missing tool, config gap) are reported as rows. They do not block the sha.
  • An empty (unborn) repo is skipped, not reported as an ERROR.
  • State and fingerprints are keyed on org/name (absolute path for --local), so two orgs sharing a state file cannot clobber each other. Old bare-name state is still read.
  • REVIEW_SCORECARD is scrubbed from the scanner env, so a scorecard row cannot pin the sha.

Safety changes in review_checks.sh

  • The sweep runs with no GitHub token and no project test or fuzz command.
  • Every tool that executes project code is now gated behind REVIEW_TEST_CMD: checkov and tflint with a repo config, eslint-security, psalm, and ansible-lint.
  • The real-hub smoke sets REVIEW_TEST_CMD=true for its hub call, so it still runs eslint-security.
  • Stdlib python calls use -I. pyflakes is imported after the cwd entry is dropped from sys.path (not -P, which needs Python 3.11), so a planted json.py or pyflakes.py no longer runs and pyflakes still works on older Python.

Second review fixes (ab141fd)

A second independent review found 0 🔴 and 3 🟡. All three are fixed, plus one 🔵:

  • A blocking not-checked row (a scanner timeout, crash or unread file) now gets its own GAP row per tool, separate from the benign COVERAGE count. The sweep exits 1 when any repo ends blocked, so a blocked repo is never reported clean. A --full scan passes --timeout 600, so a whole-repo semgrep run is not cut off at the 60 s diff default.
  • Core scanners that the hub skips silently when missing (shellcheck, semgrep, trivy, osv-scanner, ruff/pyflakes) are named in one COVERAGE "not installed" row.
  • New test: a scanner that prints [] and then exits 3 is an ERROR, and the sweep exits 1.
  • 🔵 fixed: the pyflakes isolation change above.
    The other 🔵 notes are logged for a follow-up.

Tests

Tests: 416/420 local scripts/ci-tests.d entries. The 4 failures (g1-api10-v2, semgrep-taint, semgrep-defaults, workflow-persist-credentials) fail the same way on main in this environment: no PyYAML, semgrep 1.183.0 vs the pinned 1.182.1, and semgrep.dev unreachable. CI runs them with the pinned stack.

Fix commit: Tests: 165/165. CI-parity pre-push on ab141fd: scripts/run-ci-tests.sh --affected ran 120 suites with 0 failures (121 on ff24a93).

Two existing tests no longer depend on what is installed on the host:

  • test-semgrep-pack-fallback.sh runs on a stubs-only PATH, so a scanner installed on the host cannot change its counts.
  • test-review-checks-security-folds.sh counts only the hub's own semgrep call, because an installed njsscan adds lines to the semgrep stub log.

The review fixes have one regression test each, and each test fails on the previous audit_sweep.py.

scripts/bench_tool_only.py drops DCR_CACHE_ROOT from the hub env, so the real-scanner bench reads the host semgrep p/default snapshot instead of the empty cache dir.

🤖 Generated with Claude Code

remigiusz-antczak and others added 3 commits November 19, 2026 01:44
… findings

audit_sweep.py keeps the last scanned sha and a stable fingerprint of every
finding per repo. An unchanged HEAD is skipped, a descendant HEAD is scanned
with review_checks.sh over the new range only, and a missing or non-ancestor
sha (history rewrite), a scanner change or a last --full scan 7 or more days
old forces a --full rescan (tool versions, advisory DBs and rule snapshots
change outside the tool stamp). It prints only unseen fingerprints as a
hash-only TSV (never finding text or a secret), and a failed scan gives an
ERROR row and exit 1 without advancing the state. Each repo's rows are
flushed to stdout before its state is saved, so a kill between the two
re-reports a row instead of losing it.

Coverage gaps are classified instead of silently blocking: a tool that is
not installed is listed in a COVERAGE row without holding the sha, and
installing it later triggers one rescan; config gaps (ENVGAP), including
layout and provisioning rows a rescan cannot close (no go.mod, no psalm.xml,
no go or cargo toolchain, no local kubescape, cargo-audit or retire data),
and rows every --full scan emits by design (not applicable, noise paths) do
not hold the sha either; transient failures such as a full heavy lease still
do.

The scanner runs without GH_TOKEN, GITHUB_TOKEN, the --token-env variable,
REVIEW_TEST_CMD or REVIEW_FUZZ, so a swept repo never gets a token and never
runs its own code. To make that hold, review_checks.sh now gates every tool
that executes project code behind REVIEW_TEST_CMD, in a diff review and a
--full audit alike: checkov with a .checkov.yaml or .checkov.yml in the repo
root, tflint with a .tflint.hcl, .tflint.json or .tflint.d beside the file,
eslint-security, psalm, and ansible-lint with a project config (repo-local
tsc was already gated). Their NOT CHECKED rows are ENVGAP rows in the sweep.

review_checks.sh also words the gitleaks and trufflehog fallback row as "not
installed" or "heavy lease full" depending on whether the binary is on PATH,
so the sweep can tell a missing tool from a transient skip. The tool stamp
now also hashes templates/gitleaks.toml.

The hub cds to the scanned repo's root, and python3 with -, -c or -m puts
the cwd first on sys.path, so a committed json.py or pyflakes.py ran there.
Every stdlib-only python3 call in review_checks.sh now uses -I, and pyflakes
runs as python3 -P -m pyflakes, which drops the cwd but keeps site-packages;
Python older than 3.11 rejects -P, so pyflakes then reads as not installed.
scripts/test-review-checks-python-isolation.sh plants those modules at the
repo root and asserts they never run.

tool-hub.md gains a "Recurring audit sweep" section and documents the
project-code gates; the single size-budget-raise marker is in
changelog.d/audit-sweep.md.

Closes #2282

The real-scanner bench (scripts/bench_tool_only.py) now drops DCR_CACHE_ROOT from the hub's env, so semgrep
reads its host p/default snapshot instead of the empty cache dir scrub_env.sh points at.

Review fixes: the real-hub smoke sets REVIEW_TEST_CMD=true for its hub call, because eslint-security runs the project's
own config and binary and now needs that opt-in. audit_sweep.py skips an unborn (empty) repo instead of an ERROR row and
exit 1 on every run, scrubs REVIEW_SCORECARD from the scanner env so a scorecard row cannot pin the sha, and keys state
and fingerprints on org/name (absolute path for --local) so two orgs or two same-named repos cannot clobber each other;
old bare-name state is still read. Each fix has a regression test that fails on the previous audit_sweep.py.

Local run of every scripts/ci-tests.d test: 416/420 pass. The 4 failures (g1-api10-v2, semgrep-taint,
semgrep-defaults, workflow-persist-credentials) fail the same way on origin/main f380162 in this
environment: no PyYAML, semgrep 1.183.0 vs the pinned 1.182.1, and semgrep.dev unreachable.

After the review fixes, audit-sweep.sh and the 10 ci-tests.d entries that read the changed files were
re-run: all pass.

scripts/test-provision-toolchain-dirs.sh now unsets DCR_USER_BIN. scrub_env.sh exports it empty, which
disables the toolchain dirs, so the local --affected run failed 4 cases that CI (no scrubber) passes; it
now passes 8/8 both scrubbed and unscrubbed.

Tests: 416/420
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… pyflakes on Python < 3.11

- A blocking not-checked row (a scanner timeout, crash or unread file) now
  gets its own GAP row per tool, distinct from the benign COVERAGE count, and
  the sweep exits 1 when any repo ends blocked. A --full scan passes
  --timeout 600 so a whole-repo semgrep run is not cut off at the hub's 60 s
  diff default.
- The core scanners the hub skips silently when missing (shellcheck, semgrep,
  trivy, osv-scanner, ruff/pyflakes) are named in one COVERAGE "not installed"
  row, reusing the installed() check behind the tool stamp.
- New test: a scanner that prints [] and then exits 3 is an ERROR with exit 1.
- review_checks.sh drops the cwd entry from sys.path before importing pyflakes
  instead of using -P, so pyflakes still runs on Python before 3.11 while a
  repo's pyflakes.py still cannot shadow the linter.

Tests: 165/165
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@remigiusz-antczak
remigiusz-antczak merged commit a63506b into main Nov 19, 2026
13 checks passed
@remigiusz-antczak
remigiusz-antczak deleted the sweep-v9 branch November 19, 2026 05:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recurring audit sweep with delta-only alerts

1 participant