Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
175 changes: 164 additions & 11 deletions ROADMAP.md

Large diffs are not rendered by default.

49 changes: 49 additions & 0 deletions bench/fixtures/manifests/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

62 changes: 62 additions & 0 deletions bench/fixtures/manifests/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>test.nimbus</groupId>
<artifactId>nimbus-ledger</artifactId>
<version>2.4.0</version>

<properties>
<log4j.version>2.14.1</log4j.version>
<jackson.version>2.13.0</jackson.version>
</properties>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
</dependencies>
</dependencyManagement>

<dependencies>
<!-- a ${property} that IS defined: resolved from <properties> -->
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j.version}</version>
</dependency>
<!-- version omitted, supplied by <dependencyManagement> -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>
<!-- ${project.version}: the pom's own version -->
<dependency>
<groupId>test.nimbus</groupId>
<artifactId>nimbus-common</artifactId>
<version>${project.version}</version>
</dependency>
<!-- test scope: excluded from resolution unless include-dev is set -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<!-- a ${property} that is NOT defined anywhere: must be REFUSED, never sent to OSV -->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>${spring.version}</version>
</dependency>
<!-- no version at all and not managed: inherited from a parent POM we cannot see -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
</dependencies>
</project>
3 changes: 3 additions & 0 deletions bench/fixtures/manifests/requirements-dev.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Included from requirements-with-include.txt to exercise `-r`.
pytest==7.4.3
black==23.11.0
3 changes: 3 additions & 0 deletions bench/fixtures/manifests/requirements-with-include.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
-r requirements-dev.txt
--requirement requirements.txt
mypy==1.7.1
26 changes: 26 additions & 0 deletions bench/fixtures/manifests/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Nimbus payments service — runtime dependencies.
# H2 fixture: every line here exercises one behaviour of the requirements.txt parser.

aiohttp==3.9.1 # pinned and vulnerable — the case H2 exists for
requests==2.31.0
Flask==3.0.0
PyYAML==6.0.1

# extras and case/underscore normalisation (PEP 503: Flask_Login == flask-login)
Flask_Login[extras]==0.6.2

# a marker gates WHETHER it is installed, not WHICH version — kept, and the marker recorded
importlib-metadata==6.8.0 ; python_version < "3.10"

# a hash-pinned requirement spanning continuation lines
urllib3==2.0.7 \
--hash=sha256:c97dfde1f7bd43a71c8d2a58e369e9b2bf692d1334ea9f9cae55add7d0dd8f84

# --- everything below is DELIBERATELY unresolvable, and must be reported, not guessed ---
gunicorn>=21.0 # a range: the resolver picks the version, not us
cryptography # unpinned entirely
boto3~=1.34.0 # compatible-release, still a range
django==4.2.* # wildcard is not an exact version
-e . # editable install of this project
psycopg @ https://example.test/psycopg-3.1.0.tar.gz # direct URL reference
--index-url https://pypi.example.test/simple
34 changes: 32 additions & 2 deletions docs/AUDIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,9 +187,13 @@ keeps its `run_id`, so audit entries already on disk stay joinable.
| `repo` | absolute path of the scanned repo (`root.resolve()`) |
| `repo_commit` / `repo_branch` / `repo_dirty` | `runmeta.git_provenance(repo)` — `git rev-parse HEAD`, `rev-parse --abbrev-ref HEAD`, `git status --porcelain`. **Fail-soft**: a target that is not a git checkout contributes none of these keys at all |
| `config_path` | the `config/agents*.yaml` the scan ran with — **absent** when the scan used the default config (`runmeta.write_manifest` drops `None` fields) |
| `models` | `{agent: model}` for each of `config.AGENT_NAMES` = `discover, verify, triage, generate, remediate, probe, refine` |
| `input_kind` | what was scanned: `repo`, `spec`, `manifest`, `advisory`, or a `+`-joined combination (`repo+spec`, `repo+manifest`, `repo+spec+manifest`). `advisory` is never combined — `--cve` is exclusive |
| `advisory` | H1 only: `{id, source, consulted, network_observable, fixed_version}` for a `--cve` scan |
| `spec` | H3: absolute path of the OpenAPI spec, when `--spec` was given |
| `manifests` | H2: absolute paths of the dependency manifests, when `--manifest` was given. The per-package detail is in `dependencies.json`, not here |
| `models` | `{agent: model}` for each of `config.AGENT_NAMES` = `resolve, discover, verify, triage, generate, remediate, probe, refine` |
| `caps` | `{min_confidence, max_files, max_bytes, draft_code_fixes}` — the limits the scan ran under |
| `counts` | `{candidates, verified, policies, code_fix_prs}` |
| `counts` | `{candidates, verified, policies, code_fix_prs, dependency_upgrades}`. The last two are split on `RemediationPlan.kind`: a `code_fix` is a patch drafted against a file you own and is what `pr` opens; a `dependency_upgrade` (H1/H2) is a version bump in someone else's package, so **no PR was drafted and none can be**. Counting the two together overstated the cure half of the run on every surface that renders it |
| `started` / `finished` | UTC scan bounds |
| `actor` / `host` / `tool_version` / `out_dir` | re-stamped on every manifest write |

Expand Down Expand Up @@ -326,6 +330,8 @@ ledger.json found → mitigated → remediated → retired, per fin
findings.json triage.json policies.json remediations.json
summary.json metrics.json probes.json correlations.json
lb_snapshot.json the most recent pre-change LB state
simulation.json the blast-radius replay result, when a simulation ran (G2)
dependencies.json the dependency funnel, when the scan used --manifest (H2)
report.html the standalone HTML report
policies/* the exact XC configs that were pushed
snapshots/* per-LB timestamped pre-change state (`<lb>-<UTC>.json`)
Expand All @@ -335,6 +341,30 @@ Missing members are skipped, not faked — an unscanned dir has no `findings.jso
live apply has no `snapshots/`. (`apply_timing` is an *entry inside* `audit.log`, not a member — a
CLI-driven run simply has none of those lines.)

**`dependencies.json` is not a clean bill of health**, and the manifest's `caveats` say so, because
this is the member most likely to be read as one. Four things a reviewer has to know:

- **`unpinned`** lists manifest entries whose exact version could not be established — a range like
`flask>=2.0`, an editable install, an unresolved `${property}`, a version inherited from a parent
POM. Those were **never queried**. Nothing is known about them; that is not the same as clean.
They are excluded on purpose: OSV answers a version string it cannot parse with a larger, wrong
advisory set rather than an error, so a guess would look like an answer.
- **`advisories[].disposition`** distinguishes what was assessed from what was merely found.
`exploitable` and `declined` came back from the resolve agent. `below_severity` and `capped` were
found and listed but never sent to one, so no band-aid was ever considered for them —
`settings.min_severity` and `settings.max_advisories` record the thresholds that did it.
`resolve_failed` is an advisory whose agent call errored.
- **`unchecked`** (with `funnel.packages_unchecked`) is a package OSV said has advisories and then
could not be asked about — a 429, a timeout, a transport error on the follow-up query. Its
advisories are **unknown, not absent**. Before this was its own list it appeared only as an
`error` key inside `packages[]`, contributing no row, no counter and no warning, which read
exactly like a package that came back clean.
- **`funnel`** reconciles the counts end to end, from `packages_parsed` through to `exploitable`.
Note `packages_parsed` counts the entries that **were** pinned; `packages_unpinned` is a disjoint
list, not a subset of it, so the queried count is `packages_parsed − packages_dev_excluded`.
It reflects OSV.dev **on the run date only** — `run.json`'s `finished` is the as-of timestamp, and
re-running later will legitimately produce different numbers.

`--all` produces one archive with each run under its own folder (`out-claude/`, `demo-out/`, …)
plus a top-level `index.json` listing `{out_dir, folder, events, run_id}` per run. Run dirs are
discovered by `export.find_runs`: everything matching `out*/` plus `demo/out`, keeping only dirs
Expand Down
74 changes: 73 additions & 1 deletion docs/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,78 @@ That finding is a comparison of two documents, so it skips the verify agent enti
adversarial code reviewer to confirm a vulnerability in source it cannot see got it refuted at 0.10
confidence.

### Scan a dependency manifest (H2)

`--cve` answers "can a load balancer hold the line on *this* advisory". `--manifest` asks it of
every dependency you actually have — which is the form the question normally arrives in. Nobody
hands you a CVE id; they hand you a `requirements.txt`.

```sh
vpcopilot deps ./requirements.txt # what a scan WOULD find — no model calls
vpcopilot scan --manifest ./requirements.txt --out out # the dependency tree alone
vpcopilot scan ./app --manifest ./package-lock.json --out out # …and the code, correlated together
vpcopilot scan --manifest ./requirements.txt --manifest ./pom.xml --out out # repeatable
```

`--manifest` is **additive**, like `--spec`. Alone it resolves the dependency tree; alongside a repo
the code findings and the dependency findings correlate together, so one band-aid can cover both.
Formats: `requirements.txt`, `package-lock.json` (v1/v2/v3), `pom.xml`.

**Start with `vpcopilot deps`.** It parses the manifests, asks OSV which pinned packages have
advisories, and prints the whole funnel — with no model, no credentials and no tenant. It is the
cheap way to see what a scan would cost and to tune the two knobs below before paying for one.

**Two things it will not do, and both are the point:**

- **It never guesses a version.** `flask>=2.0`, a bare `cryptography`, `-e .`, a `${spring.version}`
with no `<properties>` entry, a version inherited from a parent POM — each is listed under
`unpinned` with a reason, and never sent to OSV. This matters more than it sounds: OSV does *not*
error on a version string it cannot parse. Measured live, `aiohttp` at `not-a-version`,
`1.0.0-SNAPSHOT` and `${project.version}` each returned **81 advisories**, against 70 for the real
`3.9.1`. A guess does not fail loudly — it returns a bigger, wrong answer.
- **It never hides what it skipped.** *"We did not check this"* and *"this is clean"* must not read
the same way, so every unpinned entry and every advisory held back by the filters is in
`dependencies.json`, on the console preview and in the HTML report, carrying its reason. The same
goes for what it could not check *despite trying*: a package OSV flagged and then failed to
answer for (a 429, a timeout) is listed under `unchecked` — its advisories are unknown, not
absent — and a manifest it could not read at all is an error on every surface, never an empty
table. A `package.json` is refused outright rather than half-read: it names version *ranges*, so
there are no installed versions to check; point at `package-lock.json`.

**Bounding the agent stage.** Listing is cheap; resolving is not. `aiohttp` pinned at `3.9.1` alone
returns 70 OSV records, and a modest manifest reaches several hundred advisories. So the *listing*
is always complete and only the *resolve agent* is bounded:

| flag | default | what it does |
|---|---|---|
| `--min-severity` | `high` | floor for reaching the agent. Below it: listed, not resolved. |
| `--max-advisories` | `25` | cap on the agent stage (`0` = no cap). |
| `--include-dev` | off | also resolve dev/test-scoped deps. A build-time package is not in the request path. |

The cap is **shared out across packages**, not consumed in sort order — every vulnerable package
gives up its worst advisory before any package gives up its second. On the fixture manifests a flat
ordering gave `aiohttp` 23 of 25 slots because it sorts first and carries 35 advisories; sharing the
budget covers 7 packages instead of 3 for the same cost.

**The fixed version is code's, never a model's** — as in H1, and with one addition H2 needs. The
recommendation is the smallest published fix **strictly greater than the version you have, for the
package you have**. An advisory that names several packages fixes each at its own version (Log4Shell
fixes `log4j-core` at 2.15.0 and `pax-logging-log4j2` at 1.10.8), and one package's fix is not
installable for another. Where nothing published is newer, `dependencies.json` says so rather than
naming the closest number.

**Declining is still the load-bearing behaviour.** Most dependency advisories are not observable in
an HTTP request, and those route to `no_bandaid` in code with the residual risk stated and the
upgrade named. On the fixture manifests a typical run resolves 6 advisories to 2 exploitable and 4
declined. A tool that invented a plausible path for all 6 would be worse than no tool.

**No cure PR, and the counts say so.** As with `--cve`, the cure is a version bump in someone
else's package, so `remediate` is never called and `pr` declines with the upgrade recommendation.
Because no PR is drafted and none *can* be, an upgrade is never counted as one: `summary.json` and
`run.json` carry `code_fix_prs` and `dependency_upgrades` separately, and the report shows
"upgrades to ship (no PR)" beside the PR count rather than folding them together. Someone still has
to ship it — `reconcile` (§6) holds the band-aid and escalates at TTL.

## 4. Apply a band-aid (mutates XC — gated + reversible)
```sh
vpcopilot apply --from-scan out/policies/<artifact>.json --lb <lb> --url <host> --dry-run # preview
Expand Down Expand Up @@ -335,7 +407,7 @@ header carries a live model switcher, and each step is deep-linkable (`#mitigate

| Step | What |
|---|---|
| **① Scan** | point at a repo and run the pipeline — read-only, no XC/GitHub writes. Auto-advances to Review when it finishes |
| **① Scan** | point at a repo — or a CVE, an OpenAPI spec, or dependency manifests — and run the pipeline. Read-only, no XC/GitHub writes. **Preview (no model calls)** shows the H2 dependency funnel before you spend anything. Auto-advances to Review when it finishes |
| **② Review** | verified findings + the recommended band-aid; click a row for exploit / code / generated policy. **Open HTML report ↗** + **Download** |
| **③ Simulate** | replay a recorded sample against each candidate through a **spare** LB and report what it would block; over-threshold policies warn at the gate |
| **④ Mitigate** | apply each band-aid (or **Mitigate ALL**, one at a time, continuing past failures) and watch `before → after` stream, with a *self-healed in N attempts* badge |
Expand Down
Loading
Loading