feat(scan): classify snippet matches as false positives (opt-in)

CHANGELOG.md

@@ -7,6 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- **Snippet classification on `scan`**, opt-in. `--classify-snippets` scores each snippet match
+  with a local classifier (github.com/scanoss/snippets-classifier) and proposes whether it is a
+  false positive, recording the verdict, the probability and the model version. It downloads
+  each matched OSS file from `/v3/file-contents/`, so it needs an API key with file-contents
+  access and counts against the quota. It works with every format: `raw` adds
+  `snippet_classification` to the evidence, CycloneDX appends it to the occurrence's
+  `additionalContext`, and SPDX, which has no per-file data, does not carry it.
+  `--classify-threshold` (default 0.7, strictly between 0 and 1) sets the probability at or above
+  which a match is proposed. `--classify-filter` removes the proposed false positives, and any
+  component and advisory left with nothing; it never removes file matches, dependencies,
+  unscored matches or `bom.identify` claims. Both need `--classify-snippets`, and a bad value is
+  rejected before anything is uploaded. A summary on stderr gives how many matches were scored,
+  proposed, removed and left unscored, and says when the API refused file contents (401/403).
+  At 0.7 the classifier proposes for a person to confirm: on 8 public application repositories
+  (876 snippet matches) its false-positive precision was about 0.71 and recall 0.90, so
+  filtering drops some real matches. Use a higher threshold such as 0.9 to filter, and do not use
+  filtering in place of review. SDK: `scanpipeline.Options.FilterFalsePositives`,
+  `Result.Classification` and `sbom.Inventory.RemoveFalsePositiveSnippets`.
+
 ## [0.15.0] - 2026-10-23
 
 ### Added

CLIENT_HELP.md

@@ -166,7 +166,9 @@ default 0), `--max-size` (0 = unlimited), `--gitignore` (true),
 `--max-components` (a positive integer or `all`; unset = the server's
 defaults), `--skip-headers`
 (**default true**, `--skip-headers=false` to disable), `--skip-headers-limit`
-(0 = no limit).
+(0 = no limit), `--classify-snippets`, `--classify-threshold` (0.7),
+`--classify-filter` (see [Snippet classification](#snippet-classification---classify-snippets);
+`scan <path>` only).
 
 `-i, --identify` and `-n, --ignore` name a component list; see
 [BOM rules](#bom-rules). `--skip-headers` applies while fingerprinting, so
@@ -432,6 +434,59 @@ CycloneDX and SPDX carry the component order but not `match_index`: a CycloneDX
 `evidence.occurrences` entry admits only `location`, `line`, `offset`, `symbol`, `additionalContext`
 and `bom-ref`, and SPDX Lite has no per-file evidence. The per-file order is `raw`-only.
 
+### Snippet classification (`--classify-snippets`)
+
+A snippet match can be a false positive: the scanner reports a local range and an OSS range that
+do not correspond. `--classify-snippets` scores every snippet match with a local classifier and
+proposes whether it is one. It is **off by default** and costs API calls: scoring needs the
+matched OSS file, which the CLI downloads from `/v3/file-contents/` once per distinct file. That
+needs an API key with file-contents access, and each download counts against the key's quota. It
+needs the scanned files on disk, so it applies to `scan <path>` only.
+
+```bash
+# Propose false positives for a person to review
+scanoss-cli scan ./my-project --api-key "$SCANOSS_API_KEY" --classify-snippets
+
+# Remove the likely ones, with a stricter threshold
+scanoss-cli scan ./my-project --api-key "$SCANOSS_API_KEY" --classify-snippets \
+  --classify-threshold 0.9 --classify-filter
+```
+
+- `--classify-threshold` is the probability of a false positive at or above which a match is
+  proposed as one, strictly between 0 and 1 (default `0.7`). Any other value is rejected before
+  anything is uploaded.
+- `--classify-filter` removes the snippet matches proposed as false positives, and a component
+  left with no evidence goes with them, along with any advisory that named only that component.
+  File matches, declared and transitive dependencies, unscored snippet matches and matches a
+  `bom.identify` rule claimed are never removed.
+- Both are rejected without `--classify-snippets`.
+
+Each scored match records the verdict (`false_positive` or `real`), the probability and the model
+version:
+
+- `raw` adds `snippet_classification` to the snippet's evidence:
+  `{"verdict": "false_positive", "probability": 0.91, "model_version": "lr-d51f2e100211"}`.
+- `cyclonedx` appends it to the occurrence's `additionalContext`, after the ranges:
+  `input lines 12-48; oss lines 100-136; classifier false_positive p=0.91 model=lr-d51f2e100211`.
+- `spdx` has no per-file data, so the verdict is not carried. `--classify-filter` still applies.
+
+A match that could not be scored (no OSS file hash, an unreadable local file, a failed download)
+carries no verdict and is never filtered. After the scan a summary on stderr says how many were
+scored, proposed and removed, and how many could not be scored:
+
+```
+Snippet classifier (model lr-d51f2e100211, threshold 0.7): scored 52 of 56 snippet matches, 6 proposed as false positives; 4 could not be scored
+```
+
+When the API answers 401 or 403 for file contents, the summary says so: the key lacks
+file-contents access, and nothing was scored.
+
+> **Review, do not trust blindly.** At `0.7` the classifier proposes false positives for a person
+> to confirm. On 8 public application repositories (876 snippet matches) it reached a
+> false-positive precision of about 0.71 and recall of 0.90 at `0.7`: roughly 3 in 10 proposed
+> matches were real. `--classify-filter` therefore drops some real matches; use a higher
+> threshold such as `0.9` when filtering, and never use filtering as a substitute for review.
+
 ## Resuming a scan (`results`)
 
 Retrieve the results of a scan by the id printed during `scan` (works after a
@@ -701,6 +756,9 @@ with `--settings`. It carries BOM context and file-skip rules.
 | `--chunk-size` (scan) | `1048576` | WFP upload block size (bytes) |
 | `--poll-interval` (scan) | `2s` | Scan status poll cadence |
 | `--max-components` (scan) | unset | Not sent; the server's default applies |
+| `--classify-snippets` (scan) | `false` | No snippet match is scored; no file contents are fetched |
+| `--classify-threshold` (scan) | `0.7` | Strictly between 0 and 1; needs `--classify-snippets` |
+| `--classify-filter` (scan) | `false` | Nothing is removed; needs `--classify-snippets` |
 | `--min-size` / `--max-size` | `0` / `0` | Literal minimum; `0` max means unlimited |
 | `--chunk-size` (decoration) | `10` | PURLs per request |
 | `--workers` (decoration) | `5` | Max concurrent requests |

README.md

@@ -103,7 +103,7 @@ Stdout stays reserved for results, so logs never corrupt `--output` or piped JSO
 
 | Command | Purpose |
 |---------|---------|
-| `scan <path>` | Fingerprint a folder/file, scan against the SCANOSS v3 API, and output results (`--format raw`/`spdx`/`cyclonedx`; opt into dependency/vuln/license/crypto/geo layers with `--include`). |
+| `scan <path>` | Fingerprint a folder/file, scan against the SCANOSS v3 API, and output results (`--format raw`/`spdx`/`cyclonedx`; opt into dependency/vuln/license/crypto/geo layers with `--include`; opt into flagging likely false-positive snippet matches with `--classify-snippets`). |
 | `scan wfp <wfp>` | Scan a pre-generated WFP file (no fingerprinting). |
 | `wfp <path>` | Generate WFP fingerprints only (no upload). |
 | `results <scan-id>` | Resume or poll a scan by its id. |

cmd/classify_test.go

@@ -0,0 +1,139 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright (c) 2026, SCANOSS
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+package cmd
+
+import (
+	"errors"
+	"strings"
+	"testing"
+
+	"github.com/scanoss/scanoss.go/pkg/scanpipeline"
+	"github.com/scanoss/scanoss.go/pkg/snippetmatch"
+)
+
+// A threshold outside (0, 1) is refused before anything is fingerprinted or uploaded.
+func TestScanRejectsClassifyThresholdOutOfRange(t *testing.T) {
+	for _, bad := range []string{"0", "1", "-0.1", "1.5", "NaN"} {
+		t.Run("value="+bad, func(t *testing.T) {
+			var rec scanRequests
+			srv := rec.server(t)
+			err := runScanDirCmd(t, srv.URL, "--classify-snippets", "--classify-threshold="+bad)
+			if err == nil || !strings.Contains(err.Error(), "--classify-threshold") {
+				t.Fatalf("error = %v, want one naming --classify-threshold", err)
+			}
+			if n := len(rec.queries); n != 0 {
+				t.Errorf("server saw %d uploads, want 0", n)
+			}
+		})
+	}
+}
+
+// --classify-threshold and --classify-filter do nothing without --classify-snippets, so they are
+// refused rather than ignored in silence.
+func TestScanClassifyFlagsRequireClassifySnippets(t *testing.T) {
+	for _, args := range [][]string{
+		{"--classify-threshold", "0.9"},
+		{"--classify-filter"},
+	} {
+		t.Run(args[0], func(t *testing.T) {
+			var rec scanRequests
+			srv := rec.server(t)
+			err := runScanDirCmd(t, srv.URL, args...)
+			if err == nil || !strings.Contains(err.Error(), args[0]+" requires --classify-snippets") {
+				t.Fatalf("error = %v, want %q", err, args[0]+" requires --classify-snippets")
+			}
+			if n := len(rec.queries); n != 0 {
+				t.Errorf("server saw %d uploads, want 0", n)
+			}
+		})
+	}
+}
+
+// Classification is no longer raw-only: every format accepts it, with a valid threshold and the
+// filter.
+func TestScanClassifyAcceptsEveryFormat(t *testing.T) {
+	for _, format := range []string{"raw", "cyclonedx", "spdx"} {
+		t.Run(format, func(t *testing.T) {
+			var rec scanRequests
+			srv := rec.server(t)
+			if err := runScanDirCmd(t, srv.URL, "-f", format, "--classify-snippets", "--classify-threshold", "0.9", "--classify-filter"); err != nil {
+				t.Fatal(err)
+			}
+			if len(rec.queries) == 0 {
+				t.Error("server saw no upload")
+			}
+		})
+	}
+}
+
+func TestClassificationSummary(t *testing.T) {
+	base := snippetmatch.Summary{ModelVersion: "lr-d51f2e100211", Threshold: 0.7}
+	with := func(f func(*scanpipeline.ClassifySummary)) scanpipeline.ClassifySummary {
+		c := scanpipeline.ClassifySummary{Summary: base}
+		f(&c)
+		return c
+	}
+	for _, tc := range []struct {
+		name     string
+		in       scanpipeline.ClassifySummary
+		filtered bool
+		want     string
+	}{
+		{
+			name: "partly scored",
+			in:   with(func(c *scanpipeline.ClassifySummary) { c.Matches, c.Scored, c.FalsePositives = 56, 52, 6 }),
+			want: "Snippet classifier (model lr-d51f2e100211, threshold 0.7): scored 52 of 56 snippet matches, 6 proposed as false positives; 4 could not be scored",
+		},
+		{
+			name:     "filtered, all scored",
+			in:       with(func(c *scanpipeline.ClassifySummary) { c.Matches, c.Scored, c.FalsePositives, c.Removed = 10, 10, 3, 3 }),
+			filtered: true,
+			want:     "Snippet classifier (model lr-d51f2e100211, threshold 0.7): scored 10 of 10 snippet matches, 3 proposed as false positives, 3 removed",
+		},
+		{
+			name: "file contents denied",
+			in: with(func(c *scanpipeline.ClassifySummary) {
+				c.Matches, c.DeniedStatus, c.Threshold = 5, 403, 0.9
+			}),
+			filtered: true,
+			want:     "Snippet classifier (model lr-d51f2e100211, threshold 0.9): scored 0 of 5 snippet matches, 0 proposed as false positives, 0 removed; 5 could not be scored (the API refused file contents with HTTP 403: the API key needs file-contents access)",
+		},
+		{
+			name: "no snippets",
+			in:   with(func(*scanpipeline.ClassifySummary) {}),
+			want: "Snippet classifier (model lr-d51f2e100211, threshold 0.7): no snippet matches to score",
+		},
+		{
+			name: "could not start",
+			in:   scanpipeline.ClassifySummary{Err: errors.New("bad model")},
+			want: "Snippet classifier could not start, no snippet match was scored: bad model",
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			if got := classificationSummary(tc.in, tc.filtered); got != tc.want {
+				t.Errorf("got  %q\nwant %q", got, tc.want)
+			}
+		})
+	}
+}