Skip to content

Commit 04e41d8

Browse files
lelialelia
authored
Fix GitLab security report schema compliance + scan alert population (#182)
* Bump incremental version Signed-off-by: lelia <2418071+lelia@users.noreply.github.com> * Fix gitlab security report schema validation errors Signed-off-by: lelia <2418071+lelia@users.noreply.github.com> * Populate gitlab security report with alerts for full scans Signed-off-by: lelia <2418071+lelia@users.noreply.github.com> * Skip license-metadata API call when fetching full scan alerts Signed-off-by: lelia <2418071+lelia@users.noreply.github.com> * Consolidate e2e test workflows, add additional coverage Signed-off-by: lelia <2418071+lelia@users.noreply.github.com> * Strip logger timestamp prefix to fix e2e test Signed-off-by: lelia <2418071+lelia@users.noreply.github.com> * Include unchanged alerts in GitLab report Signed-off-by: lelia <2418071+lelia@users.noreply.github.com> * Fix commit status to count new + unchanged alerts when strict blocking enabled Signed-off-by: lelia <2418071+lelia@users.noreply.github.com> * Add comparison table of alert behaviors between GitLab + JSON/SARIF Signed-off-by: lelia <2418071+lelia@users.noreply.github.com> * Document comment-based ignore behavior differences Signed-off-by: lelia <2418071+lelia@users.noreply.github.com> * Add new unit tests for unchanged alerts Signed-off-by: lelia <2418071+lelia@users.noreply.github.com> * chore: bump npm test fixture versions Signed-off-by: lelia <2418071+lelia@users.noreply.github.com> --------- Signed-off-by: lelia <2418071+lelia@users.noreply.github.com>
1 parent 064fb7d commit 04e41d8

File tree

16 files changed

+585
-180
lines changed

16 files changed

+585
-180
lines changed

‎.github/workflows/e2e-test.yml‎

Lines changed: 57 additions & 159 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: E2E Test
1+
name: E2E Tests
22

33
on:
44
push:
@@ -10,93 +10,58 @@ permissions:
1010
contents: read
1111

1212
jobs:
13-
e2e-scan:
14-
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
15-
runs-on: ubuntu-latest
16-
steps:
17-
- uses: actions/checkout@eef61447b9ff4aafe5dcd4e0bbf5d482be7e7871
18-
with:
19-
fetch-depth: 0
20-
persist-credentials: false
21-
22-
- uses: actions/setup-python@f677139bbe7f9c59b41e40162b753c062f5d49a3
23-
with:
24-
python-version: '3.12'
25-
26-
- name: Install CLI from local repo
27-
run: |
28-
python -m pip install --upgrade pip
29-
pip install .
30-
31-
- name: Run Socket CLI scan
32-
env:
33-
SOCKET_SECURITY_API_KEY: ${{ secrets.SOCKET_CLI_API_TOKEN }}
34-
run: |
35-
set -o pipefail
36-
socketcli \
37-
--target-path tests/e2e/fixtures/simple-npm \
38-
--disable-blocking \
39-
--enable-debug \
40-
2>&1 | tee /tmp/scan-output.log
41-
42-
- name: Verify scan produced a report
43-
run: |
44-
if grep -q "Full scan report URL: https://socket.dev/" /tmp/scan-output.log; then
45-
echo "PASS: Full scan report URL found"
46-
grep "Full scan report URL:" /tmp/scan-output.log
47-
elif grep -q "Diff Url: https://socket.dev/" /tmp/scan-output.log; then
48-
echo "PASS: Diff URL found"
49-
grep "Diff Url:" /tmp/scan-output.log
50-
else
51-
echo "FAIL: No report URL found in scan output"
52-
cat /tmp/scan-output.log
53-
exit 1
54-
fi
55-
56-
e2e-sarif:
57-
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
58-
runs-on: ubuntu-latest
59-
steps:
60-
- uses: actions/checkout@eef61447b9ff4aafe5dcd4e0bbf5d482be7e7871
61-
with:
62-
fetch-depth: 0
63-
persist-credentials: false
64-
65-
- uses: actions/setup-python@f677139bbe7f9c59b41e40162b753c062f5d49a3
66-
with:
67-
python-version: '3.12'
68-
69-
- name: Install CLI from local repo
70-
run: |
71-
python -m pip install --upgrade pip
72-
pip install .
73-
74-
- name: Run Socket CLI scan with --sarif-file
75-
env:
76-
SOCKET_SECURITY_API_KEY: ${{ secrets.SOCKET_CLI_API_TOKEN }}
77-
run: |
78-
set -o pipefail
79-
socketcli \
80-
--target-path tests/e2e/fixtures/simple-npm \
81-
--sarif-file /tmp/results.sarif \
82-
--disable-blocking \
83-
2>&1 | tee /tmp/sarif-output.log
84-
85-
- name: Verify SARIF file is valid
86-
run: |
87-
python3 -c "
88-
import json, sys
89-
with open('/tmp/results.sarif') as f:
90-
data = json.load(f)
91-
assert data['version'] == '2.1.0', f'Invalid version: {data[\"version\"]}'
92-
assert '\$schema' in data, 'Missing \$schema'
93-
count = len(data['runs'][0]['results'])
94-
print(f'PASS: Valid SARIF 2.1.0 with {count} result(s)')
95-
"
96-
97-
e2e-reachability:
13+
e2e:
9814
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
9915
runs-on: ubuntu-latest
16+
strategy:
17+
fail-fast: false
18+
matrix:
19+
include:
20+
- name: scan
21+
args: >-
22+
--target-path tests/e2e/fixtures/simple-npm
23+
--disable-blocking
24+
--enable-debug
25+
validate: tests/e2e/validate-scan.sh
26+
27+
- name: sarif
28+
args: >-
29+
--target-path tests/e2e/fixtures/simple-npm
30+
--sarif-file /tmp/results.sarif
31+
--disable-blocking
32+
validate: tests/e2e/validate-sarif.sh
33+
34+
- name: reachability
35+
args: >-
36+
--target-path tests/e2e/fixtures/simple-npm
37+
--reach
38+
--disable-blocking
39+
--enable-debug
40+
validate: tests/e2e/validate-reachability.sh
41+
setup-node: "true"
42+
43+
- name: gitlab
44+
args: >-
45+
--target-path tests/e2e/fixtures/simple-npm
46+
--enable-gitlab-security
47+
--disable-blocking
48+
validate: tests/e2e/validate-gitlab.sh
49+
50+
- name: json
51+
args: >-
52+
--target-path tests/e2e/fixtures/simple-npm
53+
--enable-json
54+
--disable-blocking
55+
validate: tests/e2e/validate-json.sh
56+
57+
- name: pypi
58+
args: >-
59+
--target-path tests/e2e/fixtures/simple-pypi
60+
--disable-blocking
61+
--enable-debug
62+
validate: tests/e2e/validate-scan.sh
63+
64+
name: e2e-${{ matrix.name }}
10065
steps:
10166
- uses: actions/checkout@eef61447b9ff4aafe5dcd4e0bbf5d482be7e7871
10267
with:
@@ -108,6 +73,7 @@ jobs:
10873
python-version: '3.12'
10974

11075
- uses: actions/setup-node@39370e3970a6d050c480ffad4ff0ed4d3fdee5af
76+
if: matrix.setup-node == 'true'
11177
with:
11278
node-version: '20'
11379

@@ -117,85 +83,17 @@ jobs:
11783
pip install .
11884
11985
- name: Install uv
86+
if: matrix.setup-node == 'true'
12087
run: pip install uv
12188

122-
- name: Run Socket CLI with reachability
89+
- name: Run Socket CLI
12390
env:
12491
SOCKET_SECURITY_API_KEY: ${{ secrets.SOCKET_CLI_API_TOKEN }}
12592
run: |
12693
set -o pipefail
127-
socketcli \
128-
--target-path tests/e2e/fixtures/simple-npm \
129-
--reach \
130-
--disable-blocking \
131-
--enable-debug \
132-
2>&1 | tee /tmp/reach-output.log
133-
134-
- name: Verify reachability analysis completed
135-
run: |
136-
if grep -q "Reachability analysis completed successfully" /tmp/reach-output.log; then
137-
echo "PASS: Reachability analysis completed"
138-
grep "Reachability analysis completed successfully" /tmp/reach-output.log
139-
grep "Results written to:" /tmp/reach-output.log || true
140-
else
141-
echo "FAIL: Reachability analysis did not complete successfully"
142-
cat /tmp/reach-output.log
143-
exit 1
144-
fi
145-
146-
- name: Verify scan produced a report
147-
run: |
148-
if grep -q "Full scan report URL: https://socket.dev/" /tmp/reach-output.log; then
149-
echo "PASS: Full scan report URL found"
150-
grep "Full scan report URL:" /tmp/reach-output.log
151-
elif grep -q "Diff Url: https://socket.dev/" /tmp/reach-output.log; then
152-
echo "PASS: Diff URL found"
153-
grep "Diff Url:" /tmp/reach-output.log
154-
else
155-
echo "FAIL: No report URL found in scan output"
156-
cat /tmp/reach-output.log
157-
exit 1
158-
fi
94+
socketcli ${{ matrix.args }} 2>&1 | tee /tmp/e2e-output.log
15995
160-
- name: Run scan with --sarif-file (all results)
96+
- name: Validate results
16197
env:
16298
SOCKET_SECURITY_API_KEY: ${{ secrets.SOCKET_CLI_API_TOKEN }}
163-
run: |
164-
socketcli \
165-
--target-path tests/e2e/fixtures/simple-npm \
166-
--reach \
167-
--sarif-file /tmp/sarif-all.sarif \
168-
--sarif-scope full \
169-
--sarif-reachability all \
170-
--disable-blocking \
171-
2>/dev/null
172-
173-
- name: Run scan with --sarif-file --sarif-reachability reachable (filtered results)
174-
env:
175-
SOCKET_SECURITY_API_KEY: ${{ secrets.SOCKET_CLI_API_TOKEN }}
176-
run: |
177-
socketcli \
178-
--target-path tests/e2e/fixtures/simple-npm \
179-
--reach \
180-
--sarif-file /tmp/sarif-reachable.sarif \
181-
--sarif-scope full \
182-
--sarif-reachability reachable \
183-
--disable-blocking \
184-
2>/dev/null
185-
186-
- name: Verify reachable-only results are a subset of all results
187-
run: |
188-
test -f /tmp/sarif-all.sarif
189-
test -f /tmp/sarif-reachable.sarif
190-
python3 -c "
191-
import json
192-
with open('/tmp/sarif-all.sarif') as f:
193-
all_data = json.load(f)
194-
with open('/tmp/sarif-reachable.sarif') as f:
195-
reach_data = json.load(f)
196-
all_count = len(all_data['runs'][0]['results'])
197-
reach_count = len(reach_data['runs'][0]['results'])
198-
print(f'All results: {all_count}, Reachable-only results: {reach_count}')
199-
assert reach_count <= all_count, f'FAIL: reachable ({reach_count}) > all ({all_count})'
200-
print('PASS: Reachable-only results is a subset of all results')
201-
"
99+
run: bash ${{ matrix.validate }}

‎docs/cli-reference.md‎

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -700,17 +700,44 @@ The GitLab report includes **actionable security alerts** based on your Socket p
700700

701701
All alert types are included in the GitLab report if they're marked as `error` or `warn` by your Socket Security policy, ensuring the Security Dashboard shows only actionable findings.
702702
703+
### Alert Population: GitLab vs JSON/SARIF
704+
705+
The GitLab Security Dashboard report and the JSON/SARIF diff outputs use different alert selection strategies, reflecting their distinct purposes:
706+
707+
| Output Format | Default Alerts | With `--strict-blocking` |
708+
|:---|:---|:---|
709+
| `--enable-gitlab-security` | **All** alerts (new + existing) | All alerts (same) |
710+
| `--enable-json` | New alerts only | New + existing alerts |
711+
| `--enable-sarif` (diff scope) | New alerts only | New + existing alerts |
712+
713+
**Why the difference?** GitLab's Security Dashboard is designed to present the full security posture of a project. An empty dashboard on a scan with no dependency changes would be misleading -- the vulnerabilities still exist, they just didn't change. By contrast, JSON and SARIF in diff scope are designed to answer "what changed?" and only include existing alerts when `--strict-blocking` explicitly requests it.
714+
715+
> **Tip:** If you use `--enable-json` alongside `--enable-gitlab-security`, the GitLab report may contain more vulnerabilities than the JSON output. This is expected. To make JSON output match, add `--strict-blocking`.
716+
717+
### Alert Ignoring via PR/MR Comments
718+
719+
When using the CLI with SCM integration (`--scm github` or `--scm gitlab`), users can ignore specific alerts by reacting to Socket's PR/MR comments. Ignored alerts are removed from `--enable-json`, `--enable-sarif`, and console output.
720+
721+
However, the GitLab Security Dashboard report includes **all** alerts matching your security policy (new and existing), regardless of comment-based ignores. This ensures the Security Dashboard always reflects the full set of known issues. To suppress a vulnerability from the GitLab report, adjust the alert's policy in Socket's dashboard rather than ignoring it via a PR comment.
722+
703723
### Report Schema
704724

705-
Socket CLI generates reports compliant with [GitLab Dependency Scanning schema version 15.0.0](https://docs.gitlab.com/ee/development/integrations/secure.html). The reports include:
725+
Socket CLI generates reports compliant with [GitLab Dependency Scanning schema version 15.0.0](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/v15.0.0/dist/dependency-scanning-report-format.json). The reports include:
706726

707-
- **Scan metadata**: Analyzer and scanner information
727+
- **Scan metadata**: Analyzer and scanner information with ISO 8601 timestamps
708728
- **Vulnerabilities**: Detailed vulnerability data with:
709729
- Unique deterministic UUIDs for tracking
710730
- Package location and dependency information
711731
- Severity levels mapped from Socket's analysis
712732
- Socket-specific alert types and CVE identifiers
713733
- Links to Socket.dev for detailed analysis
734+
- **Dependency files**: Manifest files and their dependencies discovered during the scan
735+
736+
**Schema compatibility:** The v15.0.0 schema is supported across all GitLab versions 12.0+ (both self-hosted and cloud). The report includes the `dependency_files` field, which is required by v15.0.0 and accepted as an optional extra by newer schema versions, ensuring maximum compatibility across GitLab instances.
737+
738+
### Performance Notes
739+
740+
When `--enable-gitlab-security` (or `--enable-json` / `--enable-sarif`) is used with a full scan (non-diff mode), the CLI fetches package and alert data from the scan results to populate the report. This adds time proportional to the number of packages in the scan. Without these output flags, no additional data is fetched and scan performance is unchanged.
714741
715742
### Requirements
716743
@@ -726,7 +753,9 @@ Socket CLI generates reports compliant with [GitLab Dependency Scanning schema v
726753
- Ensure the report file follows the correct schema format
727754
728755
**Empty vulnerabilities array:**
729-
- This is normal if no new security issues were detected
756+
- The GitLab report includes both new and existing alerts, so repeated scans of the same repo should still populate the report as long as Socket detects actionable issues
757+
- If the report is empty, verify the Socket dashboard shows alerts for the scanned packages -- an empty report means no error/warn-level alerts exist
758+
- For full scans (non-diff mode), ensure you are using `--enable-gitlab-security` so alert data is fetched
730759
- Check Socket.dev dashboard for full analysis details
731760
732761
## Development

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ build-backend = "hatchling.build"
66

77
[project]
88
name = "socketsecurity"
9-
version = "2.2.80"
9+
version = "2.2.81"
1010
requires-python = ">= 3.11"
1111
license = {"file" = "LICENSE"}
1212
dependencies = [

‎socketsecurity/__init__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
__author__ = 'socket.dev'
2-
__version__ = '2.2.80'
2+
__version__ = '2.2.81'
33
USER_AGENT = f'SocketPythonCLI/{__version__}'

0 commit comments

Comments
 (0)