-
Notifications
You must be signed in to change notification settings - Fork 7
Extend ZAP scan to provider users and state API M2M (#1467, #1468) #1470
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
80ac360
e235d32
d7c1e27
b417642
105edda
615c41a
8373eb6
2b7df02
848d5d5
6b42fe9
76b76de
dd41ce5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,38 +1,115 @@ | ||
| # OWASP Zed Attack Proxy | ||
|
|
||
| [ZAP](https://www.zaproxy.org/) is a security pen testing tool, which this project uses for active scanning for vulnerabilities. To manually run a scan similar to the ones integrated into [our GitHub Actions](../.github/workflows), run `./manual-scan.sh` from a command line. For full details on running the scan locally, see [Manual run ](#manual-run). Below is a brief explanation of the pieces of this integration and what they are for. | ||
| [ZAP](https://www.zaproxy.org/) is a security pen testing tool, which this project uses for active scanning for vulnerabilities. To manually run a scan similar to the ones integrated into [our GitHub Actions](../.github/workflows), run `./manual-scan.sh` from a command line. For full details on running the scan locally, see [Manual run](#manual-run). Below is a brief explanation of the pieces of this integration and what they are for. | ||
|
|
||
| # Authenticator | ||
|
|
||
| The [authenticator](./authenticator) directory is a simple NodeJS script that leverages [aws-amplify](https://www.npmjs.com/package/aws-amplify) to quickly log a test user in and acquire an token for authentication. This script requires that the Cognito UserPool Client be configured to enable the [Secure Remote Pasword (SRP)](https://en.wikipedia.org/wiki/Secure_Remote_Password_protocol) authentication flow as part of a 'VULNERABLE' security profile, which can only be activated in pre-production environments. See [Set up](#set-up) for discussion of the profile. | ||
| The [authenticator](./authenticator) directory contains helpers for obtaining bearer tokens against the three Cognito pools CompactConnect uses: | ||
|
|
||
| - `main.js` — a NodeJS script leveraging [aws-amplify](https://www.npmjs.com/package/aws-amplify) for user sign-in against the **Staff Users** and **Provider Users** pools. It accepts `--mode=staff` or `--mode=provider` to select which set of `.env` variables to use. Both pools must have the [Secure Remote Password (SRP)](https://en.wikipedia.org/wiki/Secure_Remote_Password_protocol) flow enabled, which happens automatically under the `VULNERABLE` security profile (see [Set up](#set-up)). | ||
| - `get-m2m-token.sh` — a shell script that performs the OAuth2 `client_credentials` grant against the **State Auth** pool for machine-to-machine access to `state-api.test.compactconnect.org`. | ||
|
|
||
| # Data | ||
|
|
||
| The [data](./data) folder contains configuration and automation data files that are used to control ZAP's scan. This includes an HttpSender script and an automation YML file. The HTTPSender script will tell ZAP to add the token we acquired from the [authenticator](#authenticator) script into the `Authorization` header for every 'in scope' request, headed to the UI or API. The YML file defines what jobs are to be run as part of an automated scan. | ||
| The [data](./data) folder contains configuration and automation data files that are used to control ZAP's scan. This includes an HttpSender script and an automation YML file. The HttpSender script (`bearer-token.js`) routes requests to the correct bearer token based on hostname + path: requests to the state API host get the M2M token, requests under `/v1/provider-users/*`, `/v1/purchases/*`, and `GET /v1/compacts/{compact}/attestations/{attestationId}` get the provider token, everything else gets the staff token. State API requests additionally get ECDSA-SHA256 request signatures (`X-Algorithm`, `X-Timestamp`, `X-Nonce`, `X-Key-Id`, `X-Signature`) computed per-request against the signing private key — see [client_signature_auth.md](../backend/compact-connect/docs/client_signature_auth.md). The YML file defines the ZAP automation plan. | ||
|
|
||
| # Set up | ||
|
|
||
| In order for the scan to be able to run successfully, the scanned environment requires some set-up: | ||
| 1) The target environment backend needs to be deployed with the `"security_profile": "VULNERABLE"` environment context set in order to prevent ZAP from being locked out. The `VULNERABLE` profile weakens a number of security elements to allow for the scan including: | ||
| - Removing the rate limit from the WAF policies | ||
| - Enabling the SRP authentication flow | ||
| - Disabling Cognito Advanced Security | ||
| - Removing MFA requirements | ||
| In order for the scan to run successfully, the target environment needs some set-up: | ||
|
|
||
| 1. The target environment backend needs to be deployed with the `"security_profile": "VULNERABLE"` environment context set in order to prevent ZAP from being locked out. The `VULNERABLE` profile weakens a number of security elements to allow for the scan, including: | ||
| - Removing the rate limit from the WAF policies | ||
| - Enabling the SRP authentication flow (for both staff and provider pools) | ||
| - Disabling Cognito Advanced Security | ||
| - Removing MFA requirements | ||
|
|
||
| Because of this loosened security for scanning, the `VULNERABLE` security profile cannot be used in the production environment. | ||
| 2) Create a dedicated test user in the StaffUsers Cognito UserPool | ||
| 3) Define the following secrets in the GitHub repository with the corresponding information from the staff user pool, client, and test user: | ||
|
|
||
| ``` | ||
| TEST_COGNITO_USER_POOL_ID_STAFF | ||
| TEST_WEBROOT_COGNITO_CLIENT_ID_STAFF | ||
| TEST_ZAP_USERNAME_STAFF | ||
| TEST_ZAP_PASSWORD_STAFF | ||
| ``` | ||
| 2. **Staff Users pool** — create a dedicated test user with broad scope coverage (e.g. `aslp/admin` plus full `oh` and `ky` jurisdiction permissions). Define these GitHub secrets: | ||
|
|
||
| ``` | ||
| TEST_COGNITO_USER_POOL_ID_STAFF | ||
| TEST_WEBROOT_COGNITO_CLIENT_ID_STAFF | ||
| TEST_ZAP_USERNAME_STAFF | ||
| TEST_ZAP_PASSWORD_STAFF | ||
| ``` | ||
|
Comment on lines
+30
to
+35
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Add language identifiers to fenced code blocks (markdownlint MD040). These code fences should specify a language to satisfy linting and keep docs consistent. Suggested doc patch- ```
+ ```text
TEST_COGNITO_USER_POOL_ID_STAFF
TEST_WEBROOT_COGNITO_CLIENT_ID_STAFF
TEST_ZAP_USERNAME_STAFF
TEST_ZAP_PASSWORD_STAFF
```
@@
- ```
+ ```text
TEST_COGNITO_USER_POOL_ID_PROVIDER
TEST_COGNITO_CLIENT_ID_PROVIDER
TEST_ZAP_USERNAME_PROVIDER
TEST_ZAP_PASSWORD_PROVIDER
```
@@
- ```
+ ```bash
python3 backend/compact-connect/app_clients/bin/create_app_client.py -u <state-auth-pool-id>
```
@@
- ```
+ ```text
TEST_COGNITO_STATE_AUTH_DOMAIN (e.g. compact-connect-state-auth-test.auth.us-east-1.amazoncognito.com)
TEST_ZAP_STATE_AUTH_CLIENT_ID
TEST_ZAP_STATE_AUTH_CLIENT_SECRET
TEST_ZAP_STATE_AUTH_SCOPES (space-separated, e.g. "aslp/readGeneral ky/aslp.write oh/aslp.write")
```Also applies to: 39-44, 56-58, 66-71 🧰 Tools🪛 markdownlint-cli2 (0.22.0)[warning] 30-30: Fenced code blocks should have a language specified (MD040, fenced-code-language) 🤖 Prompt for AI Agents |
||
|
|
||
| 3. **Provider Users pool** — create a dedicated test provider user. The user needs a backing provider record in DynamoDB with a license in a covered jurisdiction (e.g. `ky` or `oh` within ASLP) so the `/v1/provider-users/me/*` endpoints resolve. ZAP only needs handlers to execute — response codes don't matter for vulnerability scanning, so POST endpoints (`/v1/purchases/privileges`, `/v1/provider-users/me/military-affiliation`) returning 4xx on repeat runs because records already exist is fine; no cleanup is required between scans. Define these GitHub secrets: | ||
|
|
||
| ``` | ||
| TEST_COGNITO_USER_POOL_ID_PROVIDER | ||
| TEST_COGNITO_CLIENT_ID_PROVIDER | ||
| TEST_ZAP_USERNAME_PROVIDER | ||
| TEST_ZAP_PASSWORD_PROVIDER | ||
| ``` | ||
|
|
||
| 4. **State Auth M2M pool** — provision an app client dedicated to ZAP scanning. This client needs two things that differ from a standard state onboarding client: | ||
|
|
||
| - A 60-minute access token validity (vs. the 15-minute default) so one token covers a scan run under the 45-minute active scan cap. | ||
| - Scopes to exercise all four state API endpoints: `{compact}/readGeneral` plus `{state}/{compact}.write` for each covered jurisdiction. | ||
|
|
||
| Rather than creating with defaults and patching after (`update-user-pool-client` is a full-replacement API — any attribute omitted is reset to its default, which is easy to get wrong), temporarily bump the validity in the creation script: | ||
|
|
||
| 1. In `backend/compact-connect/app_clients/bin/create_app_client.py`, change `BASE_CLIENT_CONFIG['AccessTokenValidity']` from `15` to `60`. **Do not commit this change** — it's a one-shot override for this ZAP client. | ||
| 2. Run the script against the test StateAuthUsers pool: | ||
|
|
||
| ``` | ||
| python3 backend/compact-connect/app_clients/bin/create_app_client.py -u <state-auth-pool-id> | ||
| ``` | ||
|
|
||
| At the prompts: client name `owasp-zap-v1` (increment the version on rotation), compact `aslp`, state `ky`, additional scopes `aslp/readGeneral,oh/aslp.write`. Capture the `clientId` / `clientSecret` from the output. | ||
| 3. Revert the `AccessTokenValidity` change in `create_app_client.py`. | ||
| 4. Verify the client with `aws cognito-idp describe-user-pool-client` — you should see `AccessTokenValidity: 60` and `AllowedOAuthScopes` containing `aslp/readGeneral`, `ky/aslp.write`, `oh/aslp.write`. | ||
|
|
||
| Then define these GitHub secrets: | ||
|
|
||
| ``` | ||
| TEST_COGNITO_STATE_AUTH_DOMAIN (e.g. compact-connect-state-auth-test.auth.us-east-1.amazoncognito.com) | ||
| TEST_ZAP_STATE_AUTH_CLIENT_ID | ||
| TEST_ZAP_STATE_AUTH_CLIENT_SECRET | ||
| TEST_ZAP_STATE_AUTH_SCOPES (space-separated, e.g. "aslp/readGeneral ky/aslp.write oh/aslp.write") | ||
| ``` | ||
|
|
||
| 5. **State API request signing (ECDSA)** — state-api reads of PII-heavy data (`/providers/query`, `/providers/{id}`) require an ECDSA signature on top of the M2M bearer token per [`client_signature_auth.md`](../backend/compact-connect/docs/client_signature_auth.md). Without it, those endpoints return `401 "Missing required X-Key-Id header"` and ZAP can only probe the authorizer/WAF layer. | ||
|
|
||
| We scan with signatures enabled to simulate the full insider-threat model (an attacker with access to a state IT org's credentials and signing key). Once configured, this also turns on signature enforcement for *optional*-signature endpoints like `POST /licenses`. | ||
|
|
||
| Provisioning steps: | ||
|
|
||
| 1. **Generate an ECDSA P-256 keypair in PKCS#8 PEM format** (Java loads PKCS#8 natively; OpenSSL's default `ecparam` output is SEC1, which requires an extra conversion step): | ||
|
|
||
| ```bash | ||
| openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out zap_test_private_key.pem | ||
| openssl pkey -in zap_test_private_key.pem -pubout -out zap_test_public_key.pub | ||
| ``` | ||
|
|
||
| 2. **Register the public key** for each (compact, jurisdiction) that the ZAP client will probe. With the defaults used above (`aslp`, `ky`, `oh`), that's two registrations. Rename `zap_test_public_key.pub` to `<key-id>.pub` first because the script infers the key ID from the filename: | ||
|
|
||
| ```bash | ||
| cp zap_test_public_key.pub zap-test-v1.pub | ||
| python3 backend/compact-connect/app_clients/bin/manage_signature_keys.py create \ | ||
| -t <compact-configuration-table-name-for-test-env> | ||
| # When prompted: compact=aslp, state=ky, key id=zap-test-v1 | ||
| # Repeat the create step for state=oh | ||
| ``` | ||
|
|
||
| 3. **Define these GitHub secrets**: | ||
|
|
||
| ``` | ||
| TEST_ZAP_STATE_SIGNATURE_KEY_ID (e.g. zap-test-v1) | ||
| TEST_ZAP_STATE_SIGNATURE_PRIVATE_KEY (paste the full PKCS#8 PEM contents, including the BEGIN/END lines) | ||
| ``` | ||
|
|
||
| GitHub Actions secrets support multi-line values; paste the PEM as-is. | ||
|
|
||
| 4. **Clean up the private key file from your local disk** once it's stored in GitHub secrets. Do not commit it. | ||
|
|
||
| Rotation: generate a new keypair with a new key ID (`zap-test-v2`), register it alongside the old one, update the secrets, then run `manage_signature_keys.py delete` on the old key. Signature validation supports multiple active keys during rollover. | ||
|
|
||
| # Manual run | ||
|
|
||
| If you wish to run the ZAP scan locally, you can do so buy following these steps: | ||
| 1) Install `docker` [docker](https://www.docker.com/) on your computer. | ||
| 2) From inside the [authenticator](./authenticator) folder, copy `env.example` to `.env` and update the example values with the real values from the StaffUser user pool and the test user created in [Set up](#set-up). | ||
| 3) From the repository root, run `./owasp-zap/manual-run.sh`. The scan will run inside a docker container and the report will be added to a `reports` folder at the repository root. | ||
| To run the ZAP scan locally: | ||
|
|
||
| 1. Install [docker](https://www.docker.com/). | ||
| 2. From inside the [authenticator](./authenticator) folder, copy `.env.example` to `.env` and fill in whichever credential sets you have. Missing sets (e.g. you only have staff creds) are skipped with a warning — the scan still runs, but endpoints needing the missing tokens will return 401. To also cover signature-gated state endpoints, set `STATE_SIGNATURE_KEY_ID` and paste your PKCS#8 PEM into `STATE_SIGNATURE_PRIVATE_KEY` (multi-line quoted value — see `.env.example`). | ||
| 3. From the repository root, run `./owasp-zap/manual-scan.sh`. The scan runs inside a Docker container and the report is written to a `report/` folder at the repository root. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,28 @@ | ||
| COGNITO_USER_POOL_ID=us-east-1_abcdefg | ||
| COGNITO_USER_POOL_CLIENT_ID=abcdefghijklmnop1234567890 | ||
| COGNITO_USERNAME=user@example.com | ||
| COGNITO_PASSWORD=123imapassword456 | ||
| # Copy to .env and fill in real values for whichever credential sets you want to scan. | ||
| # Missing sets will be skipped by manual-scan.sh with a warning. | ||
|
|
||
| # Staff Users pool (most endpoints on api.test.compactconnect.org) | ||
| STAFF_COGNITO_USER_POOL_ID=us-east-1_abcdefg | ||
| STAFF_COGNITO_USER_POOL_CLIENT_ID=abcdefghijklmnop1234567890 | ||
| STAFF_COGNITO_USERNAME=staff-zap@example.com | ||
| STAFF_COGNITO_PASSWORD=123imapassword456 | ||
|
|
||
| # Provider Users pool (/v1/provider-users/*, /v1/purchases/*, GET attestation-by-id) | ||
| PROVIDER_COGNITO_USER_POOL_ID=us-east-1_hijklmn | ||
| PROVIDER_COGNITO_USER_POOL_CLIENT_ID=qrstuvwxyz0987654321 | ||
| PROVIDER_COGNITO_USERNAME=provider-zap@example.com | ||
| PROVIDER_COGNITO_PASSWORD=456imapassword789 | ||
|
|
||
| # State Auth M2M pool (state-api.test.compactconnect.org) | ||
| COGNITO_STATE_AUTH_DOMAIN=compact-connect-state-auth-test.auth.us-east-1.amazoncognito.com | ||
| STATE_AUTH_CLIENT_ID=xxxxxxxxxxxxxxxxxxxxxxxxxx | ||
| STATE_AUTH_CLIENT_SECRET=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy | ||
| STATE_AUTH_SCOPES=aslp/readGeneral ky/aslp.write oh/aslp.write | ||
|
|
||
| # State API ECDSA signature auth (optional — required for /providers/query and /providers/{id} | ||
| # coverage). See owasp-zap/README.md for key generation and registration steps. | ||
| # The private key must be in PKCS#8 PEM format. | ||
| STATE_SIGNATURE_KEY_ID=zap-test-v1 | ||
| STATE_SIGNATURE_PRIVATE_KEY="-----BEGIN PRIVATE KEY----- | ||
| MIG...replace with actual key contents... | ||
| -----END PRIVATE KEY-----" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| #!/usr/bin/env bash | ||
| # | ||
| # Fetches a Cognito client_credentials access token for the State Auth M2M pool. | ||
| # Prints only the access_token to stdout on success. | ||
| # | ||
| # Requires: | ||
| # COGNITO_STATE_AUTH_DOMAIN — e.g. compact-connect-state-auth-test.auth.us-east-1.amazoncognito.com | ||
| # STATE_AUTH_CLIENT_ID | ||
| # STATE_AUTH_CLIENT_SECRET | ||
| # STATE_AUTH_SCOPES — space-separated list, e.g. "aslp/readGeneral ky/aslp.write" | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| : "${COGNITO_STATE_AUTH_DOMAIN:?COGNITO_STATE_AUTH_DOMAIN is required}" | ||
| : "${STATE_AUTH_CLIENT_ID:?STATE_AUTH_CLIENT_ID is required}" | ||
| : "${STATE_AUTH_CLIENT_SECRET:?STATE_AUTH_CLIENT_SECRET is required}" | ||
| : "${STATE_AUTH_SCOPES:?STATE_AUTH_SCOPES is required}" | ||
|
|
||
| response=$(curl -sS --fail-with-body \ | ||
| -X POST "https://${COGNITO_STATE_AUTH_DOMAIN}/oauth2/token" \ | ||
| -u "${STATE_AUTH_CLIENT_ID}:${STATE_AUTH_CLIENT_SECRET}" \ | ||
| -H "Content-Type: application/x-www-form-urlencoded" \ | ||
| --data-urlencode "grant_type=client_credentials" \ | ||
| --data-urlencode "scope=${STATE_AUTH_SCOPES}") | ||
|
|
||
| token=$(jq -r '.access_token // empty' <<<"$response") | ||
| if [[ -z "$token" ]]; then | ||
| echo "Failed to obtain M2M token. Response: $response" >&2 | ||
| exit 1 | ||
| fi | ||
|
Comment on lines
+19
to
+30
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
With 🛡️ Suggested fix-response=$(curl -sS --fail-with-body \
- -X POST "https://${COGNITO_STATE_AUTH_DOMAIN}/oauth2/token" \
- -u "${STATE_AUTH_CLIENT_ID}:${STATE_AUTH_CLIENT_SECRET}" \
- -H "Content-Type: application/x-www-form-urlencoded" \
- --data-urlencode "grant_type=client_credentials" \
- --data-urlencode "scope=${STATE_AUTH_SCOPES}")
-
-token=$(jq -r '.access_token // empty' <<<"$response")
-if [[ -z "$token" ]]; then
- echo "Failed to obtain M2M token. Response: $response" >&2
- exit 1
-fi
-printf '%s' "$token"
+http_status=0
+response=$(curl -sS -w '\n%{http_code}' \
+ -X POST "https://${COGNITO_STATE_AUTH_DOMAIN}/oauth2/token" \
+ -u "${STATE_AUTH_CLIENT_ID}:${STATE_AUTH_CLIENT_SECRET}" \
+ -H "Content-Type: application/x-www-form-urlencoded" \
+ --data-urlencode "grant_type=client_credentials" \
+ --data-urlencode "scope=${STATE_AUTH_SCOPES}") || true
+http_status="${response##*$'\n'}"
+body="${response%$'\n'*}"
+
+if [[ "$http_status" != 2* ]]; then
+ echo "Failed to obtain M2M token (HTTP $http_status). Response: $body" >&2
+ exit 1
+fi
+
+token=$(jq -r '.access_token // empty' <<<"$body")
+if [[ -z "$token" ]]; then
+ echo "Token missing from response: $body" >&2
+ exit 1
+fi
+printf '%s' "$token"🤖 Prompt for AI Agents |
||
| printf '%s' "$token" | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reminder: revert the temporary
pull_requesttrigger before merge.Per the PR description, this trigger is intended to be removed prior to merge. Two additional notes while it's present:
pull_requestevents from forked repos do not have access to repository secrets, so the three authentication steps (and thus the whole job) will hard-fail for any external contributor PR.workflow_dispatchgate if this is ever made permanent.Want me to draft a follow-up commit that removes the
pull_request:trigger (or replaces it with a path-filtered, label-gated variant) once the end-to-end validation is complete?🤖 Prompt for AI Agents