feat(datadog_agent source): add API key validation#25536
Conversation
Add two options to the `datadog_agent` source to validate incoming Datadog API keys: - `valid_api_keys`: a list of permitted API keys. When non-empty, the API key carried by a request (URL, `dd-api-key` header, or query parameter) is checked against it. When empty (the default), all keys are accepted and no validation is performed, preserving existing behavior. - `drop_on_invalid_api_key`: when `true`, requests with an unrecognized key are rejected with a 403 Forbidden response; when `false` (the default), the unrecognized key is simply not stored but the events are accepted. This supports basic authentication and prevents unknown API keys from being used. Validation is applied across the logs, metrics, and traces endpoints. Closes vectordotdev#6809
github-actions
Bot
added
docs review on hold
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0cf42c4aff
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| header: Option<String>, | ||
| query_params: Option<String>, | ||
| ) -> ApiKeyValidation { | ||
| let api_key = self.extract(path, header, query_params); |
There was a problem hiding this comment.
When store_api_key = false, extract() returns None before it ever reads the path/header/query key, so valid_api_keys cannot be used independently of metadata storage. In a config with valid_api_keys set and drop_on_invalid_api_key = true, even requests carrying an allowed key are rejected as missing; users who disable key forwarding to sinks therefore cannot enable the new validation feature.
Useful? React with 👍 / 👎.
| /// `drop_on_invalid_api_key`. | ||
| #[configurable(metadata(docs::advanced))] | ||
| #[serde(default)] | ||
| valid_api_keys: Vec<String>, |
There was a problem hiding this comment.
Datadog API keys are credentials, but storing valid_api_keys as plain Strings means they are included verbatim in the derived Debug output for DatadogAgentConfig and the generated schema is not marked sensitive. In contexts that log or expose component config, this can leak every allow-listed key; existing credential fields in this repo use SensitiveString specifically to redact these values.
Useful? React with 👍 / 👎.
| .decode(&encoding_header, body, path.as_str()) | ||
| .and_then(|body| { | ||
| decode_log_body( | ||
| body, | ||
| source.api_key_extractor.extract( | ||
| path.as_str(), | ||
| api_token, | ||
| query_params.dd_api_key, | ||
| ), | ||
| &source, | ||
| ) | ||
| let api_key = match source.api_key_extractor.extract_and_validate( |
There was a problem hiding this comment.
With drop_on_invalid_api_key = true, this endpoint still reads and decodes/decompresses the request body before running the new API-key check. An unauthenticated request with a bad key and malformed or very large compressed body will consume decode work and return the decode error instead of the intended 403; the same post-decode validation pattern appears in the metrics and traces handlers, even though validation only needs path/header/query data.
Useful? React with 👍 / 👎.
| format!("Error decoding Datadog traces: {error:?}"), | ||
| ) | ||
| }) | ||
| let api_key = match source.api_key_extractor.extract_and_validate( |
There was a problem hiding this comment.
The new validation is only added inside the trace-payload branch, while build_warp_filter still ORs in build_stats_filter() for /api/v0.2/stats, which accepts every POST with a 200 and has no access to source.api_key_extractor. With drop_on_invalid_api_key = true, an invalid or missing key is therefore rejected for /api/v0.2/traces but still accepted on the companion stats endpoint, leaving a bypass in the advertised request-level API key restriction.
Useful? React with 👍 / 👎.
1 participant
Summary
Closes #6809.
Adds API key validation to the
datadog_agentsource, allowing it to restrict which Datadog API keys are accepted. This supports basic authentication and prevents unknown API keys from being used, as requested in the issue.Two new options are added:
valid_api_keys(list of strings, default empty): the API keys permitted to send events to this source. When non-empty, the API key carried by an incoming request — extracted from the URL path, thedd-api-keyheader, or thedd-api-keyquery parameter (the existing extraction order) — is checked against this list. When empty (the default), all API keys are accepted and no validation is performed, so existing configurations are unaffected.drop_on_invalid_api_key(bool, defaultfalse): controls behavior when a request carries an API key that is not invalid_api_keys. Whentrue, the request is rejected with a403 Forbiddenresponse. Whenfalse, the unrecognized key is simply not stored in the event metadata, but the events are still accepted. Has no effect whenvalid_api_keysis empty.Example
Implementation notes
ApiKeyExtractor::extract_and_validate, which returns anApiKeyValidationenum (Accepted(Option<key>)/Rejected). The existingextractlogic and extraction order are unchanged.403 Forbiddenerror.Arc<HashSet<String>>for cheap clones and O(1) lookups.How did you test this PR?
cargo check --lib --no-default-features --features sources-datadog_agent— passescargo test --lib --no-default-features --features sources-datadog_agent datadog_agent::— passes (40 tests, including a newapi_key_validationunit test covering: no allow list, header match, URL-path match, unknown key with drop, missing key with drop, and unknown key without drop)cargo fmt --check— passescargo clippy --lib --no-default-features --features sources-datadog_agent -- -D warnings— passesChange Type
Is this a breaking change?
When
valid_api_keysis left at its default (empty), behavior is identical to before.Does this PR include user facing changes?
no-changeloglabel to this PR.References