Skip to content

[FEAT] HTTP Headers Capture#1163

Open
marco-saia-datadog wants to merge 14 commits into
developfrom
marcosaia/RUM-13466/network-request-headers
Open

[FEAT] HTTP Headers Capture#1163
marco-saia-datadog wants to merge 14 commits into
developfrom
marcosaia/RUM-13466/network-request-headers

Conversation

@marco-saia-datadog

@marco-saia-datadog marco-saia-datadog commented Feb 26, 2026

Copy link
Copy Markdown
Member

What does this PR do?

Adds out-of-the-box HTTP request and response header capture to RUM resource events. When trackResources: true is set, users can opt in to capturing headers by setting the new trackResourceHeaders option to true in their RUM configuration. Once enabled, the headerCaptureRules option (defaults to 'defaults') controls which headers are captured.

Three modes are supported:

  • 'defaults' shortcut — captures a predefined set of 10 response headers (cache-control, etag, age, expires, content-type, content-encoding, content-length, vary, server-timing, x-cache) and 2 request headers (cache-control, content-type) across all URLs. This is the default value of headerCaptureRules once trackResourceHeaders: true is set.
  • Composable rule array — an array of HeaderCaptureRule objects, each with a discriminated type field ('defaults', 'matchHeaders', 'matchRequestHeaders', 'matchResponseHeaders'). Rules can be scoped to specific URLs via a forURLs field (same format as firstPartyHosts). Multiple matching rules are merged additively.
  • Disabled (default)trackResourceHeaders defaults to false, so no headers are captured and no overhead is added, regardless of headerCaptureRules.

Captured headers appear as _dd.request_headers and _dd.response_headers context attributes on stopResource events. These attributes are omitted entirely when there is nothing to report.

Security model

Two layers of non-overridable filtering are always applied, regardless of configuration:

  1. Sensitive header blocklist — headers matching a compiled pattern (token, cookie, secret, authorization, password, credential, bearer, API/access key variants, forwarding and IP headers) are blocked at capture time and never stored in memory.
  2. SDK tracing header exclusion — the 12 headers injected by the distributed tracing system (Datadog, W3C/OTel, B3) are unconditionally excluded.

Implementation pipeline

The feature is structured as a pure pipeline applied per-request inside the XHR proxy:

  1. compileHeaderConfig.ts — converts the user-facing headerCaptureRules config into an array of pre-built RegExp + Set<string> objects at SDK initialization (run-once cost, zero per-request config traversal).
  2. captureHeaders.ts — accumulates request headers at setRequestHeader time (after security filtering), then merges matching rules and filters by URL at request completion.
  3. parseResponseHeaders.ts — parses the raw getAllResponseHeaders() string into a Record<string, string> at response time.
  4. enforceSizeLimits.ts — caps individual header values and total header payload size before the attributes are forwarded to the native layer, preventing runaway memory and event size growth.

Configuration is snapshotted per-request at open() time, so in-flight requests are never affected by config changes.

Motivation

Caching and content negotiation issues are among the most common and hardest-to-diagnose categories of network problems in mobile apps. Headers like Cache-Control, ETag, Age, Server-Timing, and Vary carry the information needed to understand whether resources are being served from cache, how long they live, and whether CDN or server configuration is correct — but none of this was previously visible in RUM resource events without custom instrumentation.

This feature makes that class of debugging available out-of-the-box, while keeping security as a first-class constraint: sensitive headers are hard-blocked at the SDK level and cannot be accidentally enabled through misconfiguration.

Additional Notes

  • Header capture is disabled by default and opt-in only — set trackResourceHeaders: true to enable it; existing integrations are unaffected.
  • The fetch API in React Native uses XMLHttpRequest under the hood, so all fetch requests go through the same interception pipeline automatically.
  • Aborted requests (XHR status 0) skip response header capture entirely. Accumulated request headers are also discarded on abort.
  • When trackResourceHeaders is false (default), the capture path is not entered at all, regardless of headerCaptureRuleszero overhead on the XHR hot path.
  • headerCaptureRules requires trackResourceHeaders: true to take effect; it defaults to 'defaults'.
  • Full design and behavior reference: packages/core/src/rum/instrumentation/resourceTracking/headerCapture/README.md

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests
  • Make sure you discussed the feature or bugfix with the maintaining team in an Issue
  • Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)
  • If this PR is auto-generated, please make sure also to manually update the code related to the change

@marco-saia-datadog marco-saia-datadog force-pushed the marcosaia/RUM-13466/network-request-headers branch from 0cf3bbb to 1c3b17e Compare February 26, 2026 14:03
@marco-saia-datadog marco-saia-datadog force-pushed the marcosaia/RUM-13466/network-request-headers branch from 1c3b17e to f9f867a Compare March 19, 2026 10:50
@marco-saia-datadog marco-saia-datadog force-pushed the marcosaia/RUM-13466/network-request-headers branch from f9f867a to eb092fd Compare June 8, 2026 10:55
@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Jun 8, 2026

Copy link
Copy Markdown

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

DataDog/dd-sdk-reactnative | test:native-ios-newarch   View in Datadog   GitLab

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 433cef2 | Docs | Datadog PR Page | Give us feedback!

@marco-saia-datadog marco-saia-datadog force-pushed the marcosaia/RUM-13466/network-request-headers branch from be7468a to 9590fd3 Compare June 8, 2026 13:51
@marco-saia-datadog marco-saia-datadog marked this pull request as ready for review June 8, 2026 14:18
@marco-saia-datadog marco-saia-datadog requested a review from a team as a code owner June 8, 2026 14:18
Copilot AI review requested due to automatic review settings June 8, 2026 14:18
@marco-saia-datadog marco-saia-datadog requested a review from a team as a code owner June 8, 2026 14:18
@marco-saia-datadog marco-saia-datadog changed the title [FEAT] [POC] HTTP Headers Capture [FEAT] HTTP Headers Capture Jun 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds opt-in HTTP request/response header capture to RUM resource events (via XHR interception), including configuration compilation, security filtering, and size limiting before forwarding captured headers as _dd.request_headers / _dd.response_headers attributes on stopResource.

Changes:

  • Introduces headerCaptureRules RUM config (defaults shortcut + composable rules), compiles it once at init, and plumbs it through resource tracking.
  • Captures request headers from setRequestHeader, parses response headers from getAllResponseHeaders(), applies sensitive/tracing header exclusion, then enforces payload size limits.
  • Adds extensive unit/integration tests plus a feature README; exposes rule types from the package entrypoint.

Reviewed changes

Copilot reviewed 35 out of 36 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
packages/core/src/sdk/DatadogProvider/tests/initialization.test.tsx Updates provider init snapshot (adds headerCaptureRules, changes RN version handling).
packages/core/src/rum/instrumentation/resourceTracking/requestProxy/XHRProxy/XHRProxy.ts Implements per-request header capture + size limiting in XHR proxy pipeline.
packages/core/src/rum/instrumentation/resourceTracking/requestProxy/XHRProxy/DatadogRumResource/ResourceReporter.ts Adds _dd.request_headers / _dd.response_headers to stop context when present.
packages/core/src/rum/instrumentation/resourceTracking/requestProxy/XHRProxy/DatadogRumResource/tests/ResourceReporter.test.ts Tests header attributes presence/absence in stop context.
packages/core/src/rum/instrumentation/resourceTracking/requestProxy/XHRProxy/tests/XHRProxy.test.ts Adds XHRProxy-level tests for header capture, security filtering, and integration into stopResource.
packages/core/src/rum/instrumentation/resourceTracking/requestProxy/interfaces/RumResource.ts Extends resource interface with optional captured headers.
packages/core/src/rum/instrumentation/resourceTracking/requestProxy/interfaces/RequestProxy.ts Extends tracking context with compiled header-capture config.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/types.ts Defines compiled header-capture config/rule runtime types.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tracingHeaderExclusion.ts Implements explicit tracing-header exclusion set.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/sensitiveHeaderBlocklist.ts Implements sensitive-header regex blocklist.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/README.md Documents configuration, behavior, security model, and constraints.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/parseResponseHeaders.ts Adds safe parsing for getAllResponseHeaders() output.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/isHeaderAllowed.ts Composes sensitive + tracing exclusion into a single check.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/enforceSizeLimits.ts Adds truncation + count + total-budget limiting for captured headers.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/compileHeaderConfig.ts Compiles user config into URL regex + sets/maps for O(1) matching.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/captureHeaders.ts Core capture logic: accumulate request headers, capture/merge/filter response headers, apply casing rules.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/tracingHeaderExclusion.test.ts Tests tracing header exclusion list behavior.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/sensitiveHeaderBlocklist.test.ts Tests sensitive header regex behavior.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/parseResponseHeaders.test.ts Tests response header parsing edge cases.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/isHeaderAllowed.test.ts Tests composed allow/deny decision.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/enforceSizeLimits.test.ts Unit tests for truncation/count/budget behavior.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/enforceSizeLimits.integration.test.ts End-to-end test ensuring limits apply before reaching native stopResource.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/compileHeaderConfig.test.ts Tests compilation behavior for rule variants and forURLs.
packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/captureHeaders.test.ts Tests request accumulation, response capture, casing, union/precedence behavior.
packages/core/src/rum/instrumentation/resourceTracking/distributedTracing/firstPartyHosts.ts Exports escapeRegExp for reuse in header URL matching.
packages/core/src/rum/instrumentation/resourceTracking/DdRumResourceTracking.tsx Plumbs headerCaptureRules into tracking start via compilation.
packages/core/src/rum/instrumentation/resourceTracking/tests/utils/XMLHttpRequestMock.ts Adjusts mock getAllResponseHeaders default return value.
packages/core/src/index.tsx Exports header-capture rule types publicly.
packages/core/src/DdSdkReactNative.tsx Wires headerCaptureRules through initialization; warns if set without trackResources.
packages/core/src/config/features/RumConfiguration.type.ts Adds headerCaptureRules option + rule type definitions.
packages/core/src/config/features/RumConfiguration.ts Adds headerCaptureRules to defaults and configuration instance.
packages/core/src/config/async/AutoInstrumentationConfiguration.ts Adds headerCaptureRules to auto-instrumentation config shapes/defaulting.
packages/core/src/config/tests/FileBasedConfiguration.test.ts Updates snapshots for new rum config field.
packages/core/src/tests/DdSdkReactNativeConfiguration.test.ts Updates snapshots for new rum config field.
packages/core/src/tests/DdSdkReactNative.test.tsx Updates expectations to include headerCaptureRules.
.gitignore Ignores .planning directory/file.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread packages/core/src/rum/instrumentation/resourceTracking/headerCapture/README.md Outdated
Comment thread packages/core/src/config/features/RumConfiguration.type.ts
Comment thread packages/core/src/config/features/RumConfiguration.type.ts Outdated
Comment thread packages/core/src/rum/instrumentation/resourceTracking/headerCapture/README.md Outdated
Copilot AI review requested due to automatic review settings June 8, 2026 14:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 35 out of 36 changed files in this pull request and generated 7 comments.

Comment thread packages/core/src/rum/instrumentation/resourceTracking/headerCapture/README.md Outdated
Comment thread packages/core/src/config/features/RumConfiguration.type.ts
Comment thread packages/core/src/config/features/RumConfiguration.type.ts
@marco-saia-datadog marco-saia-datadog force-pushed the marcosaia/RUM-13466/network-request-headers branch from 3390b99 to 89808ed Compare June 8, 2026 14:35
Copilot AI review requested due to automatic review settings June 8, 2026 14:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 35 out of 36 changed files in this pull request and generated 7 comments.

Comment thread packages/core/src/config/features/RumConfiguration.type.ts Outdated
Comment thread packages/core/src/sdk/DatadogProvider/__tests__/initialization.test.tsx Outdated
Comment thread packages/core/src/rum/instrumentation/resourceTracking/headerCapture/README.md Outdated
Copilot AI review requested due to automatic review settings June 9, 2026 14:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 37 out of 38 changed files in this pull request and generated 4 comments.

Comment thread packages/core/src/utils/stringUtils.ts
Copilot AI review requested due to automatic review settings June 9, 2026 15:03
@DataDog DataDog deleted a comment from datadog-datadog-prod-us1-2 Bot Jun 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 37 out of 38 changed files in this pull request and generated 3 comments.

Comment thread packages/core/src/utils/stringUtils.ts
Copilot AI review requested due to automatic review settings June 9, 2026 15:13
@marco-saia-datadog marco-saia-datadog force-pushed the marcosaia/RUM-13466/network-request-headers branch from d8d9645 to 0fac228 Compare June 9, 2026 15:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 38 out of 39 changed files in this pull request and generated 2 comments.

Comment thread packages/core/src/config/features/RumConfiguration.type.ts
Comment thread packages/core/src/DdSdkReactNative.tsx Outdated
Copilot AI review requested due to automatic review settings June 9, 2026 17:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 38 out of 39 changed files in this pull request and generated 6 comments.

@cdn34dd cdn34dd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall very looks good, Great job! Although I think we should align the option name before merging.

Comment thread packages/core/src/__tests__/DdSdkReactNative.test.tsx
Copilot AI review requested due to automatic review settings July 8, 2026 08:35
@marco-saia-datadog marco-saia-datadog force-pushed the marcosaia/RUM-13466/network-request-headers branch from 0dfa9a8 to 0fa9a4b Compare July 8, 2026 08:35
@marco-saia-datadog

marco-saia-datadog commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

@cdn34dd I have introduced a trackResourceHeaders parameter, and kept headerCaptureRules as an additional optional config parameter. Here -> 433cef2

@marco-saia-datadog marco-saia-datadog force-pushed the marcosaia/RUM-13466/network-request-headers branch from 0fa9a4b to 0e3fbe4 Compare July 8, 2026 08:40
@marco-saia-datadog marco-saia-datadog force-pushed the marcosaia/RUM-13466/network-request-headers branch from 0e3fbe4 to 433cef2 Compare July 8, 2026 08:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 38 out of 39 changed files in this pull request and generated 3 comments.

Comment on lines +43 to +46
const escapedHost = escapeRegExp(hostname);
// If a path prefix is given, escape it and match as prefix.
// If hostname-only, accept /, ?, # or end-of-string as valid URL terminators.
const pathSuffix = pathPrefix ? escapeRegExp(pathPrefix) : '(/|\\?|#|$)';
Copilot AI review requested due to automatic review settings July 8, 2026 08:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 38 out of 39 changed files in this pull request and generated 4 comments.

Comment on lines +219 to +226
// Remove any existing entry with different casing for same header
// (last-value-wins: both value and casing from latest call)
for (const existing of Object.keys(accumulator)) {
if (existing.toLowerCase() === lowered) {
delete accumulator[existing];
break;
}
}
Comment on lines +391 to +399
// Accumulate for header capture (only user-set, non-Datadog headers)
if (this._datadog_xhr?.capturedRequestHeaders !== undefined) {
accumulateRequestHeader(
this._datadog_xhr.capturedRequestHeaders,
header,
value
);
}
return result;
Comment on lines +23 to +24
* Silent filtering — no debug log for blocked headers (these are expected
* normal behavior, not warnings).
});
});

/** Helper: compute total bytes for all headers */
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants