Skip to content

Add HTTPXWrapper behind use_httpx flag#23822

Draft
mwdd146980 wants to merge 8 commits into
mwdd146980/httpx-migration-basefrom
mwdd146980/phase2-httpx-wrapper-mvp
Draft

Add HTTPXWrapper behind use_httpx flag#23822
mwdd146980 wants to merge 8 commits into
mwdd146980/httpx-migration-basefrom
mwdd146980/phase2-httpx-wrapper-mvp

Conversation

@mwdd146980
Copy link
Copy Markdown
Contributor

@mwdd146980 mwdd146980 commented May 22, 2026

What does this PR do?

Adds a minimum-viable HTTPXWrapper that implements the existing HTTPClientProtocol so checks can opt into an httpx-backed HTTP client via use_httpx: true in instance config. RequestsWrapper remains the default for every check that does not opt in.

Wires the selector into AgentCheck.http. Adds a parallel test suite for the wrapper. Opts kong and kuma in as the first integrations to use the new wrapper. Extends mock_http_response to also patch httpx.Client.request so existing test fixtures intercept both code paths transparently.

Motivation

This PR is part of a larger migration of datadog_checks_base's HTTP layer from requests to httpx. It ships only the minimum needed to prove the wrapper-swap mechanism works end-to-end through a real integration. It does NOT aim for RequestsWrapper feature parity. The full surface (auth tokens, proxies, UDS, Kerberos / NTLM / AWS / Digest auth, connection-pool tuning, HTTP/2, multipart uploads) is deferred to follow-up PRs.

Approach

MVP feature surface in datadog_checks_base/datadog_checks/base/utils/http_httpx.py:

  • 7 HTTP methods: get, post, put, delete, head, patch, options_method
  • Response surface via HTTPXResponseAdapter: .content, .text, .headers, .status_code, .json(), .raise_for_status(), .close(), .iter_lines(), .iter_content(), .encoding, .url, .cookies, .elapsed
  • Standard request options: headers, params, json, data, timeout, verify, cert, allow_redirects
  • Basic auth via username/password
  • TLS verify, CA bundle, client cert
  • Exception mapping from httpx.* into http_exceptions.* so existing widened catches keep working
  • Single shared httpx.Client per check instance (matches RequestsWrapper's session sharing)
  • Import-time ImportError if httpx is not installed (no soft-import branching)

AgentCheck.http selects between HTTPXWrapper and RequestsWrapper on is_affirmative(instance.get('use_httpx', False)). Default behavior is unchanged for every integration that does not opt in.

Verification

  • Parallel unit test suite at datadog_checks_base/tests/base/utils/http_httpx/ (51 tests, all green) covering methods, config, basic auth, TLS, exception mapping, response surface, and lifecycle.
  • kong and kuma test suites both pass with use_httpx: true set in their instance fixtures, exercising the OM v2 scraper path end-to-end through HTTPXWrapper.
  • Confirmed via direct construction that KumaCheck(... use_httpx=True).http returns an HTTPXWrapper instance.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

Adds an httpx-backed HTTP client wrapper that implements the
HTTPClientProtocol so a check can opt in via 'use_httpx: true'
in instance config. RequestsWrapper remains the default.

Opts kong and kuma into the new wrapper as a proof of concept.
Extends mock_http_response so the existing test fixtures
intercept both requests and httpx code paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@mwdd146980 mwdd146980 added the qa/required QA is required for this PR and will generate a QA card label May 22, 2026
@datadog-prod-us1-4
Copy link
Copy Markdown
Contributor

datadog-prod-us1-4 Bot commented May 22, 2026

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 8 Pipeline jobs failed

PR All | test / j1e84c35 / Couchbase   View in Datadog   GitHub Actions

🔧 Fix in code (Fix with Cursor). 1 failed test: AssertionError: Needed at least 1 candidates for 'couchbase.index.avg_drain_rate', got 0 in tests/test_integration.py:128

PR All | test / jcc7e1d2 / Nutanix   View in Datadog   GitHub Actions

🔧 Fix in code (Fix with Cursor). Undefined name 'mock_resp' and 'mocker' in tests/conftest.py with multiple linting errors found.

PR All | test / j5a9585a / IBM ACE   View in Datadog   GitHub Actions

🔄 Retry job. This looks flaky and may succeed on retry. Job execution exceeded the timeout limit of 60 minutes during 'Prepare for testing'.

View all 8 failed jobs.

🧪 20 Tests failed in 1 job

PR All | run   GitHub Actions

test_e2e_cisco_legacy_wlc from test_e2e_core.py   View in Datadog (Fix with Cursor)
Needed at least 1 candidates for &#39;snmp.device.reachable&#39;, got 0
Expected:
        MetricStub(name=&#39;snmp.device.reachable&#39;, type=0, value=1, tags=[&#39;agent_host:runnervmyxb0g&#39;, &#39;device_hostname:DDOGWLC&#39;, &#39;device_id:default:172.18.0.2&#39;, &#39;device_ip:172.18.0.2&#39;, &#39;device_namespace:default&#39;, &#39;device_vendor:cisco&#39;, &#39;snmp_device:172.18.0.2&#39;, &#39;snmp_host:DDOGWLC&#39;, &#39;snmp_profile:cisco-legacy-wlc&#39;], hostname=None, device=None, flush_first_value=None)
Difference to closest:
        Expected tag agent_host:runnervmyxb0g
        Found agent_host:None
        Expected tag device_hostname:DDOGWLC
        Found device_hostname:None
        Expected tag device_id:default:172.18.0.2
        Found device_id:None
...
test_e2e_cisco_nexus from test_e2e_core.py   View in Datadog (Fix with Cursor)
Needed at least 1 candidates for &#39;snmp.device.reachable&#39;, got 0
Expected:
        MetricStub(name=&#39;snmp.device.reachable&#39;, type=0, value=1, tags=[&#39;agent_host:runnervmyxb0g&#39;, &#39;device_hostname:Nexus-eu1.companyname.managed&#39;, &#39;device_id:default:172.18.0.2&#39;, &#39;device_ip:172.18.0.2&#39;, &#39;device_namespace:default&#39;, &#39;device_vendor:cisco&#39;, &#39;snmp_device:172.18.0.2&#39;, &#39;snmp_host:Nexus-eu1.companyname.managed&#39;, &#39;snmp_profile:cisco-nexus&#39;], hostname=None, device=None, flush_first_value=None)
Difference to closest:
        Expected tag agent_host:runnervmyxb0g
        Found agent_host:None
        Expected tag device_hostname:Nexus-eu1.companyname.managed
        Found device_hostname:None
        Expected tag device_id:default:172.18.0.2
        Found device_id:None
...

View all 20 test failures

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ade5123 | Docs | Datadog PR Page | Give us feedback!

@codecov
Copy link
Copy Markdown

codecov Bot commented May 22, 2026

Codecov Report

❌ Patch coverage is 92.17134% with 53 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.12%. Comparing base (dc5a6d5) to head (0d65e5a).

Additional details and impacted files
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mwdd146980 mwdd146980 self-assigned this May 22, 2026
mwdd146980 and others added 7 commits May 22, 2026 17:13
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix docstring drift on mock_http_response (patches Client.request, not send).
Add options['proxies'] = None for shape-parity with RequestsWrapper.
Document the silent kwarg-drop policy in _build_request_kwargs.
Add tests for password-only auth branch, iter_content branches,
elapsed RuntimeError fallback, and AgentCheck.http httpx dispatch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Map HTTPConnectionError alongside requests ConnectionError in OM v2 scraper
  so ignore_connection_errors keeps working for use_httpx opt-ins
- Bridge reason vs reason_phrase: prefer httpx's reason_phrase, fall back to
  the protocol-standard reason exposed by MockHTTPResponse
- Narrow HTTPXWrapper.__del__ to AttributeError, matching RequestsWrapper
- Drop unimplemented STANDARD_FIELDS keys (use_legacy_auth_encoding,
  tls_ignore_warning) so the config surface only lists what is honored
- Strengthen response-surface tests with concrete assertions
- Pin the single-shared-client invariant in test_lifecycle.py
- Move use_httpx into kuma's dd_environment yield so e2e exercises the wrapper
  the same way kong does

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cut stdlib-passthrough tests, parametrize duplicates, remove private-attr
asserts, fold test_tls.py into test_config.py, move parse_basic_auth to
common.py. 87 to 52 tests, same wrapper coverage.
@dd-octo-sts
Copy link
Copy Markdown
Contributor

dd-octo-sts Bot commented May 26, 2026

Validation Report

Validation Description Status
dep Verify dependency pins are consistent and Agent-compatible

Run ddev validate all changed --fix to attempt to auto-fix supported validations.

Passed validations (20)
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file
ci Validate CI configuration and Codecov settings
codeowners Validate every integration has a CODEOWNERS entry
config Validate default configuration files against spec.yaml
http Validate integrations use the HTTP wrapper correctly
imports Validate check imports do not use deprecated modules
integration-style Validate check code style conventions
jmx-metrics Validate JMX metrics definition files and config
labeler Validate PR labeler config matches integration directories
legacy-signature Validate no integration uses the legacy Agent check signature
license-headers Validate Python files have proper license headers
licenses Validate third-party license attribution list
metadata Validate metadata.csv metric definitions
models Validate configuration data models match spec.yaml
openmetrics Validate OpenMetrics integrations disable the metric limit
package Validate Python package metadata and naming
qa-label Validate the pull request declares whether it needs QA for the next Agent release
readmes Validate README files have required sections
saved-views Validate saved view JSON file structure and fields
version Validate version consistency between package and changelog

View full run

@mwdd146980 mwdd146980 changed the title Phase 2 MVP: add HTTPXWrapper behind use_httpx flag Add HTTPXWrapper behind use_httpx flag May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

base_package dev_package integration/kong integration/kuma qa/required QA is required for this PR and will generate a QA card

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant