Add OpenMetrics hot path benchmarks

[scottopell]

What does this PR do?

Adds focused pytest-benchmark coverage for OpenMetrics v2 hot paths identified from the Agent SMP macro investigation:

• Prometheus parser label parsing with current vs high-label payloads.
• High metric-family count payloads that exercise transformer lookup/compilation behavior.
• Transform-only benchmarks over already parsed metric families.
• Full OpenMetrics check benchmarks for the same synthetic payload shapes.

The benchmark payloads mirror the macro investigation dimensions: fixed sample count, label cardinality, and metric-family count.

Motivation

Agent SMP macro profiling showed OpenMetrics throughput degradation is driven by OpenMetrics-specific Python paths rather than generic Agent ingestion. cProfile from real SMP macro runs identified these hot paths:

• prometheus_client.parser.parse_labels and parser helpers (_next_unquoted_char, _next_term, _unquote_unescape).
• MetricTransformer.get / compile_transformer for high-family-count payloads.
• generate_sample_data label/tag handling.
• aggregator.submit_metric as a linear submission contributor.

This draft PR does not optimize behavior yet. It makes those code paths reproducible in integrations-core benchmarks so candidate fixes can be evaluated with targeted benchmark evidence alongside the existing macro SMP signal.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[datadog-prod-us1-5]

 

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 2 Pipeline jobs failed

PR | test / test (linux, ubuntu-22.04, datadog_checks_base, Datadog Checks Base on Linux (py3.13), py3.13) / Datadog Checks Base on Linux (py3.13)-py3.13     

🔄 Retry job. This looks flaky and may succeed on retry.

Job failed due to timeout during 'Prepare for testing' action.

PR | test / check     

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 27.27%
• Overall Coverage: 89.17%

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e91dce2 | Docs | Datadog PR Page | Give us feedback!}

[scottopell]

Current local benchmark values from the workspace run:

Command used:

cd datadog_checks_base
DDEV_SKIP_GENERIC_TAGS_CHECK=true \
UV_PROJECT_ENVIRONMENT=.venv-hotpath \
UV_LINK_MODE=copy \
uv run --python 3.13 \
  --with-editable . \
  --with-editable ../datadog_checks_tests_helper \
  --with pytest \
  --with pytest-benchmark \
  --with hypothesis \
  --with datadog-checks-dev \
  --with '.[deps,json,db]' \
  pytest tests/base/checks/openmetrics/test_v2/test_hotpath_bench.py \
    --benchmark-only \
    --benchmark-min-rounds=5 \
    --benchmark-disable-gc \
    --benchmark-sort=name

Note: GNU ddev installs and starts correctly, but the default datadog_checks_base ddev environment pulls the http extra and failed in this container due to missing Kerberos system tooling (krb5-config). The command above avoids the Kerberos extra while still using the local package and test helper.

Benchmark	Mean
test_hotpath_full_current_labels	42.7873 ms
test_hotpath_full_high_labels	57.0632 ms
test_hotpath_full_many_families	41.5184 ms
test_hotpath_parse_current_labels	37.4159 ms
test_hotpath_parse_high_labels	48.4737 ms
test_hotpath_parse_many_families	31.4036 ms
test_hotpath_transform_cold_current_labels	5.8100 ms
test_hotpath_transform_cold_many_families	7.4528 ms
test_hotpath_transform_current_labels	3.7863 ms
test_hotpath_transform_high_labels	4.6105 ms
test_hotpath_transform_many_families	3.0493 ms

All 11 benchmark tests passed in this environment.

[scottopell]

Updated benchmark run after adding the family-count probes requested by the Agent-side SMP investigation.

Run command:

cd datadog_checks_base
DDEV_SKIP_GENERIC_TAGS_CHECK=true \
UV_PROJECT_ENVIRONMENT=.venv-hotpath \
UV_LINK_MODE=copy \
uv run --python 3.13 \
  --with-editable . \
  --with-editable ../datadog_checks_tests_helper \
  --with pytest \
  --with pytest-benchmark \
  --with hypothesis \
  --with datadog-checks-dev \
  --with '.[deps,json,db]' \
  pytest tests/base/checks/openmetrics/test_v2/test_hotpath_bench.py \
    --benchmark-only \
    --benchmark-min-rounds=5 \
    --benchmark-disable-gc \
    --benchmark-sort=name \
    --benchmark-json=/tmp/openmetrics_hotpath_bench.json

Results from this container:

Benchmark	Mean (ms)	Min (ms)	Max (ms)	Rounds
full_current_labels	43.1697	40.7962	59.1810	23
full_high_labels	55.7119	51.2068	63.8163	18
full_many_families	40.0720	36.2285	54.6351	26
full_one_family_two_labels	18.0018	16.9488	20.3657	55
parse_current_labels	36.2271	35.1884	40.5226	27
parse_high_labels	47.2493	45.9537	48.4366	21
parse_many_families	29.9833	27.5992	34.3109	37
parse_one_family_two_labels	14.1147	13.5248	15.8565	68
transform_cold_current_labels	5.2143	5.0015	6.1062	185
transform_cold_many_families	6.3161	5.9816	7.3135	123
transform_cold_many_families_multi_scraper	168.2640	160.0930	180.6216	7
transform_cold_one_family_two_labels	3.7795	3.6369	4.5521	244
transform_current_labels	3.6708	3.4532	5.6566	190
transform_high_labels	4.3738	4.0913	7.3949	156
transform_many_families	3.0248	2.8101	4.7456	149
transformer_get_cold_many_families	1.8722	1.3740	2.7005	656
transformer_get_warm_many_families	0.1926	0.1789	0.4051	4153

Interpretation relative to the Agent macro/SMP profile:

• Parser/label handling remains the clearest benchmark-level signal: parse_high_labels is ~3.35x slower than the one-family/two-label baseline, and full_high_labels is ~3.1x slower than full_one_family_two_labels.
• Family count is now visible as a transform/cache dimension when comparing equivalent two-label shapes: transform_cold_many_families is ~1.67x transform_cold_one_family_two_labels.
• The direct cache probe isolates the per-scraper metric-family lookup/compile cost: cold MetricTransformer.get for 960 families is ~1.87 ms, warm is ~0.19 ms.
• The multi-scraper cold-cache probe scales that effect across scraper instances: 25 cold scrapers over the 960-family payload average ~168 ms total, i.e. ~6.7 ms per scraper. This better represents the macro sweep’s repeated cold scraper/check execution than a single warmed benchmark loop.
• Full many-family is still not a standalone reproduction of the Agent macro cliff; it is best treated as a code-path probe. The stronger correlation is in the cold transform/cache and multi-scraper variants.

[dd-octo-sts]

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and Codecov settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run

[codecov]

Codecov Report

❌ Patch coverage is 27.27273% with 96 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.19%. Comparing base (7e7ccb5) to head (e91dce2).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.