Measure metric metadata coverage in CI

[nubtron]

What does this PR do?

Temporarily instruments AggregatorStub.assert_metrics_using_metadata so each assertion can emit one metric metadata coverage JSON object during CI. The existing metadata assertion behavior is intentionally unchanged: submitted metrics that are not in metadata.csv, type mismatches, and symmetric missing submissions still fail only in the same situations as before.

When DD_INTEGRATION_METRIC_COVERAGE=1 is set, each assertion writes a JSON line to DD_INTEGRATION_METRIC_COVERAGE_FILE. In GitHub test-target CI, it also falls back to $TEST_RESULTS_DIR/metric-metadata-coverage.jsonl so reports are uploaded with the existing test-results artifact. Each report is also printed as a greppable line prefixed with DD_INTEGRATION_METRIC_COVERAGE . The PR enables this in the GitHub test-target unit/integration and E2E scripts and stores the JSONL file under the existing test-results artifact directory.

Key fields:

• integration / check_name: best-effort CI target/check context.
• pytest_nodeid: test node from PYTEST_CURRENT_TEST when pytest provides it.
• submitted_count: unique submitted metric names considered by the assertion after exclude is applied.
• metadata_count: metric names present in metadata.csv for the assertion input.
• covered_count: metric names present in both submissions and metadata.
• coverage_percent: covered_count / metadata_count * 100.
• missing_metric_names: metadata rows not observed in this assertion.
• emitted_not_in_metadata: submitted metric names without metadata rows.
• type_mismatches: metrics whose observed type does not match metadata.
• excluded_metrics: metrics passed through the assertion's exclude parameter.

Motivation

This is a draft PR experiment to measure integration metric metadata coverage in CI before deciding whether and where to add stricter gates. The output should make low-coverage tests and integrations visible without broadly failing on missing submitted coverage.

Interpretation guidance:

• Low coverage_percent means the test only emitted a subset of metadata-defined metrics; this is expected for some narrow tests and useful for finding E2E gaps.
• Non-empty emitted_not_in_metadata is already assertion-failing behavior today unless that metric is excluded.
• Non-empty type_mismatches is already assertion-failing behavior today when type checks are enabled.
• excluded_metrics should be treated as intentional blind spots. Per-test exclusions should include a nearby code comment explaining why each metric is excluded, for example dynamic version behavior, optional backend features, or known fixture limitations. Avoid adding unexplained broad exclusions because they reduce the usefulness of coverage stats.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[datadog-prod-us1-4]

 

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 3 Pipeline jobs failed

PR All | test / j06ca546 / SNMP     

🔧 Fix in code (Fix with Cursor).

158 tests failed due to AssertionError: Needed at least 1 candidates for 'snmp.ifNumber', got 0.

Check PR | run / Check PR changelog     

🛟 This job is unlikely to succeed on retry. Please review your pipeline configuration.

Package "datadog_checks_base" has changes that require a changelog. Please run `ddev release changelog new` to add it.

PR All | test / j6f0ded4 / Strimzi     

🛟 This job is unlikely to succeed on retry. Please review your pipeline configuration.

Resource mapping not found for 'my-cluster', as no matches for kind 'Kafka' in version 'kafka.strimzi.io/v1beta2'. Ensure CRDs are installed first.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 215a302 | Docs | Datadog PR Page | Give us feedback!}

[dd-octo-sts]

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and Codecov settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run