-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Add OTel Histogram support documentation #37526
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -17,9 +17,9 @@ aliases: | |||||
|
|
||||||
| ## Overview | ||||||
|
|
||||||
| The Datadog Agent and the OpenTelemetry Collector Datadog exporter can ingest metrics in the OpenTelemetry format (OTLP), which can be produced by OpenTelemetry-instrumented applications. | ||||||
| The Datadog Agent and the OpenTelemetry Collector OTLP Exporter can ingest metrics in the OpenTelemetry format (OTLP), which can be produced by OpenTelemetry-instrumented applications. | ||||||
|
|
||||||
| The following OTLP metric types can be ingested by the Datadog Agent and the OpenTelemetry Collector Datadog exporter: | ||||||
| The following OTLP metric types can be ingested by the Datadog Agent and the OpenTelemetry Collector OTLP Exporter: | ||||||
| - Sums | ||||||
| - Gauges | ||||||
| - Histograms | ||||||
|
|
@@ -66,19 +66,25 @@ OTLP Gauges are mapped to Datadog Gauges, since they do not provide an aggregati | |||||
| {{% /tab %}} | ||||||
| {{% tab "Histogram" %}} | ||||||
|
|
||||||
| An OTLP Histogram represents the statistical distribution of a set of values on a given time window, by storing certain aggregation metrics such as the population sum or count together with a series of bucket counts. Histograms have one feature that influences the mapping: | ||||||
| OTLP defines two histogram types for representing value distributions: | ||||||
|
|
||||||
| - **Histogram (Explicit Bucket)**: Uses fixed, user-defined or SDK-default bucket boundaries. Each bucket counts how many measured values fall within its bounds. Also stores sum, count, and optional min/max. | ||||||
| - **ExponentialHistogram**: Uses dynamically computed bucket boundaries based on a scale parameter (base = 2^(2^(-scale))), which allows high dynamic range with small relative error. Datadog supports scales in the range [-4, 8]. | ||||||
|
|
||||||
| Both types have one feature that influences the mapping: | ||||||
|
|
||||||
| - *Aggregation temporality*, which can be cumulative or delta. Delta metrics have no overlap in their time windows, while cumulative metrics represent a time window from a fixed start point in time. | ||||||
|
|
||||||
| The default mapping is as follows: | ||||||
| 1. Delta histograms are reported as Datadog distributions. [Read more about distributions][1] to understand the available aggregations. Histograms with a count of 0 are dropped. | ||||||
| 2. For cumulative histograms, the delta between consecutive points is calculated and reported to Datadog as a distribution. Deltas with a count of 0 are not reported. You may use the [`cumsum` arithmetic function][2] on individual aggregations to recover the value in the OTLP payload. | ||||||
| The default mapping for both histogram temporality types is as follows: | ||||||
|
|
||||||
| - **Delta (Default and Recommended)**: Explicit Bucket and Exponential Histograms are ingested natively and reported as Datadog distributions. The original bucket structure is preserved — no conversion to DDSketch occurs. Histograms with a count of 0 are dropped. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| - **Cumulative**: The delta between consecutive points is calculated and reported to Datadog as a distribution. Deltas with a count of 0 are not reported. | ||||||
|
|
||||||
| **Note**: Histogram metrics in OTLP are mapped by default to Distribution metrics. Because of how OTLP sends this data, percentile aggregations and the max and min (if not available on the original OTLP data) are approximations, not accurate calculations. | ||||||
| **Note**: See [Producing Delta Temporality Metrics with OpenTelemetry][3] for configuration instructions. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| Datadog distributions are powered by the [DDSketch data structure][4]. See [Distributions][5] for details on enabling percentile aggregations and threshold queries on your mapped metrics. | ||||||
| Explicit and Exponential Histograms are stored natively in Datadog, preserving the original bucket boundaries and counts. Percentile aggregations are computed directly from the native bucket structure. Min and max are stored when provided; if not present in the original data, they are derived from the bucket boundaries. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The first sentence repeats what the Delta bullet already states ("ingested natively... The original bucket structure is preserved").
Suggested change
|
||||||
|
|
||||||
| The Datadog Agent and the OpenTelemetry Collector Datadog exporter allow changing the Histogram export in the `histogram` subsection. | ||||||
| The Datadog Agent and the OpenTelemetry Collector OTLP Exporter allow changing the Histogram export in the `histogram` subsection. | ||||||
| - If the `mode` is set to `counters`, the following metrics are produced: | ||||||
|
|
||||||
| `<METRIC_NAME>.bucket`, tagged by `lower_bound` and `upper_bound` | ||||||
|
|
@@ -96,14 +102,14 @@ The Datadog Agent and the OpenTelemetry Collector Datadog exporter allow changin | |||||
| **Datadog In-App Type**: COUNT | ||||||
|
|
||||||
| `<METRIC_NAME>.min` | ||||||
| : Minimum of values submitted during the time window. Only available for delta OTLP Histograms. Available since: Datadog exporter v0.75.0 and Datadog Agent v6.45.0 and v7.45.0. <br> | ||||||
| : Minimum of values submitted during the time window. Only available for delta OTLP Histograms. Available since: OTLP Exporter v0.75.0 and Datadog Agent v6.45.0 and v7.45.0. <br> | ||||||
| **Datadog In-App Type**: GAUGE | ||||||
|
|
||||||
| `<METRIC_NAME>.max` | ||||||
| : Maximum of values submitted during the time window. Only available for delta OTLP Histograms. Available since: Datadog exporter v0.75.0 and Datadog Agent v6.45.0 and v7.45.0.<br> | ||||||
| : Maximum of values submitted during the time window. Only available for delta OTLP Histograms. Available since: OTLP Exporter v0.75.0 and Datadog Agent v6.45.0 and v7.45.0.<br> | ||||||
| **Datadog In-App Type**: GAUGE | ||||||
|
|
||||||
| **Note**: `send_aggregation_metrics` is useful only when not using the distributions mode. Before the Datadog exporter v0.75.0 and the Datadog Agent v6.45.0 and v7.45.0 use `send_count_sum_metrics` instead. | ||||||
| **Note**: `send_aggregation_metrics` is useful only when not using the distributions mode. Before the OTLP Exporter v0.75.0 and the Datadog Agent v6.45.0 and v7.45.0 use `send_count_sum_metrics` instead. | ||||||
|
|
||||||
| [1]: /metrics/distributions | ||||||
| [2]: /dashboards/functions/arithmetic/#cumulative-sum | ||||||
|
Comment on lines
114
to
115
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
@@ -131,7 +137,7 @@ An OTLP Summary is a legacy type that conveys quantile information about a popul | |||||
|
|
||||||
| OTLP supports two kinds of attributes: datapoint-level attributes and resource attributes. These attributes may follow OpenTelemetry semantic conventions and have well-known semantics. | ||||||
|
|
||||||
| The Datadog Agent and the OpenTelemetry Collector Datadog exporter map the datapoints-level attributes as tags. Resource attributes following OpenTelemetry semantic conventions are mapped to the equivalent Datadog conventions if they exist. | ||||||
| The Datadog Agent and the OpenTelemetry Collector OTLP Exporter map the datapoints-level attributes as tags. Resource attributes following OpenTelemetry semantic conventions are mapped to the equivalent Datadog conventions if they exist. | ||||||
|
|
||||||
| You may add all resource attributes as tags by using the `resource_attributes_as_tags` flag. | ||||||
|
|
||||||
|
|
||||||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -40,12 +40,13 @@ The following metric submission types are accepted: | |||||
| - [HISTOGRAM](?tab=histogram#metric-types) | ||||||
| - [DISTRIBUTION](?tab=distribution#metric-types) | ||||||
|
|
||||||
| These different metric submission types are mapped to four in-app metric types found within the Datadog web application: | ||||||
| These different metric submission types are mapped to five in-app metric types found within the Datadog web application: | ||||||
|
|
||||||
| - COUNT | ||||||
| - RATE | ||||||
| - GAUGE | ||||||
| - DISTRIBUTION | ||||||
| - (Exponential, Explicit) HISTOGRAM | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| **Note**: If you submit a metric to Datadog without a type, the metric type appears as `Not Assigned` within Datadog. The `Not Assigned` metric type cannot be further changed to another in-app type until an initial metric type is submitted. | ||||||
|
|
||||||
|
|
||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.