-
Notifications
You must be signed in to change notification settings - Fork 830
Add docs and example for OTel JVM runtime metrics #1861
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
zeitlinger
wants to merge
9
commits into
main
Choose a base branch
from
otel-metrics
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
9d0a77e
add docs for using otel metrics
zeitlinger c31243c
add example
zeitlinger 637fd5f
fix path
zeitlinger 33ca552
add shutdown hook
zeitlinger 665e62a
fix link checking
zeitlinger da9292a
improve permissions
zeitlinger 90b9f99
fix link checking
zeitlinger 5733a10
fix link checking
zeitlinger 099768f
Merge branch 'main' into otel-metrics
zeitlinger File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,236 @@ | ||
| --- | ||
| title: JVM Runtime Metrics | ||
| weight: 4 | ||
| --- | ||
|
|
||
| OpenTelemetry's | ||
| [runtime-telemetry](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/runtime-telemetry) <!-- editorconfig-checker-disable-line --> | ||
| module is an alternative to | ||
| [prometheus-metrics-instrumentation-jvm]({{< relref "../instrumentation/jvm.md" >}}) | ||
| for users who want JVM metrics following OTel semantic conventions. | ||
|
|
||
| Key advantages: | ||
|
|
||
| - Metric names follow | ||
| [OTel semantic conventions](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/) <!-- editorconfig-checker-disable-line --> | ||
| - Java 17+ JFR support (context switches, network I/O, | ||
| lock contention, memory allocation) | ||
| - Alignment with the broader OTel ecosystem | ||
|
|
||
| Since OpenTelemetry's `opentelemetry-exporter-prometheus` | ||
| already depends on this library's `PrometheusRegistry`, | ||
| no additional code is needed in this library — only the | ||
| OTel SDK wiring shown below. | ||
|
|
||
| ## Dependencies | ||
|
|
||
| {{< tabs "jvm-runtime-deps" >}} | ||
| {{< tab "Gradle" >}} | ||
|
|
||
| ```groovy | ||
| implementation 'io.opentelemetry:opentelemetry-sdk' | ||
| implementation 'io.opentelemetry:opentelemetry-exporter-prometheus' | ||
|
|
||
| // Pick ONE of the following: | ||
| // Java 8+: | ||
| implementation 'io.opentelemetry.instrumentation:opentelemetry-runtime-telemetry-java8' | ||
| // Java 17+ (adds JFR-based metrics): | ||
| // implementation 'io.opentelemetry.instrumentation:opentelemetry-runtime-telemetry-java17' | ||
| ``` | ||
|
|
||
| {{< /tab >}} | ||
| {{< tab "Maven" >}} | ||
|
|
||
| ```xml | ||
| <dependency> | ||
| <groupId>io.opentelemetry</groupId> | ||
| <artifactId>opentelemetry-sdk</artifactId> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>io.opentelemetry</groupId> | ||
| <artifactId>opentelemetry-exporter-prometheus</artifactId> | ||
| </dependency> | ||
|
|
||
| <!-- Pick ONE of the following --> | ||
| <!-- Java 8+ --> | ||
| <dependency> | ||
| <groupId>io.opentelemetry.instrumentation</groupId> | ||
| <artifactId>opentelemetry-runtime-telemetry-java8</artifactId> | ||
| </dependency> | ||
| <!-- Java 17+ (adds JFR-based metrics) --> | ||
| <!-- | ||
| <dependency> | ||
| <groupId>io.opentelemetry.instrumentation</groupId> | ||
| <artifactId>opentelemetry-runtime-telemetry-java17</artifactId> | ||
| </dependency> | ||
| --> | ||
| ``` | ||
|
|
||
| {{< /tab >}} | ||
| {{< /tabs >}} | ||
|
|
||
| ## Standalone Setup | ||
|
|
||
| If you **only** want OTel runtime metrics exposed as | ||
| Prometheus, without any Prometheus Java client metrics: | ||
|
|
||
| ```java | ||
| import io.opentelemetry.exporter.prometheus.PrometheusHttpServer; | ||
| import io.opentelemetry.instrumentation.runtimemetrics.java8.RuntimeMetrics; | ||
| import io.opentelemetry.sdk.OpenTelemetrySdk; | ||
| import io.opentelemetry.sdk.metrics.SdkMeterProvider; | ||
|
|
||
| PrometheusHttpServer prometheusServer = | ||
| PrometheusHttpServer.builder() | ||
| .setPort(9464) | ||
| .build(); | ||
|
|
||
| OpenTelemetrySdk openTelemetry = | ||
| OpenTelemetrySdk.builder() | ||
| .setMeterProvider( | ||
| SdkMeterProvider.builder() | ||
| .registerMetricReader(prometheusServer) | ||
| .build()) | ||
| .build(); | ||
|
|
||
| RuntimeMetrics runtimeMetrics = | ||
| RuntimeMetrics.builder(openTelemetry).build(); | ||
|
|
||
| // Close on shutdown to stop JMX metric collection | ||
| Runtime.getRuntime() | ||
| .addShutdownHook(new Thread(runtimeMetrics::close)); | ||
|
|
||
| // Scrape at http://localhost:9464/metrics | ||
| ``` | ||
|
|
||
| ## Combined with Prometheus Java Client Metrics | ||
|
|
||
| If you already have Prometheus Java client metrics and want to | ||
| add OTel runtime metrics to the **same** `/metrics` | ||
| endpoint, use `PrometheusMetricReader` to bridge OTel | ||
| metrics into a `PrometheusRegistry`: | ||
|
|
||
| ```java | ||
| import io.prometheus.metrics.core.metrics.Counter; | ||
| import io.prometheus.metrics.exporter.httpserver.HTTPServer; | ||
| import io.prometheus.metrics.model.registry.PrometheusRegistry; | ||
| import io.opentelemetry.exporter.prometheus.PrometheusMetricReader; | ||
| import io.opentelemetry.instrumentation.runtimemetrics.java8.RuntimeMetrics; | ||
| import io.opentelemetry.sdk.OpenTelemetrySdk; | ||
| import io.opentelemetry.sdk.metrics.SdkMeterProvider; | ||
|
|
||
| PrometheusRegistry registry = | ||
| new PrometheusRegistry(); | ||
|
|
||
| // Register Prometheus metrics as usual | ||
| Counter myCounter = Counter.builder() | ||
| .name("my_requests_total") | ||
| .register(registry); | ||
|
|
||
| // Bridge OTel metrics into the same registry | ||
| PrometheusMetricReader reader = | ||
| PrometheusMetricReader.create(); | ||
| registry.register(reader); | ||
|
|
||
| OpenTelemetrySdk openTelemetry = | ||
| OpenTelemetrySdk.builder() | ||
| .setMeterProvider( | ||
| SdkMeterProvider.builder() | ||
| .registerMetricReader(reader) | ||
| .build()) | ||
| .build(); | ||
|
|
||
| RuntimeMetrics runtimeMetrics = | ||
| RuntimeMetrics.builder(openTelemetry).build(); | ||
| Runtime.getRuntime() | ||
| .addShutdownHook(new Thread(runtimeMetrics::close)); | ||
|
|
||
| // Expose everything on one endpoint | ||
| HTTPServer.builder() | ||
| .port(9400) | ||
| .registry(registry) | ||
| .buildAndStart(); | ||
| ``` | ||
|
|
||
| The [examples/example-otel-jvm-runtime-metrics](https://github.com/prometheus/client_java/tree/main/examples/example-otel-jvm-runtime-metrics) <!-- editorconfig-checker-disable-line --> | ||
| directory has a complete runnable example. | ||
|
|
||
| ## Configuration | ||
|
|
||
| The `RuntimeMetricsBuilder` supports two configuration | ||
| options: | ||
|
|
||
| ### `captureGcCause()` | ||
|
|
||
| Adds a `jvm.gc.cause` attribute to the `jvm.gc.duration` | ||
| metric, indicating why the garbage collection occurred | ||
| (e.g. `G1 Evacuation Pause`, `System.gc()`): | ||
|
|
||
| ```java | ||
| RuntimeMetrics.builder(openTelemetry) | ||
| .captureGcCause() | ||
| .build(); | ||
| ``` | ||
|
|
||
| ### `emitExperimentalTelemetry()` | ||
|
|
||
| Enables additional experimental metrics beyond the stable | ||
| set. These are not yet part of the OTel semantic conventions | ||
| and may change in future releases: | ||
|
|
||
| - Buffer pool metrics (direct and mapped byte buffers) | ||
| - Extended CPU metrics | ||
| - Extended memory pool metrics | ||
| - File descriptor metrics | ||
|
|
||
| ```java | ||
| RuntimeMetrics.builder(openTelemetry) | ||
| .emitExperimentalTelemetry() | ||
| .build(); | ||
| ``` | ||
|
|
||
| Both options can be combined: | ||
|
|
||
| ```java | ||
| RuntimeMetrics.builder(openTelemetry) | ||
| .captureGcCause() | ||
| .emitExperimentalTelemetry() | ||
| .build(); | ||
| ``` | ||
|
|
||
| Selective per-metric registration is not supported by the | ||
| runtime-telemetry API — it is all-or-nothing with these | ||
| two toggles. | ||
|
|
||
| ## Java 17 JFR Support | ||
|
|
||
| The `opentelemetry-runtime-telemetry-java17` variant adds | ||
| JFR-based metrics. You can selectively enable features: | ||
|
|
||
| ```java | ||
| import io.opentelemetry.instrumentation.runtimemetrics.java17.JfrFeature; | ||
| import io.opentelemetry.instrumentation.runtimemetrics.java17.RuntimeMetrics; | ||
|
|
||
| RuntimeMetrics.builder(openTelemetry) | ||
| .enableFeature(JfrFeature.BUFFER_METRICS) | ||
| .enableFeature(JfrFeature.NETWORK_IO_METRICS) | ||
| .enableFeature(JfrFeature.LOCK_METRICS) | ||
| .enableFeature(JfrFeature.CONTEXT_SWITCH_METRICS) | ||
| .build(); | ||
| ``` | ||
|
|
||
| ## Metric Names | ||
|
|
||
| OTel metric names are converted to Prometheus format by | ||
| the exporter. Examples: | ||
|
|
||
| | OTel name | Prometheus name | | ||
| | ---------------------------- | ---------------------------------- | | ||
| | `jvm.memory.used` | `jvm_memory_used_bytes` | | ||
| | `jvm.gc.duration` | `jvm_gc_duration_seconds` | | ||
| | `jvm.thread.count` | `jvm_thread_count` | | ||
| | `jvm.class.loaded` | `jvm_class_loaded` | | ||
| | `jvm.cpu.recent_utilization` | `jvm_cpu_recent_utilization_ratio` | | ||
|
|
||
| See [Names]({{< relref "names.md" >}}) for full details on | ||
| how OTel names map to Prometheus names. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| # OTel JVM Runtime Metrics with Prometheus HTTPServer | ||
|
|
||
| ## Build | ||
|
|
||
| This example is built as part of the `client_java` project. | ||
|
|
||
| ```shell | ||
| ./mvnw package | ||
| ``` | ||
|
|
||
| ## Run | ||
|
|
||
| The build creates a JAR file with the example application in | ||
| `./examples/example-otel-jvm-runtime-metrics/target/`. | ||
|
|
||
| ```shell | ||
| java -jar ./examples/example-otel-jvm-runtime-metrics/target/example-otel-jvm-runtime-metrics.jar | ||
| ``` | ||
|
|
||
| ## Manually Testing the Metrics Endpoint | ||
|
|
||
| Accessing | ||
| [http://localhost:9400/metrics](http://localhost:9400/metrics) | ||
| with a Web browser should yield both a Prometheus counter metric | ||
| and OTel JVM runtime metrics on the same endpoint. | ||
|
|
||
| Prometheus counter: | ||
|
|
||
| ```text | ||
| # HELP uptime_seconds_total total number of seconds since this application was started | ||
| # TYPE uptime_seconds_total counter | ||
| uptime_seconds_total 42.0 | ||
| ``` | ||
|
|
||
| OTel JVM runtime metrics (excerpt): | ||
|
|
||
| ```text | ||
| # HELP jvm_memory_used_bytes Measure of memory used. | ||
| # TYPE jvm_memory_used_bytes gauge | ||
| jvm_memory_used_bytes{jvm_memory_pool_name="G1 Eden Space",jvm_memory_type="heap"} 4194304.0 | ||
| ``` |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not related to this PR, but a good opportunity to fix - last time I merged with this check failing and I didn't like it