Tracer Version(s)
Datadog dd-trace-java v1.65.1 (official source compared). The debug capture is from a downstream-derived build; the official binary has not been independently run in this environment.
Java Version(s)
Java 17.0.9 / IBM OpenJ9 0.41.0
JVM Vendor
IBM SDK / Semeru
Bug Report
Context and upstream comparison
TomEE/Tomcat with CXF Jakarta-RS. I compared the affected build with the official Datadog v1.65.1 tag. The Jakarta-RS annotations instrumentation source tree, Tomcat instrumentation source tree, scope-manager source tree, ExtractedSpan.java, and LogbackLoggerInstrumentation.java are byte-for-byte identical. In particular, the Jakarta-RS advice in v1.65.1 has the same scope lifecycle code.
One downstream Logback event instrumentation file differs, so I am not claiming that the exact log output or missing correlation IDs have been reproduced with the official Datadog artifact. This report focuses on the scope-stack symptom. Source equivalence also does not, by itself, prove that the official binary exhibits the same runtime behavior.
Observed sequence in a sanitized tracer debug capture
- A Jakarta-RS span starts for an annotated resource method on a Tomcat worker.
- When the outer servlet request completes, closing its scope logs "scope when not on top" and identifies the still-active Jakarta-RS span as the current top. The example Jakarta-RS span has no matching finish record in the available capture.
- Similar out-of-order closes recur on worker threads. Later, activation of new request spans reports "Scope depth limit exceeded (100). Returning NoopScope."
Please investigate whether the Jakarta-RS advice can leave a resource-method scope active after the Tomcat request finishes in a TomEE/CXF deployment, and whether this behavior is already addressed upstream. The capture does not identify which specific advice-exit path was missed.
A related Logback null TraceConfig path is already addressed by #12413; that is not the requested fix here.
Expected Behavior
The Jakarta-RS scope closes before the outer servlet request scope, and each completed request leaves the worker scope stack at its original depth.
Reproduction Code
No standalone reproducer against the official Datadog binary is available yet. No customer logs or internal URLs are attached.
Tracer Version(s)
Datadog dd-trace-java v1.65.1 (official source compared). The debug capture is from a downstream-derived build; the official binary has not been independently run in this environment.
Java Version(s)
Java 17.0.9 / IBM OpenJ9 0.41.0
JVM Vendor
IBM SDK / Semeru
Bug Report
Context and upstream comparison
TomEE/Tomcat with CXF Jakarta-RS. I compared the affected build with the official Datadog v1.65.1 tag. The Jakarta-RS annotations instrumentation source tree, Tomcat instrumentation source tree, scope-manager source tree, ExtractedSpan.java, and LogbackLoggerInstrumentation.java are byte-for-byte identical. In particular, the Jakarta-RS advice in v1.65.1 has the same scope lifecycle code.
One downstream Logback event instrumentation file differs, so I am not claiming that the exact log output or missing correlation IDs have been reproduced with the official Datadog artifact. This report focuses on the scope-stack symptom. Source equivalence also does not, by itself, prove that the official binary exhibits the same runtime behavior.
Observed sequence in a sanitized tracer debug capture
Please investigate whether the Jakarta-RS advice can leave a resource-method scope active after the Tomcat request finishes in a TomEE/CXF deployment, and whether this behavior is already addressed upstream. The capture does not identify which specific advice-exit path was missed.
A related Logback null TraceConfig path is already addressed by #12413; that is not the requested fix here.
Expected Behavior
The Jakarta-RS scope closes before the outer servlet request scope, and each completed request leaves the worker scope stack at its original depth.
Reproduction Code
No standalone reproducer against the official Datadog binary is available yet. No customer logs or internal URLs are attached.