Skip to content

add TaskBlock events for blocking intervals#570

Open
kaahos wants to merge 76 commits into
mainfrom
paul.fournillon/wallclock-taskblock
Open

add TaskBlock events for blocking intervals#570
kaahos wants to merge 76 commits into
mainfrom
paul.fournillon/wallclock-taskblock

Conversation

@kaahos

@kaahos kaahos commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?:

Adds datadog.TaskBlock JFR events for blocking intervals such as LockSupport.park, Object.wait, monitor contention, including recording APIs used by dd-trace-java for instrumented blocking operations such as Thread.sleep.

Motivation:

This builds on paul.fournillon/wallclock-suppression (#560) by preserving visibility into blocked spans as explicit duration events.

Additional Notes:

This PR does not add the dd-trace-java instrumentation itself; Thread.sleep emission depends on that side calling the new profiler APIs. Monitor callback support is HotSpot-specific, and virtual-thread carrier attribution is avoided.

How to test the change?:

  • ./.claude/commands/build-and-summarize :ddprof-test:testDebug -Ptests="*.wallclock.*TaskBlockTest"
  • ./.claude/commands/build-and-summarize :ddprof-lib:gtestDebug

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.
  • This PR doesn't touch any of that.
  • JIRA: [JIRA-14354]

Unsure? Have a question? Request a review!

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jun 1, 2026

Copy link
Copy Markdown

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

DataDog/java-profiler | reliability-chaos-amd64: [profiler, tcmalloc, 21.0.3-tem]   View in Datadog   GitLab

ℹ️ Info

🔄 Datadog auto-retried 3 jobs - 2 passed on retry View in Datadog

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 38362a0 | Docs | Datadog PR Page | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #28882397887 | Commit: 85928e0 | Duration: 14m 50s (longest job)

All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - - -
8-ibm - - -
8-j9 - -
8-librca - -
8-orcl - - -
11 - - -
11-j9 - -
11-librca - -
17 - -
17-graal - -
17-j9 - -
17-librca - -
21 - -
21-graal - -
21-librca - -
25 - -
25-graal - -
25-librca - -

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-07-07 16:52:18 UTC

@kaahos kaahos force-pushed the paul.fournillon/wallclock-taskblock branch from bcf3c2b to a46d368 Compare June 2, 2026 08:24
@kaahos kaahos force-pushed the paul.fournillon/wallclock-taskblock branch 3 times, most recently from 3f2f71a to d1e3210 Compare June 3, 2026 13:00
@kaahos

kaahos commented Jun 3, 2026

Copy link
Copy Markdown
Contributor Author

@copilot review this PR

Copilot AI commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

@copilot review this PR

Reviewed and found a flaky assertion in NativeSocketTaskBlockTest: it required zero TaskBlock events globally, which can fail when an untraced socket op emits one. Fixed in d21e493 by asserting that no TaskBlock is emitted for the traced span specifically, and re-ran :ddprof-test:testDebug -Ptests="*.wallclock.*TaskBlockTest" successfully.

Copilot finished work on behalf of kaahos June 3, 2026 13:27
@kaahos kaahos marked this pull request as ready for review June 3, 2026 13:40
@kaahos kaahos requested a review from a team as a code owner June 3, 2026 13:40

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d21e4936d2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ddprof-lib/src/main/cpp/wallClock.cpp Outdated
@kaahos kaahos force-pushed the paul.fournillon/wallclock-taskblock branch 2 times, most recently from 2101007 to a94677a Compare June 3, 2026 14:05
@kaahos kaahos force-pushed the paul.fournillon/wallclock-taskblock branch from a94677a to 681582c Compare June 3, 2026 14:38
@jbachorik

Copy link
Copy Markdown
Collaborator

🗿 🤖 🔴

[Sphinx Review — INFO] ddprof-lib/src/main/cpp/vmEntry.cpp line 557

The wallPrecheck parameter of VM::initProfilerBridge is immediately silenced with (void)wallPrecheck and never used; the actual _wall_precheck flag is read from Arguments during Profiler::start(), making the parameter misleading.

Suggestion: Either remove the wallPrecheck parameter from initProfilerBridge and its callers, or add a comment explaining that it is intentionally ignored at bridge-init time.

@jbachorik jbachorik added the sphinx:critical Sphinx: critical — human review required label Jun 30, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/122365137 Commit: b7db495593670f5e3ba4c153b225734b7124d463

⚠️ Significant outliers

  • 🟢 future-genetic (JDK 25): runtime -7.3% (2074→1923 ms)
  • 🔴 reactors (JDK 25): runtime +4.8% (17735→18593 ms)
  • ⚠️ fj-kmeans (JDK 21): wallclock unwinds skipped — latest 0 → dev 1247 (new)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10414 ms (7 iters) ✅ 10412 ms (7 iters) ≈ -0% (±20.7%) — / —
akka-uct 25 ✅ 8774 ms (8 iters) ✅ 9037 ms (8 iters) ≈ +3% (±22.5%) — / —
finagle-chirper 21 ✅ 5988 ms (11 iters) ✅ 6043 ms (11 iters) ≈ +0.9% (±45.5%) ⚠️ W:1 / ⚠️ W:1
finagle-chirper 25 ✅ 5445 ms (12 iters) ✅ 5507 ms (12 iters) ≈ +1.1% (±43.1%) ⚠️ W:1 / ⚠️ W:1
fj-kmeans 21 ✅ 2817 ms (22 iters) ✅ 2808 ms (22 iters) ≈ -0.3% (±4.4%) — / —
fj-kmeans 25 ✅ 2842 ms (22 iters) ✅ 2843 ms (22 iters) ≈ +0% (±4.5%) — / —
future-genetic 21 ✅ 2023 ms (31 iters) ✅ 2052 ms (30 iters) ≈ +1.4% (±4.3%) — / —
future-genetic 25 ✅ 2074 ms (30 iters) ✅ 1923 ms (32 iters) 🟢 -7.3% — / —
naive-bayes 21 ✅ 1232 ms (46 iters) ✅ 1233 ms (46 iters) ≈ +0.1% (±57%) — / —
naive-bayes 25 ✅ 925 ms (61 iters) 💥 0 ms (0 iters) — / —
reactors 21 ✅ 16521 ms (5 iters) ✅ 17662 ms (5 iters) ≈ +6.9% (±13.4%) — / —
reactors 25 ✅ 17735 ms (5 iters) ✅ 18593 ms (5 iters) 🔴 +4.8% — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 4 / 3 2115 / 2356 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ ✅ / 2 8348 / 8213 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ ✅ / 1 ✅ / 1247 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 1 / 2 1278 / 1278 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / 2 2918 / 2051 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / 2 1700 / 1851 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / 1 1817 / 1889 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Reliability & Chaos Results

1 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/123745876

❌ chaos: profiler tcmalloc amd64 21 0 3 temXchaos
Chaos harness crashed (RC=124)
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000
# C  [libjavaProfiler-dd-tmp9981091628585413870.so+0x14600]

@dd-octo-sts

dd-octo-sts Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 36f3db5)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/122534005 Commit: 36f3db52d66680198bd5ce6ed0dcc3fc73aca40c

⚠️ Significant outliers

  • 🔴 future-genetic (JDK 25): runtime +7.4% (1913→2054 ms)
  • 🔴 reactors (JDK 25): runtime +7.5% (17266→18553 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10381 ms (7 iters) ✅ 10543 ms (7 iters) ≈ +1.6% (±19.2%) — / —
akka-uct 25 ✅ 8848 ms (8 iters) ✅ 8978 ms (8 iters) ≈ +1.5% (±20%) — / —
finagle-chirper 21 ✅ 6063 ms (11 iters) ✅ 6089 ms (11 iters) ≈ +0.4% (±46%) ⚠️ W:1 / ⚠️ W:1
finagle-chirper 25 ✅ 5362 ms (12 iters) ✅ 5480 ms (12 iters) ≈ +2.2% (±44.8%) ⚠️ W:1 / ⚠️ W:1
fj-kmeans 21 ✅ 2804 ms (22 iters) ✅ 2798 ms (22 iters) ≈ -0.2% (±4.3%) — / —
fj-kmeans 25 ✅ 2803 ms (22 iters) ✅ 2816 ms (22 iters) ≈ +0.5% (±5%) — / —
future-genetic 21 ✅ 2051 ms (30 iters) ✅ 2111 ms (29 iters) ≈ +2.9% (±4.7%) — / —
future-genetic 25 ✅ 1913 ms (32 iters) ✅ 2054 ms (30 iters) 🔴 +7.4% — / —
naive-bayes 21 ✅ 1264 ms (45 iters) ✅ 1392 ms (42 iters) ≈ +10.1% (±57.6%) — / —
naive-bayes 25 ✅ 1022 ms (56 iters) ✅ 1001 ms (57 iters) ≈ -2.1% (±54.8%) — / —
reactors 21 ✅ 16918 ms (5 iters) ✅ 15700 ms (5 iters) ≈ -7.2% (±13.6%) — / —
reactors 25 ✅ 17266 ms (5 iters) ✅ 18553 ms (5 iters) 🔴 +7.5% — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 3 / 1 2366 / 2335 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 1 8231 / 8310 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ ✅ / 3 1267 / 1209 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 2 / 1 2825 / 2902 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 2 / 3 3501 / 3486 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 1 / 1 1730 / 1894 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 821cac8)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/122545771 Commit: 821cac8d91257bcc5696ccc715f1d9009a8e5432

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10354 ms (7 iters) ✅ 10195 ms (7 iters) ≈ -1.5% (±19.7%) — / —
akka-uct 25 ✅ 8940 ms (8 iters) ✅ 8943 ms (8 iters) ≈ +0% (±20.2%) — / —
finagle-chirper 21 ✅ 6057 ms (11 iters) ✅ 6114 ms (11 iters) ≈ +0.9% (±46.1%) ⚠️ W:1 / ⚠️ W:1
finagle-chirper 25 ✅ 5430 ms (12 iters) ✅ 5450 ms (12 iters) ≈ +0.4% (±44.3%) ⚠️ W:1 / ⚠️ W:1
fj-kmeans 21 ✅ 2745 ms (23 iters) ✅ 2799 ms (22 iters) ≈ +2% (±4.3%) — / —
fj-kmeans 25 ✅ 2828 ms (22 iters) ✅ 2835 ms (22 iters) ≈ +0.2% (±4.8%) — / —
future-genetic 21 ✅ 2127 ms (29 iters) ✅ 2039 ms (30 iters) ≈ -4.1% (±4.5%) — / —
future-genetic 25 ✅ 2069 ms (30 iters) ✅ 2104 ms (30 iters) ≈ +1.7% (±4.4%) — / —
naive-bayes 21 ✅ 1227 ms (46 iters) ✅ 1221 ms (47 iters) ≈ -0.5% (±55.9%) — / —
naive-bayes 25 ✅ 1035 ms (55 iters) ✅ 1023 ms (56 iters) ≈ -1.2% (±55.7%) — / —
reactors 21 ✅ 17340 ms (5 iters) ✅ 17849 ms (5 iters) ≈ +2.9% (±14.2%) — / —
reactors 25 ✅ 18225 ms (5 iters) ✅ 17798 ms (5 iters) ≈ -2.3% (±6.9%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 2 / ✅ 2009 / ✅ ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 1 2209 / 2322 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 2 8171 / 8480 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 3 / 2 1284 / 1225 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / 3 2928 / 2962 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 8 / 7 3490 / 3470 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 1 / 1 1835 / 1642 ✅ / ✅ ✅ / ✅

@kaahos kaahos requested review from jbachorik and rkennke July 3, 2026 12:05

@rkennke rkennke left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me (pending Jaroslav's Sphinx reviews)

@dd-octo-sts

dd-octo-sts Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit bfdc687)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/122570556 Commit: bfdc6877afabbdd6b921f29dd0dd584403621dc0

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10564 ms (7 iters) ✅ 10428 ms (7 iters) ≈ -1.3% (±24.3%) — / —
akka-uct 25 ✅ 8887 ms (8 iters) ✅ 8873 ms (8 iters) ≈ -0.2% (±20%) — / —
finagle-chirper 21 ✅ 5982 ms (11 iters) ✅ 5942 ms (11 iters) ≈ -0.7% (±46%) ⚠️ W:1 / ⚠️ W:1
finagle-chirper 25 ✅ 5370 ms (12 iters) ✅ 5503 ms (12 iters) ≈ +2.5% (±43%) ⚠️ W:1 / ⚠️ W:1
fj-kmeans 21 ✅ 2821 ms (22 iters) ✅ 2827 ms (22 iters) ≈ +0.2% (±4.3%) — / —
fj-kmeans 25 ✅ 2821 ms (22 iters) ✅ 2832 ms (22 iters) ≈ +0.4% (±4.6%) — / —
future-genetic 21 ✅ 2027 ms (31 iters) ✅ 2098 ms (30 iters) ≈ +3.5% (±4.6%) — / —
future-genetic 25 ✅ 2060 ms (30 iters) ✅ 2123 ms (29 iters) ≈ +3.1% (±4.9%) — / —
naive-bayes 21 ✅ 1226 ms (46 iters) ✅ 1231 ms (46 iters) ≈ +0.4% (±57.4%) — / —
naive-bayes 25 ✅ 1011 ms (56 iters) ✅ 1018 ms (56 iters) ≈ +0.7% (±55.8%) — / —
reactors 21 ✅ 15812 ms (5 iters) ✅ 16153 ms (5 iters) ≈ +2.2% (±12.7%) — / —
reactors 25 ✅ 18260 ms (5 iters) ✅ 18464 ms (5 iters) ≈ +1.1% (±2.2%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 4 / 4 2015 / 2290 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 2 8052 / 8333 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ 1283 / ✅ ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 5 / 3 1263 / 1279 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 1 / 1 2908 / 2905 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 2 / 13 3510 / 3498 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 2 / 1 1855 / 1720 ✅ / ✅ ✅ / ✅

@dd-octo-sts

dd-octo-sts Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit b2d5245)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/122598460 Commit: b2d524527f497eed49e0abd8c2bf4249ac2575b1

⚠️ Significant outliers

  • 🟢 future-genetic (JDK 21): runtime -5.4% (2210→2091 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10094 ms (7 iters) ✅ 10356 ms (7 iters) ≈ +2.6% (±21.2%) — / —
akka-uct 25 ✅ 8837 ms (8 iters) ✅ 8743 ms (8 iters) ≈ -1.1% (±22.7%) — / —
finagle-chirper 21 ✅ 5853 ms (11 iters) ✅ 5910 ms (11 iters) ≈ +1% (±43.6%) ⚠️ W:1 / ⚠️ W:1
finagle-chirper 25 ✅ 5478 ms (12 iters) ✅ 5368 ms (12 iters) ≈ -2% (±44%) ⚠️ W:1 / ⚠️ W:1
fj-kmeans 21 ✅ 2824 ms (22 iters) ✅ 2834 ms (22 iters) ≈ +0.4% (±4.8%) — / —
fj-kmeans 25 ✅ 2817 ms (22 iters) ✅ 2840 ms (22 iters) ≈ +0.8% (±4.8%) — / —
future-genetic 21 ✅ 2210 ms (28 iters) ✅ 2091 ms (30 iters) 🟢 -5.4% — / —
future-genetic 25 ✅ 1923 ms (32 iters) ✅ 1931 ms (32 iters) ≈ +0.4% (±4.8%) — / —
naive-bayes 21 ✅ 1260 ms (45 iters) ✅ 1229 ms (46 iters) ≈ -2.5% (±56.9%) — / —
naive-bayes 25 ✅ 1016 ms (56 iters) ✅ 1020 ms (56 iters) ≈ +0.4% (±56.3%) — / —
reactors 21 ✅ 17030 ms (5 iters) ✅ 17150 ms (5 iters) ≈ +0.7% (±12.4%) — / —
reactors 25 ✅ 18323 ms (5 iters) ✅ 18162 ms (5 iters) ≈ -0.9% (±3.7%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 4 / 4 2267 / 2432 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 2 / 2 8669 / 8358 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 2 / 4 1268 / 1225 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 7 / ✅ 3051 / ✅ ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 1 / 1 2773 / 2068 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 7 / 6 3483 / 3500 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ ✅ / 4 1846 / 1692 ✅ / ✅ ✅ / ✅

@jbachorik

Copy link
Copy Markdown
Collaborator

@copilot resolve the merge conflicts in this pull request

Copilot AI commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

@copilot resolve the merge conflicts in this pull request

Resolved. I merged origin/main into this branch and fixed the conflict in ddprof-lib/src/main/cpp/vmEntry.cpp in commit 38362a03.

@jbachorik jbachorik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did review ~50% of the code but I will need to get more info on some design decisions before I continue because things might change depending on that info.

ExecutionMode _execution_mode;
u64 _weight;
u32 _call_trace_id;
u64 _call_trace_id;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uff. Nice catch.

u64 _end;
u64 _blocker;
u64 _unblockingSpanId;
Context _ctx;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you need to carry context here.
IIUC, this event would be written at the moment a thread is ublocked. At that point its context is valid and can be retrieved and written along this event.

u64 _unblockingSpanId;
Context _ctx;
u64 _callTraceId;
u64 _correlationId;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we discussed this already (or maybe it was another place?).
For the current work the correlation ID adds only overhead. The event will capture the correct stacktrace when it is written - if the implementation is correct, the event stacktrace should be exactly the same as the one that it would get by the correlation id.
So, it would be mostly just an overhead per TaskBlockEvent

JNIEnv *env, jclass unused, jboolean delegateMonitorEvents, jboolean wallPrecheck) {
// JavaVM* has already been stored when the native library was loaded so we can pass nullptr here
return VM::initProfilerBridge(nullptr, true);
return VM::initProfilerBridge(nullptr, true, delegateMonitorEvents, wallPrecheck);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Historically, these engine specific settings would have been passed by an argument (via arguments.cpp)
I am a bit worried by introducing an alternative way of doing that here - also, the args are not engine agnostic so they kind of mix concerns here.
Could you take a look how to convert this to an arg that can be passed to the agent?

inline bool parkEnter(u64 start_ticks) {
u32 flags = __atomic_load_n(&_misc_flags, __ATOMIC_ACQUIRE);
while ((flags & FLAG_PARKED) == 0) {
_park_context = ContextApi::snapshot();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need to capture the context here. The event will be written at appropriate parkExit and since the thread is parking, there is no way the context would change.

Comment on lines +510 to +513
// Virtual-thread path: span/root ids captured at block entry are passed explicitly because
// the native OTEP TLS is carrier-scoped and cannot be trusted. Custom attributes not propagated.
Context ctx{(u64)spanId, (u64)rootSpanId};
return recordTaskBlockWithContextIfEligible(tid, (u64)startTicks, (u64)endTicks,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still true? We did a bunch of fixes in the context area and I think this might be redundant as we should always get correct context when we ask for it from the same thread.

Comment on lines +471 to +510
extern "C" DLLEXPORT void JNICALL
Java_com_datadoghq_profiler_JavaProfiler_blockExitWithSnapshot0(
JNIEnv *env, jclass unused, jlong token, jlongArray snapshotArray) {
BlockRunSnapshot snapshot{};
snapshot.active_state = OSThreadState::UNKNOWN;
snapshot.sampled_state = OSThreadState::UNKNOWN;
snapshot.owner = BlockRunOwner::NONE;
if (snapshotAndExitBlockedRun(token, &snapshot)) {
writeBlockRunSnapshot(env, snapshotArray, &snapshot);
} else {
writeBlockRunSnapshot(env, snapshotArray, nullptr);
}
ThreadFilter::SlotID slot_id = ThreadFilter::tokenSlotId(block_token);
if (current->filterSlotId() != slot_id) {
return;
}

extern "C" DLLEXPORT jboolean JNICALL
Java_com_datadoghq_profiler_JavaProfiler_recordTaskBlock0(
JNIEnv *env, jclass unused, jlong startTicks, jlong endTicks,
jlong blocker, jlong unblockingSpanId) {
int tid = ProfiledThread::currentTid();
if (tid < 0) {
return JNI_FALSE;
}
ThreadFilter *tf = Profiler::instance()->threadFilter();
if (tf->enabled()) {
tf->exitBlockedRun(slot_id, ThreadFilter::tokenGeneration(block_token));
// Context (span ids + tags) is captured from OTEP TLS via ContextApi::snapshot()
// inside recordTaskBlockLiveIfEligible, mirroring the recordQueueTime convention.
return recordTaskBlockLiveIfEligible(tid, (u64)startTicks, (u64)endTicks,
(u64)blocker, (u64)unblockingSpanId)
? JNI_TRUE
: JNI_FALSE;
}

extern "C" DLLEXPORT jboolean JNICALL
Java_com_datadoghq_profiler_JavaProfiler_recordTaskBlockWithContext0(
JNIEnv *env, jclass unused, jlong startTicks, jlong endTicks,
jlong blocker, jlong unblockingSpanId,
jlong spanId, jlong rootSpanId) {
int tid = ProfiledThread::currentTid();
if (tid < 0) {
return JNI_FALSE;
}
// Virtual-thread path: span/root ids captured at block entry are passed explicitly because

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will need some explanation about why we need to snapshot the context - are we deferring writing related events way after they were collected?


#if defined(__linux__)

class NativeFdClassifier {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add some comment block here about what the classification spec is?

});
}

ssize_t NativeSocketInterposer::write_hook(int fd, const void* buf, size_t len) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, thanks! Can we get some benchmark numbers to get the feeling about what kind of overhead we are talking here?

Comment on lines +855 to +871
bool Profiler::recordTaskBlockAsync(int tid, TaskBlockEvent *event) {
if (!_task_block_drain_running.load(std::memory_order_acquire)) {
Counters::increment(TASK_BLOCK_QUEUE_DROPPED);
return false;
}

QueuedTaskBlockEvent queued;
queued.tid = tid;
queued.generation = _task_block_generation.load(std::memory_order_acquire);
queued.event = *event;
if (!_task_block_queue.tryPush(queued)) {
Counters::increment(TASK_BLOCK_QUEUE_DROPPED);
return false;
}
return true;
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will need also some info why we must record those events asynchronously - I see this used only from monitor exit and fd interposer; both places do not seem like we could not record the event synchronously.
Could you put some explanations either to the PR description or even on the related classes, why we need this asynchronous machinery?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sphinx:critical Sphinx: critical — human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants