Reference chains for surviving live-heap samples (PROF-15341)#644
Reference chains for surviving live-heap samples (PROF-15341)#644jbachorik wants to merge 9 commits into
Conversation
- ReferenceChainTracker: JVMTI tag/GC-signal lifecycle, frontier table, bounded BFS, incremental resumption, JFR reporting - wired into Profiler start/stop/dump; referencechains= config flag - design doc, implementation plan, benchmark plan Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ure Open Question brainstorm - New gtest + companion JMC test empirically confirm JMC's parser correctly resolves the datadog.ReferenceChain "chain" field's F_CPOOL|F_ARRAY encoding (array of class-constant-pool indices) - previously untested end-to-end. - Design doc: capture proposed (not yet implemented) resolutions for Open Questions 2/3/5 - PID-controller-driven pause-time-SLO feedback loop for per-pass budget/cadence, and positive-population-slope klass ranking for buildChainEvent()'s target-selection policy.
Fixes a structural gap: a one-shot BFS search could finish walking the whole reachable graph before LivenessTracker's population-trend detection had enough epochs to flag a candidate, making later-created leaking objects permanently undiscoverable. - PainBudget (painBudget.h): leaky bucket over cost (ms), not event rate - gates how soon a restarted search may take its first pass. - ReferenceChainTracker::canAffordNewSearch()/restartSearch(): a terminal search restarts once LivenessTracker reports a leak candidate and the pain budget has drained; unaffected when generations=true isn't set. - FrontierTable::resetForRestart(), painbudget=N config sub-option (default 1%). - gtest coverage for PainBudget and the restart gate; verified against ExternalProcessReferenceChainTest (separate-process leaking-cache scenario), which now passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
CI Test ResultsRun: #29095645195 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Failed Testsmusl-amd64/debug / 21-librcaJob: View logs No detailed failure information available. Check the job logs. musl-amd64/debug / 17-librcaJob: View logs No detailed failure information available. Check the job logs. musl-aarch64/debug / 25-librcaJob: View logs No detailed failure information available. Check the job logs. musl-aarch64/debug / 21-librcaJob: View logs No detailed failure information available. Check the job logs. musl-amd64/debug / 25-librcaJob: View logs No detailed failure information available. Check the job logs. musl-aarch64/debug / 11-librcaJob: View logs No detailed failure information available. Check the job logs. glibc-aarch64/debug / 21Job: View logs No detailed failure information available. Check the job logs. glibc-aarch64/debug / 17Job: View logs No detailed failure information available. Check the job logs. glibc-aarch64/debug / 17-graalJob: View logs No detailed failure information available. Check the job logs. glibc-amd64/debug / 25Job: View logs No detailed failure information available. Check the job logs. glibc-amd64/debug / 17-graalJob: View logs No detailed failure information available. Check the job logs. musl-aarch64/debug / 17-librcaJob: View logs No detailed failure information available. Check the job logs. glibc-amd64/debug / 11Job: View logs No detailed failure information available. Check the job logs. glibc-amd64/debug / 17Job: View logs No detailed failure information available. Check the job logs. glibc-amd64/debug / 21-graalJob: View logs No detailed failure information available. Check the job logs. glibc-amd64/debug / 21Job: View logs No detailed failure information available. Check the job logs. glibc-aarch64/debug / 21-graalJob: View logs No detailed failure information available. Check the job logs. glibc-amd64/debug / 25-graalJob: View logs No detailed failure information available. Check the job logs. musl-amd64/debug / 11-librcaJob: View logs No detailed failure information available. Check the job logs. glibc-aarch64/debug / 25-graalJob: View logs No detailed failure information available. Check the job logs. glibc-aarch64/debug / 25Job: View logs No detailed failure information available. Check the job logs. Summary: Total: 32 | Passed: 11 | Failed: 21 Updated: 2026-07-10 13:47:07 UTC |
…ce-chain JFR roundtrip test Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…shared test-JVM heap Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…owth _table_size was volatile but updated via __sync_bool_compare_and_swap, which has no synchronizes-with edge under the C++ memory model - switch to std::atomic<int> with relaxed ops. Also clamp the reference-chain tests' per-round allocation growth at round 10 (trend-eligibility point) so retry rounds stop adding unbounded garbage, which was causing OOME in CI's shared test-JVM heap.
Reliability & Chaos Results✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/124004130 |
The glibc-aarch64 job never installed gtest/gmock, unlike amd64 and musl jobs, so all native gtests silently skipped there. This surfaced as a hard failure once ReferenceChainJfrParserTest started depending on the roundtrip gtest actually producing its JFR file.
…from reference-chain BFS reconstruction Both mechanisms are probabilistic and hard to reliably co-occur in one JVM run; new JNI test seams (guarded by #ifdef DEBUG) let JUnit tests seed population history and tag a known live object directly, verifying each end-to-end in isolation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Benchmark Results (commit cfec595)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/123822759 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 25 | 💥 0 ms (0 iters) | 💥 0 ms (0 iters) | — | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1949 / 2055 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / · | ✅ / · | ✅ / · | ✅ / · | ✅ / · | ✅ / · |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 8727 / 8389 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8166 / 8396 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1292 / 1256 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1288 / 1290 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2830 / 3023 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2976 / 2821 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | · / ✅ | · / ✅ | · / 3 | · / 3555 | · / ✅ | · / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 3499 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1592 / 1697 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1922 / 1854 | ✅ / ✅ | ✅ / ✅ |
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Benchmark Results (commit 0714bd3)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124004132 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10184 ms (21 iters) | ✅ 10371 ms (21 iters) | ≈ +1.8% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8891 ms (24 iters) | ✅ 8865 ms (24 iters) | ≈ -0.3% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6003 ms (33 iters) | ✅ 6034 ms (33 iters) | ≈ +0.5% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5507 ms (36 iters) | ✅ 5426 ms (36 iters) | ≈ -1.5% (±23.8%) | |
| fj-kmeans | 21 | ✅ 2834 ms (66 iters) | ✅ 2761 ms (67 iters) | 🟢 -2.6% | — / — |
| fj-kmeans | 25 | ✅ 2824 ms (66 iters) | ✅ 2838 ms (66 iters) | ≈ +0.5% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2113 ms (88 iters) | ✅ 2065 ms (89 iters) | ≈ -2.3% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2059 ms (90 iters) | ✅ 2102 ms (89 iters) | ≈ +2.1% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1283 ms (134 iters) | ✅ 1260 ms (135 iters) | ≈ -1.8% (±32%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (169 iters) | ✅ 1017 ms (168 iters) | ≈ +0.1% (±31.5%) | — / — |
| reactors | 21 | ✅ 16658 ms (15 iters) | ✅ 16371 ms (16 iters) | ≈ -1.7% (±9%) | — / — |
| reactors | 25 | ✅ 18507 ms (15 iters) | ✅ 18056 ms (15 iters) | ≈ -2.4% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1929 / 2034 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2310 / 2113 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 8529 / 8852 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8635 / 8390 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 5 | 1284 / 1281 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 2 | 1283 / 1285 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2930 / 2864 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2963 / 2953 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 7 / 6 | 3558 / 3510 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 3463 / 3469 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1564 / 1782 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1905 / 1681 | ✅ / ✅ | ✅ / ✅ |
1 participant
What does this PR do?:
Implements reference-chain reconstruction for live-heap samples that survive
past their allocation window (PROF-15341): a JVMTI
FollowReferences-basedBFS walk (
ReferenceChainTracker) tags reachable objects and recordsreferrer-type chains back toward a GC root, bridged to
LivenessTracker::selectLeakCandidates()'s population-slope leak detection.This revision also closes a structural gap found while adding end-to-end
coverage: a one-shot search could finish walking the whole reachable graph
before population-trend detection had accumulated enough GC epochs to flag a
candidate, making any object allocated after that point permanently
undiscoverable. Search restarts are now gated on an actual leak indication
existing (so a restarted walk is very likely to still find the object) plus
a
PainBudgetcooldown - a leaky bucket over cumulative safepoint timespent, not a fixed interval, so a cheap search can restart sooner than an
expensive one.
Motivation:
PROF-15341 - give live-heap samples that survive long enough to look
leak-shaped an actual referrer chain, not just "this object is still alive".
Additional Notes:
See
doc/architecture/LiveHeapReferenceChains.mdanddoc/architecture/LiveHeapReferenceChains-RemainingWorkPlan.mdfor the fulldesign history, open questions, and what's still provisional (per-pass
budget/hop-cap/TTL/frontier-cap/pain-budget-refill-rate defaults are all
round, unbenchmarked placeholders pending a future measurement pass).
How to test the change?:
./gradlew :ddprof-lib:gtestDebug- full native unit suite, including newPainBudgetTest/SearchRestartTestcoverage for the restart gate../gradlew :ddprof-test:testDebug --tests "*ExternalProcessReferenceChainTest*"-genuinely separate-process end-to-end test: runs a real leaking-cache Java
app in a child JVM and asserts the reconstructed chain's leaf class.
./gradlew :ddprof-test:testDebug --tests "*ReferenceChainTrackingTest*"-in-process coverage for the BFS engine, target-selection bridging, and
abandonment reporting.
For Datadog employees:
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!
🤖 Generated with Claude Code