chaos: vthread-context-cascade antagonist + glibc heap-corruption defaults#642
chaos: vthread-context-cascade antagonist + glibc heap-corruption defaults#642jbachorik wants to merge 12 commits into
Conversation
…ALLOC_PERTURB_ for glibc test runs Antagonist races tracer-style context propagation against virtual-thread carrier churn to target the ContextStorageMode.THREAD stale-carrier use-after-free. Since such heap corruption manifests silently until an unrelated later allocation, also turn on glibc's own corruption checks by default for chaos runs and ddprof-test suites (skipped on musl and whenever a sanitizer/allocator already owns malloc via LD_PRELOAD). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI Test ResultsRun: #29116987971 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Failed Testsmusl-amd64/debug / 25-librcaJob: View logs No detailed failure information available. Check the job logs. musl-amd64/debug / 8-librcaJob: View logs No detailed failure information available. Check the job logs. musl-aarch64/debug / 21-librcaJob: View logs No detailed failure information available. Check the job logs. musl-aarch64/debug / 25-librcaJob: View logs No detailed failure information available. Check the job logs. musl-amd64/debug / 17-librcaJob: View logs No detailed failure information available. Check the job logs. musl-aarch64/debug / 11-librcaJob: View logs No detailed failure information available. Check the job logs. musl-amd64/debug / 21-librcaJob: View logs No detailed failure information available. Check the job logs. musl-amd64/debug / 11-librcaJob: View logs No detailed failure information available. Check the job logs. musl-aarch64/debug / 8-librcaJob: View logs No detailed failure information available. Check the job logs. musl-aarch64/debug / 17-librcaJob: View logs No detailed failure information available. Check the job logs. Summary: Total: 32 | Passed: 22 | Failed: 10 Updated: 2026-07-10 19:34:20 UTC |
J9 hits a pre-existing, unrelated glibc heap-corruption bug that MALLOC_CHECK_ now surfaces as a hard abort; gate it off until that's investigated separately.
Reliability & Chaos Results❌ 1 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/123849937 ❌ chaos: profiler tcmalloc amd64 21 0 3 temXchaos |
|
isRawPointer() only checked RAW_POINTER_MASK, letting raw J9 ASGCT BCIs that never went through encode() misroute into JVMSupport::resolve(), which asserts false for non-HotSpot VMs. Require ENCODED_MASK too.
It was registered in Main.java's antagonist factory but never added to either config's ANTAGONISTS string, so it silently never ran in CI.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e1dd0fb1c7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Include ACTIVE_PHASE_MILLIS in STALE_RACER_SLEEP_MILLIS so racers wake as the next active phase begins instead of ~3s early. Bound stopGracefully's total wait to the caller's timeout via a shared deadline instead of giving each join/tryAcquire the full budget independently.
…ntext via @trace driverLoop() called JavaProfiler.getInstance() directly, but JavaProfiler is a chaosCompileOnly dependency never bundled into chaos.jar and not visible to the app classloader (the agent's copy is shaded/relocated). The resulting NoClassDefFoundError is an Error, not an Exception, so it escaped the catch (Exception e) block and silently killed the driver thread — leaving activePhase stuck true and letting the sibling pinningChurnLoop run full-throttle for the rest of the test instead of its intended active/quiet duty cycle, which is the likely driver behind the OOM seen in reliability-chaos-aarch64 [profiler, gmalloc, 21.0.3-tem]. Rewrite the antagonist to only touch the tracer through @Trace-annotated methods (the same proven-safe dd-trace-api compileOnly pattern already used by TraceContextAntagonist), letting the real tracer and its virtual-thread instrumentation drive setContext/clearContext instead of calling com.datadoghq.profiler.* directly. Drops the custom-attribute/baggage simulation in favor of nested @trace hops plus forceUnmount and stale-buffer racer threads that still race OtelContextStorage's thread-scoped fallback across carrier churn. Verified with a 90s live run (JDK 21.0.11, profiler+tracer, gmalloc): no NoClassDefFoundError, no exceptions, completed cleanly with RC=0. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without a RUNTIME arg, chaos_check.sh silently proceeds with an empty value, which becomes --duration s in the chaos harness invocation and fails 15+ seconds later — after the JDK/agent/jar setup — with a NumberFormatException buried in the harness output, reported generically as "FAIL:Chaos harness crashed (RC=1)". Check for a missing RUNTIME right after arg parsing and fail immediately with a clear usage message. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Benchmark Results (commit 3e379a7)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/123961967 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
|
Not a review, but one thing that I can't quickly see from looking at the code: does it trigger actual migrations of vthreads across carriers? Because that is the fault scenario that led to the heap corruption. |
|
@rkennke It is trying to - but it is very probabilistic. It is trying to hold back carrier threads so new ones have to be spawned, then let the old one die so the vthread will have to be remounted but DBB still point to now invalid memory. |
J9
|
I see. If you have the time/nerve(/AI tokens) for it, you could perhaps try using a custom FJP where you can control the thread lifecycles. Probabilistic is fine, though, better than nothing. |
|
@rkennke That's a great idea. Thanks! |
Extract chaos_check.sh's logic into utils/run-chaos-harness.sh for local repro. Add a #ifdef DEBUG-only watchdog in ProfiledThread's free path (threadLocalData.cpp) that detects writes into freed OTel-context memory, and beef up VirtualThreadContextCascadeAntagonist's stale-carrier racing (continuous decoupled driver, randomized race window, dead-code removal). Fixes from review: Main.java antagonist-start try/finally, watcher thread signal-mask inheritance, racer-loop backoff under saturation, redundant frame_ut.cpp assertions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2 participants
What does this PR do?:
Adds a new
vthread-context-cascadechaos antagonist that races tracer-stylecontext propagate/activate/restore semantics against virtual-thread carrier
churn, targeting the
ContextStorageMode.THREADstale-carrierDirectByteBufferuse-after-free. Also defaultsMALLOC_CHECK_/MALLOC_PERTURB_for glibc-backed test runs (chaos reliability runner andthe shared
ddprof-testGradle test-task configuration), turning silentheap corruption into an immediate, attributable
SIGABRTinstead of acrash much later in an unrelated allocation.
Also fixes a real J9-only bug (PROF-15364) uncovered while chasing the
MALLOC_CHECK_default's fallout:FrameType::isRawPointer()(frame.h)misidentified raw, unencoded J9 ASGCT BCIs as HotSpot raw-pointer-encoded
frames, routing them into
JVMSupport::resolve()'sassert(false)fornon-HotSpot VMs — a debug-only SIGABRT. This is required for this PR's own
J9 debug CI config to pass; outside this branch/config (i.e. before the
MALLOC_CHECK_default made the J9 debug suite deterministic) it wasn'teasily reproducible.
Motivation:
While building the
vthread-context-cascadereproducer it became clearthat this class of use-after-free bug corrupts glibc heap/tcache metadata
silently and only manifests as a crash far later, in an unrelated
allocation call — making it unlikely to be caught within a short chaos
run or CI test window. Enabling glibc's own heap-corruption checks by
default closes that gap cheaply, without needing a sanitizer build (which
can't run against a live JVM, see
reference_sanitizers_no_jvm).Additional Notes:
The
MALLOC_CHECK_/MALLOC_PERTURB_defaults are skipped on musl (noMALLOC_CHECK_support) and whenever a sanitizer/allocator alreadyreplaces malloc via
LD_PRELOAD(ASan/TSan/tcmalloc/jemalloc configs),mirroring the existing
gmallocvstcmalloc/jemallocmutualexclusivity in
chaos_check.sh.The
isRawPointer()fix (frame.h) makes the check also requireENCODED_MASK(bit 20) before trustingRAW_POINTER_MASK(bit 30) —RAW_POINTER_MASKis only ever set byencode(..., rawPointer=true),which is HotSpot-only and always sets
ENCODED_MASKtoo, so a rawunencoded ASGCT BCI (used on J9) can no longer false-positive just because
it happens to have bit 30 set. PROF-15360 (originally filed as suspected
"silent glibc heap corruption on J9") is closed in favor of PROF-15364,
which documents this corrected root cause.
How to test the change?:
./gradlew :ddprof-stresstest:chaosJarbuilds cleanly with the new antagonist registered../gradlew :build-logic:conventions:compileKotlincompiles cleanly with theProfilerTestPluginchange.-Dddprof.debug.context.storage.mode=thread, tuned virtual-thread scheduler flags,MALLOC_CHECK_=3/MALLOC_PERTURB_set) for several minutes; no crash reproduced yet in this window — the antagonist is being landed as a reliability-suite addition to keep exercising this race over longer soak runs, not as a guaranteed-repro smoke test.FrameTypeIsRawPointerTest.FalseForRawAsgctBciWithBit30SetButNoEncodedMarkertoframe_ut.cpp(regression test for theisRawPointer()fix);ddprof-lib:gtestDebug_frame_utpasses 18/18, including the pre-existingTrueWhenBit30IsSetcase (no regression on the legitimate HotSpot path)../gradlew :ddprof-lib:buildDebug -x testsucceeds.glibc/jdk8/j9/aarch64debug container (Assertion 'false && "Should not reach here"' failedatjvmSupport.inline.h:34) before this fix — this is the repro that led to filing and fixing PROF-15364.For Datadog employees:
If this PR touches code that signs or publishes builds or packages, or handles
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.This PR doesn't touch any of that.
JIRA: PROF-15364 (fixed by this PR); closes [PROF-15360]
Unsure? Have a question? Request a review!