fix(profiler): complete malloc/native-socket hook-prefix-skip stackwalking#648
Draft
jbachorik wants to merge 5 commits into
Draft
fix(profiler): complete malloc/native-socket hook-prefix-skip stackwalking#648jbachorik wants to merge 5 commits into
jbachorik wants to merge 5 commits into
Conversation
…ix missing native frames The event name/category did not follow the datadog.* convention used by other events. Separately, Profiler::getNativeTrace unconditionally skipped native stack unwinding for BCI_NATIVE_MALLOC under the default cstack setting, leaving only Java frames in malloc call stacks.
Same guard in Profiler::getNativeTrace that was skipping native unwinding for BCI_NATIVE_MALLOC also applied to BCI_NATIVE_SOCKET; both are non-CPU/WALL event types and were bypassed when cstack is left at its default.
Marks the socket hook wrapper symbols (send/recv/write/read_hook) with MARK_ASYNC_PROFILER, mirroring the existing malloc hook marking, and teaches convertNativeTrace to discard frames captured before reaching that mark instead of truncating the stack there, so only the real caller frames remain for BCI_NATIVE_MALLOC/BCI_NATIVE_SOCKET traces.
Adds SOCKET_SAMPLE to EventType so BCI_NATIVE_SOCKET gets its own identity in walkVM, and fixes a bug where the generic is_marked check in HotspotSupport::walkVM terminated the scan on ANY mark before the MARK_ASYNC_PROFILER dispatch below it could ever run, making the malloc hook-skip logic dead code. NativeFrameResolution now carries the actual mark value so walkVM can dispatch correctly: MARK_ASYNC_PROFILER resets depth and resumes past the hook boundary for malloc/socket samples (matching the FP/DWARF path), other marks still terminate the scan as before.
Surface silent hook-mark resolution failures via counters/logging, make convertNativeTrace's MARK_ASYNC_PROFILER reset gate on skip_hook_prefix instead of the mutable skipping flag, restore the dropped MARK_COMPILER_ENTRY dispatch in walkVM, drop the now-redundant is_marked field, de-duplicate the hook-prefixed-sample predicate, and add stack trace assertions for the socket hook boundary skip. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
CI Test ResultsRun: #29114537583 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Failed Testsglibc-amd64/debug / 11-j9Job: View logs No detailed failure information available. Check the job logs. Summary: Total: 32 | Passed: 31 | Failed: 1 Updated: 2026-07-10 18:47:56 UTC |
jbachorik
changed the title
Collaborator
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?:
Fixes up gaps in the malloc/native-socket hook-prefix-skip stackwalking logic introduced earlier on this branch (
convertNativeTraceandHotspotSupport::walkVM), which was incomplete:markAsyncProfilerHooknow returnsbool;NativeSocketSampler::start()checks it).MARK_ASYNC_PROFILERhook boundary is never found in a callchain, so a fully-discarded/empty native stack is observable rather than silent.convertNativeTrace'sMARK_ASYNC_PROFILERreset to gate on the immutableskip_hook_prefixparameter instead of the mutableskippingflag, so a second boundary-marked frame later in the same callchain is still treated as a reset point (matcheswalkVM's per-frame dispatch).MARK_COMPILER_ENTRYdispatch branch inwalkVMthat had been dropped without updating thecomp_taskflag/docs that still reference it.is_markedfield onNativeFrameResolutionin favor of a computedmark != 0accessor, and updates the stale doc comments describing per-mark dispatch semantics.isHookPrefixedSample()helper (BCI and EventType overloads) used at all 4 call sites that previously hardcoded the "is this a malloc/socket sample" predicate independently.NativeSocketStackTraceTestproving a real caller frame (not a profiler-internal hook frame) is present after unwinding.Motivation:
The hook-prefix-skip stackwalking added earlier on this branch was incomplete — this closes the gaps before the branch merges.
Additional Notes:
The restored
MARK_COMPILER_ENTRYdispatch is currently unreachable:_features.comp_taskis never set to1anywhere in the codebase (no CLI flag wires it), so this fix restores doc/dead-code consistency but doesn't change observable behavior. Revisiting that separately.How to test the change?:
./gradlew assembleDebugJarbuilds clean;./gradlew compileTestJavacompiles clean. The new/updated assertions inNativeSocketStackTraceTestare parameterized acrossvm/vmx/dwarf/fpcstack modes and cover the fixed code paths.For Datadog employees:
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!