chore(crashtracking): harden multi thread ptrace collection by gyuheon0h · Pull Request #2216 · DataDog/libdatadog

gyuheon0h · 2026-07-08T21:59:42Z

What does this PR do?

test_crash_tracking_multi_thread_collection has been failing ~10% of the time on CentOS 7 CI. The test spawns worker threads in spin loops, crashes, and validates that the receiver's ptrace-based collector captured stack frames for all threads. Failures showed ct_worker_X stack should contain a frame for 'worker_fn_X' but got: []

I built the CentOS 7 Docker image from tools/docker/Dockerfile.centos and ran the test inside it. It passed 100/100 in isolation and 500/500 at full CPU. However, when I trottled the CPU (docker --cpus=0.5) I was able to reproduced the failure at ~10% rate.

By adding some logging, I was able to see this

retry tid=120 attempt=0: transient error Attach(120, 110)   // ETIMEDOUT
retry tid=120 attempt=1: permanent error Attach(120, 1)     // EPERM

Under CPU pressure, wait_for_stop exceeded the 50ms STOP_TIMEOUT_PER_THREAD
The ETIMEDOUT path called detach_thread then retried
The retry's PTRACE_SEIZE got EPERM because the kernel hadn't fully processed the detach
EPERM was treated as permanent -> gave up -> 0 frames

Changes

STOP_TIMEOUT_PER_THREAD: 50ms to 200ms; 50ms was too short under CPU contention. The thread genuinely couldn't be scheduled to enter ptrace-stop in time.
Non-fatal wait_for_registers -- Changed from detach + return ETIMEDOUT to let _ = wait_for_registers(...). libunwind uses PTRACE_GETREGSET which can succeed even when PTRACE_PEEKUSER returns zero. This avoids the detach re-attach cycle that triggered EPERM.
waitpid drain after detach -- Added waitpid(tid, NULL, __WALL | WNOHANG) after PTRACE_DETACH to consume pending kernel events, ensuring the thread is fully released before any retry can re-seize it.

The retry-with-exponential-backoff mechanism was added in earlier iterations and remains in the final fix for additional strengthening

I modified test.yaml to run the test 100x in CI: https://github.com/DataDog/libdatadog/actions/runs/29366237931/job/87198658036?pr=2216

Prior to these changes, this test would consistently fail, but now, it does not.

Motivation

Too many flakes on these tests

Additional Notes

Anything else we should know when reviewing?

How to test the change?

Describe here in detail how the change can be validated.

github-actions · 2026-07-08T22:01:56Z

📚 Documentation Check Results

⚠️ 1088 documentation warning(s) found

📦 `libdd-crashtracker` - 1088 warning(s)

Updated: 2026-07-15 13:14:42 UTC | Commit: 75e70fe | missing-docs job results

datadog-datadog-prod-us1 · 2026-07-08T22:03:13Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 34.62%
• Overall Coverage: 74.42% (-0.05%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 7ebd2b9 | Docs | Datadog PR Page | Give us feedback!}

github-actions · 2026-07-08T22:03:47Z

🔒 Cargo Deny Results

⚠️ 2 issue(s) found, showing only errors (advisories, bans, sources)

📦 `libdd-crashtracker` - 2 error(s)

Show output

error[vulnerability]: Invalid pointer dereference in `fmt::Pointer` impl for `Atomic` and `Shared` when the underlying pointer is invalid
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:59:1
   │
59 │ crossbeam-epoch 0.9.18 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0204
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0204
   ├ Affected versions of `fmt::Display` dereference the underlying pointer. This causes a invalid pointer dereference e.g., when a pointer created with `Atomic::null` or `Shared::null`. `fmt::Debug` impls and pre-0.9 `fmt::Display` impls, which do not dereference pointers, are not affected by this issue.
   ├ Announcement: https://github.com/crossbeam-rs/crossbeam/pull/1276
   ├ Solution: Upgrade to >=0.9.20 (try `cargo update -p crossbeam-epoch`)
   ├ crossbeam-epoch v0.9.18
     └── crossbeam-deque v0.8.5
         └── rayon-core v1.12.1
             └── rayon v1.10.0
                 └── criterion v0.5.1
                     ├── (dev) libdd-crashtracker v1.0.0
                     └── (dev) libdd-ddsketch v1.1.0
                         └── libdd-telemetry v6.0.0
                             └── libdd-crashtracker v1.0.0 (*)

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:204:1
    │
204 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0097
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
    ├ It has been reported (by @lopopolo) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
      
      - The `log` and `thread_rng` features are enabled
      - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
      - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
      - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
      - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
      
      `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
    ├ Announcement: https://github.com/rust-random/rand/pull/1763
    ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
    ├ rand v0.8.5
      ├── libdd-common v5.1.0
      │   ├── libdd-capabilities-impl v3.0.0
      │   │   └── libdd-shared-runtime v2.0.0
      │   │       └── libdd-telemetry v6.0.0
      │   │           └── libdd-crashtracker v1.0.0
      │   ├── (build) libdd-crashtracker v1.0.0 (*)
      │   ├── libdd-shared-runtime v2.0.0 (*)
      │   └── libdd-telemetry v6.0.0 (*)
      ├── libdd-crashtracker v1.0.0 (*)
      └── (dev) libdd-ddsketch v1.1.0
          └── libdd-telemetry v6.0.0 (*)

advisories FAILED, bans ok, sources ok

📦 `libdd-profiling-heap-gotter` - ✅ No issues

Updated: 2026-07-15 13:15:44 UTC | Commit: ac5e8a5 | dependency-check job results

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: edd03dc8f1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

gyuheon0h · 2026-07-08T22:28:08Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: edd03dc8f1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

dd-octo-sts · 2026-07-08T22:37:44Z

Artifact Size Benchmark Report

aarch64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a	85.88 MB	85.88 MB	+0% (+3.48 KB) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so	7.88 MB	7.88 MB	0% (0 B) 👌

aarch64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so	10.61 MB	10.61 MB	+0% (+256 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a	97.10 MB	97.10 MB	+0% (+2.98 KB) 👌

libdatadog-x64-windows

Artifact	Baseline	Commit	Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll	25.45 MB	25.45 MB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib	88.44 KB	88.44 KB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb	184.54 MB	184.55 MB	+0% (+8.00 KB) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib	946.77 MB	946.77 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll	8.32 MB	8.32 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib	88.44 KB	88.44 KB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb	24.62 MB	24.62 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib	49.03 MB	49.03 MB	0% (0 B) 👌

libdatadog-x86-windows

Artifact	Baseline	Commit	Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll	22.05 MB	22.05 MB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib	89.82 KB	89.82 KB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb	188.76 MB	188.76 MB	0% (0 B) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib	935.45 MB	935.45 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll	6.43 MB	6.43 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib	89.82 KB	89.82 KB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb	26.43 MB	26.43 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib	46.66 MB	46.66 MB	0% (0 B) 👌

x86_64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a	76.58 MB	76.58 MB	+0% (+3.21 KB) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so	8.78 MB	8.78 MB	0% (0 B) 👌

x86_64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a	92.10 MB	92.11 MB	+0% (+2.67 KB) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so	10.69 MB	10.69 MB	+0% (+240 B) 👌

pr-commenter · 2026-07-09T16:14:47Z

Benchmarks

Comparison

Benchmark execution time: 2026-07-15 13:22:32

Comparing candidate commit 7ebd2b9 in PR branch gyuheon0h/multi-thread-collection-harden with baseline commit a10f667 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 1 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Candidate

Candidate benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`7ebd2b9`	1784121131	gyuheon0h/multi-thread-collection-harden

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
receiver_entry_point/report/2644	execution_time	3.344ms	3.377ms ± 0.026ms	3.368ms ± 0.013ms	3.387ms	3.428ms	3.460ms	3.472ms	3.09%	1.299	1.296	0.76%	0.002ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
receiver_entry_point/report/2644	execution_time	[3.373ms; 3.380ms] or [-0.106%; +0.106%]	None	None	None

Baseline

Baseline benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`a10f667`	1784074032	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
receiver_entry_point/report/2644	execution_time	3.339ms	3.364ms ± 0.016ms	3.360ms ± 0.008ms	3.370ms	3.393ms	3.415ms	3.425ms	1.95%	1.391	2.247	0.48%	0.001ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
receiver_entry_point/report/2644	execution_time	[3.361ms; 3.366ms] or [-0.067%; +0.067%]	None	None	None

gyuheon0h · 2026-07-14T13:41:58Z

chore(crashtracking): harden multi thread ptrace collection #2216 👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7ebd2b92ea

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

gyuheon0h requested a review from a team as a code owner July 8, 2026 21:59

gyuheon0h marked this pull request as draft July 8, 2026 21:59

github-actions Bot added the crashtracker label Jul 8, 2026

chatgpt-codex-connector Bot reviewed Jul 8, 2026

View reviewed changes

Comment thread libdd-crashtracker/src/receiver/ptrace_collector.rs Outdated

Comment thread libdd-crashtracker/src/receiver/ptrace_collector.rs Outdated

gyuheon0h changed the title ~~chore(crashtracking): harden multi thread ptrace collection~~ chore(crashtracking): harden multi thread ptrace collection Jul 8, 2026

chatgpt-codex-connector Bot reviewed Jul 8, 2026

View reviewed changes

Comment thread libdd-crashtracker/src/receiver/ptrace_collector.rs Outdated

github-actions Bot added the ci-build label Jul 9, 2026

gyuheon0h added 4 commits July 9, 2026 17:47

Harden ptrace retries

d6f07c7

Only retry register wait timeout

07507f6

Run test 100x

4f21b6e

Retry for 0 frames

dc76d1f

gyuheon0h force-pushed the gyuheon0h/multi-thread-collection-harden branch from d292c2f to dc76d1f Compare July 9, 2026 17:54

gyuheon0h force-pushed the gyuheon0h/multi-thread-collection-harden branch from 39e3d72 to 39c9f0f Compare July 14, 2026 14:26

github-actions Bot removed the ci-build label Jul 14, 2026

gyuheon0h force-pushed the gyuheon0h/multi-thread-collection-harden branch from 39c9f0f to a87179c Compare July 14, 2026 14:50

github-actions Bot added the ci-build label Jul 14, 2026

gyuheon0h force-pushed the gyuheon0h/multi-thread-collection-harden branch from a87179c to 78b3ebf Compare July 14, 2026 17:59

Trigger CI

eac1414

gyuheon0h force-pushed the gyuheon0h/multi-thread-collection-harden branch 2 times, most recently from fe8db49 to 25d74f4 Compare July 15, 2026 13:11

github-actions Bot removed the ci-build label Jul 15, 2026

Revert 100x run

7ebd2b9

gyuheon0h force-pushed the gyuheon0h/multi-thread-collection-harden branch from 25d74f4 to 7ebd2b9 Compare July 15, 2026 13:12

gyuheon0h marked this pull request as ready for review July 15, 2026 13:19

gyuheon0h requested review from gleocadie and hoolioh July 15, 2026 13:24

chatgpt-codex-connector Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread libdd-crashtracker/src/receiver/ptrace_collector.rs

gleocadie approved these changes Jul 15, 2026

View reviewed changes

hoolioh approved these changes Jul 15, 2026

View reviewed changes

gyuheon0h merged commit 11748c6 into main Jul 15, 2026
76 of 80 checks passed

gyuheon0h deleted the gyuheon0h/multi-thread-collection-harden branch July 15, 2026 14:22

Uh oh!

Conversation

gyuheon0h commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

How to test the change?

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📚 Documentation Check Results

📦 libdd-crashtracker - 1088 warning(s)

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 8, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔒 Cargo Deny Results

📦 libdd-crashtracker - 2 error(s)

📦 libdd-profiling-heap-gotter - ✅ No issues

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

gyuheon0h commented Jul 8, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Artifact Size Benchmark Report

Uh oh!

pr-commenter Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Comparison

Explanation

More details about the CI and significant changes

Candidate

Group 1

Baseline

Group 1

Uh oh!

gyuheon0h commented Jul 14, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gyuheon0h commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

📦 `libdd-crashtracker` - 1088 warning(s)

datadog-datadog-prod-us1 Bot commented Jul 8, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

📦 `libdd-crashtracker` - 2 error(s)

📦 `libdd-profiling-heap-gotter` - ✅ No issues

dd-octo-sts Bot commented Jul 8, 2026 •

edited

Loading

pr-commenter Bot commented Jul 9, 2026 •

edited

Loading