Add _dd.p.ksr propagated tag for Knuth sampling rate by bm1549 · Pull Request #10802 · DataDog/dd-trace-java

bm1549 · 2026-03-11T02:06:47Z

What Does This Do

Adds _dd.p.ksr (Knuth Sampling Rate) as a propagated tag set when agent-based or rule-based sampling decisions are made. The tag is stored in span meta (string type) with up to 6 significant digits and no trailing zeros. It propagates via x-datadog-tags header and W3C tracestate (t.ksr).

Motivation

To enable consistent sampling across tracers and backend retention filters, the backend needs to know the sampling rate applied by the tracer. Without transmitting the tracer's rate via _dd.p.ksr, backend resampling cannot correctly compute effective rates in multi-stage sampling scenarios.

See RFC: "Transmit Knuth sampling rate to backend"

Additional Notes

Key files changed:

DDSpan.java — setSamplingPriority() integration (decides when to set ksr)
PTagsFactory.java — formatKnuthSamplingRate() and updateKnuthSamplingRate(double) (owns formatting and storage)
PropagationTags.java — Abstract API accepting raw double rate
PTagsCodec.java — Encoding/decoding ksr in x-datadog-tags header
KnuthSamplingRateTest.groovy — Unit tests for formatting, agent/rule sampling, propagation
KnuthSamplingRateFormatBenchmark.java — JMH benchmark comparing formatting approaches

Design decisions:

Formatting logic (formatKnuthSamplingRate) lives in the propagation layer (PTagsFactory), not in DDSpan, since _dd.p.ksr is a propagated tag and encoding belongs with propagation
formatKnuthSamplingRate uses manual char-array arithmetic instead of String.format to avoid Formatter/stream/boxing allocations (~30x faster: 320ns → 11ns)
The TagValue is eagerly cached in updateKnuthSamplingRate() so that getKnuthSamplingRateTagValue() is a simple volatile read (~1.2ns) with zero allocation on the header-injection hot path
updateKnuthSamplingRate accepts double so callers don't need to know the formatting contract

Related PRs across tracers:

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors

Jira ticket: APMSP-1867

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Remove unused imports in KnuthSamplingRateTest (CodeNarc violations) - Update OT31ApiTest and OT33ApiTest to expect _dd.p.ksr in x-datadog-tags when agent-rate sampler runs (UNSET priority) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The _dd.p.ksr propagated tag also appears in W3C tracestate as t.ksr. Update OT31 and OT33 test expectations for the UNSET context priority case where the agent-rate sampler runs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Replace String#replaceAll (a forbidden API in this codebase) with manual character-based trailing-zero stripping logic that has the same semantics but avoids the regex-based method. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The PTagsCodec headerValue method outputs tags in order: dm, tid, ksr. The test datadogTags list had ksr before tid, causing a comparison failure. Reorder to match the actual output: dm, tid, ksr. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The ksr implementation now adds _dd.p.ksr tag to spans with agent sampling rate, so the msgpack serialization test expectations need to include it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- OpenTelemetryTest: fix x-datadog-tags ordering (tid before ksr) - DatadogPropagatorTest: add ksr to expected tags when UNSET priority - OpenTracing32Test: add ksr and tid handling for UNSET priority case All follow PTagsCodec ordering: dm → tid → ksr Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Update the test inject extract expectations to include _dd.p.ksr=1 in x-datadog-tags and t.ksr:1 in tracestate for the UNSET sampling case, following PTagsCodec ordering: dm -> tid -> ksr. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pr-commenter · 2026-03-11T04:58:54Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	brian.marks/add-ksr-tag
git_commit_date	1773845838	1773954620
git_commit_sha	`a953f33`	`e73c492`
release_version	1.61.0-SNAPSHOT~a953f33c70	1.61.0-SNAPSHOT~e73c492734

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1773956376	1773956376
ci_job_id	1523704420	1523704420
ci_pipeline_id	103615480	103615480
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-djbq719i 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-djbq719i 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 64 metrics, 7 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~e73c492734, baseline=1.61.0-SNAPSHOT~a953f33c70

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1053785
Total [baseline] (11.053 s) : 0, 11053273
Agent [candidate] (1.055 s) : 0, 1054626
Total [candidate] (11.019 s) : 0, 11019146
section appsec
Agent [baseline] (1.244 s) : 0, 1244279
Total [baseline] (11.222 s) : 0, 11221903
Agent [candidate] (1.253 s) : 0, 1252634
Total [candidate] (11.143 s) : 0, 11142706
section iast
Agent [baseline] (1.228 s) : 0, 1227567
Total [baseline] (11.268 s) : 0, 11268179
Agent [candidate] (1.233 s) : 0, 1233301
Total [candidate] (11.36 s) : 0, 11360373
section profiling
Agent [baseline] (1.182 s) : 0, 1181610
Total [baseline] (10.962 s) : 0, 10961792
Agent [candidate] (1.181 s) : 0, 1180850
Total [candidate] (10.916 s) : 0, 10916465

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.054 s	-
Agent	appsec	1.244 s	190.495 ms (18.1%)
Agent	iast	1.228 s	173.783 ms (16.5%)
Agent	profiling	1.182 s	127.826 ms (12.1%)
Total	tracing	11.053 s	-
Total	appsec	11.222 s	168.631 ms (1.5%)
Total	iast	11.268 s	214.906 ms (1.9%)
Total	profiling	10.962 s	-91.481 ms (-0.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.055 s	-
Agent	appsec	1.253 s	198.009 ms (18.8%)
Agent	iast	1.233 s	178.676 ms (16.9%)
Agent	profiling	1.181 s	126.224 ms (12.0%)
Total	tracing	11.019 s	-
Total	appsec	11.143 s	123.56 ms (1.1%)
Total	iast	11.36 s	341.226 ms (3.1%)
Total	profiling	10.916 s	-102.681 ms (-0.9%)

gantt
    title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~e73c492734, baseline=1.61.0-SNAPSHOT~a953f33c70

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.178 ms) : 0, 1178
crashtracking [candidate] (1.213 ms) : 0, 1213
BytebuddyAgent [baseline] (627.284 ms) : 0, 627284
BytebuddyAgent [candidate] (627.77 ms) : 0, 627770
AgentMeter [baseline] (29.005 ms) : 0, 29005
AgentMeter [candidate] (29.037 ms) : 0, 29037
GlobalTracer [baseline] (255.968 ms) : 0, 255968
GlobalTracer [candidate] (256.196 ms) : 0, 256196
AppSec [baseline] (31.598 ms) : 0, 31598
AppSec [candidate] (31.534 ms) : 0, 31534
Debugger [baseline] (60.054 ms) : 0, 60054
Debugger [candidate] (60.074 ms) : 0, 60074
Remote Config [baseline] (580.391 µs) : 0, 580
Remote Config [candidate] (584.365 µs) : 0, 584
Telemetry [baseline] (8.002 ms) : 0, 8002
Telemetry [candidate] (8.646 ms) : 0, 8646
Flare Poller [baseline] (4.232 ms) : 0, 4232
Flare Poller [candidate] (3.554 ms) : 0, 3554
section appsec
crashtracking [baseline] (1.186 ms) : 0, 1186
crashtracking [candidate] (1.199 ms) : 0, 1199
BytebuddyAgent [baseline] (657.075 ms) : 0, 657075
BytebuddyAgent [candidate] (662.264 ms) : 0, 662264
AgentMeter [baseline] (11.929 ms) : 0, 11929
AgentMeter [candidate] (12.071 ms) : 0, 12071
GlobalTracer [baseline] (257.334 ms) : 0, 257334
GlobalTracer [candidate] (259.451 ms) : 0, 259451
IAST [baseline] (24.149 ms) : 0, 24149
IAST [candidate] (24.384 ms) : 0, 24384
AppSec [baseline] (177.554 ms) : 0, 177554
AppSec [candidate] (177.844 ms) : 0, 177844
Debugger [baseline] (66.24 ms) : 0, 66240
Debugger [candidate] (66.438 ms) : 0, 66438
Remote Config [baseline] (618.68 µs) : 0, 619
Remote Config [candidate] (622.814 µs) : 0, 623
Telemetry [baseline] (8.322 ms) : 0, 8322
Telemetry [candidate] (8.417 ms) : 0, 8417
Flare Poller [baseline] (3.626 ms) : 0, 3626
Flare Poller [candidate] (3.603 ms) : 0, 3603
section iast
crashtracking [baseline] (1.18 ms) : 0, 1180
crashtracking [candidate] (1.186 ms) : 0, 1186
BytebuddyAgent [baseline] (796.638 ms) : 0, 796638
BytebuddyAgent [candidate] (799.148 ms) : 0, 799148
AgentMeter [baseline] (11.289 ms) : 0, 11289
AgentMeter [candidate] (11.4 ms) : 0, 11400
GlobalTracer [baseline] (247.263 ms) : 0, 247263
GlobalTracer [candidate] (248.997 ms) : 0, 248997
IAST [baseline] (25.293 ms) : 0, 25293
IAST [candidate] (25.595 ms) : 0, 25595
AppSec [baseline] (26.46 ms) : 0, 26460
AppSec [candidate] (26.829 ms) : 0, 26829
Debugger [baseline] (70.464 ms) : 0, 70464
Debugger [candidate] (71.006 ms) : 0, 71006
Remote Config [baseline] (529.592 µs) : 0, 530
Remote Config [candidate] (526.829 µs) : 0, 527
Telemetry [baseline] (9.097 ms) : 0, 9097
Telemetry [candidate] (9.203 ms) : 0, 9203
Flare Poller [baseline] (3.294 ms) : 0, 3294
Flare Poller [candidate] (3.376 ms) : 0, 3376
section profiling
crashtracking [baseline] (1.169 ms) : 0, 1169
crashtracking [candidate] (1.177 ms) : 0, 1177
BytebuddyAgent [baseline] (682.384 ms) : 0, 682384
BytebuddyAgent [candidate] (681.757 ms) : 0, 681757
AgentMeter [baseline] (8.607 ms) : 0, 8607
AgentMeter [candidate] (8.562 ms) : 0, 8562
GlobalTracer [baseline] (215.304 ms) : 0, 215304
GlobalTracer [candidate] (215.241 ms) : 0, 215241
AppSec [baseline] (32.312 ms) : 0, 32312
AppSec [candidate] (32.337 ms) : 0, 32337
Debugger [baseline] (64.204 ms) : 0, 64204
Debugger [candidate] (64.934 ms) : 0, 64934
Remote Config [baseline] (565.351 µs) : 0, 565
Remote Config [candidate] (566.423 µs) : 0, 566
Telemetry [baseline] (9.334 ms) : 0, 9334
Telemetry [candidate] (8.488 ms) : 0, 8488
Flare Poller [baseline] (3.474 ms) : 0, 3474
Flare Poller [candidate] (3.444 ms) : 0, 3444
ProfilingAgent [baseline] (93.572 ms) : 0, 93572
ProfilingAgent [candidate] (93.657 ms) : 0, 93657
Profiling [baseline] (94.131 ms) : 0, 94131
Profiling [candidate] (94.216 ms) : 0, 94216

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~e73c492734, baseline=1.61.0-SNAPSHOT~a953f33c70

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1055504
Total [baseline] (8.833 s) : 0, 8833184
Agent [candidate] (1.061 s) : 0, 1061396
Total [candidate] (8.869 s) : 0, 8869025
section iast
Agent [baseline] (1.226 s) : 0, 1226246
Total [baseline] (9.539 s) : 0, 9539028
Agent [candidate] (1.237 s) : 0, 1236801
Total [candidate] (9.571 s) : 0, 9571147

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.056 s	-
Agent	iast	1.226 s	170.741 ms (16.2%)
Total	tracing	8.833 s	-
Total	iast	9.539 s	705.844 ms (8.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.061 s	-
Agent	iast	1.237 s	175.405 ms (16.5%)
Total	tracing	8.869 s	-
Total	iast	9.571 s	702.122 ms (7.9%)

gantt
    title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~e73c492734, baseline=1.61.0-SNAPSHOT~a953f33c70

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.196 ms) : 0, 1196
crashtracking [candidate] (1.201 ms) : 0, 1201
BytebuddyAgent [baseline] (628.872 ms) : 0, 628872
BytebuddyAgent [candidate] (631.584 ms) : 0, 631584
AgentMeter [baseline] (28.988 ms) : 0, 28988
AgentMeter [candidate] (29.382 ms) : 0, 29382
GlobalTracer [baseline] (256.114 ms) : 0, 256114
GlobalTracer [candidate] (257.983 ms) : 0, 257983
AppSec [baseline] (31.455 ms) : 0, 31455
AppSec [candidate] (31.77 ms) : 0, 31770
Debugger [baseline] (59.229 ms) : 0, 59229
Debugger [candidate] (59.803 ms) : 0, 59803
Remote Config [baseline] (578.687 µs) : 0, 579
Remote Config [candidate] (581.336 µs) : 0, 581
Telemetry [baseline] (8.008 ms) : 0, 8008
Telemetry [candidate] (9.462 ms) : 0, 9462
Flare Poller [baseline] (5.006 ms) : 0, 5006
Flare Poller [candidate] (3.57 ms) : 0, 3570
section iast
crashtracking [baseline] (1.179 ms) : 0, 1179
crashtracking [candidate] (1.194 ms) : 0, 1194
BytebuddyAgent [baseline] (795.532 ms) : 0, 795532
BytebuddyAgent [candidate] (803.794 ms) : 0, 803794
AgentMeter [baseline] (11.317 ms) : 0, 11317
AgentMeter [candidate] (11.573 ms) : 0, 11573
GlobalTracer [baseline] (246.83 ms) : 0, 246830
GlobalTracer [candidate] (248.755 ms) : 0, 248755
IAST [baseline] (25.3 ms) : 0, 25300
IAST [candidate] (25.471 ms) : 0, 25471
AppSec [baseline] (26.454 ms) : 0, 26454
AppSec [candidate] (26.74 ms) : 0, 26740
Debugger [baseline] (69.67 ms) : 0, 69670
Debugger [candidate] (68.738 ms) : 0, 68738
Remote Config [baseline] (533.675 µs) : 0, 534
Remote Config [candidate] (517.637 µs) : 0, 518
Telemetry [baseline] (9.784 ms) : 0, 9784
Telemetry [candidate] (10.13 ms) : 0, 10130
Flare Poller [baseline] (3.535 ms) : 0, 3535
Flare Poller [candidate] (3.65 ms) : 0, 3650

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	brian.marks/add-ksr-tag
git_commit_date	1773845838	1773954620
git_commit_sha	`a953f33`	`e73c492`
release_version	1.61.0-SNAPSHOT~a953f33c70	1.61.0-SNAPSHOT~e73c492734

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1773956849	1773956849
ci_job_id	1523704421	1523704421
ci_pipeline_id	103615480	103615480
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-7ox31n0q 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-7ox31n0q 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 0 performance regressions! Performance is the same for 17 metrics, 17 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:petclinic:profiling:high_load	better [-1463.294µs; -431.168µs] or [-7.598%; -2.239%]	unsure [-1814.271µs; -67.620µs] or [-5.826%; -0.217%]	unstable [-18.264op/s; +36.639op/s] or [-7.621%; +15.288%]	18.312ms	30.198ms	248.844op/s	19.259ms	31.139ms	239.656op/s
scenario:load:petclinic:appsec:high_load	better [-1.595ms; -0.532ms] or [-8.189%; -2.730%]	unsure [-1.844ms; -0.182ms] or [-5.938%; -0.587%]	unstable [-16.326op/s; +38.701op/s] or [-6.861%; +16.263%]	18.410ms	30.039ms	249.156op/s	19.474ms	31.052ms	237.969op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~e73c492734, baseline=1.61.0-SNAPSHOT~a953f33c70
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.183 ms) : 1172, 1195
.   : milestone, 1183,
iast (3.231 ms) : 3187, 3276
.   : milestone, 3231,
iast_FULL (5.901 ms) : 5840, 5961
.   : milestone, 5901,
iast_GLOBAL (3.549 ms) : 3483, 3614
.   : milestone, 3549,
profiling (1.93 ms) : 1914, 1947
.   : milestone, 1930,
tracing (1.792 ms) : 1777, 1808
.   : milestone, 1792,
section candidate
no_agent (1.181 ms) : 1170, 1193
.   : milestone, 1181,
iast (3.201 ms) : 3159, 3243
.   : milestone, 3201,
iast_FULL (6.014 ms) : 5952, 6076
.   : milestone, 6014,
iast_GLOBAL (3.626 ms) : 3565, 3687
.   : milestone, 3626,
profiling (1.954 ms) : 1938, 1970
.   : milestone, 1954,
tracing (1.786 ms) : 1771, 1800
.   : milestone, 1786,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.183 ms [1.172 ms, 1.195 ms]	-
iast	3.231 ms [3.187 ms, 3.276 ms]	2.048 ms (173.1%)
iast_FULL	5.901 ms [5.84 ms, 5.961 ms]	4.717 ms (398.7%)
iast_GLOBAL	3.549 ms [3.483 ms, 3.614 ms]	2.365 ms (199.9%)
profiling	1.93 ms [1.914 ms, 1.947 ms]	747.127 µs (63.1%)
tracing	1.792 ms [1.777 ms, 1.808 ms]	609.095 µs (51.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.181 ms [1.17 ms, 1.193 ms]	-
iast	3.201 ms [3.159 ms, 3.243 ms]	2.019 ms (171.0%)
iast_FULL	6.014 ms [5.952 ms, 6.076 ms]	4.833 ms (409.1%)
iast_GLOBAL	3.626 ms [3.565 ms, 3.687 ms]	2.445 ms (207.0%)
profiling	1.954 ms [1.938 ms, 1.97 ms]	773.133 µs (65.5%)
tracing	1.786 ms [1.771 ms, 1.8 ms]	604.546 µs (51.2%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~e73c492734, baseline=1.61.0-SNAPSHOT~a953f33c70
    dateFormat X
    axisFormat %s
section baseline
no_agent (17.991 ms) : 17808, 18173
.   : milestone, 17991,
appsec (19.618 ms) : 19416, 19821
.   : milestone, 19618,
code_origins (17.868 ms) : 17693, 18042
.   : milestone, 17868,
iast (17.77 ms) : 17598, 17943
.   : milestone, 17770,
profiling (19.482 ms) : 19280, 19683
.   : milestone, 19482,
tracing (18.457 ms) : 18267, 18646
.   : milestone, 18457,
section candidate
no_agent (18.767 ms) : 18576, 18958
.   : milestone, 18767,
appsec (18.733 ms) : 18541, 18925
.   : milestone, 18733,
code_origins (17.561 ms) : 17387, 17735
.   : milestone, 17561,
iast (17.62 ms) : 17448, 17791
.   : milestone, 17620,
profiling (18.755 ms) : 18564, 18947
.   : milestone, 18755,
tracing (18.376 ms) : 18191, 18562
.   : milestone, 18376,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	17.991 ms [17.808 ms, 18.173 ms]	-
appsec	19.618 ms [19.416 ms, 19.821 ms]	1.628 ms (9.0%)
code_origins	17.868 ms [17.693 ms, 18.042 ms]	-123.102 µs (-0.7%)
iast	17.77 ms [17.598 ms, 17.943 ms]	-220.205 µs (-1.2%)
profiling	19.482 ms [19.28 ms, 19.683 ms]	1.491 ms (8.3%)
tracing	18.457 ms [18.267 ms, 18.646 ms]	466.028 µs (2.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.767 ms [18.576 ms, 18.958 ms]	-
appsec	18.733 ms [18.541 ms, 18.925 ms]	-33.851 µs (-0.2%)
code_origins	17.561 ms [17.387 ms, 17.735 ms]	-1.206 ms (-6.4%)
iast	17.62 ms [17.448 ms, 17.791 ms]	-1.147 ms (-6.1%)
profiling	18.755 ms [18.564 ms, 18.947 ms]	-11.703 µs (-0.1%)
tracing	18.376 ms [18.191 ms, 18.562 ms]	-390.422 µs (-2.1%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	brian.marks/add-ksr-tag
git_commit_date	1773845838	1773954620
git_commit_sha	`a953f33`	`e73c492`
release_version	1.61.0-SNAPSHOT~a953f33c70	1.61.0-SNAPSHOT~e73c492734

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1773956601	1773956601
ci_job_id	1523704422	1523704422
ci_pipeline_id	103615480	103615480
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-6e1rquyy 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-6e1rquyy 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~e73c492734, baseline=1.61.0-SNAPSHOT~a953f33c70
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.479 ms) : 1467, 1490
.   : milestone, 1479,
appsec (3.833 ms) : 3611, 4055
.   : milestone, 3833,
iast (2.273 ms) : 2203, 2343
.   : milestone, 2273,
iast_GLOBAL (2.315 ms) : 2245, 2385
.   : milestone, 2315,
profiling (2.125 ms) : 2067, 2182
.   : milestone, 2125,
tracing (2.083 ms) : 2029, 2137
.   : milestone, 2083,
section candidate
no_agent (1.479 ms) : 1467, 1490
.   : milestone, 1479,
appsec (3.845 ms) : 3623, 4068
.   : milestone, 3845,
iast (2.273 ms) : 2203, 2342
.   : milestone, 2273,
iast_GLOBAL (2.317 ms) : 2247, 2387
.   : milestone, 2317,
profiling (2.102 ms) : 2047, 2157
.   : milestone, 2102,
tracing (2.089 ms) : 2035, 2144
.   : milestone, 2089,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.479 ms [1.467 ms, 1.49 ms]	-
appsec	3.833 ms [3.611 ms, 4.055 ms]	2.354 ms (159.2%)
iast	2.273 ms [2.203 ms, 2.343 ms]	794.152 µs (53.7%)
iast_GLOBAL	2.315 ms [2.245 ms, 2.385 ms]	836.245 µs (56.6%)
profiling	2.125 ms [2.067 ms, 2.182 ms]	646.11 µs (43.7%)
tracing	2.083 ms [2.029 ms, 2.137 ms]	604.059 µs (40.9%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.479 ms [1.467 ms, 1.49 ms]	-
appsec	3.845 ms [3.623 ms, 4.068 ms]	2.367 ms (160.0%)
iast	2.273 ms [2.203 ms, 2.342 ms]	793.75 µs (53.7%)
iast_GLOBAL	2.317 ms [2.247 ms, 2.387 ms]	838.326 µs (56.7%)
profiling	2.102 ms [2.047 ms, 2.157 ms]	623.434 µs (42.2%)
tracing	2.089 ms [2.035 ms, 2.144 ms]	610.673 µs (41.3%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~e73c492734, baseline=1.61.0-SNAPSHOT~a953f33c70
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.501 s) : 15501000, 15501000
.   : milestone, 15501000,
appsec (14.51 s) : 14510000, 14510000
.   : milestone, 14510000,
iast (18.472 s) : 18472000, 18472000
.   : milestone, 18472000,
iast_GLOBAL (17.959 s) : 17959000, 17959000
.   : milestone, 17959000,
profiling (14.849 s) : 14849000, 14849000
.   : milestone, 14849000,
tracing (14.676 s) : 14676000, 14676000
.   : milestone, 14676000,
section candidate
no_agent (14.961 s) : 14961000, 14961000
.   : milestone, 14961000,
appsec (14.956 s) : 14956000, 14956000
.   : milestone, 14956000,
iast (18.444 s) : 18444000, 18444000
.   : milestone, 18444000,
iast_GLOBAL (17.939 s) : 17939000, 17939000
.   : milestone, 17939000,
profiling (14.883 s) : 14883000, 14883000
.   : milestone, 14883000,
tracing (14.987 s) : 14987000, 14987000
.   : milestone, 14987000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.501 s [15.501 s, 15.501 s]	-
appsec	14.51 s [14.51 s, 14.51 s]	-991.0 ms (-6.4%)
iast	18.472 s [18.472 s, 18.472 s]	2.971 s (19.2%)
iast_GLOBAL	17.959 s [17.959 s, 17.959 s]	2.458 s (15.9%)
profiling	14.849 s [14.849 s, 14.849 s]	-652.0 ms (-4.2%)
tracing	14.676 s [14.676 s, 14.676 s]	-825.0 ms (-5.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.961 s [14.961 s, 14.961 s]	-
appsec	14.956 s [14.956 s, 14.956 s]	-5.0 ms (-0.0%)
iast	18.444 s [18.444 s, 18.444 s]	3.483 s (23.3%)
iast_GLOBAL	17.939 s [17.939 s, 17.939 s]	2.978 s (19.9%)
profiling	14.883 s [14.883 s, 14.883 s]	-78.0 ms (-0.5%)
tracing	14.987 s [14.987 s, 14.987 s]	26.0 ms (0.2%)

github-actions · 2026-03-17T01:24:53Z

⚠️ New Groovy Test Files Added

The following files add Groovy tests to modules that are candidates for migration to Java / JUnit 5:

dd-trace-core/src/test/groovy/datadog/trace/core/KnuthSamplingRateTest.groovy (module: dd-trace-core)

Consider writing these tests in Java / JUnit 5 instead to help with the ongoing migration effort.

github-actions · 2026-03-17T01:24:53Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

amarziali · 2026-03-17T07:36:32Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 13b7d5e495

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java

dd-trace-core/src/main/java/datadog/trace/core/propagation/PropagationTags.java

mcculls · 2026-03-17T09:55:24Z

dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java

+  static String formatKnuthSamplingRate(double rate) {
+    // Use %.6g format: up to 6 significant digits, no trailing zeros
+    // This matches Go's strconv.FormatFloat(rate, 'g', 6, 64) and Python's f"{rate:.6g}"
+    String formatted = String.format("%.6g", rate);


This should fix the localization issue:

Suggested change

String formatted = String.format("%.6g", rate);

String formatted = String.format(Locale.ROOT, "%.6g", rate);

Note you'll also need to add an import for Locale

This is a pretty hot place to be using String.format. Profiles already show that PropagationTags is a major source of allocation. That said, I'm not going to block this PR, but I do expect that I'll end up reworking this sooner than later.

I'm wondering if we can replace that with a tiny custom formatter that rounds to 6 decimal places and trims trailing zeros? That would avoid Formatter allocations while keeping the output stable and locale-independent.

Yes, I think we'll have to. That String.format call is quite expensive in terms of allocation.
It is auto-boxing a double, then creating an array for the var-args, and then allocating a String.
I started working on a benchmark to check the overhead.

Apparently, the boxing, var-args, and String were just the tip of the iceberg. After looking into it, format ends up allocating a bunch of streams, etc.

For comparison sake, I'll see what repeatedly using a formatter does, but ultimately, I think we'll just have to roll our own lighter solution.

@dougqh just handled this in the latest commit

I've replaced String.format with manual char-array arithmetic. Benchmarked showed ~11ns vs ~320ns (30x improvement). It also eagerly caches the TagValue in updateKnuthSamplingRate() so getKnuthSamplingRateTagValue() on the header-injection hot path is just a volatile read (~1.2ns, zero allocation)

To be safe, I've also added a JMH benchmark (KnuthSamplingRateFormatBenchmark) comparing all three approaches.

dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java

…onTags - Use Locale.ROOT in String.format to prevent locale-sensitive decimal separators (e.g. comma in fr_FR) from corrupting x-datadog-tags - Move formatKnuthSamplingRate() from DDSpan to PTagsFactory so propagation encoding logic stays in the propagation layer - Change updateKnuthSamplingRate signature to accept double instead of pre-formatted String - Use indexOf('.') instead of contains(".") for minor perf improvement - Update test to exercise formatting through PropagationTags API Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

# What does this PR do? Adds `_dd.p.ksr` (Knuth Sampling Rate) as a propagated tag set when agent-based or rule-based sampling decisions are made. The tag is stored in span `meta` (string type) with up to 6 significant digits and no trailing zeros. `format_sampling_rate` now returns `Option<String>` and guards against invalid inputs (negative, >1.0, NaN, infinity), returning `None` instead of producing garbage output. # Motivation To enable consistent sampling across tracers and backend retention filters, the backend needs to know the sampling rate applied by the tracer. Without transmitting the tracer's rate via `_dd.p.ksr`, backend resampling cannot correctly compute effective rates in multi-stage sampling scenarios. See RFC: "Transmit Knuth sampling rate to backend" # Additional Notes Key files changed: - `datadog-opentelemetry/src/core/constants.rs` — Added `SAMPLING_KNUTH_RATE_TAG_KEY` constant - `datadog-opentelemetry/src/sampling/datadog_sampler.rs` — Added `format_sampling_rate()` helper (returns `Option<String>`, defensive against invalid rates) and set ksr in agent/rule sampling paths - Updated 2 snapshot JSON files Related PRs across tracers: - Java: DataDog/dd-trace-java#10802 - .NET: DataDog/dd-trace-dotnet#8287 - Ruby: DataDog/dd-trace-rb#5436 - Node.js: DataDog/dd-trace-js#7741 - PHP: DataDog/dd-trace-php#3701 - C++: DataDog/dd-trace-cpp#288 - System tests: DataDog/system-tests#6466 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

...ent/instrumentation/opentelemetry/opentelemetry-0.3/src/test/groovy/OpenTelemetryTest.groovy

dd-trace-core/src/main/java/datadog/trace/core/propagation/ptags/PTagsFactory.java

…r as primitive double Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

dougqh

Unfortunately, I think we need to replace the String.format call before this can be merged.
As is, this change increases the memory consumption in a span creation stress test by 2x.

In my local test, 16 threads x 10_000_000 traces x 2 spans, current head allocates 241 GiB. After this change, the same test is allocating 584 GiB. That corresponds to a 15 sec increase in execution time from ~65 secs to ~80 secs. Presumably mostly from GC, but I haven't verified that yet.

### What does this PR do? Fixes `_dd.p.ksr` (Knuth Sampling Rate) to only be set on spans when the agent has provided sampling rates via `readRatesJSON()`. Previously, ksr was unconditionally set in `prioritySampler.apply()`, including when the rate was the initial client-side default (1.0) before any agent response arrived. Also refactors `prioritySampler` to consolidate lock acquisitions: extracts `getRateLocked()` so `apply()` acquires `ps.mu.RLock` only once to read both the rate and `agentRatesLoaded`. ### Motivation Cross-language consistency: Python, Java, PHP, and other tracers only set ksr when actual agent rates or sampling rules are applied, not for the default fallback. This aligns Go with that behavior. See RFC: "Transmit Knuth sampling rate to backend" ### Additional Notes - Added `agentRatesLoaded` bool field to `prioritySampler`, set to `true` in `readRatesJSON()` - `apply()` now gates ksr behind `agentRatesLoaded` check - Extracted `getRateLocked()` to avoid double lock acquisition in `apply()` - Rule-based sampling path (`applyTraceRuleSampling` in span.go) unchanged — correctly always sets ksr - Tests added: `ksr-not-set-without-agent-rates` and `ksr-set-after-agent-rates-received` Related PRs across tracers: - Java: DataDog/dd-trace-java#10802 - .NET: DataDog/dd-trace-dotnet#8287 - Ruby: DataDog/dd-trace-rb#5436 - Node.js: DataDog/dd-trace-js#7741 - PHP: DataDog/dd-trace-php#3701 - Rust: DataDog/dd-trace-rs#180 - C++: DataDog/dd-trace-cpp#288 - System tests: DataDog/system-tests#6466 ### Reviewer's Checklist - [x] Changed code has unit tests for its functionality at or near 100% coverage. - [x] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag. - [ ] There is a benchmark for any new code, or changes to existing code. - [x] If this interacts with the agent in a new way, a system test has been added. - [x] New code is free of linting errors. You can check this by running `make lint` locally. - [x] New code doesn't break existing tests. You can check this by running `make test` locally. - [ ] Add an appropriate team label so this PR gets put in the right place for the release notes. - [ ] All generated files are up to date. You can check this by running `make generate` locally. - [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running `make fix-modules` locally. Unsure? Have a question? Request a review! 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Dario Castañé <dario.castane@datadoghq.com> Co-authored-by: Mikayla Toffler <46911781+mtoffl01@users.noreply.github.com>

…ay arithmetic Replace String.format(Locale.ROOT, "%.6g", rate) with manual char-array formatting that avoids Formatter/stream/boxing allocations. Benchmarks show ~30x improvement (320ns -> 11ns). Additionally, cache the TagValue in updateKnuthSamplingRate() so that getKnuthSamplingRateTagValue() is a simple volatile read (~1.2ns) instead of re-formatting and re-looking up the TagValue on every header injection. Add JMH benchmark (KnuthSamplingRateFormatBenchmark) comparing all three approaches and expand test coverage to all magnitude buckets. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Implement scientific notation (rates < 1e-4) with manual char-array formatting, removing the last String.format dependency. The method now uses zero Formatter/Locale allocations for all rate values. Add test coverage for scientific notation: 1e-05, 5e-05, 1.23457e-05, 1e-07, 5.5e-10. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add _dd.p.ksr propagated tag for Knuth sampling rate

684a837

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

bm1549 and others added 2 commits March 10, 2026 22:32

bm1549 added tag: ai generated Largely based on code generated by an AI or LLM comp: core Tracer core labels Mar 11, 2026

bm1549 mentioned this pull request Mar 11, 2026

fix(tracer): only set _dd.p.ksr after agent rates are received DataDog/dd-trace-go#4523

Merged

9 tasks

bm1549 and others added 5 commits March 10, 2026 22:47

fix: add _dd.p.ksr to expected meta maps in DDAgentApiTest

7670947

The ksr implementation now adds _dd.p.ksr tag to spans with agent sampling rate, so the msgpack serialization test expectations need to include it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

bm1549 marked this pull request as ready for review March 17, 2026 01:23

bm1549 requested review from a team as code owners March 17, 2026 01:23

bm1549 requested review from amarziali and mtoffl01 and removed request for a team March 17, 2026 01:23

bm1549 added comp: core Tracer core and removed comp: core Tracer core labels Mar 17, 2026

chatgpt-codex-connector bot reviewed Mar 17, 2026

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java Outdated Show resolved Hide resolved

amarziali reviewed Mar 17, 2026

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/propagation/PropagationTags.java Outdated Show resolved Hide resolved

mcculls reviewed Mar 17, 2026

View reviewed changes

dougqh reviewed Mar 17, 2026

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java Outdated Show resolved Hide resolved

bm1549 added the type: enhancement Enhancements and improvements label Mar 17, 2026

bm1549 and others added 3 commits March 17, 2026 14:38

ci: retry - transient JBoss 503 failure in smoke tests

f1e92a0

Merge branch 'master' into brian.marks/add-ksr-tag

81f00be

Merge branch 'master' into brian.marks/add-ksr-tag

e0ce92a

bm1549 requested review from amarziali, dougqh and mcculls March 19, 2026 15:14

dougqh reviewed Mar 19, 2026

View reviewed changes

...ent/instrumentation/opentelemetry/opentelemetry-0.3/src/test/groovy/OpenTelemetryTest.groovy Outdated Show resolved Hide resolved

dougqh reviewed Mar 19, 2026

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/propagation/ptags/PTagsFactory.java Show resolved Hide resolved

Address PR feedback: rename datadogTags to expectedDataTags, store ks…

97d7e52

…r as primitive double Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

dougqh requested changes Mar 19, 2026

View reviewed changes

bm1549 and others added 2 commits March 19, 2026 17:03

	String formatted = String.format("%.6g", rate);
	String formatted = String.format(Locale.ROOT, "%.6g", rate);

Conversation

bm1549 commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

github-actions bot commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amarziali commented Mar 17, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

mcculls Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

dougqh Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amarziali Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

dougqh Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dougqh Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bm1549 Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dougqh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bm1549 commented Mar 11, 2026 •

edited

Loading

pr-commenter bot commented Mar 11, 2026 •

edited

Loading

github-actions bot commented Mar 17, 2026 •

edited

Loading

github-actions bot commented Mar 17, 2026 •

edited

Loading

dougqh Mar 17, 2026 •

edited

Loading

dougqh Mar 18, 2026 •

edited

Loading

dougqh Mar 19, 2026 •

edited

Loading

bm1549 Mar 19, 2026 •

edited

Loading

dougqh left a comment •

edited

Loading