Add Azure Durable Functions tracing support by jcstorms1 · Pull Request #12618 · DataDog/dd-trace-java

jcstorms1 · 2026-09-23T17:12:29Z

What Does This Do

Adds tracing support for Azure Durable Functions running on the Java worker:

Instruments durable orchestrator, activity, and entity invocations through FunctionExecutionMiddleware.
Continues the W3C trace context propagated by the Azure Functions host.
Preserves Datadog sampling decisions when Azure clears the W3C sampled flag.
Suppresses successful orchestrator replays while retaining initial executions and replays caused by activity or sub-orchestration failures.
Marks application failures as errors without treating Durable SDK replay-control exceptions as failures.
Adds Durable function name, trigger type, serverless span type, and component metadata.
Adds Java/JUnit 5 coverage for both span attribute schema versions and the latest supported dependency set.

Motivation

The existing Azure Functions instrumentation covers HTTP entry points, but Java Durable orchestrators and activities execute through worker middleware and were not represented in the distributed trace. This left the asynchronous Durable workflow invisible after the starter function scheduled an instance.

This instrumentation connects the Java Durable workflow to the originating trace and avoids emitting a duplicate orchestrator span for every successful replay.

Validation

./gradlew :dd-java-agent:instrumentation:azure-functions-worker-2.7:spotlessCheck
./gradlew :dd-java-agent:instrumentation:azure-functions-worker-2.7:test
./gradlew :dd-java-agent:instrumentation:azure-functions-worker-2.7:forkedTest
./gradlew :dd-java-agent:instrumentation:azure-functions-worker-2.7:latestDepTest
The Java/JUnit 5 suite runs 18 cases in each test task, covering trigger spans, replay suppression and retention, sampling propagation, replay-control exceptions, and application failures.
./gradlew :dd-trace-core:test --tests datadog.trace.core.propagation.ExtractedContextTest
./gradlew :dd-java-agent:shadowJar :dd-java-agent:verifyAgentJarContents :dd-java-agent:verifyAgentJarIntegrations
Deployed the rebuilt agent to an Azure Java Durable Functions POC using Durable tracing V2 and the Azure Storage provider.
Verified a successful orchestration contains one orchestrator span and both activity spans.
Verified a failed activity is marked as an error and retains the failure replay of the orchestrator.

No error

With error

Additional Notes

The instrumentation targets Azure Functions Java worker 2.7+ and was validated with durabletask-azure-functions 1.0.1.
The instrumentation tests are implemented in Java/JUnit 5 using the repository's Groovy-to-Java migration conventions.
No new user-facing configuration flags are introduced.

Contributor Checklist

Use a user-facing infinitive title.
Assign type: feature and inst: azure-functions labels.
Add the new instrumentation directory to CODEOWNERS.
Add focused Java/JUnit 5 instrumentation and propagation tests.
No public configuration documentation update is required because this adds no configuration flags.

datadog-datadog-us1-prod · 2026-09-23T17:27:12Z

🎯 Code Coverage (details)
• Patch Coverage: 71.43%
• Overall Coverage: 59.27% (+0.14%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: ab2c863 | Docs | Give us feedback!}

dd-octo-sts · 2026-09-23T17:37:59Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.03 s	13.92 s	[-0.0%; +1.5%] (no difference)
startup:insecure-bank:tracing:Agent	12.96 s	13.06 s	[-1.8%; +0.1%] (no difference)
startup:petclinic:appsec:Agent	17.02 s	17.01 s	[-0.9%; +1.0%] (no difference)
startup:petclinic:iast:Agent	16.97 s	17.10 s	[-1.4%; -0.1%] (maybe better)
startup:petclinic:profiling:Agent	16.57 s	16.88 s	[-2.8%; -0.8%] (maybe better)
startup:petclinic:sca:Agent	16.95 s	16.89 s	[-0.8%; +1.5%] (no difference)
startup:petclinic:tracing:Agent	15.69 s	16.24 s	[-7.5%; +0.8%] (no difference)

Commit: ab2c863f · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

jcstorms1 · 2026-09-23T18:39:12Z

@codex review

chatgpt-codex-connector · 2026-09-23T18:39:30Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-23T20:07:08.148907Z	`12dec94`	Draft marked ready
🔒 Security Review	✅ Completed 2026-09-23T20:07:30.900452Z	`12dec94`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector · 2026-09-23T18:48:35Z

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 376c24d851

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 376c24d851

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

datadog-datadog-us1-prod

An exception with a cyclic cause chain can keep a worker thread in an endless loop. Error spans created after a suppressed replay also omit the time spent in the invocation.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 12dec94 · @DataDog review to ask questions}

AlexeyKuznetsov-DD

Left several comments. My main concern that core classes changed, we need to double check on that.

AlexeyKuznetsov-DD · 2026-09-23T20:13:02Z

+                takesArgument(
+                    0,
+                    named(
+                        "com.microsoft.azure.functions.internal.spi.middleware.MiddlewareContext")))


nit: "com.microsoft.azure.functions.internal.spi.middleware.MiddlewareContext" looks very long (and other similar strings), maybe make sense to create short consts MIDDLEWARE_CONTEXT ?

Not sure it is worth since they only appear once. Also, the repository's instrumentation guidelines recommend keeping values inline in one-shot methods such as methodAdvice()

AlexeyKuznetsov-DD · 2026-09-23T20:31:11Z

  private final Map<String, String> baggage;
  private Baggage w3cBaggage;
-  private final int samplingPriority;
+  private int samplingPriority;


This is a change in a core class, betted to discuss with dd-trace-java such changes...
@dougqh WDYT?

For context, I need to change the sampling priority here because the host runtime is .NET. We end up with sampling decisions and context that points to traces we have no access to. If we don't override the sampling we end up dropping all the spans unless we enable Opentelemetry in the Azure host runtime config. Telling customers to enable that is less than ideal since we want to have a native instrumentation. Open to other ideas, but we had to do this in all runtimes so far.

... also check with @MilanGarnier because this may overlap with the consistent probability sampling RFC. Certainly this looks like something that needs careful consideration wrt. how it might interact with other efforts.

jcstorms1 added 2 commits September 23, 2026 11:35

Add Azure Durable Functions worker instrumentation

1ecaa96

Add Azure Functions worker code owner

f6cb9cf

jcstorms1 added type: feature Enhancements and improvements inst: azure-functions Azure Functions instrumentation labels Sep 23, 2026

Convert Azure Functions worker tests to JUnit

376c24d

chatgpt-codex-connector Bot reviewed Sep 23, 2026

View reviewed changes

Fix Durable Functions replay edge cases

12dec94

jcstorms1 marked this pull request as ready for review September 23, 2026 20:00

jcstorms1 requested review from a team as code owners September 23, 2026 20:00

jcstorms1 requested review from AlexeyKuznetsov-DD, ValentinZakharov and mcculls and removed request for a team September 23, 2026 20:00

datadog-datadog-us1-prod Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread ...rc/main/java/datadog/trace/instrumentation/azure/functions/worker/DurableFunctionsUtils.java Outdated

Comment thread ...atadog/trace/instrumentation/azure/functions/worker/AzureFunctionsWorkerInstrumentation.java Outdated

AlexeyKuznetsov-DD reviewed Sep 23, 2026

View reviewed changes

jcstorms1 added 3 commits September 23, 2026 16:36

Fix Azure Functions replay error handling

8cd74a3

Address Azure Functions review feedback

e159a3d

Sort Azure Functions helper classes

ab2c863

Conversation

jcstorms1 commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Validation

No error

With error

Additional Notes

Contributor Checklist

Uh oh!

datadog-datadog-us1-prod Bot commented Sep 23, 2026 • edited by datadog-prod-us1-6 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

jcstorms1 commented Sep 23, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot commented Sep 23, 2026

🛡️ Codex Security Review · Automatically triggered

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

datadog-datadog-us1-prod Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AlexeyKuznetsov-DD left a comment

Choose a reason for hiding this comment

Uh oh!

AlexeyKuznetsov-DD Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

jcstorms1 Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexeyKuznetsov-DD Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

jcstorms1 Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mcculls Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jcstorms1 commented Sep 23, 2026 •

edited

Loading

datadog-datadog-us1-prod Bot commented Sep 23, 2026 •

edited by datadog-prod-us1-6 Bot

Loading

dd-octo-sts Bot commented Sep 23, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 23, 2026 •

edited

Loading

jcstorms1 Sep 23, 2026 •

edited

Loading

jcstorms1 Sep 23, 2026 •

edited

Loading