Skip to content

Add agentless Feature Flagging configuration source#11892

Draft
leoromanovsky wants to merge 4 commits into
masterfrom
leo.romanovsky/ffl-2693-java-agentless-configuration-source
Draft

Add agentless Feature Flagging configuration source#11892
leoromanovsky wants to merge 4 commits into
masterfrom
leo.romanovsky/ffl-2693-java-agentless-configuration-source

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

NOTE TO REVIEWERS

This PR intentionally keeps the complete ~1K LOC feature in one PR so the final architecture remains visible. The four commits are deliberately layered like a stacked PR, and each commit pairs its production change with the tests for that layer.

How to Review

  1. Review the commits in the order listed below; each link opens only that commit's diff.
  2. Treat each commit as an independently buildable review unit.
  3. After the fourth commit, use the full Files changed view for a final cross-layer integration pass.
  4. Please leave comments on the commit where the behavior is introduced when possible.

Commit Guide

LOC is rename-aware per commit against its parent.

Commit Review focus LOC (+/-)
07f94a5488 Define the canonical configuration-source contract, defaults, metadata, generic source lifecycle, and explicit Remote Configuration compatibility pins. +115 / -3
ba90dea14e Add first-party Datadog-managed agentless UFC polling, including authenticated HTTP delivery, ETag/retry/last-known-good behavior, lifecycle safety, and focused tests. +744 / -0
d784c8b5d2 Extend the same agentless source to custom HTTP endpoints, including bare-host versus exact-path behavior and tests. +57 / -1
82bb199145 Select agentless, Remote Configuration, or reserved offline mode at runtime and verify system lifecycle behavior. +118 / -6

Motivation

Java Feature Flagging needs a tracer-side configuration-source split so SDKs can fetch UFC from an agentless HTTP backend while preserving the existing Agent Remote Configuration path.

Changes

  • Adds DD_FEATURE_FLAGS_CONFIGURATION_SOURCE with agentless, remote_config, and reserved offline; default is agentless.
  • Adds agentless endpoint selection through DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL.
  • Adds agentless polling controls: DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDS and DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS.
  • Adds AgentlessConfigurationSource, an HTTP UFC poller used for both the default Datadog-managed endpoint and a configured custom HTTP endpoint.
  • Keeps remote_config on the existing RemoteConfigServiceImpl / Agent RC path.
  • Handles DD-API-KEY, ETag/304, timeout/429/5xx retry, malformed UFC rejection, last-known-good preservation, and no overlapping polls.
  • Removes the proposed DD_FEATURE_FLAGS_ENABLED and extra-header env from this PR.

Decisions

No new provider kill switch. This keeps the existing provider bootstrap behavior through DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED; changing enablement is out of scope for this PR.

Custom HTTP delivery remains under agentless. The source mode describes direct HTTP UFC delivery without the Datadog Agent. With no base URL, the SDK derives the first-party Datadog endpoint as https://api.<site>/api/v2/feature-flagging/config/server-distribution?dd_env=<env>. Setting DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL chooses a system-test, dogfood, or operator-managed HTTP endpoint without introducing a fourth source mode. A bare host uses the standard server-distribution path; a URL with a path is used as the exact UFC endpoint.

Remote Configuration remains a distinct mode. It is Agent-mediated and uses the RC protocol's security, signing, targeting, capability, and subscription lifecycle. Those semantics are materially different from direct HTTP polling, so remote_config remains a peer source mode rather than another agentless endpoint choice.

No offline factory yet. offline remains a reserved configuration-source value. This PR does not expose a startup-bytes API or offline factory because that API still needs design work.

Source selection stays SDK-local. DD_FEATURE_FLAGS_CONFIGURATION_SOURCE decides which local source implementation starts. It is not sent to the backend.

Initialization Modes

Solid connectors are implemented in this PR. The dashed offline connector previews startup-provided UFC bytes.

flowchart TD
    Gate{DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED}
    Disabled[Feature Flagging subsystem not started]
    System[FeatureFlaggingSystem.start]
    Select{DD_FEATURE_FLAGS_CONFIGURATION_SOURCE}
    Exposure[ExposureWriter starts for every enabled mode]

    subgraph AgentlessMode[agentless mode - current default]
        AgentlessSource[AgentlessConfigurationSource]
        OperationalDefaults[SDK-owned operation<br/>30s poll default; 2s request timeout default<br/>ETag; retry; last-known-good; no overlap]
        EndpointChoice{agentless base URL set?}
        DatadogManaged[Datadog-managed agentless<br/>first-party; CDN-backed]
        CustomHttp[Custom HTTP endpoint<br/>system tests; dogfood; operator backend]

        AgentlessSource --- OperationalDefaults
        AgentlessSource -- polls --> EndpointChoice
        EndpointChoice -- no --> DatadogManaged
        EndpointChoice -- yes --> CustomHttp
    end

    RemoteConfig[remote_config<br/>explicit opt-in; Agent-mediated]
    RemoteConfigSource[RemoteConfigServiceImpl]
    AgentRc[Datadog Agent Remote Configuration<br/>RC security; signing; targeting; subscriptions]

    OfflineReserved[offline today<br/>reserved; no configuration service]
    OfflineSource[Later: OfflineConfigurationSource<br/>customer UFC JSON bytes at startup; no network]

    Ufc[Shared UFC deserialize and evaluate pipeline]
    Gateway[FeatureFlaggingGateway]
    Provider[OpenFeature provider]

    Gate -- false --> Disabled
    Gate -- true --> System
    System --> Select
    System --> Exposure

    Select -- unset or agentless --> AgentlessSource
    DatadogManaged --> Ufc
    CustomHttp --> Ufc

    Select -- remote_config --> RemoteConfig
    RemoteConfig --> RemoteConfigSource
    RemoteConfigSource -- subscribes --> AgentRc
    RemoteConfigSource --> Ufc

    Select -- offline today --> OfflineReserved
    OfflineReserved -. later .-> OfflineSource
    OfflineSource -. startup UFC bytes .-> Ufc

    Ufc --> Gateway
    Gateway --> Provider
Loading

Customer Usage Example

Default Datadog-managed agentless delivery:

export DD_API_KEY=<datadog-api-key>
export DD_SITE=datadoghq.com
export DD_ENV=prod

# Optional because agentless is the default configuration source.
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=agentless
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDS=30
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS=2

java -javaagent:/path/to/dd-java-agent.jar -jar app.jar

Custom HTTP delivery keeps agentless mode and changes only the endpoint:

export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=agentless
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL=https://flags.example.com/ufc

java -javaagent:/path/to/dd-java-agent.jar -jar app.jar

Explicit Agent Remote Configuration delivery still uses the existing Agent path:

export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=remote_config
export DD_REMOTE_CONFIGURATION_ENABLED=true

java -javaagent:/path/to/dd-java-agent.jar -jar app.jar

Verification

./gradlew :products:feature-flagging:feature-flagging-agent:test :products:feature-flagging:feature-flagging-lib:test --no-daemon
result: BUILD SUCCESSFUL

./gradlew :products:feature-flagging:feature-flagging-agent:jacocoTestCoverageVerification :products:feature-flagging:feature-flagging-lib:jacocoTestCoverageVerification :products:feature-flagging:feature-flagging-agent:spotlessCheck :products:feature-flagging:feature-flagging-lib:spotlessCheck --no-daemon
result: BUILD SUCCESSFUL

./gradlew :dd-smoke-tests:openfeature:cleanTest :dd-smoke-tests:openfeature:test --tests datadog.smoketest.springboot.OpenFeatureProviderSmokeTest --no-daemon
result: BUILD SUCCESSFUL

git diff --check
result: clean

Note on CI failure (known)

Will be resolved after merging DataDog/system-tests#7297

System Test Evidence

The Java manifest activation is exercised locally before it is merged. I built
all three Java artifacts from the pre-restack head (1858f3256e):
dd-java-agent, dd-trace-api, and dd-openfeature. The narrative restack head (82bb199145) has the identical Git tree (9b945ade3d0e), so the tested code is byte-for-byte unchanged. I then ran the
manifest-enabled agentless scenario from the companion system-tests draft.

TEST_LIBRARY=java ./run.sh +v \
  FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS \
  tests/ffe/test_agentless_configuration.py
Scenario: FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS
Library: java@1.65.0-SNAPSHOT+1858f3256e
Weblog variant: spring-boot
collected 1 item
tests/ffe/test_agentless_configuration.py .                        [100%]
1 passed in 36.04s

This is the same Java manifest row proposed by the companion draft, enabled
locally against the byte-identical pre-restack tree rather than relying on the currently published Java
development artifact.

Next Steps

  • Review the draft Java manifest activation:
    DataDog/system-tests#7300.
  • Keep that PR draft until this Java behavior is available to system-tests CI.
  • After this PR lands, rerun the manifest activation in Java development CI and
    merge the activation only when that gate is green.

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
Patch Coverage: 85.81%
Overall Coverage: 56.97% (-0.08%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 82bb199 | Docs | Datadog PR Page | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.95 s 13.97 s [-1.0%; +0.7%] (no difference)
startup:insecure-bank:tracing:Agent 12.98 s 13.03 s [-1.4%; +0.6%] (no difference)
startup:petclinic:appsec:Agent 16.13 s 16.71 s [-7.7%; +0.7%] (no difference)
startup:petclinic:iast:Agent 16.38 s 16.90 s [-7.4%; +1.3%] (no difference)
startup:petclinic:profiling:Agent 16.67 s 16.72 s [-1.3%; +0.6%] (no difference)
startup:petclinic:sca:Agent 16.90 s 16.73 s [-0.0%; +2.0%] (no difference)
startup:petclinic:tracing:Agent 16.05 s 16.20 s [-1.8%; -0.1%] (maybe better)

Commit: 82bb1991 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Comment thread dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java Outdated
Comment thread metadata/supported-configurations.json Outdated
Comment thread internal-api/src/main/java/datadog/trace/api/Config.java Outdated
@leoromanovsky leoromanovsky force-pushed the leo.romanovsky/ffl-2693-java-agentless-configuration-source branch from 1858f32 to 82bb199 Compare July 11, 2026 04:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant