Skip to content

Reuse smoke test results across commits and checkouts - #13294

Draft
bric3 wants to merge 4 commits into
masterfrom
bdu/cache-smoke-test-results
Draft

bric3 wants to merge 4 commits into
masterfrom
bdu/cache-smoke-test-results

Conversation

@bric3

@bric3 bric3 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Allows smoke tests to reuse matching results across fresh checkouts and commits that change only agent commit metadata. Tests continue to launch the exact packaged agent JAR.

Motivation

Commit-stamped version resources and filesystem-dependent ordering of JMX metric configuration names invalidate smoke-test results when agent code is unchanged. Oversized application-build entries also miss the shared cache, forcing fixtures to rebuild. These changes let smoke tests benefit from the shared build cache introduced in #12604.

Additional Notes

Follows #12604, now merged into master.

Two CI runs of the same commit produced different normalised agent fingerprints. The only differing packaged resource was JMXFetch's metricconfigs.txt: Files.find supplied filenames in filesystem traversal order. Sorting those names before writing the list makes the agent reproducible while keeping this resource in the test fingerprint.

The S3 plugin skipped the 56.9 MB Vert.x application-build entry because its default limit is 50 MiB. Rebuilding that fixture changed its JAR bytes and invalidated its smoke tests. Quarkus native entries also exceeded the limit: 137 MB on GraalVM 21 and 141 MB on GraalVM 25. The WebSphere shard wrote 212 MiB across two cache entries. The shared cache now accepts entries up to 256 MiB. This allows larger transfers and stored entries while retaining a size bound.

Unused forwarded JAR declarations are removed in a separate commit: eleven API declarations across Vert.x, Spring and WildFly, and Kafka's application IAST helper. Their unused nested dependencies and lockfile entries are removed too. The WildFly WAR's published API dependency and the smoke-test harness dependencies remain. Quarkus uses GraalVM's compatibility CPU target so cached executables can run on other CI hosts of the same architecture.

The S3 prefix groups OS and os.arch; it does not distinguish CPU feature levels such as x86-64-v3. The builder identity also records the image, OS and architecture. The compatibility target keeps generated native instructions suitable for reuse across CPU models within that group.

With the default snapshot versioning, project.version remains stable across commits, for example 1.69.0-SNAPSHOT. The version-file generator appends the commit SHA separately as <version>~<SHA>. A new commit with the same normalised agent contents and other test inputs can therefore reuse the cached result.

The explicit tracerVersion input preserves invalidation when the project version changes, for example from 1.69.0-SNAPSHOT to 1.70.0-SNAPSHOT. This matters because runtime-classpath normalisation excludes complete *.version resources. It does not change the JAR passed to the application. Gradle input normalisation.

Intended behaviour, assuming the other test inputs match:

Change Effect on smoke-test result reuse
Commit suffix inside a generated version file Fingerprint unchanged
Project version Fingerprint changes
Agent class or .classdata contents Fingerprint changes
Manifest, index or other tracked resource Fingerprint changes
ZIP entry timestamps or order Fingerprint unchanged
Filesystem traversal order of JMX configurations Packaged configuration list stays byte-identical
Presence of dd-java-agent.version Fingerprint changes through the required-resource input
Other excluded .version resource paths Fingerprint unchanged

The JAR input objects expose a neededResourcePaths input alongside the normalised JAR and base version. It contains the sorted names of the required resources present in the JAR, currently only dd-java-agent.version.

Adding or removing that required resource invalidates results. The Gradle CI Visibility smoke tests exercise CiVisibilityGradleListener, which reads dd-java-agent.version during class initialisation and cannot initialise without it. Changing only its stamp leaves the resource names unchanged. Other .version resources remain excluded by normalisation. Packaging verification stays in its existing flow.

projectJar uses the consuming project's runtime-classpath normalisation for JVM and native fixture dependencies, including non-default artifacts such as the agent's shadow configuration. It tracks the producing project's version separately. Cached application JARs and native executables can retain earlier embedded agent and API commit stamps. Test tasks still fingerprint the complete application artifact; reusing its cached build keeps that artifact stable.

Both native builds receive neededResourcePaths through projectJar, so adding or removing the required agent version resource also invalidates the executable.

The Quarkus and Armeria JVM applications and both native applications now check project.hasProperty('apiJar') inside their dependency blocks. The unqualified lookup checked the dependency handler and selected the published fallback despite the forwarded property. These consumers now compile against the declared project API JAR.

Change to a forwarded fixture input Effect on application-build reuse
Excluded dependency commit suffix or ZIP metadata Fingerprint unchanged
Producing project's base version Fingerprint changes
Class, .classdata or other tracked resource Fingerprint changes
GraalVM release or CI builder identity Native-build fingerprint changes
Presence of dd-java-agent.version Native-build fingerprint changes

Agent code changes still invalidate the full-agent smoke tests and native application builds. Commit-stamp changes alone can reuse matching JVM and native fixtures. Tests on master retain their existing forced-execution policy.

Approaches considered
Approach What it enables Main tradeoff
Normalise the agent input Reuse results while continuing to launch the exact packaged JAR Requires explicit handling of excluded metadata
Produce a stable smoke-test JAR Reuse results with deterministic commit metadata Most smoke tests would exercise a slightly different artifact
Separate code packaging from stamp Reuse intermediate agent packaging before adding commit data Larger build change; test reuse still needs metadata handling
Improve fixture caching Reuse application builds when agent changes require tests Helps preparation; application startup and test execution still run

Normalising the agent input reuses Gradle's existing runtime-classpath support and preserves the packaged agent used by smoke tests. The explicit base-version and required-resource inputs preserve invalidation for version changes and the presence of resources needed by smoke tests. Commit stamps remain excluded by design. JVM and native application dependencies use the same normalisation with explicit project versions. The cache-entry bound and unused project JAR removals address observed fixture misses. Producing a separate smoke-test agent JAR and separating agent packaging from the commit stamp remain outside this PR.

Cache checks

The real JMX configuration copy task was exercised with the same configurations under opposite traversal orders. It produced different list bytes before the fix and identical list and configuration bytes afterwards.

Before the forwarded-JAR changes, the second Java 17 shard run restored all seven executable test tasks. The second GraalVM 21 run restored both native application builds and both test tasks; its Gradle run fell from 4m23s to 1m52s. The WebSphere rerun also restored its test results.

That baseline restored 359 of 429 executable test tasks across the regular smoke jobs. The 70 executions comprised 23 initial runs with the changed Vert.x fixture inputs and 47 runs of JVM fixtures that bundle stamped API or OpenFeature JARs. These are Gradle task executions across JVMs, excluding skipped tasks and tasks without sources. The normalisation and unused-dependency cleanup address those remaining JVM fixture misses.

Twenty TestKit cases check default and shadow JAR cache outcomes, including .classdata, source-project version changes, required-resource presence, excluded version resources, ZIP entry order and configuration-cache reuse. The restored fixture assertion also checks the retained dependency commit stamp and required resource.

Before making normalisation the default for native inputs, the first-pass matrix passed all 48 regular smoke jobs, all three native jobs and Semeru. Regular jobs restored 421 of 429 executable test tasks. The eight executions were Quarkus and Armeria, whose dependency lookup now selects the project API. Native jobs rebuilt their images with that commit's agent and API JARs.

Eleven individual reruns restored 95 of 95 executable test tasks, including the Quarkus, Armeria and WildFly shards. The GraalVM 21 rerun also restored both native application builds.

After making normalisation the default for JVM and native inputs, all four selected jobs passed twice at e2ff46062e, before removing verification-task dependencies, in pipeline 151233522. The second runs restored 31 of 31 executable smoke test tasks: 13 in the Quarkus shard, nine in the Armeria shard, seven in the OpenFeature shard and two in the GraalVM 21 job. Both native application builds also reported FROM-CACHE.

The project JAR inputs are covered by 20 cache scenarios and a provider-backed native-input case. SmokeAgentArguments lives in the shared smoke-test build script; its managed properties and JVM arguments were checked through Gradle. JVM and native smoke task graphs exclude verifyAgentJarContents.

The build-cache guide explains input annotations, normalisation, excluded metadata and cache-debug log comparisons.

Contributor Checklist

@bric3 bric3 added comp: tooling Build & Tooling type: feature Enhancements and improvements tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes labels Nov 6, 2026
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 59.98% (+0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f7be66e | Docs | Give us feedback!

Base automatically changed from bdu/use-s3-gradle-build-cache to master November 6, 2026 18:46
@dd-octo-sts

dd-octo-sts Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.00 s 13.89 s [+0.1%; +1.4%] (maybe worse)
startup:insecure-bank:tracing:Agent 12.91 s 13.02 s [-1.8%; +0.1%] (no difference)
startup:petclinic:appsec:Agent 17.17 s 17.08 s [-0.3%; +1.3%] (no difference)
startup:petclinic:iast:Agent 17.05 s 17.18 s [-1.4%; -0.1%] (maybe better)
startup:petclinic:profiling:Agent 16.72 s 16.96 s [-2.7%; -0.2%] (maybe better)
startup:petclinic:sca:Agent 17.24 s 17.03 s [+0.3%; +2.1%] (maybe worse)
startup:petclinic:tracing:Agent 16.28 s 16.27 s [-0.8%; +0.9%] (no difference)

Commit: f7be66ee · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@bric3
bric3 force-pushed the bdu/cache-smoke-test-results branch 2 times, most recently from a5fd6b3 to 0bf8b97 Compare November 6, 2026 20:28
@bric3

bric3 commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-06T21:10:33.977919Z 0bf8b97 Manual request
🔒 Security Review ✅ Completed 2026-11-06T21:11:25.866888Z 0bf8b97 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 0bf8b977c4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 0bf8b977c4

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@bric3
bric3 force-pushed the bdu/cache-smoke-test-results branch 3 times, most recently from 20c7218 to 351813d Compare November 7, 2026 01:26
@bric3 bric3 changed the title Reuse smoke test results across commit metadata changes Reuse smoke test results across commits and checkouts Nov 7, 2026
@bric3
bric3 force-pushed the bdu/cache-smoke-test-results branch 10 times, most recently from 00987ee to 29a660b Compare November 7, 2026 23:25
bric3 added 2 commits November 8, 2026 00:33
Commit stamps and filesystem traversal order change the packaged agent
even when its code and configuration are unchanged, invalidating smoke
test results on fresh checkouts.

Use runtime classpath normalisation for the agent input and track the
project version explicitly. Keep classes, .classdata, manifests and
indexes in the fingerprint.

Track the presence of dd-java-agent.version separately because Gradle
CI Visibility needs it to initialise. Keep the argument provider in the
shared smoke-test build script and ignore other generated version
resources without running packaging verification.

Sort JMX metric configuration names before writing the packaged list so
fresh checkouts produce identical resource bytes.

Allow S3 cache entries up to 256 MiB so smoke application builds larger
than the plugin's 50 MiB default can be stored.
Use one projectJar API with classpath fingerprints for JVM and native
application dependencies, including non-default artifact configurations.
Ignore ZIP metadata and the existing version-resource exclusions while
tracking the producing project's version separately.

Cached JVM applications and native executables may retain earlier
embedded agent and API commit stamps when only excluded metadata changes.

Track needed resource names separately so adding or removing the agent
version resource also invalidates native builds. Keep packaging
verification in its existing flow.

Qualify API-property lookups inside dependency blocks so the JVM
consumers select the forwarded project JAR instead of a fallback.

Cover cache restoration, configuration-cache reuse, project versions,
class and classdata changes, required resources and ZIP entry order for
default and shadow JARs. Re-snapshot externally rewritten test JARs.

Document input annotations, normalisation and cache fingerprint
comparisons in the build-cache guide.

Explain classpath normalisation, base versions and required resource
presence beside the nested JAR inputs.
@bric3
bric3 force-pushed the bdu/cache-smoke-test-results branch from 9529267 to be18123 Compare November 7, 2026 23:35
bric3 added 2 commits November 8, 2026 00:35
Use a compatibility CPU target for the Quarkus and Spring Boot native
fixtures so cached executables can run on other CI hosts of the same
OS and architecture.

Document why the S3 prefix does not isolate CPU features. Link the
GraalVM guidance and record stronger targets as candidates requiring
guarantees across eligible build and execution hosts.

Resolve forwarded API JAR properties on the project inside dependency
blocks so both native fixtures consume the supplied project JAR.
Remove eleven API JAR declarations and Kafka's unused application IAST
helper JAR. Drop the matching nested dependencies and obsolete lockfile
configuration names so unrelated JAR changes no longer rebuild fixtures.

Keep the smoke-test harness dependencies, the WildFly WAR's published API,
and the project JARs consumed by JVM and native applications.
@bric3
bric3 force-pushed the bdu/cache-smoke-test-results branch from be18123 to f7be66e Compare November 7, 2026 23:36

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: tooling Build & Tooling tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant