ci: upload system-tests results to Test Optimization and add tracer-release nightly by nccatoni · Pull Request #3660 · DataDog/dd-trace-php

nccatoni · 2026-02-18T14:16:02Z

Description

Two changes to the system-tests CI integration:

Upload all system-tests results to Test Optimization — adds an after_script to the .system_tests base job that copies JUnit XML files and uploads them via the existing silent-upload-junit-to-datadog.sh script.
Add tracer-release nightly — create a new system-tests workflow that uses the official system-tests reusable workflow to run the tracer-release scenario group on master.

This is meant to replace the system-tests-dashboard nightly.

Reviewer checklist

This is a CI-only change, no code changes.
Appropriate labels assigned.

Run the official system-tests reusable workflow daily at 4 AM GMT using dev mode against the latest commit on the default branch. Co-authored-by: Cursor <cursoragent@cursor.com>

datadog-official · 2026-02-18T14:22:44Z

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 13 Tests failed

testSearchPhpBinaries from integration.DDTrace\Tests\Integration\PHPInstallerTest (Datadog) (Fix with Cursor)
Risky Test
phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:52
testSimplePushAndProcess from laravel-58-test.DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 69bd5d83000000002ff5b09b9d1092d4
tid: 69bd5d8300000000
hexProcessTraceId: 2ff5b09b9d1092d4
hexProcessSpanId: d2a7c8e43ff694f8
processTraceId: 3455862471478252244
processSpanId: 15179321951726769400

phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:106
testSimplePushAndProcess from laravel-8x-test.DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 69bd5d37000000002a3d8441dd688e48
tid: 69bd5d3700000000
hexProcessTraceId: 2a3d8441dd688e48
hexProcessSpanId: 23a046075748d0c2
processTraceId: 3043734341594680904
processSpanId: 2567128784944287938
View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: f67c9a0 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

Co-authored-by: Cursor <cursoragent@cursor.com>

codecov-commenter · 2026-02-18T14:35:08Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.30%. Comparing base (147f3c2) to head (f67c9a0).
⚠️ Report is 18 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3660      +/-   ##
==========================================
- Coverage   62.40%   62.30%   -0.11%     
==========================================
  Files         142      142              
  Lines       13586    13586              
  Branches     1775     1775              
==========================================
- Hits         8479     8465      -14     
- Misses       4301     4314      +13     
- Partials      806      807       +1

see 3 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 147f3c2...f67c9a0. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Co-authored-by: Cursor <cursoragent@cursor.com>

- Add after_script to .system_tests base job to upload JUnit results to Datadog Test Optimization using the existing upload script - Add tracer-release scenario group to configure_system_tests on schedule and master runs - Remove separate GitHub Actions nightly workflow Made-with: Cursor

The silent wrapper's grep returns exit code 1 when no summary lines are found, causing the after_script to fail. Use the non-silent version with || true to prevent after_script failures. Made-with: Cursor

Replace the generic upload-junit-to-datadog.sh with a custom after_script that: - Uses --service system-tests (not dd-trace-php-tests) - References logs directly from system-tests/logs*/ - Uses the datadog-ci standalone binary (no npm needed) - Uses Python for JSON/zip (guaranteed available in the image) - Provides explicit error messages instead of silent exit 0 Made-with: Cursor

pr-commenter · 2026-02-26T17:25:52Z

Benchmarks [ tracer ]

Benchmark execution time: 2026-03-19 17:08:48

Comparing candidate commit f00dc87 in PR branch add-nightly-system-tests with baseline commit 147f3c2 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 194 metrics, 0 unstable metrics.

Made-with: Cursor

nccatoni · 2026-03-05T12:06:47Z

For this to work two things need to be done:
Adding docker/login-action and nick-fields/retry to the list of authorized actions for the repo
Adding an Datadog API key as a github repo secret

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a65519dc59

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

.github/workflows/system_tests.yml

.gitlab/generate-package.php

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b44c4ab3eb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

.gitlab/generate-package.php

Leiyks · 2026-03-12T14:52:28Z

.gitlab/generate-package.php

+  after_script:
+    - |
+      set +e
+      echo "=== Uploading system-tests JUnit results to Test Optimization ==="
+
+      # Check that there are JUnit files to upload
+      ls system-tests/logs*/reportJunit.xml >/dev/null 2>&1
+      if [ $? -ne 0 ]; then echo "No JUnit XML files found, skipping upload"; exit 0; fi
+
+      # Download datadog-ci standalone binary
+      echo "Downloading datadog-ci..."
+      curl -L --fail "https://github.com/DataDog/datadog-ci/releases/latest/download/datadog-ci_linux-x64" -o /tmp/datadog-ci
+      if [ $? -ne 0 ]; then echo "Failed to download datadog-ci"; exit 0; fi
+      chmod +x /tmp/datadog-ci
+
+      # Download and extract Vault CLI
+      echo "Downloading Vault..."
+      VAULT_VERSION="1.20.0"
+      curl -L --fail "https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip" -o /tmp/vault.zip
+      if [ $? -ne 0 ]; then echo "Failed to download Vault"; exit 0; fi
+      python3 -c "import zipfile; zipfile.ZipFile('/tmp/vault.zip').extractall('/tmp/')"
+      chmod +x /tmp/vault
+
+      # Fetch API key from Vault
+      echo "Fetching API key from Vault..."
+      VAULT_JSON=$(/tmp/vault kv get --format=json "kv/k8s/gitlab-runner/dd-trace-php/datadoghq-api-key" 2>&1)
+      if [ $? -ne 0 ]; then echo "Failed to fetch API key from Vault: $VAULT_JSON"; exit 0; fi
+      export DATADOG_API_KEY=$(echo "$VAULT_JSON" | python3 -c "import sys,json; print(json.loads(sys.stdin.read())['data']['data']['key'])")
+      export DATADOG_SITE="datadoghq.com"
+
+      # Upload JUnit results
+      echo "Uploading JUnit results..."
+      /tmp/datadog-ci junit upload \
+        system-tests/logs*/reportJunit.xml \
+        --service system-tests \
+        --env ci \
+        --verbose \
+        --xpath-tag "test.codeowners=/testcase/properties/property[@name='test.codeowners']"
+      echo "=== Upload complete ==="


We have a script that basically do all of this in .gitlab/upload-junit-to-datadog.sh

You may need to tweak it a bit to add the xpath-tag thing tho

…ible scenarios - Use vault CLI instead of curl to fetch DD_API_KEY (matching upload-junit-to-datadog.sh) - Add error handling for vault fetch failure - Use compute-workflow-parameters.py to filter out lambda scenarios unsupported by PHP

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e44dc4c4b6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

.gitlab/generate-package.php

Add nightly system-tests workflow

17ff796

Run the official system-tests reusable workflow daily at 4 AM GMT using dev mode against the latest commit on the default branch. Co-authored-by: Cursor <cursoragent@cursor.com>

Enable push_to_test_optimization

10f7eab

Co-authored-by: Cursor <cursoragent@cursor.com>

nccatoni changed the title ~~Add nightly system-tests workflow~~ ci: add nightly system-tests workflow Feb 18, 2026

nccatoni and others added 6 commits February 18, 2026 15:40

Pass secrets explicitly to reusable workflow

2f73f4b

Co-authored-by: Cursor <cursoragent@cursor.com>

Only pass TEST_OPTIMIZATION_API_KEY secret

34d46d2

Co-authored-by: Cursor <cursoragent@cursor.com>

tmp: add pull_request trigger for testing

420b281

Co-authored-by: Cursor <cursoragent@cursor.com>

Set desired_execution_time to 300s

1185a36

Co-authored-by: Cursor <cursoragent@cursor.com>

Also pass DD_API_KEY secret

4af9b76

Co-authored-by: Cursor <cursoragent@cursor.com>

nccatoni changed the title ~~ci: add nightly system-tests workflow~~ ci: upload system-tests results to Test Optimization and add tracer-release nightly Feb 26, 2026

nccatoni added 2 commits February 26, 2026 16:54

ci: fix after_script by using upload script directly

42bcd97

The silent wrapper's grep returns exit code 1 when no summary lines are found, causing the after_script to fail. Use the non-silent version with || true to prevent after_script failures. Made-with: Cursor

nccatoni added 3 commits March 2, 2026 14:20

Run only tracer-release on schedule, it already includes other groups

34f79e3

Made-with: Cursor

Temporarily run tracer-release on all triggers for testing

bafec28

Made-with: Cursor

Tracer release scenario group on main

578e0b6

nccatoni force-pushed the add-nightly-system-tests branch from 9a812c9 to 578e0b6 Compare March 5, 2026 11:08

nccatoni added 2 commits March 5, 2026 12:11

Test

4db0655

Remove ref argument

a65519d

nccatoni marked this pull request as ready for review March 5, 2026 12:06

nccatoni requested a review from a team as a code owner March 5, 2026 12:06

chatgpt-codex-connector bot reviewed Mar 5, 2026

View reviewed changes

.github/workflows/system_tests.yml Outdated Show resolved Hide resolved

.gitlab/generate-package.php Outdated Show resolved Hide resolved

nccatoni marked this pull request as draft March 9, 2026 10:15

Every day schedule (include weekends)

b44c4ab

nccatoni marked this pull request as ready for review March 12, 2026 14:37

chatgpt-codex-connector bot reviewed Mar 12, 2026

View reviewed changes

.gitlab/generate-package.php Outdated Show resolved Hide resolved

Leiyks reviewed Mar 12, 2026

View reviewed changes

nccatoni added 6 commits March 12, 2026 17:52

Use upload-junit-to-datadog.sh

5159a71

Remove github workflow

85c13ba

Test

3c214dc

Add tracer-release system tests job on GitLab

be1373a

Merge branch 'master' into add-nightly-system-tests

a258de8

Fetch DD_API_KEY from Vault for tracer-release system tests

6b14c2f

nccatoni marked this pull request as draft March 19, 2026 10:17

nccatoni added 9 commits March 19, 2026 11:37

fail when vault call fails

9d3c4ee

Fix vault path

99bd6b0

Using vault CLI

922d80d

Use compute-workflow-parameters.py to get PHP-compatible scenarios

b0982a7

Pass all computed scenarios to run.sh without filtering

f00dc87

Increase tracer-release system tests timeout to 4h

0a7ccd1

Run all scenarios regardless of individual failures

dbf442b

Putting rules back

e44dc4c

nccatoni marked this pull request as ready for review March 20, 2026 14:04

nccatoni requested a review from Leiyks March 20, 2026 14:04

chatgpt-codex-connector bot reviewed Mar 20, 2026

View reviewed changes

.gitlab/generate-package.php Outdated Show resolved Hide resolved

.gitlab/generate-package.php Show resolved Hide resolved

nccatoni marked this pull request as draft March 20, 2026 14:28

Fix JUnit artifact collisions across multiple scenarios

f67c9a0

nccatoni marked this pull request as ready for review March 20, 2026 14:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: upload system-tests results to Test Optimization and add tracer-release nightly#3660

ci: upload system-tests results to Test Optimization and add tracer-release nightly#3660
nccatoni wants to merge 32 commits intomasterfrom
add-nightly-system-tests

nccatoni commented Feb 18, 2026 •

edited

Loading

Uh oh!

datadog-official bot commented Feb 18, 2026 •

edited by datadog-datadog-prod-us1-2 bot

Loading

Uh oh!

codecov-commenter commented Feb 18, 2026 •

edited

Loading

Uh oh!

pr-commenter bot commented Feb 26, 2026 •

edited

Loading

Uh oh!

nccatoni commented Mar 5, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

Leiyks Mar 12, 2026

Uh oh!

Leiyks Mar 12, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

nccatoni commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Reviewer checklist

Uh oh!

datadog-official bot commented Feb 18, 2026 • edited by datadog-datadog-prod-us1-2 bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

codecov-commenter commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pr-commenter bot commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [ tracer ]

Uh oh!

nccatoni commented Mar 5, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Leiyks Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Leiyks Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nccatoni commented Feb 18, 2026 •

edited

Loading

datadog-official bot commented Feb 18, 2026 •

edited by datadog-datadog-prod-us1-2 bot

Loading

codecov-commenter commented Feb 18, 2026 •

edited

Loading

pr-commenter bot commented Feb 26, 2026 •

edited

Loading