feat: add matrix tests to containers

[kaahos]

What does this PR do?:

This PR allows to run all the CI tests matrix locally in containers.

Motivation:

Run the full CI tests matrix locally. It comes out to be easier to run full testsuite with agents and is also useful in case of infrastructure flakiness.

Additional Notes:

How to test the change?:

• Run the testsuite

For Datadog employees:

• If this PR touches code that signs or publishes builds or packages, or handles
  credentials of any kind, I've requested a security review (run the dd:platform-security-review
  skill, or file a request via the PSEC review form).
  bewaire also runs automatically on every PR.
• This PR doesn't touch any of that.
• JIRA: [JIRA-XXXX]

Unsure? Have a question? Request a review!

[dd-octo-sts]

CI Test Results

Run: #29108122307 | Commit: f0e2a3d | Duration: 14m 55s (longest job)

❌ 1 of 32 test jobs failed

Status Overview

JDK	glibc-aarch64/debug	glibc-amd64/debug	musl-aarch64/debug	musl-amd64/debug
8	-	❌	-	-
8-ibm	-	✅	-	-
8-j9	✅	✅	-	-
8-librca	-	-	✅	✅
8-orcl	-	✅	-	-
11	-	✅	-	-
11-j9	✅	✅	-	-
11-librca	-	-	✅	✅
17	✅	✅	-	-
17-graal	✅	✅	-	-
17-j9	✅	✅	-	-
17-librca	-	-	✅	✅
21	✅	✅	-	-
21-graal	✅	✅	-	-
21-librca	-	-	✅	✅
25	✅	✅	-	-
25-graal	✅	✅	-	-
25-librca	-	-	✅	✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Failed Tests

glibc-amd64/debug / 8

Job: View logs

No detailed failure information available. Check the job logs.

Summary: Total: 32 | Passed: 31 | Failed: 1

---

Updated: 2026-07-10 16:57:31 UTC

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 53fb055c24

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-prod-us1-3]

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 3 Pipeline jobs failed

CI Run | test-matrix / test-linux-glibc-amd64 (8, debug)     

DataDog/java-profiler | reliability-chaos-aarch64: [profiler+tracer, tcmalloc, 21.0.3-tem]     

DataDog/java-profiler | reliability-chaos-amd64: [profiler, tcmalloc, 21.0.3-tem]     

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: afa13dd | Docs | Datadog PR Page | Give us feedback!}

[dd-octo-sts]

Reliability & Chaos Results

❌ 1 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/124061214

❌ chaos: profiler tcmalloc amd64 21 0 3 temXchaos

Chaos harness crashed (RC=124)
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000
# C  [libjavaProfiler-dd-tmp13295632808171335641.so+0x20819]  CTimer::signalHandler(int, siginfo_t*, void*)+0x79

[rkennke]

Review

Scope: 3 files, +739/-25 — README.md, utils/README.md, utils/run-containers-tests.sh. No CI workflow files touched; this only extends the local dev/test script that mirrors CI's container test matrix (glibc/musl × JDK × arch × config), adding --matrix, --run, --fail-fast, and multi-value (all/comma-list) dimension filters, plus a status table and markdown/JSON summary reports.

Overview

The script gains dimension expansion (expand_dimension, expand_jdk_dimension), a skip_reason() table that mirrors known CI exclusions (musl+j9/graal, graal+asan, glibc/aarch64 8-11, musl+sanitizers), a matrix preview/confirmation flow, sequential matrix execution with --fail-fast, and report writers. Docs in both READMEs are updated consistently with the new flags.

Bug found (verified by running the script)

utils/run-containers-tests.sh:907:

if (( MATRIX_TOTAL > 1 || MATRIX_RUNNABLE > 1 || MATRIX_MODE )); then

MATRIX_MODE holds the string "true"/"false", not 1/0. In a bash arithmetic context, a variable holding a non-numeric string is itself re-evaluated as a variable name — since there's no shell variable named true, it silently resolves to 0. So MATRIX_MODE never actually contributes to this condition.

Confirmed with:

./run-containers-tests.sh --matrix --libc=glibc --jdk=21 --arch=x64 --config=debug --container=docker
# → "Error: container runtime 'docker' not found" — i.e. it fell through to the single-cell
#   execution path instead of showing the matrix preview/confirmation, even with --run.

Impact is narrow — bare --matrix or partial filters expand to >1 cell and work correctly (verified separately). But it's a real correctness bug in exactly the "preview a specific single-cell matrix explicitly" path the flag is meant to support, and "boolean flag in arithmetic context" is a bug class worth fixing before it recurs elsewhere. Fix:

if (( MATRIX_TOTAL > 1 || MATRIX_RUNNABLE > 1 )) || $MATRIX_MODE; then

Minor / non-blocking

• matrix_command() (run-containers-tests.sh:336) is defined but never called anywhere in the script — dead code, presumably superseded by the inline command-building in run_matrix(). Worth deleting.
• Heavy use of eval for indirect array access (append_unique_line, read_lines_into_array, read_string_into_array) works but is easy to get wrong later; a bash-4+ local -n nameref would be safer if the minimum bash version allows it — just flagging the tradeoff (macOS ships bash 3.2 by default, where namerefs aren't available, so eval may be the pragmatic choice here).
• No automated test coverage for the new flag-parsing/expansion logic (understandable for a bash dev-utility script, but a quick self-check invocation, e.g. a --matrix dry run, in CI would catch regressions like the one above).

Verdict

Nothing here is a security or CI-breaking risk — it's an opt-in local dev tool with no production/CI wiring changed. The MATRIX_MODE arithmetic bug is real and demonstrable but narrow in scope. Flagging as should-fix-before-merge rather than a hard blocker — it's a quick one-line fix, and the documented usage patterns all work correctly as tested.

[kaahos]

Review

Scope: 3 files, +739/-25 — README.md, utils/README.md, utils/run-containers-tests.sh. No CI workflow files touched; this only extends the local dev/test script that mirrors CI's container test matrix (glibc/musl × JDK × arch × config), adding --matrix, --run, --fail-fast, and multi-value (all/comma-list) dimension filters, plus a status table and markdown/JSON summary reports.

Overview

The script gains dimension expansion (expand_dimension, expand_jdk_dimension), a skip_reason() table that mirrors known CI exclusions (musl+j9/graal, graal+asan, glibc/aarch64 8-11, musl+sanitizers), a matrix preview/confirmation flow, sequential matrix execution with --fail-fast, and report writers. Docs in both READMEs are updated consistently with the new flags.

Bug found (verified by running the script)

utils/run-containers-tests.sh:907:

if (( MATRIX_TOTAL > 1 || MATRIX_RUNNABLE > 1 || MATRIX_MODE )); then

MATRIX_MODE holds the string "true"/"false", not 1/0. In a bash arithmetic context, a variable holding a non-numeric string is itself re-evaluated as a variable name — since there's no shell variable named true, it silently resolves to 0. So MATRIX_MODE never actually contributes to this condition.

Confirmed with:

./run-containers-tests.sh --matrix --libc=glibc --jdk=21 --arch=x64 --config=debug --container=docker
# → "Error: container runtime 'docker' not found" — i.e. it fell through to the single-cell
#   execution path instead of showing the matrix preview/confirmation, even with --run.

Impact is narrow — bare --matrix or partial filters expand to >1 cell and work correctly (verified separately). But it's a real correctness bug in exactly the "preview a specific single-cell matrix explicitly" path the flag is meant to support, and "boolean flag in arithmetic context" is a bug class worth fixing before it recurs elsewhere. Fix:

if (( MATRIX_TOTAL > 1 || MATRIX_RUNNABLE > 1 )) || $MATRIX_MODE; then

Minor / non-blocking

• matrix_command() (run-containers-tests.sh:336) is defined but never called anywhere in the script — dead code, presumably superseded by the inline command-building in run_matrix(). Worth deleting.
• Heavy use of eval for indirect array access (append_unique_line, read_lines_into_array, read_string_into_array) works but is easy to get wrong later; a bash-4+ local -n nameref would be safer if the minimum bash version allows it — just flagging the tradeoff (macOS ships bash 3.2 by default, where namerefs aren't available, so eval may be the pragmatic choice here).
• No automated test coverage for the new flag-parsing/expansion logic (understandable for a bash dev-utility script, but a quick self-check invocation, e.g. a --matrix dry run, in CI would catch regressions like the one above).

Verdict

Nothing here is a security or CI-breaking risk — it's an opt-in local dev tool with no production/CI wiring changed. The MATRIX_MODE arithmetic bug is real and demonstrable but narrow in scope. Flagging as should-fix-before-merge rather than a hard blocker — it's a quick one-line fix, and the documented usage patterns all work correctly as tested.

@rkennke thanks for the review; I've applied your suggestions in 6462618.

[dd-octo-sts]

Benchmark Results (commit afa13dd)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124061231 Commit: afa13dd56dcae17b00897bf5c3968cf5354f0303

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)

Benchmark	JDK	Latest	Dev	Δ (dev vs latest)	Issues L/D
akka-uct	21	✅ 10240 ms (21 iters)	✅ 10211 ms (21 iters)	≈ -0.3% (±11.2%)	— / —
akka-uct	25	✅ 8856 ms (24 iters)	✅ 8908 ms (24 iters)	≈ +0.6% (±9.8%)	— / —
finagle-chirper	21	✅ 5953 ms (33 iters)	✅ 5963 ms (33 iters)	≈ +0.2% (±25%)	⚠️ W:3 / ⚠️ W:3
finagle-chirper	25	✅ 5455 ms (36 iters)	✅ 5521 ms (36 iters)	≈ +1.2% (±25.4%)	⚠️ W:3 / ⚠️ W:3
fj-kmeans	21	✅ 2814 ms (66 iters)	✅ 2743 ms (68 iters)	≈ -2.5% (±2.6%)	— / —
fj-kmeans	25	✅ 2823 ms (66 iters)	✅ 2801 ms (66 iters)	≈ -0.8% (±2.6%)	— / —
future-genetic	21	✅ 2072 ms (90 iters)	✅ 2078 ms (89 iters)	≈ +0.3% (±2.6%)	— / —
future-genetic	25	✅ 2039 ms (91 iters)	✅ 2011 ms (93 iters)	≈ -1.4% (±2.6%)	— / —
naive-bayes	25	✅ 1034 ms (165 iters)	✅ 1012 ms (169 iters)	≈ -2.1% (±31.4%)	— / —
reactors	25	✅ 18700 ms (15 iters)	✅ 18422 ms (15 iters)	≈ -1.5% (±3.8%)	— / —

Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark	JDK	Dropped rec	Dropped jvmti	Dropped trace	Skipped WC	AGCT fail	Unwind fail
akka-uct	21	✅ / ✅	✅ / ✅	2 / 1	2000 / 1843	✅ / ✅	✅ / ✅
akka-uct	25	✅ / ✅	✅ / ✅	✅ / ✅	2348 / 2352	✅ / ✅	✅ / ✅
finagle-chirper	21	✅ / ✅	✅ / ✅	2 / 2	8946 / 8469	✅ / ✅	✅ / ✅
finagle-chirper	25	✅ / ✅	✅ / ✅	2 / 1	8522 / 8406	✅ / ✅	✅ / ✅
fj-kmeans	21	✅ / ✅	✅ / ✅	1 / 2	1260 / 1257	✅ / ✅	✅ / ✅
fj-kmeans	25	✅ / ✅	✅ / ✅	3 / 2	1276 / 1270	✅ / ✅	✅ / ✅
future-genetic	21	✅ / ✅	✅ / ✅	5 / 2	3014 / 2919	✅ / ✅	✅ / ✅
future-genetic	25	✅ / ✅	✅ / ✅	✅ / 2	2896 / 2784	✅ / ✅	✅ / ✅
naive-bayes	25	✅ / ✅	✅ / ✅	4 / 4	3438 / 3501	✅ / ✅	✅ / ✅
reactors	21	✅ / ✅	✅ / ✅	✅ / ✅	✅ / ✅	✅ / ✅	✅ / ✅
reactors	25	✅ / ✅	✅ / ✅	1 / ✅	1854 / 1881	✅ / ✅	✅ / ✅