[tests] Replace make-based packaged macOS test runner with C# script by rolfbjarne · Pull Request #25749 · dotnet/macios

rolfbjarne · 2026-06-19T11:06:03Z

Replace the Makefile-based test runner (packaged-macos-tests.mk + run-with-timeout + mac-test-report-generator) with a single C# script project (scripts/run-packaged-macos-tests) that:

Executes pre-built macOS and Mac Catalyst test apps with timeout and launch retry logic
Generates an HTML report with per-platform/architecture breakdown tables showing output links and [FAIL] lines inline
Generates TestSummary.md for GitHub PR comments
Handles ARM64/x64 architecture detection and skipping
Collects and links crash reports

Simplifies run_mac_tests.ps1 from ~235 lines to ~150 lines by delegating all test execution and reporting to the new C# tool.

🤖 Pull request created by Copilot

Replace the Makefile-based test runner (packaged-macos-tests.mk + run-with-timeout + mac-test-report-generator) with a single C# script project (scripts/run-packaged-macos-tests) that: - Executes pre-built macOS and Mac Catalyst test apps with timeout and launch retry logic (previously in run-with-timeout) - Generates an HTML report with per-platform/architecture breakdown tables (previously in mac-test-report-generator) - Generates TestSummary.md for GitHub PR comments - Handles ARM64/x64 architecture detection and skipping - Reports crash reports Simplify run_mac_tests.ps1 to delegate all test execution and reporting to the new C# tool, and update package-mac-tests.sh to include it in the test package. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The --autostart --autoexit arguments are only passed to Mac Catalyst test apps, matching the behavior in packaged-macos-tests.mk. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Combine stdout and stderr into a single output file per test config - Add output file link directly in the table's Output column - Show [FAIL] lines immediately after each failing test row in the table instead of in a separate list after the table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Add @media (prefers-color-scheme: dark) CSS with GitHub-dark-themed colors - Skip x64 test configs on macOS 27+ (not needed, always on Apple Silicon) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…n detection Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…code Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…os-test-runner

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…os-test-runner

Copilot

Pull request overview

This PR replaces the previous make-based packaged macOS test runner (Makefile targets + run-with-timeout + mac-test-report-generator) with a single C# script tool that runs the prebuilt test apps and generates both an HTML report and TestSummary.md, while simplifying the CI PowerShell wrapper.

Changes:

Added scripts/run-packaged-macos-tests (C# tool) to execute packaged macOS/Mac Catalyst tests with timeout/retry and to generate HTML + markdown summaries.
Updated run_mac_tests.ps1 to build and invoke the new tool instead of make targets and the old HTML report generator.
Removed mac-test-report-generator integration from the packaged tests and packaging scripts.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tools/devops/automation/scripts/run_mac_tests.ps1	Delegates packaged test execution + reporting to the new C# tool.
tests/packaged-macos-tests.mk	Removes the old HTML report generator include/target.
tests/package-mac-tests.sh	Packages the new `run-packaged-macos-tests` script instead of `mac-test-report-generator`.
scripts/run-packaged-macos-tests/run-packaged-macos-tests.csproj	New script project for the packaged test runner tool.
scripts/run-packaged-macos-tests/run-packaged-macos-tests.cs	Implements running test apps, aggregating results, and generating HTML/TestSummary outputs.
scripts/run-packaged-macos-tests/fragment.mk	Adds template.mk integration for building the new script.
scripts/mac-test-report-generator/mac-test-report-generator.cs	Deletes the old standalone HTML report generator implementation.
scripts/mac-test-report-generator/fragment.mk	Removes template.mk integration for the deleted report generator.

+
+if ($testExitCode -ne 0) {
+  # We failed, so write to the comment file why we failed.
+  Set-Content -Path "$GithubFailureCommentFile" -Value "Tests failed (see HTML report for details)"


+	var htmlDir = Path.GetDirectoryName (reportPath);
+	if (!string.IsNullOrEmpty (htmlDir))
+		Directory.CreateDirectory (htmlDir);


- Fix failure comment to not assume HTML report exists - Dispose ManualResetEvent with 'using' declaration - Use Path.GetFullPath for report path to avoid null directory Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

vs-mobiletools-engineering-service2 · 2026-06-23T17:40:38Z

✅ [PR Build #174b5b9] Build passed (Detect API changes) ✅

Pipeline on Agent
Hash: 174b5b9a62d770c1bfd53e7c79d2e89d9c98bbb7 [PR build]

vs-mobiletools-engineering-service2 · 2026-06-23T17:48:17Z

✅ [PR Build #174b5b9] Build passed (Build packages) ✅

Pipeline on Agent
Hash: 174b5b9a62d770c1bfd53e7c79d2e89d9c98bbb7 [PR build]

vs-mobiletools-engineering-service2 · 2026-06-23T17:52:59Z

✅ API diff for current PR / commit

NET (empty diffs)

iOS: vsdrops gist ( No breaking changes )
tvOS: vsdrops gist ( No breaking changes )
MacCatalyst: vsdrops gist ( No breaking changes )
macOS: vsdrops gist ( No breaking changes )

✅ API diff vs stable

NET (empty diffs)

iOS: vsdrops gist ( No breaking changes )
tvOS: vsdrops gist ( No breaking changes )
MacCatalyst: vsdrops gist ( No breaking changes )
macOS: vsdrops gist ( No breaking changes )

ℹ️ Generator diff

Generator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes)

Pipeline on Agent
Hash: 174b5b9a62d770c1bfd53e7c79d2e89d9c98bbb7 [PR build]

vs-mobiletools-engineering-service2 · 2026-06-23T18:02:17Z

✅ [PR Build #174b5b9] Build passed (Build macOS tests) ✅

Pipeline on Agent
Hash: 174b5b9a62d770c1bfd53e7c79d2e89d9c98bbb7 [PR build]

vs-mobiletools-engineering-service2 · 2026-06-23T22:14:30Z

🚀 [CI Build #174b5b9] Test results 🚀

Test results

✅ All tests passed on VSTS: test results.

🎉 All 207 tests passed 🎉

Tests counts

✅ assembly-processing: All 1 tests passed. Html Report (VSDrops) Download
✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (iOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (tvOS): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 2 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 4 tests passed. Html Report (VSDrops) Download
✅ generator: All 5 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 4 tests passed. Html Report (VSDrops) Download
✅ introspection: All 4 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ linker (iOS): All 15 tests passed. Html Report (VSDrops) Download
✅ linker (MacCatalyst): All 15 tests passed. Html Report (VSDrops) Download
✅ linker (macOS): All 21 tests passed. Html Report (VSDrops) Download
✅ linker (tvOS): All 15 tests passed. Html Report (VSDrops) Download
✅ monotouch (iOS): All 20 tests passed. Html Report (VSDrops) Download
✅ monotouch (MacCatalyst): All 19 tests passed. Html Report (VSDrops) Download
✅ monotouch (macOS): All 20 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ monotouch (tvOS): All 20 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ sharpie: All 1 tests passed. Html Report (VSDrops) Download
✅ windows: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. Html Report (VSDrops) Download
✅ xtro: All 1 tests passed. Html Report (VSDrops) Download

macOS tests

✅ Tests on macOS Monterey (12): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Ventura (13): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sequoia (15): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Tahoe (26): All 5 tests passed. Html Report (VSDrops) Download

Linux Build Verification

✅ Linux build succeeded

Pipeline on Agent
Hash: 174b5b9a62d770c1bfd53e7c79d2e89d9c98bbb7 [PR build]

rolfbjarne and others added 5 commits June 19, 2026 13:03

Fix launch arguments: use for Mac Catalyst only, not macOS

63a7071

The --autostart --autoexit arguments are only passed to Mac Catalyst test apps, matching the behavior in packaged-macos-tests.mk. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove stale sentinel file

c4c874a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Show entire [FAIL] line including the [FAIL] prefix

1af29c6

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

rolfbjarne added the copilot label Jun 19, 2026

Auto-format source code

f7220b4

This comment has been minimized.

Sign in to view

This comment was marked as outdated.

Sign in to view

This comment has been minimized.

Sign in to view

rolfbjarne and others added 7 commits June 23, 2026 10:19

Add dark mode support to HTML report and skip x64 on macOS 27+

cf7b7fa

- Add @media (prefers-color-scheme: dark) CSS with GitHub-dark-themed colors - Skip x64 test configs on macOS 27+ (not needed, always on Apple Silicon) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use sysctlbyname P/Invoke instead of sw_vers process for macOS versio…

4fc9e22

…n detection Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Show skip reason in HTML report for skipped x64 tests

ff0f6f5

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add #nullable enable to fix CS8632 warning

1fa44f7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Show skip reason instead of exit code for skipped tests in HTML report

89c957e

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Show 'Tests run: ...' summary line in Details column instead of exit …

90bb934

…code Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into dev/rolf/improved-mac…

f03abda

…os-test-runner

This comment has been minimized.

Sign in to view

rolfbjarne and others added 5 commits June 23, 2026 15:54

Remove mac-test-report-generator (replaced by run-packaged-macos-tests)

a5ffbc7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Strip prefix before 'Tests run:' in details column

b5b41c5

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Track and display test duration in HTML report

a3ec3b9

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use ProcessStartInfo.ArgumentList instead of Arguments string

54e06d1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into dev/rolf/improved-mac…

6e61151

…os-test-runner

This comment has been minimized.

Sign in to view

rolfbjarne requested a review from Copilot June 23, 2026 16:49

Copilot started reviewing on behalf of rolfbjarne June 23, 2026 16:50 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Address PR review comments

174b5b9

- Fix failure comment to not assume HTML report exists - Dispose ManualResetEvent with 'using' declaration - Use Path.GetFullPath for report path to avoid null directory Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

rolfbjarne marked this pull request as ready for review June 23, 2026 17:24

rolfbjarne enabled auto-merge (squash) June 23, 2026 17:25

This comment has been minimized.

Sign in to view

dalexsoto approved these changes Jun 24, 2026

View reviewed changes

rolfbjarne merged commit dfccfc6 into main Jun 24, 2026
56 checks passed

rolfbjarne deleted the dev/rolf/improved-macos-test-runner branch June 24, 2026 01:05

Conversation

rolfbjarne commented Jun 19, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment was marked as outdated.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

vs-mobiletools-engineering-service2 commented Jun 23, 2026

✅ [PR Build #174b5b9] Build passed (Detect API changes) ✅

Uh oh!

This comment has been minimized.

vs-mobiletools-engineering-service2 commented Jun 23, 2026

✅ [PR Build #174b5b9] Build passed (Build packages) ✅

Uh oh!

vs-mobiletools-engineering-service2 commented Jun 23, 2026

✅ API diff for current PR / commit

✅ API diff vs stable

ℹ️ Generator diff

Uh oh!

vs-mobiletools-engineering-service2 commented Jun 23, 2026

✅ [PR Build #174b5b9] Build passed (Build macOS tests) ✅

Uh oh!

This comment has been minimized.

vs-mobiletools-engineering-service2 commented Jun 23, 2026

🚀 [CI Build #174b5b9] Test results 🚀

Test results

Tests counts

macOS tests

Linux Build Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants