Deadline-aware cancellation (prototype)

[Jakub Jareš (nohwnd)]

CI hard-cancels a job at a fixed wall-clock time. When it fires the runner kills the test process, and we lose the TRX/HTML/AzDO reports and get no dump for a hanging test. This is a prototype that tells MTP when that deadline is, so it can react a little early while it still owns its shutdown.

What it does

The deadline arrives as an environment variable, an absolute instant:

• TESTINGPLATFORM_DEADLINE: ISO 8601, parsed to UTC.
• TESTINGPLATFORM_DEADLINE_STOP_MARGIN: lead time for the graceful stop. Default 60s.
• TESTINGPLATFORM_DEADLINE_DUMP_MARGIN: lead time for the hang dump. Default 30s.

Two reactions, armed off the same instant:

• At deadline - stopMargin an in-process extension asks the framework to gracefully stop scheduling new tests (IGracefulStopTestExecutionCapability). In-flight tests finish, the session ends normally, and every reporter gets to finalize.
• At deadline - dumpMargin the out-of-process HangDump controller dumps the process tree and kills the host. This is the fallback for a wedged host that never reaches the graceful stop. It reuses the controller that HangDump already runs, so it costs nothing extra when --hangdump is on.

stopMargin > dumpMargin on purpose: try the clean stop first, dump only if that did not happen in time.

It is opt-in. No deadline set, both timers stay unarmed, no behavior change. The graceful stop also degrades to a no-op if the framework does not expose the capability.

Where the deadline comes from

Out of scope here, that is the "solve timing later" part. MTP has no hardcoded timeout, it only reacts to whatever instant the environment gives it. The CI side is a few lines of YAML that compute job start + timeout and export it. Rough sketches:

GitHub Actions (job has timeout-minutes: 60):

- name: Compute deadline
  shell: pwsh
  run: echo "TESTINGPLATFORM_DEADLINE=$((Get-Date).ToUniversalTime().AddMinutes(55).ToString('o'))" >> $env:GITHUB_ENV

Azure DevOps (job has timeoutInMinutes: 60):

- pwsh: echo "##vso[task.setvariable variable=TESTINGPLATFORM_DEADLINE]$((Get-Date).ToUniversalTime().AddMinutes(55).ToString('o'))"
  displayName: Compute deadline

Both approximate "now" as the job start, which is close enough. The nicer version is Arcade computing the real deadline once and exporting it for everyone, so nobody has to copy YAML around. Left for a follow-up.

Verified

build.cmd -pack passes 0/0. New acceptance tests:

• AbortAtDeadlineTests (5): past deadline stops immediately, future deadline stops when the timer fires, the stop margin is subtracted from the deadline, no deadline stays silent, missing capability is a no-op.
• HangDumpTests.HangDump_AbsoluteDeadline_CreateDump (3 tfms): 30 minute inactivity timeout so only the deadline path can fire, and it produces a dump. Full HangDumpTests still 30/30.

This is a prototype, so LMK what you think about the shape before I polish it. Open questions:

• Names for the env vars and the margins.
• Whether the dump-at-deadline message should say "deadline" instead of reusing the inactivity-timeout text.

[Copilot]

Pull request overview

Adds a prototype “deadline-aware cancellation” mechanism to Microsoft.Testing.Platform (MTP) so CI can provide an absolute wall-clock deadline and MTP can proactively (a) request a graceful stop before the runner hard-kills the process, and (b) trigger HangDump as a fallback before the deadline.

Changes:

• Introduces DeadlineHelper + new env vars (TESTINGPLATFORM_DEADLINE, *_STOP_MARGIN, *_DUMP_MARGIN) for parsing an absolute UTC deadline and margins.
• Registers a new in-proc AbortAtDeadlineExtension that schedules a timer to request IGracefulStopTestExecutionCapability.
• Extends HangDump to arm an additional one-shot timer for the absolute deadline and adds acceptance coverage for both graceful stop and deadline-driven dump.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HangDumpTests.cs	Adds acceptance coverage ensuring HangDump can be triggered via absolute deadline env var.
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/AbortAtDeadlineTests.cs	New acceptance suite validating the graceful-stop behavior around deadlines/margins and missing capability.
src/Platform/Microsoft.Testing.Platform/InternalAPI/InternalAPI.Unshipped.txt	Tracks newly added internal APIs/constants for PublicAPIAnalyzers.
src/Platform/Microsoft.Testing.Platform/Hosts/TestHostBuilder.Framework.cs	Registers AbortAtDeadlineExtension into the message bus when enabled.
src/Platform/Microsoft.Testing.Platform/Helpers/EnvironmentVariableConstants.cs	Adds constants for the new deadline-related environment variables.
src/Platform/Microsoft.Testing.Platform/Helpers/DeadlineHelper.cs	New helper to read/parse deadline + margins from environment.
src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs	New extension that arms a timer to request graceful stop before the deadline.
src/Platform/Microsoft.Testing.Extensions.TrxReport/InternalAPI/InternalAPI.Unshipped.txt	Tracks newly added env-var constants due to shared-source inclusion.
src/Platform/Microsoft.Testing.Extensions.Retry/InternalAPI/InternalAPI.Unshipped.txt	Tracks newly added env-var constants due to shared-source inclusion.
src/Platform/Microsoft.Testing.Extensions.MSBuild/InternalAPI.Unshipped.txt	Tracks newly added env-var constants due to shared-source inclusion.
src/Platform/Microsoft.Testing.Extensions.HotReload/InternalAPI/InternalAPI.Unshipped.txt	Tracks newly added env-var constants due to shared-source inclusion.
src/Platform/Microsoft.Testing.Extensions.HangDump/Microsoft.Testing.Extensions.HangDump.csproj	Links DeadlineHelper.cs into HangDump extension for shared deadline parsing.
src/Platform/Microsoft.Testing.Extensions.HangDump/InternalAPI/InternalAPI.Unshipped.txt	Tracks DeadlineHelper + env-var constants for this assembly.
src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs	Arms a new deadline-driven dump timer and prevents double dump via _dumpTaken.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 7 comments.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs:459

• This request is not actually best-effort when the host is wedged. NamedPipeClient.RequestReplyAsync waits for a response until the supplied run token is canceled (NamedPipeClient.cs:103-105), so a connected host whose control-pipe callback no longer runs can block here indefinitely and prevent both dump creation and process-tree termination. Skip this query for deadline-triggered dumps or bound it to a short deadline-specific budget.

                GetInProgressTestsResponse tests = await _namedPipeClient.RequestReplyAsync<GetInProgressTestsRequest, GetInProgressTestsResponse>(new GetInProgressTestsRequest(), cancellationToken).ConfigureAwait(false);

[Amaury Levé (Evangelink)]

Review — deadline-aware cancellation

I ran this through several review passes (expert MTP reviewer + a bug-focused diff reviewer) and then validated every finding against the code on 14d6b88. Skipping everything the earlier Copilot round already covered and you already fixed.

The shape is good and the concurrency work after the last round (the _dumpLock gate + TriggerDumpOnce trampoline, SubtractSaturating, the timer clamp) reads correctly to me. What follows is what survived validation, roughly in priority order. The first three are the ones I'd want settled before this stops being a prototype.

Things I checked and found clean, so you don't have to re-litigate them: the empty DataTypesConsumed is legal (AsynchronousMessageBus.InitAsync just creates no processor, and the consumer is still disposed via CommonTestHost line 401's messageBus.DataConsumerServices loop); the InternalAPI.Unshipped.txt entries are exact across all five projects that link-compile EnvironmentVariableConstants.cs; the .xlf files are genuinely generated (state="new", source==target, alphabetical); [Embedded] on DeadlineHelper matches every neighbour in that folder; and no new test hard-codes a \(\d+ms\) duration pattern.

[Amaury Levé (Evangelink)]

Minor — this test can't fail for the reason it exists.

UtcNow.AddSeconds(6) is evaluated in the test process, but the timer is armed in the child after process launch + runtime startup + the MTP builder runs. On a loaded CI agent that easily exceeds 6s, at which point dueTime clamps to zero and the run takes the "already past" path — which is what WhenDeadlineIsInThePast_GracefullyStopsImmediately already covers. The assertions still pass, so the test silently stops testing "the timer fires on schedule".

Same applies to WhenStopMarginIsSubtracted (60s out, 60s margin → stop instant ≈ now).

If you want this to actually pin the timer down, have the test asset report when the stop arrived (e.g. print elapsed-since-start from StopTestExecutionAsync) and assert it's non-trivially greater than zero.

Also: await GracefulStop.Instance.TCS.Task in the asset has no timeout, so a regression in the extension turns these into hangs rather than failures — the acceptance harness timeout is what would eventually catch it. A WaitAsync/Task.WhenAny with a generous cap would fail fast with a readable message instead.

[Jakub Jareš (nohwnd)]

Half fixed in e7786b2, and I want to be honest about the other half. The unbounded wait is gone: the asset now waits on Task.WhenAny(tcs.Task, Task.Delay(2min)), so a broken stop path fails the assertions instead of hanging until the harness kills it. I did not add an elapsed-time assertion. The tests already assert the new exit code (15) and the stop message, which is what catches a stop that never happened; a wall-clock timing assertion on top of that is the kind of check this repo warns is flaky, so for the prototype I left it out. Happy to add a generous lower bound if you would rather have it.

🤖

[Copilot]

Pull request overview

Copilot reviewed 70 out of 70 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

test/UnitTests/Microsoft.Testing.Platform.UnitTests/Helpers/DeadlineHelperTests.cs:1

• This new C# file is missing the UTF-8 BOM required by .editorconfig:66-67. Add the BOM so repository encoding checks and future rewrites preserve the required format.
  src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs:246
• For the startup-wedge case described above, this still blocks until the five-minute default handshake timeout. Killing a host that never connected does not complete the server's WaitForConnectionAsync, and the controller cannot reach WaitForExitAsync/OnTestHostProcessExitedAsync to publish the dump before the CI hard deadline. Race or cancel the handshake when the deadline dump wins, then await the dump task before continuing.

            await _logger.LogDebugAsync($"Wait for test host connection to the server pipe '{_singleConnectionNamedPipeServer.PipeName.Name}'").ConfigureAwait(false);
            await _waitConnectionTask.TimeoutAfterAsync(TimeoutHelper.DefaultHangTimeSpanTimeout).ConfigureAwait(false);
            using CancellationTokenSource timeout = new(TimeoutHelper.DefaultHangTimeSpanTimeout);
            using var linkedCancellationToken = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeout.Token);
            _waitConsumerPipeName.Wait(linkedCancellationToken.Token);

[Copilot]

Pull request overview

Copilot reviewed 70 out of 70 changed files in this pull request and generated 1 comment.

Suppressed comments (5)

src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs:494

• This five-second timeout is applied once per process because TakeDumpAsync is called sequentially for every node in the process tree. For a wedged host, each request times out, so six processes consume the entire default 30-second dump margin before dump creation; CI can hard-kill the job without any dump. Query/cache the in-progress tests once for the whole tree, or disable further queries after the first failure, so the timeout is a total bound rather than a per-process bound.

                using var queryCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
                queryCts.CancelAfter(InProgressTestsQueryTimeout);
                GetInProgressTestsResponse tests = await _namedPipeClient.RequestReplyAsync<GetInProgressTestsRequest, GetInProgressTestsResponse>(new GetInProgressTestsRequest(), queryCts.Token).ConfigureAwait(false);

src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs:371

• The deadline dump can fault here before entering the try/finally that kills the process tree. If logging or output display throws, the fire-and-forget timer task ends and the wedged host remains alive until CI hard-cancels it, producing no dump—the exact fallback this path is intended to guarantee. Make diagnostics best-effort and place all deadline work after root-process acquisition under a kill-guaranteeing finally.

        await _logger.LogInformationAsync($"{dumpReason}. Taking hang dump.").ConfigureAwait(false);
        await _outputDisplay.DisplayAsync(
            new ErrorMessageOutputDeviceData(triggeredByDeadline
                ? ExtensionResources.HangDumpDeadlineApproaching
                : string.Format(CultureInfo.InvariantCulture, ExtensionResources.HangDumpTimeoutExpired, _activityTimerValue)),
            cancellationToken).ConfigureAwait(false);

test/UnitTests/Microsoft.Testing.Platform.UnitTests/Helpers/DeadlineHelperTests.cs:1

• This new C# file is missing the UTF-8 BOM required for all C# files by .editorconfig:65-68; the other new C# files include it. Add the BOM so repository encoding checks do not fail.
  src/Platform/Microsoft.Testing.Platform/Helpers/ExitCodes.cs:87
• Every other exit-code member has an XML summary, but the newly introduced code has none. Document that code 15 represents execution stopped early for an approaching deadline so generated/internal API documentation remains complete.
  src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsExitCode.cs:63
• The new exit-code reason is not covered by GitHubActionsExitCodeTests, even though that suite explicitly verifies the neighboring known mappings for codes 9, 13, and 14. Add code 15 to the known-code assertions so a missing resource/mapping cannot silently fall back to the generic reason.

            (int)ExitCode.TestExecutionStoppedAtDeadline => GitHubActionsResources.ExitCodeReasonTestExecutionStoppedAtDeadline,

[Copilot]

Pull request overview

Copilot reviewed 73 out of 73 changed files in this pull request and generated 1 comment.

Suppressed comments (6)

src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs:280

• Task.Yield() opens a window after the timer has passed both completion checks. If the framework invoker returns during that yield, the host sets IsTestExecutionCompleted, but this handler resumes and still marks the run deadline-triggered and requests a stop, so a fully completed run can incorrectly exit with code 15. Re-checking alone still leaves a check/mark race; coordinate completion and deadline admission atomically in the policy service before emitting diagnostics or invoking the capability.
  test/UnitTests/Microsoft.Testing.Platform.UnitTests/Helpers/DeadlineHelperTests.cs:1
• This new C# file is UTF-8 without the required BOM; .editorconfig:66-67 requires charset = utf-8-bom for all C# files. Re-save it as UTF-8 with BOM so repository encoding checks remain consistent.
  src/Platform/Microsoft.Testing.Platform/Helpers/DeadlineHelper.cs:35
• DateTimeOffset.TryParse accepts many invariant-culture formats that are not ISO 8601, such as slash-separated dates. That contradicts the environment-variable contract and can silently schedule a deadline for a malformed CI value instead of logging the configuration warning. Validate against the explicitly supported ISO 8601 forms before accepting the instant.
  src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs:494
• This five-second query runs inside TakeDumpAsync, which is called sequentially for every process in the tree. A wedged host therefore costs up to five seconds per process before each dump starts; with the default 30-second margin, six processes can consume the entire margin in repeated requests for the same test list. Query once per dump operation (or skip this optional metadata on the deadline path) so the timeout is bounded for the whole tree, not per process.

                using var queryCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
                queryCts.CancelAfter(InProgressTestsQueryTimeout);
                GetInProgressTestsResponse tests = await _namedPipeClient.RequestReplyAsync<GetInProgressTestsRequest, GetInProgressTestsResponse>(new GetInProgressTestsRequest(), queryCts.Token).ConfigureAwait(false);

src/Platform/Microsoft.Testing.Platform/Helpers/ExitCodes.cs:87

• Every other exit-code member documents its externally observable meaning, but the new code 15 has no XML summary. Add the corresponding summary so generated/internal API documentation explains this outcome consistently.
  src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsExitCode.cs:63
• The existing GitHubActionsExitCodeTests.GetReason_ForKnownCode_MentionsRelevantOption covers the neighboring known-code mappings, but no test exercises the new code 15 branch. Add an assertion that code 15 returns the deadline-specific reason so a missing resource mapping cannot silently fall back to the generic message.

            (int)ExitCode.TestExecutionStoppedAtDeadline => GitHubActionsResources.ExitCodeReasonTestExecutionStoppedAtDeadline,

[Copilot]

Pull request overview

Copilot reviewed 73 out of 73 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs:323

• The completion gate and deadline outcome are not committed atomically. The timer can pass both checks in OnDeadlineReached, yield here, and then the host can call NotifyTestExecutionCompleted() before this line sets IsDeadlineTriggered; a framework whose stop method succeeds as a no-op will make an already-completed run exit with code 15. The flag is also externally visible while StopTestExecutionAsync may still fault and roll it back, so exit-code computation can observe the transient value. Use a synchronized/atomic policy-state transition that arbitrates completion versus deadline claiming, and prevent exit-code finalization from observing an unaccepted stop.

[Copilot]

Pull request overview

Copilot reviewed 73 out of 73 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs:257

• When the host wedges before connecting, the deadline task can dump and kill it, but these handshake waits are not canceled by that task and can remain blocked for DefaultHangTimeSpanTimeout (five minutes). The controller lifecycle awaits OnTestHostProcessStartedAsync before it starts WaitForExitAsync/OnTestHostProcessExitedAsync (TestHostControllersTestHost.ProcessLifecycle.cs:114-123), so the dump artifact is not published before the CI deadline. Cancel/race the handshake against the deadline dump task so this method returns as soon as that path wins.

            await _logger.LogDebugAsync($"Wait for test host connection to the server pipe '{_singleConnectionNamedPipeServer.PipeName.Name}'").ConfigureAwait(false);
            await _waitConnectionTask.TimeoutAfterAsync(TimeoutHelper.DefaultHangTimeSpanTimeout).ConfigureAwait(false);
            using CancellationTokenSource timeout = new(TimeoutHelper.DefaultHangTimeSpanTimeout);
            using var linkedCancellationToken = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeout.Token);
            _waitConsumerPipeName.Wait(linkedCancellationToken.Token);

[Jakub Jareš (nohwnd)]

You are right about the gap, and I checked it rather than taking it on trust: DotnetTestConnection.IsServerMode is _transportClient?.IsConnected == true, and the SDK launches the app with --server dotnettestcli, so on a dotnet test run this condition is true and AbortAtDeadlineExtension is never registered. The feature is inert on exactly the path that matters most.

The skip itself is not accidental — BuildTestFrameworkAsync runs per request in server mode, so registering there re-arms the timer against an already-past deadline and fires on every subsequent request. That is why it is written this way, and the comment says so.

So the two halves are both true: the current placement is wrong for dotnet test, and it cannot simply be deleted. Owning the deadline at server/application scope and applying it to the active run request is the right shape, and it is a design change to the prototype rather than a patch to this line, so I want to do it deliberately and with tests for the per-request case. Leaving this thread open as the marker for that.

[Copilot]

Pull request overview

Copilot reviewed 75 out of 75 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs:479

• This diagnostic is still on the path before QueryOnceAndDumpTreeAsync. If the logger provider throws, control jumps directly to finally, killing the process tree without creating any dump. Make all diagnostics in this block best-effort so they cannot bypass the dump operation.

            await _logger.LogInformationAsync($"{dumpReason}.").ConfigureAwait(false);

src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs:711

• This recovery path logs through the same potentially failing logger without guarding it. If annotation writing failed because a provider is broken, this call can throw and skip the actual dump below, despite the comment promising metadata cannot block dump creation. Use the existing best-effort logger helper here.

                await _logger.LogDebugAsync($"Could not write the in-progress tests next to the dump of process {process.Id}. Continuing with the dump. {ex}").ConfigureAwait(false);