Feature/mtp test adapter 2803 - #3229
Conversation
Added a new guide for running benchmarks with Microsoft.Testing.Platform (MTP), covering setup, usage, and caveats. Updated the table of contents to include the new page and added a note to the VSTest docs about the MTP adapter option.
Added InternalsVisibleTo attribute in AssemblyInfo.cs to expose internal members to the BenchmarkDotNet.TestAdapter.TestingPlatform assembly, ensuring it uses the same public key as other related assemblies.
Deleted the internal static method GetUnrandomizedJobDisplayInfo from BenchmarkCaseExtensions.cs. This method handled normalization of job display info by removing randomness from job IDs for consistent benchmark referencing. No other code changes were made.
Introduced BenchmarkCaseIdentityExtensions with GetUnrandomizedJobDisplayInfo to normalize Job DisplayInfo by removing random ID components. This ensures consistent benchmark identification across processes for test adapters.
Introduce GetBenchmarksFromAssembly to extract benchmarks from an already loaded Assembly. Refactor existing logic to use this method, improving code reuse and enabling benchmark retrieval from both loaded assemblies and file paths.
Add MSBuild props to enable TestingPlatform integration, set defaults for `dotnet test` compatibility, disable parallel TFM runs, and auto-register BenchmarkDotNet builder hook.
Introduced AsyncWorkQueue, an internal sealed class in BenchmarkDotNet.TestAdapter.TestingPlatform. It enables ordered, thread-safe queuing of asynchronous work items, allowing synchronous producers and asynchronous consumers. Utilizes ConcurrentQueue and SemaphoreSlim, supports completion signaling, and implements IDisposable for resource cleanup.
Created a new .csproj targeting netstandard2.0 for the TestingPlatform adapter. Configured project metadata, packaging, and references. Integrated Microsoft.Testing.Platform.MSBuild and BenchmarkDotNet, and linked shared source files for benchmark enumeration. Set IsTestingPlatformApplication to false to avoid test app behavior.
Introduced BenchmarkDotNetExtension class implementing IExtension to provide extension metadata and enablement status for Microsoft.Testing.Platform integration.
Introduced BenchmarkEventProcessor to process BenchmarkDotNet events and translate them into test node updates for the testing platform. Handles validation errors, build results, benchmark execution, and ensures all benchmarks have published results. Includes logic for error aggregation, output formatting, and timing information.
Introduce BenchmarkTestFramework to integrate BenchmarkDotNet with Microsoft.Testing.Platform, enabling benchmark discovery and execution. Implements session management, filtering, event processing, and output routing. Handles test node updates and cancellation, with support for experimental platform features.
Introduced the internal sealed class BenchmarkTestNode to encapsulate immutable BenchmarkCase data for Microsoft.Testing.Platform integration. This includes stable UID generation, display name and path construction, property management, and support for test filtering and message bus conversion.
Introduced OutputDeviceLogger class implementing ILogger to forward BenchmarkDotNet logs to the platform output device. Handles log kinds, buffers lines, and asynchronously displays output to ensure build progress and results are visible in test run output.
Introduce TestApplicationBuilderExtensions with AddBenchmarkDotNet methods for integrating BenchmarkDotNet benchmarks into Microsoft.Testing.Platform. Includes overloads for entry assembly and specific assemblies, null checks, test framework registration, and tree node filter service support.
Introduced a static TestingPlatformBuilderHook class in the BenchmarkDotNet.TestAdapter.TestingPlatform namespace. This class provides an AddExtensions method to register BenchmarkDotNet with the test application builder, intended for use by generated code and hidden from IntelliSense.
Added a "test" section to global.json to specify "Microsoft.Testing.Platform" as the test runner. This configures the project to use the designated testing platform.
Introduce BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj targeting net10.0 as an executable. The project includes assembly metadata, references BenchmarkDotNet.TestAdapter.TestingPlatform, manually imports its build props, and uses shared common.props and common.targets for build configuration.
Introduced SampleBenchmarks class in BenchmarkDotNet.IntegrationTests.TestingPlatform. Defines Add and Multiply benchmarks with parameterized Size, categorized as "Fast" and "Slow". Uses a custom FastConfig to run benchmarks in-process with a single dry iteration for quick end-to-end testing.
Added BenchmarkDotNet.TestAdapter.TestingPlatform and BenchmarkDotNet.IntegrationTests.TestingPlatform projects to the solution. Updated MonoBenchmarks and SharedDiagnosers integration test projects to only build for the solution in Debug configuration.
Explicitly set BenchmarkDotNet.TestAdapter.TestingPlatform and BenchmarkDotNet.IntegrationTests.TestingPlatform to not build in the Debug configuration by adding <Build Solution="Debug|*" Project="false" /> in BenchmarkDotNet.slnx. No other changes made.
|
Let's name it
Why? We run tests in Release configuration. |
| /// The job is always part of the uid, otherwise two cases of the same benchmark that only differ by job would | ||
| /// collide. The parameters are already part of the method name. | ||
| /// </remarks> | ||
| public static string GetUid(BenchmarkCase benchmarkCase) |
There was a problem hiding this comment.
I though GetUid logics should be implemented on BenchmarkDotNet core project side.
Because --filter-uid option is useful for normal benchmark exe project without MTP.
I've implemented MSTest based UID generation logics on #3227.
Is it able to confirm these logics can be shared with TestAdapter?
There was a problem hiding this comment.
This has been noted and taken into consideration. I have done the fix
There was a problem hiding this comment.
#3227 is merged to master.
So GUID based UID generator is available.
|
|
||
| var properties = new List<IProperty> | ||
| { | ||
| new TestMethodIdentifierProperty( |
There was a problem hiding this comment.
Following generic benchmarks are not shown correctly on VS Test Explorer.
[InProcess]
[GenericTypeArguments(typeof(int))]
[GenericTypeArguments(typeof(int?))]
[GenericTypeArguments(typeof(int[]))]
[GenericTypeArguments(typeof(int?[]))]
[GenericTypeArguments(typeof(int[,]))]
[GenericTypeArguments(typeof(int?[,]))]
public class GenericTypeBenchmarks<T>
{
[Benchmark]
public void Benchmark() { }
}
I though TestMethodIdentifier's property require ECMA-335 compliant type names.
https://learn.microsoft.com/en/dotnet/api/microsoft.testing.platform.extensions.messages.testmethodidentifierproperty
xUnit.net example.
https://github.com/xunit/xunit/blob/rel/4.0.0/src/xunit.v3.common/Extensions/ReflectionExtensions.cs#L171
There was a problem hiding this comment.
It looks like generics are still not displayed as expected.
- Correct NuGet package and namespace in documentation - Add GetBenchmarkUid for stable benchmark identification - Change namespace in BenchmarkCaseIdentityExtensions - Update InternalsVisibleTo for TestingPlatform assembly
Deleted all source, project, and props files from BenchmarkDotNet.TestAdapter.TestingPlatform. This removes all implementation and integration for running benchmarks as tests via Microsoft.Testing.Platform, including test discovery, execution, and result processing logic.
Add BenchmarkDotNet.TestingPlatform.props to enable seamless integration with Microsoft.Testing.Platform. This includes setting required properties for Testing Platform application behavior, ensuring `dotnet test` compatibility on older SDKs, disabling parallel test execution for multi-targeted projects by default, and registering BenchmarkDotNet as a builder hook.
Introduced AsyncWorkQueue in BenchmarkDotNet.TestingPlatform to enable thread-safe, ordered queuing of asynchronous work items. Supports synchronous enqueuing, asynchronous draining, completion signaling, and resource disposal using ConcurrentQueue and SemaphoreSlim.
Introduce a new project to integrate BenchmarkDotNet with Microsoft.Testing.Platform, enabling benchmarks to be discovered and executed as tests. Implements extension identification, test framework, event processing, test node representation, and output logging. Provides builder extensions for easy registration and an MSBuild hook for automatic integration. Updates project configuration for packaging and dependencies.
Updated BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj to reference BenchmarkDotNet.TestingPlatform instead of BenchmarkDotNet.TestAdapter.TestingPlatform. Adjusted both the ProjectReference and Import paths accordingly.
Introduced a lock (buildCompleteGate) to synchronize build failure handling in BenchmarkEventProcessor. This prevents race conditions when multiple threads report build completion concurrently, ensuring safe execution in parallel build scenarios. The failure handling logic remains unchanged.
Updated the XML documentation for GetFilterableProperties to specify --treenode-filter as the correct command-line argument, replacing the inaccurate --filter reference. This improves the accuracy of usage instructions.
Updated the comment describing the tree node filter to reference the correct command-line option, `--treenode-filter`, instead of the outdated `--filter`. No functional changes were made; this is a documentation clarification.
|
@timcassell / @filzrev the last 4 issues raised have been reviewed and addressed:
|
|
Reviewed The adapter code holds up. The concurrency reasoning checks out against the runner rather than just the comments: Two things worth raising. 1. The new integration project asserts nothing, and never builds a benchmark. All three probes pin 2. Neither in-repo consumer exercises the import order The Minor: Confirmed fixed since the last round: Reviewed by Claude Opus 5 via Claude Code. |
Added a new step in run-tests.yaml to run a PowerShell smoke test on the packed BenchmarkDotNet.TestAdapter after the 'pack' task. This ensures correct NuGet build file import order, since nothing in the solution currently consumes the package directly.
Added ConsumedBenchmark.cs with a simple benchmark using BenchmarkDotNet and a custom fast config. Introduced TestAdapterConsumer.csproj targeting .NET 10.0, referencing BenchmarkDotNet.TestAdapter from a local artifacts source for smoke testing real-world usage.
A new BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures.csproj was added targeting .NET 10.0 as an executable. The project includes assembly metadata, enforces code optimization for consistent benchmarks, and references BenchmarkDotNet.TestAdapter with manual imports of its .props and .targets files. Common build property and target files are also imported to ensure correct MSBuild behavior.
Added SeparatorProbe class in BenchmarkDotNet.IntegrationTests.TestingPlatform. This benchmark uses a parameter with '/' as a tree separator, includes a Length() method, and applies a custom FastConfig with a dry job and InProcessEmitToolchain for faster execution.
Introduce BuildFailureProbe in BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures. This class uses a custom toolchain (FailingBuildConfig) with a NoopGenerator, FailingBuilder, and UnreachableExecutor to reliably simulate build failures for adapter testing, without relying on uncompilable code.
Added CollisionProbe class to test BenchmarkDotNet's behavior when benchmark parameters have identical string representations, using a custom Ambiguous type. Ensures the adapter reports collisions instead of running ambiguous benchmarks.
Added OutOfProcessProbe class in BenchmarkDotNet.IntegrationTests.TestingPlatform with an Add() benchmark method. Configured to run out-of-process using a custom OutOfProcessConfig and Job.Dry to ensure a real build/execute cycle for adapter testing, unlike in-process probes.
Added conditional project references to BenchmarkDotNet.IntegrationTests.TestingPlatform and .Failures in BenchmarkDotNet.IntegrationTests.csproj. These are included only for .NETCoreApp targets with ReferenceOutputAssembly set to false, ensuring correct build order for probe apps used in TestingPlatformAdapterTests.
Added TestingPlatformAdapterTests (under #if NETCOREAPP) using BenchmarkDotNet to perform integration tests on Microsoft.Testing.Platform probe apps. Tests cover benchmark discovery, UID consistency, filtering, build/run behavior, and error reporting by running probe apps as separate processes and asserting on their output. Introduced helper methods for process execution, output parsing, and result summarization.
Updated the Escape method in BenchmarkTestNode.cs to percent-encode '/' as '%2F' and '%' as '%25'. This prevents path segmentation issues in Microsoft.Testing.Platform, ensuring correct tree structure and benchmark addressability. Filters must now use '%2F' instead of '/'.
Updated documentation to clarify that benchmark parameter values containing slashes (/) or percent signs (%) are percent-encoded in the tree node filter path (e.g., a/b as a%2Fb, % as %25). This encoding applies only to the filter, not to the displayed benchmark name.
Added test-adapter-consumer.ps1 to perform smoke tests on the packed BenchmarkDotNet.TestAdapter NuGet package. The script restores and builds a consumer project, checks MSBuild property resolutions, and verifies benchmark discovery to ensure correct adapter behavior when used as a package. Includes detailed comments, parameter handling, and error checking.
Add <Optimize>true</Optimize> to ensure the assembly is always built with optimizations, preventing BenchmarkEnumerator from hiding out-of-process benchmarks in non-Release builds. Expand comments to clarify manual build file imports and MSBuild processing order.
Added a section to README.md describing the BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures project. The documentation explains its purpose as a collection of intentionally failing benchmarks for testing BenchmarkDotNet.TestAdapter error handling, including UID collision and build failure mapping. It also clarifies the relationship with TestingPlatformAdapterTests and the location of passing benchmarks.
Added BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures to the solution file, ensuring it is included with other integration test projects.
|
@timcassell the issues have been addressed |
timcassell
left a comment
timcassell
left a comment
There was a problem hiding this comment.
Five findings. The async orchestration in RunAsync (channel drain, cancellation, TryComplete in finally) and the event to TestNode mapping look right.
Reviewed by Claude (Opus 5), posted by @timcassell.
| <None Include="build\BenchmarkDotNet.TestAdapter.props" Pack="true" PackagePath="build;buildTransitive" /> | ||
| <None Include="build\BenchmarkDotNet.TestAdapter.targets" Pack="true" PackagePath="build;buildTransitive" /> |
There was a problem hiding this comment.
Packing the props/targets into buildTransitive (plus Microsoft.Testing.Platform.MSBuild flowing without PrivateAssets) makes these reach transitive consumers, not just the benchmark project.
MyApp (a normal exe) -> shared benchmark library -> BenchmarkDotNet.TestAdapter: MyApp now imports BenchmarkDotNet.TestAdapter.targets, hits the IsTestingPlatformApplication == '' fallback, gets GenerateProgramFile=false, and MTP.MSBuild compiles a second entry point into it (CS0017) - or, with StartupObject set, it silently becomes a dotnet test target. Before this PR the props were build-only, so only direct consumers were affected.
If the goal here was only to fix the trailing-separator/NU5129 issue, PackagePath="build" alone does that. The smoke test only covers a direct PackageReference.
| var matches = new List<List<Match>>(); | ||
| var matchesByUid = new Dictionary<string, List<Match>>(StringComparer.Ordinal); | ||
|
|
||
| foreach (var runInfo in BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly)) |
There was a problem hiding this comment.
The BenchmarkRunInfos from GetBenchmarksFromAssembly are never disposed. BenchmarkRunInfo.Dispose() -> BenchmarkCase.Dispose() -> ParameterInstance.Dispose() disposes IDisposable parameter values (added in #1383 for parameters with locking finalizers).
DiscoverAsync disposes nothing at all, and RunAsync only reaches the selected cases, via the run infos BenchmarkRunnerClean disposes in its finally - every filtered-out case leaks. So --list-tests, or running one case of a [ParamsSource] set, leaves disposable parameter values undisposed.
| private async Task PublishCollisionAsync(ExecuteRequestContext context, SessionUid sessionUid, List<Match> collision) | ||
| { | ||
| var node = collision[0].Node; | ||
| var error = |
There was a problem hiding this comment.
The message names one cause, but the uid also hashes Descriptor.DisplayInfo (TypeInfo.WorkloadMethodDisplayInfo, where the method part is the [Benchmark(Description = ...)] when one is set) and the job display info.
[Benchmark(Description = "Foo")] public int Bar() => 0;
[Benchmark] public int Foo() => 0;No parameters, still a collision - and the message sends the user off to fix ToString() on parameters that do not exist. Worth widening it to mention the description and the job.
| properties.Add(new TestFileLocationProperty(benchmarkAttribute.SourceCodeFile, new LinePositionSpan(position, position))); | ||
| } | ||
|
|
||
| foreach (var category in DefaultCategoryDiscoverer.Instance.GetCategories(benchmarkMethod)) |
There was a problem hiding this comment.
This should be benchmarkCase.Descriptor.Categories - that is what BenchmarkConverter already resolved using the config's ICategoryDiscoverer, and it is public.
With a custom ICategoryDiscoverer, BDN's own --anyCategories and the summary see the custom categories, but --treenode-filter "/*/*/*/*[Category=X]" matches nothing, because the node only carries the default-discovered ones. (The VSTest adapter has the same shape, but this is new code.)
| // Benchmarks that never reported a result still need one, unless the run was cancelled: | ||
| // the platform expects an OperationCanceledException in that case, and publishing results | ||
| // afterwards would contradict it. | ||
| if (!runCancellation.IsCancellationRequested) |
There was a problem hiding this comment.
Minor: OnStartRunBenchmark has already published InProgressTestNodeStateProperty for the benchmark that was executing, so skipping PublishOutstandingResults leaves that node with no terminal state - a server-mode consumer keeps showing it as running for the rest of the session.
MTP tolerates a missing terminal state on a cancelled request, so this may well be fine, but right now it falls out of the IsCancellationRequested gate rather than being a decision.
Updated the comment explaining the opt-out logic for IsTestingPlatformApplication to clarify that the property may already be set to true by the project or another package, making empty checks unreliable. The revised comment aligns with MSTest.TestAdapter.targets and improves accuracy. No functional code changes were made.
CategoryProbe.cs introduces a benchmark using a custom ICategoryDiscoverer to assign categories based on method names, ensuring custom categories are recognized by BenchmarkDotNet. DisposableProbe.cs adds a benchmark with disposable parameter values, tracking their creation and disposal, and writing a report on process exit to address potential runtime hangs from undisposed parameters. Both use custom ManualConfig for job and toolchain settings.
Expanded TestingPlatformAdapterTests with cases for custom categories, descriptions, and disposal of parameter values. Added tests for handling UID collisions in benchmarks. Introduced ReadDisposalReport helper and updated expected benchmark discovery list.
Introduced DescriptionCollisionProbe class with two benchmarks to test identity collisions when a benchmark's description matches another's method name. Added FastConfig for dry job using InProcessEmitToolchain.
- Update .csproj to pack .props/.targets only in build folder, preventing test platform settings from leaking to consumers - Refactor BenchmarkTestFramework.cs to track all enumerated benchmarks, dispose unused parameter values, and improve collision error messages - Add internal classes for grouping enumeration results and reference-based IDisposable comparison - Use resolved categories from descriptors in BenchmarkTestNode.cs for consistency with BenchmarkDotNet and custom discoverers
|
Please update the branch from master. |
Toolchain became abstract and gained a Runtime parameter in the toolchain rework merged from master, so the probe's fake toolchain is now a named subclass. UnknownRuntime is enough: the build always fails, so nothing ever resolves a default toolchain from it.
3 participants
#2803
@timcassell
Introduce BenchmarkTestFramework to integrate BenchmarkDotNet with Microsoft.Testing.Platform, enabling benchmark discovery and execution. Implements session management, filtering, event processing, and output routing. Handles test node updates and cancellation, with support for experimental platform features.
Added BenchmarkDotNet.TestAdapter.TestingPlatform and BenchmarkDotNet.IntegrationTests.TestingPlatform projects to the solution. Updated MonoBenchmarks and SharedDiagnosers integration test projects to only build for the solution in Debug configuration.