perf: include every pattern in benchmark matrix#334
Merged
Conversation
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Contributor
Test Results 1 files 1 suites 2m 2s ⏱️ Results for commit 6d19507. ♻️ This comment has been updated with latest results. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR expands the benchmark suite to include a complete “coverage matrix” that enumerates (a) every production-readiness catalog pattern and (b) every *Generator.cs source file, and adds production-readiness tests + docs updates to keep that matrix complete over time.
Changes:
- Added BenchmarkDotNet “matrix” benchmarks for pattern coverage and generator source coverage.
- Added TinyBDD production-readiness tests that validate the benchmark coverage inventories against the pattern catalog and generator source tree.
- Updated benchmark documentation and adjusted the benchmarks project to multi-target
net8.0;net9.0;net10.0.
Reviewed changes
Copilot reviewed 7 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/PatternKit.Examples.Tests/ProductionReadiness/PatternKitBenchmarkCoverageTests.cs | Adds production-readiness tests asserting benchmark coverage for catalog patterns and generator sources. |
| test/PatternKit.Examples.Tests/PatternKit.Examples.Tests.csproj | Links benchmark coverage source files into the test project for validation. |
| docs/guides/benchmarks.md | Documents the new full coverage matrix and its test-backed enforcement. |
| benchmarks/PatternKit.Benchmarks/README.md | Clarifies expectations for pattern-specific scenario benchmarks vs. coverage matrix benchmarks. |
| benchmarks/PatternKit.Benchmarks/PatternKitBenchmarkConfig.cs | Renames the BenchmarkDotNet job id to reflect “current” TFM execution. |
| benchmarks/PatternKit.Benchmarks/PatternKit.Benchmarks.csproj | Multi-targets benchmarks across net8/net9/net10 and references PatternKit.Examples for catalog-driven coverage. |
| benchmarks/PatternKit.Benchmarks/packages.lock.json | Updates lock file to reflect new targeting/dependency graph. |
| benchmarks/PatternKit.Benchmarks/Coverage/PatternMatrixBenchmarks.cs | Adds BenchmarkDotNet coverage benchmarks for fluent/generated construction/execution routes. |
| benchmarks/PatternKit.Benchmarks/Coverage/PatternBenchmarkCoverage.cs | Introduces catalog-driven route enumeration for pattern benchmark coverage. |
| benchmarks/PatternKit.Benchmarks/Coverage/GeneratorMatrixBenchmarks.cs | Adds BenchmarkDotNet coverage benchmarks enumerating generator source files. |
| benchmarks/PatternKit.Benchmarks/Coverage/GeneratorBenchmarkCoverage.cs | Introduces generator-source enumeration used by generator coverage benchmarks. |
| benchmarks/PatternKit.Benchmarks/Coverage/BenchmarkRoute.cs | Defines route/phase enums and route record types used by coverage matrix benchmarks. |
| benchmarks/PatternKit.Benchmarks/Coverage/BenchmarkRepository.cs | Adds repo-root discovery helper (via PatternKit.slnx) for benchmarks. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| implementation.ExampleDocumentationPath)); | ||
| } | ||
|
|
||
| return routes; |
Comment on lines
14
to
17
| public static IConfig Create() | ||
| => ManualConfig.Create(DefaultConfig.Instance) | ||
| .AddJob(Job.Default.WithId("net10.0")) | ||
| .AddJob(Job.Default.WithId("current-tfm")) | ||
| .AddDiagnoser(MemoryDiagnoser.Default) |
Comment on lines
+74
to
+86
| private static string FindRepoRoot() | ||
| { | ||
| var directory = new DirectoryInfo(AppContext.BaseDirectory); | ||
| while (directory is not null) | ||
| { | ||
| if (File.Exists(Path.Combine(directory.FullName, "PatternKit.slnx"))) | ||
| return directory.FullName; | ||
|
|
||
| directory = directory.Parent; | ||
| } | ||
|
|
||
| throw new DirectoryNotFoundException("Could not find PatternKit repository root."); | ||
| } |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #334 +/- ##
==========================================
+ Coverage 89.66% 95.65% +5.98%
==========================================
Files 484 484
Lines 39951 39951
Branches 5744 5744
==========================================
+ Hits 35823 38214 +2391
+ Misses 1873 1737 -136
+ Partials 2255 0 -2255
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Contributor
🔍 PR Validation ResultsVersion: `` ✅ Validation Steps
📊 ArtifactsDry-run artifacts have been uploaded and will be available for 7 days. This comment was automatically generated by the PR validation workflow. |
JerrettDavis
force-pushed
the
benchmarkdotnet-complete-pattern-matrix
branch
from
8f70f23 to
59994e1
Compare
JerrettDavis
force-pushed
the
benchmarkdotnet-complete-pattern-matrix
branch
from
59994e1 to
8f721e2
Compare
Comment on lines
+74
to
+86
| private static string FindRepoRoot() | ||
| { | ||
| var directory = new DirectoryInfo(AppContext.BaseDirectory); | ||
| while (directory is not null) | ||
| { | ||
| if (File.Exists(Path.Combine(directory.FullName, "PatternKit.slnx"))) | ||
| return directory.FullName; | ||
|
|
||
| directory = directory.Parent; | ||
| } | ||
|
|
||
| throw new DirectoryNotFoundException("Could not find PatternKit repository root."); | ||
| } |
Comment on lines
14
to
17
| public static IConfig Create() | ||
| => ManualConfig.Create(DefaultConfig.Instance) | ||
| .AddJob(Job.Default.WithId("net10.0")) | ||
| .AddJob(Job.Default.WithId("current-tfm")) | ||
| .AddDiagnoser(MemoryDiagnoser.Default) |
| <PropertyGroup> | ||
| <OutputType>Exe</OutputType> | ||
| <TargetFramework>net10.0</TargetFramework> | ||
| <TargetFrameworks>net8.0;net9.0;net10.0</TargetFrameworks> |
Comment on lines
19
to
+21
| Benchmark output should be reviewed as part of pattern hardening work. When a pattern has both fluent and generated APIs, the benchmark must include both routes with categories for the pattern family, pattern name, route, and phase. | ||
|
|
||
| The benchmark suite also includes coverage matrix benchmarks for every pattern in the production-readiness catalog and every source generator under `src/PatternKit.Generators`. Those matrix benchmarks are validated by TinyBDD tests so a new pattern or generator cannot be added without appearing in BenchmarkDotNet output. |
Comment on lines
5
to
15
| Run the full benchmark suite from the repository root: | ||
|
|
||
| ```powershell | ||
| dotnet run -c Release --project benchmarks/PatternKit.Benchmarks -- --artifacts artifacts/benchmarks | ||
| ``` | ||
|
|
||
| Run one pattern family: | ||
|
|
||
| ```powershell | ||
| dotnet run -c Release --project benchmarks/PatternKit.Benchmarks -- --filter *LeaderElection* --artifacts artifacts/benchmarks | ||
| ``` |
Comment on lines
+23
to
+27
| private static IReadOnlyList<PatternBenchmarkRoute> CreateRoutes() | ||
| { | ||
| var catalog = new PatternKitPatternCatalog(); | ||
| var routes = new List<PatternBenchmarkRoute>(catalog.Patterns.Count * 4); | ||
|
|
JerrettDavis
force-pushed
the
benchmarkdotnet-complete-pattern-matrix
branch
from
8f721e2 to
6d19507
Compare
Contributor
Code Coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary\n- add BenchmarkDotNet coverage matrix benchmarks for every PatternKit production catalog pattern\n- include fluent construction, fluent execution, generated construction, and generated execution routes for each catalog pattern\n- add generator matrix benchmarks that enumerate every *Generator.cs source file\n- add TinyBDD production-readiness tests that fail when catalog patterns or generator sources are missing from the benchmark matrix\n- update benchmark docs to describe the complete matrix\n\n## Validation\n- dotnet restore benchmarks/PatternKit.Benchmarks/PatternKit.Benchmarks.csproj\n- dotnet restore test/PatternKit.Examples.Tests/PatternKit.Examples.Tests.csproj\n- git diff --check\n\nLocal test/build is still blocked by the known Windows Roslyn analyzer/compiler mismatch in PatternKit.Examples; PR CI is the authoritative generator-backed validation.