Document RepeatAttribute threshold features (NUnit 5, issue #5220)

docs/articles/nunit/writing-tests/attributes/repeat.md

@@ -23,9 +23,15 @@ RepeatAttribute(int count, bool stopOnFailure)
 | Property | Type | Description | Default |
 |----------|------|-------------|---------|
 | `StopOnFailure` | `bool` | Whether to stop repeating when a test fails. When `false`, all repetitions run regardless of failures. | `true` |
+| `RequiredPassPercentage` | `int` | The minimum percentage of runs (1–100) that must pass for the test to succeed. When set below 100, `StopOnFailure` is treated as `false` (and must not be explicitly set to `true`); iterations run up to the repeat count unless `StopWhenOverallResultDetermined` is enabled. | `100` |
+| `StopWhenOverallResultDetermined` | `bool` | When `true` and `RequiredPassPercentage` is below 100, stops iterating as soon as the final outcome is determined — either because the pass threshold is already guaranteed or can no longer be reached. | `false` |
 
+-----
 > [!NOTE]
 > The `StopOnFailure` property was added in **NUnit 4.3.0**.
+-----
+> [!NOTE]
+> The `RequiredPassPercentage` and `StopWhenOverallResultDetermined` properties were added in **NUnit 5.0**.
 
 ## Applies To
 
@@ -43,13 +49,29 @@ RepeatAttribute(int count, bool stopOnFailure)
 
 [!code-csharp[RepeatWithFaultAttributeExample](~/snippets/Snippets.NUnit/Attributes/RepeatAttributeExample.cs#RepeatWithFaultAttributeExample)]
 
+### Pass Threshold
+
+When testing non-deterministic systems (such as LLM-based tests or flaky integrations), you can allow a percentage of runs to fail. The test passes as long as at least `RequiredPassPercentage` percent of the runs succeed.
+
+[!code-csharp[RepeatWithPassThresholdExample](~/snippets/Snippets.NUnit/Attributes/RepeatAttributeExample.cs#RepeatWithPassThresholdExample)]
+
+### Early Stop When Outcome Is Determined
+
+When `StopWhenOverallResultDetermined` is `true`, NUnit stops iterating as soon as it knows whether the threshold will be met or missed, saving unnecessary test runs.
+
+[!code-csharp[RepeatWithStopWhenDeterminedExample](~/snippets/Snippets.NUnit/Attributes/RepeatAttributeExample.cs#RepeatWithStopWhenDeterminedExample)]
+
 ## Notes
 
 1. When `StopOnFailure` is `true` (default), the test stops at the first failure.
 2. When `StopOnFailure` is `false`, all repetitions run and all failures are collected.
-3. If `RepeatAttribute` is used on a parameterized method, each individual test case is repeated.
-4. It is not currently possible to use `RepeatAttribute` on a `TestFixture` or any higher level suite. Only test methods may be repeated.
+3. When `RequiredPassPercentage` is set below 100, `StopOnFailure` must not be explicitly set to `true` — combining the two is a configuration error that produces `ResultState.Error` without running the test. Setting `StopOnFailure=false` explicitly alongside a threshold is valid. When `StopOnFailure` is left at its default (not explicitly set), it is silently treated as `false` so repetitions continue after failures (unless `StopWhenOverallResultDetermined` is enabled).
+4. When `RequiredPassPercentage` is below 100 and `StopWhenOverallResultDetermined` is `true`, NUnit exits the loop as soon as the result is certain:
+   - **Early success**: the number of passes already guarantees the threshold regardless of the remaining runs.
+   - **Early failure**: even if all remaining runs pass, the threshold can no longer be reached.
+5. If `RepeatAttribute` is used on a parameterized method, each individual test case is repeated independently.
+6. It is not currently possible to use `RepeatAttribute` on a `TestFixture` or any higher level suite. Only test methods may be repeated.
 
 ## See Also
 
-* [Retry Attribute](xref:attribute-retry)
+- [Retry Attribute](xref:attribute-retry)

docs/snippets/Snippets.NUnit/Attributes/RepeatAttributeExample.cs

@@ -1,4 +1,4 @@
-using NUnit.Framework;
+using NUnit.Framework;
 
 namespace Snippets.NUnit.Attributes;
 
@@ -29,7 +29,7 @@ public void TestMethod2()
     #region RepeatWithFaultAttributeExample
 
     private int count2;
-    
+
     [Test,Explicit]  // Marking the test as Explicit to avoid failing our doc build. You can skip this.
     [Repeat(5, StopOnFailure = false)]
     public void TestMethod3()
@@ -38,4 +38,33 @@ public void TestMethod3()
         Assert.That(count2, Is.Not.EqualTo(3)); // Intentional failure on 3rd iteration
     }
     #endregion
-}
+
+#if NUNIT_REPEAT_THRESHOLD
+    #region RepeatWithPassThresholdExample
+    // The test passes if at least 80% of the 10 runs succeed.
+    // Useful for non-deterministic systems where some variation is acceptable.
+    [Test]
+    [Repeat(10, RequiredPassPercentage = 80)]
+    public void NonDeterministicTest()
+    {
+        bool result = CallFlakyService();
+        Assert.That(result, Is.True);
+    }
+
+    private bool CallFlakyService() => true; // placeholder
+    #endregion
+
+    #region RepeatWithStopWhenDeterminedExample
+    // Stops as soon as NUnit can guarantee pass or failure:
+    //  - Early success: enough passes accumulated (threshold already guaranteed).
+    //  - Early failure: too many failures (threshold no longer achievable).
+    [Test]
+    [Repeat(10, RequiredPassPercentage = 80, StopWhenOverallResultDetermined = true)]
+    public void NonDeterministicTestWithEarlyStop()
+    {
+        bool result = CallFlakyService();
+        Assert.That(result, Is.True);
+    }
+    #endregion
+#endif
+}

docs/snippets/Snippets.NUnit/Snippets.NUnit.csproj

@@ -10,8 +10,8 @@
 
     <ItemGroup>
         <PackageReference Include="Microsoft.NET.Test.Sdk" Version="18.5.1" />
-        <PackageReference Include="NUnit" Version="5.0.0-alpha.100.9" />
-        <PackageReference Include="NUnit3TestAdapter" Version="6.2.0" />
+        <PackageReference Include="NUnit" Version="5.0.0-alpha.100.24" />
+        <PackageReference Include="NUnit3TestAdapter" Version="6.3.0-alpha.0.19" />
         <PackageReference Include="NUnit.Analyzers" Version="4.14.0">
             <PrivateAssets>all</PrivateAssets>
             <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>