diff --git a/docs/articles/nunit/writing-tests/attributes/repeat.md b/docs/articles/nunit/writing-tests/attributes/repeat.md index 4a7343b95..660e9d74a 100644 --- a/docs/articles/nunit/writing-tests/attributes/repeat.md +++ b/docs/articles/nunit/writing-tests/attributes/repeat.md @@ -23,9 +23,15 @@ RepeatAttribute(int count, bool stopOnFailure) | Property | Type | Description | Default | |----------|------|-------------|---------| | `StopOnFailure` | `bool` | Whether to stop repeating when a test fails. When `false`, all repetitions run regardless of failures. | `true` | +| `RequiredPassPercentage` | `int` | The minimum percentage of runs (1–100) that must pass for the test to succeed. When set below 100, `StopOnFailure` is treated as `false` (and must not be explicitly set to `true`); iterations run up to the repeat count unless `StopWhenOverallResultDetermined` is enabled. | `100` | +| `StopWhenOverallResultDetermined` | `bool` | When `true` and `RequiredPassPercentage` is below 100, stops iterating as soon as the final outcome is determined — either because the pass threshold is already guaranteed or can no longer be reached. | `false` | +----- > [!NOTE] > The `StopOnFailure` property was added in **NUnit 4.3.0**. +----- +> [!NOTE] +> The `RequiredPassPercentage` and `StopWhenOverallResultDetermined` properties were added in **NUnit 5.0**. ## Applies To @@ -43,13 +49,29 @@ RepeatAttribute(int count, bool stopOnFailure) [!code-csharp[RepeatWithFaultAttributeExample](~/snippets/Snippets.NUnit/Attributes/RepeatAttributeExample.cs#RepeatWithFaultAttributeExample)] +### Pass Threshold + +When testing non-deterministic systems (such as LLM-based tests or flaky integrations), you can allow a percentage of runs to fail. The test passes as long as at least `RequiredPassPercentage` percent of the runs succeed. + +[!code-csharp[RepeatWithPassThresholdExample](~/snippets/Snippets.NUnit/Attributes/RepeatAttributeExample.cs#RepeatWithPassThresholdExample)] + +### Early Stop When Outcome Is Determined + +When `StopWhenOverallResultDetermined` is `true`, NUnit stops iterating as soon as it knows whether the threshold will be met or missed, saving unnecessary test runs. + +[!code-csharp[RepeatWithStopWhenDeterminedExample](~/snippets/Snippets.NUnit/Attributes/RepeatAttributeExample.cs#RepeatWithStopWhenDeterminedExample)] + ## Notes 1. When `StopOnFailure` is `true` (default), the test stops at the first failure. 2. When `StopOnFailure` is `false`, all repetitions run and all failures are collected. -3. If `RepeatAttribute` is used on a parameterized method, each individual test case is repeated. -4. It is not currently possible to use `RepeatAttribute` on a `TestFixture` or any higher level suite. Only test methods may be repeated. +3. When `RequiredPassPercentage` is set below 100, `StopOnFailure` must not be explicitly set to `true` — combining the two is a configuration error that produces `ResultState.Error` without running the test. Setting `StopOnFailure=false` explicitly alongside a threshold is valid. When `StopOnFailure` is left at its default (not explicitly set), it is silently treated as `false` so repetitions continue after failures (unless `StopWhenOverallResultDetermined` is enabled). +4. When `RequiredPassPercentage` is below 100 and `StopWhenOverallResultDetermined` is `true`, NUnit exits the loop as soon as the result is certain: + - **Early success**: the number of passes already guarantees the threshold regardless of the remaining runs. + - **Early failure**: even if all remaining runs pass, the threshold can no longer be reached. +5. If `RepeatAttribute` is used on a parameterized method, each individual test case is repeated independently. +6. It is not currently possible to use `RepeatAttribute` on a `TestFixture` or any higher level suite. Only test methods may be repeated. ## See Also -* [Retry Attribute](xref:attribute-retry) +- [Retry Attribute](xref:attribute-retry) diff --git a/docs/snippets/Snippets.NUnit/Attributes/RepeatAttributeExample.cs b/docs/snippets/Snippets.NUnit/Attributes/RepeatAttributeExample.cs index 9ca0532b8..e72cbc86f 100644 --- a/docs/snippets/Snippets.NUnit/Attributes/RepeatAttributeExample.cs +++ b/docs/snippets/Snippets.NUnit/Attributes/RepeatAttributeExample.cs @@ -1,4 +1,4 @@ -using NUnit.Framework; +using NUnit.Framework; namespace Snippets.NUnit.Attributes; @@ -29,7 +29,7 @@ public void TestMethod2() #region RepeatWithFaultAttributeExample private int count2; - + [Test,Explicit] // Marking the test as Explicit to avoid failing our doc build. You can skip this. [Repeat(5, StopOnFailure = false)] public void TestMethod3() @@ -38,4 +38,33 @@ public void TestMethod3() Assert.That(count2, Is.Not.EqualTo(3)); // Intentional failure on 3rd iteration } #endregion -} \ No newline at end of file + +#if NUNIT_REPEAT_THRESHOLD + #region RepeatWithPassThresholdExample + // The test passes if at least 80% of the 10 runs succeed. + // Useful for non-deterministic systems where some variation is acceptable. + [Test] + [Repeat(10, RequiredPassPercentage = 80)] + public void NonDeterministicTest() + { + bool result = CallFlakyService(); + Assert.That(result, Is.True); + } + + private bool CallFlakyService() => true; // placeholder + #endregion + + #region RepeatWithStopWhenDeterminedExample + // Stops as soon as NUnit can guarantee pass or failure: + // - Early success: enough passes accumulated (threshold already guaranteed). + // - Early failure: too many failures (threshold no longer achievable). + [Test] + [Repeat(10, RequiredPassPercentage = 80, StopWhenOverallResultDetermined = true)] + public void NonDeterministicTestWithEarlyStop() + { + bool result = CallFlakyService(); + Assert.That(result, Is.True); + } + #endregion +#endif +} diff --git a/docs/snippets/Snippets.NUnit/Snippets.NUnit.csproj b/docs/snippets/Snippets.NUnit/Snippets.NUnit.csproj index 1630375f0..950455782 100644 --- a/docs/snippets/Snippets.NUnit/Snippets.NUnit.csproj +++ b/docs/snippets/Snippets.NUnit/Snippets.NUnit.csproj @@ -10,8 +10,8 @@ - - + + all runtime; build; native; contentfiles; analyzers; buildtransitive