Skip to content

Correct sample standard deviation calculations #1191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dwcullop wants to merge 3 commits into
base: main
Choose a base branch
from
+213 −6
Open

Correct sample standard deviation calculations #1191

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
36ef6db
Correct sample standard deviation and restore numeric coverage
dwcullop Sep 21, 2026
ce90221
Document StdDev sample deviation contract
dwcullop Sep 21, 2026
932e503
Merge branch 'main' into u/dacullop/main/stddev-sample-calculation
dwcullop Sep 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .github/instructions/dynamicdata-cache.instructions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -820,6 +820,20 @@ Filters Update changes based on reference equality or a custom predicate. If fil

---

### StdDev

Computes sample standard deviation as `sqrt(sum((value - mean)^2) / (count - 1))`. The configured fallback is returned when a changeset leaves fewer than two items. Integer selectors retain fractional means and variances.

| Input | Behavior |
|-------|----------|
| **Add** | Includes the selected value and emits the updated result. |
| **Update** | Removes the previous selected value, includes the current value, and emits the updated result. |
| **Remove** | Removes the selected value and emits the updated result or fallback. |
| **Refresh / Moved** | Does not adjust the aggregate; the current result is emitted for the changeset. |
| **OnError / OnCompleted** | Forwards the terminal notification. |

In-place property mutations require recomputation, such as `InvalidateWhen`, rather than a Refresh alone.

### Property Observation

| Operator | Behavior |
Expand Down
14 changes: 14 additions & 0 deletions .github/instructions/dynamicdata-list.instructions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -498,6 +498,20 @@ myObservable.ToObservableChangeSet(expireAfter: item => TimeSpan.FromMinutes(5))

---

### StdDev

Computes sample standard deviation as `sqrt(sum((value - mean)^2) / (count - 1))`. The configured fallback is returned when a changeset leaves fewer than two items. Integer selectors retain fractional means and variances.

| Input | Behavior |
|-------|----------|
| **Add / AddRange** | Includes the selected values and emits the updated result. |
| **Replace** | Removes the previous selected value, includes the current value, and emits the updated result. |
| **Remove / RemoveRange / Clear** | Removes the selected values and emits the updated result or fallback. |
| **Refresh / Moved** | Does not adjust the aggregate; the current result is emitted for the changeset. |
| **OnError / OnCompleted** | Forwards the terminal notification. |

In-place property mutations require recomputation, such as `InvalidateWhen`, rather than a Refresh alone.

### Property Observation

```csharp
Expand Down
175 changes: 175 additions & 0 deletions src/DynamicData.Tests/AggregationTests/StdDevFixture.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
// Copyright (c) 2011-2026 Roland Pheasant. All rights reserved.
// Roland Pheasant licenses this file to you under the MIT license.
// See the LICENSE file in the project root for full license information.

using System;
using System.Linq;
using System.Reactive.Linq;

using Bogus;

using DynamicData.Aggregation;
using DynamicData.Tests.Utilities;

using FluentAssertions;

using Xunit;
using Xunit.Abstractions;

namespace DynamicData.Tests.AggregationTests;

/// <summary>Verifies sample standard deviation for the supported numeric overloads.</summary>
public sealed class StdDevFixture
{
private const int Seed = 0x2409_1071;

private readonly Randomizer _randomizer = new(Seed);

public StdDevFixture(ITestOutputHelper output)
=> output.WriteLine($"{nameof(StdDevFixture)} seed: {Seed:X8}");

/// <summary>Verifies that cache standard deviation applies the sample divisor to the variance.</summary>
[Theory]
[InlineData(nameof(Int32))]
[InlineData(nameof(Int64))]
[InlineData(nameof(Single))]
[InlineData(nameof(Double))]
[InlineData(nameof(Decimal))]
public void Cache_ThreeEquallySpacedValues_ReportsTheirSpacing(string numericType)
{
// Arrange
var center = _randomizer.Int(-100, 100);
var spacing = _randomizer.Int(1, 30);
var values = new[] { center - spacing, center, center + spacing };
var fallback = _randomizer.Int(1, 100);
using var source = new TestSourceCache<int, int>(static value => value);
var standardDeviation = numericType switch
{
nameof(Int32) => source.Connect().StdDev(static value => value, fallback),
nameof(Int64) => source.Connect().StdDev(static value => (long)value, fallback),
nameof(Single) => source.Connect().StdDev(static value => (float)value, fallback),
nameof(Double) => source.Connect().StdDev(static value => (double)value, fallback),
nameof(Decimal) => source.Connect().StdDev(static value => (decimal)value, fallback).Select(static value => (double)value),
_ => throw new ArgumentOutOfRangeException(nameof(numericType))
};
using var subscription = standardDeviation
.ValidateSynchronization()
.RecordValues(out var results);

// Act
source.AddOrUpdate(values);

// Assert
results.Error.Should().BeNull(because: "all values are within the supported numeric ranges");
results.RecordedValues.Should().ContainSingle(because: "one source edit produces one aggregate")
.Which.Should().Be(spacing, because: "the sample variance of three equally spaced values is the square of their spacing");
}

/// <summary>Verifies that list standard deviation applies the sample divisor to the variance.</summary>
[Theory]
[InlineData(nameof(Int32))]
[InlineData(nameof(Int64))]
[InlineData(nameof(Single))]
[InlineData(nameof(Double))]
[InlineData(nameof(Decimal))]
public void List_ThreeEquallySpacedValues_ReportsTheirSpacing(string numericType)
{
// Arrange
var center = _randomizer.Int(-100, 100);
var spacing = _randomizer.Int(1, 30);
var values = new[] { center - spacing, center, center + spacing };
var fallback = _randomizer.Int(1, 100);
using var source = new TestSourceList<int>();
var standardDeviation = numericType switch
{
nameof(Int32) => source.Connect().StdDev(static value => value, fallback),
nameof(Int64) => source.Connect().StdDev(static value => (long)value, fallback),
nameof(Single) => source.Connect().StdDev(static value => (float)value, fallback),
nameof(Double) => source.Connect().StdDev(static value => (double)value, fallback),
nameof(Decimal) => source.Connect().StdDev(static value => (decimal)value, fallback).Select(static value => (double)value),
_ => throw new ArgumentOutOfRangeException(nameof(numericType))
};
using var subscription = standardDeviation
.ValidateSynchronization()
.RecordValues(out var results);

// Act
source.AddRange(values);

// Assert
results.Error.Should().BeNull(because: "all values are within the supported numeric ranges");
results.RecordedValues.Should().ContainSingle(because: "one source edit produces one aggregate")
.Which.Should().Be(spacing, because: "the sample variance of three equally spaced values is the square of their spacing");
}

/// <summary>Verifies that integer overloads retain the fractional part of the mean when calculating variance.</summary>
[Theory]
[InlineData(nameof(Int32))]
[InlineData(nameof(Int64))]
public void IntegerValues_FractionalMean_PreservesFractionalVariance(string numericType)
{
// Arrange
var start = _randomizer.Int(-100, 100);
var spacing = _randomizer.Int(2, 30);
var adjustment = _randomizer.Int(1, 2);
var values = new[] { start, start + spacing, start + spacing + spacing + adjustment };
var mean = values.Average();
var expected = Math.Sqrt(values.Sum(value => Math.Pow(value - mean, 2)) / (values.Length - 1));
var fallback = _randomizer.Int(1, 100);
using var source = new TestSourceCache<int, int>(static value => value);
var standardDeviation = numericType switch
{
nameof(Int32) => source.Connect().StdDev(static value => value, fallback),
nameof(Int64) => source.Connect().StdDev(static value => (long)value, fallback),
_ => throw new ArgumentOutOfRangeException(nameof(numericType))
};
using var subscription = standardDeviation
.ValidateSynchronization()
.RecordValues(out var results);

// Act
source.AddOrUpdate(values);

// Assert
results.Error.Should().BeNull(because: "integer inputs with a fractional mean are valid");
results.RecordedValues.Should().ContainSingle(because: "one source edit produces one aggregate")
.Which.Should().BeApproximately(expected, 1e-10, because: "integer inputs do not imply an integer-valued mean or variance");
}

/// <summary>Verifies that clearing the input returns the configured fallback rather than evaluating an undefined variance.</summary>
[Theory]
[InlineData(nameof(Int32))]
[InlineData(nameof(Int64))]
[InlineData(nameof(Single))]
[InlineData(nameof(Double))]
[InlineData(nameof(Decimal))]
public void Cache_AllItemsRemoved_ReportsFallback(string numericType)
{
// Arrange
var center = _randomizer.Int(-100, 100);
var spacing = _randomizer.Int(1, 30);
var values = new[] { center - spacing, center, center + spacing };
var fallback = _randomizer.Int(1, 100);
using var source = new TestSourceCache<int, int>(static value => value);
source.AddOrUpdate(values);
var standardDeviation = numericType switch
{
nameof(Int32) => source.Connect().StdDev(static value => value, fallback),
nameof(Int64) => source.Connect().StdDev(static value => (long)value, fallback),
nameof(Single) => source.Connect().StdDev(static value => (float)value, fallback),
nameof(Double) => source.Connect().StdDev(static value => (double)value, fallback),
nameof(Decimal) => source.Connect().StdDev(static value => (decimal)value, fallback).Select(static value => (double)value),
_ => throw new ArgumentOutOfRangeException(nameof(numericType))
};
using var subscription = standardDeviation
.ValidateSynchronization()
.RecordValues(out var results);

// Act
source.Clear();

// Assert
results.Error.Should().BeNull(because: "an empty collection has a defined fallback");
results.RecordedValues[^1].Should().Be(fallback, because: "sample variance is not calculated for fewer than two items");
}
}
16 changes: 10 additions & 6 deletions src/DynamicData/Aggregation/StdDevEx.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,12 @@
namespace DynamicData.Aggregation;

/// <summary>
/// Extensions for calculating standard deviation.
/// Extensions for calculating sample standard deviation.
/// </summary>
/// <remarks>
/// Each changeset produces the square root of the sample variance, using the item count minus one as the variance divisor.
/// When fewer than two items remain, the configured fallback value is returned instead.
/// </remarks>
public static class StdDevEx
{
/// <summary>
Expand Down Expand Up @@ -139,7 +143,7 @@ public static IObservable<double> StdDev<TObject, TKey>(this IObservable<IChange
/// <param name="valueSelector">The value selector.</param>
/// <param name="fallbackValue">The fallback value.</param>
/// <returns>An observable which emits the standard deviation value.</returns>
public static IObservable<double> StdDev<T>(this IObservable<IAggregateChangeSet<T>> source, Func<T, int> valueSelector, int fallbackValue = 0) => source.StdDevCalc(t => (long)valueSelector(t), fallbackValue, (current, item) => new StdDev<long>(current.Count + 1, current.SumOfItems + item, current.SumOfSquares + (item * item)), (current, item) => new StdDev<long>(current.Count - 1, current.SumOfItems - item, current.SumOfSquares - (item * item)), values => Math.Sqrt(values.SumOfSquares - ((values.SumOfItems * values.SumOfItems) / values.Count)) * (1.0d / (values.Count - 1)));
public static IObservable<double> StdDev<T>(this IObservable<IAggregateChangeSet<T>> source, Func<T, int> valueSelector, int fallbackValue = 0) => source.StdDevCalc(t => (long)valueSelector(t), fallbackValue, (current, item) => new StdDev<long>(current.Count + 1, current.SumOfItems + item, current.SumOfSquares + (item * item)), (current, item) => new StdDev<long>(current.Count - 1, current.SumOfItems - item, current.SumOfSquares - (item * item)), values => Math.Sqrt((values.SumOfSquares - (((double)values.SumOfItems * values.SumOfItems) / values.Count)) / (values.Count - 1)));

/// <summary>
/// Continual computation of the standard deviation of the values in the underlying data source.
Expand All @@ -150,7 +154,7 @@ public static IObservable<double> StdDev<TObject, TKey>(this IObservable<IChange
/// <param name="fallbackValue">The fallback value.</param>
/// <returns>An observable which emits the standard deviation value.</returns>
public static IObservable<double> StdDev<T>(this IObservable<IAggregateChangeSet<T>> source, Func<T, long> valueSelector, long fallbackValue = 0) =>
source.StdDevCalc(valueSelector, fallbackValue, (current, item) => new StdDev<long>(current.Count + 1, current.SumOfItems + item, current.SumOfSquares + (item * item)), (current, item) => new StdDev<long>(current.Count - 1, current.SumOfItems - item, current.SumOfSquares - (item * item)), values => Math.Sqrt(values.SumOfSquares - ((values.SumOfItems * values.SumOfItems) / values.Count)) * (1.0d / (values.Count - 1)));
source.StdDevCalc(valueSelector, fallbackValue, (current, item) => new StdDev<long>(current.Count + 1, current.SumOfItems + item, current.SumOfSquares + (item * item)), (current, item) => new StdDev<long>(current.Count - 1, current.SumOfItems - item, current.SumOfSquares - (item * item)), values => Math.Sqrt((values.SumOfSquares - (((double)values.SumOfItems * values.SumOfItems) / values.Count)) / (values.Count - 1)));

/// <summary>
/// Continual computation of the standard deviation of the values in the underlying data source.
Expand All @@ -161,7 +165,7 @@ public static IObservable<double> StdDev<T>(this IObservable<IAggregateChangeSet
/// <param name="fallbackValue">The fallback value.</param>
/// <returns>An observable which emits the standard deviation value.</returns>
public static IObservable<decimal> StdDev<T>(this IObservable<IAggregateChangeSet<T>> source, Func<T, decimal> valueSelector, decimal fallbackValue = 0M) =>
source.StdDevCalc(valueSelector, fallbackValue, (current, item) => new StdDev<decimal>(current.Count + 1, current.SumOfItems + item, current.SumOfSquares + (item * item)), (current, item) => new StdDev<decimal>(current.Count - 1, current.SumOfItems - item, current.SumOfSquares - (item * item)), values => Sqrt(values.SumOfSquares - ((values.SumOfItems * values.SumOfItems) / values.Count)) * (1.0M / (values.Count - 1)));
source.StdDevCalc(valueSelector, fallbackValue, (current, item) => new StdDev<decimal>(current.Count + 1, current.SumOfItems + item, current.SumOfSquares + (item * item)), (current, item) => new StdDev<decimal>(current.Count - 1, current.SumOfItems - item, current.SumOfSquares - (item * item)), values => Sqrt((values.SumOfSquares - ((values.SumOfItems * values.SumOfItems) / values.Count)) / (values.Count - 1)));

/// <summary>
/// Continual computation of the standard deviation of the values in the underlying data source.
Expand All @@ -171,7 +175,7 @@ public static IObservable<decimal> StdDev<T>(this IObservable<IAggregateChangeSe
/// <param name="valueSelector">The value selector.</param>
/// <param name="fallbackValue">The fallback value.</param>
/// <returns>An observable which emits the standard deviation value.</returns>
public static IObservable<double> StdDev<T>(this IObservable<IAggregateChangeSet<T>> source, Func<T, double> valueSelector, double fallbackValue = 0) => source.StdDevCalc(valueSelector, fallbackValue, (current, item) => new StdDev<double>(current.Count + 1, current.SumOfItems + item, current.SumOfSquares + (item * item)), (current, item) => new StdDev<double>(current.Count - 1, current.SumOfItems - item, current.SumOfSquares - (item * item)), values => Math.Sqrt(values.SumOfSquares - ((values.SumOfItems * values.SumOfItems) / values.Count)) * (1.0d / (values.Count - 1)));
public static IObservable<double> StdDev<T>(this IObservable<IAggregateChangeSet<T>> source, Func<T, double> valueSelector, double fallbackValue = 0) => source.StdDevCalc(valueSelector, fallbackValue, (current, item) => new StdDev<double>(current.Count + 1, current.SumOfItems + item, current.SumOfSquares + (item * item)), (current, item) => new StdDev<double>(current.Count - 1, current.SumOfItems - item, current.SumOfSquares - (item * item)), values => Math.Sqrt((values.SumOfSquares - ((values.SumOfItems * values.SumOfItems) / values.Count)) / (values.Count - 1)));

/// <summary>
/// Continual computation of the standard deviation of the values in the underlying data source.
Expand All @@ -181,7 +185,7 @@ public static IObservable<decimal> StdDev<T>(this IObservable<IAggregateChangeSe
/// <param name="valueSelector">The value selector.</param>
/// <param name="fallbackValue">The fallback value.</param>
/// <returns>An observable which emits the standard deviation value.</returns>
public static IObservable<double> StdDev<T>(this IObservable<IAggregateChangeSet<T>> source, Func<T, float> valueSelector, float fallbackValue = 0) => source.StdDevCalc(valueSelector, fallbackValue, (current, item) => new StdDev<float>(current.Count + 1, current.SumOfItems + item, current.SumOfSquares + (item * item)), (current, item) => new StdDev<float>(current.Count - 1, current.SumOfItems - item, current.SumOfSquares - (item * item)), values => Math.Sqrt(values.SumOfSquares - ((values.SumOfItems * values.SumOfItems) / values.Count)) * (1.0d / (values.Count - 1)));
public static IObservable<double> StdDev<T>(this IObservable<IAggregateChangeSet<T>> source, Func<T, float> valueSelector, float fallbackValue = 0) => source.StdDevCalc(valueSelector, fallbackValue, (current, item) => new StdDev<float>(current.Count + 1, current.SumOfItems + item, current.SumOfSquares + (item * item)), (current, item) => new StdDev<float>(current.Count - 1, current.SumOfItems - item, current.SumOfSquares - (item * item)), values => Math.Sqrt((values.SumOfSquares - ((values.SumOfItems * values.SumOfItems) / values.Count)) / (values.Count - 1)));

private static IObservable<TResult> StdDevCalc<TObject, TValue, TResult>(this IObservable<IAggregateChangeSet<TObject>> source, Func<TObject, TValue> valueSelector, TResult fallbackValue, Func<StdDev<TValue>, TValue, StdDev<TValue>> addAction, Func<StdDev<TValue>, TValue, StdDev<TValue>> removeAction, Func<StdDev<TValue>, TResult> resultAction)
{
Expand Down
Loading