Skip to content

Correct sample standard deviation calculations - #1191

Open
dwcullop wants to merge 2 commits into
mainfrom
u/dacullop/main/stddev-sample-calculation
Open

dwcullop wants to merge 2 commits into
mainfrom
u/dacullop/main/stddev-sample-calculation

Conversation

@dwcullop

Copy link
Copy Markdown
Member

Fixes #1181

Problem

Every StdDev overload computed the result with a misplaced divisor. The corrected sum of squares was divided by Count inside Math.Sqrt, and the 1 / (Count - 1) sample factor was then applied outside the square root:

Math.Sqrt(SumOfSquares - ((SumOfItems * SumOfItems) / Count)) * (1.0d / (Count - 1))

That is not the sample standard deviation, nor the population standard deviation. It is the square root of the total corrected sum of squares scaled by 1 / (n - 1), so results were wrong for every input of size 2 or more.

The int and long overloads had a second defect: (SumOfItems * SumOfItems) / Count was evaluated in integer arithmetic, truncating the mean correction before it reached the floating-point path.

Fix

The divisor now lives inside the square root, and the integer overloads widen to double before dividing:

Math.Sqrt((SumOfSquares - (((double)SumOfItems * SumOfItems) / Count)) / (Count - 1))

The decimal overload gets the same restructuring using its Sqrt helper.

⚠️ Behavioral change: returned values intentionally change

This changes the numbers returned by StdDev for all callers. Any consumer comparing against previously observed output, or with recorded baselines/snapshots, will see different values. The new values are the correct sample standard deviation (Bessel-corrected, n - 1 denominator); the old values were not a standard deviation under any convention, so the change is a correction rather than a redefinition. The public signatures are unchanged.

Fallback behavior is unchanged: when fewer than two items remain, the configured fallbackValue is returned.

Coverage

StdDev previously had no test coverage at all. This adds StdDevFixture with tests over all five numeric selector types (int, long, float, double, decimal), validating computed values against independently calculated expectations, plus incremental add/remove/update behavior and the fewer-than-two-items fallback path.

Documentation

Adds StdDev operator sections to the cache and list instruction references, documenting the sample-deviation formula, the fallback rule, and per-change-reason handling.

Add StdDev operator sections to the cache and list instruction references, covering the sample-deviation formula, fallback behavior, and per-change-reason handling.
@dwcullop

Copy link
Copy Markdown
Member Author

P2: Integer standard deviation loses small variances when the moments are converted to double.

Locations: StdDevEx.cs, line 146, and the corresponding long calculation at line 157.

Casting SumOfItems to double before calculating and subtracting the squared-mean term avoids integer division, but it also converts the subtraction to floating-point arithmetic. Large, nearly equal moments then round to the same value, losing the variance.

Reproduced with the explicit integer overload, source.Connect().StdDev(value => value, 0), and inputs [174542947, 174542949]: the PR returns 0; the base returns the correct sample standard deviation, 1.4142135623730951. Neither the sum nor the squared moments overflow long for this input.

Suggested fix: calculate variance using a numerically stable centered representation, such as a running mean and centered sum of squares with matching add/remove updates, rather than subtracting two large floating-point moments. This requires coordinated accumulator changes, not a replacement of the final cast alone. Add translation-invariance coverage for both int and long selectors: widely offset inputs with identical spacing should produce the same deviation, including after removals and replacements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: StdDev applies the sample divisor outside the square root and truncates integer variance

1 participant