Skip to content

fix(derivations): one bin per distinct value when more bins are requested than values exist - #128

Merged
cnicholas merged 1 commit into
mainfrom
fix/equal-freq-tie-cap
Sep 15, 2026
Merged

cnicholas merged 1 commit into
mainfrom
fix/equal-freq-tie-cap

Conversation

@cnicholas

Copy link
Copy Markdown
Owner

Summary

  • What: when at least as many bins are requested as there are distinct finite values, fit the edges at the midpoints between neighbouring values, one bin per value, instead of capping the count and re-taking quantiles.
  • Why: the 0.3.1 cap collapsed heavily tied columns. A column of 3,000 ones and 1,000 twos with n=4 fitted one bin, where 0.3.0 fitted two, because the halved quantile positions both landed on 1. Caught by the app's own preview test during the pin bump. Requesting far more bins than values (the case the cap was for) now also gives one bin per value instead of mostly-empty interpolated bins.
  • Scope: _fit_edges in derivations.py; two existing tests re-pinned to the fewer-bins-than-values case they were written to prove; robustness tests extended; CHANGELOG.

Contract / invariants

  • Defaults unchanged — requesting fewer bins than distinct values takes the original quantile / equal-width path with byte-identical edges (pinned). Every finite value still lands in a bin (fuzz and bin-extremes suites). validation/e2e_bishop_report.py exits 0.
  • Residuals unaffected.
  • Row/index alignment preserved.
  • Output schema compatible.
  • Pinned error strings unchanged — message wording for the collision case is unchanged; the one-per-value case has its own message.

Behaviour changes

  • n >= distinct count: edges are [v1, midpoints…, vk] and the message reads requested n bins, ties produced k (one per distinct value). Labels change from interpolated quantile edges such as [1, 1.25) to midpoint edges such as [1, 1.5); bin membership is one value per bin either way.

Methodology

  • No methodology change.

Tests

  • test_heavily_tied_column_keeps_one_bin_per_distinct_value — the regression, 3,000 ones and 1,000 twos.

  • test_more_bins_than_distinct_values_gives_one_bin_per_value — edges and counts for both methods.

  • test_fewer_bins_than_distinct_values_is_untouched — the original path pinned.

  • test_tie_drop_uses_fitted_count_labels_and_message and test_validate_label_count_against_fitted_not_requested — now request 4 bins on 5 distinct values so the quantile-collision case they document still occurs (5 on 5 now correctly fits 5).

  • pytest tests/ — 2431 passed, 10 skipped

  • ruff check . — clean

  • Golden masters untouched

🤖 Generated with Claude Code

https://claude.ai/code/session_01BKphCi1LC1hjGYteWEK9JV

…sted than values exist

## Summary
- **What:** when n >= number of distinct finite values, fit edges at the midpoints
  between neighbouring values (one bin per value) instead of capping n and re-taking
  quantiles.
- **Why:** 0.3.1's cap collapsed heavily tied columns — 3,000 ones and 1,000 twos with
  n=4 fitted ONE bin (0.3.0 fitted two) because the halved quantile positions both landed
  on 1. Caught by the app's preview test on the pin bump.
- **Scope:** derivations._fit_edges; two existing tests re-pinned to the n < distinct
  case they were written to prove; robustness tests extended; CHANGELOG.

## Contract / Invariants (must remain true)
- n < number of distinct values: byte-identical edges (original quantile / linspace path;
  pinned by test_fewer_bins_than_distinct_values_is_untouched and the bin-extremes suite).
- Every finite value lands in a bin (fuzz + bin-extremes).
- validator exit 0.

## Behavior Changes (explicit)
- n >= distinct count: edges = [v1, midpoints..., vk]; message "requested n bins, ties
  produced k (one per distinct value)". Labels change from interpolated quantile edges
  (e.g. '[1, 1.25)') to midpoint edges ('[1, 1.5)'); membership is one value per bin.

## Tests
- test_heavily_tied_column_keeps_one_bin_per_distinct_value (the regression)
- test_more_bins_than_distinct_values_gives_one_bin_per_value (edges + counts)
- test_fewer_bins_than_distinct_values_is_untouched
- test_tie_drop_uses_fitted_count_labels_and_message and
  test_validate_label_count_against_fitted_not_requested now request n=4 on 5 distinct
  values so the quantile-collision case they document still occurs.

## Manual Verification
- pytest tests/: 2431 passed; ruff clean; validator exit 0
@cnicholas
cnicholas merged commit 2799286 into main Sep 15, 2026
17 checks passed
@cnicholas
cnicholas deleted the fix/equal-freq-tie-cap branch September 15, 2026 11:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant