diff --git a/CHANGELOG.md b/CHANGELOG.md index 31d2a59..d6c07df 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed +- **The connecting line on a lane chart no longer crosses lane boundaries.** On an X or mR + chart with factors and ``by=[]`` (one chart, subgroups side by side), the last point of + each subgroup was joined to the first point of the next, so every boundary drew a steep + rise or drop that was only the sort order. The line is now its own trace with a gap at + each boundary, drawn under the markers; the markers trace keeps hover, legend and + signal highlighting exactly as before. Single-series and faceted charts are unchanged. #121 + +### Added +- Docs: a user-guide page on series length and where sigma comes from (the position, the + n = 3..30 precision table from ``validation/short_series_bands.py`` with credit to + @rabujamra, the trending-series and not-one-process cautions, the formulation lesson from + #114, and the two-audiences position on run rules), and a user-guide page stating the + derived-variables contract (never raises, what counts as a violation, what bins promise, + what construction rejects). + ## [0.3.1] - 2026-09-14 ### Fixed diff --git a/docs/getting-started/key-concepts.md b/docs/getting-started/key-concepts.md index 18a2c2c..4dbf131 100644 --- a/docs/getting-started/key-concepts.md +++ b/docs/getting-started/key-concepts.md @@ -51,7 +51,8 @@ The DS determines: The design report states this as one line under `Structure` (`study.series_length`): with any singleton cell, `Sigma for X/mR rests on T − 1 moving ranges`; with every cell replicated, `Sigma rests on within-cell replication`, and T does not enter. No threshold - or warning is attached; the analyst weighs it. + or warning is attached; the analyst weighs it. See + [Series length](../user-guide/series-length.md). - Which VAS residuals can be computed - What conclusions you can draw diff --git a/docs/myst.yml b/docs/myst.yml index 427f8a0..38aa887 100644 --- a/docs/myst.yml +++ b/docs/myst.yml @@ -67,6 +67,10 @@ project: title: Data Formulation - file: user-guide/sds-detection.md title: Design-State Lineage (PDS / ODS / ADS) + - file: user-guide/series-length.md + title: Series Length & Where Sigma Comes From + - file: user-guide/derived-variables.md + title: Derived Variables - file: user-guide/chart-types.md title: Chart Types - file: user-guide/plotting.md diff --git a/docs/user-guide/derived-variables.md b/docs/user-guide/derived-variables.md new file mode 100644 index 0000000..f588933 --- /dev/null +++ b/docs/user-guide/derived-variables.md @@ -0,0 +1,99 @@ +# Derived variables + +A derived variable is a new column computed from an existing one before a study is +formulated: a continuous-to-continuous **transform** (`log`, `sqrt`, `zscore`, …) or a +continuous-to-categorical **bin** (`equal_freq`, `equal_width`, `breaks`, `sd`). Derived +columns can be the response, a factor, or the time variable of the study. The +[derived-variables tutorial](../tutorials/derived-variables.ipynb) walks through the +workflow; this page states the contract. + +```python +import processbehavior as pb + +pbd = (pb.ProcessBehavior(df) + .transform("weight", "log") # -> weight_log + .bin("weight", n=4, label="weight_q")) # -> weight_q, ordered categorical +study = pbd.formulate(response="weight_log", factors=["weight_q"], time="hour") +``` + +Each verb returns a new `ProcessBehavior`; the original is untouched. The specs are plain +data (`pbd.derivations`, `Derivation.to_dict()` / `from_dict()`), fits are frozen when the +study is formulated (`study.derivations`), and `evaluate(spec, column)` is the single +primitive behind both the fluent verbs and the app's live preview. + +## The contract + +**`evaluate` and `validate` never raise on routine data.** Whatever the column contains — +missing values, infinities, ties, a constant, a tiny range, a huge scale — the result comes +back as data: `values`, `n_invalid`, `invalid_index`, `fitted`, and a `message`. A seeded +fuzz over every method is part of the test suite to keep it that way. Exceptions are reserved +for two things: a spec that cannot be honoured (raised at construction, see below) and the +`on_invalid='error'` commit at `formulate()`. + +**Missing input is not a violation.** A missing value passes through as missing and is never +counted in `n_invalid`. + +**A domain violation is any input for which the function has no finite real result.** For +transforms that means: the input is ±inf; the input lies outside the function's domain (log +or log10 of a value ≤ 0, sqrt of a negative, arcsin of a value outside [0, 1], inverse of 0, +a negative base to a fractional power, 0 to a negative power); or the arithmetic overflows +(square of 1e200, inverse of a denormal). All are counted in `n_invalid`, listed in +`invalid_index`, and become missing in the output. A z-score whose sigma is zero or undefined +(a constant or single-value column) makes every present value a violation, rather than a +silently all-missing column. + +Two conveniences on the boundary: a value within 1e-9 of a *closed* boundary (sqrt of +−1e-10, arcsin of 1.0000000005) is clamped to the boundary and is not a violation; and +`shift` applies before the function, so `log` with `shift=1` accepts zeros. + +**What happens to violations is the analyst's call, at formulation.** `on_invalid='error'` +(the default) makes `formulate()` raise a `ValidationError` that names the derivation, the +count, and the first row labels. `on_invalid='na'` makes the offenders missing and proceeds. + +**Bins never drop a finite value.** Every finite, non-missing input lands in a bin, including +values exactly on the minimum or maximum. Infinities cannot be placed in a finite bin: they +leave the fit, become missing, and are counted in `n_invalid` with a message. A bin has no +`on_invalid`; a missing factor value is dropped from the study with a warning at +formulation. + +**Fits are reported, not hidden.** `fitted` carries the resolved edges, labels, and bin count +(or mu and sigma for a z-score). When the fit is not what was asked for, `message` says so: +ties that collapse equal-frequency bins, more bins requested than distinct values (fitted one +per distinct value), ordinal labels above five bins (numbered instead), range labels too +close to print distinctly (numbered instead), a column with no spread (no bins), a +non-numeric source (nothing derived). + +**Construction rejects what evaluation could not honour**, as a `ValidationError` before any +data is touched: + +| Parameter | Must be | +|---|---| +| `n` | a positive integer (numpy integers accepted; booleans and floats rejected) | +| `breaks` | a non-empty, strictly ascending list of finite numbers | +| `shift`, `exponent` | finite numbers | +| `on_invalid` | `'error'` or `'na'` | +| explicit `bin_labels` | non-empty, no nulls, unique | +| output name | not an existing column and not another pending derivation | +| source column | present and numeric (bool counts as numeric; datetime and categorical do not) | + +## Reading a preview + +```python +from processbehavior import Derivation, evaluate + +r = evaluate(Derivation.bin("y", n=4), df["y"]) +r.fitted["n_bins"], r.fitted["edges"], r.fitted["labels"] +r.n_invalid, r.message +r.values.value_counts(sort=False) # bin counts, in bin order +``` + +Before attaching, `validate(spec, df)` returns the structured issues the attach path would +raise on: column not found, not numeric, output-name collision, breaks out of order, and an +explicit label count that does not match the *fitted* bin count. + +## Limits of the current design + +- Derivations are evaluated against the original columns. A derivation of a derived column + is rejected at attach time. +- Box–Cox is not offered. +- Range labels use six significant digits, rising to whatever separates the edges. diff --git a/docs/user-guide/series-length.md b/docs/user-guide/series-length.md new file mode 100644 index 0000000..20e9db0 --- /dev/null +++ b/docs/user-guide/series-length.md @@ -0,0 +1,155 @@ +# Series length and where sigma comes from + +The design report ends its structure section with one line about series length: + +``` + Structure: Complete structure + Series length: T=4. Sigma for X/mR rests on 3 moving ranges. +``` + +It names where the sigma behind the natural process limits comes from at the observed +structure and series length. It is a fact placed beside the design state, not a judgment on +it. Nothing about it changes which charts are valid, which is recommended, or which analyses +are offered, and no threshold, label, or warning is attached. The analyst weighs it. + +This page explains why the line reads the way it does, what the numbers behind it look like, +and what a short series can and cannot tell you. + +## The position + +The Variance Analysis System was built for analytic studies broadly. Statistical process +control, where data arrive quickly and action follows each point, is a special case. VAS +rests on the scientific method and the nontrivial replication of results, not on prescribed +tests or rules for how many points a chart needs. Judgment about what the data are saying +belongs to the analyst. + +Small samples and few time periods raise uncertainty. The analysis still usually gives real +insight into the causes of variation. So the library does not refuse a short series, does not +grade it, and does not withhold capability or loss-function analysis from it. It says where +sigma comes from and how many values carry it, and leaves the reading to the person who +knows what the study is for. + +## Where sigma comes from, by design state + +- **Every cell replicated (ADS 1).** Sigma rests on within-cell replication. The number of + time periods does not enter. A study with five conditions, six replicates each, and one + time period is a complete study: + + ``` + Series length: T=1. Sigma rests on within-cell replication. + ``` + +- **Any singleton cell (ADS 2, ADS 3).** Sigma for the X and mR charts rests on the moving + ranges between successive points: T − 1 of them for T time periods. + + ``` + Series length: T=4. Sigma for X/mR rests on 3 moving ranges. + Series length: T=2. Sigma for X/mR rests on a single moving range. + ``` + +- **No time variable and no replication.** Nothing to say; the line is omitted. + +The same numbers appear in code as `study.series_length`, a frozen record with `T`, +`n_moving_ranges`, `within_cell_df` (the sum over cells of N_kt − 1), `sigma_from_time`, +`mr_interval_80`, and `description`. `study.series_length_description` is the sentence alone. + +## How precise a moving-range sigma is + +For a stable normal process, the moving-range estimate of sigma (MR̄/d₂) scatters around +the truth by an amount that depends only on how many points there are. The table gives the +10th, 50th and 90th percentiles of the estimate relative to the true sigma, and the ratio of +the 90th to the 10th, for every series length from 3 to 30. It was generated by +`validation/short_series_bands.py` (100,000 replicates per length, an independent random +stream per length, seed 20260903), contributed by [@rabujamra](https://github.com/rabujamra) +in [#114](https://github.com/cnicholas/processbehavior/issues/114) and +[#119](https://github.com/cnicholas/processbehavior/pull/119). The p10 and p90 columns are +carried in the library as `MR_SIGMA_INTERVAL_80` and ride on `study.series_length` for +anyone who wants them in code; the design report prints only the sentence. + +| n | p10 | p50 | p90 | p90 / p10 | +|--:|--:|--:|--:|--:| +| 3 | 0.337 | 0.895 | 1.810 | 5.37 | +| 4 | 0.430 | 0.928 | 1.676 | 3.90 | +| 5 | 0.488 | 0.948 | 1.593 | 3.27 | +| 6 | 0.528 | 0.951 | 1.526 | 2.89 | +| 7 | 0.565 | 0.962 | 1.484 | 2.63 | +| 8 | 0.595 | 0.971 | 1.453 | 2.44 | +| 9 | 0.618 | 0.974 | 1.418 | 2.30 | +| 10 | 0.637 | 0.974 | 1.396 | 2.19 | +| 11 | 0.656 | 0.977 | 1.375 | 2.10 | +| 12 | 0.670 | 0.981 | 1.360 | 2.03 | +| 13 | 0.680 | 0.983 | 1.344 | 1.98 | +| 14 | 0.694 | 0.984 | 1.330 | 1.92 | +| 15 | 0.703 | 0.983 | 1.315 | 1.87 | +| 16 | 0.714 | 0.987 | 1.310 | 1.83 | +| 17 | 0.722 | 0.986 | 1.300 | 1.80 | +| 18 | 0.730 | 0.986 | 1.288 | 1.77 | +| 19 | 0.739 | 0.987 | 1.280 | 1.73 | +| 20 | 0.745 | 0.988 | 1.274 | 1.71 | +| 21 | 0.749 | 0.989 | 1.266 | 1.69 | +| 22 | 0.756 | 0.989 | 1.258 | 1.67 | +| 23 | 0.761 | 0.989 | 1.253 | 1.65 | +| 24 | 0.766 | 0.990 | 1.249 | 1.63 | +| 25 | 0.771 | 0.991 | 1.243 | 1.61 | +| 26 | 0.775 | 0.991 | 1.238 | 1.60 | +| 27 | 0.780 | 0.992 | 1.234 | 1.58 | +| 28 | 0.784 | 0.993 | 1.229 | 1.57 | +| 29 | 0.787 | 0.992 | 1.224 | 1.55 | +| 30 | 0.790 | 0.993 | 1.220 | 1.54 | + +Read a row as: at four points, the middle 80% of moving-range sigma estimates fall between +0.43 and 1.68 times the true sigma. Most of the improvement is spent by eight to ten +points; after that the spread narrows slowly. Nothing in the library is cut from this table. +It is here so an analyst can see what T − 1 moving ranges mean. + +`validation/short_series_sampling.py`, from the same contributor, reproduces the +single-process illustration that opened the discussion: the limits from the first T points +of one stable series, and the fraction of the true limit span they cover. + +## Two cautions a short series cannot raise for you + +**A trending series inflates its own moving range.** When successive values rise, the +differences between them carry the slope as well as the noise. The limits widen to absorb the +very pattern an analyst is looking for, and nothing inside the moving range can tell the two +apart. A test for monotone drift does not rescue this: with three differences, an independent +series is monotone with probability 2/4! = 1/12, above 5%, so no four-point series can reach +significance. The chart shows the climb; the limits cannot. + +**Four figures are not always four observations of one process.** Benchmark figures +constructed under different rules in different years, restated or risk-adjusted separately, +are not draws from one process. No statement about series length can catch that. Only the +analyst who knows how the numbers were made can. + +## The formulation matters more than the length + +The question that started this page was about 476 organizations with four annual figures +each, run as 476 separate four-point studies, each read from its summary line. Formulated as +one study with the organization as a factor and the year as time, all of them sit on one +chart at one scale, lane by lane, and the same rising pattern reproduces in nearly every +lane. That agreement across independent subgroups is replication of a result in the +scientific sense, and it is the finding. Hardly any single point falls outside its limits, +and that is expected. The picture carries what the limits cannot. + +```python +study = pb.formulate(df, response="PER CAPITA EXPENDITURE", factors=["ACO"], time="YEAR") +print(study.design()) # ADS 2 ... Series length: T=4. Sigma for X/mR rests on 3 moving ranges. +result = study.execute(chart="X", by=[], companion=True) +result.plot(chart="X") # every organisation as a lane, one scale +``` + +Before reading any limit, ask what decision the analysis serves. If action will be taken on +the historical record, the limits describe what happened. If the aim is to predict next year +and act on the prediction, the uncertainty that dominates is the extrapolation into a future +the data has not seen, not the precision of the sigma behind the limits. + +## Two audiences, one position + +In an analytic study the analyst sees every chart, and the judgment is theirs. The run +rules are optional and off unless asked for; see [WECO rules](../reference/weco-rules.md). + +In automated monitoring of a device or a telemetry stream, nobody sees every chart, and the +Western Electric rules are how the system notices on the analyst's behalf. They stay in the +library for that use. That is also why the signal summary reports a partial evaluation +honestly: an automated check that says "no signals" on a series it could not examine is +worse than one that says nothing. Points beyond the limits are always computed; rules 2 +through 8 and the chart zones are opt-in. diff --git a/processbehavior/plotting/renderers.py b/processbehavior/plotting/renderers.py index c83b9b0..f1d2830 100644 --- a/processbehavior/plotting/renderers.py +++ b/processbehavior/plotting/renderers.py @@ -35,6 +35,36 @@ # --------------------------------------------------------------------------- +def _lane_positions(lane_boundaries, chart_name: str, n_rows: int) -> list[int]: + """0-based row positions where a new lane starts (excluding row 0), for this chart. + + Accepts the flat ``list[dict]`` form or the per-stratum ``dict`` form of the + ``lane_boundaries`` metadata; returns ``[]`` when there are no boundaries. + """ + if not lane_boundaries: + return [] + if isinstance(lane_boundaries, dict): + lane_boundaries = lane_boundaries.get(chart_name) or [] + positions = sorted({int(b['position']) for b in lane_boundaries if 0 < int(b['position']) < n_rows}) + return positions + + +def _with_gaps(values: list, positions: list[int], repeat: bool = False) -> list: + """Insert a gap entry before each position so Plotly breaks the line there. + + For ``y`` the gap is ``None`` (a null y breaks the line). For ``x`` pass + ``repeat=True`` to insert the position's own x value instead, so the x array + stays numeric/categorical throughout and axis consumers never see a null. + """ + out: list = [] + cut = set(positions) + for i, v in enumerate(values): + if i in cut: + out.append(v if repeat else None) + out.append(v) + return out + + def _build_hover( data: pd.DataFrame, ctx: RenderContext, @@ -136,10 +166,32 @@ def render_control_chart( # 2. Main data trace customdata, hovertemplate = _build_hover(data, ctx, value_col) + lane_positions = _lane_positions(spec.lane_boundaries, ctx.chart_name, len(data)) + if lane_positions: + # Lane chart: the connecting line must not cross a lane boundary. The last point + # of one subgroup joined to the first of the next drew a steep "event" at every + # boundary that was only the sort order (#121). Draw the line as its own trace + # with a gap at each boundary, under the markers; the markers trace keeps the + # hover, customdata and legend exactly as before. + _add_trace( + fig, + go.Scatter( + x=_with_gaps(list(x_data), lane_positions, repeat=True), + y=_with_gaps(list(data[value_col]), lane_positions), + mode='lines', + line=dict(color=theme.data_color, width=ctx.line_width), + opacity=theme.data_opacity, + hoverinfo='skip', + showlegend=False, + name=f'{ctx.chart_name} (lanes)', + ), + row, + col, + ) trace_kw = dict( x=x_data, y=data[value_col], - mode='lines+markers', + mode='markers' if lane_positions else 'lines+markers', name=ctx.chart_name, marker=dict(size=ctx.marker_size, color=theme.data_color), line=dict(color=theme.data_color, width=ctx.line_width), diff --git a/tests/test_lane_connectors.py b/tests/test_lane_connectors.py new file mode 100644 index 0000000..c232f9b --- /dev/null +++ b/tests/test_lane_connectors.py @@ -0,0 +1,103 @@ +"""On a lane chart the connecting line breaks at every lane boundary (#121). + +One subgroup's last point used to be joined to the next subgroup's first point, so every +boundary showed a steep rise or drop that was only the sort order. The line is now its own +trace with a gap at each boundary; the markers trace keeps hover, customdata and legend. +""" + +import numpy as np +import pandas as pd + +import processbehavior as pb + + +def _lane_study(n_lanes=3, T=4): + rng = np.random.default_rng(121) + rows = [] + for k in range(n_lanes): + base = 100 * (k + 1) + for t in range(1, T + 1): + rows.append({'unit': f'U{k + 1}', 't': t, 'y': base + t + rng.normal(0, 0.1)}) + return pb.formulate(pd.DataFrame(rows), response='y', factors=['unit'], time='t') + + +def _traces(fig): + return {t.name: t for t in fig.data} + + +def test_lane_chart_draws_the_line_with_one_gap_per_boundary(): + fig = _lane_study(n_lanes=3, T=4).execute(chart='X', by=[], companion=True).plot(chart='X') + traces = _traces(fig) + line = traces['X (lanes)'] + markers = traces['X'] + assert line.mode == 'lines' and markers.mode == 'markers' + ys = list(line.y) + assert ys.count(None) == 2, 'three lanes -> two boundaries -> two gaps' + assert len([y for y in ys if y is not None]) == 12 and len(markers.y) == 12 + # The gaps sit exactly at the lane starts (positions 4 and 8). + assert [i for i, y in enumerate(ys) if y is None] == [4, 9] + + +def test_lines_never_join_two_lanes(): + """No drawn segment spans a boundary: every consecutive non-gap pair is within a lane.""" + fig = _lane_study(n_lanes=4, T=3).execute(chart='X', by=[], companion=True).plot(chart='X') + line = _traces(fig)['X (lanes)'] + ys = list(line.y) + # Lanes are at 100, 200, 300, 400 (+ small t); any segment crossing a lane would jump ~100. + for a, b in zip(ys, ys[1:], strict=False): + if a is not None and b is not None: + assert abs(a - b) < 10, (a, b) + + +def test_markers_keep_hover_and_the_line_has_none(): + fig = _lane_study().execute(chart='X', by=[], companion=True).plot(chart='X') + traces = _traces(fig) + assert traces['X'].hovertemplate + assert traces['X (lanes)'].hoverinfo == 'skip' and traces['X (lanes)'].showlegend is False + + +def test_single_series_chart_is_unchanged(): + """No lanes -> the original single lines+markers trace, no extra trace.""" + df = pd.DataFrame({'t': range(1, 13), 'y': np.arange(12, dtype=float)}) + fig = pb.formulate(df, response='y', time='t').execute(chart='X', companion=True).plot(chart='X') + names = [t.name for t in fig.data] + assert 'X (lanes)' not in names + assert _traces(fig)['X'].mode == 'lines+markers' + + +def test_faceted_charts_are_unchanged(): + """by=['unit'] gives one panel per lane: no boundaries inside a panel, so no gaps.""" + fig = _lane_study().execute(chart='X', by=['unit'], companion=True).plot(chart='X', facet=True, ncols=3) + assert not [t for t in fig.data if t.name and t.name.endswith('(lanes)')] + + +# --------------------------------------------------------------------------- +# The helpers, directly: both boundary shapes, out-of-range positions, both gap modes +# --------------------------------------------------------------------------- + + +def test_lane_positions_accepts_flat_and_per_stratum_boundaries(): + from processbehavior.plotting.renderers import _lane_positions + + flat = [{'position': 4, 'label': 'B'}, {'position': 8, 'label': 'C'}] + assert _lane_positions(flat, 'X', n_rows=12) == [4, 8] + per_stratum = {'X': flat, 'mR': [{'position': 3}]} + assert _lane_positions(per_stratum, 'X', n_rows=12) == [4, 8] + assert _lane_positions(per_stratum, 'mR', n_rows=12) == [3] + assert _lane_positions(per_stratum, 'Histogram', n_rows=12) == [] + assert _lane_positions(None, 'X', n_rows=12) == [] and _lane_positions([], 'X', n_rows=12) == [] + + +def test_lane_positions_ignores_row_zero_and_out_of_range(): + from processbehavior.plotting.renderers import _lane_positions + + raw = [{'position': 0}, {'position': 5}, {'position': 5}, {'position': 12}, {'position': 99}] + assert _lane_positions(raw, 'X', n_rows=12) == [5] + + +def test_with_gaps_inserts_none_for_y_and_repeats_x(): + from processbehavior.plotting.renderers import _with_gaps + + assert _with_gaps([10, 11, 12, 13], [2]) == [10, 11, None, 12, 13] + assert _with_gaps(['a', 'b', 'c', 'd'], [2], repeat=True) == ['a', 'b', 'c', 'c', 'd'] + assert _with_gaps([1, 2, 3], []) == [1, 2, 3]