fix(plotting): lane-chart line breaks at lane boundaries (#121); docs: series length + derived variables

CHANGELOG.md

@@ -7,6 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+- **The connecting line on a lane chart no longer crosses lane boundaries.** On an X or mR
+  chart with factors and ``by=[]`` (one chart, subgroups side by side), the last point of
+  each subgroup was joined to the first point of the next, so every boundary drew a steep
+  rise or drop that was only the sort order. The line is now its own trace with a gap at
+  each boundary, drawn under the markers; the markers trace keeps hover, legend and
+  signal highlighting exactly as before. Single-series and faceted charts are unchanged. #121
+
+### Added
+- Docs: a user-guide page on series length and where sigma comes from (the position, the
+  n = 3..30 precision table from ``validation/short_series_bands.py`` with credit to
+  @rabujamra, the trending-series and not-one-process cautions, the formulation lesson from
+  #114, and the two-audiences position on run rules), and a user-guide page stating the
+  derived-variables contract (never raises, what counts as a violation, what bins promise,
+  what construction rejects).
+
 ## [0.3.1] - 2026-09-14
 
 ### Fixed

docs/getting-started/key-concepts.md

@@ -51,7 +51,8 @@ The DS determines:
   The design report states this as one line under `Structure` (`study.series_length`): with
   any singleton cell, `Sigma for X/mR rests on T − 1 moving ranges`; with every cell
   replicated, `Sigma rests on within-cell replication`, and T does not enter. No threshold
-  or warning is attached; the analyst weighs it.
+  or warning is attached; the analyst weighs it. See
+  [Series length](../user-guide/series-length.md).
 - Which VAS residuals can be computed
 - What conclusions you can draw
 

docs/myst.yml

@@ -67,6 +67,10 @@ project:
           title: Data Formulation
         - file: user-guide/sds-detection.md
           title: Design-State Lineage (PDS / ODS / ADS)
+        - file: user-guide/series-length.md
+          title: Series Length & Where Sigma Comes From
+        - file: user-guide/derived-variables.md
+          title: Derived Variables
         - file: user-guide/chart-types.md
           title: Chart Types
         - file: user-guide/plotting.md

docs/user-guide/derived-variables.md

@@ -0,0 +1,99 @@
+# Derived variables
+
+A derived variable is a new column computed from an existing one before a study is
+formulated: a continuous-to-continuous **transform** (`log`, `sqrt`, `zscore`, …) or a
+continuous-to-categorical **bin** (`equal_freq`, `equal_width`, `breaks`, `sd`). Derived
+columns can be the response, a factor, or the time variable of the study. The
+[derived-variables tutorial](../tutorials/derived-variables.ipynb) walks through the
+workflow; this page states the contract.
+
+```python
+import processbehavior as pb
+
+pbd = (pb.ProcessBehavior(df)
+         .transform("weight", "log")                         # -> weight_log
+         .bin("weight", n=4, label="weight_q"))              # -> weight_q, ordered categorical
+study = pbd.formulate(response="weight_log", factors=["weight_q"], time="hour")
+```
+
+Each verb returns a new `ProcessBehavior`; the original is untouched. The specs are plain
+data (`pbd.derivations`, `Derivation.to_dict()` / `from_dict()`), fits are frozen when the
+study is formulated (`study.derivations`), and `evaluate(spec, column)` is the single
+primitive behind both the fluent verbs and the app's live preview.
+
+## The contract
+
+**`evaluate` and `validate` never raise on routine data.** Whatever the column contains —
+missing values, infinities, ties, a constant, a tiny range, a huge scale — the result comes
+back as data: `values`, `n_invalid`, `invalid_index`, `fitted`, and a `message`. A seeded
+fuzz over every method is part of the test suite to keep it that way. Exceptions are reserved
+for two things: a spec that cannot be honoured (raised at construction, see below) and the
+`on_invalid='error'` commit at `formulate()`.
+
+**Missing input is not a violation.** A missing value passes through as missing and is never
+counted in `n_invalid`.
+
+**A domain violation is any input for which the function has no finite real result.** For
+transforms that means: the input is ±inf; the input lies outside the function's domain (log
+or log10 of a value ≤ 0, sqrt of a negative, arcsin of a value outside [0, 1], inverse of 0,
+a negative base to a fractional power, 0 to a negative power); or the arithmetic overflows
+(square of 1e200, inverse of a denormal). All are counted in `n_invalid`, listed in
+`invalid_index`, and become missing in the output. A z-score whose sigma is zero or undefined
+(a constant or single-value column) makes every present value a violation, rather than a
+silently all-missing column.
+
+Two conveniences on the boundary: a value within 1e-9 of a *closed* boundary (sqrt of
+−1e-10, arcsin of 1.0000000005) is clamped to the boundary and is not a violation; and
+`shift` applies before the function, so `log` with `shift=1` accepts zeros.
+
+**What happens to violations is the analyst's call, at formulation.** `on_invalid='error'`
+(the default) makes `formulate()` raise a `ValidationError` that names the derivation, the
+count, and the first row labels. `on_invalid='na'` makes the offenders missing and proceeds.
+
+**Bins never drop a finite value.** Every finite, non-missing input lands in a bin, including
+values exactly on the minimum or maximum. Infinities cannot be placed in a finite bin: they
+leave the fit, become missing, and are counted in `n_invalid` with a message. A bin has no
+`on_invalid`; a missing factor value is dropped from the study with a warning at
+formulation.
+
+**Fits are reported, not hidden.** `fitted` carries the resolved edges, labels, and bin count
+(or mu and sigma for a z-score). When the fit is not what was asked for, `message` says so:
+ties that collapse equal-frequency bins, more bins requested than distinct values (fitted one
+per distinct value), ordinal labels above five bins (numbered instead), range labels too
+close to print distinctly (numbered instead), a column with no spread (no bins), a
+non-numeric source (nothing derived).
+
+**Construction rejects what evaluation could not honour**, as a `ValidationError` before any
+data is touched:
+
+| Parameter | Must be |
+|---|---|
+| `n` | a positive integer (numpy integers accepted; booleans and floats rejected) |
+| `breaks` | a non-empty, strictly ascending list of finite numbers |
+| `shift`, `exponent` | finite numbers |
+| `on_invalid` | `'error'` or `'na'` |
+| explicit `bin_labels` | non-empty, no nulls, unique |
+| output name | not an existing column and not another pending derivation |
+| source column | present and numeric (bool counts as numeric; datetime and categorical do not) |
+
+## Reading a preview
+
+```python
+from processbehavior import Derivation, evaluate
+
+r = evaluate(Derivation.bin("y", n=4), df["y"])
+r.fitted["n_bins"], r.fitted["edges"], r.fitted["labels"]
+r.n_invalid, r.message
+r.values.value_counts(sort=False)          # bin counts, in bin order
+```
+
+Before attaching, `validate(spec, df)` returns the structured issues the attach path would
+raise on: column not found, not numeric, output-name collision, breaks out of order, and an
+explicit label count that does not match the *fitted* bin count.
+
+## Limits of the current design
+
+- Derivations are evaluated against the original columns. A derivation of a derived column
+  is rejected at attach time.
+- Box–Cox is not offered.
+- Range labels use six significant digits, rising to whatever separates the edges.

docs/user-guide/series-length.md

@@ -0,0 +1,155 @@
+# Series length and where sigma comes from
+
+The design report ends its structure section with one line about series length:
+
+```
+  Structure: Complete structure
+  Series length: T=4. Sigma for X/mR rests on 3 moving ranges.
+```
+
+It names where the sigma behind the natural process limits comes from at the observed
+structure and series length. It is a fact placed beside the design state, not a judgment on
+it. Nothing about it changes which charts are valid, which is recommended, or which analyses
+are offered, and no threshold, label, or warning is attached. The analyst weighs it.
+
+This page explains why the line reads the way it does, what the numbers behind it look like,
+and what a short series can and cannot tell you.
+
+## The position
+
+The Variance Analysis System was built for analytic studies broadly. Statistical process
+control, where data arrive quickly and action follows each point, is a special case. VAS
+rests on the scientific method and the nontrivial replication of results, not on prescribed
+tests or rules for how many points a chart needs. Judgment about what the data are saying
+belongs to the analyst.
+
+Small samples and few time periods raise uncertainty. The analysis still usually gives real
+insight into the causes of variation. So the library does not refuse a short series, does not
+grade it, and does not withhold capability or loss-function analysis from it. It says where
+sigma comes from and how many values carry it, and leaves the reading to the person who
+knows what the study is for.
+
+## Where sigma comes from, by design state
+
+- **Every cell replicated (ADS 1).** Sigma rests on within-cell replication. The number of
+  time periods does not enter. A study with five conditions, six replicates each, and one
+  time period is a complete study:
+
+  ```
+  Series length: T=1. Sigma rests on within-cell replication.
+  ```
+
+- **Any singleton cell (ADS 2, ADS 3).** Sigma for the X and mR charts rests on the moving
+  ranges between successive points: T − 1 of them for T time periods.
+
+  ```
+  Series length: T=4. Sigma for X/mR rests on 3 moving ranges.
+  Series length: T=2. Sigma for X/mR rests on a single moving range.
+  ```
+
+- **No time variable and no replication.** Nothing to say; the line is omitted.
+
+The same numbers appear in code as `study.series_length`, a frozen record with `T`,
+`n_moving_ranges`, `within_cell_df` (the sum over cells of N_kt − 1), `sigma_from_time`,
+`mr_interval_80`, and `description`. `study.series_length_description` is the sentence alone.
+
+## How precise a moving-range sigma is
+
+For a stable normal process, the moving-range estimate of sigma (MR̄/d₂) scatters around
+the truth by an amount that depends only on how many points there are. The table gives the
+10th, 50th and 90th percentiles of the estimate relative to the true sigma, and the ratio of
+the 90th to the 10th, for every series length from 3 to 30. It was generated by
+`validation/short_series_bands.py` (100,000 replicates per length, an independent random
+stream per length, seed 20260903), contributed by [@rabujamra](https://github.com/rabujamra)
+in [#114](https://github.com/cnicholas/processbehavior/issues/114) and
+[#119](https://github.com/cnicholas/processbehavior/pull/119). The p10 and p90 columns are
+carried in the library as `MR_SIGMA_INTERVAL_80` and ride on `study.series_length` for
+anyone who wants them in code; the design report prints only the sentence.
+
+| n | p10 | p50 | p90 | p90 / p10 |
+|--:|--:|--:|--:|--:|
+| 3 | 0.337 | 0.895 | 1.810 | 5.37 |
+| 4 | 0.430 | 0.928 | 1.676 | 3.90 |
+| 5 | 0.488 | 0.948 | 1.593 | 3.27 |
+| 6 | 0.528 | 0.951 | 1.526 | 2.89 |
+| 7 | 0.565 | 0.962 | 1.484 | 2.63 |
+| 8 | 0.595 | 0.971 | 1.453 | 2.44 |
+| 9 | 0.618 | 0.974 | 1.418 | 2.30 |
+| 10 | 0.637 | 0.974 | 1.396 | 2.19 |
+| 11 | 0.656 | 0.977 | 1.375 | 2.10 |
+| 12 | 0.670 | 0.981 | 1.360 | 2.03 |
+| 13 | 0.680 | 0.983 | 1.344 | 1.98 |
+| 14 | 0.694 | 0.984 | 1.330 | 1.92 |
+| 15 | 0.703 | 0.983 | 1.315 | 1.87 |
+| 16 | 0.714 | 0.987 | 1.310 | 1.83 |
+| 17 | 0.722 | 0.986 | 1.300 | 1.80 |
+| 18 | 0.730 | 0.986 | 1.288 | 1.77 |
+| 19 | 0.739 | 0.987 | 1.280 | 1.73 |
+| 20 | 0.745 | 0.988 | 1.274 | 1.71 |
+| 21 | 0.749 | 0.989 | 1.266 | 1.69 |
+| 22 | 0.756 | 0.989 | 1.258 | 1.67 |
+| 23 | 0.761 | 0.989 | 1.253 | 1.65 |
+| 24 | 0.766 | 0.990 | 1.249 | 1.63 |
+| 25 | 0.771 | 0.991 | 1.243 | 1.61 |
+| 26 | 0.775 | 0.991 | 1.238 | 1.60 |
+| 27 | 0.780 | 0.992 | 1.234 | 1.58 |
+| 28 | 0.784 | 0.993 | 1.229 | 1.57 |
+| 29 | 0.787 | 0.992 | 1.224 | 1.55 |
+| 30 | 0.790 | 0.993 | 1.220 | 1.54 |
+
+Read a row as: at four points, the middle 80% of moving-range sigma estimates fall between
+0.43 and 1.68 times the true sigma. Most of the improvement is spent by eight to ten
+points; after that the spread narrows slowly. Nothing in the library is cut from this table.
+It is here so an analyst can see what T − 1 moving ranges mean.
+
+`validation/short_series_sampling.py`, from the same contributor, reproduces the
+single-process illustration that opened the discussion: the limits from the first T points
+of one stable series, and the fraction of the true limit span they cover.
+
+## Two cautions a short series cannot raise for you
+
+**A trending series inflates its own moving range.** When successive values rise, the
+differences between them carry the slope as well as the noise. The limits widen to absorb the
+very pattern an analyst is looking for, and nothing inside the moving range can tell the two
+apart. A test for monotone drift does not rescue this: with three differences, an independent
+series is monotone with probability 2/4! = 1/12, above 5%, so no four-point series can reach
+significance. The chart shows the climb; the limits cannot.
+
+**Four figures are not always four observations of one process.** Benchmark figures
+constructed under different rules in different years, restated or risk-adjusted separately,
+are not draws from one process. No statement about series length can catch that. Only the
+analyst who knows how the numbers were made can.
+
+## The formulation matters more than the length
+
+The question that started this page was about 476 organizations with four annual figures
+each, run as 476 separate four-point studies, each read from its summary line. Formulated as
+one study with the organization as a factor and the year as time, all of them sit on one
+chart at one scale, lane by lane, and the same rising pattern reproduces in nearly every
+lane. That agreement across independent subgroups is replication of a result in the
+scientific sense, and it is the finding. Hardly any single point falls outside its limits,
+and that is expected. The picture carries what the limits cannot.
+
+```python
+study = pb.formulate(df, response="PER CAPITA EXPENDITURE", factors=["ACO"], time="YEAR")
+print(study.design())                              # ADS 2 ... Series length: T=4. Sigma for X/mR rests on 3 moving ranges.
+result = study.execute(chart="X", by=[], companion=True)
+result.plot(chart="X")                             # every organisation as a lane, one scale
+```
+
+Before reading any limit, ask what decision the analysis serves. If action will be taken on
+the historical record, the limits describe what happened. If the aim is to predict next year
+and act on the prediction, the uncertainty that dominates is the extrapolation into a future
+the data has not seen, not the precision of the sigma behind the limits.
+
+## Two audiences, one position
+
+In an analytic study the analyst sees every chart, and the judgment is theirs. The run
+rules are optional and off unless asked for; see [WECO rules](../reference/weco-rules.md).
+
+In automated monitoring of a device or a telemetry stream, nobody sees every chart, and the
+Western Electric rules are how the system notices on the analyst's behalf. They stay in the
+library for that use. That is also why the signal summary reports a partial evaluation
+honestly: an automated check that says "no signals" on a series it could not examine is
+worse than one that says nothing. Points beyond the limits are always computed; rules 2
+through 8 and the chart zones are opt-in.

processbehavior/plotting/renderers.py

@@ -35,6 +35,36 @@
 # ---------------------------------------------------------------------------
 
 
+def _lane_positions(lane_boundaries, chart_name: str, n_rows: int) -> list[int]:
+    """0-based row positions where a new lane starts (excluding row 0), for this chart.
+
+    Accepts the flat ``list[dict]`` form or the per-stratum ``dict`` form of the
+    ``lane_boundaries`` metadata; returns ``[]`` when there are no boundaries.
+    """
+    if not lane_boundaries:
+        return []
+    if isinstance(lane_boundaries, dict):
+        lane_boundaries = lane_boundaries.get(chart_name) or []
+    positions = sorted({int(b['position']) for b in lane_boundaries if 0 < int(b['position']) < n_rows})
+    return positions
+
+
+def _with_gaps(values: list, positions: list[int], repeat: bool = False) -> list:
+    """Insert a gap entry before each position so Plotly breaks the line there.
+
+    For ``y`` the gap is ``None`` (a null y breaks the line). For ``x`` pass
+    ``repeat=True`` to insert the position's own x value instead, so the x array
+    stays numeric/categorical throughout and axis consumers never see a null.
+    """
+    out: list = []
+    cut = set(positions)
+    for i, v in enumerate(values):
+        if i in cut:
+            out.append(v if repeat else None)
+        out.append(v)
+    return out
+
+
 def _build_hover(
     data: pd.DataFrame,
     ctx: RenderContext,
@@ -136,10 +166,32 @@ def render_control_chart(
 
     # 2. Main data trace
     customdata, hovertemplate = _build_hover(data, ctx, value_col)
+    lane_positions = _lane_positions(spec.lane_boundaries, ctx.chart_name, len(data))
+    if lane_positions:
+        # Lane chart: the connecting line must not cross a lane boundary. The last point
+        # of one subgroup joined to the first of the next drew a steep "event" at every
+        # boundary that was only the sort order (#121). Draw the line as its own trace
+        # with a gap at each boundary, under the markers; the markers trace keeps the
+        # hover, customdata and legend exactly as before.
+        _add_trace(
+            fig,
+            go.Scatter(
+                x=_with_gaps(list(x_data), lane_positions, repeat=True),
+                y=_with_gaps(list(data[value_col]), lane_positions),
+                mode='lines',
+                line=dict(color=theme.data_color, width=ctx.line_width),
+                opacity=theme.data_opacity,
+                hoverinfo='skip',
+                showlegend=False,
+                name=f'{ctx.chart_name} (lanes)',
+            ),
+            row,
+            col,
+        )
     trace_kw = dict(
         x=x_data,
         y=data[value_col],
-        mode='lines+markers',
+        mode='markers' if lane_positions else 'lines+markers',
         name=ctx.chart_name,
         marker=dict(size=ctx.marker_size, color=theme.data_color),
         line=dict(color=theme.data_color, width=ctx.line_width),