Update loo_compare() to support new pred_measure API

.gitignore

@@ -30,3 +30,5 @@ release-prep.R
 
 # personal maintainer scratch (not shared)
 internal-notes/
+notes/loo_se.pdf
+notes/loo-compare-se-diff.md

NEWS.md

@@ -7,6 +7,10 @@
 * New predictive performance API: `insample_pred_measure()`, `loo_pred_measure()`,
   `kfold_pred_measure()`, `test_pred_measure()`, and `pred_measure()` with
   built-in measures via `measure_*()` and [supported_measures_list()].
+* `loo_compare()` now supports `loo_pred_measure` objects: paired differences
+  for all measures common to the compared models, optional `rank_by` ranking,
+  utility-scale sign conversion for loss measures, and
+  `print(compare, measures = ...)` for multi-measure tables by @florence-bockting in #380.
 
 # loo 2.10.0
 

R/loo-glossary.R

@@ -254,4 +254,108 @@
 #'
 #' See for further information on Pareto-k values the "Pareto k estimates"
 #' section.
+#'
+#' @section Multi-measure model comparisons:
+#'
+#' When comparing [`loo_pred_measure()`][loo_pred_measure] objects with
+#' `loo_compare()`, paired differences are computed for every predictive
+#' measure common to all models. Models are ranked by the `rank_by` argument
+#' (default `"elpd"`); the top-ranked model is the reference for all difference
+#' columns.
+#'
+#' ### `{measure}_diff` and `{measure}_se_diff`
+#'
+#' For each non-ELPD measure `m`, `loo_compare()` adds columns `m_diff` and
+#' `m_se_diff`. In all cases `m_diff` is the difference between the two overall
+#' estimates on a utility scale (higher is better; loss measures such as MSE,
+#' Brier score, and SRPS have their sign flipped from the raw loss orientation).
+#' Measures already returned on a utility scale (e.g. ELPD, CRPS/RPS) are not
+#' sign-flipped. Negative `m_diff` values then indicate worse performance than
+#' the reference model, which has `m_diff = 0`.
+#'
+#' How `m_se_diff` is obtained depends on the measure:
+#'
+#' * When the overall estimate is a sum or mean of pointwise contributions, it
+#'   is computed from paired pointwise differences using the same approach as
+#'   `elpd_diff` and `se_diff` (Eq 24 in VGG2017 for sums; the mean analogue for
+#'   means). This covers ELPD, `mlpd`, `ic`, `mae`, `mse`, `acc`, `brier`, and
+#'   the ranked probability scores.
+#' * When it is a transformation of such quantities, the measure supplies its
+#'   own delta-method standard error (`se_diff_fun`). For `rmse` this is the
+#'   first-order bivariate Taylor approximation propagated from the MSE scale,
+#'   which requires the covariance between the two models' pointwise squared
+#'   errors and is therefore not a paired pointwise standard deviation. For
+#'   `r2` it is the trivariate analogue, which additionally propagates the
+#'   uncertainty in the baseline `MSE(y)` shared by both models.
+#' * Otherwise `m_se_diff` is `NA`. This applies only to custom measures whose
+#'   estimate is neither a sum nor a mean of their pointwise values and which do
+#'   not attach an `se_diff_fun`.
+#'
+#' The reference model has `m_se_diff = 0` whenever an `m_se_diff` is available.
+#' Attribute `measure_higher_is_better` on each `*_pred_measure()`
+#' result records the `higher_is_better` setting used when each measure was
+#' computed; when stored values are on a loss scale, `loo_compare()` emits a
+#' short message naming those measures (see [loo_compare()]).
+#'
+#' ELPD-family measures use the column names `elpd_diff` and `se_diff` rather
+#' than a prefixed form. Only ELPD comparisons include `p_worse` and `diag_diff`;
+#' these diagnostics do not apply to other predictive measures.
+#'
+#' ### `measure_higher_is_better`
+#'
+#' Attribute on all `*_pred_measure()` and [pred_measure()] results: a named
+#' list recording the `higher_is_better` setting used for each measure (`NULL`,
+#' `TRUE`, or `FALSE`; `elpd` is always `NULL`). Used by [loo_compare()] with
+#' `measure_compare_meta` to decide whether paired differences need a sign flip
+#' when converting to a utility scale.
+#'
+#' ### `measure_compare_meta`
+#'
+#' Attribute on all `*_pred_measure()` and [pred_measure()] results: a named
+#' list of per-measure comparison metadata used by [loo_compare()]. Each entry
+#' is a list with:
+#'
+#' * `higher_is_better` — the orientation setting used when the measure was
+#'   computed (`NULL`, `TRUE`, or `FALSE`)
+#' * `loss` — whether stored values are on a loss scale (lower is better)
+#' * `diff_method` — how the standard error of the difference is obtained:
+#'   `"sum"` or `"mean"` (paired pointwise differences), `"pairwise"` (the
+#'   measure's own `se_diff_fun`), `"estimates_only"` (unavailable, `NA`), or
+#'   `"auto"` (inferred at compare time for custom measures). No built-in
+#'   measure declares `"estimates_only"`; it is reached only when
+#'   autodetection under `"auto"` cannot establish that the estimate is a sum
+#'   or mean of its pointwise values. It is not an error state — the
+#'   difference itself is still reported, and only its standard error is
+#'   marked unavailable.
+#' * `se_diff_fun` — for `diff_method = "pairwise"`, either the name of a
+#'   built-in implementation or, for custom measures, the function itself
+#' * `extra` — optional list of auxiliary data the measure stored for its
+#'   `se_diff_fun`, present only for measures that need it (`r2` stores the
+#'   pointwise baseline `(y_i - mean(y))^2`, which `y` no longer supplies by
+#'   the time [loo_compare()] runs; `bacc` stores the class index of each
+#'   observation, which its pointwise values do not determine). It is excluded
+#'   from the metadata consistency check below, since it varies with the data
+#'   rather than with how the measure was configured.
+#'
+#' Built-in measures take `loss`, `diff_method`, and `se_diff_fun` from the
+#' package measure registry; custom measures default to `loss = FALSE` and
+#' `diff_method = "auto"` unless they attach an `se_diff_fun` (see
+#' [insample_pred_measure()]).
+#' [loo_compare()] requires all models to provide matching metadata for each
+#' shared measure; mismatched `higher_is_better` settings or missing metadata on
+#' some models produce an error.
+#'
+#' ### `rank_by`, `compare_measures`, and related attributes`
+#'
+#' The `rank_by` argument selects which measure determines model ordering and
+#' the reference model for all pairwise differences. When `rank_by` is omitted,
+#' models are ranked by `"elpd"`; attribute `rank_by` is set only when `rank_by`
+#' is passed explicitly. Attribute `compare_measures` lists all measures that
+#' were compared, and `sign_converted_measures` lists loss measures whose sign
+#' was flipped onto the utility scale. The print method shows the ranking
+#' measure by default
+#' (`"elpd"` when `rank_by` was not set); use `print(x, measures = "all")` or
+#' `print(x, measures = c("rmse", "r2"))` to display additional measure tables.
+#' Printed tables label the standard-error column `se_diff` even for non-ELPD
+#' measures; the data frame columns remain `{measure}_se_diff`.
 NULL