Add margin score_type to NaiveCP (BaseConformal)

docs/api/calib/pyhealth.calib.predictionset.rst

@@ -8,10 +8,14 @@ techniques for uncertainty quantification.
 ``BaseConformal``, ``LABEL``, ``ClusterLabel``, ``CovariateLabel``, and
 ``NeighborhoodLabel`` all accept a ``score_type`` argument selecting the
 nonconformity/conformity score used for calibration and set construction:
-either ``"threshold"`` (the default, unchanged from prior releases) or
-``"aps"`` (Adaptive Prediction Sets, Romano, Sesia, and Candes 2020), which
-adapts the prediction set size to the model's per-input confidence. See
-:mod:`pyhealth.calib.predictionset.scores` for the exact score formulas.
+``"threshold"`` (LAC, Sadinle, Lei, and Wasserman 2019 -- the default for
+every one of these classes) or ``"aps"`` (Adaptive Prediction Sets, Romano,
+Sesia, and Candes 2020, which adapts the prediction set size to the
+model's per-input confidence). ``BaseConformal`` additionally supports
+``"margin"`` (Papadopoulos, Vovk, and Gammerman 2007's own score for
+neural-network classifiers). Coverage validity does not depend on this
+choice -- see :mod:`pyhealth.calib.predictionset.scores` for the exact
+score formulas.
 ``SCRIB`` and ``FavMac`` are not included since their calibration
 procedures aren't a score-then-quantile pattern.
 
@@ -30,8 +34,26 @@ Available Methods
    pyhealth.calib.predictionset.ClusterLabel
    pyhealth.calib.predictionset.NeighborhoodLabel
 
-BaseConformal (Standard Split Conformal Prediction)
-----------------------------------------------------
+BaseConformal (Standard "Naive" Split Conformal Prediction)
+--------------------------------------------------------------
+
+.. note::
+
+   ``BaseConformal`` and ``LABEL`` both build on the same generic
+   split-conformal calibration machinery (Vovk, Gammerman, and Shafer
+   2005): sort calibration nonconformity scores and take the
+   ``ceil((1-alpha)*(n+1))``-th one as the threshold. By default they use
+   the identical score too -- ``"threshold"``, i.e. ``1 - p(true class)``,
+   from Sadinle, Lei, and Wasserman (2019) -- for consistency with every
+   other class in this module. ``BaseConformal`` additionally supports
+   ``score_type="margin"``, i.e. ``max_{j!=k} p(j) - p(k)``, the
+   nonconformity measure from Papadopoulos, Vovk, and Gammerman,
+   "Conformal prediction with neural networks," 19th IEEE ICTAI 2007,
+   vol. 2, pp. 388-395 (verified directly against Section 4.2 of
+   Papadopoulos's 2008 InTech chapter restating that paper, which calls
+   this "the natural nonconformity measure" for neural-network
+   classifiers). Coverage validity does not depend on this choice; it
+   only changes prediction-set composition and average set size.
 
 .. autoclass:: pyhealth.calib.predictionset.BaseConformal
    :members:

examples/conformal_eeg/tuev_conventional_conformal.py

@@ -17,6 +17,16 @@
   python examples/conformal_eeg/tuev_conventional_conformal.py \\
       --root downloads/tuev/v2.0.1/edf --model tfm --n-seeds 5 --seed 42 --alpha 0.1 \\
       --log-file tuev_conventional_tfm_alpha0.1_5seeds.log
+
+Note: this example specifically demonstrates LABEL (Sadinle, Lei, and
+Wasserman 2019's least-ambiguous-set score, "1 - p(true class)"), which is
+also pyhealth.calib.predictionset.BaseConformal's default score_type
+(both classes default to "threshold" for consistency). BaseConformal
+additionally supports score_type="margin", the nonconformity measure from
+Papadopoulos, Vovk, and Gammerman's "Conformal prediction with neural
+networks" (ICTAI 2007) -- pass BaseConformal(model, alpha,
+score_type="margin") if you specifically want that paper's own method.
+Coverage validity does not depend on which score_type is used.
 """
 
 from __future__ import annotations

pyhealth/calib/predictionset/base_conformal/__init__.py

@@ -1,8 +1,19 @@
 """
 Base Conformal Prediction (Split Conformal)
 
-Standard split conformal prediction for multiclass classification without
-covariate shift correction. 
+Standard "naive" split conformal prediction for multiclass classification
+without covariate shift correction. Like every other score-then-quantile
+class in this package (LABEL, ClusterLabel, CovariateLabel,
+NeighborhoodLabel), the default nonconformity score is "threshold"
+(Sadinle, Lei, and Wasserman 2019's least-ambiguous-set score), for
+consistency across the module. BaseConformal additionally supports
+score_type="margin", the nonconformity measure Papadopoulos, Vovk, and
+Gammerman defined specifically for neural-network classifiers -- pass this
+if you want that paper's own method rather than LABEL's score. Coverage
+validity does not depend on which score_type is chosen (Vovk, Gammerman,
+and Shafer 2005): all of them share the same split-conformal calibration
+machinery and guarantee, and differ only in prediction-set composition
+and average set size.
 
 This method constructs prediction sets with coverage guarantees by calibrating
 score thresholds on a held-out calibration set.
@@ -11,12 +22,15 @@
     Vovk, Vladimir, Alexander Gammerman, and Glenn Shafer.
     "Algorithmic learning in a random world." Springer, 2005.
 
-    Papadopoulos, Harris, Kostas Proedrou, Volodya Vovk, and Alex Gammerman.
-    "Inductive confidence machines for regression." ECML 2002.
-
     Sadinle, Mauricio, Jing Lei, and Larry Wasserman. "Least ambiguous
     set-valued classifiers with bounded error levels." Journal of the
-    American Statistical Association (2019). [score_type="threshold"]
+    American Statistical Association (2019). [score_type="threshold",
+    the default]
+
+    Papadopoulos, Harris, Vladimir Vovk, and Alexander Gammerman.
+    "Conformal prediction with neural networks." 19th IEEE International
+    Conference on Tools with Artificial Intelligence (ICTAI 2007), vol. 2,
+    pp. 388-395. IEEE, 2007. [score_type="margin"]
 
     Romano, Yaniv, Matteo Sesia, and Emmanuel Candes. "Classification with
     valid and adaptive coverage." NeurIPS 2020. [score_type="aps"]
@@ -95,35 +109,59 @@ def _query_weighted_quantile(
 
 
 class BaseConformal(SetPredictor):
-    """Base Conformal Prediction for multiclass classification.
+    """Base ("naive") Conformal Prediction for multiclass classification.
 
     This implements standard split conformal prediction, which constructs
     prediction sets with distribution-free coverage guarantees. The method
     calibrates thresholds on a calibration set and uses them to construct
     prediction sets on test data.
 
+    By default this uses the same "threshold" nonconformity score as every
+    other class in this package (LABEL, ClusterLabel, CovariateLabel,
+    NeighborhoodLabel): Sadinle, Lei, and Wasserman's (2019) ``1 - p(k)``
+    score. BaseConformal additionally supports score_type="margin", the
+    nonconformity measure Papadopoulos, Vovk, and Gammerman (2007) defined
+    for neural-network classifiers: ``alpha(x, k) = max_{j != k} p(j) -
+    p(k)``, i.e. how much the strongest competing class beats k. Coverage
+    validity does not depend on this choice (Vovk, Gammerman, and Shafer
+    2005): any nonconformity score, under the same rank-based calibration
+    procedure, gives the same coverage guarantee -- score_type only changes
+    prediction-set composition and average set size, not validity.
+
     The method guarantees that:
     - For marginal coverage (alpha is float): P(Y not in C(X)) <= alpha
     - For class-conditional coverage (alpha is array): P(Y not in C(X) | Y=k) <= alpha[k]
 
-    where C(X) denotes the prediction set for input X.
+    where C(X) denotes the prediction set for input X. This holds regardless
+    of score_type, since validity of split conformal prediction does not
+    depend on which nonconformity measure is used (only the usefulness /
+    average set size does).
 
     Papers:
         Vovk, Vladimir, Alexander Gammerman, and Glenn Shafer.
         "Algorithmic learning in a random world." Springer, 2005.
 
-        Lei, Jing, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani,
-        and Larry Wasserman. "Distribution-free predictive inference for
-        regression." Journal of the American Statistical Association (2018).
+        Sadinle, Mauricio, Jing Lei, and Larry Wasserman. "Least ambiguous
+        set-valued classifiers with bounded error levels." Journal of the
+        American Statistical Association (2019).
+
+        Papadopoulos, Harris, Vladimir Vovk, and Alexander Gammerman.
+        "Conformal prediction with neural networks." 19th IEEE ICTAI 2007,
+        vol. 2, pp. 388-395.
 
     Args:
         model: A trained base model that outputs predicted probabilities
         alpha: Target miscoverage rate(s). Can be:
             - float: marginal coverage P(Y not in C(X)) <= alpha
             - array: class-conditional P(Y not in C(X) | Y=k) <= alpha[k]
         score_type: Type of nonconformity score to use:
-            - "threshold" (default): NC score = 1 - p(true class), the score
-              from Sadinle, Lei, and Wasserman (2019) ("LABEL").
+            - "threshold" (default): NC score = 1 - p(true class), the
+              score from Sadinle, Lei, and Wasserman (2019) ("LABEL") --
+              matches :class:`~pyhealth.calib.predictionset.LABEL`'s
+              behavior, and every other class in this package's default.
+            - "margin": NC score = max_{j!=k} p(j) - p(k), the score from
+              Papadopoulos, Vovk, and Gammerman (2007) ("naive" split
+              conformal prediction for neural networks).
             - "aps": Adaptive Prediction Sets (Romano, Sesia, and Candes
               2020). NC score for class k is the cumulative sum of predicted
               probabilities for classes ranked above k, plus a randomized
@@ -134,7 +172,8 @@ class BaseConformal(SetPredictor):
               individual input. See :mod:`pyhealth.calib.predictionset.scores`
               for the exact formula.
         random_state: Optional int seed for the RNG used by score_type="aps"
-            (the U ~ Uniform(0,1) draws). Ignored for score_type="threshold".
+            (the U ~ Uniform(0,1) draws). Ignored for score_type="threshold"
+            and score_type="margin".
         debug: Whether to use debug mode (processes fewer samples)
 
     Examples:
@@ -181,6 +220,12 @@ class BaseConformal(SetPredictor):
         >>> conformal_model_aps = BaseConformal(
         ...     model, alpha=0.1, score_type="aps", random_state=0)
         >>> conformal_model_aps.calibrate(cal_dataset=val_data)
+        >>>
+        >>> # Use Papadopoulos, Vovk, and Gammerman (2007)'s own margin
+        >>> # score instead of the default threshold (LABEL) score
+        >>> conformal_model_margin = BaseConformal(
+        ...     model, alpha=0.1, score_type="margin")
+        >>> conformal_model_margin.calibrate(cal_dataset=val_data)
     """
 
     def __init__(

pyhealth/calib/predictionset/scores.py

@@ -2,10 +2,11 @@
 
 This module separates the *score* used by a conformal-prediction-set method
 from the *calibration/thresholding procedure* it's plugged into. These are
-two independent axes: the choice of score ("threshold"/LAC vs "aps") does
-not depend on how the resulting scores get turned into a threshold (marginal
-quantile, per-class quantile, per-cluster quantile, weighted quantile for
-covariate shift, or localized weighted quantile for neighborhood methods).
+two independent axes: the choice of score ("threshold"/LAC vs "aps" vs
+"margin") does not depend on how the resulting scores get turned into a
+threshold (marginal quantile, per-class quantile, per-cluster quantile,
+weighted quantile for covariate shift, or localized weighted quantile for
+neighborhood methods).
 
 Supported score types:
 
@@ -31,9 +32,26 @@
       to the model's confidence for each individual input, which the
       "threshold" score does not.
 
-Both scores are computed here in *nonconformity* convention (higher = less
-conforming, i.e. 1 minus a probability-like quantity) since that's the
-convention BaseConformal/LABEL/ClusterLabel use internally. A *conformity*
+    - "margin" (Papadopoulos, Vovk, and Gammerman, "Conformal Prediction
+      with Neural Networks," 19th IEEE ICTAI 2007, vol. 2, pp. 388-395;
+      restated in Section 4.2 of Papadopoulos, "Inductive Conformal
+      Prediction: Theory and Application to Neural Networks," Tools in
+      Artificial Intelligence, InTech, 2008, as the "natural nonconformity
+      measure" for neural-network classifiers): the score for class k is
+      how much the best *other* class beats k::
+
+          alpha(x, k) = max_{j != k} pi(x, j) - pi(x, k)
+
+      Unlike "threshold" (which only looks at k's own probability), this
+      measure the margin between k and its strongest competitor, so it is
+      more nonconforming for an example whose true class is only narrowly
+      ahead of a rival than for one that's ahead by a landslide, even at
+      the same raw probability for k.
+
+Both "threshold" and "aps" scores are computed here in *nonconformity*
+convention (higher = less conforming, i.e. 1 minus a probability-like
+quantity) since that's the convention BaseConformal/LABEL/ClusterLabel use
+internally; "margin" is nonconformity-signed by construction. A *conformity*
 (higher = more conforming) variant is also provided for CovariateLabel/
 NeighborhoodLabel, which use the opposite sign convention internally; it is
 simply `1 - nonconformity`, preserving the same ranking of examples either
@@ -50,7 +68,7 @@
     "true_class_nc_scores",
 ]
 
-SUPPORTED_SCORE_TYPES = ("threshold", "aps")
+SUPPORTED_SCORE_TYPES = ("threshold", "aps", "margin")
 
 
 def _validate_score_type(score_type: str) -> None:
@@ -112,6 +130,38 @@ def _aps_all_class_nc_scores(
     return scores
 
 
+def _margin_all_class_nc_scores(y_prob: np.ndarray) -> np.ndarray:
+    """Computes the margin nonconformity score for every class, every row.
+
+    alpha(x, k) = max_{j != k} pi(x, j) - pi(x, k)
+
+    (Papadopoulos, Vovk, and Gammerman 2007, Eq. for the "natural
+    nonconformity measure"; see module docstring.) Vectorized via each
+    row's top-2 probabilities, with ties at the row max handled so that a
+    class tied for the top spot still has another class achieving that same
+    max value as its "best other class."
+
+    Args:
+        y_prob: Predicted probabilities, shape (N, K).
+
+    Returns:
+        Nonconformity scores of shape (N, K); higher means less conforming.
+    """
+    n, k = y_prob.shape
+    order = np.argsort(-y_prob, axis=1)
+    sorted_probs = np.take_along_axis(y_prob, order, axis=1)
+    top1 = sorted_probs[:, 0]
+    top2 = sorted_probs[:, 1] if k >= 2 else np.full(n, -np.inf)
+    is_top = y_prob == top1[:, None]
+    num_top = is_top.sum(axis=1, keepdims=True)
+    max_others = np.where(
+        is_top,
+        np.where(num_top > 1, top1[:, None], top2[:, None]),
+        top1[:, None],
+    )
+    return max_others - y_prob
+
+
 def all_class_nc_scores(
     y_prob: np.ndarray,
     score_type: str = "threshold",
@@ -122,12 +172,13 @@ def all_class_nc_scores(
 
     Args:
         y_prob: Predicted probabilities, shape (N, K).
-        score_type: "threshold" (Sadinle, Lei, and Wasserman 2019) or "aps"
-            (Romano, Sesia, and Candes 2020). Default "threshold".
+        score_type: "threshold" (Sadinle, Lei, and Wasserman 2019), "aps"
+            (Romano, Sesia, and Candes 2020), or "margin" (Papadopoulos,
+            Vovk, and Gammerman 2007). Default "threshold".
         rng: Random generator, required (and only used) if score_type="aps"
             and randomize=True.
         randomize: Whether to use the randomized ("exact coverage") variant
-            of APS. Ignored for score_type="threshold".
+            of APS. Ignored for score_type="threshold" and "margin".
 
     Returns:
         Nonconformity scores of shape (N, K).
@@ -144,10 +195,15 @@ def all_class_nc_scores(
         >>> np.round(scores, 2)
         array([[0.42, 0.82, 0.96],
                [0.72, 0.36, 0.95]])
+        >>> all_class_nc_scores(y_prob, score_type="margin")
+        array([[-0.5,  0.5,  0.6],
+               [ 0.2, -0.2,  0.3]])
     """
     _validate_score_type(score_type)
     if score_type == "threshold":
         return 1.0 - y_prob
+    if score_type == "margin":
+        return _margin_all_class_nc_scores(y_prob)
     # score_type == "aps"
     if rng is None:
         rng = np.random.default_rng()