Add margin score_type to NaiveCP (BaseConformal) - #1220
Open
lehendo wants to merge 1 commit into
Open
Conversation
… 2007), keep threshold as the uniform default across all CP classes
BaseConformal's docstring claimed to implement 'naive' split conformal
prediction per Papadopoulos, Vovk, and Gammerman (2007), but its only
score_type="threshold" computed 1 - p(true class) -- Sadinle, Lei, and
Wasserman's (2019) exact LABEL/LAC formula -- with no way to actually get
that paper's own method through this class.
Verified against the primary source (Papadopoulos's 2008 InTech chapter,
which explicitly restates the ICTAI 2007 paper's Section 4 content): the
paper's actual nonconformity measure for neural-network classifiers is a
margin, max_{j!=k} p(j) - p(k) ("the natural nonconformity measure").
Added this as a new score_type="margin" in scores.py, available on
BaseConformal alongside the existing "threshold"/"aps" options.
score_type="threshold" remains BaseConformal's default, consistent with
every other score-then-quantile class in the package (LABEL, ClusterLabel,
CovariateLabel, NeighborhoodLabel) -- coverage validity never depended on
score choice (Vovk, Gammerman, and Shafer 2005), so this is purely about
giving BaseConformal a real, working way to reproduce Papadopoulos et
al.'s specific paper on request, without changing any class's default
behavior.
Contributor
|
lgtm |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title is self explanatory.