ClusterLabel audit + tests - #1221
Open
lehendo wants to merge 1 commit into
Open
Conversation
…fied K-means design choice, add coverage tests ClusterLabel had no paper citation at all, despite being a legitimate, correctly-implemented instance of Mondrian conformal prediction (Vovk, Lindsay, Nouretdinov, and Gammerman, 'Mondrian confidence machine,' 2003) using K-means clusters as the category function. Added that citation plus Vovk/Gammerman/Shafer 2005, and documented the actual guarantee this class provides: per-cluster coverage (P(Y not in C(X) | cluster=c) <= alpha for every cluster), which is strictly stronger than plain marginal coverage -- the previous docstring didn't call this out. Investigated a theoretical concern by analogy to the NeighborhoodLabel self-inclusion bug fixed earlier this session: calibrate() fits KMeans on train+cal embeddings combined, so calibration points influence the very cluster centroids used to assign their own threshold, unlike a strict split-conformal setup where the category function would be fit on data disjoint from calibration. Verified via Monte Carlo simulation (including a calibration-set-dominated stress test, 10 train vs 300 cal points) that this does NOT introduce measurable coverage bias, unlike k-NN's self- inclusion (a hard, always-occurring artifact) -- a single point's leverage on a K-means centroid, an average over many points, is negligible. Documented this as a deliberate, verified design choice rather than leaving it unexamined, and added regression tests locking in both the per-cluster coverage guarantee and the train+cal-vs-train-only equivalence finding.
Contributor
|
The documentation promises coverage for each cluster separately, but the clusters are built using the same calibration patients that are later used to set each cluster's threshold. That means the calibration data helps define the groups and choose their thresholds, so the usual finite-sample guarantee does not apply. Fit K-means on training embeddings only, then assign calibration patients with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title is self explanatory.