Update the EnsembleDetector with multiple voting system - #114
Conversation
3a90e5e to
c136e71
Compare
c136e71 to
c22f787
Compare
|
|
||
| def load_drift_detector(cfg: Config) -> BaseDriftDetector: | ||
| """Dynamically load and instantiate a drift detector based on its name. | ||
| def _build_detector(detector_name: str, cfg: Config) -> BaseDriftDetector: |
There was a problem hiding this comment.
The Config still should still hold the detector name in it. We should continue to derive detector name from the config instead of accepting it separately. It's slightly less error prone and a bit easier to use.
Changing it to this means every caller will do:
detector_name = cfg.drift_detection.detector_name
_build_detector(detector_name, cfg)
instead of:
_build_detector(cfg)
There was a problem hiding this comment.
The load_drift_detection function is still using the detector name set in config. This change is necessary (i.e. splitting the load function into _build_detector that takes a given detector) to cover the EnsembleDetector. For the EnsembleDetector the config is like this:
detector_name = "EnsembleDetector"
ensemble_detectors = ["ADWINDetector", "KSWINDetector", "PageHinkleyDetector"]
ensemble_voting = "majority"
which means we are calling _build_detector for each detector in the list.
There was a problem hiding this comment.
Okay so for the ensemble detector, the name that we're passing to buld_detector will differ from the name in the config? That's what it looks like based on the below code snippet. Odd behavior but it's an internal only method so I suppose it's fine.
detectors=[_build_detector(name, cfg) for name in sub_names]
|
|
||
| def load_drift_detector(cfg: Config) -> BaseDriftDetector: | ||
| """Dynamically load and instantiate a drift detector based on its name. | ||
| def _build_detector(detector_name: str, cfg: Config) -> BaseDriftDetector: |
There was a problem hiding this comment.
Okay so for the ensemble detector, the name that we're passing to buld_detector will differ from the name in the config? That's what it looks like based on the below code snippet. Odd behavior but it's an internal only method so I suppose it's fine.
detectors=[_build_detector(name, cfg) for name in sub_names]
|
@andrewfayres we basically check if the name is detector_name = cfg.drift_detection.detector_name
if detector_name != "EnsembleDetector":
return _build_detector(detector_name, cfg)The other option was to allow the name of the detector to be a list, in which case we would still need to handle it differently depending on if a string or a list was given. Keeping a separate |
Co-authored-by: Rafael Zamora-Resendiz <15003285+rz4@users.noreply.github.com>
af52942 to
416dc1c
Compare
Adds the singles-vs-ensemble result: over the identical 343-window trace, ADWIN and Page-Hinkley never fire, KSWIN fires four times, and an any-vote ensemble of all three fires at exactly KSWIN's steps. The shift is in the distribution of the per-window error rather than its mean, which is what the two mean-based detectors miss. The ensemble is reported as the negative result it is: no false alarms added, no sensitivity gained. The figure follows #114's layout -- regimes shaded and named, and a headline block giving the pretrained base model against the adapted one. The percentage now reads "lower" rather than carrying a sign that looked like error rising, and the per-adaptation drops say they are measured on the arriving bundle's validation split, not on the streamed windows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the singles-vs-ensemble result: over the identical 343-window trace, ADWIN and Page-Hinkley never fire, KSWIN fires four times, and an any-vote ensemble of all three fires at exactly KSWIN's steps. The shift is in the distribution of the per-window error rather than its mean, which is what the two mean-based detectors miss. The ensemble is reported as the negative result it is: no false alarms added, no sensitivity gained. The figure follows #114's layout -- regimes shaded and named, and a headline block giving the pretrained base model against the adapted one. The percentage now reads "lower" rather than carrying a sign that looked like error rising, and the per-adaptation drops say they are measured on the arriving bundle's validation split, not on the streamed windows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the singles-vs-ensemble result: over the identical 343-window trace, ADWIN and Page-Hinkley never fire, KSWIN fires four times, and an any-vote ensemble of all three fires at exactly KSWIN's steps. The shift is in the distribution of the per-window error rather than its mean, which is what the two mean-based detectors miss. The ensemble is reported as the negative result it is: no false alarms added, no sensitivity gained. The figure follows #114's layout -- regimes shaded and named, and a headline block giving the pretrained base model against the adapted one. The percentage now reads "lower" rather than carrying a sign that looked like error rising, and the per-adaptation drops say they are measured on the arriving bundle's validation split, not on the streamed windows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The
EnsembleDetectorexisted before but was not usable, it also only accepted a voting majority as the strategy to decide when to fire. This PR extends it so it can accept two more strategies (anyreacts to the first detector that fires,unanimousrequires full agreement,majoritysits in between).This PR also fixes the lack of configuration options that made the
EnsembleDetectornot usable. After the current PR toml files can include the following options:EnsembleDetectorbuilds each name inensemble_detectorsfrom the same[drift_detection]block (so a detector type can appear at most once). An unknown voting name or an empty detector list raisesValueError.SLAC model harness (from PR #83) with EWC and KSWin fires 4 times:

with PageHinkley fires 2 times:

And with both (and

anystrategy) fires 6 times: