On the default TAP 1–10 scale, if the judge answers 0 or 11, SelfAskScaleScorer fails after one call instead of retrying:
RuntimeError: Error in scorer SelfAskScaleScorer: ... Float scale scorers must have a score value between 0 and 1. Got -0.1111111111111111
The response handler only checks that the value is a finite float. The range check happens later, when the Score is built, which is outside the JSON retry. So one sloppy answer ends the attack, and this is the default scorer for TAP and Crescendo. SelfAskGeneralFloatScaleScorer has the same problem. Likert and the true/false scorers already validate their score domain inside the parser, so a bad value there just gets retried.
Suggested fix: a small NumericRangeResponseHandler wrapper next to TrueFalseResponseHandler, used by both scale scorers.
On the default TAP 1–10 scale, if the judge answers
0or11,SelfAskScaleScorerfails after one call instead of retrying:The response handler only checks that the value is a finite float. The range check happens later, when the
Scoreis built, which is outside the JSON retry. So one sloppy answer ends the attack, and this is the default scorer for TAP and Crescendo.SelfAskGeneralFloatScaleScorerhas the same problem. Likert and the true/false scorers already validate their score domain inside the parser, so a bad value there just gets retried.Suggested fix: a small
NumericRangeResponseHandlerwrapper next toTrueFalseResponseHandler, used by both scale scorers.