classify_error_type() in evaluation/scripts/PrefEval/pref_eval.py returns Personalized Response for inputs that do not match its error checks. This includes failed judge calls and valid answers that should count as errors.
Three examples reproduce the problem. Each cell below is the answer value under that key in evaluation_results:
violate_preference |
acknowledge_preference |
hallucinate_preference |
helpful_response |
Expected result |
"" |
"" |
"" |
"" |
Judge Failure |
"yes" |
"no" |
"No" |
"yes" |
Preference-Unaware Violation |
"Yes" |
"No" |
"No" |
"No" |
Unhelpful Response |
The current function returns Personalized Response for all three rows.
The first case can occur after an API error: call_gpt4o_mini_async() returns an empty string, and parse_xml_response() also returns an empty answer. The classifier has no check for this failure.
The second case fails because the checks require exact Yes and No strings. The third fails because the unhelpful branch also requires violate_preference == "No".
These cases can inflate the reported Personalized Response percentage. Failed judge answers should have a separate result, case should not affect the label, and an unhelpful response should remain unhelpful even if it also violates a preference.
Proposed fix: #2397.
classify_error_type()inevaluation/scripts/PrefEval/pref_eval.pyreturnsPersonalized Responsefor inputs that do not match its error checks. This includes failed judge calls and valid answers that should count as errors.Three examples reproduce the problem. Each cell below is the
answervalue under that key inevaluation_results:violate_preferenceacknowledge_preferencehallucinate_preferencehelpful_response""""""""Judge Failure"yes""no""No""yes"Preference-Unaware Violation"Yes""No""No""No"Unhelpful ResponseThe current function returns
Personalized Responsefor all three rows.The first case can occur after an API error:
call_gpt4o_mini_async()returns an empty string, andparse_xml_response()also returns an empty answer. The classifier has no check for this failure.The second case fails because the checks require exact
YesandNostrings. The third fails because the unhelpful branch also requiresviolate_preference == "No".These cases can inflate the reported
Personalized Responsepercentage. Failed judge answers should have a separate result, case should not affect the label, and an unhelpful response should remain unhelpful even if it also violates a preference.Proposed fix: #2397.