fix(legacy): read JudgeRubric questions from Pydantic messages - #2341
Open
crazywriter1 wants to merge 1 commit into
Open
fix(legacy): read JudgeRubric questions from Pydantic messages#2341crazywriter1 wants to merge 1 commit into
crazywriter1 wants to merge 1 commit into
Conversation
Rollouts normalize prompts to Message objects, but judge() only read dict content and silently sent an empty Question to the judge model.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #2005 — same fix after #2303 moved v0 rubrics to
verifiers/legacy/.Summary
JudgeRubric.judge()only extractedquestionfrom dict prompts (isinstance(..., dict)).Messageobjects, so the Question block was silently empty._message_field/_content_to_text) for dict and Message prompts.Test plan
{question}.contentno longer produces empty QuestionNote
Fix
JudgeRubric.judgeto read question content from Pydantic messagesPreviously,
JudgeRubric.judgein judge_rubric.py extracted the last message's content via raw dict access, which failed for Pydantic message objects. It now usesself.parser._message_fieldandself.parser._content_to_textto normalize the content, consistent with how the rest of the parser handles messages. Behavioral Change: the extracted question string used to formatjudge_prompt(and its cache key) may differ when prompts contain Pydantic message objects.Macroscope summarized adb8c94.