Skip to content

fix(legacy): read JudgeRubric questions from Pydantic messages - #2341

Open
crazywriter1 wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
crazywriter1:fix/judge-rubric-pydantic-question-legacy
Open

fix(legacy): read JudgeRubric questions from Pydantic messages#2341
crazywriter1 wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
crazywriter1:fix/judge-rubric-pydantic-question-legacy

Conversation

@crazywriter1

@crazywriter1 crazywriter1 commented Aug 12, 2026

Copy link
Copy Markdown

Supersedes #2005 — same fix after #2303 moved v0 rubrics to verifiers/legacy/.

Summary

  • JudgeRubric.judge() only extracted question from dict prompts (isinstance(..., dict)).
  • Rollouts normalize prompts to Pydantic Message objects, so the Question block was silently empty.
  • Read content via parser helpers (_message_field / _content_to_text) for dict and Message prompts.

Test plan

  • Manual: dict prompt still yields last user content as {question}
  • Manual: Pydantic-like message with .content no longer produces empty Question
  • Manual: multimodal text parts joined into question string

Note

Fix JudgeRubric.judge to read question content from Pydantic messages

Previously, JudgeRubric.judge in judge_rubric.py extracted the last message's content via raw dict access, which failed for Pydantic message objects. It now uses self.parser._message_field and self.parser._content_to_text to normalize the content, consistent with how the rest of the parser handles messages. Behavioral Change: the extracted question string used to format judge_prompt (and its cache key) may differ when prompts contain Pydantic message objects.

Macroscope summarized adb8c94.

Rollouts normalize prompts to Message objects, but judge() only read dict content and silently sent an empty Question to the judge model.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant