Skip to content

fix(evaluation): resolve threshold via criterion in LlmAsJudge, not the deprecated EvalMetric.threshold - #6678

Open
gaurav-gandhi-2411 wants to merge 1 commit into
google:mainfrom
gaurav-gandhi-2411:fix/eval-metric-threshold-criterion-resolution
Open

fix(evaluation): resolve threshold via criterion in LlmAsJudge, not the deprecated EvalMetric.threshold#6678
gaurav-gandhi-2411 wants to merge 1 commit into
google:mainfrom
gaurav-gandhi-2411:fix/eval-metric-threshold-criterion-resolution

Conversation

@gaurav-gandhi-2411

Copy link
Copy Markdown

🔴 Required Information

Describe the Bug:
LlmAsJudge.evaluate_invocations reads self._eval_metric.threshold directly, which is the deprecated field (EvalMetric.threshold, defaults to None, docstring: "This field will be deprecated soon. Please use criterion instead."). Configuring the metric only via criterion — the documented, non-deprecated path — leaves this field None and crashes.

Steps to Reproduce:

  1. Construct an EvalMetric with criterion=LlmAsAJudgeCriterion(threshold=0.5, ...) and no top-level threshold.
  2. Run any LlmAsJudge subclass's evaluate_invocations.

Expected Behavior: Evaluation completes using the criterion's threshold.

Observed Behavior:

TypeError: '>=' not supported between instances of 'float' and 'NoneType'

raised from llm_as_judge_utils.get_eval_status, called with threshold=None.

Minimal Reproduction Code:

EvalMetric(
    metric_name="my_metric",
    criterion=LlmAsAJudgeCriterion(threshold=0.5, judge_model_options=...),
)
# threshold is left at its default (None); evaluate_invocations crashes.

Why this fix

_get_metric_threshold() (added in 2510141) is the established pattern for resolving this exact deprecated-field-vs-criterion ambiguity, already used by 8 other evaluators (trajectory_evaluator.py, response_evaluator.py, safety_evaluator.py, final_response_match_v1.py, and the multi_turn_* evaluators). This call site was missed when that pattern was introduced. This PR routes LlmAsJudge through the same helper.

Testing Plan

Added test_evaluate_invocations_with_criterion_only_threshold, which constructs the metric via criterion only and asserts evaluate_invocations completes without raising. Confirmed this test fails on main with the exact TypeError above, and passes after the fix.

tests/unittests/evaluation/test_llm_as_judge.py: 6 passed
tests/unittests/evaluation/: 784 passed

…he deprecated EvalMetric.threshold

LlmAsJudge.evaluate_invocations read self._eval_metric.threshold
directly, the deprecated field (defaults to None). Configuring the
metric only via criterion -- the documented, non-deprecated path --
left this field None and crashed with TypeError: '>=' not supported
between instances of 'float' and 'NoneType'.

_get_metric_threshold() is the established pattern for this exact
resolution, already used by 8 other evaluators since 2510141. This
call site was missed. Routes LlmAsJudge through the same helper.
@adk-bot adk-bot added the eval [Component] This issue is related to evaluation label Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval [Component] This issue is related to evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants