Skip to content

Python: Preserve numeric evaluator scores in LocalEvaluator - #10383

Open
Qiming Bao (14H034160212) wants to merge 2 commits into
microsoft:mainfrom
14H034160212:fix-10073-numeric-scores
Open

Qiming Bao (14H034160212) wants to merge 2 commits into
microsoft:mainfrom
14H034160212:fix-10073-numeric-scores

Conversation

@14H034160212

Copy link
Copy Markdown

Motivation & Context

LocalEvaluator.evaluate() recorded every check score as 1.0 or 0.0, based only on passed. A function evaluator that returns a numeric score (a float, or a dict with score) lost that number, which _coerce_result() kept only inside the reason string.

As a result, EvalResults.assert_score_at_least() gated on the pass flag rather than on the evaluator's score. An LLM-as-judge returning 0.6 passes the default 0.5 threshold, is stored as 1.0, and assert_score_at_least(0.8) returns normally. In addition, assert_score_at_least() tested score < min_score, which is False for NaN, so a provider that reports NaN (for example a judge whose output failed to parse) also passed the gate. Both are silent passes in a check meant for CI quality gates.

Description & Review Guide

  • What are the major changes?
    • CheckResult gets an optional score: float | None = None. It is the last field and has a default, so existing CheckResult(passed=..., reason=..., check_name=...) calls are unaffected.
    • _coerce_result() sets it for numeric returns and for dicts with score.
    • LocalEvaluator.evaluate() records result.score when present. Boolean checks (keyword_check, tool_called_check, ...) keep mapping to 1.0 / 0.0.
    • assert_score_at_least() uses not score.score >= min_score, so NaN counts as an offender. The recent check for an evaluator filter that matches no scores is kept.
  • What is the impact of these changes? No breaking change. Pass/fail counts and boolean checks are unchanged. For function evaluators that return a number, EvalScoreResult.score now holds that number, as its docstring ("Numeric score from the evaluator") says, so assert_score_at_least() can fail where it silently passed before.
  • What do you want reviewers to focus on? The new public field on CheckResult. The alternative raised in .NET: Python: [Bug]: LocalEvaluator drops numeric evaluator scores, so assert_score_at_least() passes below-threshold (and NaN) scores #10073 is to keep local scores 0/1 and document it; I'm happy to switch if you prefer that.

Tests (in python/packages/core/tests/core/test_local_eval.py):

  • test_numeric_evaluator_scores_are_preserved: float and {"score": ...} returns are stored as-is.
  • test_numeric_evaluator_score_below_gate_raises: a 0.6 score fails assert_score_at_least(0.8).
  • test_nan_score_raises: a NaN score fails the gate.

All three fail on main and pass with this change. After rebasing on the latest main, test_local_eval.py passes (all tests), ruff check and ruff format --check pass on the changed files, and the core suite gives 7722 passed, 25 skipped, 1 failed. The one failure, test_feature_stage.py::test_feature_id_allows_lowercase_values, also fails on main in my environment and is unrelated.

Related Issue

Fixes #10073

The issue is labelled reproduced: the triage bot reproduced both problems with the same three tests. I proposed this direction on the issue on 2026-11-05 and haven't had a reply yet, so I'm opening the PR to make review easier. No other PR is open for this issue.

AI Assistance

  • No material AI assistance was used.
  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: Claude (Anthropic) helped with the implementation, tests and this PR description.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

LocalEvaluator recorded every check score as 1.0 or 0.0 based only on
the pass/fail flag, so a function evaluator returning a numeric score
(a float, or a dict with "score") lost that value. As a result,
EvalResults.assert_score_at_least() gated on the pass flag rather than
on the evaluator's score, and a 0.6 judge score passed a 0.8 gate.

- Add an optional `score` field to CheckResult (last, with a None
  default, so existing constructor calls are unaffected).
- Set it in _coerce_result() for numeric returns and dicts with "score".
- Use it in LocalEvaluator.evaluate() when present; boolean checks keep
  mapping to 1.0 / 0.0.
- Treat NaN scores as below the threshold in assert_score_at_least().

Fixes microsoft#10073

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new positional default field breaks existing dataclass subclasses with required fields at import time; making it keyword-only would preserve their constructors.

1 open issue
What changed in this PR

This PR addresses #10073 by preserving numeric evaluator scores in Python’s local evaluation pipeline and preventing NaN scores from silently passing CI quality gates.

Changes:

  • Add an optional CheckResult.score and preserve numeric scores while retaining boolean fallback behavior.
  • Make assert_score_at_least() reject NaN scores.
  • Add regression tests for score preservation and gate failures.
File Description
python/​packages/​core/​tests/​core/​test_local_eval.py Tests numeric score preservation, below-threshold failures, and NaN rejection.
python/​packages/​core/​agent_framework/​_evaluation.py Carries numeric scores through local evaluation and strengthens score assertions.

Review effort: Balanced

passed: bool
reason: str
check_name: str
score: float | None = None

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. It is now field(default=None, kw_only=True) (44e3d3c → 40acc80), and test_check_result_subclass_can_add_required_fields defines a subclass with a required field and passes score by keyword. test_local_eval.py passes.

A positional field with a default breaks dataclass subclasses that add
required fields. Add a test for such a subclass.

This branch was successfully deployed

1 active deployment
github-app-auth — 40acc804 Deployed Nov 16, 2026 by 14H034160212 via add_label #27902
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

.NET: Python: [Bug]: LocalEvaluator drops numeric evaluator scores, so assert_score_at_least() passes below-threshold (and NaN) scores

2 participants