Repository navigation
Python: Preserve numeric evaluator scores in LocalEvaluator - #10383
Open
Qiming Bao (14H034160212) wants to merge 2 commits into
Open
Qiming Bao (14H034160212) wants to merge 2 commits into
Qiming Bao (14H034160212) wants to merge 2 commits into
Conversation
LocalEvaluator recorded every check score as 1.0 or 0.0 based only on the pass/fail flag, so a function evaluator returning a numeric score (a float, or a dict with "score") lost that value. As a result, EvalResults.assert_score_at_least() gated on the pass flag rather than on the evaluator's score, and a 0.6 judge score passed a 0.8 gate. - Add an optional `score` field to CheckResult (last, with a None default, so existing constructor calls are unaffected). - Set it in _coerce_result() for numeric returns and dicts with "score". - Use it in LocalEvaluator.evaluate() when present; boolean checks keep mapping to 1.0 / 0.0. - Treat NaN scores as below the threshold in assert_score_at_least(). Fixes microsoft#10073
Qiming Bao (14H034160212)
deployed
to
github-app-auth
November 16, 2026 04:25 — with
GitHub Actions
Active
Qiming Bao (14H034160212)
deployed
to
github-app-auth
November 16, 2026 04:25 — with
GitHub Actions
Active
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The new positional default field breaks existing dataclass subclasses with required fields at import time; making it keyword-only would preserve their constructors.
1 open issue
What changed in this PR
This PR addresses #10073 by preserving numeric evaluator scores in Python’s local evaluation pipeline and preventing NaN scores from silently passing CI quality gates.
Changes:
- Add an optional
CheckResult.scoreand preserve numeric scores while retaining boolean fallback behavior. - Make
assert_score_at_least()rejectNaNscores. - Add regression tests for score preservation and gate failures.
| File | Description |
|---|---|
python/packages/core/tests/core/test_local_eval.py |
Tests numeric score preservation, below-threshold failures, and NaN rejection. |
python/packages/core/agent_framework/_evaluation.py |
Carries numeric scores through local evaluation and strengthens score assertions. |
Review effort: Balanced
| passed: bool | ||
| reason: str | ||
| check_name: str | ||
| score: float | None = None |
A positional field with a default breaks dataclass subclasses that add required fields. Add a test for such a subclass.
Qiming Bao (14H034160212)
deployed
to
github-app-auth
November 16, 2026 04:39 — with
GitHub Actions
Active
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Motivation & Context
LocalEvaluator.evaluate()recorded every check score as1.0or0.0, based only onpassed. A function evaluator that returns a numeric score (afloat, or a dict withscore) lost that number, which_coerce_result()kept only inside thereasonstring.As a result,
EvalResults.assert_score_at_least()gated on the pass flag rather than on the evaluator's score. An LLM-as-judge returning0.6passes the default0.5threshold, is stored as1.0, andassert_score_at_least(0.8)returns normally. In addition,assert_score_at_least()testedscore < min_score, which isFalseforNaN, so a provider that reportsNaN(for example a judge whose output failed to parse) also passed the gate. Both are silent passes in a check meant for CI quality gates.Description & Review Guide
CheckResultgets an optionalscore: float | None = None. It is the last field and has a default, so existingCheckResult(passed=..., reason=..., check_name=...)calls are unaffected._coerce_result()sets it for numeric returns and for dicts withscore.LocalEvaluator.evaluate()recordsresult.scorewhen present. Boolean checks (keyword_check,tool_called_check, ...) keep mapping to1.0/0.0.assert_score_at_least()usesnot score.score >= min_score, soNaNcounts as an offender. The recent check for anevaluatorfilter that matches no scores is kept.EvalScoreResult.scorenow holds that number, as its docstring ("Numeric score from the evaluator") says, soassert_score_at_least()can fail where it silently passed before.CheckResult. The alternative raised in .NET: Python: [Bug]: LocalEvaluator drops numeric evaluator scores, so assert_score_at_least() passes below-threshold (and NaN) scores #10073 is to keep local scores 0/1 and document it; I'm happy to switch if you prefer that.Tests (in
python/packages/core/tests/core/test_local_eval.py):test_numeric_evaluator_scores_are_preserved: float and{"score": ...}returns are stored as-is.test_numeric_evaluator_score_below_gate_raises: a0.6score failsassert_score_at_least(0.8).test_nan_score_raises: aNaNscore fails the gate.All three fail on
mainand pass with this change. After rebasing on the latestmain,test_local_eval.pypasses (all tests),ruff checkandruff format --checkpass on the changed files, and the core suite gives 7722 passed, 25 skipped, 1 failed. The one failure,test_feature_stage.py::test_feature_id_allows_lowercase_values, also fails onmainin my environment and is unrelated.Related Issue
Fixes #10073
The issue is labelled
reproduced: the triage bot reproduced both problems with the same three tests. I proposed this direction on the issue on 2026-11-05 and haven't had a reply yet, so I'm opening the PR to make review easier. No other PR is open for this issue.AI Assistance
AI assistance details: Claude (Anthropic) helped with the implementation, tests and this PR description.
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.