Skip to content

Python: Fail assert_dimension_score_at_least when a filter matches nothing - #10199

Closed
Wilson_Jiang (Wilson0511) wants to merge 2 commits into
microsoft:mainfrom
Wilson0511:fix/assert-dimension-vacuous-pass
Closed

Wilson_Jiang (Wilson0511) wants to merge 2 commits into
microsoft:mainfrom
Wilson0511:fix/assert-dimension-vacuous-pass

Conversation

@Wilson0511

Copy link
Copy Markdown
Contributor

Python: Fail assert_dimension_score_at_least when a filter matches nothing

Description

EvalResults.assert_dimension_score_at_least has the same vacuous-pass defect that #10180 just fixed for assert_score_at_least: with the default require_applicable=False, an evaluator or dimension_id that matches nothing (e.g. a typo — or an empty result set) skips every score and the assertion passes silently, so a CI gate that was meant to run never checked anything.

  • When the evaluator filter matches no scores, the call now raises EvalNotPassedError listing the available evaluators (mirroring the Python: fail assert_score_at_least when evaluator filter matches no scores #10180 wording).
  • When the dimension is never seen among the matching scores, it raises listing the available dimensions.
  • require_applicable=True keeps its existing not-applicable report and pinned wording — the new failures apply only to the default flags, where the call previously passed silently. The documented default skip for a dimension that is present but not applicable is unchanged.

Tests

Three regressions in TestRubricAssertions (red on main, green here): missing evaluator raises with available evaluators; missing dimension raises with available dimensions; empty results raise with "Available dimension(s): none". Full test_local_eval.py: 100 passed, zero regressions — all existing pinned behaviors (non_applicable_skipped_by_default, require_applicable_raises_when_*, evaluator_filter_isolates_offenders) stay green.

Attribution

Developed with AI assistance (model: GLM-5.3 via ZCode; Co-Authored-By trailer on the commit). Found by sweeping the sibling assertion helpers after #10180 merged; diff reviewed in the authoring context, no separate fresh-context pass. Slow/integration tests: not run locally (no live evaluator backends needed — all tests are offline unit level).

With the default require_applicable=False, assert_dimension_score_at_least
vacuously passed when the evaluator filter or the dimension_id matched
nothing — a typo in either silently skipped the CI gate. Mirror the
assert_score_at_least fix: when no score matches the evaluator (or the
dimension is never seen among matching scores), raise EvalNotPassedError
listing the available evaluators/dimensions.

require_applicable=True keeps its existing not-applicable report (and its
pinned wording); the new failure only applies to the default flags where
the call previously passed silently.

Co-Authored-By: ZCode <zcode@bigmodel.ai>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Enabling require_applicable=True silently passes evaluations with no items, allowing an empty CI gate to succeed despite the stricter option.

1 open issue
What changed in this PR

Extends the no-match protection from #10180 to Python’s dimension-score assertions, preventing silent success when default filters find nothing.

Changes:

  • Tracks matching evaluators and dimensions, raising errors that list available values.
  • Preserves default skipping for present but non-applicable dimensions.
  • Adds three regression tests and documents the new failure conditions.
File Description
python/​packages/​core/​agent_framework/​_evaluation.py Adds no-match checks and updates assertion documentation.
python/​packages/​core/​tests/​core/​test_local_eval.py Tests missing evaluators, missing dimensions, and empty score lists.

Review effort: Balanced

# assert_score_at_least for evaluator filters). With
# require_applicable=True the not-applicable report below already
# fails the call, and its wording is pinned by existing tests.
if not require_applicable:
Copilot review: a bare EvalResults (no items) still passed silently with
require_applicable=True — missing_items stays empty when there are no
items to report, so the stricter option let an empty CI gate succeed.
Run the no-match checks whenever missing_items is empty (all-items-
applicable runs pass them harmlessly); the not-applicable report keeps
carrying non-empty missing sets with its pinned wording.

Co-Authored-By: ZCode <zcode@bigmodel.ai>
@Wilson0511

Copy link
Copy Markdown
Contributor Author

Good catch — fixed in 33c5c0b. A bare EvalResults (zero items) left missing_items empty under require_applicable=True, so the not-applicable report never fired and the empty gate passed. As suggested, the no-match checks now also run when missing_items is empty (if not require_applicable or not missing_items:); runs where every item is applicable pass those checks harmlessly, and non-empty missing sets still get the existing not-applicable report with its pinned wording. Added test_empty_results_raise_even_with_require_applicable (bare EvalResults, both with and without an evaluator filter) — red on the previous head, green now; full file 101 passed.

(Small note on the repro detail: _rubric_results([]) builds one item with no scores, which under require_applicable=True already raised not-applicable — the silent pass needed the zero-item constructor form, which the new test uses.)

@eavanvalkenburg

Copy link
Copy Markdown
Member

Wilson_Jiang (@Wilson0511), thank you for the contribution. The PR body does not link an agreed repository issue or document a valid trivial-change or repository-automation exception. Please settle the scope in the issue and wait for explicit maintainer agreement before opening a fresh implementation PR. I am closing this for now.

This branch was previously deployed

1 inactive deployment
github-app-auth — 33c5c0b0 Deployed Nov 9, 2026 by Wilson0511 via add_label #27402
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants