Repository navigation
Python: Fail assert_dimension_score_at_least when a filter matches nothing - #10199
Wilson_Jiang (Wilson0511) wants to merge 2 commits into
Conversation
With the default require_applicable=False, assert_dimension_score_at_least vacuously passed when the evaluator filter or the dimension_id matched nothing — a typo in either silently skipped the CI gate. Mirror the assert_score_at_least fix: when no score matches the evaluator (or the dimension is never seen among matching scores), raise EvalNotPassedError listing the available evaluators/dimensions. require_applicable=True keeps its existing not-applicable report (and its pinned wording); the new failure only applies to the default flags where the call previously passed silently. Co-Authored-By: ZCode <zcode@bigmodel.ai>
There was a problem hiding this comment.
🟡 Changes recommended
Enabling require_applicable=True silently passes evaluations with no items, allowing an empty CI gate to succeed despite the stricter option.
1 open issue
What changed in this PR
Extends the no-match protection from #10180 to Python’s dimension-score assertions, preventing silent success when default filters find nothing.
Changes:
- Tracks matching evaluators and dimensions, raising errors that list available values.
- Preserves default skipping for present but non-applicable dimensions.
- Adds three regression tests and documents the new failure conditions.
| File | Description |
|---|---|
python/packages/core/agent_framework/_evaluation.py |
Adds no-match checks and updates assertion documentation. |
python/packages/core/tests/core/test_local_eval.py |
Tests missing evaluators, missing dimensions, and empty score lists. |
Review effort: Balanced
| # assert_score_at_least for evaluator filters). With | ||
| # require_applicable=True the not-applicable report below already | ||
| # fails the call, and its wording is pinned by existing tests. | ||
| if not require_applicable: |
Copilot review: a bare EvalResults (no items) still passed silently with require_applicable=True — missing_items stays empty when there are no items to report, so the stricter option let an empty CI gate succeed. Run the no-match checks whenever missing_items is empty (all-items- applicable runs pass them harmlessly); the not-applicable report keeps carrying non-empty missing sets with its pinned wording. Co-Authored-By: ZCode <zcode@bigmodel.ai>
|
Good catch — fixed in 33c5c0b. A bare (Small note on the repro detail: |
|
Wilson_Jiang (@Wilson0511), thank you for the contribution. The PR body does not link an agreed repository issue or document a valid trivial-change or repository-automation exception. Please settle the scope in the issue and wait for explicit maintainer agreement before opening a fresh implementation PR. I am closing this for now. |

Python: Fail
assert_dimension_score_at_leastwhen a filter matches nothingDescription
EvalResults.assert_dimension_score_at_leasthas the same vacuous-pass defect that #10180 just fixed forassert_score_at_least: with the defaultrequire_applicable=False, anevaluatorordimension_idthat matches nothing (e.g. a typo — or an empty result set) skips every score and the assertion passes silently, so a CI gate that was meant to run never checked anything.evaluatorfilter matches no scores, the call now raisesEvalNotPassedErrorlisting the available evaluators (mirroring the Python: fail assert_score_at_least when evaluator filter matches no scores #10180 wording).require_applicable=Truekeeps its existing not-applicable report and pinned wording — the new failures apply only to the default flags, where the call previously passed silently. The documented default skip for a dimension that is present but not applicable is unchanged.Tests
Three regressions in
TestRubricAssertions(red on main, green here): missing evaluator raises with available evaluators; missing dimension raises with available dimensions; empty results raise with "Available dimension(s): none". Fulltest_local_eval.py: 100 passed, zero regressions — all existing pinned behaviors (non_applicable_skipped_by_default,require_applicable_raises_when_*,evaluator_filter_isolates_offenders) stay green.Attribution
Developed with AI assistance (model: GLM-5.3 via ZCode; Co-Authored-By trailer on the commit). Found by sweeping the sibling assertion helpers after #10180 merged; diff reviewed in the authoring context, no separate fresh-context pass. Slow/integration tests: not run locally (no live evaluator backends needed — all tests are offline unit level).