You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Python: assert_dimension_score_at_least passes vacuously when evaluator/dimension filter matches nothing or results are empty #10237
assert_dimension_score_at_least and assert_score_at_least in python/packages/core/agent_framework/_evaluation.py pass vacuously when the filter matches nothing: an unknown evaluator name or dimension_id, or an empty EvalResults (zero items), leaves the assertion a no-op instead of failing the run.
evaluator="typo" on an empty result set: the empty-missing_items path under require_applicable=True skips both the not-applicable report and the no-scores check.
evaluator filter that matches no scores on a non-empty result set: matching_scores_count == 0 only raises when evaluator is not None — but the dimension-only form (dimension_id that matches no scores) raises nothing.
Zero-item EvalResults with no evaluator: dimension_seen is never set, and the same empty-missing_items path bypasses the check.
Expected Behavior
A filter (evaluator or dimension) that matches no scores should fail the assertion — mirroring what #10180 established for assert_score_at_least's evaluator filter — and an empty result set should not silently satisfy assert_dimension_score_at_least under require_applicable=True.
Minimal Reproduction
fromagent_frameworkimportEvalResults# no exception on current main:assert_dimension_score_at_least(EvalResults(items=[]), dimension_id="accuracy", score=0.5)
Additional Context
An implementation with red-green tests was previously posted in #10199 (closed for arriving without an agreed issue — resubmitting a fresh PR once this one is agreed).
I understand that PR #10199 was closed because the implementation was submitted before the issue was agreed upon. I'd like to make sure I follow the project's contribution process.
Would it be okay for me to work on this issue and submit a fresh PR with regression tests covering:
Unknown evaluator names
Unmatched dimension IDs
Empty evaluation results
I'm comfortable working with Python and would be happy to take this up once the proposed approach is approved. Thank you!
Activity
rohithavadlamudi2005-commits commentedon Nov 13, 2026
Hi! I'd like to contribute to this issue.
I understand that PR #10199 was closed because the implementation was submitted before the issue was agreed upon. I'd like to make sure I follow the project's contribution process.
Would it be okay for me to work on this issue and submit a fresh PR with regression tests covering:
I'm comfortable working with Python and would be happy to take this up once the proposed approach is approved. Thank you!