Skip to content

Python: assert_dimension_score_at_least passes vacuously when evaluator/dimension filter matches nothing or results are empty #10237

Open

Description

Observed Behavior

assert_dimension_score_at_least and assert_score_at_least in python/packages/core/agent_framework/_evaluation.py pass vacuously when the filter matches nothing: an unknown evaluator name or dimension_id, or an empty EvalResults (zero items), leaves the assertion a no-op instead of failing the run.

  • evaluator="typo" on an empty result set: the empty-missing_items path under require_applicable=True skips both the not-applicable report and the no-scores check.
  • evaluator filter that matches no scores on a non-empty result set: matching_scores_count == 0 only raises when evaluator is not None — but the dimension-only form (dimension_id that matches no scores) raises nothing.
  • Zero-item EvalResults with no evaluator: dimension_seen is never set, and the same empty-missing_items path bypasses the check.

Expected Behavior

A filter (evaluator or dimension) that matches no scores should fail the assertion — mirroring what #10180 established for assert_score_at_least's evaluator filter — and an empty result set should not silently satisfy assert_dimension_score_at_least under require_applicable=True.

Minimal Reproduction

from agent_framework import EvalResults
# no exception on current main:
assert_dimension_score_at_least(EvalResults(items=[]), dimension_id="accuracy", score=0.5)

Additional Context

An implementation with red-green tests was previously posted in #10199 (closed for arriving without an agreed issue — resubmitting a fresh PR once this one is agreed).

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 10, 2026
  2. rohithavadlamudi2005-commits commented on Nov 13, 2026

    @rohithavadlamudi2005-commits

    Hi! I'd like to contribute to this issue.

    I understand that PR #10199 was closed because the implementation was submitted before the issue was agreed upon. I'd like to make sure I follow the project's contribution process.

    Would it be okay for me to work on this issue and submit a fresh PR with regression tests covering:

    • Unknown evaluator names
    • Unmatched dimension IDs
    • Empty evaluation results

    I'm comfortable working with Python and would be happy to take this up once the proposed approach is approved. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    pythonUsage: [Issues, PRs], Target: PythontriageUsage: [Issues], Target: All issues that still need to be triaged

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions