fix(evals): StepEfficiency negative score on redundant calls within a step - #62
Open
usegitdivyansh wants to merge 1 commit into
Open
Conversation
… step Redundancy was counted per tool call but normalized by step count, so a single step with multiple redundant calls could push redundancy_ratio below 0 and produce a negative score, out of the documented 0 to 1 range. Divide by total tool calls instead, matching the failure calculation in the same method. The per-call denominator keeps the ratio bounded by construction, so no clamp is needed. Adds a test covering multiple redundant calls within a single step. Fixes future-agi#60
Author
|
@abhiramArise this implements option 1 as discussed in #60 — redundancy now divides by total_calls, no clamp. Added a test for the multiple-calls-per-step case alongside the existing single-call test. Ready for review whenever you get a chance. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #60
What does this PR do?
Fixes StepEfficiency returning a negative score when a single trajectory step contains more than one redundant tool call.
Why?
Redundancy was counted per tool call (the inner loop runs over step.tool_calls) but the ratio divided that count by total_steps, the number of steps. When a step makes several redundant calls, redundant_count exceeds total_steps, so redundancy_ratio goes below 0 and the final score drops out of the documented 0 to 1 range. On the reproduction in #60 (one step, five identical calls) the metric returned -0.2.
What changed?
The redundancy ratio now divides by total_calls instead of total_steps, matching the failure calculation in the same method. total_calls is moved up so it is defined once before both uses. As discussed in #60, the per-call denominator keeps the ratio bounded by construction, so no clamp is added.
Tests
Adds test_redundant_calls_within_single_step, covering multiple redundant calls within a single step and asserting the score stays within 0 to 1. The existing single-call-per-step test is unchanged. Full agent suite passes locally (29 passed).