Cap the METEOR fragmentation penalty so scores stay in [0, 1] (#210) - #216
Merged
Merged
Conversation
_calculate_fragmentation_penalty divides chunk counts taken over the full token sequence (repeats included) by the number of distinct matched words, so the ratio can exceed 1 and the METEOR score can go negative. Cap the penalty at 1. Only scores that were negative change (they become 0). Also re-scores the one negative cached value (claude-3-haiku, NarrativeQA_4102). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the negative METEOR score reported in #210.
Cause.
_calculate_fragmentation_penaltycomputes0.5 * (min_chunks / total_matches) ** 2, butmin_chunkscounts runs over the full token sequence (repeated words included) whiletotal_matchescounts distinct matched words. The ratio can exceed 1, the penalty can exceed 1, andf_score * (1 - penalty)goes negative.Fix. Cap the penalty at 1. This is deliberately minimal: only scores that were negative change, and they become 0. Every other score is unchanged.
Verified by re-grading every non-LiveCodeBench row of the 15 leaderboard routers that store per-row scores (150,576 rows, including candidate rows):
Also re-scores the one negative cached value (
cached_results/claude-3-haiku-20240307.jsonl,NarrativeQA_4102: −0.033 → 0.0) throughmeteor_score; no negative scores remain in the caches.This PR does not change any published leaderboard numbers. Those will update in the planned full re-grade of all routers.
🤖 Generated with Claude Code