fix(metrics): avg@n scores raw text instead of post-processed text, and skips preprocessing

[arthi-arumugam-git]

The wrong number

On a reasoning model with the pipeline default remove_reasoning_tags=True, avg@n reports 0.0 where the correct value is 1.0, while maj@n and pass@1 on the identical generations both report 1. Nothing raises.

raw text[0]         : '<think>Could be Lyon. No, it is Paris.</think>Paris'
final_text[0]       : 'Paris'
sliced final_text[0]: '<think>Could be Lyon. No, it is Paris.</think>Paris'   <- the bug

gold: Paris, and all 4 generations answer Paris once reasoning is stripped

avg@n  (AvgAtN)  = 0.0     <- reported
maj@n  (MajAtN)  = 1       <- same data, sibling metric
pass@1 (PassAtK) = 1.0     <- same data, sibling metric
correct avg@n    = 1.0

A second, independent defect sits in the same three lines. Metrics.avg_at_n is registered as AvgAtN(strip_strings=True), but compute never calls self.preprocess, so strip_strings and normalize are silently dropped. With ordinary leading and trailing whitespace and no reasoning tags at all:

avg@n = 0.25 where the correct value is 1.0
avg@n = 0.25 where the correct value is 0.75

Both errors push the score down, and avg@n is bounded [0, 1], so the first case is the maximum possible error on the scale.

Cause

Both are the same shape: a contract that landed on every metric except one.

1. ModelResponse.__getitem__ drops text_post_processed (src/lighteval/models/model_output.py:147). It copies nine fields and omits the one that final_text depends on, so final_text on a slice falls back to self.text. AvgAtN.compute is the only metric that scores through model_response[i], which is why it is the only one affected.

git log -L shows __getitem__ and text_post_processed were introduced in the same commit, d7beacb, "Added post processing (for reasoning tokens) to pipeline (#882)". The new field was simply missed in the new method.

2. AvgAtN.compute never preprocesses (src/lighteval/metrics/metrics_sample.py:1200). MajAtN, PassAtK and GPassAtK all build a preprocessed Doc and read model_response.final_text. AvgAtN did neither, while its docstring has claimed "It applies normalisation (if needed) to model prediction and gold" the whole time.

This survived because tests/unit/metrics/test_cases/avg_at_k.json never executed: its metric_class does not match the registered enum member, so it falls into the silent-skip branch. That is already reported as #1304 / #1305, which is not mine.

The change

AvgAtN.compute now reads exactly like PassAtK.compute: preprocess the choices, read model_response.final_text, preprocess each prediction.

The explicit n is None guard is deliberate. Moving from range(self.n) to final_text[: self.n] would have converted today's loud TypeError into a silent "average whatever you were given", which is the same class of bug being fixed here. The wording matches what MajAtN already raises.

Tests

tests/unit/metrics/test_avg_at_n_final_text.py, using only symbols that exist on main.

On pristine main: 5 failed, 1 passed.
With this change: 6 passed.

The one that passes on main is the all-wrong case, which must keep returning 0.0. That is the legitimate zero, and the point of including it is to show this change does not turn every zero into a non-zero.

Wider metric suite (tests/unit/metrics/test_automated_metrics_pytest.py and tests/test_unit_base_metrics.py), run before and after on the same machine:

pristine main : 12 failed, 44 passed, 3 skipped
with this PR  : 12 failed, 44 passed, 3 skipped

The failing set is byte-identical between the two runs, and none of the 12 touch AvgAtN or ModelResponse. They are environment-only on my Windows box and reproduce on unmodified main. There are no errored tests hiding behind the passed count.

ruff check and ruff format --check both pass on all three files.

Scope, and what I deliberately did not touch

• Not MajAtN, PassAtK or GPassAtK. All three already read final_text and already preprocess. Their outputs are unchanged.
• Not the MajAtN doc.get_golds() behaviour at metrics_sample.py:1244. I ran into it, but it is already open as Fix MajAtN IndexError when the gold is not the first choice #1274 and belongs to that author.
• Not SamplingMetric.__init__ at 1122, already open as Fix string-name normalization in SamplingMetric (inspect.getmembers returns a list, not a dict) #1292.
• Not the avg_at_k.json / maj_at_k.json fixture naming, already open as avg_at_k / avg_at_k_math / maj_at_k metric regression fixtures silently no-op (metric_class name doesn't match registered Metrics enum members) #1304 / fix metric_class names in avg_at_n / avg_at_n_math / maj_at_n test cases #1305. I checked this change against those fixtures with fix metric_class names in avg_at_n / avg_at_n_math / maj_at_n test cases #1305's corrections applied and the expected values 0.5, 0.0 and 0.33 all reproduce, so the two do not collide.
• Not reasonings, which __getitem__ also drops. No metric reads it today, so it is a latent schema inconsistency rather than a wrong number. Worth knowing about, but I did not want to widen this diff.
• Not the pipeline post-processing, remove_reasoning_tags, the final_text property, any task definition, or any corpus-level aggregation.

One claim I am not making

Metrics.avg_at_n_math is used by aime24, aime24_avg, aime25 and aime25_avg, and it goes through MultilingualExtractiveMatchMetric, which is far more robust to leading reasoning text. I built two realistic reasoning traces, one with an abandoned \boxed{} candidate and one with a self-correction, and could not make the extractor diverge; both returned 1.0 with and without the reasoning stripped.

So I am claiming the confirmed wrong number on Metrics.avg_at_n and on any custom AvgAtN with a string-comparison scoring function. On the AIME tasks I have confirmed only that avg@n is fed different text from every other metric on the same run, not that the published AIME number moves. I would rather say that plainly than overstate it.