Fix DROP exact match: the length check compares gold to gold

[Ag3497120]

The defect

src/lighteval/metrics/harness_compatibility/drop.py:92 — the second conjunct of the exact-match condition compares gold to gold:

if set(pred_normalized_spans) == set(gold_normalized_spans) and len(gold_normalized_spans) == len(
    gold_normalized_spans
):
    exact_match = 1.0

len(x) == len(x) holds for every input, so the condition is set equality. The same condition in lm_eval/tasks/drop/utils.py:87, which this module exists to be compatible with, has the prediction on the left, so it is multiset equality:

if set(predicted_bags[0]) == set(gold_bags[0]) and len(predicted_bags[0]) == len(
    gold_bags[0]
):

A prediction that repeats a correct span scores as an exact match. Two lines, no fixtures:

from lighteval.metrics.harness_compatibility.drop import DropMetrics
DropMetrics()._get_metrics(["4", "4"], "4")
# main:    (1.0, 0.5)
# patched: (0.0, 0.5)

drop.py entered the repository already in this form, in 9009723 (#1065); there is no earlier revision of the file here.

Measured against lm-evaluation-harness

lighteval @ 932e1f2 against EleutherAI/lm-evaluation-harness @ 64f3d092 (its lm_eval/tasks/drop/utils.py unmodified), calling get_metrics and DropMetrics._get_metrics on the same inputs:

prediction       gold           harness em/f1  main em/f1  patched em/f1
---------------  -------------  -------------  ----------  -------------
['4']            4              1.0 / 1.00     1.0 / 1.00  1.0 / 1.00     exact single span
['5']            4              0.0 / 0.00     0.0 / 0.00  0.0 / 0.00     wrong answer
['4', 'four']    4              0.0 / 0.50     0.0 / 0.50  0.0 / 0.50     prediction adds an unrelated span
['4', 'four']    ['4', 'four']  1.0 / 1.00     1.0 / 1.00  1.0 / 1.00     two spans matching two golds
['4', '4']       4              0.0 / 0.50     1.0 / 0.50  0.0 / 0.50     prediction repeats the gold span   <-- main differs
['4', '4', '4']  4              0.0 / 0.33     1.0 / 0.33  0.0 / 0.33     prediction repeats it three times   <-- main differs
['4']            ['4', '4']     0.0 / 0.50     1.0 / 0.50  0.0 / 0.50     the gold is the side with the repeat   <-- main differs

The last row is the same bug seen from the other side: because the surviving comparison is len(gold) == len(gold), a gold that repeats a span also matches a shorter prediction.

Sweeping it: every prediction of 1–3 spans and every gold of 1–2 spans drawn from {"4", "five", "6"}, both passed as lists — 39 predictions × 12 golds = 468 pairs.

em disagrees with harness  -- main: 48   patched: 0
main em ABOVE patched: 48    main em BELOW patched: 0
f1 identical on every pair: yes (asserted, not sampled)

Over that space the patched metric agrees with the harness everywhere, main disagrees on 48 pairs, and every one of those 48 is main scoring higher. f1 never moves. So on those inputs reported DROP EM is inflated, not deflated.

The change

-        if set(pred_normalized_spans) == set(gold_normalized_spans) and len(gold_normalized_spans) == len(
+        if set(pred_normalized_spans) == set(gold_normalized_spans) and len(pred_normalized_spans) == len(
             gold_normalized_spans
         ):

The test does not render in the diff

Line 2 of .gitattributes is tests/unit/metrics/test_cases/*.json -filter -diff -merge text, so GitHub shows the fixture change as Bin 1714 -> 2332 bytes. It is quoted in full below. It is appended after DROP - Partial Match, and the three existing cases are byte-identical.

    {
      "name": "DROP - Duplicate Predicted Span",
      "metric_class": "drop",
      "metric_params": {},
      "doc": {
        "query": "What is 2 + 2?",
        "specific": {
          "golds_no_preprocessing": ["4"]
        },
        "choices": ["4"],
        "gold_index": 0,
        "task_name": "math"
      },
      "model_response": {
        "text": ["4", "4"]
      },
      "expected_output": {
        "em": 0.0,
        "f1": 0.5
      },
      "tolerance": 0.01,
      "description": "Two predicted spans against one gold span is not an exact match; lm-evaluation-harness scores this 0.0"
    }

git diff --text tests/unit/metrics/test_cases/drop.json renders it: 23 lines inserted, 0 deleted.

Red, then green

Base commit 932e1f2f4c5af3e926534f12b2a84a3ae18d6d3f (huggingface/lighteval main, 2026-08-11), working tree clean at that commit, pip install -e ".[dev]". Python 3.11.15, torch 2.13.0, transformers 5.15.1, datasets 5.0.1, numpy 2.4.6, pytest 9.1.1, ruff 0.16.4, macOS 26.5 arm64.

$ pytest tests/unit/metrics/test_automated_metrics_pytest.py -k drop -q

Unmodified at 932e1f2:

1 passed, 46 deselected in 13.61s

Fixture applied, drop.py untouched:

E   Failed: Test suite 'Drop Test Suite' failed with 1 failed tests:
E
E     - DROP - Duplicate Predicted Span: Expected {'em': 0.0, 'f1': 0.5}, got {'em': 1.0, 'f1': np.float64(0.5)}
1 failed, 46 deselected in 32.60s

One failure, so the three existing cases still pass without the fix. Fixture plus the one-line change:

1 passed, 46 deselected in 50.28s

Suite

$ pytest tests/unit/metrics --disable-pytest-warnings -q
932e1f2 : 4 failed, 283 passed
branch  : 4 failed, 283 passed

$ pytest tests/unit --disable-pytest-warnings -q
932e1f2 : 20 failed, 400 passed, 2 skipped, 191 subtests passed
branch  : 20 failed, 400 passed, 2 skipped, 191 subtests passed

The failing names are the same list before and after, and none of them are DROP:

• test_cases/extractiveness.json — ImportError on DataStatsMetric, wants lighteval[multilingual]
• three test_metric_requests.py::test_pmi_request* — TypeError: Strings must be encoded before hashing
• test_vllm_model.py (2), test_reasoning_tags.py (6), test_caching.py (2 tests + 6 subtests) — vllm and torchvision are not installed here

The passed count in tests/unit/metrics does not move, because each test-case JSON file is a single parametrised pytest item regardless of how many cases it holds.

ruff format --check and ruff check pass on drop.py; make quality is what the Quality workflow runs.

The diff is two files: one line in drop.py, one appended object in drop.json. Nothing reformatted, no other hunks, no new files.