Skip to content

Fix Dyck scoring to preserve bracket order - #1358

Open
Excelius-Wang wants to merge 1 commit into
huggingface:mainfrom
Excelius-Wang:fix/dyck-strict-scoring
Open

Fix Dyck scoring to preserve bracket order#1358
Excelius-Wang wants to merge 1 commit into
huggingface:mainfrom
Excelius-Wang:fix/dyck-strict-scoring

Conversation

@Excelius-Wang

Copy link
Copy Markdown

Summary

  • replace inspect-ai's punctuation-normalizing exact() scorer for Dyck tasks with a task-local
    whitespace-trimmed comparison
  • preserve bracket order and multiplicity for the two-, three-, and four-bracket variants
  • add regression coverage for exact output, surrounding whitespace, reordered brackets, and
    duplicated brackets

Fixes #1329.

Why

Raising the inspect-ai dependency floor does not fully fix bracket-only answers: inspect-ai 0.3.260
still normalizes reordered bracket sequences to the same value. The task-local scorer mirrors
lighteval's existing exact-match behavior by trimming surrounding whitespace only, while keeping the
existing mean and stderr metrics.

Testing

  • pytest -q tests/unit/tasks/test_dyck_language.py
  • pytest -q tests/unit/tasks/test_lighteval_task.py::test_hf_data_files tests/unit/tasks/templates tests/unit/tasks/test_dyck_language.py
  • ruff check src/lighteval/tasks/tasks/dyck_language.py tests/unit/tasks/test_dyck_language.py
  • ruff format --check src/lighteval/tasks/tasks/dyck_language.py tests/unit/tasks/test_dyck_language.py
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] dyck_language scores with inspect_ai's exact(), which ignored word order in every release up to 0.3.252

1 participant