Skip to content

Fix BBEH generative evaluation - #1333

Open
lakshayxi wants to merge 1 commit into
huggingface:mainfrom
lakshayxi:fix-bbeh-evaluation
Open

Fix BBEH generative evaluation#1333
lakshayxi wants to merge 1 commit into
huggingface:mainfrom
lakshayxi:fix-bbeh-evaluation

Conversation

@lakshayxi

Copy link
Copy Markdown

Summary

  • fix BBEH gold answers being treated as strings instead of choice lists
  • evaluate BBEH as a generative task instead of using log-likelihood accuracy
  • match the official BBEH answer normalization and equivalence rules in both LightEval and Inspect
  • add focused regression tests for gold preservation, scoring, task wiring, and Inspect scoring

The existing BBEH task was added in #1124. This is a correctness follow-up to that implementation.

Refs #600.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant