Fix DetailsLogger compatibility with xxhash 4.0 - #1332
Conversation
|
Independent confirmation this is worth merging: hit the exact same crash (identical traceback line, Worth noting for triage: it's not just |
… LLM Leaderboard eval library Converts real per-document lighteval Pipeline.get_details() output (Detail(doc, model_response, metric)) to/from EvalPort Suites and ResultSets. lighteval is the evaluation library behind the Hugging Face Open LLM Leaderboard, and is itself built directly on inspect_ai (a declared hard dependency; several of its own task definitions, e.g. gsm8k, are written using inspect_ai's Sample/solver/scorer primitives directly) -- this closes the loop on the other side of EvalPort's already merged inspect_ai integration (UKGovernmentBEIS/inspect_ai#4797). Two things verified against the actually-installed lighteval 0.13.0 package, not assumed from docs: - pipeline.get_results() does not return per-sample data (it's the aggregate summary dict) -- the real per-document Detail objects come from the separate pipeline.get_details() method. This adapter reads get_details(). - Even a classic multiple-choice task (hellaswag) is scored generatively in this version: Metrics.exact_match's real metric_name is "em", backed by ExactMatches(strip_strings=True) (confirmed by reading lighteval/metrics/metrics.py), scoring model_response.text against the gold choice, not model_response.logprobs. Mapped to EvalPort's native exact_match grader because it genuinely is exact-match semantics, not just named similarly. Also found and reported a real bug in lighteval itself while building this: a fresh `pip install lighteval` today resolves the unpinned transitive xxhash dependency to 4.0.1, and lighteval's own DetailsLogger.log() calls xxhash.xxh64(doc.query) with a raw str, which xxhash 4.0 no longer accepts -- crashing every real Pipeline.evaluate() call. Already tracked as huggingface/lighteval#1330 with a fix up as huggingface/lighteval#1332; left an independent reproduction comment there confirming all four affected call sites. Pinned xxhash<4.0 in this adapter's own test extra as a stopgap so its tests can run today. Verified locally before pushing, in two independent clean venvs: - pytest tests/ -v -> 19 passed (both times), real Pipeline.evaluate() runs against hellaswag (multiple-choice) and gsm8k (generation, inspect_ai-solver-backed), both validated against the real openeval.validate.validate_suite()/validate_result_set() - python -m build --wheel succeeds
Summary
xxhash.xxh64xxhashas a direct dependencyWhy
xxhash4.0.0 no longer accepts Python strings directly.DetailsLogger.log()andDetailsLogger.aggregate()passed strings toxxh64, which raisedTypeErrorduring evaluation logging.Closes #1330.
Validation
xxhash==4.0.0python -m py_compile src/lighteval/logging/info_loggers.py tests/unit/logging/test_evaluation_tracker.pyruff check src/lighteval/logging/info_loggers.py tests/unit/logging/test_evaluation_tracker.pyruff format --check src/lighteval/logging/info_loggers.py tests/unit/logging/test_evaluation_tracker.py