Skip to content

Fix DetailsLogger compatibility with xxhash 4.0 - #1332

Open
Honorllq wants to merge 1 commit into
huggingface:mainfrom
Honorllq:agent/fix-xxhash-compat
Open

Fix DetailsLogger compatibility with xxhash 4.0#1332
Honorllq wants to merge 1 commit into
huggingface:mainfrom
Honorllq:agent/fix-xxhash-compat

Conversation

@Honorllq

Copy link
Copy Markdown

Summary

  • encode logging values as UTF-8 bytes before passing them to xxhash.xxh64
  • use the same byte-safe helper for per-sample, per-task, and cross-task hashes
  • declare xxhash as a direct dependency
  • add a regression test that simulates bytes-only xxhash behavior with non-ASCII input

Why

xxhash 4.0.0 no longer accepts Python strings directly. DetailsLogger.log() and DetailsLogger.aggregate() passed strings to xxh64, which raised TypeError during evaluation logging.

Closes #1330.

Validation

  • targeted compatibility check with xxhash==4.0.0
  • python -m py_compile src/lighteval/logging/info_loggers.py tests/unit/logging/test_evaluation_tracker.py
  • ruff check src/lighteval/logging/info_loggers.py tests/unit/logging/test_evaluation_tracker.py
  • ruff format --check src/lighteval/logging/info_loggers.py tests/unit/logging/test_evaluation_tracker.py

@adhabnr-ux

Copy link
Copy Markdown

Independent confirmation this is worth merging: hit the exact same crash (identical traceback line, DetailsLogger.log()xxhash.xxh64(doc.query)) from a completely different angle — building an EvalPort adapter that reads real per-document Pipeline.get_details() output from lighteval 0.13.0. Reproduced from scratch on a totally fresh pip install lighteval (today's PyPI resolves xxhash to 4.0.1, same as the original report), confirmed the standalone repro (xxhash.xxh64("hello") alone raises the same TypeError, no lighteval involved), and confirmed pinning xxhash<4.0 unblocks a real Pipeline.evaluate() run end-to-end.

Worth noting for triage: it's not just log() — every one of the four xxhash.xxh64(...) call sites in info_loggers.py (hash.example, hash.input_tokens, hash.cont_tokens, and the three compiled_hash.* aggregation lines) has the same str-not-bytes issue, so a fix needs to cover all four, not just the one in the traceback — sounds like that's already what this PR does per the summary ("per-sample, per-task, and cross-task hashes"), just flagging in case it helps review. Since I had to work around this myself to get real test data (pinning xxhash<4.0 in my own project's test deps as a stopgap), happy to see this land — it currently breaks Pipeline.evaluate(), which is the method every CLI backend (accelerate/vllm/endpoint/etc.) calls under the hood, for anyone installing lighteval fresh today.

adhabnr-ux added a commit to adhabnr-ux/evalport that referenced this pull request Aug 20, 2026
… LLM Leaderboard eval library

Converts real per-document lighteval Pipeline.get_details() output
(Detail(doc, model_response, metric)) to/from EvalPort Suites and
ResultSets. lighteval is the evaluation library behind the Hugging Face
Open LLM Leaderboard, and is itself built directly on inspect_ai (a
declared hard dependency; several of its own task definitions, e.g.
gsm8k, are written using inspect_ai's Sample/solver/scorer primitives
directly) -- this closes the loop on the other side of EvalPort's already
merged inspect_ai integration (UKGovernmentBEIS/inspect_ai#4797).

Two things verified against the actually-installed lighteval 0.13.0
package, not assumed from docs:
- pipeline.get_results() does not return per-sample data (it's the
  aggregate summary dict) -- the real per-document Detail objects come
  from the separate pipeline.get_details() method. This adapter reads
  get_details().
- Even a classic multiple-choice task (hellaswag) is scored generatively
  in this version: Metrics.exact_match's real metric_name is "em", backed
  by ExactMatches(strip_strings=True) (confirmed by reading
  lighteval/metrics/metrics.py), scoring model_response.text against the
  gold choice, not model_response.logprobs. Mapped to EvalPort's native
  exact_match grader because it genuinely is exact-match semantics, not
  just named similarly.

Also found and reported a real bug in lighteval itself while building
this: a fresh `pip install lighteval` today resolves the unpinned
transitive xxhash dependency to 4.0.1, and lighteval's own
DetailsLogger.log() calls xxhash.xxh64(doc.query) with a raw str, which
xxhash 4.0 no longer accepts -- crashing every real Pipeline.evaluate()
call. Already tracked as huggingface/lighteval#1330 with a fix up as
huggingface/lighteval#1332; left an independent reproduction comment
there confirming all four affected call sites. Pinned xxhash<4.0 in this
adapter's own test extra as a stopgap so its tests can run today.

Verified locally before pushing, in two independent clean venvs:
- pytest tests/ -v -> 19 passed (both times), real Pipeline.evaluate()
  runs against hellaswag (multiple-choice) and gsm8k (generation,
  inspect_ai-solver-backed), both validated against the real
  openeval.validate.validate_suite()/validate_result_set()
- python -m build --wheel succeeds
@Honorllq
Honorllq marked this pull request as ready for review August 20, 2026 08:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Incompatibility with latest version of xxhash

2 participants