Skip to content

Fix continuation tokens after distributed gathering - #1357

Open
Excelius-Wang wants to merge 1 commit into
huggingface:mainfrom
Excelius-Wang:fix/logprob-token-normalization
Open

Fix continuation tokens after distributed gathering#1357
Excelius-Wang wants to merge 1 commit into
huggingface:mainfrom
Excelius-Wang:fix/logprob-token-normalization

Conversation

@Excelius-Wang

Copy link
Copy Markdown

Summary

  • slice gathered continuation tensors across both the choice and token dimensions
  • strip the -1 padding used only to make continuations gatherable
  • extend the existing padding test to verify every choice is preserved and token counts exclude padding

Fixes #1170.

Testing

  • pytest tests/unit/models/test_transformers_model.py::TestTransformersModelProcessing -q
  • pytest tests/unit/metrics/test_normalizations.py -q
  • ruff check src/lighteval/models/transformers/transformers_model.py tests/unit/models/test_transformers_model.py
  • ruff format --check src/lighteval/models/transformers/transformers_model.py tests/unit/models/test_transformers_model.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] LogProbTokenNorm crashes with IndexError in CFFormulation for belebele

1 participant