Skip to content

Fix few-shot sampler isolation - #1334

Open
atirna wants to merge 1 commit into
huggingface:mainfrom
atirna:fix/fewshot-sampler-isolation
Open

Fix few-shot sampler isolation#1334
atirna wants to merge 1 commit into
huggingface:mainfrom
atirna:fix/fewshot-sampler-isolation

Conversation

@atirna

@atirna atirna commented Aug 16, 2026

Copy link
Copy Markdown

Description

I fixed the few-shot sampler so sequential variance seeds rotate independent copies of the task pool. I also fixed balanced sampling to retain falsy labels and use a local seeded RNG instead of mutating process-global random state.

The existing sequential assertion now compares the sampled offset with the unchanged pool. Previously, its expected value was read from the pool after sampling had mutated it.

Fixes #1307
Fixes #1309

Why

Sequential variance runs should choose offsets from the original few-shot pool for each seed, without changing later sampling or other consumers of the task pool. Balanced sampling should preserve valid empty labels and not affect unrelated random consumers.

Verification

  • before, on current main: the candidate sampler regression tests produced 4 failures and 2 passes
  • after: uv run --no-sync pytest tests/unit/prompt/test_prompt_manager.py -q (6 passed)
  • uv run --no-sync ruff format --check src/lighteval/tasks/prompt_manager.py tests/unit/prompt/test_prompt_manager.py
  • uv run --no-sync ruff check src/lighteval/tasks/prompt_manager.py tests/unit/prompt/test_prompt_manager.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant