Skip to content

Fix inspect-ai GenerateConfig field names in lighteval eval - #1331

Open
harshasiddartha wants to merge 1 commit into
huggingface:mainfrom
harshasiddartha:fix/inspect-ai-generateconfig-field-names
Open

Fix inspect-ai GenerateConfig field names in lighteval eval#1331
harshasiddartha wants to merge 1 commit into
huggingface:mainfrom
harshasiddartha:fix/inspect-ai-generateconfig-field-names

Conversation

@harshasiddartha

Copy link
Copy Markdown

Fixes #1327.

lighteval eval crashes at startup on a fresh install, for every model and task:

ValidationError: 1 validation error for GenerateConfig
  Value error, Unknown GenerateConfig field(s): frequence_penalty, log_probs, response_format.

Three of the generation parameters main_inspect.py forwards to inspect_ai.eval_set are not GenerateConfig fields:

forwarded actual field
frequence_penalty frequency_penalty
log_probs logprobs
response_format (str) response_schema (ResponseSchema)

Worth noting these names were never correct — they are wrong on inspect-ai==0.3.140 too, which is the declared minimum. Older releases just dropped unknown fields silently, so --frequence-penalty, --log-probs and --response-format were accepted on the command line and then had no effect. Current releases validate, which turns that silent no-op into a hard startup failure. So this is a rename rather than a compatibility shim, and it works across the whole supported inspect-ai range — no upper bound needed.

Changes

  • Rename the three parameters in the eval command and in InspectAIModelConfig, which mirrors them.
  • --response-schema now parses its JSON string into a ResponseSchema, accepting either a bare JSON schema (wrapped under the default name response) or a full ResponseSchema object.
  • Add tests/unit/test_main_inspect.py, which asserts every generation parameter is a real GenerateConfig field and is exposed on the CLI. That's the regression guard — a name drifting out of sync now fails in CI instead of at a user's startup. Plus coverage for the schema parsing.

Verification

The command from the issue now runs to completion on current inspect-ai (0.3.258):

lighteval eval mockllm/model "lighteval|gsm8k|0" --max-samples 2 --log-dir out
...
Completed all tasks in 'out' successfully

|    Model    |gsm8k|
|-------------|----:|
|mockllm/model|    0|

tests/unit/test_main_inspect.py passes (8 tests); ruff check and ruff format --check are clean on the touched files.

Note on the CLI rename

--frequence-penalty--frequency-penalty and --log-probs--logprobs are user-visible flag renames. Since neither flag has ever had any effect (silently ignored before, crashing now), there is nothing depending on the old spellings. Happy to add hidden aliases for the old names if you'd rather not break them outright.

`lighteval eval` crashes at startup on a fresh install for every model and
task:

    ValidationError: 1 validation error for GenerateConfig
      Value error, Unknown GenerateConfig field(s): frequence_penalty,
      log_probs, response_format.

Three of the generation parameters forwarded to `inspect_ai.eval_set` do not
exist on `GenerateConfig`:

- `frequence_penalty` is a typo for `frequency_penalty`
- `log_probs` is spelled `logprobs`
- `response_format` is `response_schema`, and takes a `ResponseSchema`
  rather than a string

None of these names have ever been correct — older inspect-ai releases
silently dropped unknown fields, so the three options were accepted on the
command line and then ignored. Current releases reject them, which turns a
silent no-op into a hard startup failure.

Rename all three (in the CLI and in `InspectAIModelConfig`) and parse
`--response-schema` into a `ResponseSchema`, accepting either a bare JSON
schema or a full `ResponseSchema` object.

The new test asserts every generation parameter is a real `GenerateConfig`
field and is exposed on the command line, so a name drifting out of sync
fails in CI rather than at a user's startup.

Fixes huggingface#1327

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lighteval eval crashes on fresh install: GenerateConfig rejects frequence_penalty / log_probs / response_format with current inspect-ai

1 participant