Skip to content

Conversation

@rlundeen2
Copy link
Contributor

@rlundeen2 rlundeen2 commented Dec 29, 2025

This PR introduces DatasetConfiguration class that is passed to scenarios in initialize_async to address several pain points.

Big available default datasets

It was tough to change the default datasets - some were too big and some too small. As an example, Foundry used harm_bench, which is too big for the default. So it would randomly select 4 by default. But how could users run against all 100? There was no way to configure this. On the flip side garak.encoding_scenario had a large dataset that took a very long time to run, but there was no way to make it small by default and still have the entire dataset available.

This change allows scenarios to be configured with a default (e.g. name=harm_bench, max=4). But users can easily configure both the dataset name or max differently.

It also helps with our end to end tests, which were taking a million years because encoding scenario had so many datasets (this pr reduces to only 3 by default)

Allows dataset params from the front end

This also allows users to specify the dataset names they want to use from pyrit_scan and pyrit_shell. Previously, users had to use the dataset defaults.

--dataset-names DATASET_NAMES [DATASET_NAMES ...]
Dataset names to load for the scenario (overrides scenario defaults)
--max-dataset-size MAX_DATASET_SIZE
Maximum number of seed groups to use (randomly samples if dataset is larger)

Deprecates Incompatible Parameters

objective, seed_prompts are now deprecated as initialization parameters for scenarios

Tests

  • Unit tests added!
  • End to end tests passed 2c281be

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant