FEAT: Scenario DatasetConfiguration #1288
Open
+1,539
−485
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces
DatasetConfigurationclass that is passed to scenarios ininitialize_asyncto address several pain points.Big available default datasets
It was tough to change the default datasets - some were too big and some too small. As an example, Foundry used harm_bench, which is too big for the default. So it would randomly select 4 by default. But how could users run against all 100? There was no way to configure this. On the flip side garak.encoding_scenario had a large dataset that took a very long time to run, but there was no way to make it small by default and still have the entire dataset available.
This change allows scenarios to be configured with a default (e.g. name=harm_bench, max=4). But users can easily configure both the dataset name or max differently.
It also helps with our end to end tests, which were taking a million years because encoding scenario had so many datasets (this pr reduces to only 3 by default)
Allows dataset params from the front end
This also allows users to specify the dataset names they want to use from pyrit_scan and pyrit_shell. Previously, users had to use the dataset defaults.
--dataset-names DATASET_NAMES [DATASET_NAMES ...]
Dataset names to load for the scenario (overrides scenario defaults)
--max-dataset-size MAX_DATASET_SIZE
Maximum number of seed groups to use (randomly samples if dataset is larger)
Deprecates Incompatible Parameters
objective, seed_prompts are now deprecated as initialization parameters for scenarios
Tests