MAINT simplify how default adversarial and scorer targets are set in scenarios#1695
MAINT simplify how default adversarial and scorer targets are set in scenarios#1695behnam-o wants to merge 5 commits intomicrosoft:mainfrom
Conversation
behnam-o
changed the title
| from pyrit.registry import TargetRegistry | ||
|
|
||
|
|
||
| def get_default_scorer_target() -> PromptChatTarget: |
There was a problem hiding this comment.
Love this, I had a similar thought
| ValueError: If the registered target does not support multi-turn. | ||
| """ | ||
| return _get_default_chat_target(preferred_target_key="objective_scorer_chat") | ||
|
|
There was a problem hiding this comment.
Do you think we should move get_default_objective_scorer here also?
|
|
||
| def _get_default_objective_scorer(self) -> TrueFalseScorer: | ||
| # Deferred import to avoid circular dependency: | ||
| composite_scorer_questions_path = type(self).COMPOSITE_SCORER_QUESTIONS_PATH |
There was a problem hiding this comment.
this is cool, but I wonder if it's hidden? Should we do a method override?
And we could potentially make it more flexible so that individual scenario can add more scorers to the composite
There was a problem hiding this comment.
idk. maybe it's good as is, I do like how it saves a bunch of code.
| # Deferred import to avoid circular dependency. | ||
| from pyrit.setup.initializers.components.scorers import ScorerInitializerTags | ||
|
|
||
| entries = ScorerRegistry.get_registry_singleton().get_by_tag(tag=ScorerInitializerTags.DEFAULT_OBJECTIVE_SCORER) |
There was a problem hiding this comment.
We potentially want to make more use of this ^^^
Because this has metrics to be the "best" objective scorer including composite scorers.
| ADVERSARIAL_CHAT_KEY="xxxxx" | ||
| ADVERSARIAL_CHAT_MODEL="deployment-name" | ||
|
|
||
| # Objective Scorer chat target (used in scorers in scenarios) |
There was a problem hiding this comment.
We may want to look at the scorer initializer, so we can make sure to do metrics for whatever the default is
2 participants
Changes:
1- Use a common logic for resolving default adversarial and scorer targets in scenarios
2- Have a fallback mechanism that uses the target identified by OPENAI_CHAT_*** env variables. This allows someone to only define 1 endpoint and get a runnable scenario