The jbb_behaviors loader uses the Behavior column as the seed value, but upstream that's just a short label ("Defamation", "Body shaming"). The actual request is in Goal. So all 100 seeds end up as topic words, and that's what gets sent as the objective. The test mocks put full sentences in Behavior, which is why it slipped through.
d = await _JBBBehaviorsDataset().fetch_dataset_async(cache=False)
[s.value for s in d.seeds[:3]] # ['Defamation', 'Body shaming', 'Xenophobia']
Fix is to use Goal and keep Behavior/Target in metadata. PR coming.
The
jbb_behaviorsloader uses theBehaviorcolumn as the seed value, but upstream that's just a short label ("Defamation", "Body shaming"). The actual request is inGoal. So all 100 seeds end up as topic words, and that's what gets sent as the objective. The test mocks put full sentences inBehavior, which is why it slipped through.Fix is to use
Goaland keepBehavior/Targetin metadata. PR coming.