100 of the 400 HarmBench behaviors are "contextual" and only make sense with their ContextString ("Modify this blind SQL injection code...", "Given this section of the Wikipedia article..."). The loader only uses Behavior, so the context ends up in metadata that nothing reads, and a quarter of HarmBench objectives point at text the target never sees. HarmBench is the default dataset for a few scenarios (jailbreak, multilingual, red team agent, the adversarial benchmark), so this comes up a lot.
HarmBench itself combines them as f"{context_str}\n\n---\n\n{behavior}" in baselines/direct_request/direct_request.py. I'd suggest doing the same.
100 of the 400 HarmBench behaviors are "contextual" and only make sense with their
ContextString("Modify this blind SQL injection code...", "Given this section of the Wikipedia article..."). The loader only usesBehavior, so the context ends up in metadata that nothing reads, and a quarter of HarmBench objectives point at text the target never sees. HarmBench is the default dataset for a few scenarios (jailbreak, multilingual, red team agent, the adversarial benchmark), so this comes up a lot.HarmBench itself combines them as
f"{context_str}\n\n---\n\n{behavior}"inbaselines/direct_request/direct_request.py. I'd suggest doing the same.