If you pass a prepended_conversation to TAP or PAIR, the first turn never happens properly.
_TreeOfAttacksNode._is_first_turn() checks whether the objective conversation in memory is empty, but the prepended messages are written into that same conversation. So after any prepend it's never the first turn, which means:
set_adversarial_system_prompt never runs, so the adversarial chat has no system prompt at all, and the prepended context is never usednext_message is silently ignored- with a system-only prepend, the subsequent-turn path raises "Cannot proceed without an assistant response", every node gets pruned, and the attack reports FAILURE without sending anything
With mock targets, a scorer that would succeed right away, and just a system message prepended:
TreeOfAttacksWithPruningAttack: FAILURE | prompts sent to target: [] | adversarial system prompts set: 0
PAIRAttack: FAILURE | prompts sent to target: [] | adversarial system prompts set: 0
Fix: don't count the prepended messages when deciding whether it's the first turn.
If you pass a
prepended_conversationto TAP or PAIR, the first turn never happens properly._TreeOfAttacksNode._is_first_turn()checks whether the objective conversation in memory is empty, but the prepended messages are written into that same conversation. So after any prepend it's never the first turn, which means:set_adversarial_system_promptnever runs, so the adversarial chat has no system prompt at all, and the prepended context is never usednext_messageis silently ignoredWith mock targets, a scorer that would succeed right away, and just a system message prepended:
Fix: don't count the prepended messages when deciding whether it's the first turn.