FEAT: Improve adversarial benchmark dataset and scoring

[hannahwestra25]

Description

Improves the adversarial benchmark end to end: a balanced 120-objective dataset (adversarial_benchmark_v1), better objective scoring, and Role Play / Crescendo / TAP technique defaults.

The ADO pipeline gains quick and full profiles and reuses compatible results from a same-branch SQLite artifact, so a re-run only pays for objectives it has not already measured.

Attacks that never reached the objective target are no longer recorded as measured failures. When the adversarial model produces no attacker turn, the objective target is never probed, so there is no verdict to record. Those attacks now return AttackOutcome.UNDETERMINED and carry a typed marker (pyrit/executor/attack/core/attack_preparation.py) shared by RedTeamingAttack, PromptSendingAttack, and generate_simulated_conversation_async. That keeps them out of the result cache and out of resume-completion.

The typed marker exists so the producers and consumers of that signal agree on one contract without depending on each other's modules, and so the failure reason is guaranteed non-empty — previously the consumer read a magic metadata key imported from red_teaming and could end up with a None reason that silently fell through.

Provider blocks vs. adversarial-model refusals

An SDK-reported refusal and a provider content filter both surface as response_error == "blocked", so they were previously indistinguishable. AdversarialChatRefusedException now subclasses AdversarialChatResponseBlockedException, and _raise_for_adversarial_error branches on the message piece's structured refusal. The two are recorded as distinct failure kinds (ADVERSARIAL_CHAT_BLOCKED vs ADVERSARIAL_CHAT_REFUSED).

Both stay UNDETERMINED: in either case nothing was sent to the objective target, so scoring them as a defensive win would put an unsent attack in the ASR denominator. They are recorded separately so the distinction is auditable. A plain-text refusal from the adversarial model is unaffected — it carries no error, is still sent to the target, and remains a genuine measured FAILURE.

Notable behavior change

Scenario._get_completed_objective_hashes_for_attack now leaves a row pending only when it is an ERROR or carries a preparation-failure marker. This applies to all scenarios, not just this benchmark.

UNDETERMINED alone is deliberately not enough to re-run a row. A row that reached the objective target and ended UNDETERMINED — for example because no objective scorer was configured, or the scorer abstained — recorded the best verdict its configuration allows. Re-running it would re-send the objective on every resume without ever converging. Rows that reached a real verdict are unaffected.

Cache reuse break for TAP consumers

TreeOfAttacksWithPruningAttack now includes its search configuration (tree_width, tree_depth, branching_factor, on_topic_checking_enabled, desired_response_prefix) in its identifier. These parameters change what the attack actually does, so results produced under different settings were previously indistinguishable and could be reused for one another.

This changes TAP's eval hash for every consumer, not just this benchmark — including FoundryScenario, where TAP is one of the available techniques. Existing TAP results will no longer match for cache reuse and will be re-run once.

FoundryScenario.VERSION is deliberately left at 1. A changed eval hash already fails safe: it produces a cache miss, so stale results are ignored rather than incorrectly reused. Bumping the scenario version would additionally invalidate reuse for every other Foundry technique whose identifier did not change, turning a narrow and correct invalidation into a broad one.

Other changes

• Generic technique overrides. The scenario exposes a single typed --technique-args parameter taking <technique>.<argument>=<value> entries, dispatched through AttackTechniqueFactory.with_attack_kwargs. This replaces the TAP-specific tap_tree_width / tap_tree_depth / tap_branching_factor / tap_batch_size parameters introduced earlier in this same PR, so the scenario no longer hardcodes knowledge of one technique. Those parameters never shipped, so this is not a breaking change. Unknown technique names, unknown kwargs, and malformed entries all fail with a clear message. The pipeline keeps its operator-facing tapTreeWidth-style inputs and its quick/full tuning; they are now marshalled into --technique-args.
• Scorer block policy reaches the scorers that call an LLM. RAISE_IF_DEFAULT_SCORER_BLOCKS now also applies on the registry-default-scorer branch. The registered default (scale_and_refusal) is a TrueFalseCompositeScorer wrapping a FloatScaleThresholdScorer and a TrueFalseInverterScorer, so a policy applied only at the root never reached the SelfAsk* leaves that actually decide whether a blocked scoring response raises. Scorer.with_scorer_block_policy — mirroring the existing get_chat_target delegation convention — is overridden by those three wrappers so each level forwards to its own children and copies only when something changed, leaving the shared registry instance unmutated.
• Cache lookup degrades safely. If the analytics lookup against a restored cache fails partway through, the scenario warns, discards any partially collected reuse state, and continues as a cold run rather than pruning objectives from incomplete data. Identifier-construction and programming errors still propagate.

Tests and Documentation

Unit tests cover each behavior above rather than its implementation:

• A structured refusal and a provider content filter are raised and recorded as distinct kinds; both stay UNDETERMINED and round-trip their typed marker through memory. A blocked result with no outcome reason still yields a usable reason.
• Resume leaves preparation-failure rows pending while counting both measured failures and target-reached UNDETERMINED rows as complete.
• --technique-args parsing, including a non-TAP technique, unregistered technique names, unknown kwargs, malformed entries, and conflicting repeats.
• The registry-default scorer test builds the real scale_and_refusal shape and asserts the policy lands on both LLM-backed leaves while the shared instance and its leaves stay unmutated. Each wrapper also has a focused delegation test, including that it returns itself when already compliant. The fallback scorer path is covered separately.
• Cache lookup failure after partial success produces no partial pruning or reuse.

The pipeline test now verifies structurally that every benchmark variable consumed by a run step's env: block is emitted by the profile-resolution step, and that every declared env var is actually read. It no longer restates literal profile values or template strings, so profile drift is caught without the test needing an edit for every tuning change.

No documentation or notebook changes, and no documented public API surface changes, so JupyText was not run.