FEAT: Plan-Time Modality Validation and Policy for Scenario Runs

.github/instructions/scenarios.instructions.md

@@ -46,6 +46,55 @@ the constructor — no classmethod indirection required.
    abstract extension point every scenario must define (see "AtomicAttack Construction" below).
    Matrix-shaped scenarios delegate to `build_matrix_atomic_attacks(context=...)` in one line.
 
+## Modality Validation
+
+`Scenario.initialize_async` validates every `AtomicAttack` returned by
+`_build_atomic_attacks_async` before any of them is queued, and applies `MODALITY_POLICY`
+(`ModalityPolicy.SKIP` by default, also `WARN` and `RAISE`). Scenario authors get this for free
+and normally do not override it. On resume, it first reconstructs the persisted seed groups
+from the full dataset without resampling, then checks only those groups. `SKIP` fails loudly
+instead of removing an incompatible saved attack and silently changing the run plan; `WARN`
+retains it as configured.
+
+- The **request chain** projects each seed group's data types through the attack's request
+  converters; each possible final message combination is checked against the target's
+  advertised `input_modalities` combinations. A converter's multiple declared output types
+  are alternatives for one piece, not simultaneous message pieces. When only some outcomes
+  can reach the target, validation reports `UNKNOWN` rather than skipping the whole attack.
+  Projection is bounded at 256 piece-type combinations; larger searches are also `UNKNOWN`.
+  Conversion selection (including `indexes_to_apply`) is
+  **per message piece**: preserve ordered pieces through each converter configuration, then
+  collapse their resulting types to a **message-level set** for target compatibility. A single
+  text piece selected at index 0 and converted to an image leaves no text; selecting only
+  index 0 of two text pieces leaves text in the second piece. Declarations are read literally —
+  a target that accepts a lone image advertises `{image_path}` as well as `{text, image_path}`.
+  A constructor-supplied `AtomicAttack.next_message` execution override replaces the seed's
+  message before projection; an explicit `None` selects the objective-text fallback. Inputs
+  not safely representable at plan time remain `UNKNOWN`.
+- The **response chain** checks each advertised target output combination against the scorer.
+  Project alternative outputs of configured response converters separately before checking
+  the types the scorer receives.
+  When response piece indexes cannot be known, report `UNKNOWN` rather than guessing which
+  pieces were converted.
+  A scorer allowing unsupported pieces alongside readable ones needs at least one readable type
+  in each combination; a strict scorer needs to read every piece. This differs from the ability
+  to return no score for a wholly unreadable response, which composites use to assess their
+  children's applicability. If only some possible combinations can be scored,
+  the response-chain verdict is `UNKNOWN`, not a reason to skip the attack. This type check
+  does not establish that the resulting score is meaningful. After the scorer condition-routing
+  refactor, composite scorers declare their response modality compatibility as `UNKNOWN` until
+  their per-child applicability can be reconciled with the new expectation-selection contract;
+  this keeps potentially runnable attacks but defers some early incompatibility detection.
+- Anything indeterminate is `UNKNOWN` and never blocks a run.
+- Only turn 0 is checked. Media routing across later turns belongs to `_ModalityFeedbackRouter`,
+  which multi-turn attacks consult at execution time.
+- A `SequentialAttack` wrapper delegates its actual request, target, converters, and scorer to
+  its children, so the wrapper itself reports `UNKNOWN` rather than treating the absence of
+  `next_message` as a text request to its nominal target.
+
+Override `MODALITY_POLICY` to `RAISE` when an incompatible pairing means the run is
+misconfigured rather than merely narrower than intended.
+
 ## Constructor Pattern
 
 ```python

.github/instructions/scorers.instructions.md

@@ -8,6 +8,23 @@ Scorers evaluate model responses against an objective and live under `pyrit/scor
 
 **Does not own** (see [framework.md](../../doc/code/framework.md)): acting on its own result. A scorer evaluates a response and returns a score; branching on that score is the attack's job and aggregating scores across runs is analytics'. It may call a target to evaluate, but must not send the attack's objective prompt or manage the conversation. Flag such bleed in review.
 
+## Composite modality declarations
+
+`TrueFalseCompositeScorer` aggregates only applicable child scores; a child returning `[]`
+does not vote `False`. After the expectation-routing refactor, composite modality inference
+remains deferred: the composite declares `None` (`UNKNOWN`) so plan-time validation does not
+skip an attack on an unverified union of child types. This reduces early rejection but does
+not change runtime scoring or each child's condition selection. One-to-one wrappers
+(`TrueFalseInverterScorer` and `FloatScaleThresholdScorer`) still delegate their child's
+modality declaration and skip behavior.
+
+For mixed responses, `allows_unsupported_pieces` separately reports whether readable
+pieces can be scored alongside unsupported ones. `raise_on_no_valid_pieces=True` still
+allows a mixed response when `enforce_all_pieces_valid=False`, but it prevents a wholly
+unreadable response from yielding `[]`. Keep that case distinct from
+`skips_unsupported_data_types`, which is required before a composite may assume a child
+will be non-applicable rather than raise.
+
 ## Constructor contract
 
 `Scorer` subclasses MUST use the keyword-only constructor shape:

doc/code/scenarios/0_scenarios.ipynb

@@ -477,7 +477,31 @@
     "  when an unmodified-prompt comparison is valid but not useful enough to run by default.\n",
     "- **`Forbidden`** — the baseline is unavailable and passing `include_baseline=True` raises. Use\n",
     "  when the scenario's semantics make a single-shot unmodified prompt meaningless as a comparator\n",
-    "  (e.g., benchmarks comparing across adversarial models, or multi-turn-only scenarios)."
+    "  (e.g., benchmarks comparing across adversarial models, or multi-turn-only scenarios).\n",
+    "\n",
+    "### Modality Validation\n",
+    "\n",
+    "Before any attack is queued, a scenario checks that each `AtomicAttack` can actually carry its\n",
+    "payload. The seed's data types are projected through the request converters and must be accepted\n",
+    "by the objective target, and whatever the target may emit must be readable by the scorer. A\n",
+    "mismatch — a converter that produces an image for a text-only target, say — is caught during\n",
+    "`initialize_async` rather than part-way through a run.\n",
+    "\n",
+    "`MODALITY_POLICY` decides what happens to an incompatible attack:\n",
+    "\n",
+    "- **`SKIP`** (default) — the attack is dropped with a warning and the rest of the run proceeds. If\n",
+    "  every attack is dropped the scenario raises rather than reporting an empty success.\n",
+    "- **`WARN`** — the attack is kept and the problem is logged.\n",
+    "- **`RAISE`** — `initialize_async` aborts with `ModalityValidationError`, a `ValueError` subclass.\n",
+    "\n",
+    "Compatibility that cannot be determined never blocks a run: a target that does not declare its\n",
+    "capabilities, an attack that exposes no scoring config, and a scorer that never declared its data\n",
+    "types are all treated as unknown rather than incompatible. Only the first turn is checked — media\n",
+    "routing across later turns belongs to the multi-turn attacks themselves, at execution time.\n",
+    "For Tree of Attacks (TAP) with width greater than one, a text-capable objective gets sibling\n",
+    "roots starting from generated text, projected through the same converters. If only the seeded\n",
+    "media root is incompatible, the mixed-root attack reports unknown and is retained under `SKIP`.\n",
+    "If the objective requires media, TAP seeds every root; incompatible requests are still rejected."
    ]
   },
   {

doc/code/scenarios/0_scenarios.py

@@ -194,6 +194,30 @@ async def _build_atomic_attacks_async(self, *, context):
 # - **`Forbidden`** — the baseline is unavailable and passing `include_baseline=True` raises. Use
 #   when the scenario's semantics make a single-shot unmodified prompt meaningless as a comparator
 #   (e.g., benchmarks comparing across adversarial models, or multi-turn-only scenarios).
+#
+# ### Modality Validation
+#
+# Before any attack is queued, a scenario checks that each `AtomicAttack` can actually carry its
+# payload. The seed's data types are projected through the request converters and must be accepted
+# by the objective target, and whatever the target may emit must be readable by the scorer. A
+# mismatch — a converter that produces an image for a text-only target, say — is caught during
+# `initialize_async` rather than part-way through a run.
+#
+# `MODALITY_POLICY` decides what happens to an incompatible attack:
+#
+# - **`SKIP`** (default) — the attack is dropped with a warning and the rest of the run proceeds. If
+#   every attack is dropped the scenario raises rather than reporting an empty success.
+# - **`WARN`** — the attack is kept and the problem is logged.
+# - **`RAISE`** — `initialize_async` aborts with `ModalityValidationError`, a `ValueError` subclass.
+#
+# Compatibility that cannot be determined never blocks a run: a target that does not declare its
+# capabilities, an attack that exposes no scoring config, and a scorer that never declared its data
+# types are all treated as unknown rather than incompatible. Only the first turn is checked — media
+# routing across later turns belongs to the multi-turn attacks themselves, at execution time.
+# For Tree of Attacks (TAP) with width greater than one, a text-capable objective gets sibling
+# roots starting from generated text, projected through the same converters. If only the seeded
+# media root is incompatible, the mixed-root attack reports unknown and is retained under `SKIP`.
+# If the objective requires media, TAP seeds every root; incompatible requests are still rejected.
 
 # %% [markdown]
 #

pyrit/executor/attack/core/attack_strategy.py

@@ -61,6 +61,7 @@
     )
     from pyrit.executor.attack.core.attack_result_attribution import AttackResultAttribution
     from pyrit.message_normalizer import MessageListNormalizer
+    from pyrit.prompt_normalizer import ConverterConfiguration
     from pyrit.prompt_target import PromptTarget
     from pyrit.prompt_target.common.target_capabilities import CapabilityName
 
@@ -626,7 +627,7 @@ def __init__(
         if not hasattr(self, "_request_converters"):
             self._request_converters: list[Any] = []
         if not hasattr(self, "_response_converters"):
-            self._response_converters: list[Any] = []
+            self._response_converters: list[ConverterConfiguration] = []
 
     def _get_prepended_normalizer_overrides(
         self,
@@ -820,6 +821,15 @@ def get_request_converters(self) -> list[Any]:
         """
         return self._request_converters
 
+    def get_response_converters(self) -> list[ConverterConfiguration]:
+        """
+        Return response converter configurations applied before objective scoring.
+
+        Returns:
+            list[ConverterConfiguration]: The configured response converters.
+        """
+        return self._response_converters
+
     async def execute_with_context_async(self, *, context: AttackStrategyContextT) -> AttackStrategyResultT:
         """
         Execute an attack and persist its completed result after teardown.

pyrit/executor/attack/multi_turn/tree_of_attacks.py

@@ -1753,6 +1753,19 @@ def get_attack_scoring_config(self) -> AttackScoringConfig | None:
         """
         return self._attack_scoring_config
 
+    @property
+    def has_unseeded_first_turn_roots(self) -> bool:
+        """
+        Whether sibling roots generate text instead of consuming the first-turn seed.
+
+        Returns:
+            bool: True when more than one root exists and the objective permits text-only requests.
+        """
+        return (
+            self._configuration.tree_width > 1
+            and not self._modality_router.objective_target_requires_media_on_first_turn
+        )
+
     def get_attack_adversarial_config(self) -> AttackAdversarialConfig | None:
         """
         Get the effective adversarial configuration used by this strategy.

pyrit/scenario/core/__init__.py

@@ -23,6 +23,12 @@
         ResolvedDataset,
         require_nonempty,
     )
+    from pyrit.scenario.core.modality_validation import (
+        ModalityPolicy,
+        ModalityReport,
+        ModalityValidationError,
+        ModalityVerdict,
+    )
     from pyrit.scenario.core.scenario import BaselineAttackPolicy, Scenario
     from pyrit.scenario.core.scenario_target_defaults import (
         get_default_adversarial_target,
@@ -42,6 +48,10 @@
     "DatasetConstraintError": "pyrit.scenario.core.dataset_configuration",
     "DatasetSourceKind": "pyrit.scenario.core.dataset_configuration",
     "INLINE_DATASET_NAME": "pyrit.scenario.core.dataset_configuration",
+    "ModalityPolicy": "pyrit.scenario.core.modality_validation",
+    "ModalityReport": "pyrit.scenario.core.modality_validation",
+    "ModalityValidationError": "pyrit.scenario.core.modality_validation",
+    "ModalityVerdict": "pyrit.scenario.core.modality_validation",
     "Parameter": "pyrit.models.parameter",
     "ResolvedDataset": "pyrit.scenario.core.dataset_configuration",
     "require_nonempty": "pyrit.scenario.core.dataset_configuration",

pyrit/scenario/core/atomic_attack.py

@@ -182,6 +182,16 @@ def attack_technique(self) -> AttackTechnique:
         """The attack technique for this atomic attack."""
         return self._attack_technique
 
+    def get_next_message_override(self) -> tuple[bool, object]:
+        """
+        Return whether execution supplies a replacement for the seed's next message.
+
+        Returns:
+            tuple[bool, object]: Whether a constructor override exists and its value.
+                A present ``None`` overrides the seed and triggers the attack's fallback.
+        """
+        return "next_message" in self._attack_execute_params, self._attack_execute_params.get("next_message")
+
     @property
     def technique_name(self) -> str | None:
         """Catalog name of the technique that built this attack."""

pyrit/scenario/core/attack_technique_factory.py

@@ -37,7 +37,6 @@
     AttackTechniqueSeedGroup,
     ComponentIdentifier,
     Identifiable,
-    PromptDataType,
     SeedIdentifier,
     SeedPrompt,
     SeedSimulatedConversation,
@@ -48,6 +47,7 @@
 )
 from pyrit.models.seeds.seed_simulated_conversation import NextMessageSystemPromptPaths
 from pyrit.scenario.core.attack_technique import AttackTechnique
+from pyrit.scenario.core.modality_validation import project_request_chain
 from pyrit.scenario.core.scenario_target_defaults import get_default_adversarial_target
 
 if TYPE_CHECKING:
@@ -493,32 +493,15 @@ def can_append_request_converter(self, *, converter_type: type[Converter]) -> bo
         if "attack_converter_config" not in self._get_accepted_params():
             return False
 
-        output_types: set[PromptDataType] = {"text"}
         converter_config = self._attack_kwargs.get("attack_converter_config")
-        if converter_config is None:
-            return "text" in converter_type.SUPPORTED_INPUT_TYPES
-
-        for configuration in converter_config.request_converters:
-            next_output_types: set[PromptDataType] = set()
-            for output_type in output_types:
-                applies_to_type = (
-                    not configuration.prompt_data_types_to_apply
-                    or output_type in configuration.prompt_data_types_to_apply
-                )
-                if not applies_to_type:
-                    next_output_types.add(output_type)
-                    continue
-
-                converted_types: set[PromptDataType] = {output_type}
-                for built_in_converter in configuration.converters:
-                    if not all(built_in_converter.input_supported(data_type) for data_type in converted_types):
-                        return False
-                    converted_types = set(built_in_converter.supported_output_types)
-
-                next_output_types.update(converted_types)
-                if configuration.indexes_to_apply:
-                    next_output_types.add(output_type)
-            output_types = next_output_types
+        request_converters = converter_config.request_converters if converter_config is not None else []
+        output_types, failure_reason = project_request_chain(
+            start_types=["text"],
+            request_converters=request_converters,
+            piece_indexes_known=False,
+        )
+        if failure_reason is not None:
+            return False
 
         return bool(output_types) and output_types.issubset(converter_type.SUPPORTED_INPUT_TYPES)