FEAT: Plan-Time Modality Validation and Policy for Scenario Runs - #2773
Open
Victor Valbuena (ValbuenaVC) wants to merge 35 commits into
Open
Victor Valbuena (ValbuenaVC) wants to merge 35 commits into
Victor Valbuena (ValbuenaVC) wants to merge 35 commits into
Conversation
Extend the shared ``get_mock_target`` helper with keyword-only ``input_modalities`` /
``output_modalities`` so tests can build a ``MagicMock(spec=PromptTarget)`` whose
``configuration.capabilities`` carries real ``frozenset`` modality combinations instead of
MagicMock children. With no modality arguments the helper is unchanged.
Add ``modality_combos()`` to canonicalise combination shapes, and a new
``tests/unit/modality_profiles.py`` with six named profiles (text-only, vision, image-edit,
realtime audio in/out, video generation) as a shared vocabulary for capability-aware tests.
Add a contract suite for the helper and profiles: back-compat pins (including that the default
mock's capabilities are *not* real frozensets — the hazard downstream validation must
tolerate), every ``PromptDataType`` round-trips on both sides, ``{text, <media>}`` combos for
each media type, iterable coercion, and proof that invalid input propagates pydantic's
``ValidationError`` rather than being swallowed.
Migrate the ``_ModalityFeedbackRouter`` tests off their local ``_build_target`` helper onto
``get_mock_target``; no assertions change.
Groundwork for plan-time scenario modality validation.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Scorers already declare the data types they can score via ``ScorerPromptValidator(supported_data_types=...)``, but that declaration was only reachable through the private ``scorer._validator._supported_data_types`` and only enforced at score time — after an attack has already run. Expose it publicly so compatibility can be judged before a run. Add ``ScorerPromptValidator.supported_data_types`` and ``has_declared_data_types``. The latter distinguishes "declared nothing" (and so fell back to the permissive all-types default) from "deliberately declared every type" — a distinction plan-time code needs and the runtime default erases. The runtime default itself is deliberately unchanged. Add ``Scorer.supported_data_types`` returning ``None`` for "unknown", overridden by ``MessageScorer`` to read its validator. Wrapper scorers delegate: the inverter and threshold scorers report their child's declaration, and the composite reports the intersection of its children, or ``None`` if any child is unknown. Replace the two private reach-ins in ``AudioTranscriptHelper`` and ``VideoHelper`` with the public accessor. As a side effect these now accept wrapper scorers, which previously raised ``AttributeError`` for lack of a ``_validator``. Groundwork for plan-time scenario modality validation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
``AttackTechniqueFactory.can_append_request_converter`` already projected a text objective through a baked request-converter chain to decide whether another converter could be appended. Scenario-level plan-time modality validation needs the same projection, but starting from the seed's own data types and ending at the objective target rather than at an appended converter. Move the projection into ``pyrit/scenario/core/modality_validation.py`` as ``project_request_chain``, parameterised on the start types, and have the factory call it. Behaviour is unchanged: conditional configurations still preserve the unconverted type, ``indexes_to_apply`` still branches, and a converter that cannot accept the type reaching it still fails the chain. The extracted form additionally reports *which* converter broke the chain and what it does accept, and iterates start types in sorted order so that message is deterministic when several types are in play. The factory's existing tests are unchanged and still pass, which is the guard that the extraction preserved behaviour. Groundwork for plan-time scenario modality validation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add the verdict layer on top of ``project_request_chain``: given an ``AtomicAttack`` that has
been constructed but not yet queued, decide whether its payload can actually reach its target
and its scorer.
Two chains are checked, both for turn 0. The request chain takes each seed group's own data
types (the pieces of the ``next_message`` the attack will send, after merging any technique
seed group), projects them through the request converters, and requires one of the target's
advertised input modality combinations to cover the result. The response chain requires the
scorer to declare every data type the target may emit.
Seed groups are projected independently rather than unioned: two groups are two separate
requests, so demanding a single combination that covers both types would reject pairings that
run fine.
A target that advertises no bare ``{"text"}`` combination requires media on every request.
``_ModalityFeedbackRouter`` already reads that signal to decide whether turn 0 can be built and
raises when it cannot, so a text-only request to such a target is reported incompatible rather
than merely unadvertised.
Anything indeterminate is ``UNKNOWN`` and never blocks: a target whose capabilities are not
readable, an attack exposing no scoring config, a scorer that never declared its types. An
``UNKNOWN`` leg does not cancel a ``COMPATIBLE`` one.
Adds ``ModalityPolicy`` (``SKIP``/``WARN``/``RAISE``, mirroring ``ScorerOverridePolicy``) and
``ModalityValidationError``, a ``ValueError`` subclass so existing handlers keep working. The
policy is not yet applied anywhere; that follows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Apply the derived modality verdict in ``Scenario.initialize_async``, immediately after ``_build_atomic_attacks_async`` returns and before the display-group map, the persisted run plan, or any queued work is derived from the attack list. Modality mismatches surfaced only at execution time before this: a scorer rejecting a data type it cannot read, or a multi-turn attack failing to build its first request, both after the run had started spending. ``MODALITY_POLICY`` is a class attribute alongside ``BASELINE_ATTACK_POLICY``, defaulting to ``SKIP``. ``SKIP`` drops the attack and logs a warning; dropping a whole atomic attack changes what a run covers, so it is loud even though the policy allows it. ``WARN`` keeps the attack. ``RAISE`` aborts with ``ModalityValidationError``. Skipping every attack raises rather than reporting a successful run that tested nothing. Validation runs on the resume path too, since it sits before that branch returns. Attacks whose compatibility cannot be determined are always kept, so scenarios exercised against mock targets are unaffected. FigStep opts into ``WARN``. Its seeds are a single text-plus-image message and its caller-supplied technique converters are appended unscoped, so the normalizer runs them against both pieces; no converter in the library accepts ``text`` and ``image_path`` together, so any such converter raises ``Input type not supported`` at run time. The verdict is accurate but the limitation predates this check, so FigStep warns rather than dropping the attack until those converters are scoped to the text piece. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Describe ``MODALITY_POLICY`` where scenario behaviour is documented. The scenarios notebook gains a "Modality Validation" section next to the baseline-attack discussion, and the scenario authoring contract gains a section stating what the base class checks, that an indeterminate verdict never blocks a run, and that only turn 0 is covered. The notebook's ``.py`` and ``.ipynb`` are edited together and verified to round-trip through jupytext. Also replaces a Sphinx reST role in the ``modality_validation`` module docstring with a plain literal. PyRIT renders docstrings with MyST, so those roles surface as raw text in the built docs; ``check-no-rest-roles`` enforces this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merging main brought a dependency bump that raises ty from 0.0.78 to 0.0.80, which adds the ``redundant-condition-strict`` rule. It flags the ``isinstance(value, frozenset)`` check in ``_read_modalities``: ``TargetCapabilities`` validates that field, so statically the condition is always true. The check is not redundant at runtime. A target whose capabilities were never configured -- every ``MagicMock(spec=PromptTarget)`` in the suite -- yields a mock there rather than a frozenset, and iterating it produces an empty combination set that would read as incompatible and drop the attack. The guard is what turns that case into an indeterminate verdict instead. Suppress the rule on that line with a comment explaining why the redundancy is deliberate, matching the reasoning behind the existing path-scoped override for the Alembic revisions' defensive checks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Victor Valbuena (ValbuenaVC)
changed the title
``target_accepts`` previously accepted a request whose data types were a *subset* of some
advertised input combination, with a special case borrowed from ``_ModalityFeedbackRouter`` so
that a text-only request to a target advertising no bare ``{text}`` combination was still
rejected. That rule was compensating for a test fixture that under-declared what real targets
advertise: every vision profile in ``_KNOWN_CAPABILITIES`` and the ``OpenAIChatTarget`` default
list ``{image_path}`` on its own, not just ``{text, image_path}``.
Read the declarations literally instead: the request's set of data types must be exactly one
of the advertised combinations. ``{text, image_path}`` means "text with an image", not "an
image alone". This is simpler, needs no special case for the router's edit-only rule (a bare
``{text}`` request against ``{{text, image_path}}`` is simply not a match), and is more correct
for targets that honestly omit a lone-media shape, such as video generation, where the subset
test would have accepted a bare reference image that the API rejects.
The vision test profile gains ``{image_path}`` to match production declarations. Three
acceptance tests are renamed to say what they now assert, and one is added to pin that a lone
image needs its own advertised combination. Targets whose defaults omit a lone-media
combination are left as they are; a request that needs one will surface as incompatible and
the declaration can be completed where it is genuinely missing.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Victor Valbuena (ValbuenaVC)
marked this pull request as ready for review
September 22, 2026 20:33
Victor Valbuena (ValbuenaVC)
changed the title
Copilot started reviewing on behalf of
Victor Valbuena (ValbuenaVC)
September 22, 2026 20:45
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Validation currently mishandles resumed plans, response converters, compound attacks, and indexed conversion chains.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 2
Open (2)
What changed in this PR
Adds plan-time modality compatibility checks to scenario initialization, preventing unsupported attack/target/scorer combinations from silently proceeding.
Changes:
- Adds modality projection, validation, and SKIP/WARN/RAISE policies.
- Exposes scorer-supported modalities through public APIs and wrappers.
- Expands modality fixtures, tests, and scenario documentation.
| File | Description |
|---|---|
.github/instructions/scenarios.instructions.md |
Documents scenario modality validation. |
doc/code/scenarios/0_scenarios.py |
Adds modality policy guidance. |
doc/code/scenarios/0_scenarios.ipynb |
Synchronizes notebook guidance. |
pyrit/scenario/core/__init__.py |
Exports modality APIs. |
pyrit/scenario/core/attack_technique_factory.py |
Reuses converter-chain projection. |
pyrit/scenario/core/modality_validation.py |
Implements compatibility validation. |
pyrit/scenario/core/scenario.py |
Applies modality policies during initialization. |
pyrit/scenario/scenarios/garak/figstep.py |
Configures FigStep to warn. |
pyrit/score/audio_transcript_scorer.py |
Uses public scorer modality metadata. |
pyrit/score/message_scorer.py |
Reports validator-declared modalities. |
pyrit/score/scorer.py |
Adds the public modality accessor. |
pyrit/score/scorer_prompt_validator.py |
Exposes declaration state and types. |
pyrit/score/true_false/float_scale_threshold_scorer.py |
Delegates supported modalities. |
pyrit/score/true_false/true_false_composite_scorer.py |
Intersects child modalities. |
pyrit/score/true_false/true_false_inverter_scorer.py |
Delegates wrapped modalities. |
pyrit/score/video_scorer.py |
Uses public modality metadata. |
tests/unit/executor/attack/component/test_modality_router.py |
Reuses capability-aware target mocks. |
tests/unit/mocks.py |
Adds configurable target capabilities. |
tests/unit/modality_profiles.py |
Defines shared modality profiles. |
tests/unit/scenario/core/test_modality_validation.py |
Tests modality derivation. |
tests/unit/scenario/core/test_scenario_modality_policy.py |
Tests policy enforcement. |
tests/unit/score/test_scorer_modality_accessors.py |
Tests scorer accessors and delegation. |
tests/unit/test_mock_target.py |
Tests capability-aware mock targets. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Preserve message-piece positions through converter chains and compare the final message types with target capabilities. Document the piece/message boundary and track remaining review findings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24c08221-0382-4f39-aeda-ec3baac3ff1a
Replay the persisted plan before modality checks so unsampled groups cannot invalidate a valid run. Refuse to silently skip incompatible saved groups and cover plan and legacy resume paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24c08221-0382-4f39-aeda-ec3baac3ff1a
Preserve the fork's main merge before pushing modality validation fixes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24c08221-0382-4f39-aeda-ec3baac3ff1a
Union modalities for children that skip unsupported evidence and leave strict or undeclared child compatibility unknown. Preserve existing OR/AND aggregation and cover disjoint, strict, nested, and plan-time cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24c08221-0382-4f39-aeda-ec3baac3ff1a
Evaluate each advertised output combination independently instead of flattening every emitted type. Permissive scorers need one readable piece per response; strict scorers need every piece. Partially scorable alternatives remain unknown rather than dropping viable attacks. Cover SKIP, WARN, and RAISE across text, mixed text/audio, strict mixed, audio-only, and alternative outputs. Runtime SubStringScorer and further response-path tests follow in the companion commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24c08221-0382-4f39-aeda-ec3baac3ff1a
Expose response converters from attacks and project advertised output combinations through their declared conversion chains before scorer compatibility. Return UNKNOWN when indexed response conversion or a converter path cannot be modeled without the actual response pieces. Do not infer a text first turn from SequentialAttack's excluded next_message parameter: its real children own the seeds, targets, and converters. Cover actual offline audio-to-text conversion and scoring, plus successful execution of a media-seeded sequential child that the old check dropped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24c08221-0382-4f39-aeda-ec3baac3ff1a
Record the C1 through R4 fixes, their regression coverage, and the explicit UNKNOWN tradeoffs for mixed outputs, indexed response conversion, and delegated compound attacks. Preserve the limitation that an isolated main runtime matrix could not be executed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24c08221-0382-4f39-aeda-ec3baac3ff1a
A text-capable TAP target seeds root zero but generates text for its other first-turn roots. Project both paths through request converters and mark mixed runnable/unrunnable roots UNKNOWN so scenario SKIP does not discard the runnable branches. Preserve rejection for width-one TAP and media-required targets. Add focused regression coverage and update the scenario guide and reviewer tracker. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24c08221-0382-4f39-aeda-ec3baac3ff1a
Preserve recent main changes before publishing the TAP modality fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24c08221-0382-4f39-aeda-ec3baac3ff1a
Comment on lines
+301
to
+305
| start_types = _effective_start_types( | ||
| seed_group=seed_group, | ||
| seed_technique=technique.seed_technique, | ||
| reads_next_message=reads_next_message, | ||
| ) |
Contributor
There was a problem hiding this comment.
🔴 Must fix: account for execution overrides before projecting the request.
AtomicAttack accepts next_message through its execution kwargs. run_async forwards it to the executor, and AttackParameters.from_seed_group_async applies it after extracting the seed's message. This projection only sees the seed.
For example, an image seed plus next_message=Message.from_prompt(prompt="actual text request", role="user") runs successfully against a text-only target: the target receives only the override text. This check instead projects the unused image, marks the attack incompatible, and the default policy drops it. I reproduced that with a real AtomicAttack.run_async and an offline target.
Please derive modalities from the effective execution inputs, including constructor-supplied overrides. If those inputs cannot be determined safely, return UNKNOWN rather than rejecting based on a message that will not be sent. Add coverage that compares validation with the actual overridden request.
Contributor
Author
There was a problem hiding this comment.
Fixed in 36648c6. The default behavior is now to run the check after considering overrides (where applicable), and I added regression tests for this behavior. Note that I did this by adding get_next_message_override to AtomicAttack so the modality validation has access to the next message.
Comment on lines
+247
to
+250
| if scorer.skips_unsupported_data_types: | ||
| scorable = [bool(combination & declared) for combination in projected_modalities] | ||
| else: | ||
| scorable = [bool(combination) and combination <= declared for combination in projected_modalities] |
Contributor
There was a problem hiding this comment.
🔴 Must fix: distinguish filtering pieces from rejecting an empty score.
The default mixed-response case is fixed, but this valid configuration still gets rejected:
ScorerPromptValidator(
supported_data_types=["text"],
enforce_all_pieces_valid=False,
raise_on_no_valid_pieces=True,
)A SubStringScorer using this validator successfully scores the text in a text/audio response. raise_on_no_valid_pieces only raises when there are zero readable pieces; it does not require every piece to be readable. However, it makes skips_unsupported_data_types false, so this branch requires both types and the default SKIP policy drops a working attack. The same happens through TrueFalseInverterScorer.
Please distinguish per-piece strictness from the composite scorer's ability to decline an entirely unsupported response, and cover this flag combination in the compatibility tests.
Comment on lines
+164
to
+166
| output_types: set[PromptDataType] = set() | ||
| for types in piece_types: | ||
| output_types.update(types) |
Contributor
There was a problem hiding this comment.
🔴 Must fix: keep alternative converter outputs separate.
A converter returns one ConverterResult with one output_type. Its SUPPORTED_OUTPUT_TYPES lists possible outputs, not pieces that will all appear together. This union loses that distinction.
For example, a custom converter declaring ("text", "image_path") can turn one text piece into either one text piece or one image piece. A target advertising {text} and {image_path} separately accepts both results. I ran both paths successfully, but validation invents a two-type {text, image_path} request and the default SKIP policy rejects the attack.
Please preserve the possible message combinations through projection, or return UNKNOWN when one set cannot represent the alternatives safely. Add a regression with multiple declared output types so a valid converter extension does not require changes to scenario validation.
Keep the review working notes out of the feature branch; only the tracked REVIEWER_COMMENTS.md file is removed. No product code or other files change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24c08221-0382-4f39-aeda-ec3baac3ff1a
Expose whether AtomicAttack supplies a constructor-level next_message override without changing execution behavior. Project the override's ordered converted piece types through request converters instead of a seed message that will never be sent; distinguish an explicit None from an absent override and keep unmodelable inputs indeterminate. Cover every declared prompt data type, an actual text-only target receiving text despite an image seed, inverse incompatibility, explicit None, converter projection, and unchanged no-override behavior. Update scenario guidance and mock atomic attacks for the new read-only contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24c08221-0382-4f39-aeda-ec3baac3ff1a
Separate acceptance of unsupported pieces alongside readable evidence from the ability to return no score for wholly unreadable evidence. Plan-time validation now accepts mixed text/audio for a text scorer with raise_on_no_valid_pieces=True, without treating that scorer as safely skippable inside a composite. Forward the new read-only capability through scorer wrappers and add regressions for direct scoring, inversion, strict validators, composite applicability, and scenario modality policy. Runtime scoring is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24c08221-0382-4f39-aeda-ec3baac3ff1a
Preserve upstream seed-expectation and per-child condition routing. Restore leaf and one-to-one wrapper modality metadata, while treating composite compatibility as UNKNOWN until its post-refactor applicability contract can be reviewed. Adjust modality tests and contributor guidance to document the loss of early composite rejection without changing runtime scoring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24c08221-0382-4f39-aeda-ec3baac3ff1a
Project each piece's possible output types through the ordered converter chain without treating alternative results as pieces of the same message. Compare each possible final message to the target's exact input combinations; accept all-compatible outcomes, reject only wholly incompatible outcomes, and keep partial or excessive branching indeterminate. Use the same alternatives for response-to-scorer projection while retaining the conservative union for factory append checks. Cover runtime output selection, multi-piece indexes, converter-chain failures, response conversion, and bounded branching; document the 256-combination limit and the scorer-refactor UNKNOWN tradeoff. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24c08221-0382-4f39-aeda-ec3baac3ff1a
| root_reasons.append(failure_reason) | ||
| for projected in combinations: | ||
| projected_all.update(projected) | ||
| request_verdict = target_accepts(target=target, request_types=set(projected)) |
Contributor
There was a problem hiding this comment.
🔴 Must fix: check the target-facing modalities after normalization.
This compares request-converter output with the target's native modalities before its normalizers run. PromptTarget.send_prompt_async normalizes first, then validates.
For example, take a seed with a system instruction followed by an image-only user message, and a target accepting {text, image_path} with SYSTEM_PROMPT=ADAPT. The built-in GenericSystemSquashNormalizer inserts the instruction as a text piece, so the target receives a valid text+image request. I reproduced a successful AtomicAttack.run_async with that exact normalized shape, but this check sees only the image and the default SKIP policy drops the attack.
Please account for target-side normalization, or return UNKNOWN when its effect on the request cannot be established safely. Add a regression covering system-prompt adaptation with a media-only seed.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Introduces modality validation for scenario runs in
initialize_async, ensuring thatAtomicAttacks with invalid modality conversions (e.g. text to image for a target that doesn't accept image inputs) are noticed before anything is sent. Also adds a policy for what to do when a modality compatibility failure is detected.Today these mismatches surface only once a run is under way, e.g. as a scorer rejecting a data type it cannot read, or a multi-turn attack failing to build its first request.
Scenario.initialize_asyncnow validates everyAtomicAttackthe moment it is built before anything else calls it. This also applies to scenario resumption.The PR introduces two main ideas:
Validation:
next_messagethe attack will send, after merging any technique seed group) are projected through the request converters and checked against the objective target's advertisedinput_modalities.Policy
Scenario.MODALITY_POLICYsits alongsideBASELINE_ATTACK_POLICYand reusesScorerOverridePolicy's vocabulary. The policy has three values:SKIP(default)WARNRAISEinitialize_asyncwithModalityValidationError(aValueErrorsubclass).SKIPwarns rather than staying silent because dropping a whole atomic attack changes what arun covers.
Additional caveats:
An unknown modality never blocks a run.
A target that does not declare capabilities, an attack exposing no
scoring config, or a scorer that never declared its data types all resolve to
UNKNOWNand arekept.
Scorer modality is now public.
Scorers already declared their data
types via
ScorerPromptValidator, but only through the privatescorer._validator._supported_data_types, and only enforced after a run. This PR addsScorer.supported_data_types(wrappers delegate; the composite intersects its children) andScorerPromptValidator.has_declared_data_types, and replaces the two existing private reach-insin
AudioTranscriptHelperandVideoHelper. The previously permissive runtime behavior for scoringmodality compatibilty is unchanged.
Scorers that wrap other scorers change their behavior.
Wrapper scorers
(
TrueFalseInverterScorer,TrueFalseCompositeScorer,FloatScaleThresholdScorer) are nowaccepted by
AudioTranscriptHelper/VideoHelperwhen their delegate (supported_data_types)declares the required type, where previously they raised
AttributeErrorregardless. Wrappers whose delegate does notdeclare it are rejected with the same
ValueErrora bare scorer gets.Small change to AttackTechniqueFactory.
AttackTechniqueFactory.can_append_request_converteralready projected a converterchain for its own purposes prior to this PR. That projection logic was extracted as a new
project_request_chainmethod and sharedso it can't drift between the technique factory and the base scenario class.
Deferred for follow-ups. Per-
AtomicAttackpolicy overrides, a runtimemodality_policyparameter,tightening the runtime scorer default (this PR only changes the scenario run validation), adversarial-target chain modality validation, media scoring semantics,
and per-scenario
VERSIONbumps (scenario identity excludes atomic attacks).Credit to jbolor21, whose modality-scenarios proposal informed this PR.
Tests and Documentation
Tests. 151 new unit tests across four new files:
tests/unit/test_mock_target.py(57) —get_mock_targetnow builds realTargetCapabilitieson request; everyPromptDataTyperound-trips; invalid types are rejected.tests/unit/score/test_scorer_modality_accessors.py(36) — declared / undeclared / wrapperdelegation / composite intersection; the runtime default is pinned unchanged.
tests/unit/scenario/core/test_modality_validation.py(45) — converter-chain projection,target acceptance, scorer acceptance, and per-
AtomicAttackverdicts including technique-seedmerging and attacks that exclude
next_message.tests/unit/scenario/core/test_scenario_modality_policy.py(13) — default policy, override,each policy's behaviour, ordering against
TARGET_REQUIREMENTS, no prompt sent on failure.Shared fixtures:
tests/unit/modality_profiles.py(named target profiles) and the extendedget_mock_targetintests/unit/mocks.py.Documentation.
doc/code/scenarios/0_scenarios.pyand.ipynb— new "Modality Validation" section besidethe baseline-attack discussion.
.github/instructions/scenarios.instructions.md— new "Modality Validation" section in thescenario authoring contract.
🤖 Generated with Claude Code