[Richard Lundeen (richlundeen)]

Part -1 of the ASSERT stack. Design and how to use it: ASSERT in PyRIT.

	PR
-1	#4132	Judge dimensions, presets and shared techniques	this PR
-2	#4130	Trace evidence (OpenTelemetry)
-3	#4131	Headline metrics
-4	#4133	Staged generation (SDK)
-5	#4134	Generation jobs (backend)
-6	#4135	Generate from the GUI
-7	#4144	In-process agents (docs)
-8	#4180	Scenario-aware dataset compatibility
-9	#4226	Generated-category result groups
-10	#4228	Background dataset loading

Each PR targets main and includes the layers above it in this list, so its commit list and full diff also contain earlier layers. Review this layer's own changes as a source comparison: Files changed for -1. -8 also keeps prototype commits in its ancestry, so for later layers compare source between layers rather than reading commit lists.

The "Unchanged/deferred" wording below predates the publication of -9 and -10; they are now open drafts #4226 and #4228.

Description

ASSERT judging supports custom judge dimensions, ASSERT's reusable judge presets (safety-extended, tool-use, grounding, and others) and repeated judge calls with ASSERT's vote, both in the standalone scorer and in foundry.assert_policy. foundry.assert_policy also accepts your own objective scorer, runs any registered attack technique instead of its own single_turn and multi_turn techniques, and adds an assert_package technique that uses ASSERT's own tester prompt.

• Models: AssertJudgeDimension and AssertJudgePreset (ASSERT's judge_preset YAML format). A dimension is a boolean bad-event flag by default and accepts ASSERT's allow_not_applicable and ordinal scale options. A scale also needs violation_at, a PyRIT key naming the first grade, in declared order, that counts as a violation, because PyRIT judges every dimension as a violation or not. A scale without it is rejected with a clear error.
• Presets: ASSERT's 11 judge presets ship under pyrit/datasets/score/assert/judge_presets/ (MIT, provenance recorded). list_assert_judge_presets() lists them. judge_presets accepts only these built-in names, so callers can't name other server paths. Your own preset files go through load_assert_judge_preset_file() or the Python-only judge_preset_files parameter. Dimensions merge as in ASSERT: built-ins, then presets, then preset files, with disabled_judge_dimensions removed after.
• Scoring: build_assert_policy_scorer accepts a custom AssertJudgeDimension and a samples count, and each dimension gets one judge call that returns a true/false score. A custom dimension's judge asks for one of its grades (false/true for a flag) with a new ASSERT dimension template, and FloatScaleThresholdScorer turns the grade's position into a verdict at violation_at. A not-applicable answer is undetermined. Built-in dimensions keep their existing prompts and identities. build_assert_judge_scorers in pyrit.score.assert_judge_presets binds a resolved judge configuration to one stored policy; the scenario uses it to build its default judges.
• Judgment data: AssertDimensionResult (dimension, skip reason, not-applicable answer, grade and scale), AssertCategoryJudgment (per-category AssertNodeJudgments, original verdict, correction and warnings) and AssertSampleVotes in pyrit.models are the only readers and writers of the ASSERT score metadata keys, through from_metadata() and to_metadata(). A missing key is unset and malformed data raises. The custom-dimension judge's request comes from a YAML SeedPrompt, and the ASSERT response handlers convert the decoded response through JsonSchemaResponseHandler._convert_response instead of parsing it twice.
• Repeated samples: samples above 1 wraps the judge in TrueFalseCompositeScorer with an ASSERT vote. It's a majority vote with ties counted as failures, and undetermined and not-applicable samples are left out. As in ASSERT, a sample whose judgment is still invalid after its retries is dropped, and scoring fails only when every sample does. The result is not applicable only when every sample is. In category_validated mode, per-category judgments are voted on (relevance, then violation), and policy_violation is recomputed from them, with an override warning when the recomputed verdict differs.
• Scenario scoring (foundry.assert_policy): by default each stored policy gets a policy_violation judge (the objective score) and an overrefusal judge (an auxiliary score), category-validated.
  • AssertPolicy(objective_scorer=...) takes your own TrueFalseScorer and uses it unchanged for every group, with no default judges added. It owns its own target and settings, so combining it with judge_target, a different scoring_target, or any non-default judge setting is rejected with an explicit error.
  • New SDK/API parameters judge_presets, judge_preset_files (Python only), disabled_judge_dimensions, judge_samples and objective_dimension configure the default judges, alongside the existing judgment_mode and target_system_prompt. The objective dimension decides success; every other enabled dimension is an auxiliary score.
  • The default judges use the explicit scoring_target run parameter, then judge_target, then the registered scorer target. Scenario readiness reports that judge target, or a missing one, before any data is read; a supplied scorer needs none.
  • Non-default judge settings are recorded in the scenario identity, so a changed setting is detected on resume and saved API launches restore it. Default settings leave existing identities unchanged. A supplied scorer's own identity replaces the default judges' identity, and resuming with a different scorer is rejected. To support this, Scenario gets a small _get_identity_params hook.
• Techniques ([BREAKING]): foundry.assert_policy builds its technique class from the shared AttackTechniqueRegistry, as airt.cyber does, so any registered technique can be selected. The baseline is ASSERT's prompt case and the default assert_package technique is its scenario case (next bullet). The baseline is now on by default and sends each group's stored system and user prompts. max_turns applies to techniques whose attack takes a turn count, unless the registered technique sets its own. Those techniques need a target that natively keeps multi-turn history, because the judges read the whole conversation; other targets skip or reject them per incompatible_technique_policy. Results are grouped by technique.
  • Migration: AssertTechnique is removed. Replace single_turn with the baseline (include_baseline, on by default) and multi_turn with assert_package (the default) or red_teaming. AssertPolicy.VERSION is now 2, so runs started with version 1 can't be resumed; start a new run.
• assert_package technique: offered only by foundry.assert_policy, not registered globally. It runs PyRIT's RedTeamingAttack, but the tester's system prompt is ASSERT's tester prompt (pyrit/datasets/executors/red_teaming/assert_tester_system.yaml, from pinned revision 04b713fe) filled in with each group's stored tester instructions and max_turns. ASSERT's opening request and <target_response> reply framing are reused. Groups without tester instructions are skipped, or rejected under incompatible_technique_policy=raise. red_teaming stays selectable and uses the group's stored PyRIT tester prompt.
  • Remaining differences, documented: a stored user prompt opens the conversation instead of the tester, the tester sees no [Turn N/max] counter, and it replies as JSON with next_message instead of plain text.
• Technique converters: foundry.assert_policy now accepts technique_converters, with none by default.
• Docs: doc/scanner/foundry has a new "Judge dimensions and presets" section with a preset example adapted from ASSERT's travel-planner budget example, a note that the GUI uses default scoring and that Python can supply its own objective scorer, and an updated "Differences from ASSERT" list. The ASSERT section of doc/code/scoring/1_true_false_scorers covers custom dimensions and samples.

Own GUI changes

These apply only to the foundry.assert_policy launch form. "Before" is this PR's base, main at c3cff1831; "after" is f696a86f7. Both captures use the same synthetic fixture (two generated ASSERT groups, an offline target, the baseline plus the default technique). The default technique is multi_turn before and assert_package after.

Behavior	Before (c3cff1831)	After (f696a86f7)
judgment_mode and target_system_prompt	Rendered as form fields and sent	Not rendered, sent or previewed. The form always uses the default judges: category-validated, with no target-owned instructions. Set these through the SDK, pyrit_scan or the API.
judge_presets, disabled_judge_dimensions, judge_samples, objective_dimension	Didn't exist	New SDK/API settings; never rendered or sent by the form
Objective scorer picker	Not shown; the scenario took no scorer	Still not shown, although the scenario now accepts an SDK scorer
Preset or saved run with custom scoring (an objective scorer, a carried scoring_target, or a non-default judge setting)	Saved judgment_mode and target_system_prompt filled their fields and were sent	Refused with "…uses a custom scorer…" or "…has custom judge settings. Use the SDK to preserve them; this form uses default scoring." No estimate is requested, and nothing is previewed or launched with defaults in its place. The saved preset is left unchanged.
Preset or saved run with default-valued judge settings	Sent as saved	Dropped; the launch proceeds with default scoring
Techniques	single_turn and multi_turn; baseline off by default	The registered technique catalog with the scenario-local assert_package recommended. Each selected technique is a separate evaluation. The baseline is on by default and sends each group's stored system and user prompts.

The superseded published draft of this PR (0012d164b) rendered the four new judge settings as generic fields. The final source never shows them, so that draft isn't a before/after for this PR. Its earlier combined screenshot, which also shows trace controls from #4130, is kept for history only and is not evidence for this source: historical combined screenshot.

ASSERT parameters. Before (c3cff1831), judgment_mode and target_system_prompt follow max_turns. After (f696a86f7), max_turns is followed directly by Adversarial Target.

Before (c3cff1831)	After (f696a86f7)

Run preview. Before (c3cff1831), the techniques are baseline and multi_turn, and the parameters include judgment_mode and target_system_prompt. After (f696a86f7), the techniques are baseline and assert_package, and the only parameters are incompatible_technique_policy and max_turns.

Before (c3cff1831)	After (f696a86f7)

Saved custom scoring (f696a86f7). A saved preset with scenario_params: {scoring_target: "registered-judge"} opens with a warning, and no estimate or launch uses default scoring in its place.

Shared/non-ASSERT impact

The changes are in shared launch-form code, but they are limited to foundry.assert_policy; other scenarios keep their objective-scorer picker, parameters and saved-setting behavior. The ordinary technique.execute launch form renders identically before and after, with the same image hash at both SHAs.

On the server, foundry.assert_policy now reports supports_objective_scorer, so the server's existing objective-scorer selection passes a registered scorer to it as objective_scorer, which is used unchanged. No shared scenario, scorer or target behavior changes for other scenarios.

Unchanged/deferred behavior

• The judge model and endpoint are unchanged: the default judges still use the server-configured scorer target. There is no judge target picker.
• The legacy judge settings stay available through the SDK, pyrit_scan and the API; judge_preset_files stays Python-only.
• No trace settings (added in -2), no dataset actions or required-dataset presentation (-6), no native cases or Spanish variant (-8), and no target-creation, operation, preference or numeric-limit changes.
• Generated-category result groups (-9) and nonblocking dataset loading (-10) are separate planned PRs, not yet opened.

[Copilot]

🟡 Changes recommended

Unresolved issues permit stale evidence reuse in sampled aggregates, discard successful votes when one judgment fails, and block the event loop during preset loading.

3 open issues

• Resolve judge presets off the event loop · New
• Preserve successful votes when a sample exhausts validation retries · New
• Track every sample's observations and invocation in aggregate reuse · New

What changed in this PR

Extends PyRIT’s ASSERT integration with custom judge dimensions and reusable presets for standalone scoring and foundry.assert_policy.

Changes:

• Adds typed boolean and ordinal dimensions, preset loading, and 11 bundled presets.
• Adds repeated-sample aggregation, ordinal normalization, and not-applicable results.
• Integrates judge settings into scenario execution and resume identity, with tests, documentation, and provenance updates.

File	Description
third_party/​assert-provenance.json	Records sources and adaptations for judge features.
THIRD_PARTY_NOTICES.txt	Updates attribution to include judge presets.
tests/​unit/​score/​test_assert_judge_presets.py	Tests preset loading, merging, and validation.
tests/​unit/​score/​test_assert_dimension_scorer.py	Tests custom dimensions and sampled aggregation.
tests/​unit/​scenario/​foundry/​test_assert_policy.py	Tests judge configuration, execution, and identity.
tests/​unit/​models/​test_assert_judge.py	Tests dimension, scale, and preset models.
tests/​unit/​backend/​test_assert_policy_resume.py	Tests saved-launch restoration and file-path rejection.
pyrit/​score/​true_false/​assert_policy_scorer.py	Extends the builder with custom dimensions and sampling.
pyrit/​score/​assert_sampled_scorer.py	Implements repeated-judge aggregation.
pyrit/​score/​assert_judge_presets.py	Loads presets and resolves enabled dimensions.
pyrit/​score/​assert_dimension_scorer.py	Implements boolean and ordinal dimension judges.
pyrit/​score/​__init__.py	Exports preset helpers and metadata constants.
pyrit/​scenario/​scenarios/​foundry/​assert_policy.py	Configures objective and auxiliary dimension judges.
pyrit/​scenario/​core/​scenario.py	Adds a scenario identity-parameter hook.
pyrit/​models/​score/​assert_judge.py	Defines validated judge configuration models.
pyrit/​models/​__init__.py	Exports the new judge models.
pyrit/​datasets/​score/​assert/​judge_presets/​tool-use.yaml	Adds tool-selection and invocation checks.
pyrit/​datasets/​score/​assert/​judge_presets/​safety-extended.yaml	Adds actionable-harm and privacy checks.
pyrit/​datasets/​score/​assert/​judge_presets/​safety-core.yaml	References the built-in safety dimensions.
pyrit/​datasets/​score/​assert/​judge_presets/​robustness.yaml	Adds injection and operational-boundary checks.
pyrit/​datasets/​score/​assert/​judge_presets/​policy-adherence.yaml	Adds policy adherence checks.
pyrit/​datasets/​score/​assert/​judge_presets/​operational.yaml	Adds workflow and escalation checks.
pyrit/​datasets/​score/​assert/​judge_presets/​multi-turn.yaml	Adds context-retention and goal-drift checks.
pyrit/​datasets/​score/​assert/​judge_presets/​instruction-following.yaml	Adds constraint and completion checks.
pyrit/​datasets/​score/​assert/​judge_presets/​grounding.yaml	Adds hallucination and attribution checks.
pyrit/​datasets/​score/​assert/​judge_presets/​communication.yaml	Adds communication-quality checks.
pyrit/​datasets/​score/​assert/​judge_presets/​alignment.yaml	Adds sycophancy and verification checks.
pyrit/​datasets/​score/​assert/​dimension_system_prompt.yaml	Adds the custom-dimension judge prompt.
doc/​scanner/​foundry.py	Documents scenario judge settings and preset examples.
doc/​scanner/​foundry.ipynb	Mirrors the Foundry documentation updates.
doc/​code/​scoring/​1_true_false_scorers.py	Documents custom dimensions and repeated samples.
doc/​code/​scoring/​1_true_false_scorers.ipynb	Mirrors the standalone scoring documentation updates.

_{Review effort: Balanced}

[Roman Lutz (romanlutz)]

Reviewed the judge dimensions/presets/sampling layer and the registry migration. The new models, preset security posture, and identity/resume handling are solid and well-tested (all 122 PR tests pass locally). Two default-behavior changes from the registry migration need discussion — both verified empirically, details inline.

[Roman Lutz (romanlutz)]

Re-review of the two new commits (graded/not-applicable dimensions, assert_package). The instruction-less-groups concern from my earlier review is addressed well: assert_package skips such groups and an all-skipped run fails loudly with guidance, with good test coverage. The graded-dimension implementation (exact-typed grade matching, threshold at violation_at, N/A as undetermined) looks correct. Two remaining items inline; the single-turn-target gap from my earlier review is still open.

[Richard Lundeen (richlundeen)]

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

[P3] local:documentor-foundry-stale-output: Still OPEN P3, nonblocking: genuine notebook execution was attempted twice in one bounded repair assignment; supplied notebook-repair record reports terminal model JSON with an invalid taxonomy enum and ScenarioPartialFailureException, not a successful output refresh. The UTF-8 retry recovered the diagnostic hidden by the first wrapper's cp1252 failure. Independently compared every code cell's persisted outputs/execution_count from reviewed ancestor to repaired head: identical. Old version/category-split displays therefore remain, although the paired markdown now documents explicit tasks and TAP incompatibility. Do not mark fixed, fabricate/strip outputs, or treat notebook pairing as execution success.