[BREAKING] FEAT: ASSERT judge dimensions and presets (-1)

THIRD_PARTY_NOTICES.txt

@@ -9,8 +9,9 @@ PyRIT includes adapted assignment-sampling algorithms, generation schemas,
 policy-scoring rules, and prompt templates from ASSERT v0.3.0, revision
 04b713fe73879208369750b3bd5ad53544f3eecc (2026-09-04). Microsoft Corporation adapted these
 portions to native PyRIT targets, generation strategies, typed seed datasets,
-and true/false conversation scorers. Scoring templates are adaptations, not
-verbatim copies of ASSERT's full judge or its XML citation protocol.
+conversation scorers, and judge presets. Scoring templates are adaptations, not
+verbatim copies of ASSERT's full judge or its XML citation protocol. The tester
+prompt keeps ASSERT's text except for its turn-counter sentence.
 ASSERT is not a runtime dependency.
 
 The provenance inventory in third_party/assert-provenance.json identifies the

doc/code/scoring/1_true_false_scorers.ipynb

@@ -675,7 +675,7 @@
     "import uuid\n",
     "\n",
     "from pyrit.memory import CentralMemory\n",
-    "from pyrit.models import AssertTaxonomy, ConversationScorable\n",
+    "from pyrit.models import AssertCategoryJudgment, AssertTaxonomy, ConversationScorable\n",
     "from pyrit.score import AssertJudgmentMode, AssertScoringDimension, build_assert_policy_scorer\n",
     "\n",
     "taxonomy = AssertTaxonomy.model_validate(\n",
@@ -811,8 +811,9 @@
     "            )[0]\n",
     "            assert_scores.append(score)\n",
     "            print(f\"[ASSERT/{mode.value}/{dimension.value}] {case}: {score.get_value()}\")\n",
-    "            if \"assert_correction_reason\" in score.score_metadata:\n",
-    "                print(f\"  Correction: {score.score_metadata['assert_correction_reason']}\")"
+    "            judgment = AssertCategoryJudgment.from_metadata(metadata=score.score_metadata)\n",
+    "            if judgment is not None and judgment.correction_reason:\n",
+    "                print(f\"  Correction: {judgment.correction_reason}\")"
    ]
   },
   {
@@ -823,15 +824,23 @@
    },
    "source": [
     "Each score has its dimension category, rationale, judge metadata, and saved observations.\n",
-    "Category-mode metadata also contains `assert_node_judgments` as JSON, `assert_original_score_value`,\n",
-    "and any `assert_correction_reason` or `assert_judgment_warnings`. The raw response remains retained.\n",
+    "In category mode, `AssertCategoryJudgment.from_metadata(metadata=score.score_metadata)` returns the\n",
+    "per-category judgments, the judge's original verdict, and any correction or warnings. The raw response\n",
+    "remains retained.\n",
     "`get_assert_failure_criterion(dimension)` exposes the shared failure definition for other consumers.\n",
     "\n",
     "Category validation establishes structural consistency, not whether the model selected the right\n",
     "categories or interpreted the evidence correctly. Verdicts can still conflict with their reasoning.\n",
-    "This does not port ASSERT's XML citation resolution or multi-judge aggregation, and it does not\n",
-    "establish accuracy parity or calibration. A scenario can configure repeated judgments with\n",
-    "`TrueFalseCompositeScorer` without changing this scorer's policy or merging the two dimensions.\n"
+    "This does not port ASSERT's XML citation resolution, and it does not establish accuracy parity or\n",
+    "calibration.\n",
+    "\n",
+    "`dimension` also accepts a custom `AssertJudgeDimension`, such as one from ASSERT's judge presets\n",
+    "(`list_assert_judge_presets()` or `load_assert_judge_preset()`) or your own preset file\n",
+    "(`load_assert_judge_preset_file()`). A dimension with a graded `scale` is a violation at or past\n",
+    "its `violation_at` grade, and a not-applicable answer is undetermined. `samples` repeats the judge\n",
+    "call and combines the verdicts as ASSERT does: a majority vote with ties counted as failures, leaving\n",
+    "out undetermined samples and samples whose judgment is still invalid after retries. See\n",
+    "[Foundry scenarios](../../scanner/foundry.ipynb) for the preset file format.\n"
    ]
   },
   {

doc/code/scoring/1_true_false_scorers.py

@@ -402,7 +402,7 @@
 import uuid
 
 from pyrit.memory import CentralMemory
-from pyrit.models import AssertTaxonomy, ConversationScorable
+from pyrit.models import AssertCategoryJudgment, AssertTaxonomy, ConversationScorable
 from pyrit.score import AssertJudgmentMode, AssertScoringDimension, build_assert_policy_scorer
 
 taxonomy = AssertTaxonomy.model_validate(
@@ -474,20 +474,29 @@
             )[0]
             assert_scores.append(score)
             print(f"[ASSERT/{mode.value}/{dimension.value}] {case}: {score.get_value()}")
-            if "assert_correction_reason" in score.score_metadata:
-                print(f"  Correction: {score.score_metadata['assert_correction_reason']}")
+            judgment = AssertCategoryJudgment.from_metadata(metadata=score.score_metadata)
+            if judgment is not None and judgment.correction_reason:
+                print(f"  Correction: {judgment.correction_reason}")
 
 # %% [markdown]
 # Each score has its dimension category, rationale, judge metadata, and saved observations.
-# Category-mode metadata also contains `assert_node_judgments` as JSON, `assert_original_score_value`,
-# and any `assert_correction_reason` or `assert_judgment_warnings`. The raw response remains retained.
+# In category mode, `AssertCategoryJudgment.from_metadata(metadata=score.score_metadata)` returns the
+# per-category judgments, the judge's original verdict, and any correction or warnings. The raw response
+# remains retained.
 # `get_assert_failure_criterion(dimension)` exposes the shared failure definition for other consumers.
 #
 # Category validation establishes structural consistency, not whether the model selected the right
 # categories or interpreted the evidence correctly. Verdicts can still conflict with their reasoning.
-# This does not port ASSERT's XML citation resolution or multi-judge aggregation, and it does not
-# establish accuracy parity or calibration. A scenario can configure repeated judgments with
-# `TrueFalseCompositeScorer` without changing this scorer's policy or merging the two dimensions.
+# This does not port ASSERT's XML citation resolution, and it does not establish accuracy parity or
+# calibration.
+#
+# `dimension` also accepts a custom `AssertJudgeDimension`, such as one from ASSERT's judge presets
+# (`list_assert_judge_presets()` or `load_assert_judge_preset()`) or your own preset file
+# (`load_assert_judge_preset_file()`). A dimension with a graded `scale` is a violation at or past
+# its `violation_at` grade, and a not-applicable answer is undetermined. `samples` repeats the judge
+# call and combines the verdicts as ASSERT does: a majority vote with ties counted as failures, leaving
+# out undetermined samples and samples whose judgment is still invalid after retries. See
+# [Foundry scenarios](../../scanner/foundry.ipynb) for the preset file format.
 #
 # %% [markdown]
 # ## External classifier integrations