Richard Lundeen (richlundeen) · Nov 10, 2026 · Nov 10, 2026 · Nov 10, 2026 · Nov 10, 2026 · Nov 10, 2026
diff --git a/THIRD_PARTY_NOTICES.txt b/THIRD_PARTY_NOTICES.txt
@@ -9,8 +9,11 @@ PyRIT includes adapted assignment-sampling algorithms, generation schemas,
 policy-scoring rules, and prompt templates from ASSERT v0.3.0, revision
 04b713fe73879208369750b3bd5ad53544f3eecc (2026-09-04). Microsoft Corporation adapted these
 portions to native PyRIT targets, generation strategies, typed seed datasets,
-and true/false conversation scorers. Scoring templates are adaptations, not
-verbatim copies of ASSERT's full judge or its XML citation protocol.
+conversation scorers, and judge presets. Scoring templates are adaptations, not
+verbatim copies of ASSERT's full judge or its XML citation protocol. The tester
+prompt keeps ASSERT's text except for its turn-counter sentence.
+PyRIT also redistributes ASSERT's behavior and scenario library presets, unchanged
+except for added attribution headers, as bundled behavior and suite presets.
 ASSERT is not a runtime dependency.
 
 The provenance inventory in third_party/assert-provenance.json identifies the

diff --git a/doc/code/analytics/0_attack_results.md b/doc/code/analytics/0_attack_results.md
@@ -213,6 +213,45 @@ supported. The reported difference is descriptive and does not establish that
 the attack caused an outcome to change. No confidence interval is provided,
 because small and constant paired samples need careful uncertainty treatment.
 
+### ASSERT policy metrics
+
+`compute_assert_policy_metrics_async` reports ASSERT's headline rates for a
+`foundry.assert_policy` result: **impermissible behavior violated** and
+**permissible behavior violated**. Each counts a judged conversation once, as
+flagged when any relevant category with that permissibility was violated, and
+as not applicable when none was relevant. Relevance comes from the judge's
+per-category judgments in `category_validated` mode, and permissibility from
+the conversation's own stored taxonomy, not the category it was generated for.
+Rows are the effective execution units that `compute_scenario_statistics`
+counts. Rates are reported separately for each attack technique, by name, and
+for the baseline, and are never pooled. Each technique also reports a violation
+rate for each judge dimension, read independently of the others: not-applicable
+answers and judge replies without a verdict are counted outside its denominator,
+and graded dimensions add grade counts and the median. It also reports a violation
+rate for each taxonomy category over the conversations that judged it relevant,
+kept separate for each policy the judge used (its taxonomy, application context,
+and target instructions), even when policies share a behavior or category name, and why
+unjudged conversations have no verdict. Every rate is an `AssertBinaryRate`
+with ASSERT's 90% Wilson interval and a flag for fewer than 5 applicable
+conversations.
+
+The result must come from `foundry.assert_policy`: its saved registry name
+decides, and an older result without one must come from PyRIT's `AssertPolicy`
+class and module, not another class with the same name. A conversation's policy
+comes from the stored seed group with exactly the seeds the result recorded, and
+only when that group's taxonomy matches the one ASSERT's built-in policy judge
+recorded in its saved scorer identifier. Otherwise, for example after the dataset
+is replaced or its policy edited, or when the policy score has no such record,
+the conversation is counted as having no matching stored policy and left out of
+the stance and category rates; its dimension rates still count it. The judge
+recomputes `policy_violation` from its category judgments, so every violation
+names a violated category; unlike ASSERT, no violation is attributed to the
+category a conversation was generated for, which the stored seeds alone can't
+prove. A policy score whose verdict disagrees with its category judgments is
+rejected as malformed. A supplied `objective_scorer`'s verdicts are reported as the `objective`
+dimension, whatever its score category. See the [Foundry scenarios](../../scanner/foundry.ipynb) page for
+an example.
+
 ### Scenario producer accounting
 
 Scenario counts also expose `producer_counts`, separating target-facing,

diff --git a/doc/code/executor/7_promptgen.ipynb b/doc/code/executor/7_promptgen.ipynb
@@ -158,6 +158,52 @@
    "cell_type": "markdown",
    "id": "5",
    "metadata": {},
+   "source": [
+    "### Review the policy before generating seed groups\n",
+    "\n",
+    "`advance_async` runs the same stages and returns an `AssertGenerationCheckpoint`. Stop after the\n",
+    "dimensions stage to review the taxonomy, apply edits with `with_reviewed_policy`, then continue.\n",
+    "Each group's provenance records the edited stages. Pass `on_checkpoint` to store progress and\n",
+    "resume after an interruption. `AssertGenerator.estimate` reports the remaining model calls and an\n",
+    "approximate token range without calling a target.\n",
+    "\n",
+    "The bundled ASSERT behavior library (`list_assert_behavior_presets`) and suites\n",
+    "(`list_assert_suite_presets`) provide ready-made behavior descriptions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pyrit.datasets import get_assert_behavior_preset\n",
+    "from pyrit.models import AssertGenerationCheckpoint, AssertGenerationStage\n",
+    "\n",
+    "print(get_assert_behavior_preset(\"prompt_injection\").summary)\n",
+    "\n",
+    "draft = AssertGenerationCheckpoint(config=assert_config)\n",
+    "print(f\"Up to {AssertGenerator.estimate(checkpoint=draft).max_calls} model calls\")\n",
+    "draft = await assert_generator.advance_async(checkpoint=draft, stop_after=AssertGenerationStage.DIMENSIONS)  # type: ignore\n",
+    "for category in draft.taxonomy.behavior_categories:\n",
+    "    print(f\"{'Permissible' if category.permissible else 'Impermissible'}: {category.name}\")\n",
+    "\n",
+    "first_category, *other_categories = draft.taxonomy.behavior_categories\n",
+    "revised_category = first_category.model_copy(update={\"definition\": f\"{first_category.definition} Cite the policy.\"})\n",
+    "reviewed = draft.with_reviewed_policy(\n",
+    "    taxonomy=draft.taxonomy.model_copy(update={\"behavior_categories\": (revised_category, *other_categories)}),\n",
+    "    dimensions=draft.dimensions,\n",
+    ")\n",
+    "reviewed = await assert_generator.advance_async(checkpoint=reviewed)  # type: ignore\n",
+    "reviewed_result = assert_generator.build_result(checkpoint=reviewed)\n",
+    "print(f\"{len(reviewed_result.seed_groups)} seed groups from the reviewed policy\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7",
+   "metadata": {},
    "source": [
     "## Anecdoctor Generator\n",
     "\n",
@@ -167,7 +213,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "6",
+   "id": "8",
    "metadata": {},
    "outputs": [
     {
@@ -225,7 +271,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7",
+   "id": "9",
    "metadata": {},
    "source": [
     "Few-shot prompting produces passable attack prompts,\n",
@@ -236,7 +282,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "8",
+   "id": "10",
    "metadata": {},
    "outputs": [
     {
@@ -264,7 +310,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9",
+   "id": "11",
    "metadata": {},
    "source": [
     "The use of knowledge graph-augmented prompting can be particularly valuable when working with data across languages and cultures.\n",
@@ -278,7 +324,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "10",
+   "id": "12",
    "metadata": {},
    "outputs": [
     {
@@ -321,7 +367,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "11",
+   "id": "13",
    "metadata": {},
    "source": [
     "## GPTFuzzer Generator (optional)\n",
@@ -332,7 +378,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "12",
+   "id": "14",
    "metadata": {},
    "outputs": [
     {

diff --git a/doc/code/executor/7_promptgen.py b/doc/code/executor/7_promptgen.py
@@ -101,6 +101,40 @@
 # or score attacks. To run stored groups and judge them against their policy, use
 # the [`foundry.assert_policy` scenario](../../scanner/foundry.ipynb).
 
+# %% [markdown]
+# ### Review the policy before generating seed groups
+#
+# `advance_async` runs the same stages and returns an `AssertGenerationCheckpoint`. Stop after the
+# dimensions stage to review the taxonomy, apply edits with `with_reviewed_policy`, then continue.
+# Each group's provenance records the edited stages. Pass `on_checkpoint` to store progress and
+# resume after an interruption. `AssertGenerator.estimate` reports the remaining model calls and an
+# approximate token range without calling a target.
+#
+# The bundled ASSERT behavior library (`list_assert_behavior_presets`) and suites
+# (`list_assert_suite_presets`) provide ready-made behavior descriptions.
+
+# %%
+from pyrit.datasets import get_assert_behavior_preset
+from pyrit.models import AssertGenerationCheckpoint, AssertGenerationStage
+
+print(get_assert_behavior_preset("prompt_injection").summary)
+
+draft = AssertGenerationCheckpoint(config=assert_config)
+print(f"Up to {AssertGenerator.estimate(checkpoint=draft).max_calls} model calls")
+draft = await assert_generator.advance_async(checkpoint=draft, stop_after=AssertGenerationStage.DIMENSIONS)  # type: ignore
+for category in draft.taxonomy.behavior_categories:
+    print(f"{'Permissible' if category.permissible else 'Impermissible'}: {category.name}")
+
+first_category, *other_categories = draft.taxonomy.behavior_categories
+revised_category = first_category.model_copy(update={"definition": f"{first_category.definition} Cite the policy."})
+reviewed = draft.with_reviewed_policy(
+    taxonomy=draft.taxonomy.model_copy(update={"behavior_categories": (revised_category, *other_categories)}),
+    dimensions=draft.dimensions,
+)
+reviewed = await assert_generator.advance_async(checkpoint=reviewed)  # type: ignore
+reviewed_result = assert_generator.build_result(checkpoint=reviewed)
+print(f"{len(reviewed_result.seed_groups)} seed groups from the reviewed policy")
+
 # %% [markdown]
 # ## Anecdoctor Generator
 #

diff --git a/doc/code/framework.md b/doc/code/framework.md
@@ -306,6 +306,11 @@ If you are contributing to PyRIT, that work will most likely land in one of the
   undetermined, not false. For a `MessageScorable`, the scoring layer resolves
   outbound request trace links, regardless of chat role, through the scored response.
   Attacks pass message evidence and route expectations according to scorer support.
+- `OtelExecutionTraceSource` normalizes model, tool, and agent spans, with bounded tool
+  arguments, results, and messages, into an execution snapshot. A conversation scorer given a
+  trace source renders those steps after the turns they produced and owns citation and coverage
+  handling. A `trace_requirement` lets it decline to judge, with an undetermined result, when
+  the evidence it needs is missing.
 - Surface sources read what a location holds for a `SurfaceScorable`, such as files under one
   root directory. `FileWriteScorer` evaluates its `ContentWritten` condition against that
   evidence. The caller owns workspace isolation; a snapshot does not identify its writer.
@@ -335,7 +340,8 @@ If you are contributing to PyRIT, that work will most likely land in one of the
   When the judge's response is blocked, conversation scoring handles direct and message-triggered
   calls the same way. If it returns an undetermined score, it retains the evidence snapshot.
 - The local SDK exporter supports caller-owned, in-process capture, not a remote collector or
-  durable store.
+  durable store. `LocalTraceCapture` bundles it with a provider and completion tracking for
+  in-process targets.
 - Observation capture requires durable scored evidence. A custom general-scorer template that reads `message_piece` fields does not emit an observation for a loose `ContentScorable`.
 - `Score.scored_expectation` records the complete expectation used for the verdict. `Score.objective` is its read-only compatibility view.
 - Scorer trees check that all conditions have a matching leaf. Wrappers route supported subsets
@@ -382,6 +388,7 @@ The below talks about responsibilities of most modules in the PyRIT library
 - Shared analytics contracts (filters, dimensions, typed values, reports, facets, result pages, and `AttackStats`) live in `pyrit.models.analytics`. They validate data without querying memory or calculating statistics. `AttackResultSelection` defines selection modes without changing existing callers.
 - `compute_outcome_statistics` owns both success-rate denominators, totals, and shares. `success_rate_decided` (the explicit alias of `success_rate`) divides by successes plus failures; `success_rate_all` divides by all outcomes. `combine_outcome_statistics` revalidates and combines disjoint counts, never averages rates. Attack and scenario analytics share the same `OutcomeStatistics` model and calculations after independently selecting their populations. Models reject contradictory supplied totals/rates; they do not select populations or replace the supplied numbers.
 - `analyze_results` and `compute_technique_stats_async` remain maintained APIs. Their `include_outcome_statistics=True` option exposes the shared statistics directly; default `AttackStats` results retain their six-field shape. Existing scenario percentage fields also retain their defaults. Only the already-deprecated sync memory wrappers and `ScenarioResult.objective_achieved_rate` are scheduled for removal in 1.4.0.
+- `compute_assert_policy_metrics_async` owns ASSERT's permissible/impermissible split, judge-dimension rates, and per-category rates for `foundry.assert_policy` results, reading stored scores and seed metadata over the same execution units. The scenario and scorers only record judgments; the GUI backend serializes the SDK result.
 - [`AttackResultAnalytics`](./analytics/0_attack_results.md) provides async saved-result reports, lightweight result pages, and facet lookups. It supplies both shared success rates, display labels and exact additional drill-down predicates, and counts every saved result ID rather than latest scenario execution units. Memory owns cohort SQL and consistent projections; the SDK owns their interpretation.
 - Analytics facades sharing a memory instance share one loop-bound, bounded report/quick-query controller. Close that shared SDK lifetime before replacing or disposing memory; cancellation does not free a running query's slot before its actual session cleanup. SDK lifecycle is independent of backend runtime integration.
 - Filter-bound cursor and label-normalization helpers live in `pyrit.common.pagination`. The backend pagination module retains compatibility exports, including History's invalid-cursor first-page fallback.

diff --git a/doc/code/scoring/0_scoring.ipynb b/doc/code/scoring/0_scoring.ipynb
@@ -288,7 +288,8 @@
     "Media observation capture remains deferred until its evidence can be snapshotted.\n",
     "Trace-backed tool observations are covered in [Tool-call scoring](5_tool_call_scorer.ipynb), and file-system\n",
     "observations in [File-write scoring](6_file_write_scorer.ipynb), and sandboxed program runs in\n",
-    "[Program-execution scoring](7_program_execution_scorer.ipynb).\n",
+    "[Program-execution scoring](7_program_execution_scorer.ipynb). Execution traces with model calls,\n",
+    "tool arguments, and results are covered in [Judging with execution traces](8_trace_evidence_scoring.ipynb).\n",
     "\n",
     "Replaying a judgment is different from evaluating a stored run against a new expectation.\n",
     "A retained target judgment answers the original expectation; changing that expectation\n",

diff --git a/doc/code/scoring/0_scoring.py b/doc/code/scoring/0_scoring.py
@@ -175,7 +175,8 @@
 # Media observation capture remains deferred until its evidence can be snapshotted.
 # Trace-backed tool observations are covered in [Tool-call scoring](5_tool_call_scorer.ipynb), and file-system
 # observations in [File-write scoring](6_file_write_scorer.ipynb), and sandboxed program runs in
-# [Program-execution scoring](7_program_execution_scorer.ipynb).
+# [Program-execution scoring](7_program_execution_scorer.ipynb). Execution traces with model calls,
+# tool arguments, and results are covered in [Judging with execution traces](8_trace_evidence_scoring.ipynb).
 #
 # Replaying a judgment is different from evaluating a stored run against a new expectation.
 # A retained target judgment answers the original expectation; changing that expectation

diff --git a/doc/code/scoring/1_true_false_scorers.ipynb b/doc/code/scoring/1_true_false_scorers.ipynb
@@ -675,7 +675,7 @@
     "import uuid\n",
     "\n",
     "from pyrit.memory import CentralMemory\n",
-    "from pyrit.models import AssertTaxonomy, ConversationScorable\n",
+    "from pyrit.models import AssertCategoryJudgment, AssertTaxonomy, ConversationScorable\n",
     "from pyrit.score import AssertJudgmentMode, AssertScoringDimension, build_assert_policy_scorer\n",
     "\n",
     "taxonomy = AssertTaxonomy.model_validate(\n",
@@ -811,8 +811,9 @@
     "            )[0]\n",
     "            assert_scores.append(score)\n",
     "            print(f\"[ASSERT/{mode.value}/{dimension.value}] {case}: {score.get_value()}\")\n",
-    "            if \"assert_correction_reason\" in score.score_metadata:\n",
-    "                print(f\"  Correction: {score.score_metadata['assert_correction_reason']}\")"
+    "            judgment = AssertCategoryJudgment.from_metadata(metadata=score.score_metadata)\n",
+    "            if judgment is not None and judgment.correction_reason:\n",
+    "                print(f\"  Correction: {judgment.correction_reason}\")"
    ]
   },
   {
@@ -823,15 +824,23 @@
    },
    "source": [
     "Each score has its dimension category, rationale, judge metadata, and saved observations.\n",
-    "Category-mode metadata also contains `assert_node_judgments` as JSON, `assert_original_score_value`,\n",
-    "and any `assert_correction_reason` or `assert_judgment_warnings`. The raw response remains retained.\n",
+    "In category mode, `AssertCategoryJudgment.from_metadata(metadata=score.score_metadata)` returns the\n",
+    "per-category judgments, the judge's original verdict, and any correction or warnings. The raw response\n",
+    "remains retained.\n",
     "`get_assert_failure_criterion(dimension)` exposes the shared failure definition for other consumers.\n",
     "\n",
     "Category validation establishes structural consistency, not whether the model selected the right\n",
     "categories or interpreted the evidence correctly. Verdicts can still conflict with their reasoning.\n",
-    "This does not port ASSERT's XML citation resolution or multi-judge aggregation, and it does not\n",
-    "establish accuracy parity or calibration. A scenario can configure repeated judgments with\n",
-    "`TrueFalseCompositeScorer` without changing this scorer's policy or merging the two dimensions.\n"
+    "This does not port ASSERT's XML citation resolution, and it does not establish accuracy parity or\n",
+    "calibration.\n",
+    "\n",
+    "`dimension` also accepts a custom `AssertJudgeDimension`, such as one from ASSERT's judge presets\n",
+    "(`list_assert_judge_presets()` or `load_assert_judge_preset()`) or your own preset file\n",
+    "(`load_assert_judge_preset_file()`). A dimension with a graded `scale` is a violation at or past\n",
+    "its `violation_at` grade, and a not-applicable answer is undetermined. `samples` repeats the judge\n",
+    "call and combines the verdicts as ASSERT does: a majority vote with ties counted as failures, leaving\n",
+    "out undetermined samples and samples whose judgment is still invalid after retries. See\n",
+    "[Foundry scenarios](../../scanner/foundry.ipynb) for the preset file format.\n"
    ]
   },
   {