FEAT: Trace-grounded ASSERT judging with OpenTelemetry (-2)

THIRD_PARTY_NOTICES.txt

@@ -9,8 +9,9 @@ PyRIT includes adapted assignment-sampling algorithms, generation schemas,
 policy-scoring rules, and prompt templates from ASSERT v0.3.0, revision
 04b713fe73879208369750b3bd5ad53544f3eecc (2026-09-04). Microsoft Corporation adapted these
 portions to native PyRIT targets, generation strategies, typed seed datasets,
-and true/false conversation scorers. Scoring templates are adaptations, not
-verbatim copies of ASSERT's full judge or its XML citation protocol.
+conversation scorers, and judge presets. Scoring templates are adaptations, not
+verbatim copies of ASSERT's full judge or its XML citation protocol. The tester
+prompt keeps ASSERT's text except for its turn-counter sentence.
 ASSERT is not a runtime dependency.
 
 The provenance inventory in third_party/assert-provenance.json identifies the

doc/code/framework.md

@@ -306,6 +306,11 @@ If you are contributing to PyRIT, that work will most likely land in one of the
   undetermined, not false. For a `MessageScorable`, the scoring layer resolves
   outbound request trace links, regardless of chat role, through the scored response.
   Attacks pass message evidence and route expectations according to scorer support.
+- `OtelExecutionTraceSource` normalizes model, tool, and agent spans, with bounded tool
+  arguments, results, and messages, into an execution snapshot. A conversation scorer given a
+  trace source renders those steps after the turns they produced and owns citation and coverage
+  handling. A `trace_requirement` lets it decline to judge, with an undetermined result, when
+  the evidence it needs is missing.
 - Surface sources read what a location holds for a `SurfaceScorable`, such as files under one
   root directory. `FileWriteScorer` evaluates its `ContentWritten` condition against that
   evidence. The caller owns workspace isolation; a snapshot does not identify its writer.
@@ -335,7 +340,8 @@ If you are contributing to PyRIT, that work will most likely land in one of the
   When the judge's response is blocked, conversation scoring handles direct and message-triggered
   calls the same way. If it returns an undetermined score, it retains the evidence snapshot.
 - The local SDK exporter supports caller-owned, in-process capture, not a remote collector or
-  durable store.
+  durable store. `LocalTraceCapture` bundles it with a provider and completion tracking for
+  in-process targets.
 - Observation capture requires durable scored evidence. A custom general-scorer template that reads `message_piece` fields does not emit an observation for a loose `ContentScorable`.
 - `Score.scored_expectation` records the complete expectation used for the verdict. `Score.objective` is its read-only compatibility view.
 - Scorer trees check that all conditions have a matching leaf. Wrappers route supported subsets

doc/code/scoring/0_scoring.ipynb

@@ -288,7 +288,8 @@
     "Media observation capture remains deferred until its evidence can be snapshotted.\n",
     "Trace-backed tool observations are covered in [Tool-call scoring](5_tool_call_scorer.ipynb), and file-system\n",
     "observations in [File-write scoring](6_file_write_scorer.ipynb), and sandboxed program runs in\n",
-    "[Program-execution scoring](7_program_execution_scorer.ipynb).\n",
+    "[Program-execution scoring](7_program_execution_scorer.ipynb). Execution traces with model calls,\n",
+    "tool arguments, and results are covered in [Judging with execution traces](8_trace_evidence_scoring.ipynb).\n",
     "\n",
     "Replaying a judgment is different from evaluating a stored run against a new expectation.\n",
     "A retained target judgment answers the original expectation; changing that expectation\n",

doc/code/scoring/0_scoring.py

@@ -175,7 +175,8 @@
 # Media observation capture remains deferred until its evidence can be snapshotted.
 # Trace-backed tool observations are covered in [Tool-call scoring](5_tool_call_scorer.ipynb), and file-system
 # observations in [File-write scoring](6_file_write_scorer.ipynb), and sandboxed program runs in
-# [Program-execution scoring](7_program_execution_scorer.ipynb).
+# [Program-execution scoring](7_program_execution_scorer.ipynb). Execution traces with model calls,
+# tool arguments, and results are covered in [Judging with execution traces](8_trace_evidence_scoring.ipynb).
 #
 # Replaying a judgment is different from evaluating a stored run against a new expectation.
 # A retained target judgment answers the original expectation; changing that expectation

doc/code/scoring/1_true_false_scorers.ipynb

@@ -675,7 +675,7 @@
     "import uuid\n",
     "\n",
     "from pyrit.memory import CentralMemory\n",
-    "from pyrit.models import AssertTaxonomy, ConversationScorable\n",
+    "from pyrit.models import AssertCategoryJudgment, AssertTaxonomy, ConversationScorable\n",
     "from pyrit.score import AssertJudgmentMode, AssertScoringDimension, build_assert_policy_scorer\n",
     "\n",
     "taxonomy = AssertTaxonomy.model_validate(\n",
@@ -811,8 +811,9 @@
     "            )[0]\n",
     "            assert_scores.append(score)\n",
     "            print(f\"[ASSERT/{mode.value}/{dimension.value}] {case}: {score.get_value()}\")\n",
-    "            if \"assert_correction_reason\" in score.score_metadata:\n",
-    "                print(f\"  Correction: {score.score_metadata['assert_correction_reason']}\")"
+    "            judgment = AssertCategoryJudgment.from_metadata(metadata=score.score_metadata)\n",
+    "            if judgment is not None and judgment.correction_reason:\n",
+    "                print(f\"  Correction: {judgment.correction_reason}\")"
    ]
   },
   {
@@ -823,15 +824,23 @@
    },
    "source": [
     "Each score has its dimension category, rationale, judge metadata, and saved observations.\n",
-    "Category-mode metadata also contains `assert_node_judgments` as JSON, `assert_original_score_value`,\n",
-    "and any `assert_correction_reason` or `assert_judgment_warnings`. The raw response remains retained.\n",
+    "In category mode, `AssertCategoryJudgment.from_metadata(metadata=score.score_metadata)` returns the\n",
+    "per-category judgments, the judge's original verdict, and any correction or warnings. The raw response\n",
+    "remains retained.\n",
     "`get_assert_failure_criterion(dimension)` exposes the shared failure definition for other consumers.\n",
     "\n",
     "Category validation establishes structural consistency, not whether the model selected the right\n",
     "categories or interpreted the evidence correctly. Verdicts can still conflict with their reasoning.\n",
-    "This does not port ASSERT's XML citation resolution or multi-judge aggregation, and it does not\n",
-    "establish accuracy parity or calibration. A scenario can configure repeated judgments with\n",
-    "`TrueFalseCompositeScorer` without changing this scorer's policy or merging the two dimensions.\n"
+    "This does not port ASSERT's XML citation resolution, and it does not establish accuracy parity or\n",
+    "calibration.\n",
+    "\n",
+    "`dimension` also accepts a custom `AssertJudgeDimension`, such as one from ASSERT's judge presets\n",
+    "(`list_assert_judge_presets()` or `load_assert_judge_preset()`) or your own preset file\n",
+    "(`load_assert_judge_preset_file()`). A dimension with a graded `scale` is a violation at or past\n",
+    "its `violation_at` grade, and a not-applicable answer is undetermined. `samples` repeats the judge\n",
+    "call and combines the verdicts as ASSERT does: a majority vote with ties counted as failures, leaving\n",
+    "out undetermined samples and samples whose judgment is still invalid after retries. See\n",
+    "[Foundry scenarios](../../scanner/foundry.ipynb) for the preset file format.\n"
    ]
   },
   {

doc/code/scoring/1_true_false_scorers.py

@@ -402,7 +402,7 @@
 import uuid
 
 from pyrit.memory import CentralMemory
-from pyrit.models import AssertTaxonomy, ConversationScorable
+from pyrit.models import AssertCategoryJudgment, AssertTaxonomy, ConversationScorable
 from pyrit.score import AssertJudgmentMode, AssertScoringDimension, build_assert_policy_scorer
 
 taxonomy = AssertTaxonomy.model_validate(
@@ -474,20 +474,29 @@
             )[0]
             assert_scores.append(score)
             print(f"[ASSERT/{mode.value}/{dimension.value}] {case}: {score.get_value()}")
-            if "assert_correction_reason" in score.score_metadata:
-                print(f"  Correction: {score.score_metadata['assert_correction_reason']}")
+            judgment = AssertCategoryJudgment.from_metadata(metadata=score.score_metadata)
+            if judgment is not None and judgment.correction_reason:
+                print(f"  Correction: {judgment.correction_reason}")
 
 # %% [markdown]
 # Each score has its dimension category, rationale, judge metadata, and saved observations.
-# Category-mode metadata also contains `assert_node_judgments` as JSON, `assert_original_score_value`,
-# and any `assert_correction_reason` or `assert_judgment_warnings`. The raw response remains retained.
+# In category mode, `AssertCategoryJudgment.from_metadata(metadata=score.score_metadata)` returns the
+# per-category judgments, the judge's original verdict, and any correction or warnings. The raw response
+# remains retained.
 # `get_assert_failure_criterion(dimension)` exposes the shared failure definition for other consumers.
 #
 # Category validation establishes structural consistency, not whether the model selected the right
 # categories or interpreted the evidence correctly. Verdicts can still conflict with their reasoning.
-# This does not port ASSERT's XML citation resolution or multi-judge aggregation, and it does not
-# establish accuracy parity or calibration. A scenario can configure repeated judgments with
-# `TrueFalseCompositeScorer` without changing this scorer's policy or merging the two dimensions.
+# This does not port ASSERT's XML citation resolution, and it does not establish accuracy parity or
+# calibration.
+#
+# `dimension` also accepts a custom `AssertJudgeDimension`, such as one from ASSERT's judge presets
+# (`list_assert_judge_presets()` or `load_assert_judge_preset()`) or your own preset file
+# (`load_assert_judge_preset_file()`). A dimension with a graded `scale` is a violation at or past
+# its `violation_at` grade, and a not-applicable answer is undetermined. `samples` repeats the judge
+# call and combines the verdicts as ASSERT does: a majority vote with ties counted as failures, leaving
+# out undetermined samples and samples whose judgment is still invalid after retries. See
+# [Foundry scenarios](../../scanner/foundry.ipynb) for the preset file format.
 #
 # %% [markdown]
 # ## External classifier integrations