FEAT: Load selected scanner datasets without blocking launch (-10_

.github/instructions/scenarios.instructions.md

@@ -199,6 +199,8 @@ Options:
   before overrides, validation, and preparation. Do not rewrite user selections in validators.
 - Declare `DATASET_OPTIONS` (or parameter-aware `get_dataset_options()`): `ANY` only for
   generic named populations, `SUBSET` for approved alternatives, `FIXED` for ingredients.
+  Inline `seeds` / `seed_groups` report `FIXED` automatically, since named sources can't
+  replace them; their limits and filters keep the declared capabilities.
   Disable limits that would truncate ingredients. Compound source edits are ambiguous;
   preserve child shaping/caps and allow only supported global edits.
 - Estimates expose `dataset_defaults`, `effective_datasets`, and `dataset_options`.

.github/workflows/frontend_tests.yml

@@ -106,6 +106,10 @@ jobs:
             config: playwright.scenario-techniques.config.ts
             shard: 1
             total: 1
+          - project: prompt-generation-desktop
+            config: playwright.prompt-generation.config.ts
+            shard: 1
+            total: 1
     outputs:
       expected-reports: ${{ strategy.job-total }}
     permissions:
@@ -151,7 +155,7 @@ jobs:
         run: npx playwright install --with-deps chromium
 
       - name: Build the isolated fixture frontend
-        if: matrix.project == 'analytics' || matrix.project == 'paired-outcomes' || matrix.project == 'scenario-hierarchy' || startsWith(matrix.project, 'scenario-datasets-') || matrix.project == 'scenario-techniques' || matrix.project == 'scan-launch'
+        if: matrix.project == 'analytics' || matrix.project == 'paired-outcomes' || matrix.project == 'scenario-hierarchy' || startsWith(matrix.project, 'scenario-datasets-') || matrix.project == 'scenario-techniques' || matrix.project == 'prompt-generation-desktop' || matrix.project == 'scan-launch'
         run: npm run build
 
       - name: Run E2E shard (no credentials needed)

THIRD_PARTY_NOTICES.txt

@@ -9,8 +9,11 @@ PyRIT includes adapted assignment-sampling algorithms, generation schemas,
 policy-scoring rules, and prompt templates from ASSERT v0.3.0, revision
 04b713fe73879208369750b3bd5ad53544f3eecc (2026-09-04). Microsoft Corporation adapted these
 portions to native PyRIT targets, generation strategies, typed seed datasets,
-and true/false conversation scorers. Scoring templates are adaptations, not
-verbatim copies of ASSERT's full judge or its XML citation protocol.
+conversation scorers, and judge presets. Scoring templates are adaptations, not
+verbatim copies of ASSERT's full judge or its XML citation protocol. The tester
+prompt keeps ASSERT's text except for its turn-counter sentence.
+PyRIT also redistributes ASSERT's behavior and scenario library presets, unchanged
+except for added attribution headers, as bundled behavior and suite presets.
 ASSERT is not a runtime dependency.
 
 The provenance inventory in third_party/assert-provenance.json identifies the

doc/code/analytics/0_attack_results.md

@@ -213,6 +213,45 @@ supported. The reported difference is descriptive and does not establish that
 the attack caused an outcome to change. No confidence interval is provided,
 because small and constant paired samples need careful uncertainty treatment.
 
+### ASSERT policy metrics
+
+`compute_assert_policy_metrics_async` reports ASSERT's headline rates for a
+`foundry.assert_policy` result: **impermissible behavior violated** and
+**permissible behavior violated**. Each counts a judged conversation once, as
+flagged when any relevant category with that permissibility was violated, and
+as not applicable when none was relevant. Relevance comes from the judge's
+per-category judgments in `category_validated` mode, and permissibility from
+the conversation's own stored taxonomy, not the category it was generated for.
+Rows are the effective execution units that `compute_scenario_statistics`
+counts. Rates are reported separately for each attack technique, by name, and
+for the baseline, and are never pooled. Each technique also reports a violation
+rate for each judge dimension, read independently of the others: not-applicable
+answers and judge replies without a verdict are counted outside its denominator,
+and graded dimensions add grade counts and the median. It also reports a violation
+rate for each taxonomy category over the conversations that judged it relevant,
+kept separate for each policy the judge used (its taxonomy, application context,
+and target instructions), even when policies share a behavior or category name, and why
+unjudged conversations have no verdict. Every rate is an `AssertBinaryRate`
+with ASSERT's 90% Wilson interval and a flag for fewer than 5 applicable
+conversations.
+
+The result must come from `foundry.assert_policy`: its saved registry name
+decides, and an older result without one must come from PyRIT's `AssertPolicy`
+class and module, not another class with the same name. A conversation's policy
+comes from the stored seed group with exactly the seeds the result recorded, and
+only when that group's taxonomy matches the one ASSERT's built-in policy judge
+recorded in its saved scorer identifier. Otherwise, for example after the dataset
+is replaced or its policy edited, or when the policy score has no such record,
+the conversation is counted as having no matching stored policy and left out of
+the stance and category rates; its dimension rates still count it. The judge
+recomputes `policy_violation` from its category judgments, so every violation
+names a violated category; unlike ASSERT, no violation is attributed to the
+category a conversation was generated for, which the stored seeds alone can't
+prove. A policy score whose verdict disagrees with its category judgments is
+rejected as malformed. A supplied `objective_scorer`'s verdicts are reported as the `objective`
+dimension, whatever its score category. See the [Foundry scenarios](../../scanner/foundry.ipynb) page for
+an example.
+
 ### Scenario producer accounting
 
 Scenario counts also expose `producer_counts`, separating target-facing,

doc/code/datasets/0_dataset.md

@@ -66,6 +66,41 @@ A `SeedDataset` is a collection of related `SeedGroups` that you want to test to
 
 Datasets can be loaded from local YAML files or fetched remotely from sources like HuggingFace, making it easy to share and version test cases across teams.
 
+## Background scanner preparation
+
+The scanner automatically prepares only selected missing sources. Discovery and estimates
+remain read-only. Stored contents win, including edited and generated datasets; filters and
+selection limits never trigger a replacement or another provider call.
+
+Loading does not disable the form or Launch. Confirming Launch creates a run immediately;
+its normal queue/execution lifecycle shows dataset preparation before any attacks start.
+The backend shares missing-only provider work across selections, tabs, and launch within
+one runtime and memory lifetime. Registered providers share by provider class and source;
+configured providers share only when they are the same live instance, not merely the same
+dataset name. Distinct configured providers are not assumed equivalent.
+Successful fetches are shared through their missing-only commit, not cached for the runtime:
+deleting a stored source permits a fresh fetch. Refreshed scanner metadata also invalidates
+the previous selection's prepared state when a source is no longer stored.
+
+Provider failures stay visible until **Retry loading datasets** is selected. They cannot
+silently restart on a parameter edit or execute an incomplete run. Cancellation stops that
+run's wait without cancelling work another caller may need. Runtime replacement refuses
+active preparation, and shutdown drains owned provider work before disposing memory.
+This sticky-failure behavior is limited to automatic GUI preparation. Repeating an explicit
+legacy API load or eager API run makes a new attempt after a transient provider failure.
+
+The scanner uses `if_missing`. Saved `never`/`always` settings are visibly normalized for
+that scan when overrides are permitted; protected policies are rejected instead. Preset
+editing never prepares data and preserves its SDK-capable loading settings on save. Python
+and ordinary API launches retain `never`, `if_missing`, and explicit `always` refresh.
+Normalization uses parameter-resolved settings, not static catalog source names.
+Source settings report whether a global loading-policy override can control them; unknown
+inheritance is guarded rather than guessed. Confirmation can wait for this read-only
+configuration projection, but never for an exact attack count or provider completion.
+Inline-only scanners without dataset inputs or named sources keep their eager launch path;
+estimate failures remain advisory. Fixed named/required sources still use preparation even
+when the scenario exposes no dataset overrides.
+
 ## Generated Datasets
 
 A generation provider uses a configured `PromptTarget` to create a dataset. Store the result

doc/code/executor/7_promptgen.ipynb

@@ -158,6 +158,53 @@
    "cell_type": "markdown",
    "id": "5",
    "metadata": {},
+   "source": [
+    "### Review the policy before generating seed groups\n",
+    "\n",
+    "`advance_async` runs the same stages and returns an `AssertGenerationCheckpoint`. Stop after the\n",
+    "dimensions stage to review the taxonomy, apply edits with `with_reviewed_policy`, then continue.\n",
+    "Each group's provenance records the edited stages. Pass `on_checkpoint` to store progress and\n",
+    "resume after an interruption. `AssertGenerator.estimate` reports the remaining model calls and an\n",
+    "approximate token range without calling a target.\n",
+    "\n",
+    "The bundled ASSERT behavior library (`list_assert_behavior_presets`) and suites\n",
+    "(`list_assert_suite_presets`) provide ready-made behavior descriptions. The GUI runs this flow\n",
+    "from **Datasets > Generate dataset**."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pyrit.datasets import get_assert_behavior_preset\n",
+    "from pyrit.models import AssertGenerationCheckpoint, AssertGenerationStage\n",
+    "\n",
+    "print(get_assert_behavior_preset(\"prompt_injection\").summary)\n",
+    "\n",
+    "draft = AssertGenerationCheckpoint(config=assert_config)\n",
+    "print(f\"Up to {AssertGenerator.estimate(checkpoint=draft).max_calls} model calls\")\n",
+    "draft = await assert_generator.advance_async(checkpoint=draft, stop_after=AssertGenerationStage.DIMENSIONS)  # type: ignore\n",
+    "for category in draft.taxonomy.behavior_categories:\n",
+    "    print(f\"{'Permissible' if category.permissible else 'Impermissible'}: {category.name}\")\n",
+    "\n",
+    "first_category, *other_categories = draft.taxonomy.behavior_categories\n",
+    "revised_category = first_category.model_copy(update={\"definition\": f\"{first_category.definition} Cite the policy.\"})\n",
+    "reviewed = draft.with_reviewed_policy(\n",
+    "    taxonomy=draft.taxonomy.model_copy(update={\"behavior_categories\": (revised_category, *other_categories)}),\n",
+    "    dimensions=draft.dimensions,\n",
+    ")\n",
+    "reviewed = await assert_generator.advance_async(checkpoint=reviewed)  # type: ignore\n",
+    "reviewed_result = assert_generator.build_result(checkpoint=reviewed)\n",
+    "print(f\"{len(reviewed_result.seed_groups)} seed groups from the reviewed policy\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7",
+   "metadata": {},
    "source": [
     "## Anecdoctor Generator\n",
     "\n",
@@ -167,7 +214,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "6",
+   "id": "8",
    "metadata": {},
    "outputs": [
     {
@@ -225,7 +272,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7",
+   "id": "9",
    "metadata": {},
    "source": [
     "Few-shot prompting produces passable attack prompts,\n",
@@ -236,7 +283,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "8",
+   "id": "10",
    "metadata": {},
    "outputs": [
     {
@@ -264,7 +311,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9",
+   "id": "11",
    "metadata": {},
    "source": [
     "The use of knowledge graph-augmented prompting can be particularly valuable when working with data across languages and cultures.\n",
@@ -278,7 +325,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "10",
+   "id": "12",
    "metadata": {},
    "outputs": [
     {
@@ -321,7 +368,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "11",
+   "id": "13",
    "metadata": {},
    "source": [
     "## GPTFuzzer Generator (optional)\n",
@@ -332,7 +379,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "12",
+   "id": "14",
    "metadata": {},
    "outputs": [
     {

doc/code/executor/7_promptgen.py

@@ -101,6 +101,41 @@
 # or score attacks. To run stored groups and judge them against their policy, use
 # the [`foundry.assert_policy` scenario](../../scanner/foundry.ipynb).
 
+# %% [markdown]
+# ### Review the policy before generating seed groups
+#
+# `advance_async` runs the same stages and returns an `AssertGenerationCheckpoint`. Stop after the
+# dimensions stage to review the taxonomy, apply edits with `with_reviewed_policy`, then continue.
+# Each group's provenance records the edited stages. Pass `on_checkpoint` to store progress and
+# resume after an interruption. `AssertGenerator.estimate` reports the remaining model calls and an
+# approximate token range without calling a target.
+#
+# The bundled ASSERT behavior library (`list_assert_behavior_presets`) and suites
+# (`list_assert_suite_presets`) provide ready-made behavior descriptions. The GUI runs this flow
+# from **Datasets > Generate dataset**.
+
+# %%
+from pyrit.datasets import get_assert_behavior_preset
+from pyrit.models import AssertGenerationCheckpoint, AssertGenerationStage
+
+print(get_assert_behavior_preset("prompt_injection").summary)
+
+draft = AssertGenerationCheckpoint(config=assert_config)
+print(f"Up to {AssertGenerator.estimate(checkpoint=draft).max_calls} model calls")
+draft = await assert_generator.advance_async(checkpoint=draft, stop_after=AssertGenerationStage.DIMENSIONS)  # type: ignore
+for category in draft.taxonomy.behavior_categories:
+    print(f"{'Permissible' if category.permissible else 'Impermissible'}: {category.name}")
+
+first_category, *other_categories = draft.taxonomy.behavior_categories
+revised_category = first_category.model_copy(update={"definition": f"{first_category.definition} Cite the policy."})
+reviewed = draft.with_reviewed_policy(
+    taxonomy=draft.taxonomy.model_copy(update={"behavior_categories": (revised_category, *other_categories)}),
+    dimensions=draft.dimensions,
+)
+reviewed = await assert_generator.advance_async(checkpoint=reviewed)  # type: ignore
+reviewed_result = assert_generator.build_result(checkpoint=reviewed)
+print(f"{len(reviewed_result.seed_groups)} seed groups from the reviewed policy")
+
 # %% [markdown]
 # ## Anecdoctor Generator
 #