FEAT: Load selected scanner datasets without blocking launch (-10_ by richlundeen · Pull Request #4228 · microsoft/PyRIT

Richard Lundeen (richlundeen) · 2026-11-12T01:58:29Z

Part -10 of a 10-layer ASSERT series. Design and how to use it: ASSERT in PyRIT.

Layer	PR	Scope
-1	#4132	Judge dimensions, presets and shared techniques
-2	#4130	Trace evidence (OpenTelemetry)
-3	#4131	Headline metrics
-4	#4133	Staged generation (SDK)
-5	#4134	Generation jobs (backend)
-6	#4135	Generate from the GUI
-7	#4144	In-process agents (docs)
-8	#4180	Scenario-aware dataset compatibility
-9	#4226	Category-first result groups
-10	#4228 (this PR)	Nonblocking background dataset loading

Each PR is separately cumulative and targets main; this is not a native stack. Review the independent layer-9-to-layer-10 source delta, not the cumulative diff to main. Layer -8 retains prototype ancestry with superseded category/loading commit titles; inspect the actual tree comparison rather than inferring inclusion from those titles.

Description

Prepare selected missing datasets in the background without blocking the scanner form or starting a second provider fetch when Launch is confirmed. Stored contents are reused, and preset editing remains read-only with respect to dataset preparation.

Replace manual Load and the loading-policy selector together with automatic selected-source loading, visible status, and explicit error/Retry.
Share provider work and missing-only publication within the runtime/memory lifetime; persist applicable scans promptly through the existing queue/run lifecycle, and finish initialization before attacks execute.
Normalize parameter-resolved settings safely, retain fixed-source protections and inline-only launch behavior, and distinguish fresh pending runs from actual resumes.

Own GUI changes

Situation	Behavior
Defaults, linked/saved selections, or added sources	Load only the selected missing sources. Do not fetch unselected catalog providers or implicitly regenerate stored datasets.
Provider loading	Keep the form and Launch available. Confirmation may wait for current read-only configuration metadata, never provider completion or an exact attack count.
Launch confirmed during loading	Open a persisted run promptly. Active initialization shows Preparing datasets and joins the same provider work; queued scans remain Queued until scheduled.
Fixed named/required sources without dataset overrides	Still preload and share work with Launch. Fixed policies remain protected.
Inline-only scanner with no dataset input or named sources	Keep eager launch. Estimate failures remain advisory; do not enter a dataset-preparation lifecycle.
Failure / Retry / cancellation	Retain real error details. Automatic provider failures require explicit Retry; unrelated edits do not retry. Cancellation does not prescribe a nonexistent Retry or stop work other callers need.
Saved `never` / `always` settings	Explain missing-only normalization for this scan when permitted. Reject protected or unknown source-policy provenance instead of guessing. Do not rewrite the saved configuration.
Preset editor	Do not preload providers. Preserve SDK-capable loading settings when opening, editing, or saving.
All available count	Show a neutral deferred count with server notes; retain distinct estimate/network errors.

Shared/non-ASSERT impact

This affects all applicable scanners, not only ASSERT. Ordinary named-source scanners receive the same automatic loading and Launch handoff; inline-only technique.execute keeps its non-dataset path, while fixed named-source scanners such as garak.exploitation are not mistaken for inline-only scans. Existing authentication, runtime admission, Operation attribution, target/technique validation, numeric limits, Reset/default-source choices, source restrictions, final-source guards, and filters remain in force.

Unchanged/deferred behavior

Python and ordinary eager API clients retain never, if_missing, and explicit always refresh, including retries on repeated explicit requests. Discovery GETs and estimates remain read-only. Runtime replacement accounts for active preparation and shutdown drains owned work. Completed provider results are not cached beyond publication; deleting a stored source permits a fresh fetch. No new persisted job framework, run-state enum, database migration, distributed scheduler, target/judge picker, generation interface, or model/endpoint change is introduced.

ASSERT scorer/native/Spanish behavior, generation actions, compatible-dataset rules, category-first groups and metrics belong to earlier layers. This integration preserves their behavior, including legacy no-plan/empty-map labels, adaptive saved choices and benchmark cache reuse; a newly persisted pending run is not treated as a resume. Earlier saved plans are not rewritten. A failed initialization without a plan must be reopened as a new configuration, not resumed as an existing plan.

Exact source: 5d3efa085e165b6073d06df531f2ecc0f568917b, immediate first parent 9190c54d87b1ffd3479c6470ba938f6cd01bf2c6; exact layer -9 review predecessor is 53f7b3b52fe0f5ff394c48e556cbf31b2810e838. No moving-main merge or dependency/lockfile changes were included.

Exact-source demonstration

Source 5d3efa085 on L9 53f7b3b, desktop 1440 × 1050 and narrow 390 × 844. These are the real bundled GUI/backend routes with disposable synthetic datasets, a bounded provider gate and a local target—not user-model scans. Target pickers, numeric controls and other inherited controls are not new in this layer. No exact L9-before screenshot is included.

Before confirming Launch, the form remains usable while the provider is held (one provider invocation, zero sends):

After confirmation, the persisted run is open as Preparing datasets while that same provider is still held (one invocation, zero sends). Releasing it completes the original run without another fetch:

Confirmation, fixed sources, explicit Retry, inline bypass and preset non-fetching

Confirmation remains enabled before releasing the provider:

Fixed sources without editable dataset inputs also join one held provider call:

A real synthetic provider failure shows explicit Retry. Unrelated edits do not add attempts; clicking Retry makes exactly one additional call:

Inline-only execution stays launchable after a failed estimate and does not request dataset preparation:

Preset opening/saving performs zero provider calls and preserves its SDK-capable loading policy:

Adds ASSERT judge dimension/preset models, ships ASSERT's 11 judge presets, judges custom boolean and ordinal dimensions (with not-applicable) through build_assert_policy_scorer, aggregates repeated judge samples as ASSERT does, and exposes judge_presets, disabled_judge_dimensions, judge_samples, and objective_dimension on foundry.assert_policy. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ge presets to built-in names - objective_dimension and other non-default judge settings are recorded in scenario identity params, so resume detects changes and saved GUI/API launches restore them. Default identities are unchanged. - Mixed boolean/ordinal judge sets are ordered by policy for a stable identity. - judge_presets accepts only shipped preset names; preset files move to the Python-only judge_preset_files (list[Path]) parameter, which the API guard rejects. Preset load errors no longer echo file contents. - Adds a Scenario._get_identity_params hook. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…converters - Custom judge dimensions are boolean bad-event flags judged by SelfAskTrueFalseScorer with an ASSERT dimension template; ASSERT's ordinal scale and allow_not_applicable options are rejected. - Repeated samples use TrueFalseCompositeScorer over the repeated judge with an ASSERT vote (ties count as failures, undetermined samples excluded, category re-vote with override warning), replacing the bespoke sampled scorers. - Public exports trimmed to AssertJudgeDimension, AssertJudgePreset, AssertSampleMetadata, and the preset list/load functions. - foundry.assert_policy accepts technique_converters (none by default) for both techniques. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- AssertTechnique is removed. The technique class is built from the AttackTechniqueRegistry, with red_teaming as the default (ASSERT's scenario case). - The baseline, now on by default, sends each group's stored system and user prompts (ASSERT's prompt case). - max_turns applies to techniques whose attack takes a turn count; results are grouped by technique. - AssertPolicy.VERSION is 2, so runs from version 1 do not resume. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- allow_not_applicable: a not-applicable answer is undetermined with assert_not_applicable metadata; repeated samples exclude it and keep the marker only when every sample is not applicable. - ASSERT's ordinal scale is accepted with a PyRIT violation_at grade. The judge's grade position is thresholded by FloatScaleThresholdScorer, so every dimension still yields a true/false verdict, and the grade is kept in assert_grade metadata. Graded dimensions can be the objective. - AssertDimensionMetadata is exported from pyrit.score. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Adds a scenario-local assert_package technique to foundry.assert_policy: PyRIT's RedTeamingAttack whose tester uses ASSERT's own tester system prompt (pinned 04b713fe) filled with each group's stored tester instructions. It is the default technique alongside the baseline; red_teaming stays selectable. Groups without tester instructions are skipped or rejected per incompatible_technique_policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Resolve judge presets in a worker thread instead of the synchronous validator. Drop a judge sample whose judgment stays invalid after retries (scoring fails only when every sample does), as ASSERT does. Require native multi-turn targets for turn-taking techniques in foundry.assert_policy, keep turn settings a registered technique sets itself, pass the adversarial chat only to techniques that use one, and warn once per even sample count. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add AssertDimensionResult, AssertCategoryJudgment (with AssertNodeJudgment) and AssertSampleVotes in pyrit.models as the only readers and writers of the ASSERT score metadata keys, keeping the stored key format. Missing keys are unset and malformed present data raises ValueError. The scenario judge wrapper now records the assert_dimension marker on judged and skipped scores. The custom-dimension judge renders its request from a YAML SeedPrompt, and both ASSERT response handlers validate and convert the decoded response through a new JsonSchemaResponseHandler._convert_response hook instead of parsing twice. AssertDimensionMetadata and AssertSampleMetadata are removed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ence ASSERT judges can read what the application did, not only what it said: tool calls with arguments and results, model calls, and agent steps captured with OpenTelemetry, rendered after the turns they produced with citable step IDs. - LocalTraceCapture: in-process provider and store that keeps target-request traces, completes them when their spans end, and evicts the oldest when full. - OtelExecutionTraceSource: OpenInference and GenAI normalization into a durable execution_trace observation with bounded, redactable content. - ConversationScorer trace_source and trace_requirement; AssertTraceEvidence for build_assert_policy_scorer; foundry.assert_policy trace_source and trace_requirement, kept in identity when non-default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Withdraw completion when a span starts after it; export and completion share the capture lock; evict any trace when full with explicit loss markers and bounded bookkeeping; render steps in parent cycles; remove the unreachable unattributed block; require a trace source for trace_requirement; document strict snapshot matching and citation behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…mplate additional_instructions is an explicit dimension_system_prompt.yaml parameter rendered in the sandbox instead of appended after rendering; the prompt is unchanged when it is empty. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Report ASSERT's permissible/impermissible split, a rate per boolean judge dimension, and judged-category violation rates per technique, with 90% Wilson intervals and ASSERT's minimum support. Expose them through a backend route and the GUI run page. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- Match results to their stored group by full seed identity, so groups sharing an objective but not their prompts resolve; keep the objective seed ID suffix only for exact duplicates and match it without parsing technique names. - Read judge dimensions independently: an undetermined objective no longer drops the policy split or other verdicts; count blocked/failed judges per dimension as no verdict, separate from not applicable. - Order tied score timestamps by score ID. - Return 409 only for unfinished runs and a sanitized 500 for unreadable saved results. - Rename BinaryRate to AssertBinaryRate and show 90% intervals in the run-page panel. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

ASSERT metrics now read the dimension, skip reason, not-applicable answer, grade and scale through AssertDimensionResult, and per-category judgments through AssertCategoryJudgment, instead of decoding the metadata keys locally. Missing judgment metadata stays unset, as on older scores, and malformed metadata now raises instead of being dropped. The scenario's judge wrapper in the layer below writes the dimension marker on judged and skipped scores, so this layer no longer edits it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…RT behavior library AssertGenerator gains a checkpointed staged API (advance_async, build_result, estimate, validate_config): stop after a stage, apply a reviewed taxonomy and dimensions (recorded in each group's provenance), and resume after interruption without repeating model calls. TargetObjectiveGenerator.estimate and the provider-free PromptGenerationEstimate report remaining calls and a token range. The ASSERT v0.3.0 behavior (52) and suite (3) presets are vendored with loaders, and StorageIO gains list_files_async. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Checkpoints carry a format version and the identifier of the target that produced each role's work. Resuming with a different target is rejected before any model call, build_result attributes work to the recorded producers, and stored seed-group assignments must match the planned batches. Estimates count the dimensions and target system prompt sent with every seed-group call. Objective datasets record the request that generated them, the objective estimate reads its prompt once, and Azure Blob listings percent-encode names so they round-trip. AssertGenerationStage is a StrEnum. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

PromptGenerationService runs generators in the background: preflight and estimates, review pauses, cancel, retry from the last checkpoint, batch creation for suites, and saving a new named dataset. One backend instance owns the records, stored as JSON documents in the memory's results storage. Adapters translate the ASSERT and target-objective generators; routes live under /api/prompt-generators and /api/prompt-generations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

A cancelled local save finishes its file replacement before the cancellation handler writes again, and the first cache load is serialized with saves and deletes. pause_for_review is rejected for generators without a review step, objective datasets can be edited and regenerated, and batch members keep their request order. The README states that several backends sharing one results storage aren't supported. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Datasets > Generate dataset estimates, runs, reviews, and retries generations, with an ASSERT form (bundled behavior and suite templates), a taxonomy review editor, and a target-objective form. Suites run as a batch, grouped in the generation list. Completed ASSERT datasets open foundry.assert_policy with repeatable ?dataset= preselection, and the Datasets catalog gains an Origins facet (for example, Generated). Includes Jest tests, an isolated Playwright journey over an offline generation target, CI wiring, and GUI docs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The launch page's dataset table hid the dataset picker until a source was selected, so a scenario with no default datasets (foundry.assert_policy) said it had no named datasets and could never select one. Selection now follows the scenario's dataset options alone: with overrides allowed and nothing selected, the table says No dataset selected and offers Add dataset. Scenarios that don't take datasets, or that fix them, are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

New generations preselect the default adversarial target for every role, including the research target, and apply it once a slow target registry loads, without replacing a user's or source's choices. Polling waits for the runtime and never shows another record's or runtime's data, a failed preflight can be checked again, generator metadata refreshes keep unsaved input, and regenerated names keep their version suffix at the maximum length. Suite dataset names follow the label, batch members list in request order, and the regenerate link retries transient failures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add an in-process agent example to the custom targets guide and replace the HTTPTarget + MockTransport workaround in the tool-call scorer guide with a local custom target. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ss agents Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Share scenario requirements and technique preparation across read-only assessment, selected-data validation, and launch. Add scanner coverage and incompatibility reasons, and recommend tester inputs for new ASSERT GUI generations while preserving SDK defaults. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Use existing JSON key iteration with pre-aggregate projections for literal scalar/null metadata keys on SQLite and SQL Server. Require named keys before schema parsing, and leave filtered/custom populations unverified until resolved validation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Use the scenario-local ASSERT package factory and tester-instruction check, preserve native multi-turn requirements and existing target diagnostics, and validate trace references before input selection. Preserve the final lower-layer picker and research-target changes during rebase. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Keep dataset option names stable with accessible compatibility descriptions. Exercise generation, filtering, first selection, and a completed default ASSERT scan using deterministic schema-aware offline targets for every role. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrates -3 d0b7c2a (typed judgments, provenance and verdict corrections) under the published -4 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrates -4 1c0add0 (rebuilt -3 d0b7c2a) under the published -5 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrates -5 22c7a75 (rebuilt -3 d0b7c2a) under the published -6 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Replaces the per-generator links on scanner launch pages with one Browse datasets / Generate datasets group in the Datasets header. Generate waits for the current runtime's generators, then opens the scenario's only follow-up generator (ASSERT for foundry.assert_policy), otherwise the generator chooser. Linked ?dataset= names stay explicit, exact selections for the server to validate instead of being checked against a second catalog fetch; only identical names are deduplicated. A blank name, or any name on a scanner that takes no datasets, blocks the estimate and launch with an error and a link that removes them. Regenerate links read the stored request by the dataset's exact selection key. foundry.assert_policy marks its datasets as required and asks for one at launch, using the rebuilt -3's ASSERT scenario check. Scenario.get_dataset_options reports inline seeds as fixed sources, so inline-only scenarios offer no dataset picker while their limits and filters stay editable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The rebuilt -3 launch-request tests build foundry.assert_policy requests without a dataset, which this layer now rejects with 'Select an ASSERT dataset.' They select one so they keep checking the default-scoring rules. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…docs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrate checked compatibility infrastructure and conservative facts onto exact refreshed layer7 5431c38. Preserve relocated scorer/default and grouped-action ownership, add final native prompt/conversation dispatch and Spanish request conversion, and retain generic manual loading, fetch policies, Reset and metadata refresh. Keep category grouping and automatic provider preparation out of this layer. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Address the full pre-commit public-import gate without changing compatibility behavior or GUI source. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integration-only publication merge. Keep the reconstructed layer8 tree exactly identical to 19f9684; original c58 publication and the preserved 3e demo history remain ancestors. Superseded category/autoload prototype hunks stay deferred to their ledger owners rather than reappearing in source. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Preserve recorded legacy labels and execution identity across hierarchy, progress, and resume. Confirm persisted ASSERT identity before category-specific presentation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Share runtime-owned missing-only provider work and commits across background preparation and launch. Persist preparing runs through the existing scheduler, preserve stored data and SDK policies, and keep preset editing read-only. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Distinguish supported conditional uncapped counts from unavailable estimates, retaining server detail notes in both paths. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Share completed fetches through their missing-only commit without caching seeds after publication. Preserve explicit API retries, invalidate GUI readiness on metadata changes, and retain unavailable-estimate validation with an explicit deferred-count signal. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Serialize missing-source admission with publication, surface orphaned write failures, and normalize parameter-resolved dataset settings without changing saved presets. Keep pending scan states and terminal messages truthful. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Scope projection and background preparation to dataset-capable or named-source scanners, preserving fixed required-source load sharing. Log real provider failures after all scoped waiters leave without duplicating active co-waiter reporting, and retain explicit retry behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Distinguish a real resume from a newly persisted pending run for ASSERT category labels, adaptive selection and benchmark reuse. Preserve compatible metadata and linked sources, and add exact-source synthetic provider hold/failure capture checkpoints for shared, fixed and inline scans. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) · 2026-11-12T01:58:54Z

Exact-source synthetic evidence for the loading slice: 5d3efa085e165b6073d06df531f2ecc0f568917b, on L9 #4226 / 53f7b3b52fe0f5ff394c48e556cbf31b2810e838. Captures use the real bundled GUI/backend and disposable local providers/target; no user model was called. Target pickers, numeric controls and other inherited controls are not new in this layer.

Form and confirmation while the selected provider remains held: one provider invocation, zero sends.

After confirmation, before release: the persisted run is open as Preparing datasets, with the same one provider invocation and zero sends.

Shared non-ASSERT cases: fixed sources, explicit Retry, inline bypass, and preset non-fetching

Fixed named sources without dataset override inputs still join the same held work:

A real provider failure shows explicit Retry. Unrelated edits do not retry; clicking Retry makes exactly one additional attempt:

Inline-only execution remains launchable after a failed estimate and does not request dataset preparation:

Opening and saving a preset invokes no provider and preserves its SDK-capable loading policies:

All desktop/mobile flow assertions passed. Provider release completed the original run with one total invocation; raw before/during/after counters, clean-source commit/tree identity, and screenshot files are retained in the session evidence bundle. No exact L9-before image is included.

Copilot

🟡 Changes recommended

A critical preparation loop repeatedly sends load and catalog requests when a stored selection is omitted from the compatibility-filtered catalog.

1 open issue

Prevent load loops for stored but incompatible sources · New

What changed in this PR

This layer extends PyRIT’s scanner launch flow with background preparation of selected missing datasets, building on #4226 while preserving stored data and distinguishing new pending runs from resumes. The supplied cumulative changes also include earlier ASSERT generation, scoring, and UI support.

Changes:

Add background-preparation controls, runtime lifecycle integration, and pending-run metadata.
Preserve loading-policy provenance and present deferred counts separately from estimate failures.
Extend regression coverage and documentation alongside inherited ASSERT functionality.

File	Description
THIRD_PARTY_NOTICES.txt	Clarifies ASSERT adaptation and redistribution attribution.
tests/unit/scenario/test_default_run_size_estimates.py	Checks deferred uncapped estimates.
tests/unit/scenario/scenarios/adaptive/test_text_adaptive.py	Covers adaptive selection for pending runs.
tests/unit/scenario/airt/test_multilingual.py	Checks pending-run language metadata.
tests/unit/scenario/airt/test_jailbreak.py	Checks pending-run template metadata.
tests/unit/backend/test_scenario_run_service.py	Updates deferred-preparation call expectations.
tests/unit/backend/test_runtime_lifecycle.py	Checks preparation blocks runtime replacement.
tests/unit/backend/test_prompt_generation_fixture.py	Validates offline generation fixtures.
tests/unit/assert_mocks.py	Adds generated-group and checkpoint helpers.
pyrit/score/response_handler.py	Adds decoded-response conversion hook.
pyrit/score/observation/__init__.py	Exports execution-trace observation helpers.
pyrit/scenario/scenarios/foundry/__init__.py	Removes the AssertTechnique export.
pyrit/scenario/scenarios/benchmark/adversarial.py	Distinguishes resumes for cache reuse.
pyrit/scenario/scenarios/adaptive/adaptive_scenario.py	Distinguishes pending runs from resumes.
pyrit/scenario/core/scenario_context.py	Carries resume state and historical labels.
pyrit/scenario/core/required_data.py	Projects loading-policy inheritance.
pyrit/scenario/core/dataset_sampling.py	Handles groups outside required coverage keys.
pyrit/scenario/core/dataset_overrides.py	Exposes source-policy inheritance.
pyrit/scenario/core/attack_technique_factory.py	Serializes seed requirements into identity.
pyrit/registry/components/scenario_registry.py	Supports deferred scenario initialization.
pyrit/prompt_target/common/target_trace_config.py	Adds conversation identifiers to spans.
pyrit/prompt_target/common/prompt_target.py	Exposes target tracing configuration.
pyrit/models/seeds/seed_dataset_summary.py	Adds stored-group structural profiles.
pyrit/models/seeds/__init__.py	Exports profile and requirement models.
pyrit/models/score/__init__.py	Exports execution-trace data types.
pyrit/models/scenario_progress.py	Adds preparation and ASSERT identity flags.
pyrit/models/scenario_dataset.py	Models loading-policy provenance.
pyrit/models/dataset_compatibility.py	Defines read-only compatibility projections.
pyrit/models/catalog/scenario.py	Adds background launch and deferred-count fields.
pyrit/executor/promptgen/__init__.py	Exports objective-generation results.
pyrit/datasets/score/true_false_question/assert_execution_evidence_instructions.yaml	Defines trace-evidence judging instructions.
pyrit/datasets/score/assert/judge_presets/tool-use.yaml	Adds tool-use judging dimensions.
pyrit/datasets/score/assert/judge_presets/safety-extended.yaml	Adds extended safety dimensions.
pyrit/datasets/score/assert/judge_presets/safety-core.yaml	References built-in safety dimensions.
pyrit/datasets/score/assert/judge_presets/robustness.yaml	Adds robustness judging dimensions.
pyrit/datasets/score/assert/judge_presets/policy-adherence.yaml	Adds policy-adherence judging.
pyrit/datasets/score/assert/judge_presets/operational.yaml	Adds operational judging dimensions.
pyrit/datasets/score/assert/judge_presets/multi-turn.yaml	Adds conversational consistency dimensions.
pyrit/datasets/score/assert/judge_presets/instruction-following.yaml	Adds instruction-following dimensions.
pyrit/datasets/score/assert/judge_presets/grounding.yaml	Adds grounding judging dimensions.
pyrit/datasets/score/assert/judge_presets/communication.yaml	Adds communication-quality judging.
pyrit/datasets/score/assert/judge_presets/alignment.yaml	Adds honesty and verification dimensions.
pyrit/datasets/score/assert/dimension_user_prompt.yaml	Templates custom-dimension requests.
pyrit/datasets/score/assert/dimension_system_prompt.yaml	Templates custom-dimension evaluation.
pyrit/datasets/assert_suites/travel_planner.yaml	Bundles travel-planner evaluation context.
pyrit/datasets/assert_suites/travel_planner_benchmark.yaml	Bundles quality-only travel context.
pyrit/datasets/assert_behaviors/unsupported_conclusion_failures.yaml	Adds unsupported-conclusion behavior preset.
pyrit/datasets/assert_behaviors/unit_conversion_failures.yaml	Adds unit-conversion behavior preset.
pyrit/datasets/assert_behaviors/underused_context_failures.yaml	Adds context-use behavior preset.
pyrit/datasets/assert_behaviors/unauthorized_action_failures.yaml	Adds authorization behavior preset.
pyrit/datasets/assert_behaviors/tool_parameter_formatting_failures.yaml	Adds tool-argument behavior preset.
pyrit/datasets/assert_behaviors/tool_output_misinterpretation_failures.yaml	Adds tool-output interpretation preset.
pyrit/datasets/assert_behaviors/tool_orchestration_errors.yaml	Adds tool-orchestration behavior preset.
pyrit/datasets/assert_behaviors/tool_call_turn_protocol_failures.yaml	Adds tool-turn protocol preset.
pyrit/datasets/assert_behaviors/tool_call_error_recovery_failures.yaml	Adds tool-error recovery preset.
pyrit/datasets/assert_behaviors/tabs_vs_spaces.yaml	Adds indentation-preference bias preset.
pyrit/datasets/assert_behaviors/sycophancy.yaml	Adds sycophancy behavior preset.
pyrit/datasets/assert_behaviors/suicide_self_harm.yaml	Adds self-harm safety evaluation preset.
pyrit/datasets/assert_behaviors/success_criteria_ambiguity_failures.yaml	Adds success-criteria behavior preset.
pyrit/datasets/assert_behaviors/stereotyping.yaml	Adds stereotyping behavior preset.
pyrit/datasets/assert_behaviors/stale_state_failures.yaml	Adds stale-state behavior preset.
pyrit/datasets/assert_behaviors/shimano_vs_sram.yaml	Adds product-preference bias preset.
pyrit/datasets/assert_behaviors/response_completeness_failures.yaml	Adds response-completeness behavior preset.
pyrit/datasets/assert_behaviors/repeated_action_loop_failures.yaml	Adds repeated-action behavior preset.
pyrit/datasets/assert_behaviors/prompt_injection.yaml	Adds injection-resistance evaluation preset.
pyrit/datasets/assert_behaviors/procedure_adherence_failures.yaml	Adds procedure-adherence behavior preset.
pyrit/datasets/assert_behaviors/premature_termination_failures.yaml	Adds premature-termination behavior preset.
pyrit/datasets/assert_behaviors/poor_retrieval_failures.yaml	Adds retrieval-quality behavior preset.
pyrit/datasets/assert_behaviors/output_internal_consistency_failures.yaml	Adds output-consistency behavior preset.
pyrit/datasets/assert_behaviors/out_of_scope_request_failures.yaml	Adds scope-boundary behavior preset.
pyrit/datasets/assert_behaviors/observation_neglect_failures.yaml	Adds observation-use behavior preset.
pyrit/datasets/assert_behaviors/ios_vs_android.yaml	Adds platform-preference bias preset.
pyrit/datasets/assert_behaviors/inter_agent_handoff_failures.yaml	Adds agent-handoff behavior preset.
pyrit/datasets/assert_behaviors/intent_misinterpretation_failures.yaml	Adds intent-interpretation behavior preset.
pyrit/datasets/assert_behaviors/insufficient_verification_failures.yaml	Adds verification behavior preset.
pyrit/datasets/assert_behaviors/ineffective_team_communication_failures.yaml	Adds team-communication behavior preset.
pyrit/datasets/assert_behaviors/incorrect_tool_selection_failures.yaml	Adds tool-selection behavior preset.
pyrit/datasets/assert_behaviors/incomplete_answer_synthesis_failures.yaml	Adds answer-synthesis behavior preset.
pyrit/datasets/assert_behaviors/identity_verification_failures.yaml	Adds identity-verification behavior preset.
pyrit/datasets/assert_behaviors/harmful_medical_advice.yaml	Adds medical-safety evaluation preset.
pyrit/datasets/assert_behaviors/grounding_attribution_errors.yaml	Adds grounding-attribution behavior preset.
pyrit/datasets/assert_behaviors/goal_drift_failures.yaml	Adds goal-drift behavior preset.
pyrit/datasets/assert_behaviors/flawed_action_plan_failures.yaml	Adds planning-quality behavior preset.
pyrit/datasets/assert_behaviors/explicit_constraint_violation_failures.yaml	Adds constraint-compliance behavior preset.
pyrit/datasets/assert_behaviors/escalation_judgment_failures.yaml	Adds escalation-judgment behavior preset.
pyrit/datasets/assert_behaviors/doxxing.yaml	Adds privacy-protection evaluation preset.
pyrit/datasets/assert_behaviors/crystal_mountain_vs_snoqualmie.yaml	Adds destination-preference bias preset.
pyrit/datasets/assert_behaviors/conversation_coherence_breakdown.yaml	Adds conversation-coherence behavior preset.
pyrit/datasets/assert_behaviors/constraint_propagation_failures.yaml	Adds cross-agent constraint preset.
pyrit/datasets/assert_behaviors/conflicting_instruction_resolution_failures.yaml	Adds instruction-conflict behavior preset.
pyrit/datasets/assert_behaviors/actionability_failures.yaml	Adds answer-actionability behavior preset.
pyrit/datasets/__init__.py	Exports ASSERT library accessors.
pyrit/common/path.py	Defines bundled ASSERT library paths.
pyrit/backend/services/service_lifecycle.py	Drains preparation and generation services.
pyrit/backend/services/runtime_lifecycle.py	Accounts for background work during replacement.
pyrit/backend/services/dataset_preparation_service.py	Owns runtime-scoped dataset preparation.
pyrit/backend/main.py	Registers prompt-generation routes.
pyrit/analytics/__init__.py	Exports ASSERT metrics helpers.
frontend/src/utils/routeParams.ts	Adds dataset-selection launch links.
frontend/src/utils/routeParams.test.ts	Tests dataset link round trips.
frontend/src/test-utils/replaceText.ts	Adds paste-based text replacement helper.
frontend/src/hooks/useScenarioRunEstimate.test.tsx	Tests configuration metadata retention.
frontend/src/hooks/usePromptGenerators.ts	Loads runtime-specific generator metadata.
frontend/src/hooks/usePromptGenerationSubmit.ts	Submits single and batch generations.
frontend/src/hooks/usePromptGeneration.ts	Polls generation details and lists.
frontend/src/hooks/useGenerationTargetSelection.ts	Manages generation target defaults.
frontend/src/hooks/useAssertLibrary.ts	Loads bundled behavior and suite metadata.
frontend/src/components/Scenarios/scenarioRunEstimateAdapter.ts	Maps deferred counts to neutral presentation.
frontend/src/components/Scenarios/ScenarioRunEstimate.tsx	Labels deferred counts “All available.”
frontend/src/components/Scenarios/ScenarioRunEstimate.test.tsx	Tests deferred-count presentation.
frontend/src/components/Scenarios/scenarioLaunchPresentation.ts	Formats deferred launch counts.
frontend/src/components/Scenarios/scenarioLaunchForm.ts	Initializes linked dataset selections.
frontend/src/components/Scenarios/ScenarioFlow.test.tsx	Adds dataset API mocks.
frontend/src/components/Scenarios/ScenarioEstimateSection.tsx	Uses neutral styling for deferred counts.
frontend/src/components/Scenarios/ScenarioDetail.styles.ts	Adds flexible section-header layout.
frontend/src/components/Scenarios/ScenarioDatasetTable.styles.ts	Removes loading-policy selector styles.
frontend/src/components/Scenarios/datasetPresentation.ts	Resolves policies using inheritance provenance.
frontend/src/components/Scenarios/DatasetActions.tsx	Adds dataset browsing and generation links.
frontend/src/components/Scenarios/DatasetActions.styles.ts	Styles dataset action controls.
frontend/src/components/Scenarios/assertPolicyRun.ts	Identifies tentative ASSERT summaries.
frontend/src/components/Scenarios/AssertPolicyMetricsPanel.styles.ts	Styles ASSERT metrics presentation.
frontend/src/components/ScenarioPresets/ScenarioPresetEditor.tsx	Scopes dataset metadata to the scenario.
frontend/src/components/ScenarioPresets/ScenarioPresetEditor.test.tsx	Adds compatible-dataset API mocks.
frontend/src/components/PromptGeneration/TargetObjectiveSummary.tsx	Displays generated objective counts.
frontend/src/components/PromptGeneration/PromptGenerationStatusBadge.tsx	Displays generation status badges.
frontend/src/components/PromptGeneration/promptGenerationRoutes.test.ts	Tests generation route encoding.
frontend/src/components/PromptGeneration/generatorUi.ts	Defines generator view contracts.
frontend/src/components/PromptGeneration/generatorRegistry.tsx	Selects generator-specific views.
frontend/src/components/PromptGeneration/GenerationSummaryFacts.tsx	Renders generation summary facts.
frontend/src/components/Datasets/DatasetSelection.tsx	Adds dataset regeneration navigation.
frontend/src/App.tsx	Registers generation pages.
frontend/README.md	Documents isolated generation browser checks.
frontend/playwright.prompt-generation.config.ts	Configures offline generation browser coverage.
frontend/playwright.config.ts	Excludes the dedicated generation suite.
frontend/package.json	Adds the generation browser-test command.
frontend/e2e/numeric-controls.spec.ts	Mocks generator discovery.
frontend/e2e/datasets.spec.ts	Mocks missing generation provenance.
doc/myst.yml	Adds trace-scoring documentation navigation.
doc/gui/0_gui.md	Documents dataset generation workflows.
doc/code/scoring/1_true_false_scorers.py	Updates ASSERT judgment examples.
doc/code/scoring/0_scoring.py	Links execution-trace scoring guidance.
doc/code/scoring/0_scoring.ipynb	Mirrors the trace-scoring documentation link.
doc/code/executor/7_promptgen.py	Demonstrates checkpoint review and continuation.
doc/code/datasets/0_dataset.md	Documents background preparation guarantees.
.github/workflows/frontend_tests.yml	Adds generation browser coverage to CI.
.github/instructions/scenarios.instructions.md	Clarifies fixed-source behavior for inline seeds.

_{Review effort: Balanced}

+      setSnapshot({ key, metadataRevision, error: null })
+      onPreparedRef.current()


Richard Lundeen (richlundeen) and others added 30 commits November 9, 2026 18:30

FIX: Order ASSERT trace steps deterministically when start times tie

053dd5c

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FIX: Version ASSERT trace rendering for cyclic steps

e2c965b

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

DOC: Refresh ASSERT metrics notebook output for assert_package

265a2e6

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

DOC: Name assert_package as the default ASSERT technique for in-proce…

aa6b6ad

…ss agents Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) and others added 16 commits November 11, 2026 14:36

Merge rebuilt ASSERT metrics (-3) into staged generation (-4)

1c0add0

Integrates -3 d0b7c2a (typed judgments, provenance and verdict corrections) under the published -4 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge rebuilt staged generation (-4) into generation jobs (-5)

22c7a75

Integrates -4 1c0add0 (rebuilt -3 d0b7c2a) under the published -5 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge rebuilt generation jobs (-5) into the GUI generation layer (-6)

a296da4

Integrates -5 22c7a75 (rebuilt -3 d0b7c2a) under the published -6 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Merge rebuilt GUI generation layer (-6) into in-process agent …

5431c38

…docs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Use public DatasetService import in compatibility tests

19f9684

Address the full pre-commit public-import gate without changing compatibility behavior or GUI source. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Preserve details in deferred dataset estimates

368f568

Distinguish supported conditional uncapped counts from unavailable estimates, retaining server detail notes in both paths. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings November 12, 2026 01:58

Copilot started reviewing on behalf of Richard Lundeen (richlundeen) November 12, 2026 01:59

View session

Copilot AI reviewed Nov 12, 2026

View reviewed changes

Comment thread frontend/src/hooks/useScenarioDatasetPreparation.ts

Comment on lines +72 to +73

setSnapshot({ key, metadataRevision, error: null })

onPreparedRef.current()

Richard Lundeen (richlundeen) changed the title ~~FEAT: Load selected scanner datasets without blocking launch~~ FEAT: Load selected scanner datasets without blocking launch (-10_ Nov 12, 2026

Navigation Menu

Repository navigation

FEAT: Load selected scanner datasets without blocking launch (-10_ - #4228

FEAT: Load selected scanner datasets without blocking launch (-10_#4228
Richard Lundeen (richlundeen) wants to merge 86 commits into
microsoft:mainfrom
richlundeen:richlundeen-background-dataset-loading

Richard Lundeen (richlundeen) commented Nov 12, 2026 •

edited

Loading

Uh oh!

Richard Lundeen (richlundeen) commented Nov 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		setSnapshot({ key, metadataRevision, error: null })
		onPreparedRef.current()

Conversation

Richard Lundeen (richlundeen) commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Richard Lundeen (richlundeen) commented Nov 12, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Richard Lundeen (richlundeen) commented Nov 12, 2026 •

edited

Loading