Repository navigation
FEAT: Group ASSERT results by generated category - #4226
Draft
Richard Lundeen (richlundeen) wants to merge 80 commits into
Draft
Richard Lundeen (richlundeen) wants to merge 80 commits into
Richard Lundeen (richlundeen) wants to merge 80 commits into
Conversation
Adds ASSERT judge dimension/preset models, ships ASSERT's 11 judge presets, judges custom boolean and ordinal dimensions (with not-applicable) through build_assert_policy_scorer, aggregates repeated judge samples as ASSERT does, and exposes judge_presets, disabled_judge_dimensions, judge_samples, and objective_dimension on foundry.assert_policy. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ge presets to built-in names - objective_dimension and other non-default judge settings are recorded in scenario identity params, so resume detects changes and saved GUI/API launches restore them. Default identities are unchanged. - Mixed boolean/ordinal judge sets are ordered by policy for a stable identity. - judge_presets accepts only shipped preset names; preset files move to the Python-only judge_preset_files (list[Path]) parameter, which the API guard rejects. Preset load errors no longer echo file contents. - Adds a Scenario._get_identity_params hook. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…converters - Custom judge dimensions are boolean bad-event flags judged by SelfAskTrueFalseScorer with an ASSERT dimension template; ASSERT's ordinal scale and allow_not_applicable options are rejected. - Repeated samples use TrueFalseCompositeScorer over the repeated judge with an ASSERT vote (ties count as failures, undetermined samples excluded, category re-vote with override warning), replacing the bespoke sampled scorers. - Public exports trimmed to AssertJudgeDimension, AssertJudgePreset, AssertSampleMetadata, and the preset list/load functions. - foundry.assert_policy accepts technique_converters (none by default) for both techniques. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- AssertTechnique is removed. The technique class is built from the AttackTechniqueRegistry, with red_teaming as the default (ASSERT's scenario case). - The baseline, now on by default, sends each group's stored system and user prompts (ASSERT's prompt case). - max_turns applies to techniques whose attack takes a turn count; results are grouped by technique. - AssertPolicy.VERSION is 2, so runs from version 1 do not resume. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- allow_not_applicable: a not-applicable answer is undetermined with assert_not_applicable metadata; repeated samples exclude it and keep the marker only when every sample is not applicable. - ASSERT's ordinal scale is accepted with a PyRIT violation_at grade. The judge's grade position is thresholded by FloatScaleThresholdScorer, so every dimension still yields a true/false verdict, and the grade is kept in assert_grade metadata. Graded dimensions can be the objective. - AssertDimensionMetadata is exported from pyrit.score. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Adds a scenario-local assert_package technique to foundry.assert_policy: PyRIT's RedTeamingAttack whose tester uses ASSERT's own tester system prompt (pinned 04b713fe) filled with each group's stored tester instructions. It is the default technique alongside the baseline; red_teaming stays selectable. Groups without tester instructions are skipped or rejected per incompatible_technique_policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve judge presets in a worker thread instead of the synchronous validator. Drop a judge sample whose judgment stays invalid after retries (scoring fails only when every sample does), as ASSERT does. Require native multi-turn targets for turn-taking techniques in foundry.assert_policy, keep turn settings a registered technique sets itself, pass the adversarial chat only to techniques that use one, and warn once per even sample count. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add AssertDimensionResult, AssertCategoryJudgment (with AssertNodeJudgment) and AssertSampleVotes in pyrit.models as the only readers and writers of the ASSERT score metadata keys, keeping the stored key format. Missing keys are unset and malformed present data raises ValueError. The scenario judge wrapper now records the assert_dimension marker on judged and skipped scores. The custom-dimension judge renders its request from a YAML SeedPrompt, and both ASSERT response handlers validate and convert the decoded response through a new JsonSchemaResponseHandler._convert_response hook instead of parsing twice. AssertDimensionMetadata and AssertSampleMetadata are removed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ence ASSERT judges can read what the application did, not only what it said: tool calls with arguments and results, model calls, and agent steps captured with OpenTelemetry, rendered after the turns they produced with citable step IDs. - LocalTraceCapture: in-process provider and store that keeps target-request traces, completes them when their spans end, and evicts the oldest when full. - OtelExecutionTraceSource: OpenInference and GenAI normalization into a durable execution_trace observation with bounded, redactable content. - ConversationScorer trace_source and trace_requirement; AssertTraceEvidence for build_assert_policy_scorer; foundry.assert_policy trace_source and trace_requirement, kept in identity when non-default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Withdraw completion when a span starts after it; export and completion share the capture lock; evict any trace when full with explicit loss markers and bounded bookkeeping; render steps in parent cycles; remove the unreachable unattributed block; require a trace source for trace_requirement; document strict snapshot matching and citation behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…mplate additional_instructions is an explicit dimension_system_prompt.yaml parameter rendered in the sandbox instead of appended after rendering; the prompt is unchanged when it is empty. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Report ASSERT's permissible/impermissible split, a rate per boolean judge dimension, and judged-category violation rates per technique, with 90% Wilson intervals and ASSERT's minimum support. Expose them through a backend route and the GUI run page. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Match results to their stored group by full seed identity, so groups sharing an objective but not their prompts resolve; keep the objective seed ID suffix only for exact duplicates and match it without parsing technique names. - Read judge dimensions independently: an undetermined objective no longer drops the policy split or other verdicts; count blocked/failed judges per dimension as no verdict, separate from not applicable. - Order tied score timestamps by score ID. - Return 409 only for unfinished runs and a sanitized 500 for unreadable saved results. - Rename BinaryRate to AssertBinaryRate and show 90% intervals in the run-page panel. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ASSERT metrics now read the dimension, skip reason, not-applicable answer, grade and scale through AssertDimensionResult, and per-category judgments through AssertCategoryJudgment, instead of decoding the metadata keys locally. Missing judgment metadata stays unset, as on older scores, and malformed metadata now raises instead of being dropped. The scenario's judge wrapper in the layer below writes the dimension marker on judged and skipped scores, so this layer no longer edits it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…RT behavior library AssertGenerator gains a checkpointed staged API (advance_async, build_result, estimate, validate_config): stop after a stage, apply a reviewed taxonomy and dimensions (recorded in each group's provenance), and resume after interruption without repeating model calls. TargetObjectiveGenerator.estimate and the provider-free PromptGenerationEstimate report remaining calls and a token range. The ASSERT v0.3.0 behavior (52) and suite (3) presets are vendored with loaders, and StorageIO gains list_files_async. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Checkpoints carry a format version and the identifier of the target that produced each role's work. Resuming with a different target is rejected before any model call, build_result attributes work to the recorded producers, and stored seed-group assignments must match the planned batches. Estimates count the dimensions and target system prompt sent with every seed-group call. Objective datasets record the request that generated them, the objective estimate reads its prompt once, and Azure Blob listings percent-encode names so they round-trip. AssertGenerationStage is a StrEnum. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
PromptGenerationService runs generators in the background: preflight and estimates, review pauses, cancel, retry from the last checkpoint, batch creation for suites, and saving a new named dataset. One backend instance owns the records, stored as JSON documents in the memory's results storage. Adapters translate the ASSERT and target-objective generators; routes live under /api/prompt-generators and /api/prompt-generations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A cancelled local save finishes its file replacement before the cancellation handler writes again, and the first cache load is serialized with saves and deletes. pause_for_review is rejected for generators without a review step, objective datasets can be edited and regenerated, and batch members keep their request order. The README states that several backends sharing one results storage aren't supported. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Datasets > Generate dataset estimates, runs, reviews, and retries generations, with an ASSERT form (bundled behavior and suite templates), a taxonomy review editor, and a target-objective form. Suites run as a batch, grouped in the generation list. Completed ASSERT datasets open foundry.assert_policy with repeatable ?dataset= preselection, and the Datasets catalog gains an Origins facet (for example, Generated). Includes Jest tests, an isolated Playwright journey over an offline generation target, CI wiring, and GUI docs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The launch page's dataset table hid the dataset picker until a source was selected, so a scenario with no default datasets (foundry.assert_policy) said it had no named datasets and could never select one. Selection now follows the scenario's dataset options alone: with overrides allowed and nothing selected, the table says No dataset selected and offers Add dataset. Scenarios that don't take datasets, or that fix them, are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
New generations preselect the default adversarial target for every role, including the research target, and apply it once a slow target registry loads, without replacing a user's or source's choices. Polling waits for the runtime and never shows another record's or runtime's data, a failed preflight can be checked again, generator metadata refreshes keep unsaved input, and regenerated names keep their version suffix at the maximum length. Suite dataset names follow the label, batch members list in request order, and the regenerate link retries transient failures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add an in-process agent example to the custom targets guide and replace the HTTPTarget + MockTransport workaround in the tool-call scorer guide with a local custom target. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ss agents Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Share scenario requirements and technique preparation across read-only assessment, selected-data validation, and launch. Add scanner coverage and incompatibility reasons, and recommend tester inputs for new ASSERT GUI generations while preserving SDK defaults. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use existing JSON key iteration with pre-aggregate projections for literal scalar/null metadata keys on SQLite and SQL Server. Require named keys before schema parsing, and leave filtered/custom populations unverified until resolved validation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use the scenario-local ASSERT package factory and tester-instruction check, preserve native multi-turn requirements and existing target diagnostics, and validate trace references before input selection. Preserve the final lower-layer picker and research-target changes during rebase. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep dataset option names stable with accessible compatibility descriptions. Exercise generation, filtering, first selection, and a completed default ASSERT scan using deterministic schema-aware offline targets for every role. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…distinct ASSERT metrics now let a saved registry name decide whether foundry.assert_policy produced a result, so another scanner's AssertPolicy class isn't mistaken for ASSERT. Older results without a registry name must come from PyRIT's AssertPolicy module. The metrics route answers other scenarios with a 400 problem of type /errors/not-assert-policy-run. The run page mounts the panel on a matching registry name, or tentatively on the class name for older runs, and hides it only for that problem type; compatibility, validation and other errors still show. A supplied objective scorer's uncategorized verdicts are reported as the objective dimension, not as ASSERT's policy judge. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep legacy NULL prompt roles as possible user inputs until hydrated validation resolves their sequence role. Preserve explicit non-user negative evidence and bounded SQLite/SQL Server query structure; compare real SQLite candidate hints with canonical ASSERT metadata and hydrated role validation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Apply shared ANY, SUBSET and FIXED source rules before the empty-requirements fast path, keeping required ingredient metadata visible. Verify real Scam, FigStep and ApiKey services against isolated stored/provider metadata without fetching providers or hydrating seed bodies. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Group bounded compatibility facts by an explicit grouped/ungrouped discriminator as well as the UUID, matching canonical seed grouping without changing compressed profile output. Compare real SQLite UUID collisions with hydrated attack groups and retain portable SQL Server projections. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…sions A result's policy now comes only from the stored seed group with exactly the seeds it recorded, and only when that group's taxonomy matches the taxonomy the policy judge recorded in its saved scorer identifier, which survives sampling and trace wrappers. A replaced or edited dataset leaves the conversation without a matching stored policy instead of reporting another group's categories. Unmarked legacy scores count as ASSERT dimensions only with that judge provenance, so a supplied scorer that uses the policy_violation category is reported as the objective dimension. Category rates are kept per policy, with a policy_id, so policies that share a behavior or category name aren't pooled, and the panel names them apart. Results without a scenario module are no longer accepted as AssertPolicy by class name alone. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Do not apply assembled-population requirements to mandatory template-only or unloaded ingredient sources. Keep strict validation for selectable stored populations and exclude malformed, missing-input and unloaded populations without fetching or assembling providers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A stored seed group's policy is used only when the policy score's saved identifier comes from the scenario's built-in judge wrapper and records the same taxonomy; a policy score without that evidence leaves the conversation without a matching stored policy, while its dimension rates still count it. Only questions under that wrapper count as judge evidence, so a supplied SelfAskTrueFalseScorer whose question carries ASSERT-shaped metadata and the policy_violation category is reported as the objective dimension. Category rates are keyed by the judged policy (taxonomy, application context and target instructions), so policies that share a taxonomy but differ in context or instructions aren't pooled. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The metrics no longer read a stored group's assigned category. ASSERT's judges recompute policy_violation from their category judgments, so every violation verdict already names a violated category; ASSERT's attribution of category-less violations to the generated category isn't needed and would depend on seed metadata that can be edited after a run. A stored policy verdict that disagrees with its category judgments is now rejected as malformed instead of being attributed, and AssertCategoryMetrics drops attributed_by_fallback. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrates -3 d0b7c2a (typed judgments, provenance and verdict corrections) under the published -4 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replaces the per-generator links on scanner launch pages with one Browse datasets / Generate datasets group in the Datasets header. Generate waits for the current runtime's generators, then opens the scenario's only follow-up generator (ASSERT for foundry.assert_policy), otherwise the generator chooser. Linked ?dataset= names stay explicit, exact selections for the server to validate instead of being checked against a second catalog fetch; only identical names are deduplicated. A blank name, or any name on a scanner that takes no datasets, blocks the estimate and launch with an error and a link that removes them. Regenerate links read the stored request by the dataset's exact selection key. foundry.assert_policy marks its datasets as required and asks for one at launch, using the rebuilt -3's ASSERT scenario check. Scenario.get_dataset_options reports inline seeds as fixed sources, so inline-only scenarios offer no dataset picker while their limits and filters stay editable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The rebuilt -3 launch-request tests build foundry.assert_policy requests without a dataset, which this layer now rejects with 'Select an ASSERT dataset.' They select one so they keep checking the default-scoring rules. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…docs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate checked compatibility infrastructure and conservative facts onto exact refreshed layer7 5431c38. Preserve relocated scorer/default and grouped-action ownership, add final native prompt/conversation dispatch and Spanish request conversion, and retain generic manual loading, fetch policies, Reset and metadata refresh. Keep category grouping and automatic provider preparation out of this layer. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Address the full pre-commit public-import gate without changing compatibility behavior or GUI source. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integration-only publication merge. Keep the reconstructed layer8 tree exactly identical to 19f9684; original c58 publication and the preserved 3e demo history remain ancestors. Superseded category/autoload prototype hunks stay deferred to their ledger owners rather than reappearing in source. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve recorded legacy labels and execution identity across hierarchy, progress, and resume. Confirm persisted ASSERT identity before category-specific presentation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The E2E mock omits the scenario-scoped dataset endpoint, causing the scanner to hit the abort-and-throw branch and fail the numeric-control tests.
1 open issue
What changed in this PR
This cumulative ASSERT PR introduces category-first result grouping, bringing baseline and selected techniques together while preserving execution identity, historical labels, and judged-category metrics. The reviewed files also include supporting changes inherited from earlier layers.
Changes:
- Add recorded display-group context and confirmed ASSERT identity to scenario progress.
- Integrate generation workflows, compatible dataset selection, trace evidence, and reusable judge presets.
- Extend fixtures, browser checks, documentation, and third-party attribution.
| File | Description |
|---|---|
THIRD_PARTY_NOTICES.txt |
Expand ASSERT attribution. |
tests/unit/scenario/core/test_scenario_dataset_overrides.py |
Cover inline-seed override restrictions. |
tests/unit/backend/test_prompt_generation_fixture.py |
Check offline generation targets. |
tests/unit/assert_mocks.py |
Add generation checkpoint fixtures. |
pyrit/score/response_handler.py |
Add decoded-response conversion hook. |
pyrit/score/observation/__init__.py |
Export execution-trace capture APIs. |
pyrit/scenario/scenarios/foundry/__init__.py |
Remove static ASSERT technique export. |
pyrit/scenario/core/scenario_context.py |
Carry historical display-group maps. |
pyrit/scenario/core/dataset_sampling.py |
Handle optional coverage keys. |
pyrit/scenario/core/attack_technique_factory.py |
Serialize seed requirements into identity. |
pyrit/prompt_target/common/target_trace_config.py |
Attach conversation IDs to traces. |
pyrit/prompt_target/common/prompt_target.py |
Expose target trace configuration. |
pyrit/models/seeds/seed_dataset_summary.py |
Add structural dataset profiles. |
pyrit/models/seeds/__init__.py |
Export dataset requirement models. |
pyrit/models/score/__init__.py |
Export execution-trace models. |
pyrit/models/scenario_progress.py |
Add confirmed ASSERT identity flag. |
pyrit/models/dataset_compatibility.py |
Define compatibility response models. |
pyrit/executor/promptgen/__init__.py |
Export objective-generation results. |
pyrit/datasets/score/true_false_question/assert_execution_evidence_instructions.yaml |
Define trace-evidence judging instructions. |
pyrit/datasets/score/assert/judge_presets/tool-use.yaml |
Add tool-use judging dimensions. |
pyrit/datasets/score/assert/judge_presets/safety-extended.yaml |
Add harm and privacy dimensions. |
pyrit/datasets/score/assert/judge_presets/safety-core.yaml |
Reference built-in safety dimensions. |
pyrit/datasets/score/assert/judge_presets/robustness.yaml |
Add robustness judging dimensions. |
pyrit/datasets/score/assert/judge_presets/policy-adherence.yaml |
Add policy-adherence rubric. |
pyrit/datasets/score/assert/judge_presets/operational.yaml |
Add operational judging dimensions. |
pyrit/datasets/score/assert/judge_presets/multi-turn.yaml |
Add conversational continuity dimensions. |
pyrit/datasets/score/assert/judge_presets/instruction-following.yaml |
Add instruction-following dimensions. |
pyrit/datasets/score/assert/judge_presets/grounding.yaml |
Add grounding judging dimensions. |
pyrit/datasets/score/assert/judge_presets/communication.yaml |
Add communication-quality rubric. |
pyrit/datasets/score/assert/judge_presets/alignment.yaml |
Add alignment judging dimensions. |
pyrit/datasets/score/assert/dimension_user_prompt.yaml |
Template custom-dimension judge requests. |
pyrit/datasets/score/assert/dimension_system_prompt.yaml |
Define custom-dimension evaluation instructions. |
pyrit/datasets/assert_suites/travel_planner.yaml |
Bundle travel-planner evaluation suite. |
pyrit/datasets/assert_suites/travel_planner_benchmark.yaml |
Bundle travel-quality benchmark suite. |
pyrit/datasets/assert_behaviors/unsupported_conclusion_failures.yaml |
Add unsupported-conclusion preset. |
pyrit/datasets/assert_behaviors/unit_conversion_failures.yaml |
Add unit-conversion preset. |
pyrit/datasets/assert_behaviors/underused_context_failures.yaml |
Add context-underuse preset. |
pyrit/datasets/assert_behaviors/unauthorized_action_failures.yaml |
Add unauthorized-action preset. |
pyrit/datasets/assert_behaviors/tool_parameter_formatting_failures.yaml |
Add tool-argument formatting preset. |
pyrit/datasets/assert_behaviors/tool_output_misinterpretation_failures.yaml |
Add tool-output interpretation preset. |
pyrit/datasets/assert_behaviors/tool_orchestration_errors.yaml |
Add tool-orchestration preset. |
pyrit/datasets/assert_behaviors/tool_call_turn_protocol_failures.yaml |
Add tool-turn protocol preset. |
pyrit/datasets/assert_behaviors/tool_call_error_recovery_failures.yaml |
Add tool-error recovery preset. |
pyrit/datasets/assert_behaviors/tabs_vs_spaces.yaml |
Add indentation preference-bias preset. |
pyrit/datasets/assert_behaviors/sycophancy.yaml |
Add sycophancy preset. |
pyrit/datasets/assert_behaviors/suicide_self_harm.yaml |
Add self-harm safety preset. |
pyrit/datasets/assert_behaviors/success_criteria_ambiguity_failures.yaml |
Add completion-criteria preset. |
pyrit/datasets/assert_behaviors/stereotyping.yaml |
Add stereotyping preset. |
pyrit/datasets/assert_behaviors/stale_state_failures.yaml |
Add stale-state preset. |
pyrit/datasets/assert_behaviors/shimano_vs_sram.yaml |
Add cycling-brand preference preset. |
pyrit/datasets/assert_behaviors/response_completeness_failures.yaml |
Add response-completeness preset. |
pyrit/datasets/assert_behaviors/repeated_action_loop_failures.yaml |
Add repeated-action preset. |
pyrit/datasets/assert_behaviors/prompt_injection.yaml |
Add prompt-injection evaluation preset. |
pyrit/datasets/assert_behaviors/procedure_adherence_failures.yaml |
Add procedure-adherence preset. |
pyrit/datasets/assert_behaviors/premature_termination_failures.yaml |
Add premature-termination preset. |
pyrit/datasets/assert_behaviors/poor_retrieval_failures.yaml |
Add retrieval-quality preset. |
pyrit/datasets/assert_behaviors/output_internal_consistency_failures.yaml |
Add output-consistency preset. |
pyrit/datasets/assert_behaviors/out_of_scope_request_failures.yaml |
Add scope-boundary preset. |
pyrit/datasets/assert_behaviors/observation_neglect_failures.yaml |
Add observation-neglect preset. |
pyrit/datasets/assert_behaviors/ios_vs_android.yaml |
Add mobile-platform preference preset. |
pyrit/datasets/assert_behaviors/inter_agent_handoff_failures.yaml |
Add agent-handoff preset. |
pyrit/datasets/assert_behaviors/intent_misinterpretation_failures.yaml |
Add intent-interpretation preset. |
pyrit/datasets/assert_behaviors/insufficient_verification_failures.yaml |
Add insufficient-verification preset. |
pyrit/datasets/assert_behaviors/ineffective_team_communication_failures.yaml |
Add agent-communication preset. |
pyrit/datasets/assert_behaviors/incorrect_tool_selection_failures.yaml |
Add tool-selection preset. |
pyrit/datasets/assert_behaviors/incomplete_answer_synthesis_failures.yaml |
Add answer-synthesis preset. |
pyrit/datasets/assert_behaviors/identity_verification_failures.yaml |
Add identity-verification preset. |
pyrit/datasets/assert_behaviors/harmful_medical_advice.yaml |
Add medical-advice safety preset. |
pyrit/datasets/assert_behaviors/grounding_attribution_errors.yaml |
Add grounding-attribution preset. |
pyrit/datasets/assert_behaviors/goal_drift_failures.yaml |
Add goal-drift preset. |
pyrit/datasets/assert_behaviors/flawed_action_plan_failures.yaml |
Add action-planning preset. |
pyrit/datasets/assert_behaviors/explicit_constraint_violation_failures.yaml |
Add constraint-violation preset. |
pyrit/datasets/assert_behaviors/escalation_judgment_failures.yaml |
Add escalation-decision preset. |
pyrit/datasets/assert_behaviors/doxxing.yaml |
Add personal-information protection preset. |
pyrit/datasets/assert_behaviors/crystal_mountain_vs_snoqualmie.yaml |
Add resort preference-bias preset. |
pyrit/datasets/assert_behaviors/conversation_coherence_breakdown.yaml |
Add conversational-coherence preset. |
pyrit/datasets/assert_behaviors/constraint_propagation_failures.yaml |
Add constraint-propagation preset. |
pyrit/datasets/assert_behaviors/conflicting_instruction_resolution_failures.yaml |
Add instruction-conflict preset. |
pyrit/datasets/assert_behaviors/actionability_failures.yaml |
Add answer-actionability preset. |
pyrit/datasets/__init__.py |
Export ASSERT library accessors. |
pyrit/common/path.py |
Define bundled library paths. |
pyrit/backend/services/service_lifecycle.py |
Close generation service during cleanup. |
pyrit/backend/services/runtime_lifecycle.py |
Track generation work during shutdown. |
pyrit/backend/main.py |
Register generation routes. |
pyrit/analytics/__init__.py |
Export ASSERT metrics and identity helpers. |
frontend/src/utils/routeParams.ts |
Add dataset-preselection URL helpers. |
frontend/src/test-utils/replaceText.ts |
Add efficient text-input test helper. |
frontend/src/hooks/usePromptGenerators.ts |
Load runtime-aware generator descriptors. |
frontend/src/hooks/usePromptGeneration.ts |
Poll generation records and lists. |
frontend/src/hooks/useGenerationTargetSelection.ts |
Preserve target choices and defaults. |
frontend/src/hooks/useAssertLibrary.ts |
Load bundled behavior and suite presets. |
frontend/src/components/Scenarios/scenarioLaunchForm.ts |
Seed dataset selections from links. |
frontend/src/components/Scenarios/ScenarioFlow.test.tsx |
Mock scenario-compatible dataset requests. |
frontend/src/components/Scenarios/ScenarioDetail.styles.ts |
Style dataset section actions. |
frontend/src/components/Scenarios/DatasetActions.tsx |
Add dataset browsing and generation links. |
frontend/src/components/Scenarios/DatasetActions.styles.ts |
Style accessible dataset actions. |
frontend/src/components/Scenarios/assertPolicyRun.ts |
Identify tentative ASSERT metrics candidates. |
frontend/src/components/Scenarios/AssertPolicyMetricsPanel.styles.ts |
Style ASSERT metric tables. |
frontend/src/components/ScenarioPresets/ScenarioPresetEditor.tsx |
Scope dataset metadata by scenario. |
frontend/src/components/ScenarioPresets/ScenarioPresetEditor.test.tsx |
Mock compatible datasets for presets. |
frontend/src/components/PromptGeneration/TargetObjectiveSummary.tsx |
Display generated objective totals. |
frontend/src/components/PromptGeneration/PromptGenerationStatusBadge.tsx |
Display generation status badges. |
frontend/src/components/PromptGeneration/promptGenerationRoutes.test.ts |
Cover generation URL encoding. |
frontend/src/components/PromptGeneration/generatorUi.ts |
Define generator UI contracts. |
frontend/src/components/PromptGeneration/GenerationSummaryFacts.tsx |
Render generation summary facts. |
frontend/src/components/Datasets/DatasetSelection.tsx |
Link datasets to regeneration. |
frontend/src/App.tsx |
Register generation pages. |
frontend/README.md |
Document isolated generation browser checks. |
frontend/playwright.prompt-generation.config.ts |
Configure isolated generation journeys. |
frontend/playwright.config.ts |
Exclude fixture-specific generation journeys. |
frontend/package.json |
Add generation browser-check command. |
frontend/e2e/numeric-controls.spec.ts |
Mock generator discovery requests. |
frontend/e2e/datasets.spec.ts |
Mock missing generation provenance. |
doc/myst.yml |
Register trace-evidence guide. |
doc/gui/0_gui.md |
Explain GUI dataset generation. |
doc/code/scoring/0_scoring.py |
Link execution-trace scoring documentation. |
doc/code/scoring/0_scoring.ipynb |
Synchronize trace-scoring documentation link. |
doc/code/executor/7_promptgen.py |
Demonstrate staged policy review. |
.github/workflows/frontend_tests.yml |
Run generation browser checks in CI. |
.github/instructions/scenarios.instructions.md |
Clarify inline-seed dataset capabilities. |
Review effort: Balanced
| '/api/labels': { source: 'attacks', labels: {} }, | ||
| '/api/scorers': { items: [], pagination: { has_more: false } }, | ||
| '/api/datasets': { items: datasetItems }, | ||
| '/api/prompt-generators': { items: [] }, |
This was referenced Nov 12, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part -9 of a 10-layer ASSERT series. Design and how to use it: ASSERT in PyRIT.
Each PR is separately cumulative and targets
main; this is not a native stack. Review the independent layer-8-to-layer-9 source delta.Layer-8 ancestry warning: Review the layer-8-only Files changed comparison, not only the commit list: retained prototype ancestry includes superseded category/loading commits, but their changes are absent from layer 8's final tree. Layer 9 adds category grouping through the independent delta above; background loading belongs to #4228, not this PR's net source.
Publication clarification: The preserved body below predates publication of #4228; its references to layer -10 as Planned are historical. Layer -10 is now published as #4228 and remains outside this PR's net source.
Description
Organize new ASSERT result groups by generated category, with baseline and selected techniques together, without changing execution identity or judged-category metrics. Preserve recorded grouping when resuming or displaying historical runs.
Own GUI changes
New ASSERT runs show generated-category groups instead of technique groups. Each group contains baseline, prompt-sending and red-teaming rows when those techniques are present. Policy-qualified headers distinguish otherwise identical category/behavior names without exposing system instructions, serialized arrays or hashes.
The explanation explicitly distinguishes generated-category organization from judged-relevance category rates. Historical ASSERT runs retain their recorded grouping, including delimiter/quote/suffix-looking labels. A no-plan map-only run keeps its saved label and an unmapped atomic-name fallback; incomplete historical planned totals remain unavailable rather than being fabricated.
Shared/non-ASSERT impact
The shared hierarchy/progress readers accept recorded display-map evidence only for a proven ASSERT run. The shared result table receives optional readable labels only for confirmed ASSERT presentation. Ordinary scenarios retain their existing grouping, row labels and logical-unit/retry accounting.
No-registry canonical-module legacy ASSERT is accepted. A same-named class from an unrelated or unknown module gets no category-specific explanation or row reinterpretation; its separately tentative metrics request receives the existing typed non-ASSERT rejection and the metrics panel hides.
Unchanged/deferred behavior
Judges, headline/rate formulas, confidence intervals, trace evidence, generation, native prompt/conversation execution, Spanish conversion and dataset compatibility are inherited lower-layer behavior. Category grouping does not turn unscored fixture outcomes into judgments. Dataset auto-loading is Planned layer -10, not included in this layer's net source; manual loading and other scenario behavior are unchanged.
Exact source: after
53f7b3b52fe0f5ff394c48e556cbf31b2810e838, first parent published layer -8fa20157a6a1fdd484bea1a5dc41d957bfc2cfad3. The retained layer-8 prototype ancestry contains superseded category/loading commit titles; inspect the tree comparison, not those titles. No moving-main merge or dependency/lockfile changes were included.Source-specific synthetic before/after
All images are genuine browser captures of separately built exact sources: before = published layer -8
fa20157a6a1fdd484bea1a5dc41d957bfc2cfad3, after = layer -953f7b3b52fe0f5ff394c48e556cbf31b2810e838. Both use isolated fresh SQLite fixtures, the real backend/lifespan/REST readers and bundled SPA, Chromium 156 at 1440x1050. Small ASSERT runs execute the real pipeline against scripted in-process responses; paging and historical records are explicitly synthetic. No API/DOM substitution, real provider/model calls, user data/configuration or running demo was used.Before: three technique groups, twelve executions.
After: two generated-category groups with baseline and both techniques, the same twelve executions.
Distinct policy-qualified headers, before and after.
Rates remain judged-relevance rates, not generated-group counts. The fixture intentionally has six executions in each generated category but judged applicable denominators of three permissible and one impermissible conversation per technique.
Historical no-plan map fix and unchanged recorded atomic labels
The 504-unit no-plan map fixture records 503 atomic units under
Recorded map-only groupand leaves one original atomic name unmapped. Before ignores the map; after preserves it and the fallback. Actual REST pages are 500+4 with unchanged completed totals and disjoint result IDs. Historical planned totals remain unknown.The separate canonical-module/no-registry legacy fixture retains its three original delimiter/quote/suffix-looking labels, without a backing dataset or a fabricated complete plan.
Ordinary scenario and hard negative identity comparisons
The ordinary retry fixture keeps its existing non-ASSERT presentation and server logical accounting.
Same class name, unrelated module: no ASSERT grouping explanation or metrics panel.
Same class name, unknown module: also not proof of ASSERT identity.
Genuine persisted 504-unit paging before and after Load more results
Each source has 504 initialized native logical units and 504 stored synthetic unscored target-facing attempts. The real hierarchy produces 500 roots, then four disjoint roots after the actual Load more interaction. Full progress/hierarchy totals and per-atomic counts are identical across pages; after-source generated groups remain 252 each. Screenshots are previews, not a claim that all 504 rows are simultaneously rendered.