Repository navigation
FEAT: Show who saved each converter, scorer and technique (part 15) - #4067
Open
varunj-msft wants to merge 415 commits into
Open
varunj-msft wants to merge 415 commits into
varunj-msft wants to merge 415 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
A critical registry conflict raises without recording a refusal, allowing initialization to succeed and a saved technique to run instead of the script’s intended technique.
1 open issue
What changed in this PR
Extends saved-entry attribution from targets to converters, scorers, and techniques, helping users identify their entries in shared PyRIT deployments. The stacked diff also includes persistence infrastructure from earlier PRs.
Changes:
- Adds saver display names and caller-specific ownership flags without returning user object IDs.
- Adds shared ownership captions, origin badges, and “Only … I saved” filters.
- Includes saved-entry validation, restore safeguards, documentation, and regression coverage.
| File | Description |
|---|---|
| tests/unit/setup/test_targets_initializer.py | Tests saved-target collision protection. |
| tests/unit/setup/test_scorer_initializer.py | Tests saved-scorer tagging safeguards. |
| tests/unit/setup/test_converter_initializer.py | Tests saved-converter replacement refusal. |
| tests/unit/setup/test_configuration_loader.py | Tests recipe storage configuration validation. |
| tests/unit/registry/test_instance_registry.py | Tests origins and recorded replacement refusals. |
| tests/unit/registry/test_converter_registry.py | Tests saved-converter name clashes. |
| tests/unit/registry/test_attack_technique_registry.py | Tests saved-technique registration safeguards. |
| tests/unit/backend/test_technique_edit_service.py | Updates keyword-only service calls. |
| tests/unit/backend/test_scorer_routes.py | Tests persistence errors and caller forwarding. |
| tests/unit/backend/test_scenario_suite_routes.py | Tests sanitized storage failures. |
| tests/unit/backend/test_scenario_suite_execution_service.py | Tests saved-entry launch conflict reporting. |
| tests/unit/backend/test_scenario_run_service.py | Tests launch-script replacement safeguards. |
| tests/unit/backend/test_scenario_preset_routes.py | Tests sanitized storage errors and logs. |
| tests/unit/backend/test_saved_entries.py | Tests temporary dependency rejection. |
| tests/unit/backend/test_mappers.py | Tests origins, attribution, and object-ID omission. |
| tests/unit/backend/test_initializer_service.py | Tests local storage failure handling. |
| tests/unit/backend/test_auth_middleware.py | Tests authenticated caller ID extraction. |
| tests/unit/backend/test_analytics_benchmark.py | Tests benchmark recipe storage isolation. |
| tests/unit/backend/conftest.py | Isolates saved entries during backend tests. |
| tests/end_to_end/conftest.py | Gives test backends isolated recipe storage. |
| test_support/analytics_benchmark.py | Configures isolated benchmark recipe storage. |
| pyrit/setup/initializers/tools.py | Rejects saved-entry substitutions during tool setup. |
| pyrit/setup/initializers/targets.py | Guards round-robin names against saved targets. |
| pyrit/setup/initializers/scorers.py | Guards preferred scorer tags against saved substitutions. |
| pyrit/setup/configuration_loader.py | Adds recipe storage source configuration. |
| pyrit/score/true_false/local_refusal_classifier_scorer.py | Declares local classifier persistence restrictions. |
| pyrit/score/true_false/audio_true_false_scorer.py | Declares audio scorer persistence restrictions. |
| pyrit/score/scorer.py | Adds scorer saveability metadata. |
| pyrit/score/float_scale/azure_content_filter_scorer.py | Declares server credential fallback behavior. |
| pyrit/score/float_scale/audio_float_scale_scorer.py | Declares audio scorer persistence restrictions. |
| pyrit/registry/components/scorer_registry.py | Reserves the scorer types route name. |
| pyrit/registry/components/attack_technique_registry.py | Adds admission checks and saved-entry protection. |
| pyrit/prompt_target/websocket_copilot_target.py | Declares account-dependent persistence restrictions. |
| pyrit/prompt_target/prompt_shield_target.py | Declares credential and destination constraints. |
| pyrit/prompt_target/openai/openai_target.py | Declares credential, header, and destination constraints. |
| pyrit/prompt_target/litellm_chat_target.py | Declares API-key fallback behavior. |
| pyrit/prompt_target/hugging_face/hugging_face_chat_target.py | Declares token and model persistence restrictions. |
| pyrit/prompt_target/http_target/http_target.py | Disallows saving credential-bearing raw requests. |
| pyrit/prompt_target/hack_a_prompt_target.py | Declares session and cookie fallback behavior. |
| pyrit/prompt_target/github_copilot_target.py | Declares GitHub token fallback behavior. |
| pyrit/prompt_target/common/prompt_target.py | Defines target persistence constraint metadata. |
| pyrit/prompt_target/azure_ml_chat_target.py | Declares credential and endpoint constraints. |
| pyrit/prompt_target/azure_blob_storage_target.py | Declares token and container URL constraints. |
| pyrit/prompt_target/a2a_target.py | Declares endpoint and agent-card destinations. |
| pyrit/models/catalog/target.py | Adds target origin and attribution fields. |
| pyrit/models/catalog/scorer.py | Adds scorer origin and attribution fields. |
| pyrit/models/catalog/registry_recipe.py | Defines recipes, ownership, and restore reports. |
| pyrit/models/catalog/__init__.py | Exports recipe and restore models lazily. |
| pyrit/converter/converter.py | Defines converter credential fallback metadata. |
| pyrit/converter/azure_speech_text_to_audio_converter.py | Declares Speech credential fallback behavior. |
| pyrit/converter/azure_speech_audio_to_text_converter.py | Declares Speech credential fallback behavior. |
| pyrit/backend/services/scenario_suite_execution_service.py | Exposes safe saved-entry launch conflict messages. |
| pyrit/backend/services/scenario_run_service.py | Detects saved-entry refusals during launch scripts. |
| pyrit/backend/services/runtime_lifecycle.py | Restores saved entries and reports failures. |
| pyrit/backend/routes/techniques.py | Forwards caller identity and handles persistence errors. |
| pyrit/backend/routes/targets.py | Adds attribution and persistence route handling. |
| pyrit/backend/routes/scorers.py | Adds attribution and persistence route handling. |
| pyrit/backend/routes/scenarios.py | Documents saved-entry launch conflicts. |
| pyrit/backend/routes/scenario_suites.py | Sanitizes storage diagnostics. |
| pyrit/backend/routes/scenario_presets.py | Sanitizes storage diagnostics. |
| pyrit/backend/routes/initializers.py | Handles and sanitizes local storage failures. |
| pyrit/backend/routes/converters.py | Adds attribution, save errors, and removal protection. |
| pyrit/backend/routes/common.py | Shares owner extraction and save-refusal responses. |
| pyrit/backend/README.md | Documents persistence and attribution API behavior. |
| pyrit/backend/models/techniques.py | Adds persistence validation and attribution fields. |
| pyrit/backend/models/targets.py | Adds persistence validation and saveability fields. |
| pyrit/backend/models/scorers.py | Adds scorer persistence validation. |
| pyrit/backend/models/converters.py | Adds persistence, saveability, and attribution fields. |
| pyrit/backend/middleware/auth.py | Exposes authenticated caller ID dependency. |
| pyrit/backend/mappers/technique_mappers.py | Maps technique origin and saver attribution. |
| pyrit/backend/mappers/target_mappers.py | Maps target origin and saver attribution. |
| pyrit/backend/mappers/converter_mappers.py | Maps converter origin and saver attribution. |
| frontend/src/utils/registryOrigin.ts | Shares origin labels and ownership captions. |
| frontend/src/utils/registryName.ts | Generates valid, available registry names. |
| frontend/src/utils/registryName.test.ts | Tests name validation and suggestions. |
| frontend/src/types/index.ts | Adds persistence and attribution API types. |
| frontend/src/test-utils/targetFixtures.ts | Supports origin and attribution in fixtures. |
| frontend/src/services/errors.ts | Recognizes typed save-refusal responses. |
| frontend/src/services/errors.test.ts | Tests save-refusal classification. |
| frontend/src/components/Techniques/TechniqueEditor.test.tsx | Updates converter save-button expectations. |
| frontend/src/components/Registry/TechniqueRegistry.tsx | Shows origins, attribution, and ownership filtering. |
| frontend/src/components/Registry/ScorerRegistry.tsx | Shows origins, attribution, and ownership filtering. |
| frontend/src/components/Registry/ScorerRegistry.test.tsx | Tests attribution captions and ownership filtering. |
| frontend/src/components/Registry/SavedBy.tsx | Provides a shared saver caption component. |
| frontend/src/components/Registry/SavedBy.styles.ts | Styles saver captions. |
| frontend/src/components/Registry/OriginBadge.tsx | Provides shared entry-origin badges. |
| frontend/src/components/Registry/ConverterRegistry.tsx | Adds attribution filtering and saved-removal guidance. |
| frontend/src/components/Registry/ConverterRegistry.test.tsx | Tests attribution, filtering, and removal protection. |
| frontend/src/components/Converters/SaveConverterDialog.tsx | Clarifies that converter copies remain temporary. |
| frontend/src/components/Converters/SaveConverterDialog.test.tsx | Tests temporary-copy messaging. |
| frontend/src/components/Config/TargetTable.tsx | Adds origin labels and ownership filtering. |
| frontend/src/components/Config/TargetTable.test.tsx | Tests attribution and combined filters. |
| frontend/src/components/Config/TargetTable.styles.ts | Styles captions and visibility controls. |
| frontend/playwright.config.ts | Isolates seeded backend recipe storage. |
| frontend/e2e/fixtures/manual_send_backend.py | Isolates offline backend recipe storage. |
| frontend/e2e/config.spec.ts | Tests target attribution and save requests. |
| frontend/e2e/_targets.ts | Adds attribution fields to browser mocks. |
| doc/gui/0_gui.md | Explains saved entries and ownership filters. |
| doc/getting_started/pyrit_conf.md | Documents recipe storage and restore behavior. |
| .pyrit_conf_example | Adds commented recipe storage configuration. |
Review effort: Balanced
…hanges A 409 for a taken name or tag kept Save disabled until the name changed, so correcting the tag didn't help. The refusal now applies only to the name and tags it was given for.
# Conflicts: # frontend/src/components/Registry/TechniqueRegistry.test.tsx
A launch script that caught the refusal of a technique whose name is a saved technique's tag, or whose tag is a saved technique's name or another spelling of its tag, could leave the saved technique answering that selector. Those clashes are now recorded like a name clash, so the scan stops.
varunj-msft
requested review from
hannahwestra25 and
Richard Lundeen (richlundeen)
November 8, 2026 23:56
This was referenced Nov 9, 2026
varunj-msft
changed the title
FileDocumentStorage now raises DocumentStorageUnavailableError when the folder or container can't be used, and DocumentUnreadableError when one document exists but can't be read. The message names the location, with any SAS token hidden, and a short reason. One backend handler maps both to 503 and logs the location and reason, with the traceback at debug level, so the per-route storage helpers are gone. A single read, a conditional save and a delete now follow the listing's rules: a folder, pipe or broken link named like a document is unreadable rather than missing, and on Blob only a deleted blob counts as missing. Blob sign-in, permission, network and missing-container errors stop the listing instead of marking every file unreadable. Each unreadable report keeps the real file name and the original error, and the listing's documents are read-only. Initializer delete removes a script that can't be read or decoded, and registering one no longer turns a storage write failure into a 400. Tests patch module-level aliases instead of os.
List recipes escapes the OS text it reports, such as a file name that isn't UTF-8, and a recipe whose JSON holds an unpaired surrogate escape is reported as invalid, so the models no longer need a JSON-only serializer. Recipes may start with a byte-order mark. A document without schema_version reads as version 1, and each file is checked against the format version of the model that reads it. Validation summaries shorten each location part and the whole message, since a location can repeat a stored key. Recipes and StoredRecipe are no longer frozen, since their params and tags never were, and StoredRecipe checks that its recipe type matches its registry. The listing's recipes are read-only, and it keeps the error behind each recipe that couldn't be read, for the log.
Listing catches only storage errors, so a bug is raised instead of being reported as unreadable storage, and a bug while planning is a build error rather than an invalid recipe. Each failure is logged once, with the error behind it and URL credentials removed. The report shows the error's text with each URL's user name, query and fragment removed, any SAS signature hidden, and the length capped. A class that moved within PyRIT is restored; one now registered from outside PyRIT still fails. A technique whose name or tag clashes is a name clash, and anything else registration refuses is an invalid recipe. A recipe that sets a server file or folder is refused before it is built. Class metadata comes from the registry's cache. The class docstring lists the steps the API's create runs that restore skips.
The runtime becomes ready even when some saved entries fail; the failures stay in the runtime status. Each failed entry keeps its name in its registry: looking it up, or taking the name, raises SavedEntryUnavailableError, which the backend answers with 409, so nothing else stands in for it. Administrators see the full message and everyone else a fixed explanation of the reason. A folder that can't be listed holds every unregistered name of its kind, and each entry it left unbuilt is reported. While a saved technique is held, a launch or estimate that picks techniques by all, default or a tag is refused with 409 for scenarios whose techniques come from the registry, since the broken technique's tags may be unknown. Naming techniques still works; naming the held one fails with its reason. Route catch-alls let the error through instead of turning it into a 500.
A launch script that registers a name held for a saved entry that couldn't be restored is refused, and the refusal is recorded even when the script catches it, so the launch stops instead of running with the script's entry in the saved one's place. A suite entry that fails on a held entry, or on a broad technique selection, shows the reason without the error's details.
# Conflicts: # frontend/src/components/Config/CreateTargetDialog.tsx
A suite launch runs its launch scripts during preflight and sends its scans without them, so the clash is raised there. Map it to 409 with its message, as a single launch does, and drop the per-entry message that a clash can no longer reach.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is one of the changes for saving Registry entries across restarts. Until the earlier PRs merge, the diff here also shows their changes.
The Targets table already says who saved each target (#4028). Converters, scorers and techniques didn't, so in a shared deployment you couldn't tell your saved entries from someone else's.
ConverterInstance,ScorerInstanceandTechniqueInstanceaddsaved_by, the name of the person who saved the entry when sign-in was on, andsaved_by_me. The server works outsaved_by_mefrom the signed-in caller, so no object ID leaves the server. The list and get routes for the three kinds pass the caller in.What stays the same: with sign-in off, nobody is recorded, so no captions or checkbox appear. Who saved an entry doesn't change who can use it.
Screenshots: the Converters tab with "Saved by you" and "Saved by" captions, and with Only converters I saved checked.
Not covered: