FEAT Add backend APIs by romanlutz · Pull Request #1354 · Azure/PyRIT

romanlutz · 2026-02-05T21:35:13Z

Description

Adding backend APIs to support upcoming frontend development. This is based on an initial proposal and review.

Tests and Documentation

Includes tests for all APIs.

…backend_apis

frontend/dev.py

pyrit/backend/models/attacks.py

Copilot

Pull request overview

This PR adds comprehensive backend REST APIs to support upcoming frontend development, implementing an attack-centric design where all user interactions (including manual conversations) are modeled as "attacks". The implementation includes service layers, API routes, Pydantic models, error handling middleware, registries for managing target/converter instances, a CLI tool, and comprehensive test coverage.

Changes:

Adds three main services (AttackService, ConverterService, TargetService) for managing attacks, converters, and targets
Implements REST API routes for attacks, targets, converters, labels, health, and version endpoints
Creates Pydantic models for request/response validation with RFC 7807 error handling
Adds instance registries for targets and converters with singleton pattern
Implements pyrit_backend CLI command for starting the server with initializer support
Includes 900+ lines of comprehensive unit tests for all services and routes

Reviewed changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated 21 comments.

Show a summary per file

File	Description
pyrit/backend/services/attack_service.py	Service layer for managing attack lifecycle, messages, and scoring (586 lines)
pyrit/backend/services/converter_service.py	Service for managing converter instances and previewing conversions (303 lines)
pyrit/backend/services/target_service.py	Service for managing target instance creation and retrieval (187 lines)
pyrit/backend/routes/attacks.py	REST API endpoints for attack CRUD operations and messaging (249 lines)
pyrit/backend/routes/targets.py	REST API endpoints for target instance management (101 lines)
pyrit/backend/routes/converters.py	REST API endpoints for converter instances and preview (134 lines)
pyrit/backend/routes/labels.py	REST API endpoint for retrieving filter label options (88 lines)
pyrit/backend/models/attacks.py	Pydantic models for attack requests/responses (201 lines)
pyrit/backend/models/targets.py	Pydantic models for target instances (52 lines)
pyrit/backend/models/converters.py	Pydantic models for converter instances and preview (98 lines)
pyrit/backend/models/common.py	Common models including RFC 7807 error responses and sensitive field filtering (93 lines)
pyrit/backend/middleware/error_handlers.py	RFC 7807 compliant error handler middleware (182 lines)
pyrit/registry/instance_registries/target_registry.py	Registry for managing target instances (88 lines)
pyrit/registry/instance_registries/converter_registry.py	Registry for managing converter instances (108 lines)
pyrit/cli/pyrit_backend.py	CLI command for starting the backend server with initialization support (217 lines)
tests/unit/backend/*.py	Comprehensive unit tests for all services, routes, models, and error handlers (2000+ lines)
pyrit/backend/main.py	Updated to register new routes and error handlers
pyproject.toml	Adds pyrit_backend CLI entry point and documentation exemption
frontend/dev.py	Updated to use pyrit_backend CLI instead of direct uvicorn

pyrit/cli/pyrit_backend.py

pyrit/backend/services/attack_service.py

pyrit/backend/services/converter_service.py

pyrit/backend/services/target_service.py

pyrit/backend/services/attack_service.py

tests/unit/backend/test_attack_service.py

tests/unit/backend/test_converter_service.py

tests/unit/backend/test_target_service.py

pyrit/backend/routes/attacks.py

bashirpartovi

Great job Roman, this is my first pass at your PR, I do have a few more comments but need to think about them more

pyrit/backend/routes/attacks.py

pyrit/backend/main.py

pyrit/backend/services/attack_service.py

pyrit/backend/main.py

pyrit/backend/models/targets.py

pyrit/backend/routes/targets.py

pyrit/backend/routes/attacks.py

…backend_apis

…/PyRIT into romanlutz/backend_apis

bashirpartovi

Looks great Roman, just a couple of questions and a call-out :)

pyrit/backend/services/attack_service.py

pyrit/backend/routes/labels.py

pyrit/backend/models/attacks.py

…backend_apis

…acking Labels ------ - Source labels from PromptMemoryEntry.labels instead of AttackResult.metadata - Add _collect_labels_from_pieces helper to derive labels from pieces - Add get_unique_attack_labels() to MemoryInterface using JOIN+DISTINCT - Simplify /labels route to delegate to get_unique_attack_labels() - Remove labels from attack metadata in create_attack - Forward labels to prepended pieces and add_message pieces - Inherit labels from existing pieces when adding new messages Pagination ---------- - Refactor list_attacks_async into two phases: Phase 1: query + filter + sort on lightweight AttackResult objects Phase 2: fetch pieces only for the final paginated page - Rename _paginate to _paginate_attack_results (operates on AttackResult) Lineage tracking ---------------- - Add original_prompt_id to MessagePieceRequest DTO - Forward original_prompt_id through mapper to MessagePiece domain object Validation & style ------------------ - Add max_length=50 on PrependedMessageRequest.pieces and AddMessageRequest.pieces - Add max_length=200 on CreateAttackRequest.prepended_conversation - Add TOCTOU comment on sequence read-then-write in add_message_async - Add import placement rule to style guide - Move contextlib.closing import to top of memory_interface.py - Remove unused TYPE_CHECKING import from labels route Tests ----- - Add get_unique_attack_labels tests (empty, single, merge, no pieces, no labels, non-attack pieces, non-string values, sorted keys) - Add pagination test verifying pieces fetched only for page - Add original_prompt_id forwarding and default tests in mappers - Add labels stamping tests for prepended pieces and add_message - Add prepend ordering + lineage preservation test - Add _collect_labels_from_pieces tests - Update existing tests to match new label source (pieces, not metadata)

Replace opaque Dict[str, Any] fields in API DTOs with explicit typed fields to prevent internal PyRIT core structures from leaking through the REST API boundary. Models: - AttackSummary: replace attack_identifier dict with attack_type, attack_specific_params, target_unique_name, target_type, converters - TargetInstance: replace identifier dict with target_type, endpoint, model_name, temperature, top_p, max_requests_per_minute, target_specific_params - ConverterInstance: replace type/params with converter_type, supported_input_types, supported_output_types, converter_specific_params, sub_converter_ids - CreateConverterResponse: use converter_type/display_name Mappers: - Extract specific fields from identifier objects via attribute access instead of calling .to_dict() + filter_sensitive_fields() - Add dedicated mapper modules for targets and converters API enhancements: - Add attack list filtering by attack_class, converter_classes, outcome, labels, min/max turns with SQL-level filtering - Add /attacks/attack-options and /attacks/converter-options endpoints - Support three-state converter_classes: None=no filter, []=no converters, non-empty=must have all listed - Stamp source:gui label via labels.setdefault() in create_attack_async - Add get_unique_attack_labels endpoint for label discovery Memory layer: - Add get_unique_attack_class_names, get_unique_converter_class_names, get_unique_attack_labels to MemoryInterface, SQLiteMemory, and AzureSQLMemory - SQL-level filtering for attack results by labels and harm categories Tests: - Expand mapper tests with coverage for no-target, converters extraction, attack_specific_params passthrough, None input/output types - Expand attack service tests for filtering, options, pagination - Update all test mocks to use attribute-based identifiers - Fix mypy errors (Sequence[Any] for pieces, ChatMessageRole for role)

…backend_apis

- Widen role fields in DTOs from Literal["user", "assistant", "system"] to ChatMessageRole so simulated_assistant (and other roles like tool, developer) flow through to the frontend without remapping - Switch mapper from deprecated .role property (which collapses simulated_assistant → assistant) to get_role_for_storage() which preserves the actual stored role - Add TestDomainModelFieldsExist: 16 parametrized tests that verify every field the mappers access still exists on AttackIdentifier, TargetIdentifier, and ConverterIdentifier dataclasses - Update mock pieces in test_mappers.py and test_attack_service.py to configure get_role_for_storage() Files changed: pyrit/backend/models/attacks.py pyrit/backend/mappers/attack_mappers.py tests/unit/backend/test_mappers.py tests/unit/backend/test_attack_service.py tests/unit/backend/test_converter_service.py

pyrit/backend/mappers/attack_mappers.py

pyrit/backend/mappers/target_mappers.py

- Fix .lower() crash on int log_level in pyrit_backend CLI - Guard against double initialization in lifespan (CentralMemory check) - Remove CREATE_NEW_PROCESS_GROUP in dev.py (broke Ctrl+C on Windows) - Add sys.stdout/stderr.reconfigure(errors="replace") for Windows cp1252 - Route e2e tests through Vite proxy with beforeAll health polling - Use 127.0.0.1 instead of localhost to avoid IPv6 ECONNREFUSED - Suppress noisy proxy error logs via custom Vite logger - Return 502 on proxy errors so polling doesn't hang - Increase Playwright webServer timeout to 120s for CI - Add test for lifespan skip-init when CentralMemory is already set

Copilot

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated 2 comments.

pyrit/memory/memory_interface.py

pyrit/registry/instance_registries/converter_registry.py

- Add `from __future__ import annotations` to attack_mappers.py and remove string quotes from ChatMessageRole type hint - Use ScorerIdentifier.class_name instead of dict .get("__type__") for scorer type extraction (ScorerIdentifier is a dataclass, not dict) - Convert score_value str→float when building Score DTO - Annotate prompt_metadata as Optional[Dict[str, str | int]] to match MessagePiece parameter type - Fix type: ignore comments in pyrit_backend.py (attr-defined → union-attr) - Replace placeholder ConverterIdentifier in converter_registry with the real one from pyrit.identifiers, passing supported_input/output_types - Remove redundant string quotes from TargetRegistry return type - Update test mock to use real ScorerIdentifier instead of plain dict All 384 tests pass, pre-commit (including mypy strict) clean.

romanlutz added 14 commits January 25, 2026 06:15

backend with APIs that will be needed in the short-term.

9947fd1

add updated backend after review

560c7b9

more refinements to simplify and remove obsolete parts

cef47db

Merge branch 'main' of https://github.com/Azure/PyRIT into romanlutz/…

10bb4dd

…backend_apis

explicitly remove playwright-report

a06e9b8

explicitly remove last-run.json

d5679b3

use normalizer properly, clean up, refactor

7be9760

update with latest PR feedback, cleanup

27ca930

more simplifications

5fb3f2a

Merge branch 'main' of https://github.com/Azure/PyRIT into romanlutz/…

e83d94e

…backend_apis

update with target identifier

9b53976

Merge branch 'main' of https://github.com/Azure/PyRIT into romanlutz/…

a5f8882

…backend_apis

fix test, no default initializer

cf018b9

Merge branch 'main' into romanlutz/backend_apis

1800a8f

rlundeen2 self-assigned this Feb 6, 2026