diff --git a/docs/specs/004-python-function-calling-loop.md b/docs/specs/004-python-function-calling-loop.md index b7ad58ceb90..9f0d8a4fde6 100644 --- a/docs/specs/004-python-function-calling-loop.md +++ b/docs/specs/004-python-function-calling-loop.md @@ -758,6 +758,7 @@ that manually replay messages own the equivalent rule: do not resend an approval | Opaque reasoning signature replay | Provider-specific opaque reasoning metadata is captured and restored on reconstructed calls. | `packages/gemini/tests/test_gemini_client.py::test_function_call_part_captures_thought_signature_as_reasoning_content`, `test_reconstructed_function_call_replays_thought_signature_from_reasoning_content` | | Chat Completions approval wrappers | Framework approval wrappers are not sent as chat messages. | `packages/openai/tests/openai/test_openai_chat_completion_client.py` approval serialization tests | | DevUI conversation-bound approval ownership | A pending function approval is bound to the conversation that originated it. A response from another conversation is rejected without consuming the pending approval, and identical request IDs in different conversations remain independent. | `packages/devui/tests/devui/test_approval_validation.py::test_approval_request_must_be_bound_to_originating_conversation`, `test_same_approval_request_id_is_scoped_per_conversation` | +| AG-UI durable approval opt-in | Trusted scoped private authority survives cold restore; claim-write failure prevents execution, interrupted/unsaved execution cannot regain a grant, and results keep original call identity. Single-writer storage is required; replay-only history cannot establish authority. | `packages/ag-ui/tests/ag_ui/test_cold_approval_resume.py`, `test_private_snapshot_preserves_expiration_across_downtime`, `test_private_snapshot_restore_respects_scope_capacity` | | AG-UI approval result event | Approved result emits once with content and persists in snapshot. | `packages/ag-ui/tests/ag_ui/test_approval_result_event.py::test_approved_call_emits_one_live_result_under_original_identity`, `packages/ag-ui/tests/ag_ui/test_endpoint.py::test_endpoint_agent_approval_resume_persists_replayable_tool_results`, `test_endpoint_agent_approval_replayed_resume_entry_reprojects_retained_result` | | AG-UI rejection/mixed decision | Transport emits only the events defined for approved and rejected calls without duplicates. | `packages/ag-ui/tests/ag_ui/test_approval_result_event.py::test_rejected_call_does_not_execute_or_emit_live_result`, `test_mixed_batch_preserves_approved_result_identity_and_order`, `packages/ag-ui/tests/ag_ui/test_endpoint.py::test_endpoint_agent_approval_rejection_releases_already_approved_sibling` | | AG-UI approval-time follow-up | The full grouped user-input pause remains in message history and emits no synthetic `TOOL_CALL_RESULT`. | `packages/ag-ui/tests/ag_ui/test_approval_result_event.py::test_approval_follow_up_group_remains_in_history_without_live_tool_result` | diff --git a/python/packages/ag-ui/AGENTS.md b/python/packages/ag-ui/AGENTS.md index bd79ecda92a..4db4e36b0b1 100644 --- a/python/packages/ag-ui/AGENTS.md +++ b/python/packages/ag-ui/AGENTS.md @@ -93,6 +93,10 @@ AG-UI protocol integration for building agent UIs with the AG-UI standard. ids alone; an older request generation for the same occurrence is still invalid. - Approval responses for tools injected during `before_run` are deferred to the in-run approval middleware rather than executed or rejected by the transport before those tools exist. +- `AgentFrameworkAgent(durable_approvals=True)` retains private lifecycle authority in the scoped Thread Snapshot. + Restore only that private record, never client Shared State, replay messages, or interrupt projections. Commit claims + before side effects and propagate claim-write failures; cold unfinished claims fail closed as indeterminate. + This requires application-owned atomic durable storage and single-writer serialization, not cross-replica coordination. - `_approval_lifecycle.py` is the sole owner of approval occurrence registration, trusted aliases, authority validation, claims, terminal outcomes, and retry deduplication. Runner code normalizes AG-UI protocol values and projects lifecycle outcomes but must not maintain a parallel pending-approval registry. diff --git a/python/packages/ag-ui/README.md b/python/packages/ag-ui/README.md index 6b7769c9fc8..04e486b2bb6 100644 --- a/python/packages/ag-ui/README.md +++ b/python/packages/ag-ui/README.md @@ -433,6 +433,41 @@ For applications that need server-authoritative thread history, configure scoped mode only accepts user turns and results for backend-issued tool calls when extending stored history. It complements endpoint authentication and authorization; it does not replace them. +## Native approval recovery after restart + +Approval authority is process-local by default. A replayable interrupt is not sufficient to authorize a tool after +restart. Opt in on the agent wrapper when the application provides a trusted, scoped durable snapshot store: + +```python +from agent_framework.ag_ui import AgentFrameworkAgent, add_agent_framework_fastapi_endpoint + +runner = AgentFrameworkAgent(agent, durable_approvals=True) +add_agent_framework_fastapi_endpoint( + app, + runner, + snapshot_store=durable_store, + authorization_scope_resolver=resolve_authenticated_scope, +) +``` + +The store must retain the complete snapshot, including private `session_state`, and atomically commit writes before +returning. Its scope must identify the same authorized owner after restart. Serialize all mutations for a thread +across its entire run; this option supports a single writer and does not implement locks or compare-and-swap across +replicas. `InMemoryAGUIThreadSnapshotStore` does not provide process-restart durability. + +New snapshots retain private server-issued approval occurrences, their original function and provider call identities, +arguments, middleware continuation, and terminal results. This authority is never reconstructed from client messages, +Shared State, or the visible interrupt. Old snapshots without the private authority record remain non-resumable. +Native SDK-only histories use `AgentSession.to_dict()` / `AgentSession.from_dict()` separately; importing their visible +approval controls does not create AG-UI approval authority. + +Before an approved side effect starts, the runner commits its claim. A failed claim write prevents execution. If a +process stops after the claim or the terminal result cannot be saved, cold recovery treats that claim as indeterminate +and refuses to execute it again. Applications must reconcile uncertain side effects outside this retry path; this is +not a guarantee of completing a tool exactly once across arbitrary crashes. Completed persisted approvals retain their +canonical result, and retries do not rerun the tool during the configured lifecycle retention window. Retention ages +include downtime. Default pending retention is 24 hours and terminal replay retention is 15 minutes. + ## AG-UI Thread Snapshots AG-UI Thread Snapshot persistence is opt-in and disabled by default. Existing endpoints keep their current behavior diff --git a/python/packages/ag-ui/agent_framework_ag_ui/_agent.py b/python/packages/ag-ui/agent_framework_ag_ui/_agent.py index ab179da165c..d06b5da9b41 100644 --- a/python/packages/ag-ui/agent_framework_ag_ui/_agent.py +++ b/python/packages/ag-ui/agent_framework_ag_ui/_agent.py @@ -30,6 +30,7 @@ def __init__( service_session_id_from_thread_id: bool = False, emit_messages_snapshot: bool = True, legacy_session_id_from_thread_id: bool = False, + durable_approvals: bool = False, ): """Initialize agent configuration. @@ -51,6 +52,9 @@ def __init__( emit_messages_snapshot: Whether to emit a terminal MessagesSnapshotEvent at the end of runs. Defaults to True for backward compatibility. Set to False when using HistoryProvider to prevent redundant full-transcript rewrites on the client. + durable_approvals: Persist private approval authority in a trusted scoped snapshot store. + Requires durable atomic writes and a single writer per thread. Incomplete claims after + restart fail closed; this does not coordinate concurrent replicas. legacy_session_id_from_thread_id: Deprecated compatibility option that uses the client-owned AG-UI Thread id directly as the internal Agent Session id. """ @@ -60,6 +64,7 @@ def __init__( self.service_session_id_from_thread_id = service_session_id_from_thread_id self.require_confirmation = require_confirmation self.snapshot_store = snapshot_store + self.durable_approvals = durable_approvals self.a2ui_config = a2ui_config self.emit_messages_snapshot = emit_messages_snapshot self.legacy_session_id_from_thread_id = legacy_session_id_from_thread_id @@ -113,6 +118,7 @@ def __init__( service_session_id_from_thread_id: bool = False, emit_messages_snapshot: bool = True, legacy_session_id_from_thread_id: bool = False, + durable_approvals: bool = False, ): """Initialize the AG-UI compatible agent wrapper. @@ -134,6 +140,9 @@ def __init__( a2ui_config: Optional backend A2UI config consumed by auto-injection. emit_messages_snapshot: Whether to emit a terminal MessagesSnapshotEvent at the end of runs. Defaults to True. Set to False when using HistoryProvider. + durable_approvals: Persist private approval authority in a trusted scoped snapshot store. + Requires durable atomic writes and a single writer per thread. Incomplete claims after + restart fail closed; this does not coordinate concurrent replicas. legacy_session_id_from_thread_id: Deprecated compatibility option that uses the client-owned AG-UI Thread id directly as the internal Agent Session id. This disables Snapshot Scope isolation for context-provider state. @@ -161,6 +170,7 @@ def __init__( a2ui_config=a2ui_config, emit_messages_snapshot=emit_messages_snapshot, legacy_session_id_from_thread_id=legacy_session_id_from_thread_id, + durable_approvals=durable_approvals, ) # Server-side Approval State. Populated when approval requests are emitted diff --git a/python/packages/ag-ui/agent_framework_ag_ui/_agent_run.py b/python/packages/ag-ui/agent_framework_ag_ui/_agent_run.py index db662fe90e8..f9090b0ed8d 100644 --- a/python/packages/ag-ui/agent_framework_ag_ui/_agent_run.py +++ b/python/packages/ag-ui/agent_framework_ag_ui/_agent_run.py @@ -89,7 +89,12 @@ InRunApprovalExecution, InRunPendingToolTransitionOwner, ) -from ._approval_state import _APPROVAL_SCOPE_INPUT_KEY, InMemoryAGUIApprovalStateStore, approval_state_thread_id +from ._approval_state import ( + _APPROVAL_SCOPE_INPUT_KEY, + _APPROVAL_SNAPSHOT_STATE_KEY, + InMemoryAGUIApprovalStateStore, + approval_state_thread_id, +) from ._message_adapters import _APPROVAL_DECISION_IS_BOOLEAN_KEY, normalize_agui_input_messages from ._predictive_state import PredictiveStateHandler from ._tooling import collect_server_tools, merge_tools @@ -2575,6 +2580,7 @@ def _restore_session_continuation_state( if snapshot is None or snapshot.session_state is None: return serialized_state = copy.deepcopy(snapshot.session_state) + serialized_state.pop(_APPROVAL_SNAPSHOT_STATE_KEY, None) service_session_id = serialized_state.pop(_PROVIDER_SERVICE_SESSION_ID_STATE_KEY, None) for key in excluded_state_keys: serialized_state.pop(key, None) @@ -2641,6 +2647,7 @@ def _request_state_protected_keys(agent: SupportsAgentRun) -> set[str]: agent_middleware = categorize_middleware(getattr(agent, "middleware", None))["agent"] return { _TOOL_APPROVAL_STATE_KEY, + _APPROVAL_SNAPSHOT_STATE_KEY, InMemoryHistoryProvider.DEFAULT_SOURCE_ID, MESSAGE_INJECTION_PENDING_MESSAGES_STATE_KEY, *(provider.source_id for provider in context_providers), @@ -2730,6 +2737,7 @@ def _serialize_session_continuation_state( excluded_keys = { *shared_state_keys, _TOOL_APPROVAL_STATE_KEY, + _APPROVAL_SNAPSHOT_STATE_KEY, _PROVIDER_SERVICE_SESSION_ID_STATE_KEY, *(provider.source_id for provider in context_providers if isinstance(provider, HistoryProvider)), } @@ -3004,6 +3012,12 @@ async def _run_agent_stream( ) stored_snapshot = snapshot_session.stored + if config.durable_approvals and not snapshot_session.enabled: + raise ValueError("durable_approvals requires a scoped snapshot store.") + if config.durable_approvals and stored_snapshot is not None and stored_snapshot.session_state is not None: + authority = stored_snapshot.session_state.get(_APPROVAL_SNAPSHOT_STATE_KEY) + if authority is not None: + approval_state_store.restore_thread(approval_thread_id, authority) stored_pending_approval_interrupt_ids: set[str] = set() seeded_resume_from_snapshot = False if stored_snapshot is not None: @@ -3342,6 +3356,18 @@ async def _run_agent_stream( _restore_tool_approval_state(session, approval_state_store, approval_thread_id) approval_middleware_pipeline = _approval_observer_middleware_pipeline(agent, session) + def private_snapshot_state() -> dict[str, Any] | None: + continuation = _safe_serialize_session_continuation_state( + session, + agent, + shared_state_keys=set(flow.current_state).difference(protected_session_state_keys), + include_service_session_id=config.use_service_session, + ) + if config.durable_approvals: + continuation = continuation or {} + continuation[_APPROVAL_SNAPSHOT_STATE_KEY] = approval_state_store.export_thread(approval_thread_id) + return continuation + async def save_thread_snapshot( *, persisted_messages: list[dict[str, Any]], @@ -3353,12 +3379,7 @@ async def save_thread_snapshot( messages=_bound_host_payload_history(_persistable_host_payload_history(persisted_messages)), state=state, interrupt=interrupt, - session_state=_safe_serialize_session_continuation_state( - session, - agent, - shared_state_keys=set(flow.current_state).difference(protected_session_state_keys), - include_service_session_id=config.use_service_session, - ), + session_state=private_snapshot_state(), ) authenticated_cancellations = [ @@ -3393,12 +3414,7 @@ async def save_thread_snapshot( messages=stored_after_cancellation.messages, state=stored_after_cancellation.state, interrupt=stored_after_cancellation.interrupt, - session_state=_safe_serialize_session_continuation_state( - session, - agent, - shared_state_keys=set(flow.current_state).difference(protected_session_state_keys), - include_service_session_id=config.use_service_session, - ), + session_state=private_snapshot_state(), ) yield _build_run_finished_event(run_id=run_id, thread_id=thread_id) return @@ -3533,6 +3549,11 @@ async def save_thread_snapshot( code="APPROVAL_INVOCATION_DISABLED", ) return + if authorized_executions and config.durable_approvals: + # Commit the claim before any side effect. A crash or failed terminal save + # leaves an indeterminate claim on cold restore, never a fresh grant. + _save_tool_approval_state(session, approval_state_store, approval_thread_id) + await snapshot_session.save_approval_claim(session_state=private_snapshot_state() or {}) validated_approved_responses: list[Content] = [] replacement_approval_requests: list[Content] = [] newly_resolved_approval_results = await _resolve_approval_responses( diff --git a/python/packages/ag-ui/agent_framework_ag_ui/_approval_lifecycle.py b/python/packages/ag-ui/agent_framework_ag_ui/_approval_lifecycle.py index c380acc4a9d..2d28141efe1 100644 --- a/python/packages/ag-ui/agent_framework_ag_ui/_approval_lifecycle.py +++ b/python/packages/ag-ui/agent_framework_ag_ui/_approval_lifecycle.py @@ -8,11 +8,11 @@ from asyncio import CancelledError from collections.abc import Awaitable, Callable, Iterator from contextlib import ExitStack -from dataclasses import dataclass, field +from dataclasses import asdict, dataclass, field from enum import Enum from functools import wraps from threading import RLock -from time import monotonic +from time import monotonic, time from typing import Any, TypeVar from uuid import uuid4 @@ -441,6 +441,111 @@ def occurrences_for_thread(self, *, thread_id: str) -> tuple[ApprovalOccurrence, self._purge_expired_terminal() return tuple(occurrence for occurrence in self._occurrences.values() if thread_id in occurrence.thread_ids) + def export_thread(self, *, thread_id: str) -> dict[str, Any]: + """Serialize private authority, never the client-visible interrupt projection.""" + with self._index_lock: + now = self._clock() + records: list[dict[str, Any]] = [] + for occurrence in self.occurrences_for_thread(thread_id=thread_id): + record = { + key: getattr(occurrence, key) + for key in ( + "thread_ids", + "name", + "arguments", + "function_call_id", + "active_interrupt_id", + "scope", + "aliases", + "response_id", + "already_approved_requests", + "server_label", + "idempotency_key", + "requires_client_resume", + ) + } + record.update( + identity=asdict(occurrence.identity), + owner=occurrence.owner.value, + status=occurrence.status.value, + pending_age=max(0, now - occurrence.pending_since), + terminal_age=None if occurrence.terminal_at is None else max(0, now - occurrence.terminal_at), + decision=asdict(occurrence.decision) if occurrence.decision else None, + results=[result.content.to_dict() for result in occurrence.replayable_results], + result_group=( + [content.to_dict() for content in occurrence.outcome.result_group] + if occurrence.outcome is not None + else None + ), + ) + records.append(record) + return {"version": 1, "saved_at": time(), "thread_id": thread_id, "occurrences": records} + + def restore_thread(self, *, thread_id: str, payload: dict[str, Any]) -> None: + """Restore trusted storage once; uncertain claims never regain execution authority. + + This is restart recovery for a single writer, not coordination between replicas. + Live in-process lifecycle state always takes precedence over an older snapshot. + """ + with self._index_lock: + if self.occurrences_for_thread(thread_id=thread_id): + return + if payload["version"] != 1 or payload["thread_id"] != thread_id: + raise ValueError("Stored approval authority has an unsupported version or scope.") + elapsed = max(0, time() - payload["saved_at"]) + now = self._clock() + restored: list[ApprovalOccurrence] = [] + for saved in payload["occurrences"]: + record = dict(saved) + identity = ApprovalOccurrenceIdentity(**record.pop("identity")) + if identity.thread_id != thread_id or tuple(record["thread_ids"]) != (thread_id,): + raise ValueError("Stored approval occurrence belongs to another scope.") + owner = ApprovalExecutionOwner(record.pop("owner")) + status = ApprovalStatus(record.pop("status")) + pending_since = now - record.pop("pending_age") - elapsed + terminal_age = record.pop("terminal_age") + terminal_at = None if terminal_age is None else now - terminal_age - elapsed + decision = record.pop("decision") + results = [ReplayableToolResult(Content.from_dict(content)) for content in record.pop("results")] + result_group = record.pop("result_group") + if status in {ApprovalStatus.CLAIMED, ApprovalStatus.EXECUTING}: + status = ApprovalStatus.INDETERMINATE + terminal_at = now + occurrence = ApprovalOccurrence( + identity=identity, + owner=owner, + status=status, + pending_since=pending_since, + terminal_at=terminal_at, + decision=ResumeDecision(**decision) if decision else None, + replayable_results=results, + **{ + **record, + "thread_ids": tuple(record["thread_ids"]), + "aliases": tuple(record["aliases"]), + "already_approved_requests": tuple(record["already_approved_requests"]), + }, + ) + if result_group is not None: + occurrence.outcome = ApprovalOutcome( + identity=identity, + replayable_results=tuple(results), + result_group=tuple(Content.from_dict(content) for content in result_group), + snapshot_reconciliation=self._snapshot_reconciliation(occurrence), + ) + restored.append(occurrence) + for scope in {occurrence.scope for occurrence in restored}: + count = sum(occurrence.scope == scope for occurrence in (*self._occurrences.values(), *restored)) + if count > self._max_entries: + raise ApprovalCapacityError("Stored approval authority exceeds capacity.") + for occurrence in restored: + self._occurrences[occurrence.identity] = occurrence + self._locks_by_identity[occurrence.identity] = RLock() + index = self._terminal_by_interrupt if occurrence.status.is_terminal else self._pending_by_interrupt + for alias in occurrence.aliases: + index[(thread_id, alias)] = occurrence.identity + self._purge_expired_terminal() + def pending_interrupt_ids(self, *, thread_id: str) -> set[str]: """Return canonical interrupt identities with pending authority for one thread.""" with self._index_lock: diff --git a/python/packages/ag-ui/agent_framework_ag_ui/_approval_state.py b/python/packages/ag-ui/agent_framework_ag_ui/_approval_state.py index 3b3253cbb6f..e3fb5f32009 100644 --- a/python/packages/ag-ui/agent_framework_ag_ui/_approval_state.py +++ b/python/packages/ag-ui/agent_framework_ag_ui/_approval_state.py @@ -17,6 +17,7 @@ DEFAULT_PENDING_RETENTION_SECONDS = 86_400 DEFAULT_INDETERMINATE_RETENTION_SECONDS = 604_800 DEFAULT_TERMINAL_RETENTION_SECONDS = 900 +_APPROVAL_SNAPSHOT_STATE_KEY = "__ag_ui_approval_authority" _APPROVAL_SCOPE_INPUT_KEY = "__ag_ui_approval_scope" @@ -131,3 +132,18 @@ def has_tool_approval_state(self, thread_id: str) -> bool: """Return whether middleware approval state exists for one scoped thread.""" with self._lock: return thread_id in self._tool_approval_states + + def export_thread(self, thread_id: str) -> dict[str, Any]: + """Export server-owned lifecycle and middleware state for trusted snapshot storage.""" + return { + "lifecycle": self.lifecycle.export_thread(thread_id=thread_id), + "middleware": self.get_tool_approval_state(thread_id), + } + + def restore_thread(self, thread_id: str, payload: dict[str, Any]) -> None: + """Restore private authority without replacing live in-process state.""" + if self.lifecycle.occurrences_for_thread(thread_id=thread_id): + return + self.lifecycle.restore_thread(thread_id=thread_id, payload=payload["lifecycle"]) + if payload.get("middleware") is not None: + self.set_tool_approval_state(thread_id, payload["middleware"]) diff --git a/python/packages/ag-ui/agent_framework_ag_ui/_snapshot_session.py b/python/packages/ag-ui/agent_framework_ag_ui/_snapshot_session.py index 85546830245..d1e1f22920a 100644 --- a/python/packages/ag-ui/agent_framework_ag_ui/_snapshot_session.py +++ b/python/packages/ag-ui/agent_framework_ag_ui/_snapshot_session.py @@ -194,6 +194,15 @@ async def save( self._thread_id, ) + async def save_approval_claim(self, *, session_state: dict[str, Any]) -> None: + """Persist authority before execution; unlike replay saves, failures are fatal.""" + if self._store is None or self._scope is None: + return + snapshot = copy.deepcopy(self._stored) if self._stored is not None else AGUIThreadSnapshot(messages=[]) + snapshot.session_state = session_state + await self._store.save(scope=self._scope, thread_id=self._thread_id, snapshot=snapshot) + self._stored = snapshot + async def clear_interrupts(self, *, interrupt_ids: set[str] | None = None) -> None: """Remove completed interrupts from the latest stored snapshot. diff --git a/python/packages/ag-ui/tests/ag_ui/cold_approval_worker.py b/python/packages/ag-ui/tests/ag_ui/cold_approval_worker.py new file mode 100644 index 00000000000..3eacd735509 --- /dev/null +++ b/python/packages/ag-ui/tests/ag_ui/cold_approval_worker.py @@ -0,0 +1,121 @@ +# Copyright (c) Microsoft. All rights reserved. + +"""One process in the disk-backed approval lifecycle regression.""" + +from __future__ import annotations + +import asyncio +import json +import os +import sys +from dataclasses import asdict +from pathlib import Path +from typing import Any + +from agent_framework import Agent, AgentSession, ChatResponseUpdate, Content, FunctionTool, Message +from conftest import StreamingChatClientStub +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from agent_framework_ag_ui import AgentFrameworkAgent, add_agent_framework_fastapi_endpoint +from agent_framework_ag_ui._snapshots import AGUIThreadSnapshot + + +def main(root: Path, action: str) -> None: + class DiskSnapshots: + saves = 0 + + async def get(self, *, scope: str, thread_id: str) -> AGUIThreadSnapshot | None: + path = root / "snapshot.json" + return AGUIThreadSnapshot(**json.loads(path.read_text())) if path.exists() else None + + async def save(self, *, scope: str, thread_id: str, snapshot: AGUIThreadSnapshot) -> None: + self.saves += 1 + if action == "fail-claim" or (action == "fail-result" and self.saves > 1): + raise OSError("Storage unavailable") + temporary = root / "snapshot.tmp" + temporary.write_text(json.dumps(asdict(snapshot))) + temporary.replace(root / "snapshot.json") + + async def delete(self, **kwargs: Any) -> bool: + raise AssertionError("No deletion expected") + + async def clear(self, **kwargs: Any) -> None: + raise AssertionError("No clearing expected") + + def get_weather(city: str) -> str: + with (root / "executions.txt").open("a") as file: + file.write(city + "\n") + if action == "crash-approve": + print("[]", flush=True) + os._exit(0) + return json.dumps({"status": "pending_approval", "city": city}, separators=(",", ":")) + + async def stream(messages: Any, options: Any, **kwargs: Any): + if action in {"pause", "sdk-pause"}: + yield ChatResponseUpdate( + contents=[ + Content.from_function_call( + id="native-occurrence", + call_id="provider-call", + name="get_weather", + arguments={"city": "Seattle"}, + ) + ] + ) + else: + yield ChatResponseUpdate(contents=[Content.from_text("Done.")]) + + agent = Agent( + client=StreamingChatClientStub(stream), + tools=[ + FunctionTool(name="get_weather", description="Weather", func=get_weather, approval_mode="always_require") + ], + ) + if action.startswith("sdk-"): + + async def run_sdk() -> None: + path = root / "session.json" + session = AgentSession.from_dict(json.loads(path.read_text())) if path.exists() else agent.create_session() + if action == "sdk-pause": + incoming = "Weather?" + else: + request_content = Content.from_dict(json.loads((root / "request.json").read_text())) + incoming = [ + Message( + role="user", + contents=[request_content.to_function_approval_response(approved=action == "sdk-approve")], + ) + ] + response = await agent.run(incoming, session=session, stream=True).get_final_response() + contents = [content for message in response.messages for content in message.contents] + if action == "sdk-pause": + approval = next(content for content in contents if content.type == "function_approval_request") + (root / "request.json").write_text(json.dumps(approval.to_dict())) + path.write_text(json.dumps(session.to_dict())) + print(json.dumps([content.to_dict() for content in contents])) + + asyncio.run(run_sdk()) + return + app = FastAPI() + add_agent_framework_fastapi_endpoint( + app, + AgentFrameworkAgent(agent, durable_approvals=True), + path="/agent", + snapshot_store=DiskSnapshots(), + snapshot_scope_resolver=lambda request: "tenant-b" if action == "other-scope" else "tenant-a", + ) + request: dict[str, Any] = {"threadId": "thread-a", "runId": action, "messages": []} + if action == "pause": + request["messages"] = [{"id": "question", "role": "user", "content": "Weather?"}] + elif action != "hydrate": + request["resume"] = [ + {"interruptId": "native-occurrence", "status": "resolved", "payload": {"approved": action != "decline"}} + ] + response = TestClient(app).post("/agent", json=request) + response.raise_for_status() + print(json.dumps([json.loads(line[6:]) for line in response.text.splitlines() if line.startswith("data: ")])) + + +if __name__ == "__main__": + main(Path(sys.argv[1]), sys.argv[2]) diff --git a/python/packages/ag-ui/tests/ag_ui/test_approval_lifecycle.py b/python/packages/ag-ui/tests/ag_ui/test_approval_lifecycle.py index e58d786a7e2..2d2ca2b75d8 100644 --- a/python/packages/ag-ui/tests/ag_ui/test_approval_lifecycle.py +++ b/python/packages/ag-ui/tests/ag_ui/test_approval_lifecycle.py @@ -1331,3 +1331,52 @@ def test_expired_authority_cannot_be_claimed() -> None: ) assert lifecycle.get(occurrence.identity).status is ApprovalStatus.EXPIRED + + +def test_private_snapshot_preserves_expiration_across_downtime(monkeypatch: pytest.MonkeyPatch) -> None: + import agent_framework_ag_ui._approval_lifecycle as module + + wall_time = [1000.0] + monkeypatch.setattr(module, "time", lambda: wall_time[0]) + lifecycle = ApprovalLifecycle(pending_retention_seconds=10, clock=lambda: 100.0) + lifecycle.register( + owner=ApprovalExecutionOwner.LOCAL, + thread_id="thread", + interrupt_id="approval", + call_id="provider", + name="tool", + arguments="{}", + ) + saved = lifecycle.export_thread(thread_id="thread") + wall_time[0] += 11 + restarted = ApprovalLifecycle(pending_retention_seconds=10, clock=lambda: 900.0) + restarted.restore_thread(thread_id="thread", payload=saved) + assert restarted.pending_interrupt_ids(thread_id="thread") == set() + assert restarted.occurrence_for_alias(thread_id="thread", interrupt_id="approval").status is ApprovalStatus.EXPIRED + + +def test_private_snapshot_restore_respects_scope_capacity() -> None: + source = ApprovalLifecycle() + source.register( + owner=ApprovalExecutionOwner.LOCAL, + thread_id="second", + interrupt_id="two", + call_id="provider-two", + name="tool", + arguments="{}", + scope="tenant", + ) + target = ApprovalLifecycle(max_entries=1) + target.register( + owner=ApprovalExecutionOwner.LOCAL, + thread_id="first", + interrupt_id="one", + call_id="provider-one", + name="tool", + arguments="{}", + scope="tenant", + ) + with pytest.raises(ApprovalCapacityError): + target.restore_thread(thread_id="second", payload=source.export_thread(thread_id="second")) + assert target.pending_interrupt_ids(thread_id="first") == {"one"} + assert target.pending_interrupt_ids(thread_id="second") == set() diff --git a/python/packages/ag-ui/tests/ag_ui/test_cold_approval_resume.py b/python/packages/ag-ui/tests/ag_ui/test_cold_approval_resume.py new file mode 100644 index 00000000000..b0dc4ddae89 --- /dev/null +++ b/python/packages/ag-ui/tests/ag_ui/test_cold_approval_resume.py @@ -0,0 +1,128 @@ +# Copyright (c) Microsoft. All rights reserved. + +"""Native approvals survive independent endpoint processes with a JSON snapshot store.""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +from pathlib import Path + +import pytest + + +def _process(root: Path, action: str) -> list[dict]: + result = subprocess.run( + [sys.executable, str(Path(__file__).with_name("cold_approval_worker.py")), str(root), action], + capture_output=True, + text=True, + check=True, + env=os.environ.copy(), + ) + return json.loads(result.stdout) + + +@pytest.mark.parametrize("decision", ["approve", "decline"]) +def test_native_approval_cold_resume(tmp_path: Path, decision: str) -> None: + paused = _process(tmp_path, "pause") + interrupt = paused[-1]["outcome"]["interrupts"][0] + assert interrupt["id"] == "native-occurrence" + assert "__ag_ui_approval_authority" not in json.dumps(paused) + resumed = _process(tmp_path, decision) + assert not [event for event in resumed if event["type"] == "RUN_ERROR"] + expected = ( + '{"status":"pending_approval","city":"Seattle"}' + if decision == "approve" + else "Error: Tool call invocation was rejected by user." + ) + snapshot = json.loads((tmp_path / "snapshot.json").read_text()) + results = [message for message in snapshot["messages"] if message.get("tool_call_id") == "provider-call"] + assert [message["content"] for message in results] == [expected] + assert not snapshot["interrupt"] + live_results = [event for event in resumed if event["type"] == "TOOL_CALL_RESULT"] + if decision == "approve": + assert [(event["toolCallId"], event["content"]) for event in live_results] == [("provider-call", expected)] + assert resumed.index(live_results[0]) < next( + index for index, event in enumerate(resumed) if event["type"] == "TEXT_MESSAGE_CONTENT" + ) + retry = _process(tmp_path, decision) + assert not [event for event in retry if event["type"] == "RUN_ERROR"] + hydrated = _process(tmp_path, "hydrate") + assert "outcome" not in hydrated[-1] + replay = next(event["messages"] for event in hydrated if event["type"] == "MESSAGES_SNAPSHOT") + assert [ + (message["toolCallId"], message["content"]) + for message in replay + if message.get("toolCallId") == "provider-call" + ] == [("provider-call", expected)] + execution_file = tmp_path / "executions.txt" + assert (execution_file.read_text() if execution_file.exists() else "") == ( + "Seattle\n" if decision == "approve" else "" + ) + + +def test_cold_resume_claim_write_failure_executes_nothing(tmp_path: Path) -> None: + _process(tmp_path, "pause") + failed = _process(tmp_path, "fail-claim") + assert [event for event in failed if event["type"] == "RUN_ERROR"] + assert not (tmp_path / "executions.txt").exists() + resumed = _process(tmp_path, "approve") + assert not [event for event in resumed if event["type"] == "RUN_ERROR"] + assert (tmp_path / "executions.txt").read_text() == "Seattle\n" + + +@pytest.mark.parametrize("action", ["crash-approve", "fail-result"]) +def test_cold_resume_uncertain_execution_cannot_run_twice(tmp_path: Path, action: str) -> None: + _process(tmp_path, "pause") + _process(tmp_path, action) + assert (tmp_path / "executions.txt").read_text() == "Seattle\n" + retry = _process(tmp_path, "approve") + assert [event for event in retry if event["type"] == "RUN_ERROR"] + assert (tmp_path / "executions.txt").read_text() == "Seattle\n" + + +@pytest.mark.parametrize("action", ["approve", "other-scope"]) +def test_replay_data_cannot_recreate_authority(tmp_path: Path, action: str) -> None: + _process(tmp_path, "pause") + if action == "approve": + path = tmp_path / "snapshot.json" + snapshot = json.loads(path.read_text()) + private = snapshot.pop("session_state") + snapshot["session_state"] = None + snapshot["state"] = private # Client/shared state is never an authority source. + path.write_text(json.dumps(snapshot)) + response = _process(tmp_path, action) + assert [event for event in response if event["type"] == "RUN_ERROR"] + assert not (tmp_path / "executions.txt").exists() + + +@pytest.mark.parametrize("decision", ["approve", "decline"]) +def test_native_sdk_session_cold_resume(tmp_path: Path, decision: str) -> None: + """SDK AgentSession persistence is a separate path from AG-UI snapshot authority.""" + _process(tmp_path, "sdk-pause") + response = _process(tmp_path, "sdk-" + decision) + results = [content for content in response if content["type"] == "function_result"] + expected = ( + '{"status":"pending_approval","city":"Seattle"}' + if decision == "approve" + else "Error: Tool call invocation was rejected by user." + ) + assert [(content["call_id"], content["result"]) for content in results] == [("provider-call", expected)] + before = json.loads((tmp_path / "session.json").read_text()) + _process(tmp_path, "sdk-" + decision) + after = json.loads((tmp_path / "session.json").read_text()) + assert before["session_id"] == after["session_id"] + for saved in (before, after): + persisted = [ + content + for message in saved["state"]["in_memory"]["messages"] + for content in message["contents"] + if content["type"] == "function_result" + ] + assert [(content["call_id"], content["result"]) for content in persisted] == [("provider-call", expected)] + execution_file = tmp_path / "executions.txt" + assert (execution_file.read_text() if execution_file.exists() else "") == ( + "Seattle\n" if decision == "approve" else "" + )