diff --git a/docs/specs/004-python-function-calling-loop.md b/docs/specs/004-python-function-calling-loop.md index b35a4fe23d9..c957bd9b86c 100644 --- a/docs/specs/004-python-function-calling-loop.md +++ b/docs/specs/004-python-function-calling-loop.md @@ -18,12 +18,14 @@ It covers: - approved, rejected, mixed, and replayed approval rounds; - reasoning content and opaque reasoning signatures bound to function calls; - history persistence and service-side continuation; +- session-backed recovery of local tool results during Responses background operations; - error, user-input, middleware-termination, middleware-failure, and loop-limit paths; - provider and transport serialization of function calls and results. The primary implementation is in `python/packages/core/agent_framework/_tools.py`. History replay behavior in `python/packages/core/agent_framework/_sessions.py`, provider serializers, hosting packages, and UI transports are -part of the same contract when they carry function-call loop content. +part of the same contract when they carry function-call loop content. Background execution and result-delivery +evidence is managed by `python/packages/core/agent_framework/_tool_recovery.py`. ## Maintenance policy @@ -594,6 +596,63 @@ that manually replay messages own the equivalent rule: do not resend an approval final response transcript, each positioned before the group it replaces, so history loaded with `skip_excluded` keeps the summarized content; summaries of caller-owned input messages stay out of the returned transcript. +### Background tool-result recovery + +Recovery retries result delivery, not completed tool execution. It applies to automatic local tools in OpenAI +Responses background operations and continuation-token polls, including Azure OpenAI and Foundry adapters, not ordinary +foreground calls or other protocols. The caller must retain the authoritative `AgentSession` after failures. +Sessionless and ephemeral-session calls remain supported without cross-poll recovery or replay protection. + +- Claim each occurrence by stable `Content.id` in `executing` before middleware or tool invocation, then save results + independently as calls complete. Missing response identities or missing/repeated occurrence identities fail before invocation. + A failed or cancelled sibling must not erase completed results. +- A ready checkpoint requires results for every claimed occurrence. Session serialization must preserve the prepared + messages, caller-visible transcript, original input, identities, reasoning, usage, error count, consumed budgets, + tool choice, continuation mode, and effective tool registry independently of successful-run history. +- Retry requires the original polling token and unchanged caller input. Use checkpointed messages, not regenerated + provider context, and submit results without retrieving the completed response or rerunning middleware/tools. + Preserve conversation-object versus response-ID continuation without conflicting aliases, and do not reset consumed + limits or elapsed time. Retrieval failure before any execution claim does not make tool execution indeterminate. +- Evidence is scoped to provider, endpoint, model, organization, and project; Foundry also includes effective agent + identity (name/version or an applicable `agent_reference` override). Pending or handled evidence cannot cross scopes; + empty state may. Matching token-only polls retain the originating model when no override is supplied. +- Restore the checkpointed tool registry in order, including additions/removals, using matching trusted implementations + supplied on retry. Do not reconstruct executable tools from stored data. Missing or changed definitions reject + delivery but leave the ready checkpoint available for a corrected retry. +- Checkpoint completed nonterminal streaming batches before yielding tool results, so closing before dispatch leaves + delivery ready. Both modes preserve the same ordered reasoning/call/result transcript and cumulative usage; + successful recovery persists one pair per occurrence. Streaming may repeat previously exposed updates. +- Invalid, unsupported-version, incomplete, or indeterminate evidence fails before replay; uncertain execution or + provider acceptance requires reconciliation. Ordinary tool-error results keep their existing contract. +- After pending delivery clears, retain consumed polling/response identities and reject stale tokens. A full + handled-response ledger refuses further execution rather than evicting replay protection. +- `_background_tool_recovery` is provider-owned: preserve inherited ownership declarations and exclude it from delegated + `Agent.as_tool(propagate_session=True)` state in both directions while propagating application state. AG-UI retains + it privately in both history modes and rejects request Shared State replacement, including through opaque wrappers. + +Completing a tool does not prove provider acceptance. Delivery follows these rules: + +| State | Event | Required outcome | +|---|---|---| +| `executing` | All claimed results and delivery context are checkpointed | Enter `ready`. | +| `ready` | Retry restoration, local preparation failure, or stream close before dispatch | Stay `ready`; retained results remain reusable. | +| `ready` | Prepared result request is dispatched | Enter `submitting` immediately before the provider call, after local validation/preparation. | +| `submitting` | Successful provider response | Clear pending delivery; retain consumed identities. | +| `submitting` | HTTP 429/503 before any submission-stream update | Return to `ready`. | +| `submitting` | Other failure/cancellation, or failure after any submission-stream update | Enter `indeterminate`; do not retry automatically. | +| `executing` | Exception/cancellation escapes before a complete checkpoint | Enter `indeterminate`; retain completed siblings. | +| Any pending state | Normal completion, including an early terminal return | Clear pending state and retain consumed identities; this alone does not prove provider acceptance. | + +Concurrent polls against one live session must be rejected, with ownership released on return, error, cancellation, +or stream close. Hosts must authorize and tenant-scope storage, protect retained results, persist session mutations +after failures, and serialize ownership across restored workers. This is not a distributed lock or transaction: +crashes between side effects and durable storage still require tool idempotency or reconciliation. Recovery promises +neither exactly-once provider processing nor exactly-once external side effects across crashes or lost/shared state. + +Existing approval, Host-pause, fatal-middleware, and provider-invalidation rules still apply. Execution during initial +approval/Host resolution, before a new provider response, is outside this delivery-recovery contract (#8574). +Recovery must not manufacture approval authority. + ## Scenario-to-test matrix ### Normal function invocation @@ -769,13 +828,45 @@ that manually replay messages own the equivalent rule: do not resend an approval | AG-UI malformed `confirm_changes` metadata | Non-list tool-call metadata and malformed argument JSON are ignored without guessing a target call. | `test_confirm_changes_target_ignores_non_list_tool_calls`, `test_confirm_changes_target_rejects_malformed_arguments_json` | | Compaction pair integrity | Adjacent and non-adjacent pairs, including assistant-embedded results and completed reused-id occurrences, remain atomic without pairing ambiguous or out-of-order ids. | `packages/core/tests/core/test_compaction.py::test_group_annotations_keep_tool_call_and_tool_result_atomic`, `test_group_annotations_include_reasoning_in_tool_call_group`, `test_group_annotations_pair_nonadjacent_function_result_by_call_id`, `test_group_annotations_pair_multiple_nonadjacent_results_with_declaration`, `test_group_annotations_pair_completed_reused_call_id_occurrences`, `test_group_annotations_close_assistant_embedded_result_before_reused_call_id`, `test_sliding_window_does_not_retain_orphan_result_after_assistant_embedded_result`, `test_sliding_window_keeps_reused_call_id_occurrences_atomic`, `test_group_annotations_do_not_pair_ambiguous_duplicate_call_ids` | +### Background tool-result recovery + +The following references are relative to `python/`. "All consumption modes" means non-streaming, explicit streaming +iteration, and streaming `get_final_response()`-only consumption. Other rows state their response-mode coverage; +a non-streaming test is not evidence for the corresponding streaming path. + +| Scenario | Required invariant | Primary regression test | +|---|---|---| +| Retrieval failure before local execution | With `failure_stage="retrieve"`, HTTP 429/503 executes no tool; a later successful poll may retrieve and execute once. Covered in both response modes, with live and JSON-restored sessions. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_poll_retry_does_not_repeat_completed_tools` | +| Rejected result submission and repeated delivery failure | With `failure_stage="submit"`, HTTP 429/503 before submission output permits retry of identical saved results without tool re-execution, including persistent failure. Successful recovery preserves usage, occurrence pairing, history, and stale-token rejection. Covered in both modes, with live and JSON-restored sessions. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_poll_retry_does_not_repeat_completed_tools` | +| Pending input, scope, or checkpoint mismatch | New input, changed model/endpoint, unsupported state version, malformed budget, missing result evidence, changed tool schema, or busy state fails before another provider request or tool execution. All consumption modes. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_recovery_rejects_unsafe_retries` | +| Ambiguous or cancelled submission | The `ambiguous-delivery` and `cancelled-submission` cases leave evidence non-retryable after JSON session restoration; completed tools do not run again. All consumption modes. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_recovery_rejects_unsafe_retries` | +| Stateless compatibility | Direct-client and ephemeral-agent background calls still execute normally in both modes; this does not establish cross-poll recovery guarantees. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_tools_without_a_reusable_session_remain_supported` | +| Partial batch and concurrent ownership | A concurrent poll is rejected; cancellation retains completed siblings and leaves the incomplete batch indeterminate after restoration. All consumption modes. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_partial_batch_and_concurrent_poll_fail_closed` | +| Reused call IDs, reasoning, middleware, and consumed limits | Distinct occurrences with the same provider `call_id` retain distinct results and reasoning; recovery does not repeat middleware or reset function-call/iteration limits. Final responses and persisted history retain both results and cumulative usage is preserved. All consumption modes. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_recovery_preserves_reused_call_ids_reasoning_and_limits` | +| Malformed execution identity | Missing response/occurrence identity rejects middleware and tool execution; a repeated occurrence cannot execute again or erase its recorded result. Incomplete evidence remains non-retryable after JSON restoration. Covered at the shared execution boundary with concurrent and sequential invocation configured. | `packages/core/tests/core/test_tools.py::test_background_recovery_rejects_malformed_execution_identity` | +| Interrupted submission stream | Once a submission update is exposed, a later HTTP 429 is indeterminate rather than retryable, including after restoration. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_recovery_rejects_retry_after_partial_submission_stream` | +| Refreshed provider context | A completed TodoProvider mutation may change regenerated context, but retry accepts unchanged caller input and submits the identical checkpointed messages without duplicating the mutation. Both recovery modes. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_recovery_accepts_refreshed_todo_context` | +| Conversation continuation aliases | `conversation_id`, native `conversation` strings, and conversation objects retain the original conversation without conflicting `previous_response_id`. Both recovery modes after JSON restoration. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_recovery_preserves_conversation_mode` | +| Token-only model retention | An omitted model on polling/recovery retains the initial per-run override rather than reverting to the client default. Both recovery modes after JSON restoration. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_token_only_poll_retains_model_override` | +| Empty-state scope change | A background request that produced no execution or handled-response evidence does not prevent a later model/endpoint change. Non-streaming coverage. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_without_execution_evidence_can_change_scope` | +| Local preparation failure before dispatch | Invalid request options do not poison a ready checkpoint; a corrected retry delivers without repeating retrieval or execution. Both recovery modes. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_recovery_survives_local_request_validation_failure` | +| Close immediately after a streamed tool result | The completed batch is already checkpointed; close before dispatch followed by JSON restoration permits non-streaming delivery without re-execution. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_stream_close_after_tool_result_keeps_delivery_ready` | +| Dynamic tool registry | Recovery restores additions/removals in effective order. Missing added implementations reject delivery without losing the ready checkpoint; supplying them permits retry. Both recovery modes after JSON restoration. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_recovery_restores_dynamic_tool_registry` | +| Stream-close ownership cleanup | Closing a retrieval stream before local execution releases live-session ownership so another poll can retrieve normally. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_stream_close_releases_poll_ownership` | +| Explicit approval boundary | A background-polled approval-required tool waits for an explicit bound approval and executes once on resume in both modes; this is not coverage of delivery failure after approval execution. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_recovery_preserves_explicit_approval_resume` | +| Handled-response capacity | Stale tokens remain rejected; a full ledger refuses a new tool batch instead of evicting consumed-token evidence. Non-streaming coverage. | `packages/openai/tests/openai/test_openai_chat_client.py::test_background_recovery_capacity_does_not_evict_consumed_tokens` | +| Foundry agent scope | Changed name/version or an effective non-preview agent-reference override cannot reuse another agent's execution evidence. A preview-only ignored override does not change scope. Non-streaming coverage. | `packages/foundry/tests/foundry/test_foundry_agent.py::test_foundry_background_recovery_rejects_another_agent` | +| Foundry delegated recovery-state ownership | Raw, fully layered, and generic-agent/client entry points isolate parent/child recovery state while preserving application state on success and failure. | `packages/foundry/tests/foundry/test_foundry_agent.py::test_foundry_agent_tools_isolate_service_state_between_children` | +| AG-UI private recovery state | Both history modes retain trusted evidence in private Session Continuation State and reject request-state replacement, with or without discoverable ownership declarations. | `packages/ag-ui/tests/ag_ui/test_run.py::test_background_recovery_state_is_private_and_cannot_be_overwritten` | + ## Required coverage gaps -These scenarios are required but are not fully covered by merged tests on `main`: +These scenarios are required but are not fully covered by the regressions named above: | Gap | Tracking | |---|---| | Service-owned `previous_response_id` continuation cannot execute a terminal approval again on a later turn. | #6851 | +| Result-delivery failure after local execution during initial approval/Host resume needs its own recovery contract and dedicated regressions; successful approval resume does not cover it. | #8574 | Do not mark these rows covered by nearby tests; each needs a dedicated regression at the owning layer. @@ -819,6 +910,9 @@ Before accepting an update, reviewers must confirm: - caller messages and previous responses remain immutable; - streaming updates and final response agree with non-streaming output; - history replay does not reintroduce approval authority; +- background recovery retries delivery without repeating completed tools or middleware, and ambiguous outcomes fail + closed without discarding evidence; +- recovery state remains private, scoped to its owning session/provider, and isolated from delegated agents; - full package, syntax, source typing, and test typing checks were run. ## Related issues @@ -838,3 +932,5 @@ Before accepting an update, reviewers must confirm: - #6074 / #7233 — reasoning-paired tool-call replay - #6450 / #6794 — provider message and tool-result serialization - #8099 — in-run compaction summaries in the returned transcript +- #9031 — repeated local side effects after background result-submission failure +- #8574 — result-delivery recovery for approval/Host resume diff --git a/python/packages/ag-ui/AGENTS.md b/python/packages/ag-ui/AGENTS.md index ab01c7005a3..317236ffbac 100644 --- a/python/packages/ag-ui/AGENTS.md +++ b/python/packages/ag-ui/AGENTS.md @@ -181,6 +181,8 @@ AG-UI protocol integration for building agent UIs with the AG-UI standard. object AG-UI is handed, so a wrapper agent that does not forward it would silently drop the protection. `_RESERVED_SERVICE_SESSION_STATE_KEYS` therefore reserves such keys unconditionally; add a key there whenever it names a remote resource that the server's own credentialed call addresses. +- Background tool recovery evidence (`_background_tool_recovery`) is protected from request Shared State even through + opaque wrappers, but retained in private Session Continuation State in both history modes. - `ToolApprovalMiddleware` names its session state with an application-chosen `source_id`, which a wrapper can hide from the agent AG-UI is handed. Tool approval authority is therefore also protected by shape, not only by name: a client Shared State value holding a non-empty `rules`, `queued_approval_requests` or `collected_approval_responses` diff --git a/python/packages/ag-ui/agent_framework_ag_ui/_agent_run.py b/python/packages/ag-ui/agent_framework_ag_ui/_agent_run.py index 75a78aa49ab..827d53bcf18 100644 --- a/python/packages/ag-ui/agent_framework_ag_ui/_agent_run.py +++ b/python/packages/ag-ui/agent_framework_ag_ui/_agent_run.py @@ -162,6 +162,7 @@ AG_UI_INTERNAL_METADATA_KEYS = {"ag_ui_thread_id", "ag_ui_run_id", "current_state", "forwarded_props"} _COLLECTED_APPROVAL_RESPONSES_KEY = "collected_approval_responses" _PROVIDER_SERVICE_SESSION_ID_STATE_KEY = "__ag_ui_provider_service_session_id" +_BACKGROUND_TOOL_RECOVERY_STATE_KEY = "_background_tool_recovery" # Provider-owned session-state keys reserved unconditionally, independent of what the agent declares through # ``service_session_state_keys``. Agents are expected to declare their own keys, but that resolves through the @@ -2653,6 +2654,7 @@ def _request_state_protected_keys(agent: SupportsAgentRun) -> set[str]: _TOOL_APPROVAL_STATE_KEY, InMemoryHistoryProvider.DEFAULT_SOURCE_ID, MESSAGE_INJECTION_PENDING_MESSAGES_STATE_KEY, + _BACKGROUND_TOOL_RECOVERY_STATE_KEY, *(provider.source_id for provider in context_providers), *_provider_service_session_state_keys(agent), *_fides_session_state_keys(agent), @@ -2725,6 +2727,8 @@ def _provider_service_session_state_keys(agent: SupportsAgentRun) -> set[str]: declared = getattr(agent, "service_session_state_keys", ()) if isinstance(declared, (list, tuple, set, frozenset)): keys.update(key for key in cast(Collection[Any], declared) if isinstance(key, str)) + # Recovery evidence stays private and durable even when model history is client-managed. + keys.discard(_BACKGROUND_TOOL_RECOVERY_STATE_KEY) return keys diff --git a/python/packages/ag-ui/tests/ag_ui/test_run.py b/python/packages/ag-ui/tests/ag_ui/test_run.py index e462e1dbbb3..9e7d05e0f1a 100644 --- a/python/packages/ag-ui/tests/ag_ui/test_run.py +++ b/python/packages/ag-ui/tests/ag_ui/test_run.py @@ -3811,6 +3811,55 @@ def capture_state(*args: Any, **kwargs: Any) -> Any: assert observed_state == [{"client_value": "available"}] +@pytest.mark.parametrize("use_service_session", [False, True]) +@pytest.mark.parametrize("declares_keys", [False, True], ids=["opaque-wrapper", "declared-agent"]) +async def test_background_recovery_state_is_private_and_cannot_be_overwritten( + use_service_session: bool, declares_keys: bool +) -> None: + from agent_framework_ag_ui import AgentFrameworkAgent, InMemoryAGUIThreadSnapshotStore + + stub = StubAgent() + key = "_background_tool_recovery" + if declares_keys: + setattr(stub, "service_session_state_keys", frozenset({key})) + observed_state: list[dict[str, Any]] = [] + original_run = stub.run + trusted_state = {"pending": {"token": "trusted-response"}, "handled": ["already-executed"]} + + def capture_state(*args: Any, **kwargs: Any) -> Any: + session = kwargs["session"] + observed_state.append(dict(session.state)) + session.state[key] = trusted_state + return original_run(*args, **kwargs) + + stub.run = capture_state # type: ignore[assignment, method-assign] # ty: ignore[invalid-assignment] + store = InMemoryAGUIThreadSnapshotStore() + agent = AgentFrameworkAgent( + agent=stub, + snapshot_store=store, + use_service_session=use_service_session, + service_session_id_from_thread_id=use_service_session, + ) + payload: dict[str, Any] = { + "thread_id": "recovery-thread", + "__ag_ui_snapshot_scope": "test", + "messages": [{"role": "user", "content": "Hello"}], + "state": {key: {"pending": None, "handled": []}, "client_value": "available"}, + } + _ = [event async for event in agent.run({**payload, "run_id": "first-run"})] + snapshot = await store.get(scope="test", thread_id="recovery-thread") + assert snapshot is not None + assert snapshot.session_state is not None + assert snapshot.session_state[key] == trusted_state + assert snapshot.state is not None + assert snapshot.state[key] == payload["state"][key] + _ = [event async for event in agent.run({**payload, "run_id": "second-run"})] + assert observed_state == [ + {"client_value": "available"}, + {key: trusted_state, "client_value": "available"}, + ] + + def test_tool_approval_authority_is_recognized_by_shape_not_encoding() -> None: """Any non-empty approval list is authority whatever its item encoding; inert shapes are not.""" authority_states: list[Any] = [ diff --git a/python/packages/core/AGENTS.md b/python/packages/core/AGENTS.md index b66484a40ec..19995556bfc 100644 --- a/python/packages/core/AGENTS.md +++ b/python/packages/core/AGENTS.md @@ -13,6 +13,7 @@ agent_framework/ ├── _clients.py # Chat client base classes and protocols ├── _types.py # Core types (Message, ChatResponse, Content, etc.) ├── _tools.py # Tool definitions and function invocation +├── _tool_recovery.py # Session-backed execution evidence and delivery checkpoints for background responses ├── _vectors.py # Vector store models, CRUD/search abstractions, and protocols ├── _middleware.py # Middleware system for request/response interception ├── _sessions.py # AgentSession and context provider abstractions diff --git a/python/packages/core/agent_framework/_agents.py b/python/packages/core/agent_framework/_agents.py index fb1e94f3841..d84837d1a9f 100644 --- a/python/packages/core/agent_framework/_agents.py +++ b/python/packages/core/agent_framework/_agents.py @@ -1752,6 +1752,8 @@ async def _prepare_run_context( from ._tools import _APPROVAL_SESSION_IS_AUTHORITATIVE_KEY # pyright: ignore[reportPrivateUsage] effective_client_kwargs.pop(_APPROVAL_SESSION_IS_AUTHORITATIVE_KEY, None) + if callable(getattr(self.client, "_background_tool_recovery_scope", None)): + effective_client_kwargs["_background_tool_recovery_inputs"] = input_messages if active_session is not None: effective_client_kwargs["session"] = active_session effective_client_kwargs[_APPROVAL_SESSION_IS_AUTHORITATIVE_KEY] = not framework_created_session diff --git a/python/packages/core/agent_framework/_tool_recovery.py b/python/packages/core/agent_framework/_tool_recovery.py new file mode 100644 index 00000000000..e4f8ba984f6 --- /dev/null +++ b/python/packages/core/agent_framework/_tool_recovery.py @@ -0,0 +1,334 @@ +# Copyright (c) Microsoft. All rights reserved. + +from __future__ import annotations + +import copy +import json +from collections.abc import Mapping, Sequence +from math import isfinite +from time import perf_counter, time +from types import TracebackType +from typing import Any, Literal, cast + +import msgspec + +from ._serialization import make_json_safe +from ._sessions import AgentSession +from ._types import ChatResponse, ChatResponseUpdate, Content, Message +from .exceptions import ChatClientException, ChatClientInvalidRequestException + +BACKGROUND_TOOL_RECOVERY_KEY = "_background_tool_recovery" +_MAX_HANDLED_RESPONSES = 256 + + +class _PendingToolDelivery(msgspec.Struct, kw_only=True, forbid_unknown_fields=True): + token: str + response_id: str + status: Literal["executing", "ready", "submitting", "indeterminate"] + calls: dict[str, list[str] | None] + inputs: list[str] + prepared_messages: list[str] + transcript: list[str] + budget: dict[str, Any] + errors_in_a_row: int + usage: dict[str, Any] | None + tool_choice: Any + continuation: dict[str, Any] + tools: list[str] + + +class _BackgroundToolState(msgspec.Struct, kw_only=True, forbid_unknown_fields=True): + version: Literal[1] + scope: str + busy: bool + handled: list[str] + pending: _PendingToolDelivery | None + model: str | None = None + token: str | None = None + + +class BackgroundToolRecovery: + """Retain local execution evidence independently of successful-run history.""" + + def __init__( + self, + *, + scope: str, + session: AgentSession | None, + authoritative: bool, + options: Mapping[str, Any], + messages: Sequence[Message], + model: str | None, + ) -> None: + self._scope = scope + self.model = model + self._session = session if authoritative else None + raw_token = options.get("continuation_token") + token = cast(Mapping[str, Any], raw_token).get("response_id") if isinstance(raw_token, Mapping) else None + self._token: str | None = token if isinstance(token, str) else None + if raw_token is not None and (not isinstance(self._token, str) or not self._token): + raise ChatClientInvalidRequestException("Background continuation requires a non-empty response_id.") + self._inputs = [message.to_json() for message in messages] + self._state: _BackgroundToolState | None = None + self._response: ChatResponse[Any] | None = None + self._streamed_delivery = False + self.resuming = False + + def __enter__(self) -> BackgroundToolRecovery: + if self._session is None: + return self + raw = self._session.state.get(BACKGROUND_TOOL_RECOVERY_KEY) + if BACKGROUND_TOOL_RECOVERY_KEY not in self._session.state: + self._state = _BackgroundToolState( + version=1, scope=self._scope, busy=False, handled=[], pending=None, model=self.model + ) + else: + try: + self._state = msgspec.convert(raw, type=_BackgroundToolState, strict=True) + except (TypeError, ValueError) as exc: + raise ChatClientInvalidRequestException( + "Invalid background tool recovery state; refusing replay." + ) from exc + state = self._state + if state.busy: + raise ChatClientInvalidRequestException( + "Another background poll owns this session; concurrent polls are unsafe." + ) + if state.scope != self._scope: + if state.pending is not None or state.handled: + raise ChatClientInvalidRequestException("Background recovery belongs to a different provider or model.") + state.scope = self._scope + state.token = None + state.model = self.model + if state.pending is not None: + pending = state.pending + if self._token != pending.token or self._inputs != pending.inputs: + raise ChatClientInvalidRequestException( + "Pending tool results must be recovered with the original token and no new input." + ) + if pending.status != "ready": + raise ChatClientException( + "Background tool execution or result delivery has an indeterminate outcome. " + "Reconcile it with the external service; tools will not be executed again." + ) + if not pending.calls or any(result is None for result in pending.calls.values()): + raise ChatClientInvalidRequestException("Incomplete background execution evidence; refusing replay.") + try: + for key in ("elapsed_seconds", "paused_at"): + value = pending.budget[key] + if type(value) not in (int, float) or not isfinite(value) or value < 0: + raise ValueError(f"Invalid background budget {key}.") + for key in ("attempt_count", "total_function_calls"): + value = pending.budget[key] + if type(value) is not int or value < 0: + raise ValueError(f"Invalid background budget {key}.") + if not pending.prepared_messages or not pending.transcript: + raise ValueError("Missing background result transcript.") + for message in (*pending.prepared_messages, *pending.transcript): + Message.from_json(message) + except (KeyError, TypeError, ValueError) as exc: + raise ChatClientInvalidRequestException( + "Invalid background delivery checkpoint; refusing replay." + ) from exc + self.resuming = True + elif self._token in state.handled: + raise ChatClientInvalidRequestException("This background token is stale; its tools have already completed.") + state.busy = True + self._save() + return self + + def __exit__( + self, + exc_type: type[BaseException] | None, + exc: BaseException | None, + traceback: TracebackType | None, + ) -> None: + if self._state is None: + return + pending = self._state.pending + if pending is not None: + if exc is None: + self._remember(pending.token, pending.response_id) + self._state.pending = None + elif pending.status == "submitting": + cause = exc.__cause__ if isinstance(exc, ChatClientException) else exc + status_code = getattr(cause, "status_code", None) + pending.status = ( + "ready" if status_code in (429, 503) and not self._streamed_delivery else "indeterminate" + ) + elif pending.status != "ready": + pending.status = "indeterminate" + self._state.busy = False + self._save() + + def _save(self) -> None: + if self._session is not None and self._state is not None: + self._session.state[BACKGROUND_TOOL_RECOVERY_KEY] = msgspec.to_builtins(self._state) + + def _remember(self, *response_ids: str) -> None: + if self._state is not None: + for response_id in response_ids: + if response_id not in self._state.handled: + self._state.handled.append(response_id) + + def restore( + self, options: dict[str, Any], request_kwargs: dict[str, Any], budget: dict[str, Any] + ) -> _PendingToolDelivery | None: + if not self.resuming or self._state is None or self._state.pending is None: + return None + pending = self._state.pending + definitions = self._tool_definitions(options.get("tools") or []) + restored_tools: list[Any] = [] + available_tools = list(options.get("tools") or []) + for definition in pending.tools: + try: + index = definitions.index(definition) + except ValueError as exc: + raise ChatClientInvalidRequestException( + "Background recovery requires the checkpointed tool registry; supply its tool implementations." + ) from exc + restored_tools.append(available_tools.pop(index)) + definitions.pop(index) + snapshot = copy.deepcopy(pending.budget) + elapsed = float(snapshot.pop("elapsed_seconds")) + paused_at = float(snapshot.pop("paused_at")) + payload_budget = budget.get("_function_result_payload_budget") + if isinstance(payload_budget, dict): + payload_budget = cast(dict[str, Any], payload_budget) + payload_budget.clear() + payload_budget.update(snapshot.get("_function_result_payload_budget", {})) + snapshot["_function_result_payload_budget"] = payload_budget + budget.clear() + budget.update(snapshot) + budget["start_time"] = perf_counter() - elapsed - max(0.0, time() - paused_at) + options.pop("continuation_token", None) + for key in ("conversation_id", "previous_response_id", "conversation"): + options.pop(key, None) + options.update(copy.deepcopy(pending.continuation)) + options["tool_choice"] = copy.deepcopy(pending.tool_choice) + options["tools"] = restored_tools + request_kwargs.pop("conversation_id", None) + if conversation_id := options.get("conversation_id"): + request_kwargs["conversation_id"] = conversation_id + return pending + + def before_request(self) -> None: + if self._state is not None and self._state.pending is not None: + pending = self._state.pending + if pending.status == "ready": + pending.status = "submitting" + self._streamed_delivery = False + self._save() + + def observe_update(self, update: ChatResponseUpdate) -> None: + if self._state is not None and self._state.pending is not None and self._state.pending.status == "submitting": + self._streamed_delivery = True + + def observe_response(self, response: ChatResponse[Any]) -> None: + if self._state is not None: + if self._state.pending is not None: + pending = self._state.pending + if pending.status == "submitting": + self._remember(pending.token, pending.response_id) + self._state.pending = None + token = response.continuation_token + self._state.token = token.get("response_id") if isinstance(token, dict) else None + self._save() + self._response = response + + def start_call(self, function_call: Content) -> bool: + # Approval/Host resume executes before a new provider response and keeps its existing recovery contract. + if self._response is None: + return False + if self._session is None or self._state is None: + return False + if not self._response.response_id: + raise ChatClientInvalidRequestException("Background tools require a provider response identity.") + state = self._state + if len(state.handled) >= _MAX_HANDLED_RESPONSES: + raise ChatClientInvalidRequestException( + "Background recovery capacity reached; start a new session instead of forgetting executed tools." + ) + if state.pending is None: + state.pending = _PendingToolDelivery( + token=self._token or self._response.response_id, + response_id=self._response.response_id, + status="executing", + calls={}, + inputs=copy.deepcopy(self._inputs), + prepared_messages=[], + transcript=[], + budget={}, + errors_in_a_row=0, + usage=None, + tool_choice=None, + continuation={}, + tools=[], + ) + occurrence_id = function_call.id + if not occurrence_id or occurrence_id in state.pending.calls: + raise ChatClientInvalidRequestException( + "Missing or repeated background tool occurrence; refusing execution." + ) + state.pending.calls[occurrence_id] = None + self._save() + return True + + def complete_call(self, function_call: Content, contents: Sequence[Content]) -> None: + if self._state is None or self._state.pending is None or function_call.id is None: + raise RuntimeError("Background tool execution was not claimed.") + self._state.pending.calls[function_call.id] = [ + msgspec.json.encode(content.to_dict()).decode() for content in contents + ] + self._save() + + def checkpoint( + self, + *, + prepared_messages: Sequence[Message], + transcript: Sequence[Message], + budget: Mapping[str, Any], + errors_in_a_row: int, + usage: Mapping[str, Any] | None, + options: Mapping[str, Any], + ) -> None: + if self._state is None or self._state.pending is None: + return + pending = self._state.pending + if pending.status != "executing" or any(result is None for result in pending.calls.values()): + raise ChatClientException("Background tool batch did not produce complete execution evidence.") + # Opaque JSON prevents AgentSession restoration from eagerly rehydrating nested type discriminators. + pending.prepared_messages = [message.to_json() for message in prepared_messages] + pending.transcript = [message.to_json() for message in transcript] + pending.budget = copy.deepcopy(dict(budget)) + pending.errors_in_a_row = errors_in_a_row + pending.usage = dict(usage) if usage is not None else None + pending.tool_choice = copy.deepcopy(options.get("tool_choice")) + conversation = options.get("conversation") + conversation_id = options.get("conversation_id") + if isinstance(conversation, Mapping): + conversation = cast(Mapping[str, Any], conversation).get("id") + if isinstance(conversation, str) and conversation: + conversation_id = conversation + pending.continuation = {"conversation_id": conversation_id or pending.response_id} + pending.tools = self._tool_definitions(options.get("tools") or []) + pending.status = "ready" + self._save() + + @staticmethod + def _tool_definitions(tools: Sequence[Any]) -> list[str]: + from ._tools import FunctionTool + + definitions: list[str] = [] + for item in tools: + if isinstance(item, FunctionTool): + definition = { + "tool": item.to_dict(exclude={"invocation_count", "invocation_exception_count"}), + "additional_properties": make_json_safe(item.additional_properties), + } + elif isinstance(item, Mapping): + definition = make_json_safe(item) + else: + raise ChatClientInvalidRequestException("Background recovery cannot checkpoint this tool type.") + definitions.append(json.dumps(definition, sort_keys=True)) + return definitions diff --git a/python/packages/core/agent_framework/_tools.py b/python/packages/core/agent_framework/_tools.py index 151b6b662ee..0292b277691 100644 --- a/python/packages/core/agent_framework/_tools.py +++ b/python/packages/core/agent_framework/_tools.py @@ -14,6 +14,7 @@ import warnings from collections import Counter, deque from collections.abc import ( + AsyncGenerator, AsyncIterable, Awaitable, Callable, @@ -22,7 +23,7 @@ Mapping, Sequence, ) -from contextlib import contextmanager, suppress +from contextlib import aclosing, contextmanager, suppress from dataclasses import dataclass from datetime import date, datetime, timedelta, timezone, tzinfo from datetime import time as datetime_time @@ -85,6 +86,7 @@ MiddlewareTypes, ) from ._sessions import AgentSession + from ._tool_recovery import BackgroundToolRecovery from ._types import ( ChatOptions, ChatResponse, @@ -2646,6 +2648,7 @@ async def _try_execute_function_call_groups( approval_session_is_authoritative: bool = True, middleware_pipeline: FunctionMiddlewarePipeline | None = None, host_payload_budget: _FunctionResultPayloadBudget | None = None, + recovery: BackgroundToolRecovery | None = None, ) -> tuple[list[list[Content]], bool]: """Execute multiple function calls concurrently while preserving per-call result groups. @@ -2658,6 +2661,7 @@ async def _try_execute_function_call_groups( approval_session_is_authoritative: Whether the invocation session owns persistent approval state. middleware_pipeline: Optional middleware pipeline to apply during execution. host_payload_budget: Shared request budget for retained Host-only function result payloads. + recovery: Optional session-owned background execution and delivery evidence. Returns: A tuple of: @@ -2725,19 +2729,26 @@ async def _try_execute_function_call_groups( # Each locally executable call gets its own task so context changes made by # one tool cannot leak into another. + async def execute_call(function_call: Content) -> tuple[list[Content], bool]: + claimed = recovery is not None and recovery.start_call(_underlying_function_call(function_call)) + result = await _execute_single_function_call( + function_call, + custom_args=custom_args, + config=config, + tool_map=tool_map, + invocation_session=invocation_session, + middleware_pipeline=middleware_pipeline, + live_tools=live_tools, + host_payload_budget=host_payload_budget, + ) + if claimed and recovery is not None: + recovery.complete_call(_underlying_function_call(function_call), result[0]) + return result + def create_execution_task(function_call: Content) -> asyncio.Task[tuple[list[Content], bool]]: return contextvars.copy_context().run( asyncio.create_task, - _execute_single_function_call( - function_call, - custom_args=custom_args, - config=config, - tool_map=tool_map, - invocation_session=invocation_session, - middleware_pipeline=middleware_pipeline, - live_tools=live_tools, - host_payload_budget=host_payload_budget, - ), + execute_call(function_call), ) async def execute_local_calls(function_calls_to_execute: Sequence[Content]) -> list[tuple[list[Content], bool]]: @@ -2916,6 +2927,7 @@ async def _execute_function_calls( approval_session_is_authoritative: bool = True, middleware_pipeline: FunctionMiddlewarePipeline | None = None, host_payload_budget: _FunctionResultPayloadBudget | None = None, + recovery: BackgroundToolRecovery | None = None, ) -> _FunctionExecutionBatch: tools = _extract_tools(options) if not tools: @@ -2929,6 +2941,7 @@ async def _execute_function_calls( middleware_pipeline=middleware_pipeline, config=config, host_payload_budget=host_payload_budget, + recovery=recovery, ) return _FunctionExecutionBatch( result_groups=result_groups, @@ -5810,20 +5823,28 @@ async def _get_response_with_function_invocation( budget_state: dict[str, Any], max_errors: int, middleware_pipeline: FunctionMiddlewarePipeline | None = None, + recovery: BackgroundToolRecovery | None = None, ) -> ChatResponse[Any]: """Run the non-streaming function invocation loop.""" from ._compaction import _reconcile_compaction_summaries # pyright: ignore[reportPrivateUsage] from ._middleware import MiddlewareFailure - from ._types import ChatResponse, add_usage_details + from ._types import ChatResponse, Message, add_usage_details - errors_in_a_row = 0 + pending = recovery.restore(options, request_kwargs, budget_state) if recovery is not None else None + errors_in_a_row = pending.errors_in_a_row if pending is not None else 0 total_function_calls = int(budget_state.get("total_function_calls", 0) or 0) max_function_calls = self.function_invocation_configuration.get("max_function_calls") max_duration_seconds = self.function_invocation_configuration.get("max_duration_seconds") - prepared_messages = _copy_messages_for_function_invocation(messages) - function_call_messages: list[Message] = [] + prepared_messages = ( + [Message.from_json(message) for message in pending.prepared_messages] + if pending is not None + else _copy_messages_for_function_invocation(messages) + ) + function_call_messages: list[Message] = ( + [Message.from_json(message) for message in pending.transcript] if pending is not None else [] + ) response: ChatResponse[Any] | None = None - aggregated_usage: UsageDetails | None = None + aggregated_usage = cast("UsageDetails | None", pending.usage) if pending is not None else None max_iterations = self.function_invocation_configuration.get("max_iterations", DEFAULT_MAX_ITERATIONS) attempt_start = int(budget_state.get("attempt_count", 0) or 0) @@ -5923,6 +5944,8 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non session=invocation_session, options=options, ) + if recovery is not None: + recovery.observe_response(response) terminal_prefix_length = len(function_call_messages) try: @@ -5978,6 +6001,15 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non errors_in_a_row = function_processing.errors_in_a_row _reset_required_tool_choice(options) _prepare_messages_for_next_iteration(prepared_messages, response) + if recovery is not None: + recovery.checkpoint( + prepared_messages=prepared_messages, + transcript=function_call_messages, + budget=_portable_budget_snapshot(budget_state), + errors_in_a_row=errors_in_a_row, + usage=aggregated_usage, + options=options, + ) # Phase 3: the iteration budget is exhausted, so request one final response with tools disabled. if response is not None: @@ -6015,6 +6047,8 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non session=invocation_session, options=options, ) + if recovery is not None: + recovery.observe_response(response) response.usage_details = aggregated_usage _prepend_function_call_messages(response, function_call_messages) _clear_budget_state_from_session(invocation_session, budget_state) @@ -6036,22 +6070,34 @@ async def _stream_response_with_function_invocation( max_errors: int, middleware_pipeline: FunctionMiddlewarePipeline | None = None, invalidation_error: list[ResponseInvalidatedException], - ) -> AsyncIterable[ChatResponseUpdate]: + recovery: BackgroundToolRecovery | None = None, + ) -> AsyncGenerator[ChatResponseUpdate]: """Run the streaming function invocation loop.""" from ._middleware import MiddlewareFailure - from ._types import ChatResponseUpdate, Content, add_usage_details + from ._types import ChatResponse, ChatResponseUpdate, Content, Message, add_usage_details - errors_in_a_row = 0 + pending = recovery.restore(options, request_kwargs, budget_state) if recovery is not None else None + errors_in_a_row = pending.errors_in_a_row if pending is not None else 0 total_function_calls = int(budget_state.get("total_function_calls", 0) or 0) max_function_calls = self.function_invocation_configuration.get("max_function_calls") max_duration_seconds = self.function_invocation_configuration.get("max_duration_seconds") - prepared_messages = _copy_messages_for_function_invocation(messages) + prepared_messages = ( + [Message.from_json(message) for message in pending.prepared_messages] + if pending is not None + else _copy_messages_for_function_invocation(messages) + ) + function_call_messages = ( + [Message.from_json(message) for message in pending.transcript] if pending is not None else [] + ) response: ChatResponse[Any] | None = None - completed_usage: UsageDetails | None = None + completed_usage = cast("UsageDetails | None", pending.usage) if pending is not None else None turn_usage: UsageDetails | None = None last_projected_usage: UsageDetails | None = None max_iterations = self.function_invocation_configuration.get("max_iterations", DEFAULT_MAX_ITERATIONS) attempt_start = int(budget_state.get("attempt_count", 0) or 0) + if pending is not None: + for update in ChatResponse(messages=function_call_messages, usage_details=completed_usage).to_updates(): + yield update def project_cumulative_usage(update: ChatResponseUpdate) -> ChatResponseUpdate: """Project one provider turn's usage as a cumulative outer-stream snapshot.""" @@ -6143,6 +6189,7 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non middleware_pipeline=middleware_pipeline, settle_dangling_calls=settle_approval_replay_calls, ) + function_call_messages.extend(approval_processing.response_messages) errors_in_a_row = approval_processing.errors_in_a_row total_function_calls = _record_function_calls( budget_state, @@ -6189,6 +6236,8 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non warned_empty_call_ids: set[str] = set() async with inner_stream: async for update in _iterate_provider_stream(inner_stream, on_invalidated): + if recovery is not None: + recovery.observe_update(update) for content in update.contents: if content.type != "function_call": continue @@ -6257,6 +6306,8 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non session=invocation_session, options=options, ) + if recovery is not None: + recovery.observe_response(response) if not any( item.type == "function_approval_request" or _is_actionable_function_call(item) @@ -6305,10 +6356,10 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non total_function_calls, function_processing.function_call_count, ) - for update in function_processing.streaming_updates: - yield update if function_processing.action != "continue" and function_processing.action != "stop": # "return" action: model produced a terminal response. + for update in function_processing.streaming_updates: + yield update _clear_budget_state_from_session(invocation_session, budget_state) return _apply_batch_limit_decision( @@ -6321,6 +6372,18 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non ) _reset_required_tool_choice(options) _prepare_messages_for_next_iteration(prepared_messages, response) + if recovery is not None: + function_call_messages.extend(response.messages) + recovery.checkpoint( + prepared_messages=prepared_messages, + transcript=function_call_messages, + budget=_portable_budget_snapshot(budget_state), + errors_in_a_row=errors_in_a_row, + usage=completed_usage, + options=options, + ) + for update in function_processing.streaming_updates: + yield update # Phase 3: the iteration budget is exhausted, so stream one final response with tools disabled. if response is not None: @@ -6347,6 +6410,8 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non last_projected_usage = None async with final_inner_stream: async for update in _iterate_provider_stream(final_inner_stream, on_invalidated): + if recovery is not None: + recovery.observe_update(update) update = _drop_unexecutable_tool_contents_from_update(update) if update is None: continue @@ -6361,6 +6426,8 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non session=invocation_session, options=options, ) + if recovery is not None: + recovery.observe_response(final_response) if fallback_added: yield _function_invocation_limit_fallback_update() _clear_budget_state_from_session(invocation_session, budget_state) @@ -6491,6 +6558,45 @@ def get_response( "max_consecutive_errors_per_request", DEFAULT_MAX_CONSECUTIVE_ERRORS_PER_REQUEST ) + recovery: BackgroundToolRecovery | None = None + recovery_inputs = request_kwargs.pop("_background_tool_recovery_inputs", messages) + recovery_model: str | None = None + recovery_scope = getattr(self, "_background_tool_recovery_scope", None) + if recovery_scope is not None and self.function_invocation_configuration.get("enabled", True): + from ._tool_recovery import BACKGROUND_TOOL_RECOVERY_KEY, BackgroundToolRecovery + + stored_recovery = ( + invocation_session.state.get(BACKGROUND_TOOL_RECOVERY_KEY) if invocation_session is not None else None + ) + has_pending_recovery = ( + isinstance(stored_recovery, dict) and cast(dict[str, Any], stored_recovery).get("pending") is not None + ) + if (options and (options.get("background") or options.get("continuation_token"))) or has_pending_recovery: + recovery_options = dict(options or {}) + model_option = recovery_options.get("model") or getattr(self, "model", None) + recovery_model = model_option if isinstance(model_option, str) else None + if isinstance(stored_recovery, dict) and recovery_options.get("model") is None: + saved_state = cast(dict[str, Any], stored_recovery) + token = recovery_options.get("continuation_token") + token_id = cast(Mapping[str, Any], token).get("response_id") if isinstance(token, Mapping) else None + pending_state = saved_state.get("pending") + pending_token = ( + cast(dict[str, Any], pending_state).get("token") if isinstance(pending_state, dict) else None + ) + if token_id is not None and token_id in (saved_state.get("token"), pending_token): + recovery_model = saved_state.get("model") + if recovery_model is not None: + recovery_options["model"] = recovery_model + recovery = BackgroundToolRecovery( + scope=recovery_scope(recovery_options), + session=invocation_session, + authoritative=approval_session_is_authoritative, + options=options or {}, + messages=recovery_inputs, + model=recovery_model, + ) + request_kwargs["_background_tool_recovery_before_request"] = recovery.before_request + # Bind one executor with the run's custom arguments, middleware, configuration, and session. execute_function_calls = partial( _execute_function_calls, @@ -6500,11 +6606,14 @@ def get_response( approval_session_is_authoritative=approval_session_is_authoritative, middleware_pipeline=function_middleware_pipeline, host_payload_budget=host_payload_budget, + recovery=recovery, ) # Give the loop private mutable options and one shared run-local tool list for progressive tool changes. # Make options mutable so we can update conversation_id during function invocation loop mutable_options: dict[str, Any] = dict(options) if options else {} + if recovery_model: + mutable_options["model"] = recovery_model # Remove additional_function_arguments from options passed to underlying chat client # It's for tool invocation only and not recognized by chat service APIs mutable_options.pop("additional_function_arguments", None) @@ -6528,7 +6637,8 @@ def get_response( # Dispatch to the shape-specific loop; both loops follow the same approval -> model -> execution phases. if not stream: - return self._get_response_with_function_invocation( + invoke_response = partial( + self._get_response_with_function_invocation, super_get_response=super_get_response, messages=messages, options=mutable_options, @@ -6541,7 +6651,16 @@ def get_response( budget_state=budget_state, max_errors=max_errors, middleware_pipeline=function_middleware_pipeline, + recovery=recovery, ) + if recovery is None: + return invoke_response() + + async def run_response() -> ChatResponse[Any]: + with recovery: + return await invoke_response() + + return run_response() response_format = mutable_options.get("response_format") invalidation_error: list[ResponseInvalidatedException] = [] @@ -6551,24 +6670,32 @@ def finalize_stream(updates: Sequence[ChatResponseUpdate]) -> ChatResponse[Any]: raise invalidation_error[0] return ChatResponse.from_updates(updates, output_format_type=response_format) - return ResponseStream( - self._stream_response_with_function_invocation( - super_get_response=super_get_response, - messages=messages, - options=mutable_options, - request_kwargs=request_kwargs, - compaction_strategy=compaction_strategy, - tokenizer=tokenizer, - execute_function_calls=execute_function_calls, - invocation_session=invocation_session, - approval_session_is_authoritative=approval_session_is_authoritative, - budget_state=budget_state, - max_errors=max_errors, - middleware_pipeline=function_middleware_pipeline, - invalidation_error=invalidation_error, - ), - finalizer=finalize_stream, + updates = self._stream_response_with_function_invocation( + super_get_response=super_get_response, + messages=messages, + options=mutable_options, + request_kwargs=request_kwargs, + compaction_strategy=compaction_strategy, + tokenizer=tokenizer, + execute_function_calls=execute_function_calls, + invocation_session=invocation_session, + approval_session_is_authoritative=approval_session_is_authoritative, + budget_state=budget_state, + max_errors=max_errors, + middleware_pipeline=function_middleware_pipeline, + invalidation_error=invalidation_error, + recovery=recovery, ) + if recovery is None: + return ResponseStream(updates, finalizer=finalize_stream) + + async def run_stream() -> AsyncIterable[ChatResponseUpdate]: + with recovery: + async with aclosing(updates): + async for update in updates: + yield update + + return ResponseStream(run_stream(), finalizer=finalize_stream) # Alias for the @tool decorator, used by security tools and samples diff --git a/python/packages/core/tests/core/test_tools.py b/python/packages/core/tests/core/test_tools.py index 39b2688501e..813d11a6c5c 100644 --- a/python/packages/core/tests/core/test_tools.py +++ b/python/packages/core/tests/core/test_tools.py @@ -212,6 +212,88 @@ def test_format_tool_parameters_rejects_unknown_format(parameter_format): _format_tool_parameters({"type": "object", "properties": {}}, parameter_format=parameter_format) +@pytest.mark.parametrize("invalid", ["missing-response", "missing-occurrence", "repeated-occurrence"]) +@pytest.mark.parametrize("allow_concurrent_invocation", [False, True]) +async def test_background_recovery_rejects_malformed_execution_identity( + invalid: str, allow_concurrent_invocation: bool +) -> None: + from agent_framework import AgentSession, ChatResponse + from agent_framework._middleware import FunctionMiddlewarePipeline + from agent_framework._tool_recovery import BackgroundToolRecovery + from agent_framework.exceptions import ChatClientException, ChatClientInvalidRequestException + + executions: list[str] = [] + middleware_calls: list[str] = [] + + @tool(approval_mode="never_require") + def record_action() -> str: + """Record one local action.""" + executions.append("done") + return "done" + + async def record_middleware(context: FunctionInvocationContext, call_next: Any) -> None: + middleware_calls.append(context.function.name) + await call_next() + + session = AgentSession() + options = {"continuation_token": {"response_id": "resp_bg"}} + recovery = BackgroundToolRecovery( + scope="test-scope", session=session, authoritative=True, options=options, messages=[], model="test-model" + ) + function_call = Content.from_function_call(call_id="call_1", name="record_action", arguments="{}") + function_call.id = None if invalid == "missing-occurrence" else "occurrence_1" + pipeline = FunctionMiddlewarePipeline(record_middleware) + config = normalize_function_invocation_configuration({"allow_concurrent_invocation": allow_concurrent_invocation}) + expected_error = "response identity" if invalid == "missing-response" else "occurrence" + with pytest.raises(ChatClientInvalidRequestException, match=expected_error), recovery: + recovery.observe_response( + ChatResponse(messages=[], response_id=None if invalid == "missing-response" else "resp_bg") + ) + if invalid == "repeated-occurrence": + await _try_execute_function_call_groups( + custom_args={}, + function_calls=[function_call], + tools=[record_action], + config=config, + invocation_session=session, + middleware_pipeline=pipeline, + recovery=recovery, + ) + await _try_execute_function_call_groups( + custom_args={}, + function_calls=[function_call], + tools=[record_action], + config=config, + invocation_session=session, + middleware_pipeline=pipeline, + recovery=recovery, + ) + + assert executions == (["done"] if invalid == "repeated-occurrence" else []) + assert middleware_calls == (["record_action"] if invalid == "repeated-occurrence" else []) + restored = AgentSession.from_dict(json.loads(json.dumps(session.to_dict()))) + state = restored.state["_background_tool_recovery"] + assert state["busy"] is False + if invalid != "missing-response": + assert state["pending"]["status"] == "indeterminate" + with ( + pytest.raises(ChatClientException, match="indeterminate"), + BackgroundToolRecovery( + scope="test-scope", + session=restored, + authoritative=True, + options=options, + messages=[], + model="test-model", + ), + ): + pytest.fail("Malformed execution evidence must not become replayable.") + if invalid == "repeated-occurrence": + result = json.loads(state["pending"]["calls"]["occurrence_1"][0]) + assert result["result"] == "done" + assert result["id"] == "occurrence_1" + + async def test_sequential_function_invocation_runs_calls_in_model_order() -> None: execution_order: list[str] = [] diff --git a/python/packages/foundry/agent_framework_foundry/_agent.py b/python/packages/foundry/agent_framework_foundry/_agent.py index a860c74c8f5..5d8a3130ca1 100644 --- a/python/packages/foundry/agent_framework_foundry/_agent.py +++ b/python/packages/foundry/agent_framework_foundry/_agent.py @@ -73,6 +73,10 @@ AzureTokenProvider = Callable[[], str | Awaitable[str]] AzureCredentialTypes = TokenCredential | AsyncTokenCredential +_FOUNDRY_SERVICE_SESSION_STATE_KEYS: frozenset[str] = RawOpenAIChatClient.service_session_state_keys | { + FOUNDRY_HOSTED_AGENT_SESSION_ID_KEY +} + class FoundryAgentSettings(TypedDict, total=False): """Settings for Microsoft FoundryAgentClient resolved from args and environment. @@ -182,9 +186,22 @@ class MyClient(FunctionInvocationLayer, RawFoundryAgentChatClient): OTEL_PROVIDER_NAME: ClassVar[str] = "azure.ai.foundry" _FEATURE_USAGE_INDEX: ClassVar[int | None] = FeatureIndex.FOUNDRY_AGENT - service_session_state_keys: ClassVar[frozenset[str]] = frozenset({FOUNDRY_HOSTED_AGENT_SESSION_ID_KEY}) + service_session_state_keys: ClassVar[frozenset[str]] = _FOUNDRY_SERVICE_SESSION_STATE_KEYS """Service-owned state keys, including when this client is used by a generic Agent.""" + def _background_tool_recovery_scope(self, options: Mapping[str, Any]) -> str: + from hashlib import sha256 + + extra_body = options.get("extra_body") + agent_reference = ( + cast(Mapping[str, Any], extra_body).get("agent_reference") + if isinstance(extra_body, Mapping) and not self.allow_preview + else None + ) + identity: Mapping[str, Any] = agent_reference or _build_agent_reference(self.agent_name, self.agent_version) + scope = (super()._background_tool_recovery_scope(options), sorted(identity.items())) + return sha256(repr(scope).encode()).hexdigest() + def __init__( self, *, @@ -690,7 +707,7 @@ class RawFoundryAgent( result = await agent.run("Hello!") """ - service_session_state_keys: ClassVar[frozenset[str]] = frozenset({FOUNDRY_HOSTED_AGENT_SESSION_ID_KEY}) + service_session_state_keys: ClassVar[frozenset[str]] = _FOUNDRY_SERVICE_SESSION_STATE_KEYS """Session-state keys this agent owns, which untrusted input must never supply. Holds the Foundry hosted-agent session ID. Despite the attribute name, that value is not a conversation or diff --git a/python/packages/foundry/tests/foundry/test_foundry_agent.py b/python/packages/foundry/tests/foundry/test_foundry_agent.py index a547f7dd57a..cc1ea3cce93 100644 --- a/python/packages/foundry/tests/foundry/test_foundry_agent.py +++ b/python/packages/foundry/tests/foundry/test_foundry_agent.py @@ -45,6 +45,7 @@ from agent_framework_foundry._agent import ( FoundryAgent, + FoundryAgentOptions, RawFoundryAgent, RawFoundryAgentChatClient, _FoundryAgentChatClient, @@ -1398,8 +1399,9 @@ async def test_raw_foundry_agent_prepare_run_context_injects_agent_session_id_fr @pytest.mark.parametrize("agent_type", [RawFoundryAgent, FoundryAgent, None]) @pytest.mark.parametrize("parent_handle", [None, "parent-agent-session"]) +@pytest.mark.parametrize("fails", [False, True], ids=["success", "failure"]) async def test_foundry_agent_tools_isolate_service_state_between_children( - agent_type: type[RawFoundryAgent] | None, parent_handle: str | None + agent_type: type[RawFoundryAgent] | None, parent_handle: str | None, fails: bool ) -> None: """Delegated Foundry calls retain application state without sharing service handles.""" project = MagicMock() @@ -1412,6 +1414,7 @@ async def test_foundry_agent_tools_isolate_service_state_between_children( ] parent = AgentSession(service_session_id="parent-response") parent.state["ordinary"] = "parent-value" + parent.state["_background_tool_recovery"] = {"parent": "checkpoint"} if parent_handle is not None: parent.state[FOUNDRY_HOSTED_AGENT_SESSION_ID_KEY] = parent_handle observed_options: list[dict[str, Any]] = [] @@ -1423,6 +1426,8 @@ def respond(**kwargs: Any) -> ResponseStream[ChatResponseUpdate, ChatResponse]: call_number = len(observed_options) async def stream() -> AsyncIterator[ChatResponseUpdate]: + if fails: + raise RuntimeError("Simulated child failure") yield ChatResponseUpdate( role="assistant", contents=[Content.from_text(text="done")], @@ -1435,27 +1440,81 @@ async def stream() -> AsyncIterator[ChatResponseUpdate]: with patch.object(RawFoundryAgentChatClient, "_inner_get_response", side_effect=respond): for child in (children[0], children[1], children[0]): delegated_tool = child.as_tool(propagate_session=True, stream_callback=updates.append) - result = await delegated_tool.invoke( - context=FunctionInvocationContext( - function=delegated_tool, - arguments={"task": "Run child"}, - session=parent, - parent_service_session_state_keys={FOUNDRY_HOSTED_AGENT_SESSION_ID_KEY}, - ) + original_run = child.run + + def capture_session(*args: Any, _run: Callable[..., Any] = original_run, **kwargs: Any) -> Any: + child_session = kwargs["session"] + assert "_background_tool_recovery" not in child_session.state + child_session.state["_background_tool_recovery"] = {"child": "checkpoint"} + return _run(*args, **kwargs) + + context = FunctionInvocationContext( + function=delegated_tool, + arguments={"task": "Run child"}, + session=parent, + parent_service_session_state_keys={FOUNDRY_HOSTED_AGENT_SESSION_ID_KEY}, ) - assert result[0].text == "done" + with patch.object(child, "run", side_effect=capture_session): + if fails: + with pytest.raises(RuntimeError, match="Simulated child failure"): + await delegated_tool.invoke(context=context) + else: + result = await delegated_tool.invoke(context=context) + assert result[0].text == "done" assert len(observed_options) == 3 assert all("agent_session_id" not in options.get("extra_body", {}) for options in observed_options) assert all(options.get("conversation_id") is None for options in observed_options) - assert len([update for update in updates if update.contents]) == 3 - expected_state = {"ordinary": "parent-value"} + assert len([update for update in updates if update.contents]) == (0 if fails else 3) + expected_state: dict[str, Any] = { + "ordinary": "parent-value", + "_background_tool_recovery": {"parent": "checkpoint"}, + } if parent_handle is not None: expected_state[FOUNDRY_HOSTED_AGENT_SESSION_ID_KEY] = parent_handle assert parent.state == expected_state assert parent.service_session_id == "parent-response" +@pytest.mark.parametrize("allow_preview", [False, True]) +@pytest.mark.parametrize("change", ["name", "version", "reference"]) +async def test_foundry_background_recovery_rejects_another_agent(allow_preview: bool, change: str) -> None: + from agent_framework.exceptions import ChatClientInvalidRequestException + + project = MagicMock() + project.get_openai_client.return_value = AsyncOpenAI(api_key="test-key") + client = _FoundryAgentChatClient( + project_client=project, agent_name="first-agent", agent_version="1", allow_preview=allow_preview + ) + options = FoundryAgentOptions(background=True) + original_scope = client._background_tool_recovery_scope(options) + session = AgentSession() + session.state["_background_tool_recovery"] = { + "version": 1, + "scope": original_scope, + "busy": False, + "handled": ["resp_executed"], + "pending": None, + } + if change == "name": + client.agent_name = "second-agent" + elif change == "version": + client.agent_version = "2" + else: + options["extra_body"] = {"agent_reference": {"name": "second-agent", "type": "agent_reference"}} + if allow_preview: + assert client._background_tool_recovery_scope(options) == original_scope + return + with ( + patch.object(client.client.responses.with_raw_response, "create", new=AsyncMock()) as create, + pytest.raises(ChatClientInvalidRequestException, match="different provider or model"), + ): + await client.get_response( + [Message("user", ["New request"])], options=options, client_kwargs={"session": session} + ) + assert create.await_count == 0 + + def test_foundry_agent_updates_session_from_response_ids() -> None: """Test that response and hosted-agent session IDs persist independently.""" diff --git a/python/packages/openai/AGENTS.md b/python/packages/openai/AGENTS.md index 1c8b0f4280e..730241f5b41 100644 --- a/python/packages/openai/AGENTS.md +++ b/python/packages/openai/AGENTS.md @@ -24,6 +24,14 @@ agent_framework_openai/ All clients follow the Raw + Full-Featured pattern (e.g., `RawOpenAIChatClient` + `OpenAIChatClient`). +The shared Responses adapter supplies the scope for session-backed background tool-result recovery and declares +`_background_tool_recovery` as provider-owned session state. The core function loop records execution separately +from result delivery, including for inherited Azure OpenAI and Foundry clients. Background calls without a reusable +`AgentSession` remain supported, but have no cross-poll recovery or replay protection. With a caller-retained session, +supported delivery retries reuse completed results, never function middleware or tool bodies. Ambiguous execution/delivery +fails closed. Recovery rebinds checkpointed tools to caller-supplied implementations and rejects missing or changed +definitions; dynamic additions must be supplied again on retry. + For Responses API continuation with service-side storage, a prior hosted `function_approval_request` is server-issued and must not be replayed inline, while the new hosted `function_approval_response` is serialized as `mcp_approval_response` so the user's approved or rejected decision diff --git a/python/packages/openai/agent_framework_openai/_chat_client.py b/python/packages/openai/agent_framework_openai/_chat_client.py index c71d20d0d26..c1c002fb288 100644 --- a/python/packages/openai/agent_framework_openai/_chat_client.py +++ b/python/packages/openai/agent_framework_openai/_chat_client.py @@ -628,8 +628,21 @@ class RawOpenAIChatClient( "conversation", ) SUPPORTS_RICH_FUNCTION_OUTPUT: ClassVar[bool] = True + service_session_state_keys: ClassVar[frozenset[str]] = frozenset({"_background_tool_recovery"}) _FEATURE_USAGE_INDEX: ClassVar[int | None] = FeatureIndex.OPENAI + def _background_tool_recovery_scope(self, options: Mapping[str, Any]) -> str: + from hashlib import sha256 + + scope = ( + self.OTEL_PROVIDER_NAME, + str(self.client.base_url), + options.get("model", self.model), + self.client.organization, + self.client.project, + ) + return sha256(repr(scope).encode()).hexdigest() + # Azure OpenAI Responses API may include this header in responses naming the actual model that # served the request (e.g. ``gpt-5-nano-2025-08-07``), which can differ from the deployment alias # that the request was addressed to and that ``response.model`` reports. When present, we use it @@ -914,6 +927,7 @@ def _inner_get_response( ) -> Awaitable[ChatResponse] | ResponseStream[ChatResponseUpdate, ChatResponse]: continuation_token: OpenAIContinuationToken | None = options.get("continuation_token") extra_headers = cast("Mapping[str, Any] | None", kwargs.get("extra_headers")) + before_request = cast("Callable[[], None] | None", kwargs.get("_background_tool_recovery_before_request")) if stream: function_call_ids: dict[int, tuple[str, str]] = {} @@ -992,6 +1006,8 @@ async def _stream() -> AsyncIterable[ChatResponseUpdate]: run_options["extra_headers"] = dict(extra_headers) response_format = validated_options.get("response_format") try: + if before_request is not None: + before_request() if "text_format" in run_options: # The SDK's ``responses.stream(text_format=...)`` helper preserves # client-side ``output_parsed`` partial parsing for structured outputs, @@ -1061,6 +1077,8 @@ async def _get_response() -> ChatResponse: if extra_headers is not None: run_options["extra_headers"] = dict(extra_headers) try: + if before_request is not None: + before_request() if "text_format" in run_options: raw_response = await client.responses.with_raw_response.parse(stream=False, **run_options) else: @@ -1300,8 +1318,7 @@ def _prepare_tools_for_openai( ) continue if isinstance(tool_item, FunctionTool): - params = tool_item.parameters() - params["additionalProperties"] = False + params: dict[str, Any] = {**tool_item.parameters(), "additionalProperties": False} response_tools.append( FunctionToolParam( name=tool_item.name, diff --git a/python/packages/openai/tests/openai/conftest.py b/python/packages/openai/tests/openai/conftest.py index c68e523bf6f..589c2355d15 100644 --- a/python/packages/openai/tests/openai/conftest.py +++ b/python/packages/openai/tests/openai/conftest.py @@ -1,8 +1,8 @@ # Copyright (c) Microsoft. All rights reserved. -from collections.abc import Callable, Generator +from collections.abc import Awaitable, Callable, Generator from typing import Any, cast -from unittest.mock import patch +from unittest.mock import AsyncMock, MagicMock, patch from openai import AsyncAzureOpenAI from opentelemetry.sdk.trace.export import SimpleSpanProcessor, SpanExporter @@ -16,6 +16,49 @@ ) +@fixture +def consume_response() -> Callable[[Any, str], Awaitable[Any]]: + async def consume(response: Any, mode: str) -> Any: + if mode == "non-streaming": + return await response + if mode == "iterate": + async for _ in response: + pass + return await response.get_final_response() + + return consume + + +@fixture +def response_stream_factory() -> Callable[[Any], MagicMock]: + def create(response: Any) -> MagicMock: + events: list[Any] = [] + for index, item in enumerate(response.output): + events.append(MagicMock(type="response.output_item.added", output_index=index, item=item)) + if item.type == "function_call": + events.append( + MagicMock( + type="response.function_call_arguments.done", + output_index=index, + item_id=item.id, + arguments=item.arguments, + ) + ) + elif item.type == "message": + for part in item.content: + events.append(MagicMock(type="response.output_text.delta", delta=part.text, logprobs=None)) + events.append(MagicMock(type="response.output_item.done", output_index=index, item=item)) + events.append(MagicMock(type="response.completed", response=response)) + stream = MagicMock(headers={}) + stream.parse.return_value = stream + stream.__aiter__.return_value = events + stream.__aenter__ = AsyncMock(return_value=stream) + stream.__aexit__ = AsyncMock(return_value=False) + return stream + + return create + + @fixture(autouse=True) def _skip_openai_integration_tests(request: FixtureRequest) -> None: """Skip OpenAI integration tests while OPENAI_INTEGRATION_TESTS_SKIP_REASON is set.""" diff --git a/python/packages/openai/tests/openai/test_openai_chat_client.py b/python/packages/openai/tests/openai/test_openai_chat_client.py index 445958e354f..8d085dbd605 100644 --- a/python/packages/openai/tests/openai/test_openai_chat_client.py +++ b/python/packages/openai/tests/openai/test_openai_chat_client.py @@ -53,7 +53,7 @@ ChatClientInvalidRequestException, SettingNotFoundError, ) -from openai import AsyncOpenAI, BadRequestError, DefaultAsyncHttpxClient +from openai import AsyncOpenAI, BadRequestError, DefaultAsyncHttpxClient, InternalServerError, RateLimitError from openai.types.responses import ( ResponseComputerToolCall, ResponseComputerToolCallOutputItem, @@ -91,6 +91,7 @@ OpenAIChatClient, OpenAIChatCompletionClient, OpenAIChatOptions, + OpenAIContinuationToken, RawOpenAIChatClient, RawOpenAIChatCompletionClient, ) @@ -10664,7 +10665,12 @@ def send_email(to: str) -> str: patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve), patch.object(client.client.responses.with_raw_response, "create", new=create), ): - stream = client.get_response([Message(role="user", contents=["email bob"])], stream=True, options=options) + stream = client.get_response( + [Message(role="user", contents=["email bob"])], + stream=True, + options=options, + client_kwargs={"session": AgentSession()}, + ) async for _ in stream: pass final = await stream.get_final_response() @@ -10738,12 +10744,14 @@ def send_email(to: str) -> str: ): messages = [Message(role="user", contents=["email bob"])] if stream: - response_stream = client.get_response(messages, stream=True, options=options) + response_stream = client.get_response( + messages, stream=True, options=options, client_kwargs={"session": AgentSession()} + ) async for _ in response_stream: pass final = await response_stream.get_final_response() else: - final = await client.get_response(messages, options=options) + final = await client.get_response(messages, options=options, client_kwargs={"session": AgentSession()}) assert executions == ["bob"] assert retrieve.await_count == 1 @@ -10751,6 +10759,1010 @@ def send_email(to: str) -> str: assert final.text == "Email sent." +@pytest.mark.parametrize("stream", [False, True], ids=["non-streaming", "streaming"]) +@pytest.mark.parametrize("status_code", [429, 503]) +@pytest.mark.parametrize("retry_succeeds", [False, True], ids=["persistent-failure", "retry-success"]) +@pytest.mark.parametrize("failure_stage", ["retrieve", "submit"]) +@pytest.mark.parametrize("restore_session", [False, True], ids=["live-session", "restored-session"]) +async def test_background_poll_retry_does_not_repeat_completed_tools( + stream: bool, + status_code: int, + retry_succeeds: bool, + failure_stage: str, + restore_session: bool, +) -> None: + """A caller-held token must not repeat a completed tool after result delivery fails (#9031).""" + executions: list[str] = [] + + @tool(approval_mode="never_require") + def send_email(to: str) -> str: + """Record a simulated email without contacting an external service.""" + executions.append(to) + return f"sent-{len(executions)}" + + queued_response = _completed_response("resp_bg", "message") + queued_response.status = "queued" + queued_response.output = [] + usage = SimpleNamespace( + input_tokens=3, output_tokens=2, total_tokens=5, input_tokens_details=None, output_tokens_details=None + ) + + if stream: + function_call_added = MagicMock() + function_call_added.type = "response.output_item.added" + function_call_added.output_index = 0 + function_call_added.item.type = "function_call" + function_call_added.item.call_id = "call_1" + function_call_added.item.name = "send_email" + arguments_delta = MagicMock() + arguments_delta.type = "response.function_call_arguments.delta" + arguments_delta.output_index = 0 + arguments_delta.item_id = "fc_1" + arguments_delta.delta = '{"to": "bob"}' + tool_completed = _completed_response_event("resp_bg", "function_call") + tool_completed.response.usage = usage + retrieved_response = _FakeAsyncEventStream([ + function_call_added, + arguments_delta, + tool_completed, + ]) + text_delta = MagicMock() + text_delta.type = "response.output_text.delta" + text_delta.delta = "Email sent." + text_delta.logprobs = None + final_completed = _completed_response_event("resp_next", "message") + final_completed.response.usage = usage + final_response = _FakeAsyncEventStream([ + text_delta, + final_completed, + ]) + else: + retrieved_response = _completed_response("resp_bg", "function_call") + final_response = _completed_response("resp_next", "message") + retrieved_response.usage = usage + final_response.usage = usage + + request = _OPENAI_HTTPX.Request( + "GET" if failure_stage == "retrieve" else "POST", + "https://unit-test.invalid/v1/responses/resp_bg", + ) + error_response = _OPENAI_HTTPX.Response(status_code=status_code, request=request) + error_type = RateLimitError if status_code == 429 else InternalServerError + request_error = error_type("Simulated request failure", response=error_response, body=None) + retrieve = AsyncMock(return_value=retrieved_response) + if failure_stage == "retrieve": + retrieve.side_effect = [request_error, retrieved_response if retry_succeeds else request_error] + create = AsyncMock(side_effect=[queued_response, final_response]) + else: + create = AsyncMock( + side_effect=[queued_response, request_error, final_response if retry_succeeds else request_error] + ) + + client = OpenAIChatClient(model="test-model", api_key="test-key") + history_provider = InMemoryHistoryProvider(load_messages=False) + agent = Agent(client=client, tools=[send_email], context_providers=[history_provider]) + session = agent.create_session() + with ( + patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve), + patch.object(client.client.responses.with_raw_response, "create", new=create), + ): + initial = await agent.run("Email bob.", session=session, options=OpenAIChatOptions(background=True)) + assert initial.continuation_token is not None + held_token = cast(OpenAIContinuationToken, initial.continuation_token) + + with pytest.raises(ChatClientException) as first_error: + if stream: + await agent.run( + session=session, stream=True, options=OpenAIChatOptions(continuation_token=held_token) + ).get_final_response() + else: + await agent.run(session=session, options=OpenAIChatOptions(continuation_token=held_token)) + + assert first_error.value.__cause__ is request_error + assert executions == (["bob"] if failure_stage == "submit" else []) + if restore_session: + session = AgentSession.from_dict(json.loads(json.dumps(session.to_dict()))) + + if retry_succeeds: + if stream: + retry_stream = agent.run( + session=session, stream=True, options=OpenAIChatOptions(continuation_token=held_token) + ) + if restore_session: + async for _ in retry_stream: + pass + final = await retry_stream.get_final_response() + else: + final = await agent.run(session=session, options=OpenAIChatOptions(continuation_token=held_token)) + assert final.text == "Email sent." + assert final.continuation_token is None + assert final.usage_details is not None + assert final.usage_details["total_token_count"] == 10 + calls = [ + content for message in final.messages for content in message.contents if content.type == "function_call" + ] + results = [ + content + for message in final.messages + for content in message.contents + if content.type == "function_result" + ] + assert len(calls) == len(results) == 1 + assert calls[0].id is not None + assert calls[0].id == results[0].id + stored_results = [ + content + for message in session.state[history_provider.source_id]["messages"] + for content in message.contents + if content.type == "function_result" + ] + assert [result.id for result in stored_results] == [results[0].id] + with pytest.raises(ChatClientInvalidRequestException, match="stale"): + await agent.run(session=session, options=OpenAIChatOptions(continuation_token=held_token)) + else: + with pytest.raises(ChatClientException) as retry_error: + if stream: + await agent.run( + session=session, stream=True, options=OpenAIChatOptions(continuation_token=held_token) + ).get_final_response() + else: + await agent.run(session=session, options=OpenAIChatOptions(continuation_token=held_token)) + assert retry_error.value.__cause__ is request_error + + assert executions == (["bob"] if failure_stage == "submit" or retry_succeeds else []) + submitted_outputs = [ + item["output"] + for request_call in create.await_args_list[1:] + for item in request_call.kwargs["input"] + if item["type"] == "function_call_output" + ] + expected_submissions = 2 if failure_stage == "submit" else int(retry_succeeds) + assert submitted_outputs == ["sent-1"] * expected_submissions + for request_call in create.await_args_list[1:]: + assert request_call.kwargs["previous_response_id"] == "resp_bg" + + +@pytest.mark.parametrize("mode", ["non-streaming", "iterate", "final-only"]) +@pytest.mark.parametrize( + "retry_change", + [ + "new-input", + "model", + "endpoint", + "version", + "budget", + "results", + "tool-schema", + "busy", + "ambiguous-delivery", + "cancelled-submission", + ], +) +async def test_background_recovery_rejects_unsafe_retries( + retry_change: str, + mode: str, + consume_response: Callable[[Any, str], Awaitable[Any]], + response_stream_factory: Callable[[Any], MagicMock], +) -> None: + executions: list[str] = [] + + @tool(approval_mode="never_require") + def send_email(to: str) -> str: + """Record a harmless local action.""" + executions.append(to) + return "sent-once" + + client = OpenAIChatClient(model="test-model", api_key="test-key") + session = AgentSession() + options = OpenAIChatOptions(continuation_token={"response_id": "resp_bg"}, tools=[send_email]) + request = _OPENAI_HTTPX.Request("POST", "https://unit-test.invalid/v1/responses") + request_error: BaseException = RateLimitError( + "Simulated rejection", response=_OPENAI_HTTPX.Response(429, request=request), body=None + ) + if retry_change == "ambiguous-delivery": + request_error = _OPENAI_HTTPX.ReadTimeout("Acceptance unknown", request=request) + elif retry_change == "cancelled-submission": + request_error = asyncio.CancelledError() + + completed = _completed_response("resp_bg", "function_call") + retrieve = AsyncMock(return_value=response_stream_factory(completed) if mode != "non-streaming" else completed) + create = AsyncMock(side_effect=request_error) + with ( + patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve), + patch.object(client.client.responses.with_raw_response, "create", new=create), + ): + with pytest.raises( + asyncio.CancelledError if isinstance(request_error, asyncio.CancelledError) else ChatClientException + ): + await consume_response( + client.get_response( # type: ignore[call-overload] + [], stream=mode != "non-streaming", options=options, client_kwargs={"session": session} + ), + mode, + ) + session = AgentSession.from_dict(json.loads(json.dumps(session.to_dict()))) + state = session.state["_background_tool_recovery"] + messages: list[Message] = [] + if retry_change == "new-input": + messages = [Message(role="user", contents=["A different request."])] + elif retry_change == "model": + options["model"] = "another-model" + elif retry_change == "endpoint": + client.client.base_url = "https://another-provider.invalid/v1" + elif retry_change == "version": + state["version"] = 2 + elif retry_change == "budget": + state["pending"]["budget"].pop("attempt_count") + elif retry_change == "results": + state["pending"]["calls"] = {} + elif retry_change == "tool-schema": + send_email.parameters()["properties"]["to"]["type"] = "integer" + elif retry_change == "busy": + state["busy"] = True + with pytest.raises(ChatClientException): + await consume_response( + client.get_response( # type: ignore[call-overload] + messages, stream=mode != "non-streaming", options=options, client_kwargs={"session": session} + ), + mode, + ) + assert executions == ["bob"] + assert retrieve.await_count == create.await_count == 1 + + +@pytest.mark.parametrize("stream", [False, True]) +@pytest.mark.parametrize("through_agent", [False, True], ids=["direct-client", "ephemeral-agent-session"]) +async def test_background_tools_without_a_reusable_session_remain_supported(stream: bool, through_agent: bool) -> None: + executions: list[str] = [] + + @tool(approval_mode="never_require") + def send_email(to: str) -> str: + """Record a harmless local action.""" + executions.append(to) + return "sent" + + client = OpenAIChatClient(model="test-model", api_key="test-key") + agent = Agent(client=client, tools=[send_email]) + completed: Any = _completed_response("resp_bg", "function_call") + final: Any = _completed_response("resp_final", "message") + if stream: + added = MagicMock(type="response.output_item.added", output_index=0) + added.item.type = "function_call" + added.item.call_id = "call_1" + added.item.name = "send_email" + arguments = MagicMock( + type="response.function_call_arguments.delta", + output_index=0, + item_id="fc_1", + delta='{"to": "bob"}', + ) + completed = _FakeAsyncEventStream([ + added, + arguments, + _completed_response_event("resp_bg", "function_call"), + ]) + final = _FakeAsyncEventStream([_completed_response_event("resp_final", "message")]) + create = AsyncMock(side_effect=[completed, final]) + with patch.object(client.client.responses.with_raw_response, "create", new=create): + if through_agent: + if stream: + await agent.run( + "Email bob.", stream=True, options=OpenAIChatOptions(background=True) + ).get_final_response() + else: + await agent.run("Email bob.", options=OpenAIChatOptions(background=True)) + elif stream: + await client.get_response( + [Message(role="user", contents=["Email bob."])], + stream=True, + options=OpenAIChatOptions(background=True, tools=[send_email]), + ).get_final_response() + else: + await client.get_response( + [Message(role="user", contents=["Email bob."])], + options=OpenAIChatOptions(background=True, tools=[send_email]), + ) + assert executions == ["bob"] + assert create.await_count == 2 + + +@pytest.mark.parametrize("mode", ["non-streaming", "iterate", "final-only"]) +async def test_background_partial_batch_and_concurrent_poll_fail_closed( + mode: str, + consume_response: Callable[[Any, str], Awaitable[Any]], + response_stream_factory: Callable[[Any], MagicMock], +) -> None: + executions: list[str] = [] + waiting = asyncio.Event() + + @tool(approval_mode="never_require") + async def send_email(to: str) -> str: + """Record one completion and keep its sibling in flight.""" + executions.append(to) + if to == "waiting": + waiting.set() + await asyncio.Event().wait() + return "completed" + + completed = _completed_response("resp_bg", "function_call") + completed.output[0].arguments = '{"to": "completed"}' + second = copy.copy(completed.output[0]) + second.call_id = "call_2" + second.id = "fc_2" + second.arguments = '{"to": "waiting"}' + completed.output.append(second) + client = OpenAIChatClient(model="test-model", api_key="test-key") + session = AgentSession() + options = OpenAIChatOptions(continuation_token={"response_id": "resp_bg"}, tools=[send_email]) + retrieve = AsyncMock(return_value=response_stream_factory(completed) if mode != "non-streaming" else completed) + with patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve): + first_poll = asyncio.ensure_future( + consume_response( + client.get_response( # type: ignore[call-overload] + [], stream=mode != "non-streaming", options=options, client_kwargs={"session": session} + ), + mode, + ) + ) + try: + await asyncio.wait_for(waiting.wait(), timeout=5) + with pytest.raises(ChatClientInvalidRequestException, match="concurrent"): + await consume_response( + client.get_response( # type: ignore[call-overload] + [], stream=mode != "non-streaming", options=options, client_kwargs={"session": session} + ), + mode, + ) + finally: + first_poll.cancel() + with pytest.raises(asyncio.CancelledError): + await first_poll + session = AgentSession.from_dict(json.loads(json.dumps(session.to_dict()))) + with pytest.raises(ChatClientException, match="indeterminate"): + await consume_response( + client.get_response( # type: ignore[call-overload] + [], stream=mode != "non-streaming", options=options, client_kwargs={"session": session} + ), + mode, + ) + assert executions == ["completed", "waiting"] + assert retrieve.await_count == 1 + recorded_results = list(session.state["_background_tool_recovery"]["pending"]["calls"].values()) + assert recorded_results[0] is not None + assert recorded_results[1] is None + + +@pytest.mark.parametrize("mode", ["non-streaming", "iterate", "final-only"]) +async def test_background_recovery_preserves_reused_call_ids_reasoning_and_limits( + mode: str, + consume_response: Callable[[Any, str], Awaitable[Any]], + response_stream_factory: Callable[[Any], MagicMock], +) -> None: + from agent_framework import FunctionInvocationContext, FunctionMiddleware + + executions: list[str] = [] + middleware_calls: list[str] = [] + + class CountInvocations(FunctionMiddleware): + async def process(self, context: FunctionInvocationContext, call_next: Any) -> None: + middleware_calls.append(context.function.name) + await call_next() + + @tool(approval_mode="never_require") + def send_email(to: str) -> str: + """Record distinct logical tool occurrences.""" + executions.append(to) + return f"sent-{len(executions)}" + + first_response = _completed_response("resp_bg", "function_call") + first_response.output.insert( + 0, + ResponseReasoningItem( + id="rs_1", + type="reasoning", + summary=[Summary(type="summary_text", text="Reasoned before acting.")], + encrypted_content="opaque-reasoning", + ), + ) + second_response = _completed_response("resp_second", "function_call") + client = OpenAIChatClient(model="test-model", api_key="test-key", middleware=[CountInvocations()]) + client.function_invocation_configuration.update({"max_iterations": 2, "max_function_calls": 2}) + history = InMemoryHistoryProvider(load_messages=False) + agent = Agent(client=client, tools=[send_email], context_providers=[history]) + session = AgentSession() + options = OpenAIChatOptions(continuation_token={"response_id": "resp_bg"}) + request = _OPENAI_HTTPX.Request("POST", "https://unit-test.invalid/v1/responses") + error = RateLimitError("Rejected", response=_OPENAI_HTTPX.Response(429, request=request), body=None) + final_response = _completed_response("resp_final", "message") + usage = SimpleNamespace( + input_tokens=3, output_tokens=2, total_tokens=5, input_tokens_details=None, output_tokens_details=None + ) + for response in (first_response, second_response, final_response): + response.usage = usage + retrieve = AsyncMock( + return_value=response_stream_factory(first_response) if mode != "non-streaming" else first_response + ) + create = AsyncMock( + side_effect=[ + response_stream_factory(second_response) if mode != "non-streaming" else second_response, + error, + response_stream_factory(final_response) if mode != "non-streaming" else final_response, + ] + ) + with ( + patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve), + patch.object(client.client.responses.with_raw_response, "create", new=create), + ): + with pytest.raises(ChatClientException): + await consume_response( + agent.run(session=session, stream=mode != "non-streaming", options=options), # type: ignore[call-overload] + mode, + ) + session = AgentSession.from_dict(json.loads(json.dumps(session.to_dict()))) + final = await consume_response( + agent.run(session=session, stream=mode != "non-streaming", options=options), # type: ignore[call-overload] + mode, + ) + + assert executions == ["bob", "bob"] + assert middleware_calls == ["send_email", "send_email"] + assert retrieve.await_count == 1 + calls = [content for message in final.messages for content in message.contents if content.type == "function_call"] + results = [ + content for message in final.messages for content in message.contents if content.type == "function_result" + ] + assert [content.call_id for content in calls] == ["call_1", "call_1"] + assert calls[0].id != calls[1].id + assert [content.id for content in results] == [content.id for content in calls] + assert [content.result for content in results] == ["sent-1", "sent-2"] + assert "opaque-reasoning" in final.to_json() + assert final.usage_details["total_token_count"] == 15 + stored = session.state[history.source_id]["messages"] + stored_results = [ + content for message in stored for content in message.contents if content.type == "function_result" + ] + assert [content.id for content in stored_results] == [content.id for content in results] + assert [content.result for content in stored_results] == ["sent-1", "sent-2"] + assert create.await_args_list[1].kwargs["tool_choice"] == "none" + assert create.await_args_list[2].kwargs["tool_choice"] == "none" + assert create.await_args_list[2].kwargs["previous_response_id"] == "resp_second" + assert [ + item["output"] + for request_call in create.await_args_list + for item in request_call.kwargs["input"] + if item["type"] == "function_call_output" + ] == ["sent-1", "sent-2", "sent-2"] + + +async def test_background_recovery_rejects_retry_after_partial_submission_stream() -> None: + executions: list[str] = [] + + @tool(approval_mode="never_require") + def send_email(to: str) -> str: + """Record a harmless local action.""" + executions.append(to) + return "sent-once" + + client = OpenAIChatClient(model="test-model", api_key="test-key") + session = AgentSession() + options = OpenAIChatOptions(continuation_token={"response_id": "resp_bg"}, tools=[send_email]) + request = _OPENAI_HTTPX.Request("POST", "https://unit-test.invalid/v1/responses") + error = RateLimitError("Interrupted", response=_OPENAI_HTTPX.Response(429, request=request), body=None) + + async def interrupted_events() -> AsyncIterable[Any]: + yield MagicMock(type="response.output_text.delta", delta="Partial reply", logprobs=None) + raise error + + parsed_stream = AsyncMock() + parsed_stream.__aenter__.return_value = parsed_stream + parsed_stream.__aiter__.side_effect = interrupted_events + raw_stream = MagicMock(headers={}) + raw_stream.parse.return_value = parsed_stream + retrieve = AsyncMock(return_value=_completed_response("resp_bg", "function_call")) + create = AsyncMock(side_effect=[error, raw_stream]) + with ( + patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve), + patch.object(client.client.responses.with_raw_response, "create", new=create), + ): + with pytest.raises(ChatClientException): + await client.get_response([], options=options, client_kwargs={"session": session}) + response_stream = client.get_response([], stream=True, options=options, client_kwargs={"session": session}) + updates: list[ChatResponseUpdate] = [] + with pytest.raises(ChatClientException): + async for update in response_stream: + updates.append(update) + assert any(update.text == "Partial reply" for update in updates) + session = AgentSession.from_dict(json.loads(json.dumps(session.to_dict()))) + with pytest.raises(ChatClientException, match="indeterminate"): + await client.get_response([], options=options, client_kwargs={"session": session}) + assert executions == ["bob"] + assert retrieve.await_count == 1 + assert create.await_count == 2 + + +@pytest.mark.parametrize("stream", [False, True]) +async def test_background_recovery_accepts_refreshed_todo_context(stream: bool) -> None: + from agent_framework import TodoProvider + + client = OpenAIChatClient(model="test-model", api_key="test-key") + provider = TodoProvider() + agent = Agent(client=client, context_providers=[provider]) + session = agent.create_session() + completed = _completed_response("resp_bg", "function_call") + completed.output[0].name = "todos_add" + completed.output[0].arguments = '{"todos": [{"title": "Deliver once"}]}' + request = _OPENAI_HTTPX.Request("POST", "https://unit-test.invalid/v1/responses") + error = RateLimitError("Rejected", response=_OPENAI_HTTPX.Response(429, request=request), body=None) + final: Any = _completed_response("resp_next", "message") + if stream: + final = _FakeAsyncEventStream([_completed_response_event("resp_next", "message")]) + create = AsyncMock(side_effect=[error, final]) + retrieve = AsyncMock(return_value=completed) + options = OpenAIChatOptions(continuation_token={"response_id": "resp_bg"}) + with ( + patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve), + patch.object(client.client.responses.with_raw_response, "create", new=create), + ): + with pytest.raises(ChatClientException): + await agent.run(session=session, options=options) + session = AgentSession.from_dict(json.loads(json.dumps(session.to_dict()))) + if stream: + response = await agent.run(session=session, stream=True, options=options).get_final_response() + else: + response = await agent.run(session=session, options=options) + assert response.continuation_token is None + assert retrieve.await_count == 1 + assert create.await_args_list[0].kwargs["input"] == create.await_args_list[1].kwargs["input"] + assert len(session.state[provider.source_id]["items"]) == 1 + + +@pytest.mark.parametrize("stream", [False, True]) +@pytest.mark.parametrize("alias", ["conversation_id", "conversation", "conversation-object"]) +async def test_background_recovery_preserves_conversation_mode(stream: bool, alias: str) -> None: + executions: list[str] = [] + + @tool(approval_mode="never_require") + def send_email(to: str) -> str: + """Record a harmless action.""" + executions.append(to) + return "sent-once" + + client = OpenAIChatClient(model="test-model", api_key="test-key") + session = AgentSession() + options = cast( + OpenAIChatOptions, + { + "continuation_token": {"response_id": "resp_bg"}, + "conversation" if alias == "conversation-object" else alias: ( + {"id": "conv_original"} if alias == "conversation-object" else "conv_original" + ), + "tools": [send_email], + }, + ) + completed = _completed_response("resp_bg", "function_call") + completed.conversation = SimpleNamespace(id="conv_original") + request = _OPENAI_HTTPX.Request("POST", "https://unit-test.invalid/v1/responses") + error = RateLimitError("Rejected", response=_OPENAI_HTTPX.Response(429, request=request), body=None) + final: Any = _completed_response("resp_next", "message") + if stream: + final = _FakeAsyncEventStream([_completed_response_event("resp_next", "message")]) + create = AsyncMock(side_effect=[error, final]) + retrieve = AsyncMock(return_value=completed) + with ( + patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve), + patch.object(client.client.responses.with_raw_response, "create", new=create), + ): + with pytest.raises(ChatClientException): + await client.get_response([], options=options, client_kwargs={"session": session}) + session = AgentSession.from_dict(json.loads(json.dumps(session.to_dict()))) + if stream: + await client.get_response( + [], stream=True, options=options, client_kwargs={"session": session} + ).get_final_response() + else: + await client.get_response([], options=options, client_kwargs={"session": session}) + assert executions == ["bob"] + for submission in create.await_args_list: + conversation = submission.kwargs["conversation"] + assert (conversation["id"] if isinstance(conversation, dict) else conversation) == "conv_original" + assert "previous_response_id" not in submission.kwargs + + +@pytest.mark.parametrize("stream", [False, True]) +async def test_background_token_only_poll_retains_model_override(stream: bool) -> None: + @tool(approval_mode="never_require") + def send_email(to: str) -> str: + """Simulate an email.""" + return "sent-once" + + client = OpenAIChatClient(model="default-model", api_key="test-key") + session = AgentSession() + queued = _completed_response("resp_bg", "message") + queued.status = "queued" + queued.output = [] + request = _OPENAI_HTTPX.Request("POST", "https://unit-test.invalid/v1/responses") + error = RateLimitError("Rejected", response=_OPENAI_HTTPX.Response(429, request=request), body=None) + final: Any = _completed_response("resp_next", "message") + if stream: + final = _FakeAsyncEventStream([_completed_response_event("resp_next", "message")]) + create = AsyncMock(side_effect=[queued, error, final]) + retrieve = AsyncMock(return_value=_completed_response("resp_bg", "function_call")) + with ( + patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve), + patch.object(client.client.responses.with_raw_response, "create", new=create), + ): + initial = await client.get_response( + [Message("user", ["Email bob."])], + options={"background": True, "model": "override-model", "tools": [send_email]}, + client_kwargs={"session": session}, + ) + assert initial.continuation_token is not None + options = OpenAIChatOptions( + continuation_token=cast(OpenAIContinuationToken, initial.continuation_token), tools=[send_email] + ) + with pytest.raises(ChatClientException): + await client.get_response([], options=options, client_kwargs={"session": session}) + session = AgentSession.from_dict(json.loads(json.dumps(session.to_dict()))) + if stream: + await client.get_response( + [], stream=True, options=options, client_kwargs={"session": session} + ).get_final_response() + else: + await client.get_response([], options=options, client_kwargs={"session": session}) + assert retrieve.await_count == 1 + assert [call.kwargs["model"] for call in create.await_args_list] == ["override-model"] * 3 + + +@pytest.mark.parametrize("change", ["model", "endpoint"]) +async def test_background_without_execution_evidence_can_change_scope(change: str) -> None: + client = OpenAIChatClient(model="first-model", api_key="test-key") + session = AgentSession() + create = AsyncMock(return_value=_completed_response("resp_done", "message")) + with patch.object(client.client.responses.with_raw_response, "create", new=create): + await client.get_response( + [Message("user", ["First request"])], options={"background": True}, client_kwargs={"session": session} + ) + if change == "model": + client.model = "second-model" + else: + client.client.base_url = "https://other.invalid/v1" + await client.get_response( + [Message("user", ["Unrelated request"])], options={"background": True}, client_kwargs={"session": session} + ) + assert create.await_count == 2 + + +@pytest.mark.parametrize("stream", [False, True]) +async def test_background_recovery_survives_local_request_validation_failure(stream: bool) -> None: + executions: list[str] = [] + + @tool(approval_mode="never_require") + def send_email(to: str) -> str: + """Record one harmless action.""" + executions.append(to) + return "sent-once" + + client = OpenAIChatClient(model="test-model", api_key="test-key") + session = AgentSession() + options = OpenAIChatOptions(continuation_token={"response_id": "resp_bg"}, tools=[send_email]) + request = _OPENAI_HTTPX.Request("POST", "https://unit-test.invalid/v1/responses") + error = RateLimitError("Rejected", response=_OPENAI_HTTPX.Response(429, request=request), body=None) + final: Any = _completed_response("resp_next", "message") + if stream: + final = _FakeAsyncEventStream([_completed_response_event("resp_next", "message")]) + create = AsyncMock(side_effect=[error, final]) + retrieve = AsyncMock(return_value=_completed_response("resp_bg", "function_call")) + with ( + patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve), + patch.object(client.client.responses.with_raw_response, "create", new=create), + ): + with pytest.raises(ChatClientException): + await client.get_response([], options=options, client_kwargs={"session": session}) + invalid_options = cast(OpenAIChatOptions, {**options, "response_format": 123}) + with pytest.raises(ChatClientInvalidRequestException, match="response_format"): + if stream: + await client.get_response( + [], stream=True, options=invalid_options, client_kwargs={"session": session} + ).get_final_response() + else: + await client.get_response([], options=invalid_options, client_kwargs={"session": session}) + assert create.await_count == 1 + if stream: + await client.get_response( + [], stream=True, options=options, client_kwargs={"session": session} + ).get_final_response() + else: + await client.get_response([], options=options, client_kwargs={"session": session}) + assert executions == ["bob"] + assert create.await_count == 2 + assert retrieve.await_count == 1 + + +async def test_background_stream_close_after_tool_result_keeps_delivery_ready() -> None: + executions: list[str] = [] + + @tool(approval_mode="never_require") + def send_email(to: str) -> str: + """Record a harmless action.""" + executions.append(to) + return "sent-once" + + client = OpenAIChatClient(model="test-model", api_key="test-key") + session = AgentSession() + options = OpenAIChatOptions(continuation_token={"response_id": "resp_bg"}, tools=[send_email]) + added = MagicMock(type="response.output_item.added", output_index=0) + added.item.type = "function_call" + added.item.call_id = "call_1" + added.item.name = "send_email" + arguments = MagicMock( + type="response.function_call_arguments.delta", output_index=0, item_id="fc_1", delta='{"to": "bob"}' + ) + retrieve = AsyncMock( + return_value=_FakeAsyncEventStream([added, arguments, _completed_response_event("resp_bg", "function_call")]) + ) + create = AsyncMock(return_value=_completed_response("resp_next", "message")) + with ( + patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve), + patch.object(client.client.responses.with_raw_response, "create", new=create), + ): + response_stream = client.get_response([], stream=True, options=options, client_kwargs={"session": session}) + async for update in response_stream: + if any(content.type == "function_result" for content in update.contents): + break + await response_stream.close() + assert create.await_count == 0 + session = AgentSession.from_dict(json.loads(json.dumps(session.to_dict()))) + response = await client.get_response([], options=options, client_kwargs={"session": session}) + assert response.continuation_token is None + assert executions == ["bob"] + assert retrieve.await_count == create.await_count == 1 + + +@pytest.mark.parametrize("stream", [False, True]) +@pytest.mark.parametrize("change", ["add", "remove"]) +async def test_background_recovery_restores_dynamic_tool_registry(stream: bool, change: str) -> None: + from agent_framework import FunctionInvocationContext + + @tool(approval_mode="never_require") + def extra_tool() -> str: + """Offer a dynamically exposed capability.""" + return "extra" + + @tool(approval_mode="never_require") + def send_email(to: str, context: FunctionInvocationContext) -> str: + """Change tool exposure while simulating an email.""" + if change == "add": + context.add_tools(extra_tool) + else: + context.remove_tools(extra_tool) + return "sent-once" + + client = OpenAIChatClient(model="test-model", api_key="test-key") + session = AgentSession() + options = OpenAIChatOptions( + continuation_token={"response_id": "resp_bg"}, + tools=[send_email] if change == "add" else [send_email, extra_tool], + ) + request = _OPENAI_HTTPX.Request("POST", "https://unit-test.invalid/v1/responses") + error = RateLimitError("Rejected", response=_OPENAI_HTTPX.Response(429, request=request), body=None) + final: Any = _completed_response("resp_next", "message") + if stream: + final = _FakeAsyncEventStream([_completed_response_event("resp_next", "message")]) + create = AsyncMock(side_effect=[error, final]) + retrieve = AsyncMock(return_value=_completed_response("resp_bg", "function_call")) + with ( + patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve), + patch.object(client.client.responses.with_raw_response, "create", new=create), + ): + with pytest.raises(ChatClientException): + await client.get_response([], options=options, client_kwargs={"session": session}) + session = AgentSession.from_dict(json.loads(json.dumps(session.to_dict()))) + if change == "add": + with pytest.raises(ChatClientInvalidRequestException, match="tool registry"): + await client.get_response([], options=options, client_kwargs={"session": session}) + options["tools"] = [send_email, extra_tool] + if stream: + await client.get_response( + [], stream=True, options=options, client_kwargs={"session": session} + ).get_final_response() + else: + await client.get_response([], options=options, client_kwargs={"session": session}) + expected_names = ["send_email", "extra_tool"] if change == "add" else ["send_email"] + assert [[item["name"] for item in call.kwargs["tools"]] for call in create.await_args_list] == [expected_names] * 2 + assert retrieve.await_count == 1 + + +async def test_background_stream_close_releases_poll_ownership() -> None: + client = OpenAIChatClient(model="test-model", api_key="test-key") + session = AgentSession() + options = OpenAIChatOptions(continuation_token={"response_id": "resp_bg"}) + event = MagicMock(type="response.output_text.delta", delta="First part", logprobs=None) + retrieve = AsyncMock( + side_effect=[ + _FakeAsyncEventStream([event, _completed_response_event("resp_bg", "message")]), + _completed_response("resp_bg", "message"), + ] + ) + with patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve): + response_stream = client.get_response([], stream=True, options=options, client_kwargs={"session": session}) + async for _ in response_stream: + break + await response_stream.close() + final = await client.get_response([], options=options, client_kwargs={"session": session}) + assert final.text == "Email sent." + assert retrieve.await_count == 2 + + +@pytest.mark.parametrize("stream", [False, True]) +async def test_background_recovery_preserves_explicit_approval_resume(stream: bool) -> None: + executions: list[str] = [] + + @tool(approval_mode="always_require") + def send_email(to: str) -> str: + """Record a harmless action after explicit approval.""" + executions.append(to) + return "approved-once" + + client = OpenAIChatClient(model="test-model", api_key="test-key") + session = AgentSession() + retrieve = AsyncMock(return_value=_completed_response("resp_bg", "function_call")) + create = AsyncMock(return_value=_completed_response("resp_done", "message")) + if stream: + create.return_value = _FakeAsyncEventStream([ + MagicMock(type="response.output_text.delta", delta="Email sent.", logprobs=None), + _completed_response_event("resp_done", "message"), + ]) + with ( + patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve), + patch.object(client.client.responses.with_raw_response, "create", new=create), + ): + paused = await client.get_response( + [], + options=OpenAIChatOptions(continuation_token={"response_id": "resp_bg"}, tools=[send_email]), + client_kwargs={"session": session}, + ) + assert executions == [] + request = next( + content + for message in paused.messages + for content in message.contents + if content.type == "function_approval_request" + ) + messages = [Message(role="user", contents=[request.to_function_approval_response(approved=True)])] + options = OpenAIChatOptions(background=True, tools=[send_email], conversation_id="resp_bg") + if stream: + final = await client.get_response( + messages, stream=True, options=options, client_kwargs={"session": session} + ).get_final_response() + else: + final = await client.get_response(messages, options=options, client_kwargs={"session": session}) + assert final.text == "Email sent." + assert executions == ["bob"] + assert retrieve.await_count == create.await_count == 1 + + +@pytest.mark.parametrize("mode", ["non-streaming", "iterate", "final-only"]) +async def test_background_recovery_preserves_approved_results_in_history( + mode: str, + consume_response: Callable[[Any, str], Awaitable[Any]], + response_stream_factory: Callable[[Any], MagicMock], +) -> None: + executions: list[str] = [] + + @tool(approval_mode="always_require") + def send_email(to: str) -> str: + """Record an approved action.""" + executions.append(f"send:{to}") + return "approved-once" + + @tool(approval_mode="never_require") + def record_delivery(to: str) -> str: + """Record a later automatic action.""" + executions.append(f"record:{to}") + return "recorded-once" + + client = OpenAIChatClient(model="test-model", api_key="test-key") + history = InMemoryHistoryProvider() + agent = Agent(client=client, tools=[send_email, record_delivery], context_providers=[history]) + session = AgentSession() + retrieve = AsyncMock(return_value=_completed_response("resp_bg", "function_call")) + second_response = _completed_response("resp_second", "function_call") + second_response.output[0].name = "record_delivery" + second_response.output[0].call_id = "call_2" + second_response.output[0].id = "fc_2" + final_response = _completed_response("resp_done", "message") + request = _OPENAI_HTTPX.Request("POST", "https://unit-test.invalid/v1/responses") + error = RateLimitError("Rejected", response=_OPENAI_HTTPX.Response(429, request=request), body=None) + create = AsyncMock( + side_effect=[ + response_stream_factory(second_response) if mode != "non-streaming" else second_response, + error, + response_stream_factory(final_response) if mode != "non-streaming" else final_response, + ] + ) + with ( + patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve), + patch.object(client.client.responses.with_raw_response, "create", new=create), + ): + paused = await agent.run( + session=session, options=OpenAIChatOptions(continuation_token={"response_id": "resp_bg"}) + ) + approval = next( + content + for message in paused.messages + for content in message.contents + if content.type == "function_approval_request" + ) + assert executions == [] + messages = [Message(role="user", contents=[approval.to_function_approval_response(approved=True)])] + options = OpenAIChatOptions(background=True, conversation_id="resp_bg") + with pytest.raises(ChatClientException): + await consume_response( + agent.run( # type: ignore[call-overload] + messages, session=session, stream=mode != "non-streaming", options=options + ), + mode, + ) + assert executions == ["send:bob", "record:bob"] + session = AgentSession.from_dict(json.loads(json.dumps(session.to_dict()))) + options["continuation_token"] = {"response_id": "resp_second"} + final = await consume_response( + agent.run( # type: ignore[call-overload] + messages, session=session, stream=mode != "non-streaming", options=options + ), + mode, + ) + + expected_results = [("call_1", "approved-once"), ("call_2", "recorded-once")] + results = [ + content for message in final.messages for content in message.contents if content.type == "function_result" + ] + assert [(content.call_id, content.result) for content in results] == expected_results + stored = session.state[history.source_id]["messages"] + stored_calls = [content for message in stored for content in message.contents if content.type == "function_call"] + stored_results = [ + content for message in stored for content in message.contents if content.type == "function_result" + ] + assert [(content.call_id, content.result) for content in stored_results] == expected_results + assert [content.id for content in stored_calls] == [content.id for content in stored_results] + assert executions == ["send:bob", "record:bob"] + assert retrieve.await_count == 1 + assert create.await_count == 3 + + +async def test_background_recovery_capacity_does_not_evict_consumed_tokens() -> None: + executions: list[str] = [] + + @tool(approval_mode="never_require") + def send_email(to: str) -> str: + """Record a harmless local action.""" + executions.append(to) + return "sent-once" + + client = OpenAIChatClient(model="test-model", api_key="test-key") + session = AgentSession() + options = OpenAIChatOptions(continuation_token={"response_id": "resp_bg"}, tools=[send_email]) + retrieve = AsyncMock(return_value=_completed_response("resp_bg", "function_call")) + create = AsyncMock(return_value=_completed_response("resp_done", "message")) + with ( + patch.object(client.client.responses.with_raw_response, "retrieve", new=retrieve), + patch.object(client.client.responses.with_raw_response, "create", new=create), + ): + await client.get_response([], options=options, client_kwargs={"session": session}) + handled = session.state["_background_tool_recovery"]["handled"] + handled.extend(f"resp_old_{index}" for index in range(255)) + with pytest.raises(ChatClientInvalidRequestException, match="stale"): + await client.get_response([], options=options, client_kwargs={"session": session}) + options["continuation_token"] = {"response_id": "resp_new"} + retrieve.return_value = _completed_response("resp_new", "function_call") + with pytest.raises(ChatClientInvalidRequestException, match="capacity"): + await client.get_response([], options=options, client_kwargs={"session": session}) + assert executions == ["bob"] + assert "resp_bg" in session.state["_background_tool_recovery"]["handled"] + assert retrieve.await_count == 2 + assert create.await_count == 1 + + async def test_streaming_resume_drops_continuation_token_before_the_terminal_update() -> None: """A consumer that stops at the terminal update must not leave the token in its options.""" retrieve = AsyncMock( diff --git a/python/samples/02-agents/background_responses.py b/python/samples/02-agents/background_responses.py index f6adc30ea3a..75af6bfc904 100644 --- a/python/samples/02-agents/background_responses.py +++ b/python/samples/02-agents/background_responses.py @@ -90,6 +90,7 @@ async def streaming_with_resumption() -> None: if chunk_count >= 3: print("\n [simulated interruption]") break + await stream.close() # 4. Resume from the last continuation token. if last_token is not None: diff --git a/skills/agent-framework-builder-python/references/agent-builder.md b/skills/agent-framework-builder-python/references/agent-builder.md index b90e83a99a5..898d5c59f24 100644 --- a/skills/agent-framework-builder-python/references/agent-builder.md +++ b/skills/agent-framework-builder-python/references/agent-builder.md @@ -65,6 +65,9 @@ Create an `AgentSession` through the agent and pass the same session to each run - Treat persisted session state as sensitive data. - Read [session documentation](https://learn.microsoft.com/agent-framework/concepts/agents/conversations/session) for serialization and provider-specific continuation behavior. +- For background Responses calls that execute local tools, retain the same session and continuation token across polls + and supported result-delivery retries. Calls without a reusable session remain supported but have no cross-poll + recovery or replay protection. Re-supply dynamically added tools on retry; missing or changed definitions are rejected. ## History