diff --git a/docs/decisions/0055-python-foundry-hosting-contracts.md b/docs/decisions/0055-python-foundry-hosting-contracts.md index 2115cd476a4..8a99f2a94a6 100644 --- a/docs/decisions/0055-python-foundry-hosting-contracts.md +++ b/docs/decisions/0055-python-foundry-hosting-contracts.md @@ -170,9 +170,20 @@ approvals and rejections. Exact pre-header HTTP 400 behavior remains blocked on validation hook under #9537; the native approval migration does not claim to implement that requirement. Represent invocation-time OAuth consent as structured consent with incomplete response status. -Pair each emitted function call with one terminal result, preserve completed sibling results, and stop -later protected calls. Require `store=true` so continuation can retain the required state; +Pair each OAuth-interrupted invocation with one terminal result and preserve every authorized batch result. +OAuth consent does not suspend sibling calls: let the batch finish under its configured concurrency, +then return all consent requests with the complete results. Existing pre-execution approval gating +remains unchanged; eager execution before approval is separate vNext work tracked in #10334. +Require `store=true` so continuation can retain the required state; do not support ephemeral consent under `store=false`. +Preserve results rather than cancelling or automatically replaying completed calls. +For inference-service history, reconcile a fully terminal batch before publishing consent and retain the accepted +continuation even when the service's reconciliation output is rejected. When another call still requires approval +or other input, publish both requests and retain completed results and unsent caller input in the native pending +batch until that call is resolved. Submit the real results together on continuation without replaying completed +calls or fabricating a terminal result for a pending call. +This behavior depends on the coordinated native Core change; select and validate the released Core +dependency floor during release preparation rather than adding a hosting-only compatibility fallback. ### Unresolved boundaries diff --git a/docs/specs/004-python-function-calling-loop.md b/docs/specs/004-python-function-calling-loop.md index 646b36748a7..697ae435808 100644 --- a/docs/specs/004-python-function-calling-loop.md +++ b/docs/specs/004-python-function-calling-loop.md @@ -397,6 +397,23 @@ that manually replay messages own the equivalent rule: do not resend an approval cancellation remains cancellation rather than becoming this provider signal. - Every actionable local `function_call` produces exactly one terminal `function_result`, unless execution pauses for a new user-input request or the run is aborted by a fail-closed error. +- A tool-raised OAuth consent request is a terminally paired interruption, not an unresolved function call. + Return one consent-required result for its call and preserve the ordered results of the complete authorized batch. + OAuth consent alone does not suspend or cancel sibling calls, including MCP calls waiting for admission. Let + eligible calls finish under the configured concurrency, collect all consent requests, and count attempted calls + normally. Do not automatically replay protected operations. Existing pre-execution approval gating is unchanged, + and provider-issued consent keeps its separate ownership. + With service-owned history and a fully terminal batch, submit the complete locally produced result batch once + in the foreground with tools disabled before publishing the consent pause, advance continuation state, and account + for usage without exposing the model's reconciliation prose. If another call still requires approval or other user + input, return the requests together and retain completed results and unsent caller input in the native pending + batch instead. Submit all real results once every pending call is resolved; do not reconcile partial outputs, + fabricate a result for a pending call, or replay a completed sibling. Manual-history callers retain the equivalent + pending transcript. This applies to ordinary execution and approved-tool replay in both response modes. + Unfinished reconciliation responses and failures propagate; an accepted service continuation remains advanced if + its returned output is rejected. Cancellation or an ambiguous transport failure does not authorize an automatic retry. + Approval-time reconciliation preserves new caller input alongside normalized results without sending OAuth + actions or replaying resolved local approval controls to the service. - An ordinary exception raised by function middleware or a tool body becomes one terminal error `function_result` and the loop continues. `MiddlewareFailure` is the execution-time fail-closed escape: it is never converted into a tool result, the in-flight parallel batch is cancelled, no further tool call starts, no further model turn is @@ -645,7 +662,7 @@ that manually replay messages own the equivalent rule: do not resend an approval | Foundry native approval publication | Only requests matching native pending state are published, after successful native finalization and session persistence; failures publish no approval item. | `packages/foundry_hosting/tests/test_responses.py::TestFunctionApprovalRoundTrip::test_non_streaming_emits_mcp_approval_request_and_persists_native_session`, `test_streaming_emits_mcp_approval_request_and_persists_native_session`, `test_custom_agent_cannot_publish_approval_without_matching_native_state`, `test_failed_native_finalization_does_not_publish_approval`, `test_session_save_failure_does_not_publish_resumable_approval` | | Foundry deferred approval output | Output order is preserved across finalization; count and serialized-byte limits fail explicitly without publishing the buffered approval, and an overflowing producer is closed in both response modes. | `packages/foundry_hosting/tests/test_responses.py::TestFunctionApprovalRoundTrip::test_deferred_approval_preserves_streamed_output_order`, `test_deferred_approval_output_has_count_and_byte_limits`, `test_deferred_approval_output_enforces_default_limits`, `test_deferred_approval_limit_closes_an_unending_stream` | | Agent-tool child approval | A child `ToolApprovalMiddleware` can auto-approve runtime tool requests inside one delegated invocation; unresolved child approvals execute nothing and do not enter the caller's approval state or dispatch a same-named parent tool, including with propagated application state. Mixed batches preserve non-approval user-input requests. Shared parent and child state rejects overlapping `ToolApprovalMiddleware.source_id` values before running the child; parent approval state, including state of middleware passed to a single parent run, is neither visible nor mutable inside the child (a child cannot add standing rules or forge a decision for a pending request), and custom child approval queues do not leak into later delegations. | `packages/core/tests/core/test_agents.py::test_chat_agent_as_tool_auto_approves_child_tool_with_middleware`, `test_chat_agent_as_tool_fails_closed_for_unresolved_child_approval`, `test_chat_agent_as_tool_child_approval_does_not_dispatch_same_named_parent_tool`, `test_chat_agent_as_tool_preserves_non_approval_requests_from_mixed_child_batch`, `test_chat_agent_as_tool_shared_session_requires_distinct_tool_approval_source_ids`, `test_chat_agent_as_tool_isolates_custom_parent_approval_state`, `test_chat_agent_as_tool_isolates_run_level_parent_approval_state`, `test_chat_agent_as_tool_run_level_approval_state_cannot_authorize_pending_request`, `test_chat_agent_as_tool_run_level_approval_ownership_is_per_invocation`, `test_chat_agent_as_tool_approved_delegation_does_not_confuse_framework_approval_state`, `test_chat_agent_as_tool_does_not_restore_custom_approval_queue_on_fresh_delegation` | -| Approval-time user input | Every user-input request from one approved execution returns in order with assistant role and no extra model call; the execution consumes one call-budget unit. | `packages/core/tests/core/test_harness_tool_approval.py::test_approval_resume_returns_all_user_input_requests_without_another_model_call`, `packages/core/tests/core/test_function_invocation_logic.py::test_approval_resume_user_input_counts_toward_function_call_budget` | +| Approval-time user input | Every user-input request from one approved execution returns in order with assistant role; OAuth also retains its paired tool-role result. Local history needs no extra model call; service-owned OAuth reconciliation follows the contract below. The execution consumes one call-budget unit. | `packages/core/tests/core/test_harness_tool_approval.py::test_approval_resume_returns_all_user_input_requests_without_another_model_call`, `packages/core/tests/core/test_function_invocation_logic.py::test_approval_resume_user_input_counts_toward_function_call_budget` | | Mixed terminal result and follow-up input | Completed siblings remain tool-role while only follow-up input requests use assistant-role messages/updates. | `packages/core/tests/core/test_function_invocation_logic.py::test_approval_resume_separates_terminal_results_from_follow_up_requests`, `packages/openai/tests/openai/test_openai_chat_completion_client.py::test_mixed_approval_resume_roles_serialize_function_result_as_tool`, `packages/core/tests/core/test_harness_tool_approval.py::test_dynamic_policy_approval_partitions_safe_sibling_result_roles` | | Approval-time middleware termination | Terminal result returns with no extra model call in either response mode. | `packages/core/tests/core/test_function_invocation_logic.py::test_approval_resume_honors_middleware_termination` | | Approval re-entry after iteration budget | Pending approved calls resolve once even when prior model calls consumed `max_iterations`. | `packages/core/tests/core/test_harness_tool_approval.py::test_auto_approval_resolves_after_iteration_budget_is_exhausted` | @@ -736,6 +753,9 @@ that manually replay messages own the equivalent rule: do not resend an approval | Pre-validation middleware repair | Schema-compatible calls retain normalized middleware arguments without duplicate validation. Prepared-value reuse does not require validator outputs to be copyable, and unchanged NaNs remain stable. Identity-only opaque values may be reused for ordinary execution, but security authority rejects them before downstream middleware or the tool body because in-place mutation cannot be detected safely. When provisional validation fails, function middleware observes raw parsed arguments, may repair them before final validation, and the body receives normalized values; short-circuiting skips final validation and execution. Repair after security middleware fails closed using recursive type-aware, float-bit-exact comparison, including invalid and short-circuited mutations. Security inspects exact normalized values, and validation errors after hidden-value resolution do not disclose resolved values or mapping keys, including validator `TypeError` paths. | `test_function_middleware_keeps_normalized_arguments_for_valid_calls`, `test_prepared_arguments_support_noncopyable_validator_output`, `test_nan_prepared_and_approval_snapshots_are_stable`, `test_function_middleware_repairs_raw_arguments_before_validation`, `test_function_middleware_can_short_circuit_before_argument_validation`, `test_invalid_arguments_produced_by_middleware_keep_argument_error_contract`, `packages/core/tests/test_security.py::TestVariableArgumentPolicy::test_security_rejects_opaque_mutable_validator_output`, `test_argument_mutation_after_security_middleware_fails_closed`, `test_security_snapshot_accepts_unchanged_nan`, `test_argument_mutation_after_security_short_circuit_fails_closed`, `test_security_policy_observes_custom_validator_transform_once`, `test_hidden_argument_validation_error_does_not_disclose_resolved_value`, `test_hidden_mapping_key_is_not_disclosed_by_validation_error`, `test_hidden_value_is_not_disclosed_by_validator_type_error`, `test_hidden_argument_can_be_normalized_after_security_check` | | Approved middleware repair | Approval binds to the normalized middleware-entry representation, so ordinary Pydantic coercion still completes in one approval round. A changed approval-bound call executes zero times under the old grant, returns a persisted occurrence-bound replacement request in both response modes, and executes once only after the replacement is approved. Recursive type-aware, float-bit-exact comparison treats booleans and numbers, and positive and negative zero, as distinct while keeping unchanged NaNs stable; set elements and mapping entries are counted, so removing one of several NaNs is detected. Opaque mutable normalized values fail closed before approval authority is established. The replacement holds JSON-native arguments that survive session serialization, and a repair whose JSON form would prepare to different values fails closed. The same replacement rule applies when middleware short-circuits instead of calling the tool. Security expansion preserves approval-visible placeholders. | `test_approved_coercing_arguments_execute_without_replacement`, `test_approved_argument_repair_requires_replacement_approval`, `test_approved_argument_repair_short_circuit_requires_replacement_approval`, `test_approval_snapshot_distinguishes_exact_values`, `test_approval_rejects_opaque_mutable_validator_output`, `test_approval_detects_changes_to_converted_values`, `test_approval_change_that_json_cannot_represent_fails_closed`, `test_argument_tokens_count_items_that_share_a_token`, `test_approved_repair_replacement_round_trips_through_json`, `packages/core/tests/test_security.py::TestVariableArgumentPolicy::test_hidden_argument_resolution_does_not_require_reapproval` | | Approved success | Successful approved execution returns one result. | `test_approved_function_call_successful_execution` | +| Invocation-time OAuth consent | Ordinary and approved batches finish under configured concurrency, retain one terminal result per attempted call, collect all consent requests in model order, preserve successful and failed sibling results, and count every attempt normally. Consent does not block queued MCP siblings or change pre-execution approval gating. | `packages/core/tests/core/test_function_invocation_logic.py::test_oauth_consent_preserves_all_batch_results`, `test_oauth_consent_preserves_already_dispatched_sibling`, `test_oauth_consent_supplied_terminal_result_stays_single_and_immutable`, `packages/foundry_hosting/tests/test_toolbox.py::test_toolbox_oauth_consent_allows_an_internally_queued_mcp_dispatch`, `test_toolbox_oauth_consent_remains_distinct_from_native_policy_approval`, `test_toolbox_oauth_consent_collects_requests_and_preserves_sibling_results` | +| OAuth service-history reconciliation | Submit a fully terminal batch once before exposing consent, advance the service continuation in streaming and final output, and preserve the accepted continuation if new service tool calls are rejected. A mixed pending batch retains completed results, unsent input, and its budget until the remaining approval/input is resolved, then submits all real results without replay. Failures and cancellation never publish a successful consent pause or retry settlement. Provider-owned consent is not locally settled again. | `packages/core/tests/core/test_function_invocation_logic.py::test_oauth_consent_reconciles_real_results_and_advances_service_history`, `test_oauth_mixed_pause_retains_partial_input_and_budget`, `test_oauth_consent_settlement_failure_withholds_user_input`, `test_provider_oauth_consent_is_not_locally_settled`, `packages/foundry_hosting/tests/test_toolbox.py::test_toolbox_oauth_consent_pairs_results_and_continues`, `test_toolbox_oauth_consent_preserves_a_mixed_pending_batch`, `test_toolbox_oauth_consent_reconciliation_failure_is_not_an_incomplete_response`, `test_toolbox_oauth_consent_background_run_reconciles_in_foreground` | +| Foundry Toolbox OAuth boundary | Only validated structured Toolbox consent becomes a consent action. Generic MCP and malformed failures keep their error contract, links obey host policy, unstored consent fails explicitly without persistence, and stored response/conversation continuation does not replay completed siblings. | `packages/foundry_hosting/tests/test_toolbox.py::test_toolbox_oauth_consent_does_not_classify_unvalidated_errors`, `test_generic_mcp_does_not_translate_toolbox_consent`, `test_toolbox_oauth_consent_origin_allowlist_is_enforced`, `test_toolbox_oauth_consent_rejects_unstored_requests`, `test_toolbox_oauth_consent_pairs_results_and_continues` | | Consecutive error cap | Error threshold stops repeated failures, submits collected results, and makes only the required final no-tool model call. | `test_function_invocation_config_max_consecutive_errors`, `test_streaming_function_invocation_config_max_consecutive_errors`, `test_approval_resume_error_limit_forces_final_no_tool_response` | | Unknown call handling | Configured false returns an error result; configured true raises before any call executes. A service-managed continuation settles every call in the aborted batch and advances to the settlement response before the `KeyError` propagates; without one, including local per-service-call history persistence, no extra request is made. | `test_function_invocation_config_terminate_on_unknown_calls_false`, `test_function_invocation_config_terminate_on_unknown_calls_true`, streaming equivalents, `test_fatal_unknown_settles_service_conversation`, `test_fatal_unknown_with_local_history_makes_no_settlement_request` | | Middleware termination | Normal non-approval loop stops without a second model call. | `test_terminate_loop_single_function_call`, `test_terminate_loop_multiple_function_calls_one_terminates`, `test_terminate_loop_streaming_single_function_call` | @@ -775,7 +795,7 @@ that manually replay messages own the equivalent rule: do not resend an approval | DevUI conversation-bound approval ownership | A pending function approval is bound to the conversation that originated it. A response from another conversation is rejected without consuming the pending approval, and identical request IDs in different conversations remain independent. | `packages/devui/tests/devui/test_approval_validation.py::test_approval_request_must_be_bound_to_originating_conversation`, `test_same_approval_request_id_is_scoped_per_conversation` | | AG-UI approval result event | Approved result emits once with content and persists in snapshot. | `packages/ag-ui/tests/ag_ui/test_approval_result_event.py::test_approved_call_emits_one_live_result_under_original_identity`, `packages/ag-ui/tests/ag_ui/test_endpoint.py::test_endpoint_agent_approval_resume_persists_replayable_tool_results`, `test_endpoint_agent_approval_replayed_resume_entry_reprojects_retained_result` | | AG-UI rejection/mixed decision | Transport emits only the events defined for approved and rejected calls without duplicates. | `packages/ag-ui/tests/ag_ui/test_approval_result_event.py::test_rejected_call_does_not_execute_or_emit_live_result`, `test_mixed_batch_preserves_approved_result_identity_and_order`, `packages/ag-ui/tests/ag_ui/test_endpoint.py::test_endpoint_agent_approval_rejection_releases_already_approved_sibling` | -| AG-UI approval-time follow-up | The full grouped user-input pause remains in message history and emits no synthetic `TOOL_CALL_RESULT`. | `packages/ag-ui/tests/ag_ui/test_approval_result_event.py::test_approval_follow_up_group_remains_in_history_without_live_tool_result` | +| AG-UI approval-time follow-up | The full grouped user-input pause remains in message history. OAuth emits its single paired terminal `TOOL_CALL_RESULT`; other follow-up input does not fabricate a terminal result. | `packages/ag-ui/tests/ag_ui/test_approval_result_event.py::test_approval_follow_up_group_preserves_terminal_result_contract` | | AG-UI approval execution failure | A grouped executor failure becomes one deterministic terminal error result for the approved call. | `packages/ag-ui/tests/ag_ui/test_approval_result_event.py::test_approval_execution_failure_emits_one_terminal_error_result` | | AG-UI no-approval path | Ordinary tool results do not gain an extra approval result event. | `packages/ag-ui/tests/ag_ui/test_approval_result_event.py::test_no_approval_path_emits_no_approval_specific_duplicate_result` | | AG-UI client-tool request isolation | Client tool declarations are validated before use and remain request-scoped; a rejected collision or earlier successful request cannot change a later request's server-tool execution. | `packages/ag-ui/tests/ag_ui/test_endpoint.py::test_endpoint_failed_client_tool_collision_does_not_affect_next_request`, `test_endpoint_client_tools_do_not_persist_into_next_request` | diff --git a/python/packages/ag-ui/tests/ag_ui/test_approval_result_event.py b/python/packages/ag-ui/tests/ag_ui/test_approval_result_event.py index 32c4e5065cc..9489271e56b 100644 --- a/python/packages/ag-ui/tests/ag_ui/test_approval_result_event.py +++ b/python/packages/ag-ui/tests/ag_ui/test_approval_result_event.py @@ -7,6 +7,7 @@ import json from typing import Any +import pytest from ag_ui.core import ToolCallResultEvent from agent_framework import AgentResponseUpdate, Content, FunctionInvocationContext, FunctionTool from agent_framework.exceptions import UserInputRequiredException @@ -262,31 +263,41 @@ async def test_mixed_batch_preserves_approved_result_identity_and_order() -> Non assert [(event.tool_call_id, event.content) for event in results] == [("call-seattle", "Sunny in Seattle")] -async def test_approval_follow_up_group_remains_in_history_without_live_tool_result() -> None: - """Approval-time user-input requests remain grouped without becoming terminal tool results.""" +@pytest.mark.parametrize("oauth", [False, True], ids=["user-input", "oauth"]) +async def test_approval_follow_up_group_preserves_terminal_result_contract(oauth: bool) -> None: + """OAuth has a paired terminal result; other follow-up input remains pending.""" + prompts = ( + [ + Content.from_oauth_consent_request(consent_link="https://example.com/first"), + Content.from_oauth_consent_request(consent_link="https://example.com/second"), + ] + if oauth + else [Content("text", text="Please provide more input.", user_input_request=True)] + ) def request_consent() -> str: - raise UserInputRequiredException( - contents=[ - Content.from_oauth_consent_request(consent_link="https://example.com/first"), - Content.from_oauth_consent_request(consent_link="https://example.com/second"), - ] - ) + raise UserInputRequiredException(contents=prompts) events, agent = await _run_custom_approval( FunctionTool(name="request_consent", description="Request consent", func=request_consent) ) - assert not [event for event in events if getattr(event, "type", None) == "TOOL_CALL_RESULT"] + results = [event for event in events if isinstance(event, ToolCallResultEvent)] + if oauth: + assert len(results) == 1 + assert results[0].tool_call_id == "call-custom" + assert "OAuth consent is required" in results[0].content + assert "https://" not in results[0].content + else: + assert not results requests = [ content for message in agent.messages_received for content in message.contents if isinstance(content, Content) and content.user_input_request ] - assert [request.consent_link for request in requests] == [ - "https://example.com/first", - "https://example.com/second", + assert [(request.type, request.consent_link, request.text) for request in requests] == [ + (prompt.type, prompt.consent_link, prompt.text) for prompt in prompts ] diff --git a/python/packages/core/agent_framework/_mcp.py b/python/packages/core/agent_framework/_mcp.py index 41e37a0d9ff..3dbfca07fdc 100644 --- a/python/packages/core/agent_framework/_mcp.py +++ b/python/packages/core/agent_framework/_mcp.py @@ -50,7 +50,7 @@ Content, Message, ) -from .exceptions import ToolException, ToolExecutionException +from .exceptions import ToolException, ToolExecutionException, UserInputRequiredException from .observability import ( OtelAttr, create_mcp_client_span, @@ -3742,6 +3742,8 @@ async def _call_tool_with_retries( span_error_recorded = True raise _mcp_tool_execution_exception(text or str(parsed)) return parser(result) + except UserInputRequiredException: + raise except ToolExecutionException as tool_ex: if span.is_recording() and not span_error_recorded: set_mcp_span_error(span, type(tool_ex).__name__, str(tool_ex)) diff --git a/python/packages/core/agent_framework/_tools.py b/python/packages/core/agent_framework/_tools.py index 9a1f142cfb7..47da0cd7fe3 100644 --- a/python/packages/core/agent_framework/_tools.py +++ b/python/packages/core/agent_framework/_tools.py @@ -2645,9 +2645,33 @@ async def _execute_single_function_call( except UserInputRequiredException as exc: source_function_call = _underlying_function_call(function_call) call_id = source_function_call.call_id - propagated_contents = [item for item in exc.contents if isinstance(item, Content)] if exc.contents else [] + propagated_contents = ( + [copy.copy(item) for item in exc.contents if isinstance(item, Content)] if exc.contents else [] + ) + if any(item.type == "oauth_consent_request" for item in propagated_contents): + terminal_results = [item for item in propagated_contents if item.type == "function_result"] + if len(terminal_results) > 1: + raise ValueError( + "An OAuth consent interruption must contain at most one terminal function result." + ) from exc + if not terminal_results: + propagated_contents.insert( + 0, + Content.from_function_result( + call_id=call_id, # type: ignore[arg-type] + id=source_function_call.id, + result=( + "Error: OAuth consent is required. This tool invocation did not complete. " + "The user must authorize access before this tool can be invoked in a later request." + ), + exception="OAuthConsentRequired", + additional_properties=source_function_call.additional_properties, + ), + ) for item in propagated_contents: item.call_id = call_id + if item.type == "function_result": + item.id = source_function_call.id if not item.id: item.id = source_function_call.id or call_id if propagated_contents: @@ -3543,8 +3567,9 @@ def _store_pending_mixed_pause_batch( *, deferred_function_calls: Sequence[Content] = (), function_call_order: Sequence[Mapping[str, str | None]] | None = None, + deferred_messages: Sequence[Message] = (), ) -> None: - """Persist the active ordered pause batch and any deferred local siblings.""" + """Persist the active ordered pause batch, completed results, and deferred local siblings.""" state = _get_tool_approval_state(invocation_session) if state is None: return @@ -3553,21 +3578,34 @@ def _store_pending_mixed_pause_batch( kinds: set[str] = set() for group in pause_groups: for content in group: + if content.type == "function_result": + kinds.add("result") + items.append({"kind": "result", "response": content.to_dict()}) + continue if content.type == "function_approval_request": kind = "approval" - elif content.type == "function_call" and content.user_input_request: + elif content.user_input_request and content.type != "oauth_consent_request": kind = "host" else: continue kinds.add(kind) items.append({"kind": kind, "request": content.to_dict()}) - if kinds == {"approval", "host"} or (kinds == {"host"} and deferred_function_calls): + if ( + kinds == {"approval", "host"} + or (kinds == {"host"} and deferred_function_calls) + or ("result" in kinds and bool(kinds & {"approval", "host"})) + ): batch: dict[str, Any] = {"items": items} if deferred_function_calls: batch[_DEFERRED_FUNCTION_CALLS_KEY] = [function_call.to_dict() for function_call in deferred_function_calls] if function_call_order is not None: batch[_FUNCTION_CALL_ORDER_KEY] = [dict(item) for item in function_call_order] + if deferred_messages: + retained_messages = _copy_messages_for_function_invocation(deferred_messages) + for message in retained_messages: + message.contents = [content for content in message.contents if content.type != "oauth_consent_request"] + batch["messages"] = [message.to_dict() for message in retained_messages if message.contents] state[_PENDING_MIXED_PAUSE_BATCH_KEY] = batch else: state.pop(_PENDING_MIXED_PAUSE_BATCH_KEY, None) @@ -3593,7 +3631,7 @@ def _match_mixed_pause_responses( host_items_by_occurrence: dict[str, int] = {} host_items_by_call: dict[str, list[int]] = {} for index, item in enumerate(items): - request = _content_from_state(item.get("request")) + request = _content_from_state(item.get("request", item.get("response"))) if request is None: continue if item.get("kind") == "approval": @@ -3607,7 +3645,7 @@ def _match_mixed_pause_responses( approval_items[identity] = index elif approval_items[identity] != index: approval_items[identity] = None - elif item.get("kind") == "host" and request.call_id is not None: + elif item.get("kind") in {"host", "result"} and request.call_id is not None: host_items_by_call.setdefault(request.call_id, []).append(index) if request.id is not None: host_items_by_occurrence[request.id] = index @@ -3645,7 +3683,11 @@ def _match_mixed_pause_responses( candidate = response if response.id is not None: item_index = host_items_by_occurrence.get(response.id) - request = _content_from_state(items[item_index].get("request")) if item_index is not None else None + request = ( + _content_from_state(items[item_index].get("request", items[item_index].get("response"))) + if item_index is not None + else None + ) if request is None or request.call_id != response.call_id: item_index = None else: @@ -3714,7 +3756,7 @@ def _match_mixed_pause_responses( if response is None: return matched_content_ids, True, [], set() ordered_responses.append(response) - if item.get("kind") == "host": + if item.get("kind") in {"host", "result"}: host_result_ids.add(id(response)) return matched_content_ids, False, ordered_responses, host_result_ids @@ -3741,13 +3783,23 @@ def _stage_pending_mixed_pause_responses( if not items: state.pop(_PENDING_MIXED_PAUSE_BATCH_KEY, None) return False, False, set() + raw_messages = cast(Mapping[str, Any], raw_batch).get("messages") + if isinstance(raw_messages, list): + messages[:0] = _copy_messages_for_function_invocation([ + raw_message if isinstance(raw_message, Message) else Message.from_dict(cast(dict[str, Any], raw_message)) + for raw_message in cast(list[Any], raw_messages) + ]) def bind_approval_response(response: Content) -> Content | None: - return _bind_approval_response_to_pending_request( + bound = _bind_approval_response_to_pending_request( response, invocation_session, consume=False, ) + if bound is not None: + # Binding runs again after staging; retain replacement-request identity until then. + bound.id = bound.additional_properties[_APPROVAL_REQUEST_ID_KEY] + return bound def attributed_staged_replay_item_index( content: Content, @@ -3772,7 +3824,7 @@ def attributed_staged_replay_item_index( candidate_state, ): continue - request = _content_from_state(item.get("request")) + request = _content_from_state(item.get("request", item.get("response"))) if request is None: continue if item.get("kind") == "approval" and candidate.type == "function_approval_response": @@ -3795,7 +3847,7 @@ def attributed_staged_replay_item_index( if request_ids.intersection(candidate_ids): return item_index elif ( - item.get("kind") == "host" + item.get("kind") in {"host", "result"} and candidate.type == "function_result" and candidate.call_id == request.call_id and (candidate.id is None or candidate.id == request.id) @@ -3806,9 +3858,8 @@ def attributed_staged_replay_item_index( host_requests = [ request for item in items - if item.get("kind") == "host" - and item.get("response") is None - and (request := _content_from_state(item.get("request"))) is not None + if (item.get("kind") == "host" and item.get("response") is None) or item.get("kind") == "result" + if (request := _content_from_state(item.get("request", item.get("response")))) is not None and request.call_id is not None ] approval_request_ids = { @@ -3925,12 +3976,18 @@ def attributed_staged_replay_item_index( messages[:] = filtered_messages updated_batch = dict(cast(Mapping[str, Any], raw_batch)) updated_batch["items"] = items + if incomplete and any(item.get("kind") == "result" for item in items): + updated_batch["messages"] = [message.to_dict() for message in messages] state[_PENDING_MIXED_PAUSE_BATCH_KEY] = updated_batch if incomplete: return True, False, set() - messages.append(Message(role="user", contents=ordered_responses)) + if any(item.get("kind") == "result" for item in items): + response_messages, _ = _messages_and_updates_for_terminal_contents(ordered_responses) + messages.extend(response_messages) + else: + messages.append(Message(role="user", contents=ordered_responses)) return False, True, host_result_ids @@ -4890,9 +4947,7 @@ def _clear_budget_state_from_session( if pending: return pending_pause_batch = tool_state.get(_PENDING_MIXED_PAUSE_BATCH_KEY) - if isinstance(pending_pause_batch, Mapping) and cast(Mapping[str, Any], pending_pause_batch).get( - _DEFERRED_FUNCTION_CALLS_KEY - ): + if isinstance(pending_pause_batch, Mapping): updated_batch = dict(cast(Mapping[str, Any], pending_pause_batch)) updated_batch[_DEFERRED_BUDGET_STATE_KEY] = _portable_budget_snapshot(budget_state) tool_state[_PENDING_MIXED_PAUSE_BATCH_KEY] = updated_batch @@ -5145,6 +5200,7 @@ def _handle_function_call_results( errors_in_a_row=errors_in_a_row, action="return", function_call_count=function_call_count, + response_messages=response_messages, streaming_updates=streaming_updates, ) @@ -5746,6 +5802,97 @@ def _update_function_invocation_continuation_state( ): session.service_session_id = conversation_id + async def _settle_oauth_consent_results( + self, + *, + super_get_response: Callable[..., Any], + response: ChatResponse[Any], + result_messages: Sequence[Message], + options: dict[str, Any], + request_kwargs: dict[str, Any], + compaction_strategy: CompactionStrategy | None, + tokenizer: TokenizerProtocol | None, + invocation_session: AgentSession | None, + input_messages: Sequence[Message] | None = None, + ) -> ChatResponse[Any] | None: + """Reconcile terminal OAuth batches and retain mixed pauses until their remaining calls resolve.""" + from ._sessions import is_local_history_conversation_id + from ._types import ChatResponse, Message + + if not any( + content.type == "oauth_consent_request" for message in result_messages for content in message.contents + ): + return None + continuation_id = response.conversation_id or cast("str | None", options.get("conversation_id")) + if continuation_id is None or is_local_history_conversation_id(continuation_id): + return None + results = [ + copy.copy(content) + for message in result_messages + for content in message.contents + if content.type == "function_result" + ] + if not results: + raise RuntimeError("OAuth consent reconciliation requires terminal function results.") + completed_calls = {(content.id, content.call_id) for content in results} + pending_calls: set[tuple[str | None, str | None]] = set() + pause_contents: list[Content] = [] + for message in result_messages: + for content in message.contents: + if content.type == "function_result": + pause_contents.append(content) + elif content.type == "function_approval_request" or ( + content.user_input_request and content.type != "oauth_consent_request" + ): + source_call = content.function_call or content + identity = (source_call.id, source_call.call_id) + if identity not in completed_calls and identity not in pending_calls: + pause_contents.append(content) + pending_calls.add(identity) + if pending_calls: + _store_pending_mixed_pause_batch( + invocation_session, [pause_contents], deferred_messages=input_messages or () + ) + return None + settlement_options = {**options, "conversation_id": continuation_id, "tool_choice": "none"} + if "background" in settlement_options: + settlement_options["background"] = False + settlement_messages = ( + _copy_messages_for_function_invocation(input_messages) + if input_messages is not None + else [Message(role="tool", contents=results)] + ) + for message in settlement_messages: + message.contents = [content for content in message.contents if content.type != "oauth_consent_request"] + settlement = await super_get_response( + messages=[message for message in settlement_messages if message.contents], + stream=False, + options=settlement_options, + compaction_strategy=compaction_strategy, + tokenizer=tokenizer, + client_kwargs=request_kwargs, + ) + if not isinstance(settlement, ChatResponse) or settlement.conversation_id is None: + raise RuntimeError("OAuth consent reconciliation did not return a service continuation ID.") + settlement = cast("ChatResponse[Any]", settlement) + # The service accepted the outputs. Preserve that advance even if its new output is invalid. + self._update_function_invocation_continuation_state( + request_kwargs, settlement, session=invocation_session, options=options + ) + response.conversation_id = settlement.conversation_id + if settlement.response_id is not None: + response.response_id = settlement.response_id + if settlement.continuation_token is not None: + raise RuntimeError("OAuth consent reconciliation returned an unfinished service response.") + if any( + content.type in {"function_call", "function_approval_request", "computer_tool_call", "shell_tool_call"} + or content.user_input_request + for message in settlement.messages + for content in message.contents + ): + raise RuntimeError("The service requested new tool execution or user input while settling OAuth consent.") + return settlement + async def _settle_dangling_service_function_calls( self, *, @@ -5920,6 +6067,19 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non ) if approval_processing.action == "return": response = ChatResponse(messages=list(function_call_messages)) + settlement = await self._settle_oauth_consent_results( + super_get_response=super_get_response, + response=response, + result_messages=approval_processing.response_messages, + options=options, + request_kwargs=request_kwargs, + compaction_strategy=compaction_strategy, + tokenizer=tokenizer, + invocation_session=invocation_session, + input_messages=prepared_messages, + ) + if settlement is not None: + aggregated_usage = add_usage_details(aggregated_usage, settlement.usage_details) response.usage_details = aggregated_usage _clear_budget_state_from_session(invocation_session, budget_state) return _clear_internal_conversation_id(response) @@ -6010,6 +6170,18 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non function_processing.function_call_count, ) if function_processing.action == "return": + settlement = await self._settle_oauth_consent_results( + super_get_response=super_get_response, + response=response, + result_messages=function_processing.response_messages, + options=options, + request_kwargs=request_kwargs, + compaction_strategy=compaction_strategy, + tokenizer=tokenizer, + invocation_session=invocation_session, + ) + if settlement is not None: + aggregated_usage = add_usage_details(aggregated_usage, settlement.usage_details) response.usage_details = aggregated_usage _prepend_function_call_messages(response, function_call_messages[:terminal_prefix_length]) _clear_budget_state_from_session(invocation_session, budget_state) @@ -6086,7 +6258,7 @@ async def _stream_response_with_function_invocation( ) -> AsyncIterable[ChatResponseUpdate]: """Run the streaming function invocation loop.""" from ._middleware import MiddlewareFailure - from ._types import ChatResponseUpdate, Content, add_usage_details + from ._types import ChatResponse, ChatResponseUpdate, Content, add_usage_details errors_in_a_row = 0 total_function_calls = int(budget_state.get("total_function_calls", 0) or 0) @@ -6164,6 +6336,38 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non invocation_session=invocation_session, ) + async def settle_oauth_pause( + response: ChatResponse[Any], + result_messages: Sequence[Message], + *, + input_messages: Sequence[Message] | None = None, + ) -> ChatResponseUpdate | None: + nonlocal completed_usage + settlement = await self._settle_oauth_consent_results( + super_get_response=super_get_response, + response=response, + result_messages=result_messages, + options=options, + request_kwargs=request_kwargs, + compaction_strategy=compaction_strategy, + tokenizer=tokenizer, + invocation_session=invocation_session, + input_messages=input_messages, + ) + if settlement is None: + return None + if settlement.usage_details is not None: + completed_usage = add_usage_details(completed_usage, settlement.usage_details) + return ChatResponseUpdate( + response_id=settlement.response_id, + conversation_id=settlement.conversation_id, + contents=( + [Content.from_usage(copy.deepcopy(completed_usage), merge_mode="replace")] + if completed_usage is not None + else [] + ), + ) + # Apply limit decisions early to prevent execution during replay if limits are already breached. _apply_batch_limit_decision( "continue", @@ -6196,6 +6400,14 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non total_function_calls, approval_processing.function_call_count, ) + if approval_processing.action == "return": + settlement_update = await settle_oauth_pause( + ChatResponse(messages=list(approval_processing.response_messages)), + approval_processing.response_messages, + input_messages=prepared_messages, + ) + if settlement_update is not None: + yield settlement_update for update in approval_processing.streaming_updates: yield update if approval_processing.action == "return": @@ -6352,6 +6564,10 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non total_function_calls, function_processing.function_call_count, ) + if function_processing.action == "return": + settlement_update = await settle_oauth_pause(response, function_processing.response_messages) + if settlement_update is not None: + yield settlement_update for update in function_processing.streaming_updates: yield update if function_processing.action != "continue" and function_processing.action != "stop": diff --git a/python/packages/core/agent_framework/exceptions.py b/python/packages/core/agent_framework/exceptions.py index a908b5bce5d..93af9021388 100644 --- a/python/packages/core/agent_framework/exceptions.py +++ b/python/packages/core/agent_framework/exceptions.py @@ -194,6 +194,9 @@ class UserInputRequiredException(ToolException): invocation layer can propagate them instead of swallowing them as a generic tool error. ``Agent.as_tool()`` handles unresolved child function approvals separately and fails the delegated invocation. + An OAuth consent request also produces one terminal result for the interrupted + function call. The loop lets the authorized batch finish and preserves all + results before returning the consent requests. Args: contents: The user-input-request Content items from the sub-agent response. diff --git a/python/packages/core/tests/core/test_clients.py b/python/packages/core/tests/core/test_clients.py index bdb770dbc3f..d5663b8ba49 100644 --- a/python/packages/core/tests/core/test_clients.py +++ b/python/packages/core/tests/core/test_clients.py @@ -941,8 +941,13 @@ def terminal_tool() -> str: assert "London" in (response.messages[summary_index].text or "") assert sum(content.call_id == "call_1" for content in correlated_contents) == 2 assert sum(content.call_id == "call_2" for content in correlated_contents) == 2 - assert sum(content.call_id == "call_terminal" for content in correlated_contents) == ( - 1 if terminal_kind == "approval" else 2 + expected_terminal_types = { + "approval": ["function_call"], + "user-input": ["function_call", "function_result", "oauth_consent_request"], + "middleware-termination": ["function_call", "function_result"], + } + assert [content.type for content in correlated_contents if content.call_id == "call_terminal"] == ( + expected_terminal_types[terminal_kind] ) if terminal_kind == "approval": assert any( diff --git a/python/packages/core/tests/core/test_function_invocation_logic.py b/python/packages/core/tests/core/test_function_invocation_logic.py index 54fafe4c72b..9f052871383 100644 --- a/python/packages/core/tests/core/test_function_invocation_logic.py +++ b/python/packages/core/tests/core/test_function_invocation_logic.py @@ -1,6 +1,7 @@ # Copyright (c) Microsoft. All rights reserved. import asyncio +import copy import json import logging import math @@ -10814,7 +10815,8 @@ def guarded_input_tool() -> str: resumed_updates = [update async for update in resumed_stream] resumed_response = await resumed_stream.get_final_response() assert [[content.type for content in update.contents] for update in resumed_updates] == [ - ["oauth_consent_request", "oauth_consent_request"] + ["function_result"], + ["oauth_consent_request", "oauth_consent_request"], ] assert len(chat_client_base.streaming_responses) == 1 # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] else: @@ -10921,7 +10923,7 @@ def paused_tool() -> str: resumed_updates = [update async for update in resumed_stream] resumed_response = await resumed_stream.get_final_response() assert [(update.role, [content.type for content in update.contents]) for update in resumed_updates] == [ - ("tool", ["function_result"]), + ("tool", ["function_result", "function_result"]), ("assistant", ["oauth_consent_request"]), ] assert len(chat_client_base.streaming_responses) == 1 # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] @@ -10936,7 +10938,7 @@ def paused_tool() -> str: assert [ (message.role, [content.type for content in message.contents]) for message in resumed_response.messages ] == [ - ("tool", ["function_result"]), + ("tool", ["function_result", "function_result"]), ("assistant", ["oauth_consent_request"]), ] assert resumed_response.messages[0].contents[0].call_id == "call_completed" @@ -11494,6 +11496,597 @@ def test_dict_overwrites_existing_conversation_id(self): # endregion +@pytest.mark.parametrize("streaming", [False, True], ids=["non-streaming", "streaming"]) +@pytest.mark.parametrize("concurrent", [False, True], ids=["sequential", "concurrent"]) +@pytest.mark.parametrize("approval_resume", [False, True], ids=["ordinary", "approved-replay"]) +@pytest.mark.parametrize("later_outcome", ["result", "consent", "error"]) +async def test_oauth_consent_preserves_all_batch_results( + chat_client_base: SupportsChatGetResponse, + streaming: bool, + concurrent: bool, + approval_resume: bool, + later_outcome: str, +) -> None: + from agent_framework import function_middleware + from agent_framework._tools import _FUNCTION_INVOCATION_BUDGET_STATE_KEY + from agent_framework.exceptions import UserInputRequiredException + + executions: list[str] = [] + middleware_calls: list[str] = [] + + @function_middleware + async def record_functions(context: FunctionInvocationContext, call_next: Callable[[], Awaitable[None]]) -> None: + middleware_calls.append(context.function.name) + await call_next() + + @tool(approval_mode="always_require" if approval_resume else "never_require") + async def completed_tool() -> str: + executions.append("completed") + return "completed-result" + + @tool(approval_mode="always_require" if approval_resume else "never_require") + async def consent_tool() -> str: + executions.append("consent") + raise UserInputRequiredException([Content.from_oauth_consent_request("https://consent.example.com/authorize")]) + + @tool(approval_mode="always_require" if approval_resume else "never_require") + async def later_tool() -> str: + executions.append("later") + if later_outcome == "consent": + raise UserInputRequiredException([Content.from_oauth_consent_request("https://consent.example.com/later")]) + if later_outcome == "error": + raise RuntimeError("later tool failed") + return "later-result" + + function_calls = [ + Content.from_function_call(name, name, arguments={}, id=f"occurrence-{name}") + for name in ("completed_tool", "consent_tool", "later_tool") + ] + chat_client_base.function_invocation_configuration["allow_concurrent_invocation"] = concurrent # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + budget_state: dict[str, int] = {} + session = AgentSession() + options: ChatOptions = {"tools": [completed_tool, consent_tool, later_tool]} + client_kwargs: dict[str, Any] = { + "session": session, + "middleware": [record_functions], + _FUNCTION_INVOCATION_BUDGET_STATE_KEY: budget_state, + } + updates: list[ChatResponseUpdate] = [] + if streaming: + chat_client_base.streaming_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + [ChatResponseUpdate(role="assistant", contents=function_calls)] + ] + stream = chat_client_base.get_response( + [Message("user", contents=["run all"])], + stream=True, + options=options, + client_kwargs=client_kwargs, + ) + updates = [update async for update in stream] + response = await stream.get_final_response() + else: + chat_client_base.run_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + ChatResponse(messages=[Message("assistant", contents=function_calls)]) + ] + response = await chat_client_base.get_response( + [Message("user", contents=["run all"])], + options=options, + client_kwargs=client_kwargs, + ) + + if approval_resume: + decisions = [ + content.to_function_approval_response(approved=True) + for message in response.messages + for content in message.contents + if content.type == "function_approval_request" + ] + assert len(decisions) == 3 + assert not executions + if streaming: + stream = chat_client_base.get_response( + [Message("user", contents=decisions)], + stream=True, + options=options, + client_kwargs=client_kwargs, + ) + updates = [update async for update in stream] + response = await stream.get_final_response() + else: + response = await chat_client_base.get_response( + [Message("user", contents=decisions)], + options=options, + client_kwargs=client_kwargs, + ) + if streaming: + streamed_results = [ + content for update in updates for content in update.contents if content.type == "function_result" + ] + assert [content.call_id for content in streamed_results] == [call.call_id for call in function_calls] + results = [ + content for message in response.messages for content in message.contents if content.type == "function_result" + ] + assert executions == ["completed", "consent", "later"] + assert middleware_calls == ["completed_tool", "consent_tool", "later_tool"] + assert [(result.call_id, result.id) for result in results] == [(call.call_id, call.id) for call in function_calls] + assert results[0].result == "completed-result" + assert "OAuth consent is required" in results[1].result + if later_outcome == "consent": + assert "OAuth consent is required" in results[2].result + elif later_outcome == "error": + assert results[2].exception is not None + else: + assert results[2].result == "later-result" + assert all("https://" not in result.result for result in results) + assert budget_state["total_function_calls"] == 3 + assert [ + content.consent_link + for message in response.messages + for content in message.contents + if content.type == "oauth_consent_request" + ] == ( + ["https://consent.example.com/authorize", "https://consent.example.com/later"] + if later_outcome == "consent" + else ["https://consent.example.com/authorize"] + ) + + +@pytest.mark.parametrize("streaming", [False, True], ids=["non-streaming", "streaming"]) +async def test_oauth_mixed_pause_retains_partial_input_and_budget( + chat_client_base: SupportsChatGetResponse, + streaming: bool, +) -> None: + from agent_framework.exceptions import UserInputRequiredException + + executions: list[str] = [] + requests: list[dict[str, Any]] = [] + + @chat_middleware + async def record_requests(context: ChatContext, call_next: Callable[[], Awaitable[None]]) -> None: + requests.append({"options": dict(context.options or {}), "messages": copy.deepcopy(context.messages)}) + await call_next() + + @tool + async def completed_tool() -> str: + executions.append("completed") + return "preserved-result" + + @tool + async def consent_tool() -> str: + executions.append("consent") + raise UserInputRequiredException([Content.from_oauth_consent_request("https://consent.example.com/authorize")]) + + @tool + async def input_tool() -> str: + executions.append("input") + raise UserInputRequiredException([Content("text", text="Supply input.", user_input_request=True)]) + + calls = [ + Content.from_function_call(call_id, name, arguments={}, id=f"occurrence-{call_id}") + for call_id, name in ( + ("completed", "completed_tool"), + ("consent", "consent_tool"), + ("input-1", "input_tool"), + ("input-2", "input_tool"), + ) + ] + chat_client_base.chat_middleware = [record_requests] # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + chat_client_base.function_invocation_configuration["max_function_calls"] = 4 # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + if streaming: + chat_client_base.streaming_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + [ChatResponseUpdate(role="assistant", contents=calls, conversation_id="service-calls")], + ] + else: + chat_client_base.run_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + ChatResponse(messages=[Message("assistant", contents=calls)], conversation_id="service-calls"), + ] + options: ChatOptions = {"tools": [completed_tool, consent_tool, input_tool]} + session = AgentSession() + if streaming: + first = await chat_client_base.get_response( + [Message("user", contents=["Run the tools."])], + stream=True, + options=options, + client_kwargs={"session": session}, + ).get_final_response() + else: + first = await chat_client_base.get_response( + [Message("user", contents=["Run the tools."])], options=options, client_kwargs={"session": session} + ) + assert any(content.type == "oauth_consent_request" for message in first.messages for content in message.contents) + assert len(requests) == 1 + options["conversation_id"] = "service-calls" + partial_input = Message( + "user", + contents=[ + Content.from_text("Keep the first instruction."), + Content.from_function_result("input-1", result="first-input"), + ], + ) + original_input = partial_input.to_dict() + if streaming: + await chat_client_base.get_response( + [partial_input], stream=True, options=options, client_kwargs={"session": session} + ).get_final_response() + else: + await chat_client_base.get_response([partial_input], options=options, client_kwargs={"session": session}) + assert len(requests) == 1 + assert partial_input.to_dict() == original_input + session = AgentSession.from_dict(session.to_dict()) + final_input = Message( + "user", + contents=[ + Content.from_text("Keep the last instruction."), + Content.from_function_result("input-2", result="last-input"), + ], + ) + if streaming: + final = await chat_client_base.get_response( + [final_input], stream=True, options=options, client_kwargs={"session": session} + ).get_final_response() + else: + final = await chat_client_base.get_response([final_input], options=options, client_kwargs={"session": session}) + assert not any(content.user_input_request for message in final.messages for content in message.contents) + assert len(requests) == 2 + assert requests[1]["options"]["tool_choice"] == "none" + model_messages = requests[1]["messages"] + results = [ + content for message in model_messages for content in message.contents if content.type == "function_result" + ] + assert [content.call_id for content in results] == [call.call_id for call in calls] + assert results[0].result == "preserved-result" + assert [content.result for content in results[2:]] == ["first-input", "last-input"] + assert all( + message.role == "tool" + for message in model_messages + if any(c.type == "function_result" for c in message.contents) + ) + assert "Keep the first instruction." in " ".join(message.text for message in model_messages) + assert "Keep the last instruction." in " ".join(message.text for message in model_messages) + assert not any( + content.type == "oauth_consent_request" for message in model_messages for content in message.contents + ) + assert executions == ["completed", "consent", "input", "input"] + + +@pytest.mark.parametrize("streaming", [False, True], ids=["non-streaming", "streaming"]) +@pytest.mark.parametrize("approval_resume", [False, True], ids=["ordinary", "approved-replay"]) +async def test_oauth_consent_reconciles_real_results_and_advances_service_history( + chat_client_base: SupportsChatGetResponse, + streaming: bool, + approval_resume: bool, +) -> None: + from agent_framework.exceptions import UserInputRequiredException + + requests: list[dict[str, Any]] = [] + + @chat_middleware + async def record_requests(context: ChatContext, call_next: Callable[[], Awaitable[None]]) -> None: + requests.append({ + "conversation_id": (context.options or {}).get("conversation_id"), + "tool_choice": (context.options or {}).get("tool_choice"), + "contents": [content for message in context.messages for content in message.contents], + }) + await call_next() + if len(requests) == 2: + context.result = settlement + + @tool(approval_mode="always_require" if approval_resume else "never_require") + async def completed_tool() -> str: + return "preserved-result" + + @tool(approval_mode="always_require" if approval_resume else "never_require") + async def consent_tool() -> str: + raise UserInputRequiredException([Content.from_oauth_consent_request("https://consent.example.com/authorize")]) + + function_calls = [ + Content.from_function_call("completed", "completed_tool", arguments={}), + Content.from_function_call("consent", "consent_tool", arguments={}), + ] + settlement = ChatResponse( + messages=[Message("assistant", contents=["hidden settlement reply"])], + response_id="resp_settled", + conversation_id="resp_settled", + usage_details={"input_token_count": 3, "output_token_count": 2}, + ) + chat_client_base.chat_middleware = [record_requests] # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + if streaming: + chat_client_base.streaming_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + [ + ChatResponseUpdate( + role="assistant", + contents=function_calls, + response_id="resp_calls", + conversation_id="resp_calls", + ) + ] + ] + chat_client_base.run_responses = [settlement] # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + else: + chat_client_base.run_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + ChatResponse( + messages=[Message("assistant", contents=function_calls)], + response_id="resp_calls", + conversation_id="resp_calls", + ), + settlement, + ] + session = AgentSession() + options: ChatOptions = {"tools": [completed_tool, consent_tool]} + updates: list[ChatResponseUpdate] = [] + if streaming: + stream = chat_client_base.get_response( + [Message("user", contents=["run both"])], + stream=True, + options=options, + client_kwargs={"session": session}, + ) + updates = [update async for update in stream] + response = await stream.get_final_response() + else: + response = await chat_client_base.get_response( + [Message("user", contents=["run both"])], options=options, client_kwargs={"session": session} + ) + + if approval_resume: + decisions = [ + content.to_function_approval_response(approved=True) + for message in response.messages + for content in message.contents + if content.type == "function_approval_request" + ] + assert len(decisions) == 2 + assert len(requests) == 1 + assert session.service_session_id == "resp_calls" + options["conversation_id"] = "resp_calls" + resume_message = Message("user", contents=[Content.from_text("Keep this instruction."), *decisions]) + if streaming: + stream = chat_client_base.get_response( + [resume_message], + stream=True, + options=options, + client_kwargs={"session": session}, + ) + updates = [update async for update in stream] + response = await stream.get_final_response() + else: + response = await chat_client_base.get_response( + [resume_message], + options=options, + client_kwargs={"session": session}, + ) + if streaming: + assert any(content.type == "oauth_consent_request" for update in updates for content in update.contents) + assert len(requests) == 2 + assert requests[1]["conversation_id"] == "resp_calls" + assert requests[1]["tool_choice"] == "none" + settled_results = [content for content in requests[1]["contents"] if content.type == "function_result"] + assert settled_results[0].result == "preserved-result" + assert [content.call_id for content in settled_results] == ["completed", "consent"] + assert not any(content.type == "oauth_consent_request" for content in requests[1]["contents"]) + if approval_resume: + assert any(content.text == "Keep this instruction." for content in requests[1]["contents"]) + else: + assert [content.type for content in requests[1]["contents"]] == ["function_result", "function_result"] + assert response.conversation_id == session.service_session_id == "resp_settled" + assert response.response_id == "resp_settled" + assert "hidden settlement reply" not in response.text + + +@pytest.mark.parametrize("streaming", [False, True], ids=["non-streaming", "streaming"]) +@pytest.mark.parametrize("outcome", ["failure", "cancellation", "unexpected-call", "unfinished"]) +async def test_oauth_consent_settlement_failure_withholds_user_input( + chat_client_base: SupportsChatGetResponse, + streaming: bool, + outcome: str, +) -> None: + from agent_framework.exceptions import UserInputRequiredException + + requests = 0 + executions = 0 + + @chat_middleware + async def settlement_outcome(context: ChatContext, call_next: Callable[[], Awaitable[None]]) -> None: + nonlocal requests + requests += 1 + if requests == 2: + if outcome == "failure": + raise RuntimeError("settlement refused") + if outcome == "cancellation": + raise asyncio.CancelledError + context.result = ChatResponse( + messages=( + [ + Message( + "assistant", contents=[Content.from_function_call("new-call", "consent_tool", arguments={})] + ) + ] + if outcome == "unexpected-call" + else [] + ), + conversation_id="resp_accepted", + continuation_token={} if outcome == "unfinished" else None, + ) + return + await call_next() + + @tool(approval_mode="never_require") + async def consent_tool() -> str: + nonlocal executions + executions += 1 + raise UserInputRequiredException([Content.from_oauth_consent_request("https://consent.example.com/authorize")]) + + function_call = Content.from_function_call("consent", "consent_tool", arguments={}) + chat_client_base.chat_middleware = [settlement_outcome] # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + if streaming: + chat_client_base.streaming_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + [ChatResponseUpdate(role="assistant", contents=[function_call], conversation_id="resp_calls")] + ] + else: + chat_client_base.run_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + ChatResponse(messages=[Message("assistant", contents=[function_call])], conversation_id="resp_calls") + ] + session = AgentSession() + updates: list[ChatResponseUpdate] = [] + expected_error = asyncio.CancelledError if outcome == "cancellation" else RuntimeError + with pytest.raises(expected_error): + if streaming: + async for update in chat_client_base.get_response( + [Message("user", contents=["run"])], + stream=True, + options={"tools": [consent_tool]}, + client_kwargs={"session": session}, + ): + updates.append(update) + else: + await chat_client_base.get_response( + [Message("user", contents=["run"])], + options={"tools": [consent_tool]}, + client_kwargs={"session": session}, + ) + + assert requests == 2 + assert executions == 1 + assert not any(content.type == "oauth_consent_request" for update in updates for content in update.contents) + assert session.service_session_id == ( + "resp_accepted" if outcome in ("unexpected-call", "unfinished") else "resp_calls" + ) + + +@pytest.mark.parametrize("streaming", [False, True], ids=["non-streaming", "streaming"]) +async def test_oauth_consent_preserves_already_dispatched_sibling( + chat_client_base: SupportsChatGetResponse, streaming: bool +) -> None: + from agent_framework.exceptions import UserInputRequiredException + + started = asyncio.Event() + finish = asyncio.Event() + completed = False + + @tool(approval_mode="never_require") + async def inflight_tool() -> str: + nonlocal completed + started.set() + await finish.wait() + completed = True + return "settled sibling" + + @tool(approval_mode="never_require") + async def consent_tool() -> str: + await started.wait() + finish.set() + raise UserInputRequiredException([Content.from_oauth_consent_request("https://consent.example.com/authorize")]) + + function_calls = [ + Content.from_function_call("inflight", "inflight_tool", arguments={}), + Content.from_function_call("consent", "consent_tool", arguments={}), + ] + options: ChatOptions = {"tools": [inflight_tool, consent_tool]} + if streaming: + chat_client_base.streaming_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + [ChatResponseUpdate(role="assistant", contents=function_calls)] + ] + stream = chat_client_base.get_response([Message("user", contents=["run"])], stream=True, options=options) + _ = [update async for update in stream] + response = await stream.get_final_response() + else: + chat_client_base.run_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + ChatResponse(messages=[Message("assistant", contents=function_calls)]) + ] + response = await chat_client_base.get_response([Message("user", contents=["run"])], options=options) + + results = [ + content for message in response.messages for content in message.contents if content.type == "function_result" + ] + assert completed + assert [content.call_id for content in results] == ["inflight", "consent"] + assert results[0].result == "settled sibling" + + +@pytest.mark.parametrize("streaming", [False, True], ids=["non-streaming", "streaming"]) +async def test_provider_oauth_consent_is_not_locally_settled( + chat_client_base: SupportsChatGetResponse, streaming: bool +) -> None: + @tool(approval_mode="never_require") + async def unused_tool() -> str: + pytest.fail("A provider-owned consent request must not execute a local tool.") + + contents = [Content.from_oauth_consent_request("https://consent.example.com/provider")] + session = AgentSession() + if streaming: + chat_client_base.streaming_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + [ChatResponseUpdate(role="assistant", contents=contents, conversation_id="resp_provider_consent")] + ] + stream = chat_client_base.get_response( + [Message("user", contents=["run"])], + stream=True, + options={"tools": [unused_tool]}, + client_kwargs={"session": session}, + ) + _ = [update async for update in stream] + response = await stream.get_final_response() + else: + chat_client_base.run_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + ChatResponse(messages=[Message("assistant", contents=contents)], conversation_id="resp_provider_consent") + ] + response = await chat_client_base.get_response( + [Message("user", contents=["run"])], + options={"tools": [unused_tool]}, + client_kwargs={"session": session}, + ) + assert chat_client_base.call_count == 1 # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + assert session.service_session_id == "resp_provider_consent" + assert [content.type for message in response.messages for content in message.contents] == ["oauth_consent_request"] + + +@pytest.mark.parametrize("duplicate_result", [False, True]) +async def test_oauth_consent_supplied_terminal_result_stays_single_and_immutable( + chat_client_base: SupportsChatGetResponse, duplicate_result: bool +) -> None: + from agent_framework.exceptions import UserInputRequiredException + + contents = [Content.from_function_result("unbound", result="custom consent result", id="caller-result")] + if duplicate_result: + contents.append(Content.from_function_result("other", result="duplicate")) + contents.append(Content.from_oauth_consent_request("https://consent.example.com/authorize")) + original = [content.to_dict() for content in contents] + + @tool(approval_mode="never_require") + async def request_consent() -> str: + raise UserInputRequiredException(contents) + + chat_client_base.run_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + ChatResponse( + messages=[ + Message( + "assistant", + contents=[Content.from_function_call("bound", "request_consent", arguments={}, id="occurrence")], + ) + ] + ) + ] + if duplicate_result: + with pytest.raises(ValueError, match="at most one terminal function result") as failure: + await chat_client_base.get_response( + [Message("user", contents=["run"])], options={"tools": [request_consent]} + ) + assert isinstance(failure.value.__cause__, UserInputRequiredException) + else: + response = await chat_client_base.get_response( + [Message("user", contents=["run"])], options={"tools": [request_consent]} + ) + results = [ + content + for message in response.messages + for content in message.contents + if content.type == "function_result" + ] + assert [(result.call_id, result.id, result.result) for result in results] == [ + ("bound", "occurrence", "custom consent result") + ] + assert [content.to_dict() for content in contents] == original + + async def test_user_input_request_propagates_through_as_tool(chat_client_base: SupportsChatGetResponse): """Test that user_input_request content from a sub-agent wrapped as a tool propagates to the parent response.""" from agent_framework.exceptions import UserInputRequiredException diff --git a/python/packages/core/tests/core/test_harness_tool_approval.py b/python/packages/core/tests/core/test_harness_tool_approval.py index d2170fbff4d..329aaa83ba6 100644 --- a/python/packages/core/tests/core/test_harness_tool_approval.py +++ b/python/packages/core/tests/core/test_harness_tool_approval.py @@ -1568,7 +1568,13 @@ def oauth_tool() -> str: "https://example.com/consent-2", "https://example.com/consent-3", ] - assert resumed_response.messages[0].role == "assistant" + assert [ + (message.role, [content.type for content in message.contents]) for message in resumed_response.messages + ] == [ + ("tool", ["function_result"]), + ("assistant", ["oauth_consent_request", "oauth_consent_request", "oauth_consent_request"]), + ] + assert resumed_response.messages[0].contents[0].call_id == "call_oauth" assert calls == 1 diff --git a/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses_agent.py b/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses_agent.py index 600d27ee96a..df0f9b314f9 100644 --- a/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses_agent.py +++ b/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses_agent.py @@ -68,18 +68,15 @@ _await_before_signal, # pyright: ignore[reportPrivateUsage] _create_response_event_stream, # pyright: ignore[reportPrivateUsage] _hosted_internal_metadata, # pyright: ignore[reportPrivateUsage] - _is_allowed_oauth_consent_link, # pyright: ignore[reportPrivateUsage] _items_to_messages, # pyright: ignore[reportPrivateUsage] _log_response_failure, # pyright: ignore[reportPrivateUsage] _native_approval_id, # pyright: ignore[reportPrivateUsage] - _normalize_allowed_oauth_consent_origins, # pyright: ignore[reportPrivateUsage] _output_items_to_messages, # pyright: ignore[reportPrivateUsage] _OutputItemTracker, # pyright: ignore[reportPrivateUsage] _PublicResponseFailure, # pyright: ignore[reportPrivateUsage] _reject_busy_conversation, # pyright: ignore[reportPrivateUsage] _set_agent_session, # pyright: ignore[reportPrivateUsage] _SignalledIterator, # pyright: ignore[reportPrivateUsage] - consent_url_from_error, ) from ._scope import FoundryRequestScope from ._shared import ( @@ -99,6 +96,11 @@ validate_request_options, validate_unsupported_options, ) +from ._toolbox import ( + _is_allowed_oauth_consent_link, # pyright: ignore[reportPrivateUsage] + _normalize_allowed_oauth_consent_origins, # pyright: ignore[reportPrivateUsage] + consent_url_from_error, +) logger = logging.getLogger(__name__) @@ -1139,6 +1141,10 @@ async def publish_update( or content.user_input_request for content in update.contents ): + if any(content.type == "oauth_consent_request" for content in update.contents): + raise _PublicResponseFailure( + ValueError("OAuth consent continuation requires store=true.") + ) raise ValueError("Approval and user-input continuation requires store=true.") if deferred_updates or any( content.type == "function_approval_request" for content in update.contents diff --git a/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses_shared.py b/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses_shared.py index 9f60e5cedf8..1639c3d575a 100644 --- a/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses_shared.py +++ b/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses_shared.py @@ -6,10 +6,8 @@ import asyncio import base64 -import ipaddress import json import logging -import re import uuid from collections.abc import ( AsyncGenerator, @@ -23,8 +21,7 @@ from contextlib import suppress from dataclasses import asdict, dataclass, is_dataclass from decimal import Decimal -from typing import Any, Generic, Literal, TypeGuard, TypeVar, cast -from urllib.parse import urlparse +from typing import Any, Generic, Literal, TypeVar, cast from agent_framework import ( AgentResponse, @@ -97,7 +94,8 @@ RefusalContentBuilder, TextContentBuilder, ) -from mcp import MCPError + +from ._toolbox import _is_allowed_oauth_consent_link # pyright: ignore[reportPrivateUsage] logger = logging.getLogger(__name__) @@ -638,160 +636,6 @@ async def aclose(self) -> None: _RESERVED_RESPONSE_METADATA_KEY = "_internal_metadata" -# Foundry Toolbox Auth integration -# Consent-URL error code returned by the Foundry MCP gateway when calling `/list` -CONSENT_ERROR_CODE = -32006 - -_OAUTH_HOST_PATTERN = re.compile(r"^[A-Za-z0-9._~-]+$") - - -@dataclass -class ConsentError: - name: str - consent_url: str - - -def _is_safe_oauth_consent_link(consent_link: object) -> TypeGuard[str]: - """Return whether a consent link is an absolute HTTPS URL safe to expose as an action.""" - if not isinstance(consent_link, str) or not consent_link: - return False - if any(char.isspace() or ord(char) < 0x20 or ord(char) == 0x7F for char in consent_link): - return False - - try: - parsed = urlparse(consent_link) - hostname = parsed.hostname - _ = parsed.port - except ValueError: - return False - - if parsed.scheme.lower() != "https" or not hostname or parsed.username is not None or parsed.password is not None: - return False - - if "%" in hostname: - return False - authority = parsed.netloc - if authority.startswith("["): - closing_bracket = authority.find("]") - if closing_bracket == -1: - return False - ipv6_literal = authority[1:closing_bracket] - suffix = authority[closing_bracket + 1 :] - if suffix and (not suffix.startswith(":") or not suffix[1:].isdigit()): - return False - try: - ipaddress.IPv6Address(ipv6_literal) - except ValueError: - return False - return True - if "[" in authority or "]" in authority or ":" in hostname: - return False - return _OAUTH_HOST_PATTERN.fullmatch(hostname) is not None - - -def _normalize_oauth_consent_origin(value: str, *, require_origin_only: bool) -> str: - """Normalize an HTTPS consent URL or configured origin for exact matching.""" - if not _is_safe_oauth_consent_link(value): - raise ValueError("OAuth consent origins must be absolute HTTPS URLs without user information.") - - parsed = urlparse(value) - if require_origin_only and (parsed.path not in ("", "/") or parsed.params or parsed.query or parsed.fragment): - raise ValueError("OAuth consent origin entries must not include a path, query, parameters, or fragment.") - - hostname = parsed.hostname - if hostname is None: # pragma: no cover - _is_safe_oauth_consent_link already proved this. - raise ValueError("OAuth consent origin must include a hostname.") - normalized_host = f"[{hostname.lower()}]" if ":" in hostname else hostname.lower() - port = parsed.port - return f"https://{normalized_host}" if port in (None, 443) else f"https://{normalized_host}:{port}" - - -def _normalize_allowed_oauth_consent_origins( # pyright: ignore[reportUnusedFunction] - allowed_origins: Sequence[str] | None, -) -> frozenset[str] | None: - """Normalize the host-configured consent origin allowlist, or return ``None`` when none is configured.""" - if allowed_origins is None: - return None - return frozenset(_normalize_oauth_consent_origin(origin, require_origin_only=True) for origin in allowed_origins) - - -def _is_allowed_oauth_consent_link(consent_link: object, allowed_origins: frozenset[str] | None) -> TypeGuard[str]: - """Return whether the link is safe and, when an allowlist is configured, has an allowed origin. - - ``allowed_origins`` must come from ``_normalize_allowed_oauth_consent_origins``. ``None`` keeps the - safe-HTTPS check without restricting the origin; an empty set rejects every link. - """ - if not _is_safe_oauth_consent_link(consent_link): - return False - if allowed_origins is None: - return True - return _normalize_oauth_consent_origin(consent_link, require_origin_only=False) in allowed_origins - - -def consent_url_from_error(exc: BaseException) -> list[ConsentError] | None: - """Return the consent URLs when ``exc`` wraps Foundry MCP gateway consent errors. - - Args: - exc: The exception to inspect. - - Returns: - The consent URL(s) extracted from the error, or ``None`` if no consent error was found. - """ - inner_exception = next((arg for arg in exc.args if isinstance(arg, MCPError)), None) - if inner_exception is not None and inner_exception.code == CONSENT_ERROR_CODE: - # Parse the error message - # The error message is structured with the following format: - # "tools/list failed for 1 tool source(s), succeeded for 0 tool source(s) {"errors":[{"name": ..." - # where the second part is a JSON string that can be deserialized into an object with the following shape: - # ruff: disable[commented-out-code] - # { - # "errors" : [ - # { - # "name": "Name of the MCP tool that requires consent", - # "type" : "mcp" | "a2a_preview", - # "error": { - # "code": "CONSENT_REQUIRED", - # "message": consent_url, - # } - # } - # ] - # } - # ruff: enable[commented-out-code] - try: - consent_errors: list[ConsentError] = [] - error_message_start = inner_exception.error.message.find("{") - if error_message_start == -1: - logger.warning("Consent error message does not contain JSON: %s", inner_exception.error.message) - return None - consent_details_json = inner_exception.error.message[error_message_start:] - consent_details = json.loads(consent_details_json) - if "errors" not in consent_details or not isinstance(consent_details["errors"], list): - logger.warning("Consent error message JSON does not contain 'errors' list: %s", consent_details_json) - return None - for error in consent_details["errors"]: - if ( - isinstance(error, dict) - and error.get("type") in ("mcp", "a2a_preview") # type: ignore - and "error" in error - and isinstance(error["error"], dict) - and error["error"].get("code") == "CONSENT_REQUIRED" # type: ignore - and "message" in error["error"] - ): - consent_url = error["error"]["message"] # type: ignore - if isinstance(consent_url, str): - consent_errors.append(ConsentError(name=error.get("name", "Unknown"), consent_url=consent_url)) # type: ignore - else: - logger.warning("Consent URL in error message is not a valid URL: %s", consent_url) # type: ignore - if consent_errors: - return consent_errors - except json.JSONDecodeError: - logger.warning("Failed to parse consent details JSON: %s", inner_exception.error.message) - return None - - -# endregion Foundry Toolbox Auth integration - - # region Active Builder State @@ -1308,7 +1152,9 @@ async def handle( yield builder.emit_done(oauth_item) elif content.type == "usage": - self._usage_details = add_usage_details(self._usage_details, content.usage_details) + self._usage_details = add_usage_details( + None if content.merge_mode == "replace" else self._usage_details, content.usage_details + ) else: for event in self._close(): diff --git a/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses_workflow.py b/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses_workflow.py index fdf550fed5d..f63307c281b 100644 --- a/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses_workflow.py +++ b/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses_workflow.py @@ -51,7 +51,6 @@ _hosted_internal_metadata, # pyright: ignore[reportPrivateUsage] _item_to_message, # pyright: ignore[reportPrivateUsage] _native_approval_id, # pyright: ignore[reportPrivateUsage] - _normalize_allowed_oauth_consent_origins, # pyright: ignore[reportPrivateUsage] _OutputItemTracker, # pyright: ignore[reportPrivateUsage] _request_without_reserved_metadata, # pyright: ignore[reportPrivateUsage] _SignalledIterator, # pyright: ignore[reportPrivateUsage] @@ -75,6 +74,7 @@ validate_workflow_source, workflow_agents, ) +from ._toolbox import _normalize_allowed_oauth_consent_origins # pyright: ignore[reportPrivateUsage] logger = logging.getLogger(__name__) diff --git a/python/packages/foundry_hosting/agent_framework_foundry_hosting/_toolbox.py b/python/packages/foundry_hosting/agent_framework_foundry_hosting/_toolbox.py index d9a02501c56..d82a69c85e1 100644 --- a/python/packages/foundry_hosting/agent_framework_foundry_hosting/_toolbox.py +++ b/python/packages/foundry_hosting/agent_framework_foundry_hosting/_toolbox.py @@ -4,17 +4,22 @@ import asyncio import inspect +import ipaddress +import json import logging import os +import re from collections.abc import Callable, Mapping, Sequence from contextlib import _AsyncGeneratorContextManager # pyright: ignore[reportPrivateUsage] from contextvars import ContextVar -from typing import TYPE_CHECKING, Any, Literal, cast -from urllib.parse import urlsplit +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any, Literal, TypeGuard, cast +from urllib.parse import urlparse, urlsplit import httpx2 as httpx from agent_framework import ( CachingSkillsSource, + Content, DeduplicatingSkillsSource, MCPSkillsSource, MCPStreamableHTTPTool, @@ -23,8 +28,9 @@ SkillsSourceContext, ) from agent_framework._telemetry import mark_feature_used -from agent_framework.exceptions import ToolExecutionException +from agent_framework.exceptions import ToolExecutionException, UserInputRequiredException from azure.ai.agentserver.core import get_request_context +from mcp import MCPError from mcp.types import Tool from typing_extensions import override @@ -34,12 +40,13 @@ from collections.abc import AsyncGenerator, Generator from datetime import timedelta - from agent_framework import Content, Skill + from agent_framework import Skill from agent_framework._mcp import MCPSpecificApproval from azure.core.credentials import AccessToken, TokenCredential from azure.core.credentials_async import AsyncTokenCredential from mcp.client.session import ClientSession - from mcp.types import Prompt, ReadResourceResult + from mcp.types import CallToolResult, Prompt, ReadResourceResult + from opentelemetry import context as otel_context from pydantic import AnyUrl AzureCredentialTypes = TokenCredential | AsyncTokenCredential @@ -59,6 +66,131 @@ "strict_toolbox_header_provider", default=False, ) +CONSENT_ERROR_CODE = -32006 +_OAUTH_HOST_PATTERN = re.compile(r"^[A-Za-z0-9._~-]+$") + + +@dataclass +class ConsentError: + name: str + consent_url: str + + +def _is_safe_oauth_consent_link(consent_link: object) -> TypeGuard[str]: + """Return whether a consent link is an absolute HTTPS URL safe to expose as an action.""" + if not isinstance(consent_link, str) or not consent_link: + return False + if any(char.isspace() or ord(char) < 0x20 or ord(char) == 0x7F for char in consent_link): + return False + + try: + parsed = urlparse(consent_link) + hostname = parsed.hostname + _ = parsed.port + except ValueError: + return False + + if parsed.scheme.lower() != "https" or not hostname or parsed.username is not None or parsed.password is not None: + return False + + if "%" in hostname: + return False + authority = parsed.netloc + if authority.startswith("["): + closing_bracket = authority.find("]") + if closing_bracket == -1: + return False + ipv6_literal = authority[1:closing_bracket] + suffix = authority[closing_bracket + 1 :] + if suffix and (not suffix.startswith(":") or not suffix[1:].isdigit()): + return False + try: + ipaddress.IPv6Address(ipv6_literal) + except ValueError: + return False + return True + if "[" in authority or "]" in authority or ":" in hostname: + return False + return _OAUTH_HOST_PATTERN.fullmatch(hostname) is not None + + +def _normalize_oauth_consent_origin(value: str, *, require_origin_only: bool) -> str: + if not _is_safe_oauth_consent_link(value): + raise ValueError("OAuth consent origins must be absolute HTTPS URLs without user information.") + + parsed = urlparse(value) + if require_origin_only and (parsed.path not in ("", "/") or parsed.params or parsed.query or parsed.fragment): + raise ValueError("OAuth consent origin entries must not include a path, query, parameters, or fragment.") + + hostname = parsed.hostname + if hostname is None: # pragma: no cover - validated above. + raise ValueError("OAuth consent origin must include a hostname.") + normalized_host = f"[{hostname.lower()}]" if ":" in hostname else hostname.lower() + port = parsed.port + return f"https://{normalized_host}" if port in (None, 443) else f"https://{normalized_host}:{port}" + + +def _normalize_allowed_oauth_consent_origins( # pyright: ignore[reportUnusedFunction] + allowed_origins: Sequence[str] | None, +) -> frozenset[str] | None: + """Normalize exact HTTPS origins; ``None`` retains the safe-link-only policy.""" + if allowed_origins is None: + return None + return frozenset(_normalize_oauth_consent_origin(origin, require_origin_only=True) for origin in allowed_origins) + + +def _is_allowed_oauth_consent_link( # pyright: ignore[reportUnusedFunction] + consent_link: object, allowed_origins: frozenset[str] | None +) -> TypeGuard[str]: + """Validate a link against safety rules and the optional normalized origin allowlist.""" + if not _is_safe_oauth_consent_link(consent_link): + return False + if allowed_origins is None: + return True + return _normalize_oauth_consent_origin(consent_link, require_origin_only=False) in allowed_origins + + +def consent_url_from_error(exc: BaseException) -> list[ConsentError] | None: + """Extract requirements only from a consent-only Foundry MCP gateway error or its wrapper.""" + inner = exc if isinstance(exc, MCPError) else next((arg for arg in exc.args if isinstance(arg, MCPError)), None) + if inner is None or inner.code != CONSENT_ERROR_CODE: + return None + start = inner.error.message.find("{") + if start == -1: + logger.warning("Foundry consent error does not contain a structured payload.") + return None + try: + payload: object = json.loads(inner.error.message[start:]) + except json.JSONDecodeError: + logger.warning("Foundry consent error contains invalid JSON.") + return None + if not isinstance(payload, dict): + logger.warning("Foundry consent error payload must be an object.") + return None + errors = cast(dict[str, object], payload).get("errors") + if not isinstance(errors, list): + logger.warning("Foundry consent error payload must contain an errors list.") + return None + + requirements: list[ConsentError] = [] + for raw_error in cast(list[object], errors): + if not isinstance(raw_error, dict): + return None + error = cast(dict[str, object], raw_error) + if error.get("type") not in ("mcp", "a2a_preview"): + return None + raw_details = error.get("error") + if not isinstance(raw_details, dict): + return None + details = cast(dict[str, object], raw_details) + if details.get("code") != "CONSENT_REQUIRED": + return None + name = error.get("name", "Unknown") + link = details.get("message") + if not isinstance(name, str) or not name or not isinstance(link, str): + return None + requirements.append(ConsentError(name=name, consent_url=link)) + return requirements or None def _build_toolbox_features_header(additional_features: str | None) -> str: @@ -265,6 +397,12 @@ class FoundryToolbox(MCPStreamableHTTPTool): The private metadata parser provides no security or compatibility guarantees when reused outside Foundry Toolbox; it is not a general-purpose MCP approval policy. + A validated invocation-time OAuth consent error raises + :class:`~agent_framework.exceptions.UserInputRequiredException`. The native + function loop returns a paired terminal result and consent action, retaining + completed sibling results without automatically retrying protected operations. + Responses hosting requires ``store=true`` for this continuation. + Like any MCP tool, the connection lifecycle is driven by the agent: the hosting server enters a factory-created agent for its request and closes its toolbox and owned HTTP client afterward. A long-lived Toolbox is also supported: it @@ -418,6 +556,29 @@ async def call_tool(self, tool_name: str, **kwargs: Any) -> str | list[Content]: finally: _strict_toolbox_header_provider.reset(token) + @override + async def _invoke_tool( + self, tool_name: str, kwargs: dict[str, Any], trace_context: otel_context.Context + ) -> CallToolResult: + try: + return await super()._invoke_tool(tool_name, kwargs, trace_context) + except MCPError as exc: + requirements = consent_url_from_error(exc) + if not requirements or any( + not _is_safe_oauth_consent_link(requirement.consent_url) for requirement in requirements + ): + raise + raise UserInputRequiredException( + contents=[ + Content.from_oauth_consent_request( + requirement.consent_url, + additional_properties={"openai": {"server_label": requirement.name}}, + ) + for requirement in requirements + ], + message="Tool invocation requires OAuth consent.", + ) from exc + @override async def _call_prompt_with_runtime_kwargs( self, diff --git a/python/packages/foundry_hosting/tests/foundry_hosting/toolbox_approval_helpers.py b/python/packages/foundry_hosting/tests/foundry_hosting/toolbox_approval_helpers.py index 81fd0550e76..6990c67df2b 100644 --- a/python/packages/foundry_hosting/tests/foundry_hosting/toolbox_approval_helpers.py +++ b/python/packages/foundry_hosting/tests/foundry_hosting/toolbox_approval_helpers.py @@ -1,27 +1,44 @@ # Copyright (c) Microsoft. All rights reserved. -"""Offline protocol boundaries for Toolbox approval regressions.""" +"""Offline protocol boundaries for Toolbox approval and OAuth consent regressions.""" from __future__ import annotations +import asyncio +import copy import json -from collections.abc import AsyncGenerator, AsyncIterator, Awaitable, Mapping, Sequence +from collections.abc import AsyncGenerator, AsyncIterator, Awaitable, Callable, Mapping, Sequence from contextlib import asynccontextmanager -from typing import Any +from importlib import import_module +from typing import Any, Literal +import httpx as host_httpx import httpx2 as httpx from agent_framework import ( BaseChatClient, ChatResponse, ChatResponseUpdate, Content, + FunctionInvocationContext, FunctionInvocationLayer, + FunctionMiddleware, + FunctionTool, Message, ResponseStream, + SessionStore, ) +from agent_framework.exceptions import UserInputRequiredException +from agent_framework.openai import OpenAIChatClient, OpenAIChatOptions +from azure.ai.agentserver.core import AgentConfig, FoundryAgentRequestContext from azure.core.credentials import AccessToken +from openai import AsyncOpenAI, DefaultAsyncHttpxClient +from pydantic import BaseModel -from agent_framework_foundry_hosting import FoundryToolbox +from agent_framework_foundry_hosting import FoundryToolbox, ResponsesHostServer +from agent_framework_foundry_hosting._shared import StoreProvider + +TOOL_NAMES = ("learn___completed", "learn___protected", "learn___later") +CONSENT_LINK = "https://consent.example.com/authorize" class ToolboxCredential: @@ -150,3 +167,300 @@ async def complete() -> ChatResponse: return ChatResponse(messages=[Message("assistant", contents=contents)]) return ResponseStream(updates(), finalizer=ChatResponse.from_updates) if stream else complete() + + +class ToolboxConsentServer(ToolboxApprovalServer): + def __init__(self) -> None: + super().__init__() + self.tools = [toolbox_tool("never", name=name) for name in TOOL_NAMES] + self.authorized = False + self.before_consent: Callable[[], Awaitable[object]] | None = None + self.error: dict[str, Any] = { + "code": -32006, + "message": "tools/call failed " + + json.dumps({ + "errors": [ + { + "name": "learn", + "type": "mcp", + "error": {"code": "CONSENT_REQUIRED", "message": CONSENT_LINK}, + } + ] + }), + } + + async def handle(self, request: httpx.Request) -> httpx.Response: + if request.method == "POST": + payload = json.loads(request.content) + if ( + payload.get("method") == "tools/call" + and payload["params"]["name"] == "learn___protected" + and not self.authorized + ): + self.calls.append(payload["params"]) + if self.before_consent is not None: + await self.before_consent() + return httpx.Response( + 200, + json={"jsonrpc": "2.0", "id": payload["id"], "error": self.error}, + ) + return await super().handle(request) + + +class ConsentModelServer: + def __init__(self, tool_names: Sequence[str] = TOOL_NAMES) -> None: + self.tool_names = tuple(tool_names) + self.requests: list[dict[str, Any]] = [] + self.chains: dict[str, list[dict[str, Any]]] = {} + self.history_errors: list[str] = [] + self.fail_settlement = False + self.unexpected_settlement_call = False + self.sdk_httpx: Any = import_module(DefaultAsyncHttpxClient.__mro__[1].__module__.partition(".")[0]) + + async def handle(self, request: Any) -> Any: + payload = json.loads(request.content) + self.requests.append(payload) + previous = payload.get("previous_response_id") + history = copy.deepcopy(self.chains[previous]) if previous is not None else [] + inputs = payload["input"] + history.extend(inputs if isinstance(inputs, list) else [{"type": "message", "content": inputs}]) + pending: set[str] = set() + for item in history: + if item.get("type") == "function_call": + pending.add(item["call_id"]) + elif item.get("type") == "function_call_output": + if item["call_id"] not in pending: + self.history_errors.append(f"Unexpected tool result for {item['call_id']}") + pending.discard(item["call_id"]) + if pending or self.history_errors: + self.history_errors.extend(f"No tool output found for {call_id}" for call_id in sorted(pending)) + return self.sdk_httpx.Response( + 400, + json={"error": {"message": self.history_errors[-1], "type": "invalid_request_error"}}, + ) + settling = payload.get("tool_choice") == "none" + if settling and self.fail_settlement: + return self.sdk_httpx.Response( + 500, json={"error": {"message": "settlement refused", "type": "server_error"}} + ) + index = len(self.requests) + names = self.tool_names if index == 1 else () + authorized_retry = any(item.get("role") == "user" and "I have signed in" in json.dumps(item) for item in inputs) + if ( + not settling + and authorized_retry + and not any( + item.get("type") == "function_call_output" and item["call_id"].startswith("call_retry_") + for item in inputs + ) + ): + names = ("learn___protected",) + if settling and self.unexpected_settlement_call: + names = ("learn___later",) + output: list[dict[str, Any]] = [ + { + "type": "function_call", + "id": f"fc_{index}_{position}", + "call_id": f"call_{'initial' if index == 1 else 'retry'}_{position}", + "name": name, + "arguments": "{}", + "status": "completed", + } + for position, name in enumerate(names) + ] + if not output: + output = [ + { + "type": "message", + "id": f"msg_{index}", + "role": "assistant", + "status": "completed", + "content": [ + { + "type": "output_text", + "text": "hidden settlement reply" if settling else "done", + "annotations": [], + "logprobs": [], + } + ], + } + ] + response: dict[str, Any] = { + "id": f"resp_model_{index}", + "object": "response", + "created_at": 1_700_000_000, + "model": "test-model", + "status": "completed", + "output": output, + "error": None, + "incomplete_details": None, + "instructions": None, + "metadata": {}, + "parallel_tool_calls": True, + "temperature": 1.0, + "tool_choice": payload.get("tool_choice", "auto"), + "tools": [], + "top_p": 1.0, + "usage": { + "input_tokens": 5, + "output_tokens": 2, + "total_tokens": 7, + "input_tokens_details": {"cached_tokens": 0}, + "output_tokens_details": {"reasoning_tokens": 0}, + }, + "store": payload.get("store", True), + "background": False, + "conversation": None, + } + if response["store"]: + self.chains[response["id"]] = history + copy.deepcopy(output) + if not payload.get("stream"): + return self.sdk_httpx.Response(200, json=response) + events: list[dict[str, Any]] = [ + { + "type": "response.created", + "response": {**response, "status": "in_progress", "output": [], "usage": None}, + } + ] + for position, item in enumerate(output): + events.append({ + "type": "response.output_item.added", + "output_index": position, + "item": {**item, "arguments": ""} if item["type"] == "function_call" else {**item, "content": []}, + }) + if item["type"] == "function_call": + events.append({ + "type": "response.function_call_arguments.delta", + "item_id": item["id"], + "output_index": position, + "delta": item["arguments"], + }) + else: + events.append({ + "type": "response.output_text.delta", + "item_id": item["id"], + "output_index": position, + "content_index": 0, + "delta": item["content"][0]["text"], + "logprobs": [], + }) + events.append({"type": "response.output_item.done", "output_index": position, "item": item}) + events.append({"type": "response.completed", "response": response}) + for sequence_number, event in enumerate(events): + event["sequence_number"] = sequence_number + text = "".join(f"event: {event['type']}\ndata: {json.dumps(event)}\n\n" for event in events) + return self.sdk_httpx.Response( + 200, headers={"content-type": "text/event-stream"}, text=text + "data: [DONE]\n\n" + ) + + +@asynccontextmanager +async def consent_model_client( + server: ConsentModelServer, *, concurrent: bool = False +) -> AsyncGenerator[OpenAIChatClient[OpenAIChatOptions]]: + async with AsyncOpenAI( + api_key="offline-key", + base_url="https://model.invalid/v1", + max_retries=0, + http_client=DefaultAsyncHttpxClient(transport=server.sdk_httpx.MockTransport(server.handle)), + ) as api_client: + yield OpenAIChatClient[OpenAIChatOptions]( + model="test-model", + async_client=api_client, + function_invocation_configuration={"allow_concurrent_invocation": concurrent}, + ) + + +class ConsentSessionStoreProvider(StoreProvider[SessionStore]): + def __init__(self) -> None: + self.store = SessionStore() + + def get_store(self, *, config: AgentConfig, platform_context: FoundryAgentRequestContext) -> SessionStore: + return self.store + + +class ConsentPendingInput(FunctionMiddleware): + def __init__(self, kind: Literal["input", "approval"], *, require_approval: bool = False) -> None: + self.kind = kind + self.executions = 0 + self.function = FunctionTool( + name="awaiting_input", + func=self.invoke, + approval_mode="always_require" if require_approval else "never_require", + ) + + async def invoke(self) -> str: + self.executions += 1 + if self.kind == "input": + raise UserInputRequiredException([ + Content("text", text="Please supply the missing input.", user_input_request=True), + Content("text", text="Return the input as this call's result.", user_input_request=True), + ]) + return "pending-result" + + async def process(self, context: FunctionInvocationContext, call_next: Callable[[], Awaitable[None]]) -> None: + if context.function.name == self.function.name and self.kind == "approval": + approval = context.metadata.get("approval_response") + approved_call = approval.function_call if isinstance(approval, Content) else None + if approved_call is None or not approved_call.additional_properties.get("test_runtime_approval"): + occurrence_id = context.metadata["function_call_occurrence_id"] + context.result = Content.from_function_approval_request( + id=f"runtime-{occurrence_id}", + function_call=Content.from_function_call( + call_id=context.metadata["call_id"], + name=context.function.name, + arguments=( + context.arguments.model_dump() + if isinstance(context.arguments, BaseModel) + else context.arguments + ), + id=occurrence_id, + additional_properties={"test_runtime_approval": True}, + ), + additional_properties={"_replacement_approval_request": True}, + ) + return + await call_next() + + +async def post_consent_response( + host: ResponsesHostServer, payload: dict[str, Any] +) -> tuple[dict[str, Any], list[dict[str, Any]]]: + async with host_httpx.AsyncClient(transport=host_httpx.ASGITransport(app=host), base_url="http://test") as client: + response = await client.post("/responses", json={"model": "test-model", **payload}) + if payload.get("background"): + assert response.status_code == 200, response.text + body = response.json() + async with asyncio.timeout(5): + while body["status"] in ("queued", "in_progress"): + await asyncio.sleep(0.01) + response = await client.get(f"/responses/{body['id']}") + assert response.status_code == 200, response.text + body = response.json() + return body, [] + assert response.status_code == 200, response.text + if not payload.get("stream"): + return response.json(), [] + events = [json.loads(line[6:]) for line in response.text.splitlines() if line.startswith("data: {")] + terminal = [ + event["response"] + for event in events + if event["type"] in ("response.completed", "response.incomplete", "response.failed") + ] + assert len(terminal) == 1, events + return terminal[0], events + + +class ConsentCallerHeaders(FunctionMiddleware): + def __init__(self) -> None: + self.later_queued = asyncio.Event() + + async def process(self, context: FunctionInvocationContext, call_next: Callable[[], Awaitable[None]]) -> None: + context.kwargs["caller"] = "later" if context.function.name.endswith("later") else "first" + await call_next() + + def headers(self, kwargs: dict[str, Any]) -> dict[str, str]: + caller = kwargs["caller"] + if caller == "later": + self.later_queued.set() + return {"x-test-caller": caller} diff --git a/python/packages/foundry_hosting/tests/test_responses.py b/python/packages/foundry_hosting/tests/test_responses.py index 80c465c9db9..72cd8b3fd28 100644 --- a/python/packages/foundry_hosting/tests/test_responses.py +++ b/python/packages/foundry_hosting/tests/test_responses.py @@ -88,29 +88,31 @@ from agent_framework_foundry_hosting._responses_shared import ( _INCOMPLETE_REASON_KEY, # pyright: ignore[reportPrivateUsage] _LATEST_CHECKPOINT_ID_KEY, # pyright: ignore[reportPrivateUsage] - CONSENT_ERROR_CODE, - ConsentError, _agent_response_updates, # pyright: ignore[reportPrivateUsage] _annotations_to_output_text, # pyright: ignore[reportPrivateUsage] _await_before_signal, # pyright: ignore[reportPrivateUsage] _container_file_citations_from_function_result, # pyright: ignore[reportPrivateUsage] _create_response_event_stream, # pyright: ignore[reportPrivateUsage] _hosted_internal_metadata, # pyright: ignore[reportPrivateUsage] - _is_allowed_oauth_consent_link, # pyright: ignore[reportPrivateUsage] _item_to_message, # pyright: ignore[reportPrivateUsage] _json_safe_to_str, # pyright: ignore[reportPrivateUsage] - _normalize_allowed_oauth_consent_origins, # pyright: ignore[reportPrivateUsage] _output_item_to_message, # pyright: ignore[reportPrivateUsage] _output_items_to_messages, # pyright: ignore[reportPrivateUsage] _OutputItemTracker, # pyright: ignore[reportPrivateUsage] _PublicResponseFailure, # pyright: ignore[reportPrivateUsage] _stringify_mcp_output, # pyright: ignore[reportPrivateUsage] - consent_url_from_error, ) from agent_framework_foundry_hosting._shared import ( AgentSessionStoreProvider, FoundryAgentSessionStore, ) +from agent_framework_foundry_hosting._toolbox import ( + CONSENT_ERROR_CODE, + ConsentError, + _is_allowed_oauth_consent_link, + _normalize_allowed_oauth_consent_origins, + consent_url_from_error, +) _OPENAI_HTTPX = cast(Any, import_module(DefaultAsyncHttpxClient.__mro__[1].__module__.partition(".")[0])) _PRIVATE_ERROR_DETAIL = "test-token-value at /srv/private/tool.py" @@ -9236,6 +9238,39 @@ def test_returns_consent_url_for_a2a_preview_source(self) -> None: ConsentError(name="work-iq", consent_url="https://example.com/a2a-consent") ] + @pytest.mark.parametrize( + "other_error", + [ + {"type": "mcp", "error": {"code": "INTERNAL_ERROR", "message": "Upstream failure"}}, + {"type": "other", "error": {"code": "CONSENT_REQUIRED", "message": "https://example.com/consent"}}, + None, + {"type": "mcp", "error": None}, + {"type": "mcp", "error": {"code": "CONSENT_REQUIRED", "message": 42}}, + { + "name": "", + "type": "mcp", + "error": {"code": "CONSENT_REQUIRED", "message": "https://example.com/consent"}, + }, + ], + ) + @pytest.mark.parametrize("consent_first", [False, True]) + @pytest.mark.parametrize("wrapped", [False, True]) + def test_returns_none_for_mixed_consent_errors( + self, other_error: object, consent_first: bool, wrapped: bool + ) -> None: + from agent_framework.exceptions import ToolExecutionException + + consent = { + "name": "learn", + "type": "mcp", + "error": {"code": "CONSENT_REQUIRED", "message": "https://example.com/consent"}, + } + errors = [consent, other_error] if consent_first else [other_error, consent] + inner = MCPError(CONSENT_ERROR_CODE, "tools/call failed " + json.dumps({"errors": errors})) + exc = ToolExecutionException("MCP failure", inner_exception=inner) if wrapped else inner + + assert consent_url_from_error(exc) is None + def test_returns_none_when_no_mcp_error_in_args(self) -> None: assert consent_url_from_error(Exception("boom")) is None @@ -9244,10 +9279,7 @@ def test_returns_none_when_mcp_error_has_different_code(self) -> None: exc = Exception("wrapped", inner) assert consent_url_from_error(exc) is None - def test_returns_none_for_bare_mcp_error_without_wrapping(self) -> None: - # `args` of a bare MCPError holds the message string, not an MCPError - # instance, so it does not match the wrapping pattern produced by the - # MCP client when it bubbles consent errors up. + def test_returns_none_for_unstructured_bare_mcp_error(self) -> None: bare = MCPError(CONSENT_ERROR_CODE, "https://x") assert consent_url_from_error(bare) is None diff --git a/python/packages/foundry_hosting/tests/test_toolbox.py b/python/packages/foundry_hosting/tests/test_toolbox.py index 6396df53df1..6dc701fc5bd 100644 --- a/python/packages/foundry_hosting/tests/test_toolbox.py +++ b/python/packages/foundry_hosting/tests/test_toolbox.py @@ -10,7 +10,7 @@ from collections.abc import Awaitable, Callable from datetime import datetime, timedelta, timezone from types import SimpleNamespace -from typing import Any, cast +from typing import Any, Literal, cast from unittest.mock import AsyncMock, Mock, patch import httpx as host_httpx @@ -18,6 +18,7 @@ import pytest from agent_framework import ( Agent, + AgentSession, FunctionInvocationContext, MCPStreamableHTTPTool, Message, @@ -31,6 +32,7 @@ reset_request_context, set_request_context, ) +from azure.ai.agentserver.responses import InMemoryResponseProvider from mcp.client.session import ClientSession from mcp.types import Prompt, Tool from pydantic import AnyUrl @@ -45,8 +47,16 @@ ) from .foundry_hosting.toolbox_approval_helpers import ( # pyrefly: ignore[missing-import] - pytest's importlib namespace + CONSENT_LINK, + ConsentCallerHeaders, + ConsentModelServer, + ConsentPendingInput, + ConsentSessionStoreProvider, ToolboxApprovalClient, ToolboxApprovalServer, + ToolboxConsentServer, + consent_model_client, + post_consent_response, toolbox_connection, toolbox_tool, ) @@ -1639,3 +1649,474 @@ async def test_caller_can_wrap_the_foundry_metadata_policy() -> None: caller_policy.return_value = "always_require" await toolbox.load_tools() assert toolbox.functions[0].approval_mode == "always_require" + + +@pytest.mark.parametrize("stream", [False, True]) +@pytest.mark.parametrize("history_source", ["host_managed", "inference_service"]) +@pytest.mark.parametrize("continuation", ["response", "conversation"]) +async def test_toolbox_oauth_consent_pairs_results_and_continues( + stream: bool, + history_source: Literal["host_managed", "inference_service"], + continuation: str, +) -> None: + mcp = ToolboxConsentServer() + model = ConsentModelServer() + sessions = ConsentSessionStoreProvider() + async with toolbox_connection(mcp) as toolbox, consent_model_client(model) as chat_client: + host = ResponsesHostServer( + Agent(client=chat_client, tools=toolbox), + history_source=history_source, + response_store=InMemoryResponseProvider(), + agent_session_store_provider=sessions, + ) + payload: dict[str, Any] = {"input": "Run the tools", "stream": stream, "store": True} + if continuation == "conversation": + payload["conversation"] = "oauth-conversation" + first, events = await post_consent_response(host, payload) + + assert first["status"] == "incomplete", first + assert [item["consent_link"] for item in first["output"] if item["type"] == "oauth_consent_request"] == [ + CONSENT_LINK + ] + calls = [item for item in first["output"] if item["type"] == "function_call"] + results = [item for item in first["output"] if item["type"] == "function_call_output"] + assert len(calls) == len(results) == 3, first + assert [item["call_id"] for item in results] == [item["call_id"] for item in calls] + assert results[0]["output"] == "search-result" + assert "OAuth consent is required" in results[1]["output"] + assert results[2]["output"] == "search-result" + assert [call["name"] for call in mcp.calls] == ["learn___completed", "learn___protected", "learn___later"] + assert "hidden settlement reply" not in json.dumps(first["output"]) + if stream: + assert ( + sum( + event["type"] == "response.output_item.done" and event["item"]["type"] == "oauth_consent_request" + for event in events + ) + == 1 + ) + saved = await sessions.store.get(first["id"]) + assert saved is not None + if history_source == "inference_service": + assert len(model.requests) == 2 + assert model.requests[1]["tool_choice"] == "none" + assert [item["type"] for item in model.requests[1]["input"]] == ["function_call_output"] * 3 + assert model.requests[1]["input"][0]["output"] == "search-result" + assert model.requests[1]["input"][2]["output"] == "search-result" + assert saved.service_session_id == "resp_model_2" + assert first["usage"]["input_tokens"] == 10 + assert first["usage"]["output_tokens"] == 4 + else: + assert len(model.requests) == 1 + assert saved.service_session_id is None + + mcp.authorized = True + payload["input"] = "I have signed in. Continue." + if continuation == "response": + payload["previous_response_id"] = first["id"] + second, _ = await post_consent_response(host, payload) + assert second["status"] == "completed", second + assert not any(item["type"] == "oauth_consent_request" for item in second["output"]) + assert [call["name"] for call in mcp.calls] == [ + "learn___completed", + "learn___protected", + "learn___later", + "learn___protected", + ] + assert not model.history_errors + + +@pytest.mark.parametrize("stream", [False, True]) +@pytest.mark.parametrize("history_source", ["host_managed", "inference_service"]) +@pytest.mark.parametrize("pending_kind", ["input", "approval"]) +@pytest.mark.parametrize("approval_resume", [False, True], ids=["ordinary", "approved-replay"]) +async def test_toolbox_oauth_consent_preserves_a_mixed_pending_batch( + stream: bool, + history_source: Literal["host_managed", "inference_service"], + pending_kind: Literal["input", "approval"], + approval_resume: bool, +) -> None: + mcp = ToolboxConsentServer() + mcp.tools = [ + toolbox_tool("always" if approval_resume else "never", name=name) + for name in ("learn___completed", "learn___protected") + ] + pending = ConsentPendingInput(pending_kind, require_approval=approval_resume) + names = ("learn___completed", "learn___protected", pending.function.name) + if pending_kind == "approval": + names = (pending.function.name, "learn___completed", "learn___protected") + model = ConsentModelServer(names) + sessions = ConsentSessionStoreProvider() + async with toolbox_connection(mcp) as toolbox, consent_model_client(model, concurrent=True) as chat_client: + host = ResponsesHostServer( + Agent(client=chat_client, tools=[toolbox, pending.function], middleware=[pending]), + history_source=history_source, + response_store=InMemoryResponseProvider(), + agent_session_store_provider=sessions, + ) + first, _ = await post_consent_response(host, {"input": "Run the tools", "stream": stream}) + if approval_resume: + approvals = [item for item in first["output"] if item["type"] == "mcp_approval_request"] + assert len(approvals) == 3 + assert not mcp.calls and not pending.executions + first, _ = await post_consent_response( + host, + { + "input": [ + { + "type": "message", + "role": "user", + "content": [{"type": "input_text", "text": "Keep this instruction until input arrives."}], + }, + *[ + {"type": "mcp_approval_response", "approval_request_id": approval["id"], "approve": True} + for approval in approvals + ], + ], + "previous_response_id": first["id"], + "stream": stream, + }, + ) + assert first["status"] == "incomplete", first + assert [item["consent_link"] for item in first["output"] if item["type"] == "oauth_consent_request"] == [ + CONSENT_LINK + ] + results = [item for item in first["output"] if item["type"] == "function_call_output"] + expected_call_ids = [f"call_initial_{position}" for position in range(3)] + pending_call_id = expected_call_ids[names.index(pending.function.name)] + assert [item["call_id"] for item in results] == [ + call_id for call_id in expected_call_ids if call_id != pending_call_id + ] + assert results[0]["output"] == "search-result" + assert "OAuth consent is required" in results[1]["output"] + assert [call["name"] for call in mcp.calls] == ["learn___completed", "learn___protected"] + assert len(model.requests) == 1 + assert pending.executions == (1 if pending_kind == "input" else 0) + if pending_kind == "approval": + approval = next(item for item in first["output"] if item["type"] == "mcp_approval_request") + decision = {"type": "mcp_approval_response", "approval_request_id": approval["id"], "approve": True} + else: + assert "Please supply the missing input." in json.dumps(first["output"]) + decision = {"type": "function_call_output", "call_id": pending_call_id, "output": "provided-input"} + + saved = await sessions.store.get(first["id"]) + assert saved is not None + await sessions.store.set(first["id"], AgentSession.from_dict(saved.to_dict())) + second, _ = await post_consent_response( + host, {"input": [decision], "previous_response_id": first["id"], "stream": stream} + ) + assert second["status"] == "completed", second + assert len(model.requests) == 2 + outputs = [item for item in model.requests[1]["input"] if item["type"] == "function_call_output"] + if history_source == "inference_service": + assert [item["call_id"] for item in outputs] == expected_call_ids + else: + assert len(outputs) == 3 + assert {item["call_id"]: item["output"] for item in outputs} == { + **{item["call_id"]: item["output"] for item in results}, + pending_call_id: "pending-result" if pending_kind == "approval" else "provided-input", + } + assert not model.history_errors + if approval_resume: + assert "Keep this instruction until input arrives." in json.dumps(model.requests[1]["input"]) + assert pending.executions == 1 + assert not any(item["type"] == "oauth_consent_request" for item in second["output"]) + + third, _ = await post_consent_response( + host, {"input": "Continue.", "previous_response_id": second["id"], "stream": stream} + ) + assert third["status"] == "completed", third + assert len(model.requests) == 3 + if history_source == "inference_service": + assert not any(item["type"] == "function_call_output" for item in model.requests[2]["input"]) + assert [call["name"] for call in mcp.calls] == ["learn___completed", "learn___protected"] + assert pending.executions == 1 + assert not model.history_errors + + +@pytest.mark.parametrize("stream", [False, True]) +@pytest.mark.parametrize("history_source", ["host_managed", "inference_service"]) +async def test_toolbox_oauth_consent_rejects_unstored_requests( + stream: bool, history_source: Literal["host_managed", "inference_service"] +) -> None: + mcp = ToolboxConsentServer() + model = ConsentModelServer() + sessions = ConsentSessionStoreProvider() + provider = InMemoryResponseProvider() + async with toolbox_connection(mcp) as toolbox, consent_model_client(model) as chat_client: + host = ResponsesHostServer( + Agent(client=chat_client, tools=toolbox), + history_source=history_source, + response_store=provider, + agent_session_store_provider=sessions, + ) + response, _ = await post_consent_response(host, {"input": "Run the tools", "stream": stream, "store": False}) + assert response["status"] == "failed", response + assert "requires store=true" in response["error"]["message"] + assert not any(item["type"] == "oauth_consent_request" for item in response["output"]) + assert await sessions.store.get(response["id"]) is None + assert not model.chains + async with host_httpx.AsyncClient( + transport=host_httpx.ASGITransport(app=host), base_url="http://test" + ) as client: + stored = await client.get(f"/responses/{response['id']}") + assert stored.status_code == 404 + + +@pytest.mark.parametrize("stream", [False, True]) +@pytest.mark.parametrize("unexpected_call", [False, True], ids=["refused", "unexpected-call"]) +async def test_toolbox_oauth_consent_reconciliation_failure_is_not_an_incomplete_response( + stream: bool, unexpected_call: bool +) -> None: + mcp = ToolboxConsentServer() + model = ConsentModelServer() + model.fail_settlement = not unexpected_call + model.unexpected_settlement_call = unexpected_call + sessions = ConsentSessionStoreProvider() + async with toolbox_connection(mcp) as toolbox, consent_model_client(model) as chat_client: + host = ResponsesHostServer( + Agent(client=chat_client, tools=toolbox), + history_source="inference_service", + response_store=InMemoryResponseProvider(), + agent_session_store_provider=sessions, + ) + response, events = await post_consent_response(host, {"input": "Run the tools", "stream": stream}) + assert response["status"] == "failed", response + assert not any(item["type"] == "oauth_consent_request" for item in response["output"]) + assert not any(event.get("item", {}).get("type") == "oauth_consent_request" for event in events) + assert len(model.requests) == 2 + assert [call["name"] for call in mcp.calls] == ["learn___completed", "learn___protected", "learn___later"] + if unexpected_call: + saved = await sessions.store.get(response["id"]) + assert saved is not None + assert saved.service_session_id == "resp_model_2" + + +@pytest.mark.parametrize("stream", [False, True]) +@pytest.mark.parametrize("history_source", ["host_managed", "inference_service"]) +async def test_toolbox_oauth_consent_allows_an_internally_queued_mcp_dispatch( + stream: bool, history_source: Literal["host_managed", "inference_service"] +) -> None: + mcp = ToolboxConsentServer() + model = ConsentModelServer(("learn___protected", "learn___later")) + headers = ConsentCallerHeaders() + mcp.before_consent = headers.later_queued.wait + async with ( + toolbox_connection(mcp, header_provider=headers.headers) as toolbox, + consent_model_client(model, concurrent=True) as chat_client, + ): + host = ResponsesHostServer( + Agent(client=chat_client, tools=toolbox, middleware=[headers]), + response_store=InMemoryResponseProvider(), + history_source=history_source, + ) + response, _ = await post_consent_response(host, {"input": "Run the tools", "stream": stream}) + assert response["status"] == "incomplete", response + assert headers.later_queued.is_set() + assert [call["name"] for call in mcp.calls] == ["learn___protected", "learn___later"] + results = [item for item in response["output"] if item["type"] == "function_call_output"] + assert len(results) == 2 + assert results[1]["output"] == "search-result" + + +@pytest.mark.parametrize( + "error", + [ + {"code": -32000, "message": "ordinary tool failure"}, + {"code": -32006, "message": "https://consent.example.com/unstructured"}, + {"code": -32006, "message": '{"errors": {}}'}, + {"code": -32006, "message": '{"errors": [null, 1, {"type": "mcp", "error": null}]}'}, + {"code": -32006, "message": '{"errors": [{"type": "mcp", "error": {"code": "OTHER"}}]}'}, + { + "code": -32006, + "message": json.dumps({ + "errors": [{"type": "mcp", "error": {"code": "CONSENT_REQUIRED", "message": "javascript:x"}}] + }), + }, + ], +) +async def test_toolbox_oauth_consent_does_not_classify_unvalidated_errors(error: dict[str, Any]) -> None: + mcp = ToolboxConsentServer() + mcp.error = error + model = ConsentModelServer() + async with toolbox_connection(mcp) as toolbox, consent_model_client(model) as chat_client: + host = ResponsesHostServer(Agent(client=chat_client, tools=toolbox), response_store=InMemoryResponseProvider()) + response, _ = await post_consent_response(host, {"input": "Run the tools"}) + assert response["status"] == "completed", response + assert not any(item["type"] == "oauth_consent_request" for item in response["output"]) + assert len(mcp.calls) == 3 + + +@pytest.mark.parametrize("stream", [False, True]) +@pytest.mark.parametrize("history_source", ["host_managed", "inference_service"]) +async def test_toolbox_mixed_consent_errors_remain_function_failures( + stream: bool, history_source: Literal["host_managed", "inference_service"] +) -> None: + mcp = ToolboxConsentServer() + mcp.error["message"] = "tools/call failed " + json.dumps({ + "errors": [ + {"name": "learn", "type": "mcp", "error": {"code": "CONSENT_REQUIRED", "message": CONSENT_LINK}}, + {"name": "calendar", "type": "mcp", "error": {"code": "INTERNAL_ERROR", "message": "Upstream failure"}}, + ] + }) + model = ConsentModelServer() + async with toolbox_connection(mcp) as toolbox, consent_model_client(model) as chat_client: + host = ResponsesHostServer( + Agent(client=chat_client, tools=toolbox), + history_source=history_source, + response_store=InMemoryResponseProvider(), + ) + response, events = await post_consent_response(host, {"input": "Run the tools", "stream": stream}) + + assert response["status"] == "completed", response + assert not any(item["type"] == "oauth_consent_request" for item in response["output"]) + assert not any(event.get("item", {}).get("type") == "oauth_consent_request" for event in events) + results = [item for item in response["output"] if item["type"] == "function_call_output"] + assert len(results) == 3 + assert results[0]["output"] == results[2]["output"] == "search-result" + assert results[1]["output"].startswith("Error:") + assert "OAuth consent is required" not in results[1]["output"] + assert CONSENT_LINK not in json.dumps(response["output"]) + assert [call["name"] for call in mcp.calls] == ["learn___completed", "learn___protected", "learn___later"] + assert len(model.requests) == 2 + assert model.requests[1].get("tool_choice") != "none" + assert not model.history_errors + + +async def test_toolbox_oauth_consent_origin_allowlist_is_enforced() -> None: + mcp = ToolboxConsentServer() + model = ConsentModelServer() + async with toolbox_connection(mcp) as toolbox, consent_model_client(model) as chat_client: + host = ResponsesHostServer( + Agent(client=chat_client, tools=toolbox), + response_store=InMemoryResponseProvider(), + allowed_oauth_consent_origins=["https://other.example.com"], + ) + response, _ = await post_consent_response(host, {"input": "Run the tools"}) + assert response["status"] == "failed", response + assert not any(item["type"] == "oauth_consent_request" for item in response["output"]) + + +@pytest.mark.parametrize("stream", [False, True]) +@pytest.mark.parametrize("history_source", ["host_managed", "inference_service"]) +async def test_toolbox_oauth_consent_remains_distinct_from_native_policy_approval( + stream: bool, history_source: Literal["host_managed", "inference_service"] +) -> None: + mcp = ToolboxConsentServer() + mcp.tools = [ + toolbox_tool("always", name=name) for name in ("learn___completed", "learn___protected", "learn___later") + ] + model = ConsentModelServer() + sessions = ConsentSessionStoreProvider() + async with toolbox_connection(mcp) as toolbox, consent_model_client(model) as chat_client: + host = ResponsesHostServer( + Agent(client=chat_client, tools=toolbox), + history_source=history_source, + response_store=InMemoryResponseProvider(), + agent_session_store_provider=sessions, + ) + first, _ = await post_consent_response(host, {"input": "Run the tools", "stream": stream}) + approvals = [item for item in first["output"] if item["type"] == "mcp_approval_request"] + assert len(approvals) == 3 + assert not mcp.calls + assert not any(item["type"] == "oauth_consent_request" for item in first["output"]) + second, _ = await post_consent_response( + host, + { + "input": [ + { + "type": "message", + "role": "user", + "content": [{"type": "input_text", "text": "Keep this instruction."}], + }, + *[ + {"type": "mcp_approval_response", "approval_request_id": approval["id"], "approve": True} + for approval in approvals + ], + ], + "previous_response_id": first["id"], + "stream": stream, + }, + ) + assert second["status"] == "incomplete", second + assert [item["consent_link"] for item in second["output"] if item["type"] == "oauth_consent_request"] == [ + CONSENT_LINK + ] + results = [item for item in second["output"] if item["type"] == "function_call_output"] + assert len(results) == 3 + assert results[0]["output"] == "search-result" + assert results[2]["output"] == "search-result" + assert [call["name"] for call in mcp.calls] == ["learn___completed", "learn___protected", "learn___later"] + assert not model.history_errors + saved = await sessions.store.get(second["id"]) + assert saved is not None + if history_source == "inference_service": + assert saved.service_session_id == "resp_model_2" + assert len(model.requests) == 2 + provider_input = model.requests[1]["input"] + assert sum(item["type"] == "function_call_output" for item in provider_input) == 3 + assert not any(item["type"] in ("function_call", "mcp_approval_response") for item in provider_input) + assert any("Keep this instruction." in json.dumps(item) for item in provider_input) + + +async def test_generic_mcp_does_not_translate_toolbox_consent() -> None: + mcp = ToolboxConsentServer() + async with ( + httpx.AsyncClient(transport=httpx.MockTransport(mcp.handle)) as http_client, + MCPStreamableHTTPTool(name="generic", url="https://toolbox.invalid/mcp", http_client=http_client) as generic, + ): + with pytest.raises(ToolExecutionException): + await generic.call_tool("learn___protected") + + +async def test_toolbox_oauth_consent_background_run_reconciles_in_foreground() -> None: + mcp = ToolboxConsentServer() + model = ConsentModelServer() + async with toolbox_connection(mcp) as toolbox, consent_model_client(model) as chat_client: + host = ResponsesHostServer( + Agent(client=chat_client, tools=toolbox), + history_source="inference_service", + background_source="inference_service", + response_store=InMemoryResponseProvider(), + ) + response, _ = await post_consent_response(host, {"input": "Run the tools", "background": True}) + assert response["status"] == "incomplete", response + assert any(item["type"] == "oauth_consent_request" for item in response["output"]) + assert len(model.requests) == 2 + assert model.requests[0]["background"] is True + assert model.requests[1]["background"] is False + assert model.requests[1]["tool_choice"] == "none" + assert [item["type"] for item in model.requests[1]["input"]] == ["function_call_output"] * 3 + + +@pytest.mark.parametrize("stream", [False, True]) +@pytest.mark.parametrize("history_source", ["host_managed", "inference_service"]) +async def test_toolbox_oauth_consent_collects_requests_and_preserves_sibling_results( + stream: bool, history_source: Literal["host_managed", "inference_service"] +) -> None: + mcp = ToolboxConsentServer() + second_link = "https://calendar.example.com/authorize" + mcp.error["message"] = "tools/call failed " + json.dumps({ + "errors": [ + {"name": "learn", "type": "mcp", "error": {"code": "CONSENT_REQUIRED", "message": CONSENT_LINK}}, + {"name": "calendar", "type": "mcp", "error": {"code": "CONSENT_REQUIRED", "message": second_link}}, + ] + }) + model = ConsentModelServer() + async with toolbox_connection(mcp) as toolbox, consent_model_client(model) as chat_client: + host = ResponsesHostServer( + Agent(client=chat_client, tools=toolbox), + history_source=history_source, + response_store=InMemoryResponseProvider(), + ) + response, _ = await post_consent_response(host, {"input": "Run the tools", "stream": stream}) + assert response["status"] == "incomplete", response + assert [item["consent_link"] for item in response["output"] if item["type"] == "oauth_consent_request"] == [ + CONSENT_LINK, + second_link, + ] + results = [item for item in response["output"] if item["type"] == "function_call_output"] + assert len(results) == 3 + assert results[0]["output"] == results[2]["output"] == "search-result" + assert "OAuth consent is required" in results[1]["output"] + assert [call["name"] for call in mcp.calls] == ["learn___completed", "learn___protected", "learn___later"]