Repository navigation
Python: Add the realtime contract types - #45
adamdougal merged 4 commits into
Conversation
Adds the types every realtime piece builds on, in agent-framework-core under ExperimentalFeature.REALTIME, in a new private agent_framework/_realtime/ subpackage: - the realtime_event content type, Content.from_realtime_event(...), and Content's kind and service_event_type fields, serialized only on realtime-event content; - RealtimeKindLiteral, the open RealtimeKind, and one Realtime<Kind>Attributes TypedDict per kind, whose neutral attributes live under additional_properties["core"]; - RealtimeEvent, RealtimeOptions and its nested TypedDicts, and the RealtimeClientProtocol and RealtimeClientSessionProtocol protocols, whose send() takes a raw RealtimeEvent or a Message; - the realtime exception hierarchy, with RealtimeConnectionException added. Additive changes to stable APIs: Content.from_text(status=...), text merges that keep the latest status, the nine modality UsageDetails keys, and ExperimentalFeature.REALTIME and FOUNDRY_VOICE_AGENTS. Spec 006 gains the attribute keys, the session send() mapping, the acknowledgement-timeout owner, the raw-command field and the response.done event order. test_harness_memory.py carries the core-wide ruff format the commit hook applies. Fixes #32 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Content's constructor now requires a non-empty string kind for realtime_event content, so Content.from_dict rejects serialized realtime-event content with a missing or malformed kind, and RealtimeEvent rejects a kind that isn't a string. Unknown non-empty kinds are still accepted. Found by a GPT-family review. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Content.__setstate__ now applies the same kind rule as the constructor to realtime_event content, beside its existing hosted validation. Found by the GPT-family re-review. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
GPT-family review (gpt-6.1-sol) on two axes, before Adam's review:
A final re-review of the full diff found no remaining material issues. |
|
Local checks on
The one failure is A full CI on this PR passes: Python Tests on 3.11 to 3.15 for Ubuntu and Windows, and |
There was a problem hiding this comment.
🟡 Changes recommended
Non-realtime content accepts realtime-only fields that serialization silently drops, causing metadata loss on round trips and equality with untagged content.
2 open issues
What changed in this PR
Adds the foundational Python realtime contracts that future provider clients and live conversations will share, without adding live conversation behavior.
Changes:
- Define experimental event kinds, attribute schemas, options, protocols, and exceptions.
- Extend core content serialization, text-status merging, and modality-specific usage counts.
- Add contract tests and align contributor guidance and spec 006.
| File | Description |
|---|---|
python/packages/core/tests/core/test_realtime_types.py |
Tests construction, serialization, merging, protocols, and exports. |
python/packages/core/tests/core/test_harness_memory.py |
Whitespace-only formatting of two assertions. |
python/packages/core/AGENTS.md |
Documents realtime modules and text-status behavior. |
python/packages/core/agent_framework/exceptions.py |
Adds the experimental realtime exception hierarchy. |
python/packages/core/agent_framework/_types.py |
Adds realtime content, status preservation, and modality usage keys. |
python/packages/core/agent_framework/_realtime/_options.py |
Defines neutral realtime configuration types. |
python/packages/core/agent_framework/_realtime/_kinds.py |
Defines event kinds and neutral attribute schemas. |
python/packages/core/agent_framework/_realtime/_events.py |
Adds the provider event container. |
python/packages/core/agent_framework/_realtime/_clients.py |
Defines client and session contracts. |
python/packages/core/agent_framework/_realtime/__init__.py |
Establishes the private realtime package. |
python/packages/core/agent_framework/_feature_stage.py |
Adds realtime and Foundry voice feature identifiers. |
python/packages/core/agent_framework/__init__.pyi |
Adds realtime exports to the typing surface. |
python/packages/core/agent_framework/__init__.py |
Registers lazy realtime exports. |
docs/specs/006-python-realtime-agents.md |
Clarifies event layouts, input mapping, timeouts, and exceptions. |
Review effort: Max
- Tests: imports move to the top of the file, and tests that only asserted declarations (enum members, the kind list, TypedDict keys, experimental metadata, exception parents, protocol shapes and root-export lists) are removed. The behaviour tests remain. - Content rejects kind and service_event_type on non-realtime content types, which would otherwise drop them silently on serialization, as it already rejects hosted on non-call content. - RealtimeClientProtocol.start_session documents that a client closes what it opened when opening fails or is cancelled, and that rejected initial settings raise RealtimeControlException. Spec 006 says the same. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Local checks on
The whole-repo results in the earlier comment still apply: this commit changes only core, its tests, and one sentence of spec 006. |
c5b461c
into
feature-python-realtime-agents


Motivation & Context
Python realtime agents (ADR 0054, spec 006) land as fifteen pieces on the fork's
feature-python-realtime-agentsbranch, which goes upstream later as one PR. This is the first piece. It adds the contract types every other realtime
piece compiles against, with no behaviour beyond construction, merging and serialization. Live conversation (#34) and
Voice Live (#29) start in parallel as soon as it merges: #29 writes each event kind's attributes and #34 reads them, so
this piece pins those shapes instead of leaving each piece to invent them.
The prototype (#10) monkeypatched
Contentto get arealtime_eventcontent type and transcript status. Those patchesbecome real, additive changes to core here.
Description & Review Guide
What are the major changes?
agent_framework/_realtime/subpackage, one module per concern, so parallel pieces rarely collide.Everything public in it is experimental under
ExperimentalFeature.REALTIMEand exported lazily from the root,with
__all__and__init__.pyikept in step:_kinds.py:RealtimeKindLiteral(spec 006's 19 kinds), the openRealtimeKindNewType, and oneRealtime<Kind>AttributesTypedDict for each of the 15 kinds that have neutral attributes, withRequired[...]on the keys every provider must supply.
_events.py:RealtimeEvent, which carries one provider event from a session to the conversation, or a rawcommand whose
raw_representationis the wire event._options.py:RealtimeOptionswithRealtimeTurnDetection,RealtimeTranscriptionandRealtimeNoiseReduction._clients.py:RealtimeClientProtocolandRealtimeClientSessionProtocol, runtime-checkable. Their docstringscarry the contract Build the Voice Live client on the shared realtime session #29 implements and Build the live realtime conversation #34 calls.
realtime_eventcontent type andContent.from_realtime_event(kind, ...).Contentgainskindandservice_event_typefields, which only realtime-event content serializes, so every other content type's dicts areunchanged. A kind's neutral attributes live under
additional_properties["core"], and provider extras under theprovider package's namespace, as spec 005 reserves. Every path that creates realtime-event content (the factory,
the constructor,
from_dictand unpickling) requires a non-empty string kind and accepts kinds it doesn't know.Other content types reject
kindandservice_event_type, as they already rejecthosted.exceptions.py, each class experimental:RealtimeException,RealtimeConnectionException(close_code=),RealtimeControlException(code=, param=)withRealtimeControlTimeoutExceptionandRealtimeActiveResponseException,RealtimeUnsupportedException,RealtimeConversationClosedExceptionandRealtimeBackpressureException.Content.from_text(status=...); text merges, through+and responseaggregation, that keep the latest status that isn't
None, as reasoning text already did; the nine{text,audio,image}_{input,output,cache_read_input}_token_countUsageDetailskeys, with their telemetry mappingleft to Build realtime telemetry #43; and
ExperimentalFeature.REALTIMEandFOUNDRY_VOICE_AGENTS.packages/core/tests/core/test_realtime_types.py, and a realtime section inpackages/core/AGENTS.md. There's no sample, because nothing runs yet.packages/core/tests/core/test_harness_memory.pyarrived unformatted fromupstream. The commit hook formats all of core, so this PR carries the whitespace-only fix, as Python: Record cancelled tool calls as errors #44 does. Identical
edits in the two PRs merge cleanly.
content carries
kindandservice_event_typeasContentfields, with neutral attributes underadditional_properties["core"], andRealtimeEventuses the same layout. Oneresponse.doneemitsusage,then
errorif it failed, thenresponse_done.sendbecomessend(value: RealtimeEvent | Message), with a rule for mappingeach content type. Spec 006 had lost ADR 0035's way of handing input to a session. The exception tree gains
RealtimeConnectionException, and "timings" becomesprefix_padding_msandsilence_duration_ms.asyncio.timeout(), and sessioncontrol methods must be cancellation-safe. A raw command's wire event goes in
raw_representation.Contentfields.Usage. None of this existed before; the prototype patched
Contentat runtime.What is the impact of these changes? Everything new is experimental and additive. Chat users see one behaviour
change: text merges now keep a status where they used to drop it, and nothing in the repository sets a status on
text yet.
Content.__init__gains two keyword parameters, which only realtime-event content accepts andserializes. Content pickled before this change reads both fields as
None. ag-ui's model-replay check, which acceptsany
ContentType, now also acceptsrealtime_event.What do you want reviewers to focus on?
design: neutral attributes go
under
additional_properties["core"], not at the top level; the session'ssend()takes aMessagerather thanadding outbound kinds; and there's one TypedDict per kind, which adds 24 root exports in all.
expires_at,error_codeanderror_messagein placeof an
errormapping, andtruncated_bywith the values"client"or"provider"._kinds.pyomitsfrom __future__ import annotations. With it,Required[...]stays a string, so a TypedDict'sruntime
__required_keys__comes out empty. The existingComputerSafetyCheckin_types.pyhas that gap today;it's left as is here.
timeout, so session control methods must be cancellation-safe; and the new
RealtimeConnectionExceptioncoversfailed connects and dead sockets, which spec 006's tree had no class for.
Content.from_dictand unpickling accepted realtime-event content with a missingor malformed kind. Two follow-up commits validate the kind on every construction path, and the re-review found
nothing further.
Related Issue
Fixes #32. This PR targets
feature-python-realtime-agents, so the ticket is closed by hand when it merges.AI Assistance
AI assistance details: GitHub Copilot wrote the design notes, implementation, tests, the spec 006 edits and this
description. Adam approved the design and the test list before implementation.
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.