Skip to content

Python: Add the realtime contract types - #45

Merged
adamdougal merged 4 commits into
feature-python-realtime-agentsfrom
adamdougal-realtime-contract-types
Nov 10, 2026
Merged

adamdougal merged 4 commits into
feature-python-realtime-agentsfrom
adamdougal-realtime-contract-types

Conversation

@adamdougal

@adamdougal adamdougal commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Motivation & Context

Python realtime agents (ADR 0054, spec 006) land as fifteen pieces on the fork's feature-python-realtime-agents
branch, which goes upstream later as one PR. This is the first piece. It adds the contract types every other realtime
piece compiles against, with no behaviour beyond construction, merging and serialization. Live conversation (#34) and
Voice Live (#29) start in parallel as soon as it merges: #29 writes each event kind's attributes and #34 reads them, so
this piece pins those shapes instead of leaving each piece to invent them.

The prototype (#10) monkeypatched Content to get a realtime_event content type and transcript status. Those patches
become real, additive changes to core here.

Description & Review Guide

  • What are the major changes?

    • A private agent_framework/_realtime/ subpackage, one module per concern, so parallel pieces rarely collide.
      Everything public in it is experimental under ExperimentalFeature.REALTIME and exported lazily from the root,
      with __all__ and __init__.pyi kept in step:
      • _kinds.py: RealtimeKindLiteral (spec 006's 19 kinds), the open RealtimeKind NewType, and one
        Realtime<Kind>Attributes TypedDict for each of the 15 kinds that have neutral attributes, with Required[...]
        on the keys every provider must supply.
      • _events.py: RealtimeEvent, which carries one provider event from a session to the conversation, or a raw
        command whose raw_representation is the wire event.
      • _options.py: RealtimeOptions with RealtimeTurnDetection, RealtimeTranscription and
        RealtimeNoiseReduction.
      • _clients.py: RealtimeClientProtocol and RealtimeClientSessionProtocol, runtime-checkable. Their docstrings
        carry the contract Build the Voice Live client on the shared realtime session #29 implements and Build the live realtime conversation #34 calls.
    • The realtime_event content type and Content.from_realtime_event(kind, ...). Content gains kind and
      service_event_type fields, which only realtime-event content serializes, so every other content type's dicts are
      unchanged. A kind's neutral attributes live under additional_properties["core"], and provider extras under the
      provider package's namespace, as spec 005 reserves. Every path that creates realtime-event content (the factory,
      the constructor, from_dict and unpickling) requires a non-empty string kind and accepts kinds it doesn't know.
      Other content types reject kind and service_event_type, as they already reject hosted.
    • The realtime exception hierarchy in exceptions.py, each class experimental: RealtimeException,
      RealtimeConnectionException(close_code=), RealtimeControlException(code=, param=) with
      RealtimeControlTimeoutException and RealtimeActiveResponseException, RealtimeUnsupportedException,
      RealtimeConversationClosedException and RealtimeBackpressureException.
    • Additive changes to stable APIs: Content.from_text(status=...); text merges, through + and response
      aggregation, that keep the latest status that isn't None, as reasoning text already did; the nine
      {text,audio,image}_{input,output,cache_read_input}_token_count UsageDetails keys, with their telemetry mapping
      left to Build realtime telemetry #43; and ExperimentalFeature.REALTIME and FOUNDRY_VOICE_AGENTS.
    • New tests in packages/core/tests/core/test_realtime_types.py, and a realtime section in
      packages/core/AGENTS.md. There's no sample, because nothing runs yet.
    • Unrelated reformat: two asserts in packages/core/tests/core/test_harness_memory.py arrived unformatted from
      upstream. The commit hook formats all of core, so this PR carries the whitespace-only fix, as Python: Record cancelled tool calls as errors #44 does. Identical
      edits in the two PRs merge cleanly.
    • Spec 006 corrections. They fill gaps the spec left open and reverse no decision on the map:
      • Event kinds: the table now names each kind's neutral attributes and marks the required ones. Realtime-event
        content carries kind and service_event_type as Content fields, with neutral attributes under
        additional_properties["core"], and RealtimeEvent uses the same layout. One response.done emits usage,
        then error if it failed, then response_done.
      • API shapes: the session protocol's send becomes send(value: RealtimeEvent | Message), with a rule for mapping
        each content type. Spec 006 had lost ADR 0035's way of handing input to a session. The exception tree gains
        RealtimeConnectionException, and "timings" becomes prefix_padding_ms and silence_duration_ms.
      • Control calls: the conversation enforces the acknowledgement timeout with asyncio.timeout(), and session
        control methods must be cancellation-safe. A raw command's wire event goes in raw_representation.
      • Provider clients: if opening a session fails or is cancelled, the client closes whatever it opened.
      • Changes to existing packages: it lists the attribute TypedDicts and the new Content fields.
      • ADR 0054 is unchanged.

    Usage. None of this existed before; the prototype patched Content at runtime.

    from agent_framework import Content, RealtimeEvent
    
    # A provider client tags a server event with a kind and its neutral attributes.
    event = RealtimeEvent(
        "speech_started",
        service_event_type="input_audio_buffer.speech_started",
        additional_properties={"core": {"item_id": "item_1", "audio_start_ms": 120}},
    )
    
    # The conversation turns a signal into realtime-event content, which round-trips.
    content = Content.from_realtime_event(
        event.kind,
        service_event_type=event.service_event_type,
        additional_properties=dict(event.additional_properties),
    )
    assert Content.from_dict(content.to_dict()).kind == "speech_started"
    
    # A final transcript keeps its status when it merges with the deltas before it.
    delta = Content.from_text("Hel", merge_mode="append")
    final = Content.from_text("Hello", merge_mode="replace", status="completed")
    assert (delta + final).status == "completed"
  • What is the impact of these changes? Everything new is experimental and additive. Chat users see one behaviour
    change: text merges now keep a status where they used to drop it, and nothing in the repository sets a status on
    text yet. Content.__init__ gains two keyword parameters, which only realtime-event content accepts and
    serializes. Content pickled before this change reads both fields as None. ag-ui's model-replay check, which accepts
    any ContentType, now also accepts realtime_event.

  • What do you want reviewers to focus on?

    • The design calls Adam approved in the
      design: neutral attributes go
      under additional_properties["core"], not at the top level; the session's send() takes a Message rather than
      adding outbound kinds; and there's one TypedDict per kind, which adds 24 root exports in all.
    • The attribute key names and which are required, especially expires_at, error_code and error_message in place
      of an error mapping, and truncated_by with the values "client" or "provider".
    • _kinds.py omits from __future__ import annotations. With it, Required[...] stays a string, so a TypedDict's
      runtime __required_keys__ comes out empty. The existing ComputerSafetyCheck in _types.py has that gap today;
      it's left as is here.
    • Two contract calls that later pieces build on: the conversation, not the protocol, owns the acknowledgement
      timeout, so session control methods must be cancellation-safe; and the new RealtimeConnectionException covers
      failed connects and dead sockets, which spec 006's tree had no class for.
    • A GPT-family review found that Content.from_dict and unpickling accepted realtime-event content with a missing
      or malformed kind. Two follow-up commits validate the kind on every construction path, and the re-review found
      nothing further.

Related Issue

Fixes #32. This PR targets feature-python-realtime-agents, so the ticket is closed by hand when it merges.

AI Assistance

  • No material AI assistance was used.
  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: GitHub Copilot wrote the design notes, implementation, tests, the spec 006 edits and this
description. Adam approved the design and the test list before implementation.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

adamdougal and others added 3 commits November 9, 2026 16:27
Adds the types every realtime piece builds on, in agent-framework-core under
ExperimentalFeature.REALTIME, in a new private agent_framework/_realtime/
subpackage:

- the realtime_event content type, Content.from_realtime_event(...), and
  Content's kind and service_event_type fields, serialized only on
  realtime-event content;
- RealtimeKindLiteral, the open RealtimeKind, and one Realtime<Kind>Attributes
  TypedDict per kind, whose neutral attributes live under
  additional_properties["core"];
- RealtimeEvent, RealtimeOptions and its nested TypedDicts, and the
  RealtimeClientProtocol and RealtimeClientSessionProtocol protocols, whose
  send() takes a raw RealtimeEvent or a Message;
- the realtime exception hierarchy, with RealtimeConnectionException added.

Additive changes to stable APIs: Content.from_text(status=...), text merges
that keep the latest status, the nine modality UsageDetails keys, and
ExperimentalFeature.REALTIME and FOUNDRY_VOICE_AGENTS.

Spec 006 gains the attribute keys, the session send() mapping, the
acknowledgement-timeout owner, the raw-command field and the response.done
event order. test_harness_memory.py carries the core-wide ruff format the
commit hook applies.

Fixes #32

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Content's constructor now requires a non-empty string kind for
realtime_event content, so Content.from_dict rejects serialized
realtime-event content with a missing or malformed kind, and
RealtimeEvent rejects a kind that isn't a string. Unknown non-empty kinds
are still accepted. Found by a GPT-family review.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Content.__setstate__ now applies the same kind rule as the constructor to
realtime_event content, beside its existing hosted validation. Found by
the GPT-family re-review.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@adamdougal

Copy link
Copy Markdown
Owner Author

GPT-family review (gpt-6.1-sol) on two axes, before Adam's review:

  • Spec (against the approved design, Build the realtime contract types #32 and spec 006): no significant issues.
  • Standards found one issue. Only Content.from_realtime_event() checked the kind, so Content.from_dict accepted realtime-event content with a missing, empty or non-string kind, and RealtimeEvent(7) was accepted. That breaks CODING_STANDARD's rule that trust-boundary inputs must be validated. 07c9723 moves the check into Content.__init__ and makes the shared guard require a non-empty str. A re-review then found that Content.__setstate__ still bypassed it, and 4010052 adds the same check there, beside the existing hosted validation. Both have parametrized regression tests, and unknown non-empty kinds are still accepted.

A final re-review of the full diff found no remaining material issues.

@adamdougal

Copy link
Copy Markdown
Owner Author

Local checks on 401005255, Python 3.13.11, run from python/:

Command Result
uv run poe check Syntax 86/86, pyright 43/43 and sample checks 215/215 passed; package tests 42/43, with the one failure the flaky test below. The run stops there, so I ran the remaining steps separately.
uv run poe check -S Passed: samples ruff, then ty, pyrefly and pyright.
uv run poe markdown-code-lint Passed.
uv run poe check -P core Passed: syntax, pyright, the test checkers (mypy, pyrefly, ty, zuban) and tests, 9629 passed, 23 skipped, 2 xfailed. Core coverage is 92%, and every new _realtime module and exceptions.py is at 100%.
uv run pytest packages/core/tests/core/test_realtime_types.py 109 passed.

The one failure is packages/tools's test_windows_timeout_sentinel_does_not_reuse_exiting_process, which expected exit code 130 and got 124. It's the timing-sensitive ShellSession test that #44 found flaky on the base branch too, and nothing in this PR reaches it. uv run poe test -P tools then failed it 1 time in 3 and passed the other 2.

A full uv run poe check on the first commit's tree passed every task, markdown lint included. On Python 3.14, declarative's pyright fails because powerfx is pinned to python_version < '3.14', so the checks above use a 3.13 venv, as #44's did.

CI on this PR passes: Python Tests on 3.11 to 3.15 for Ubuntu and Windows, and python-tests-coverage.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Non-realtime content accepts realtime-only fields that serialization silently drops, causing metadata loss on round trips and equality with untagged content.

2 open issues
What changed in this PR

Adds the foundational Python realtime contracts that future provider clients and live conversations will share, without adding live conversation behavior.

Changes:

  • Define experimental event kinds, attribute schemas, options, protocols, and exceptions.
  • Extend core content serialization, text-status merging, and modality-specific usage counts.
  • Add contract tests and align contributor guidance and spec 006.
File Description
python/​packages/​core/​tests/​core/​test_realtime_types.py Tests construction, serialization, merging, protocols, and exports.
python/​packages/​core/​tests/​core/​test_harness_memory.py Whitespace-only formatting of two assertions.
python/​packages/​core/​AGENTS.md Documents realtime modules and text-status behavior.
python/​packages/​core/​agent_framework/​exceptions.py Adds the experimental realtime exception hierarchy.
python/​packages/​core/​agent_framework/​_types.py Adds realtime content, status preservation, and modality usage keys.
python/​packages/​core/​agent_framework/​_realtime/​_options.py Defines neutral realtime configuration types.
python/​packages/​core/​agent_framework/​_realtime/​_kinds.py Defines event kinds and neutral attribute schemas.
python/​packages/​core/​agent_framework/​_realtime/​_events.py Adds the provider event container.
python/​packages/​core/​agent_framework/​_realtime/​_clients.py Defines client and session contracts.
python/​packages/​core/​agent_framework/​_realtime/​__init__.py Establishes the private realtime package.
python/​packages/​core/​agent_framework/​_feature_stage.py Adds realtime and Foundry voice feature identifiers.
python/​packages/​core/​agent_framework/​__init__.pyi Adds realtime exports to the typing surface.
python/​packages/​core/​agent_framework/​__init__.py Registers lazy realtime exports.
docs/​specs/​006-python-realtime-agents.md Clarifies event layouts, input mapping, timeouts, and exceptions.

Review effort: Max

- Tests: imports move to the top of the file, and tests that only asserted
  declarations (enum members, the kind list, TypedDict keys, experimental
  metadata, exception parents, protocol shapes and root-export lists) are
  removed. The behaviour tests remain.
- Content rejects kind and service_event_type on non-realtime content types,
  which would otherwise drop them silently on serialization, as it already
  rejects hosted on non-call content.
- RealtimeClientProtocol.start_session documents that a client closes what it
  opened when opening fails or is cancelled, and that rejected initial
  settings raise RealtimeControlException. Spec 006 says the same.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@adamdougal

Copy link
Copy Markdown
Owner Author

Local checks on 71b98a3cb (review changes; core only), Python 3.13.11, run from python/:

Command Result
uv run poe check -P core Passed: syntax, pyright, the test checkers (mypy, pyrefly, ty, zuban) and tests, 9560 passed, 23 skipped, 2 xfailed. Core coverage is 92%, and every new _realtime module and exceptions.py is still at 100%.
uv run pytest packages/core/tests/core/test_realtime_types.py 40 passed.

The whole-repo results in the earlier comment still apply: this commit changes only core, its tests, and one sentence of spec 006.

@adamdougal
adamdougal merged commit c5b461c into feature-python-realtime-agents Nov 10, 2026
23 of 26 checks passed

This branch had an error being deployed

1 failed deployment
github-app-auth — 71b98a3c Deployed Nov 10, 2026 by adamdougal via add_label #15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants