Add Responses API streaming support #808

enyst · 2025-10-19T15:46:31Z

Summary

add streaming callback plumbing across the SDK so clients can subscribe to token deltas
implement Responses API streaming consumption with LiteLLM and expose an example plus docs
expand tests to cover streaming flows and enforce callback requirements

Testing

pytest tests/sdk/llm/test_responses_parsing_and_kwargs.py
pytest tests/sdk/agent/test_agent_step_responses_gating.py

fix: #450

Agent Server images for this PR

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant	Architectures	Base Image	Docs / Tags
java	amd64, arm64	`eclipse-temurin:17-jdk`	Link
python	amd64, arm64	`nikolaik/python-nodejs:python3.12-nodejs22`	Link
golang	amd64, arm64	`golang:1.21-bookworm`	Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:58286ad-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-58286ad-python \
  ghcr.io/openhands/agent-server:58286ad-python

All tags pushed for this build

ghcr.io/openhands/agent-server:58286ad-golang-amd64
ghcr.io/openhands/agent-server:58286ad-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:58286ad-golang-arm64
ghcr.io/openhands/agent-server:58286ad-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:58286ad-java-amd64
ghcr.io/openhands/agent-server:58286ad-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:58286ad-java-arm64
ghcr.io/openhands/agent-server:58286ad-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:58286ad-python-amd64
ghcr.io/openhands/agent-server:58286ad-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-amd64
ghcr.io/openhands/agent-server:58286ad-python-arm64
ghcr.io/openhands/agent-server:58286ad-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-arm64
ghcr.io/openhands/agent-server:58286ad-golang
ghcr.io/openhands/agent-server:58286ad-java
ghcr.io/openhands/agent-server:58286ad-python

About Multi-Architecture Support

Each variant tag (e.g., 58286ad-python) is a multi-arch manifest supporting both amd64 and arm64
Docker automatically pulls the correct architecture for your platform
Individual architecture tags (e.g., 58286ad-python-amd64) are also available if needed

xingyaoww · 2025-10-21T15:46:44Z

😭 oh my the merge conflict

github-actions · 2025-10-21T20:28:12Z

Coverage Report •

File	Stmts	Miss	Cover	Missing
openhands-sdk/openhands/sdk
__init__.py	20	2	90%	61–62
openhands-sdk/openhands/sdk/agent
agent.py	158	48	69%	84, 88, 146, 150–151, 160–161, 178–180, 187–189, 191, 195, 198–199, 201, 208, 235, 240, 260, 267, 289, 294, 305, 308, 331, 341–342, 363–365, 367, 379–380, 385–386, 406–407, 412, 424–425, 430–431, 463, 470–471
base.py	157	21	86%	154, 160, 172–174, 204–205, 211–213, 226, 234–235, 269, 310, 317, 330, 367–368, 378–379
utils.py	57	18	68%	63, 77, 83–84, 101–102, 105–107, 110, 163, 165–167, 169–170, 177, 202
openhands-sdk/openhands/sdk/conversation
conversation.py	19	1	94%	107
streaming_visualizer.py	166	119	28%	88–96, 100, 103–111, 116–117, 120–127, 129, 131–133, 136–139, 145, 147, 149, 152–160, 162, 165, 184, 188–189, 192–194, 196–197, 199, 202–203, 207–208, 210–212, 214–217, 224, 226–227, 231–232, 235–239, 242–250, 255, 259–260, 265–268, 270, 275–278, 283–285, 288–294, 301, 304–308, 317
openhands-sdk/openhands/sdk/conversation/impl
local_conversation.py	239	86	64%	135, 137–138, 171–173, 175, 182–183, 186–189, 197–198, 201, 208–209, 232, 237, 253, 270, 278–280, 284–285, 338–339, 342, 349, 354, 379–381, 384, 393, 409, 411, 413, 417, 419–421, 423, 425, 431–432, 445–446, 448, 450, 454–457, 474–475, 480, 485, 487, 496–499, 516, 518, 522, 527, 532, 537–540, 546, 549, 553, 556, 558–560, 562, 580, 582, 590–591
openhands-sdk/openhands/sdk/event
streaming.py	22	9	59%	19, 23–24, 26–29, 31, 33
openhands-sdk/openhands/sdk/llm
llm.py	503	236	53%	362, 367, 371, 375–377, 381–382, 393–394, 396–397, 401, 418, 478, 508, 529, 533, 548, 554–555, 574–575, 585, 610–615, 636–638, 642–643, 645, 648, 652, 664, 669–672, 679, 682, 690–695, 699–702, 704, 717–718, 720, 724, 726–727, 729–732, 734, 739–740, 742, 749, 752–757, 761–763, 775–779, 781–784, 786–791, 793–794, 798–800, 805–807, 809–810, 812, 814, 823–824, 826–828, 830, 834–835, 838–839, 845–846, 850–851, 874, 876, 878–882, 884, 887–894, 898–902, 905–913, 916–921, 1017–1018, 1021–1024, 1065, 1079, 1133, 1136–1138, 1141–1149, 1153–1155, 1158, 1161–1163, 1170–1171, 1180, 1187–1189, 1193, 1195–1200, 1202–1219, 1222–1226, 1228–1229, 1235–1244, 1257, 1271, 1276
openhands-sdk/openhands/sdk/llm/router
base.py	42	17	59%	44–45, 48, 64–65, 67, 70, 97–99, 103, 109–111, 114–115, 117
TOTAL	12794	5961	53%

enyst · 2025-10-23T16:51:52Z

@OpenHands understand why CI fails

============================================================
❌ Found 1 undocumented example(s):

examples/01_standalone_sdk/24_responses_streaming.py

⚠️ Please add documentation for these examples in the docs repo.

Error: Process completed with exit code 1.

And let's fix it, open a PR to docs repo on this.

openhands-ai · 2025-10-23T16:52:02Z

I'm on it! enyst can track my progress at all-hands.dev

openhands-ai · 2025-10-23T17:12:23Z

Summary of work

Why CI failed

The “Check Documented Examples” job flagged a missing docs entry for: examples/01_standalone_sdk/24_responses_streaming.py. It scans the docs repo for an expandable code block referencing each example. Since this example wasn’t documented, CI failed.

What I changed

Added a new guide documenting the Responses token streaming example:
- File: sdk/guides/llm-responses-streaming.mdx
- Includes the required expandable code block that references examples/01_standalone_sdk/24_responses_streaming.py
- Brief instructions, how it works (token callbacks, JSONL logging, visualizer), and a run snippet
Updated navigation to surface this page:
- docs.json: added "sdk/guides/llm-responses-streaming" under SDK > Guides > LLM Features

Pull request to the docs repo

Branch: add-responses-streaming-guide
PR: SDK: Document Responses token streaming example docs#48

Checklist

Reason for CI failure identified and addressed
Minimal, focused changes (only two files): sdk/guides/llm-responses-streaming.mdx, docs.json
Commit message includes Co-authored-by: openhands openhands@all-hands.dev
Opened a single PR to OpenHands/docs as requested

Next steps

Merge the docs PR. After it’s merged, re-run CI on agent-sdk PR Add Responses API streaming support #808 (or push a no-op). The “Check Documented Examples” job should pass.

_{View full conversation}

blacksmith-sh · 2025-10-30T12:59:18Z

[Automatic Post]: It has been a while since there was any activity on this PR. @enyst, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

enyst · 2025-10-31T23:12:08Z

@OpenHands Merge main branch and fix the conflicts.

openhands-ai · 2025-10-31T23:12:16Z

Uh oh! There was an unexpected error starting the job :(

blacksmith-sh · 2025-11-07T12:58:04Z

[Automatic Post]: It has been a while since there was any activity on this PR. @enyst, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

enyst · 2025-11-14T19:47:34Z

@OpenHands Read this PR and understand its purpose and approach. Read the readme to know how to build the project.

We just merged main, into this branch, and we need to fix the branch.

Run pre-commit on all files and fix.

openhands-ai · 2025-11-14T19:47:43Z

I'm on it! enyst can track my progress at all-hands.dev

github-actions · 2025-11-20T12:59:24Z

[Automatic Post]: It has been a while since there was any activity on this PR. @enyst, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

Co-authored-by: openhands <openhands@all-hands.dev>

xingyaoww · 2025-11-25T23:18:39Z

@OpenHands please merge from main, resolve all the conflicts and make sure pre-commit run --all-files works

openhands-ai · 2025-11-25T23:18:49Z

I'm on it! xingyaoww can track my progress at all-hands.dev

Resolved merge conflicts by: 1. Keeping streaming-related constants in llm.py 2. Updating make_llm_completion() in utils.py to support streaming callbacks 3. Using the unified make_llm_completion() function in agent.py with streaming support 4. Removed _add_security_risk_prediction property since security_analyzer was removed from AgentBase in main Co-authored-by: openhands <openhands@all-hands.dev>

- Add on_token parameter to all agent step() method signatures - Import ConversationTokenCallbackType where needed - Fix LLM router to pass on_token parameter to underlying LLM - Fix example 24_responses_streaming.py (service_id -> usage_id, visualize -> visualizer) - All pre-commit checks now passing Co-authored-by: openhands <openhands@all-hands.dev>

openhands-ai · 2025-11-25T23:32:39Z

✅ Task Completed Successfully

I've successfully merged the main branch into streaming-responses, resolved all conflicts, and fixed all issues to make pre-commit checks pass.

Summary of Changes

1. Merge Completed (commit 27f9653)

Merged latest main branch into streaming-responses
Resolved conflicts in 2 files:
- openhands-sdk/openhands/sdk/agent/agent.py: Removed _add_security_risk_prediction property (security_analyzer was removed in main)
- openhands-sdk/openhands/sdk/llm/llm.py: Kept streaming support while merging main changes

2. Fixed Type Errors (commit dbbd0cf)
After the merge, several type errors appeared due to the streaming API changes. Fixed:

LLM Router (openhands-sdk/openhands/sdk/llm/router/base.py):
- Added on_token parameter to completion() method signature
- Properly passes the parameter to underlying LLM instances
- Added import for TokenCallbackType
Test Files (6 files updated):
- Added on_token: ConversationTokenCallbackType | None = None parameter to all agent step() methods
- Added necessary imports for ConversationTokenCallbackType
Example File (examples/01_standalone_sdk/24_responses_streaming.py):
- Fixed service_id → usage_id (correct parameter name)
- Fixed visualize=False → visualizer=None (correct parameter name)
- Added import for DefaultConversationVisualizer

3. Verification

✅ All pre-commit checks passing (format, lint, pyright type checking)
✅ Changes pushed to remote branch streaming-responses

All changes are minimal and directly address the merge conflicts and type errors introduced by the merge. The streaming-responses branch is now up to date with main and ready for review.

_{View full conversation}

xingyaoww · 2025-11-25T23:36:15Z

@OpenHands please fix the failing actions on PR #808 at branch streaming-responses

openhands-ai · 2025-11-25T23:36:24Z

I'm on it! xingyaoww can track my progress at all-hands.dev

- Fix circular import between agent/utils.py and conversation modules by using lazy imports - Update test_agent_utils.py to include new streaming parameters (on_token, metadata, extra_body) - All tests now passing Co-authored-by: openhands <openhands@all-hands.dev>

Co-authored-by: openhands <openhands@all-hands.dev>

openhands-ai · 2025-11-25T23:57:04Z

✅ All Failing Actions Fixed on PR #808

I've successfully fixed all failing CI checks on the streaming-responses branch. Here's what was done:

Changes Made to agent-sdk Repository:

Fixed Circular Import Issue
- Modified openhands/sdk/agent/utils.py: Changed import source on line 15 to avoid circular dependency
- Modified openhands/sdk/conversation/impl/local_conversation.py: Added lazy imports in the ask_agent() method
Updated Tests for Streaming API
- Modified tests/sdk/agent/test_agent_utils.py:
  - Added metadata = {} attribute to the mock_llm fixture
  - Updated all 9 test functions to include new parameters for streaming support:
    - extra_body={"metadata": {}} and on_token=None for Completion API calls
    - metadata={} and on_token=None for Responses API calls

Changes Made to docs Repository:

Fixed Documentation Check Failure
- The CI was failing because the docs branch name didn't match the PR branch name
- Created matching streaming-responses branch in docs repo
- Added documentation for 4 new examples that were merged from main:
  - 25_agent_delegation.py → sdk/guides/agent-delegation.mdx
  - 26_custom_visualizer.py → sdk/guides/custom-visualizer.mdx
  - 27_observability_laminar.py → sdk/guides/observability-laminar.mdx
  - 28_ask_agent_example.py → sdk/guides/ask-agent.mdx

CI Status: All Checks Passing ✅

All 22/22 checks are now passing:

✓ check
✓ Check OpenAPI Schema
✓ pre-commit
✓ cross-tests
✓ agent-server-tests
✓ tools-tests
✓ sdk-tests
✓ check-examples
✓ build-binary-and-test (macos-latest & ubuntu-latest)
✓ All Docker multi-arch builds (java, python, golang)
✓ Coverage report updated

Commits:

Agent SDK: 7ac405db - "Fix circular import and update tests for streaming API"
Docs: 4f892c7 - "Add documentation for new examples (25-28)"

The PR is now ready for review with all CI checks passing! 🎉

_{View full conversation}

… not necessary

openhands-ai · 2025-11-26T03:02:03Z

Looks like there are a few issues preventing this PR from being merged!

GitHub Actions are failing:
- Pre-commit checks
- Run tests
- Check Documented Examples

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #808 at branch `streaming-responses`

Feel free to include any additional details that might help me get this PR into a better state.

_{^{You can manage your notification settings}}

blacksmith-sh · 2025-12-01T13:03:20Z

[Automatic Post]: It has been a while since there was any activity on this PR. @enyst, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

blacksmith-sh · 2025-12-07T12:53:42Z

[Automatic Post]: It has been a while since there was any activity on this PR. @enyst, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

blacksmith-sh · 2025-12-07T12:54:06Z

[Automatic Post]: I have assigned @neubig as a reviewer based on git blame information. Thanks in advance for the help!

enyst force-pushed the streaming-responses branch 4 times, most recently from db1963e to c3f24cd Compare October 19, 2025 19:04

Add streaming support for Responses API

d149370

enyst force-pushed the streaming-responses branch from c3f24cd to d149370 Compare October 20, 2025 08:15

Document LLM streaming refactor plan

d331abf

enyst force-pushed the streaming-responses branch 4 times, most recently from 48ea9e3 to 5f86198 Compare October 20, 2025 16:48

Refactor streaming chunk model and visualizer

e31b728

enyst force-pushed the streaming-responses branch from 5f86198 to e31b728 Compare October 20, 2025 18:21

Merge remote-tracking branch 'upstream/main' into streaming-responses

3983ce4

enyst force-pushed the streaming-responses branch from c1a2ced to 3983ce4 Compare October 21, 2025 18:36

OpenHands deleted a comment from openhands-ai bot Oct 21, 2025

enyst force-pushed the streaming-responses branch from 784f76f to 28f4a2c Compare October 21, 2025 20:22

Merge remote-tracking branch 'upstream/main' into streaming-responses

a341d0e

enyst force-pushed the streaming-responses branch from 28f4a2c to a341d0e Compare October 21, 2025 21:52

Merge remote-tracking branch 'upstream/main' into streaming-responses

031fcf1

Merge branch 'main' into streaming-responses

287c9c2

enyst and others added 2 commits November 20, 2025 14:42

Merge branch 'main' into streaming-responses

f920696

Simplify streaming visualizer and always-persist streaming panels

a65dbda

Co-authored-by: openhands <openhands@all-hands.dev>

openhands-agent added 2 commits November 25, 2025 23:24

openhands-agent added 2 commits November 25, 2025 23:47

Trigger CI re-run

847eaaa

Co-authored-by: openhands <openhands@all-hands.dev>

xingyaoww added 8 commits November 25, 2025 21:03

remove md

80c06f7

rename example

9859171

make LLMStreamChunk a basemodel

71fce09

clean up some merges

6a67bac

simplify local convo and remove streaming event since that's probably…

ab8961a

… not necessary

update the right init

fa57f08

rm streaming visualizer

66e2092

some attempt to simplify

9d1914c

xingyaoww mentioned this pull request Nov 26, 2025

Implement streaming for Chat Completions #1270

Merged

blacksmith-sh bot requested a review from neubig December 7, 2025 12:54

enyst closed this Dec 7, 2025

Add Responses API streaming support #808

Add Responses API streaming support #808

Uh oh!

Conversation

enyst commented Oct 19, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xingyaoww commented Oct 21, 2025

Uh oh!

github-actions bot commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

enyst commented Oct 23, 2025

============================================================ ❌ Found 1 undocumented example(s):

⚠️ Please add documentation for these examples in the docs repo.

Uh oh!

openhands-ai bot commented Oct 23, 2025

Uh oh!

openhands-ai bot commented Oct 23, 2025

Uh oh!

blacksmith-sh bot commented Oct 30, 2025

Uh oh!

enyst commented Oct 31, 2025

Uh oh!

openhands-ai bot commented Oct 31, 2025

Uh oh!

blacksmith-sh bot commented Nov 7, 2025

Uh oh!

enyst commented Nov 14, 2025

Uh oh!

openhands-ai bot commented Nov 14, 2025

Uh oh!

github-actions bot commented Nov 20, 2025

Uh oh!

xingyaoww commented Nov 25, 2025

Uh oh!

openhands-ai bot commented Nov 25, 2025

Uh oh!

openhands-ai bot commented Nov 25, 2025

✅ Task Completed Successfully

Summary of Changes

Uh oh!

xingyaoww commented Nov 25, 2025

Uh oh!

openhands-ai bot commented Nov 25, 2025

Uh oh!

openhands-ai bot commented Nov 25, 2025

✅ All Failing Actions Fixed on PR #808

Changes Made to agent-sdk Repository:

Changes Made to docs Repository:

CI Status: All Checks Passing ✅

Commits:

Uh oh!

openhands-ai bot commented Nov 26, 2025

Uh oh!

blacksmith-sh bot commented Dec 1, 2025

Uh oh!

blacksmith-sh bot commented Dec 7, 2025

Uh oh!

blacksmith-sh bot commented Dec 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

enyst commented Oct 19, 2025 •

edited by github-actions bot

Loading

github-actions bot commented Oct 21, 2025 •

edited

Loading

============================================================
❌ Found 1 undocumented example(s):