From 230839752dbf77a59483dd0309012769f72aa9eb Mon Sep 17 00:00:00 2001 From: Rick Staa Date: Thu, 30 Jul 2026 13:10:51 +0200 Subject: [PATCH] refactor(api-proxy): read the image from a buffered call One bounded call returning a 120 KB image has nothing to stream: stream=True was there because call_runner's buffered path assumed a JSON object and raised on image/jpeg. With that fixed upstream the async with / aiter_bytes block collapses into result.content. The buffered path bounds the whole request, which streaming did not, so the call now passes a timeout the model can actually meet. The nginx Accept pin stays: the runner advertises an image at a fixed price, so the format is its choice, not a caller's. Closes #47 Co-Authored-By: Claude Opus 5 (1M context) --- api-proxy/README.md | 2 +- api-proxy/client.py | 25 +++++++++++-------------- 2 files changed, 12 insertions(+), 15 deletions(-) diff --git a/api-proxy/README.md b/api-proxy/README.md index 8ebb83d..003a895 100644 --- a/api-proxy/README.md +++ b/api-proxy/README.md @@ -15,7 +15,7 @@ Prerequisites (Docker, `uv`, and the not-yet-released `livepeer-gateway` SDK — ## How it's wired -The app is attached as a **static runner**: the orchestrator reads [runners.json](runners.json) via `-liveRunnerConfig` — app id, runner URL, single-shot mode, and the fixed price — and health-polls `/health` (an nginx `return 200`). The `/proxy` location proxies to the pinned model URL (`MODEL` in [compose.yml](compose.yml)) with `Authorization: Bearer ` added. The caller's body is the [Hugging Face text-to-image payload](https://huggingface.co/docs/inference-providers/tasks/text-to-image) forwarded verbatim — `{"inputs": ""}` — and the image comes back as **raw JPEG bytes**. The client calls it with `runner_selector` → `call_runner(..., stream=True)` ([client.py](client.py)) — discover, then one **single-shot** call per image, reading the bytes with `aiter_bytes()`; the orchestrator reserves a session per call and releases it when the response returns. Grep `# Livepeer:` in client.py to see the exact calls. +The app is attached as a **static runner**: the orchestrator reads [runners.json](runners.json) via `-liveRunnerConfig` — app id, runner URL, single-shot mode, and the fixed price — and health-polls `/health` (an nginx `return 200`). The `/proxy` location proxies to the pinned model URL (`MODEL` in [compose.yml](compose.yml)) with `Authorization: Bearer ` added. The caller's body is the [Hugging Face text-to-image payload](https://huggingface.co/docs/inference-providers/tasks/text-to-image) forwarded verbatim — `{"inputs": ""}` — and the image comes back as **raw JPEG bytes**. The client calls it with `runner_selector` → `call_runner` ([client.py](client.py)) — discover, then one **single-shot** call per image, reading the bytes from `result.content`; the orchestrator reserves a session per call and releases it when the response returns. Grep `# Livepeer:` in client.py to see the exact calls. ## Offering an API as a capability — what this shows diff --git a/api-proxy/client.py b/api-proxy/client.py index 3941679..305c3c2 100644 --- a/api-proxy/client.py +++ b/api-proxy/client.py @@ -1,16 +1,16 @@ #!/usr/bin/env python3 -"""api-proxy client: discover a runner, send a prompt, stream the image back. +"""api-proxy client: discover a runner, send a prompt, get the image back. The request body is the Hugging Face text-to-image payload as-is ({"inputs": ""}); the runner forwards it verbatim and the image comes -back as raw JPEG bytes, received with call_runner's streaming mode. +back as raw JPEG bytes in `result.content`. Livepeer integration (grep `# Livepeer:`): - 1. runner_selector() — discover orchestrators advertising the app - 2. call_runner(stream=True) — call the app through the orchestrator and read the - raw response bytes; on the paid path it answers the - 402 payment challenge inline (one fixed payment per - image). Single-shot needs no reserve/stop. + 1. runner_selector() — discover orchestrators advertising the app + 2. call_runner() — call the app through the orchestrator; a non-JSON response + arrives in `result.content`, and on the paid path the call + answers the 402 payment challenge inline (one fixed payment + per image). Single-shot needs no reserve/stop. """ from __future__ import annotations @@ -56,21 +56,18 @@ async def main() -> None: runner = cursor.candidates[0] log.info("app_url=%s", runner.url) - # NOTE: Streamed because the buffered path still assumes JSON; once - # livepeer-python-gateway#51 lands it returns result.raw. - stream = await call_runner( # Livepeer: 2 + result = await call_runner( # Livepeer: 2 runner=runner, # discovery metadata tells call_runner the price unit runner_url=runner.url.rstrip("/") + "/proxy", payload={"inputs": args.prompt}, # the HF payload, forwarded as-is signer_url=args.signer.strip() or None, - stream=True, # the image comes back as raw bytes, not JSON + timeout=180, # a hosted diffusion model can take tens of seconds ) - async with stream: - image = b"".join([chunk async for chunk in stream.aiter_bytes()]) + image = result.content or b"" # the image comes back as raw bytes, not JSON out_path = Path(args.output).expanduser() out_path.write_bytes(image) - log.info("wrote %s (%d bytes, %s)", out_path, len(image), stream.content_type) + log.info("wrote %s (%d bytes, %s)", out_path, len(image), result.content_type) except LivepeerGatewayError as exc: raise SystemExit(f"ERROR: {exc}") from exc