refactor(api-proxy): read the image from a buffered call - #49
Draft
rickstaa wants to merge 1 commit into
Draft
Conversation
One bounded call returning a 120 KB image has nothing to stream: stream=True was there because call_runner's buffered path assumed a JSON object and raised on image/jpeg. With that fixed upstream the async with / aiter_bytes block collapses into result.content. The buffered path bounds the whole request, which streaming did not, so the call now passes a timeout the model can actually meet. The nginx Accept pin stays: the runner advertises an image at a fixed price, so the format is its choice, not a caller's. Closes #47 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Blocked on livepeer/livepeer-python-gateway#51 and #52 landing in
ja/live-runner. The example pins that branch, so nopyproject.tomlchange is needed; mark ready and merge once both are in.One bounded call returning a 120 KB image has nothing to stream.
stream=Truewas there becausecall_runner's buffered path assumed a JSON object and raised onimage/jpeg. With #51 the body arrives inresult.content, so theasync with/aiter_bytes()block collapses to one line.One thing worth knowing: the buffered path bounds the whole request with
call_runner's 5s default, which streaming did not (open_streamsetssock_read=None). Droppingstream=Truewithout a timeout fails after ~5s withfailed to reach endpointand no mention of a timeout, so the call passestimeout=180.The nginx
Acceptpin stays. It is not a workaround for the SDK bug: the runner advertises an image at a fixed price, so the response format is its choice, and the pin still holds for callers that are not this SDK or are on an older pin.Tested
End to end on the paid path against the stacked SDK branch (#51 + #52), with the pin in place: a real Hugging Face call returned a 1024x1024
image/jpeg(56313 bytes) intoresult.content. The temporary SDK pin used for that run was reverted before committing.Closes #47