Repository navigation
Description
Problem
TypeSafe released Jev 1.14 (jev-1.14.0, jev-preview) with image inputs and a new Box question type that locates a target in an image. The agent-framework-typesafe connector rejects every non-text message content, so Agent Framework users cannot send an image to Jev, and Box questions cannot be used at all because they require an image. TypeSafe also requires the TypeSafe-Danger-Zone-Images: true header on any request with an image or a Box question, which the connector never sends.
Use Case and Evidence
Use Jev from an Agent Framework agent on document and image scenarios, for example checking whether a scanned form is signed (Noul) and locating the signature (Box) before cropping. Today the only option is to bypass the connector and call the TypeSafe SDK directly, losing middleware, telemetry and function invocation.
Desired Outcome
Users can pass images as Content.from_data(..., media_type="image/...") in messages, use typesafe_sdk.Box questions in response_format, and read the BoxAnswer from response.value, with the opt-in header handled by the connector.
Alternatives Considered
Calling typesafe_sdk.AsyncTypeSafeClient.system_one directly works (typesafe-sdk 0.7.5, already within the connector's >=0.7.1,<0.8.0 range, exposes Image, Box and BoxAnswer), but it bypasses the Agent Framework client contract.
Proposed Direction
- Map inline image data content to
typesafe_sdk.Imageassets inside the existingmessagesstate. - Send
TypeSafe-Danger-Zone-Images: truethrough the SDK'sextra_headersonly when the request contains an image or aBoxquestion; text-only requests are unchanged. - Keep rejecting remote image URIs (the SDK needs the bytes) and non-text function result items.
- Document the feature as experimental, following TypeSafe's guidance.
I have an implementation with unit and transport-level tests ready on my fork (https://github.com/Zakariakhchiche/agent-framework/tree/typesafe-images-box) and will open the PR once the direction is agreed.
Scope and Non-Goals
- No support for remote image URLs or rich (image) function results.
- No change to streaming (still rejected) or to the terminal text, which keeps consolidating
Choice/Scoredecisions only;Boxanswers stay available throughresponse.value. - No long-context header (
TypeSafe-Danger-Zone-Long-Context); that can be a separate change.
Language
Python
Contribution Intent
I am willing to implement this
Acknowledgements
- I searched existing issues and did not find a duplicate.
- I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.
Metadata
Metadata
Labels
Type
Projects
- StatusShow more project fieldsNo status
Activity
eavanvalkenburg commentedon Oct 27, 2026
Thanks for the proposal. We agree this capability is worth considering within the existing TypeSafe integration: inline-image inputs and Box questions/answers, scoped to the supported provider contract. We will plan the experimental opt-in behavior and SDK/model compatibility alongside the existing #9411 before agreeing implementation details. This is separate from the Foundry-based, TypeSafe-independent direction in #9489, and is not a promise of implementation or timing.
eavanvalkenburg commentedon Oct 27, 2026
/plan Coordinate the accepted inline-image and Box-question capability with existing PR #9411, not a competing implementation. Verify current official TypeSafe/Jev image and Box contracts and the actual minimum SDK/model versions; the declared SDK floor is 0.7.1, so availability in 0.7.5 alone is insufficient. Decide whether the experimental image header requires explicit client opt-in rather than automatic content-triggered consent. Preserve text-only behavior, remote-URI and rich-tool-result rejection, non-streaming scope and BoxAnswer access via response.value. Cover independent image/Box header triggers, transport bytes, SDK-floor compatibility and clear unsupported-input errors. Keep this TypeSafe-hosted extension separate from the Foundry-based, TypeSafe-independent #9489 design. Agree these details before marking implementation ready.
eavanvalkenburg commentedon Nov 12, 2026
/plan Coordinate the accepted inline-image and Box-question capability with existing PR #9411, not a competing implementation. Verify current official TypeSafe/Jev image and Box contracts and the actual minimum SDK/model versions; the declared SDK floor is 0.7.1, so availability in 0.7.5 alone is insufficient. Decide whether the experimental image header requires explicit client opt-in rather than automatic content-triggered consent. Preserve text-only behavior, remote-URI and rich-tool-result rejection, non-streaming scope and BoxAnswer access via response.value. Cover independent image/Box header triggers, transport bytes, SDK-floor compatibility and clear unsupported-input errors. Keep this TypeSafe-hosted extension separate from the Foundry-based, TypeSafe-independent #9489 design. Agree these details before marking implementation ready.
github-actions commentedon Nov 12, 2026
Summary
The ask is to let the TypeSafe connector (
agent-framework-typesafe) send inline images to Jev and acceptBoxquestions.BoxAnswerwould be read fromresponse.value. The maintainers accepted the capability in principle. The open question is how to agree the details with PR #9411 first, so there is one implementation and not two. This is separate from the Foundry-based design in #9489.What I could check
text,text_reasoning,function_callandfunction_result(_chat_client.py~L388-397).User-Agentheader, and no experimental headers (~L169).typesafe-sdk>=0.7.1,<0.8.0(pyproject.toml).Not verified: this run had no network or package access. I could not check TypeSafe's official Jev 1.14 image/
Boxdocumentation. I also could not check which SDK release first addedImage,BoxandBoxAnswer, or which model versions accept them. The issue only says they exist in 0.7.5. I did not review PR #9411's code, as required.Recommendation
Code changes are warranted. Images and
Boxcannot be reached through the connector today, and the only workaround is to bypass it, which loses middleware, telemetry and function invocation. Keep it as one implementation, in #9411. It should not be marked ready for implementation until the points below are agreed. Because the facts in points 1-3 are not yet verified, I am not giving a final plan.Questions before a plan
Real SDK floor. The package declares
>=0.7.1, but the issue only confirms 0.7.5. Which is the first SDK version that exportsImage,Box,BoxAnswerand supportsextra_headers? Please cite the changelog or release tag. If it is above 0.7.1, we must either raise the floor inpyproject.tomlor guard the feature. A guard would raise a clear error when the SDK lacks image/Boxsupport, and the connector would import the new types lazily. Which do you prefer? Raising the floor is simpler. A guard keeps text-only users on older SDKs.Official contract. Please link TypeSafe's documentation for the
TypeSafe-Danger-Zone-Imagesheader. It should state:jev-1.14and later only, or alsojev-latest?);Boxalone needs the header.Without this we cannot write the error messages or tests.
Explicit opt-in or automatic consent. The header is named "Danger Zone", so TypeSafe treats it as a deliberate acceptance of experimental behavior. Sending it just because an image appears in a message would give consent on the user's behalf. The options are:
allow_experimental_images=True, on the client or in per-request options. Without it, images andBoxquestions raise a clearChatClientInvalidRequestException.I lean towards A, since the maintainers raised it. It is safer, and the connector is not hiding what the user agreed to. The cost is one extra setting. Which do you want? If A, should the flag be sent for both triggers, or have separate flags?
Independent header triggers. Please confirm the rules. The header is sent if the request has an image, or if it has a
Boxquestion, even with no image. In the second case the provider will reject the request, so we should fail early with a clear message. Text-only requests must send no extra headers at all. Please also confirm that headers from the user, such asUser-Agent, are merged and not replaced.Scope guardrails. Please confirm these as written in the issue, each with a specific error message and a test:
image/*media type;Choice/Scoreonly), withBoxAnswerread fromresponse.value;Transport bytes. Tests should check that the bytes sent match the original image exactly, with no re-encoding or truncation. They should also check the media type and the
Imageplacement in themessagesstate. Is a transport-level test (a mocked HTTP layer) enough, or do you also want an opt-in integration test against a live Jev?Coordination. Please confirm that the PR author keeps Python: TypeSafe: support inline images and Box questions (Jev 1.14) #9411 as the only implementation and updates it to match the answers above. The package README,
AGENTS.mdand a sample should mark the feature as experimental. Nothing should reference Python: [Feature]: Support Microsoft-Decision-1 (Microsoft Foundry) through the TypeSafe connector #9489 beyond a note that it is a separate design.Proposed plan once these are settled (proposal only, nothing changed)
Imageassets, and add the opt-in check and header logic (questions 3 and 4).It is done when all of this passes and the answers to questions 1-3 are written into the PR.
The answers will help shape the final plan.