Skip to content

Python: TypeSafe connector should support image inputs and Box questions (Jev 1.14) #9410

Description

Problem

TypeSafe released Jev 1.14 (jev-1.14.0, jev-preview) with image inputs and a new Box question type that locates a target in an image. The agent-framework-typesafe connector rejects every non-text message content, so Agent Framework users cannot send an image to Jev, and Box questions cannot be used at all because they require an image. TypeSafe also requires the TypeSafe-Danger-Zone-Images: true header on any request with an image or a Box question, which the connector never sends.

Use Case and Evidence

Use Jev from an Agent Framework agent on document and image scenarios, for example checking whether a scanned form is signed (Noul) and locating the signature (Box) before cropping. Today the only option is to bypass the connector and call the TypeSafe SDK directly, losing middleware, telemetry and function invocation.

Desired Outcome

Users can pass images as Content.from_data(..., media_type="image/...") in messages, use typesafe_sdk.Box questions in response_format, and read the BoxAnswer from response.value, with the opt-in header handled by the connector.

Alternatives Considered

Calling typesafe_sdk.AsyncTypeSafeClient.system_one directly works (typesafe-sdk 0.7.5, already within the connector's >=0.7.1,<0.8.0 range, exposes Image, Box and BoxAnswer), but it bypasses the Agent Framework client contract.

Proposed Direction

  • Map inline image data content to typesafe_sdk.Image assets inside the existing messages state.
  • Send TypeSafe-Danger-Zone-Images: true through the SDK's extra_headers only when the request contains an image or a Box question; text-only requests are unchanged.
  • Keep rejecting remote image URIs (the SDK needs the bytes) and non-text function result items.
  • Document the feature as experimental, following TypeSafe's guidance.

I have an implementation with unit and transport-level tests ready on my fork (https://github.com/Zakariakhchiche/agent-framework/tree/typesafe-images-box) and will open the PR once the direction is agreed.

Scope and Non-Goals

  • No support for remote image URLs or rich (image) function results.
  • No change to streaming (still rejected) or to the terminal text, which keeps consolidating Choice/Score decisions only; Box answers stay available through response.value.
  • No long-context header (TypeSafe-Danger-Zone-Long-Context); that can be a separate change.

Language

Python

Contribution Intent

I am willing to implement this

Acknowledgements

  • I searched existing issues and did not find a duplicate.
  • I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 15, 2026
  2. removed
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 15, 2026
  3. eavanvalkenburg commented on Oct 27, 2026

    @eavanvalkenburg
    Member

    Thanks for the proposal. We agree this capability is worth considering within the existing TypeSafe integration: inline-image inputs and Box questions/answers, scoped to the supported provider contract. We will plan the experimental opt-in behavior and SDK/model compatibility alongside the existing #9411 before agreeing implementation details. This is separate from the Foundry-based, TypeSafe-independent direction in #9489, and is not a promise of implementation or timing.

  4. eavanvalkenburg commented on Oct 27, 2026

    @eavanvalkenburg
    Member

    /plan Coordinate the accepted inline-image and Box-question capability with existing PR #9411, not a competing implementation. Verify current official TypeSafe/Jev image and Box contracts and the actual minimum SDK/model versions; the declared SDK floor is 0.7.1, so availability in 0.7.5 alone is insufficient. Decide whether the experimental image header requires explicit client opt-in rather than automatic content-triggered consent. Preserve text-only behavior, remote-URI and rich-tool-result rejection, non-streaming scope and BoxAnswer access via response.value. Cover independent image/Box header triggers, transport bytes, SDK-floor compatibility and clear unsupported-input errors. Keep this TypeSafe-hosted extension separate from the Foundry-based, TypeSafe-independent #9489 design. Agree these details before marking implementation ready.

  5. eavanvalkenburg commented on Nov 12, 2026

    @eavanvalkenburg
    Member

    /plan Coordinate the accepted inline-image and Box-question capability with existing PR #9411, not a competing implementation. Verify current official TypeSafe/Jev image and Box contracts and the actual minimum SDK/model versions; the declared SDK floor is 0.7.1, so availability in 0.7.5 alone is insufficient. Decide whether the experimental image header requires explicit client opt-in rather than automatic content-triggered consent. Preserve text-only behavior, remote-URI and rich-tool-result rejection, non-streaming scope and BoxAnswer access via response.value. Cover independent image/Box header triggers, transport bytes, SDK-floor compatibility and clear unsupported-input errors. Keep this TypeSafe-hosted extension separate from the Foundry-based, TypeSafe-independent #9489 design. Agree these details before marking implementation ready.

  6. github-actions commented on Nov 12, 2026

    @github-actions
    Contributor

    Summary

    The ask is to let the TypeSafe connector (agent-framework-typesafe) send inline images to Jev and accept Box questions. BoxAnswer would be read from response.value. The maintainers accepted the capability in principle. The open question is how to agree the details with PR #9411 first, so there is one implementation and not two. This is separate from the Foundry-based design in #9489.

    What I could check

    • The connector currently rejects any content type other than text, text_reasoning, function_call and function_result (_chat_client.py ~L388-397).
    • It rejects non-text function results (~L433).
    • It rejects streaming (~L210).
    • It sets only a User-Agent header, and no experimental headers (~L169).
    • The dependency range is typesafe-sdk>=0.7.1,<0.8.0 (pyproject.toml).

    Not verified: this run had no network or package access. I could not check TypeSafe's official Jev 1.14 image/Box documentation. I also could not check which SDK release first added Image, Box and BoxAnswer, or which model versions accept them. The issue only says they exist in 0.7.5. I did not review PR #9411's code, as required.

    Recommendation

    Code changes are warranted. Images and Box cannot be reached through the connector today, and the only workaround is to bypass it, which loses middleware, telemetry and function invocation. Keep it as one implementation, in #9411. It should not be marked ready for implementation until the points below are agreed. Because the facts in points 1-3 are not yet verified, I am not giving a final plan.

    Questions before a plan

    1. Real SDK floor. The package declares >=0.7.1, but the issue only confirms 0.7.5. Which is the first SDK version that exports Image, Box, BoxAnswer and supports extra_headers? Please cite the changelog or release tag. If it is above 0.7.1, we must either raise the floor in pyproject.toml or guard the feature. A guard would raise a clear error when the SDK lacks image/Box support, and the connector would import the new types lazily. Which do you prefer? Raising the floor is simpler. A guard keeps text-only users on older SDKs.

    2. Official contract. Please link TypeSafe's documentation for the TypeSafe-Danger-Zone-Images header. It should state:

      • which model versions accept images (is it jev-1.14 and later only, or also jev-latest?);
      • the allowed image media types and size limits;
      • whether Box alone needs the header.

      Without this we cannot write the error messages or tests.

    3. Explicit opt-in or automatic consent. The header is named "Danger Zone", so TypeSafe treats it as a deliberate acceptance of experimental behavior. Sending it just because an image appears in a message would give consent on the user's behalf. The options are:

      • A: a client option, for example allow_experimental_images=True, on the client or in per-request options. Without it, images and Box questions raise a clear ChatClientInvalidRequestException.
      • B: automatic, content-triggered consent (the issue's current proposal).

    I lean towards A, since the maintainers raised it. It is safer, and the connector is not hiding what the user agreed to. The cost is one extra setting. Which do you want? If A, should the flag be sent for both triggers, or have separate flags?

    1. Independent header triggers. Please confirm the rules. The header is sent if the request has an image, or if it has a Box question, even with no image. In the second case the provider will reject the request, so we should fail early with a clear message. Text-only requests must send no extra headers at all. Please also confirm that headers from the user, such as User-Agent, are merged and not replaced.

    2. Scope guardrails. Please confirm these as written in the issue, each with a specific error message and a test:

      • only inline data content with an image/* media type;
      • remote image URIs rejected, because the SDK needs the bytes;
      • image items in function results rejected;
      • streaming still rejected;
      • the response text unchanged (Choice/Score only), with BoxAnswer read from response.value;
      • no long-context header.
    3. Transport bytes. Tests should check that the bytes sent match the original image exactly, with no re-encoding or truncation. They should also check the media type and the Image placement in the messages state. Is a transport-level test (a mocked HTTP layer) enough, or do you also want an opt-in integration test against a live Jev?

    4. Coordination. Please confirm that the PR author keeps Python: TypeSafe: support inline images and Box questions (Jev 1.14) #9411 as the only implementation and updates it to match the answers above. The package README, AGENTS.md and a sample should mark the feature as experimental. Nothing should reference Python: [Feature]: Support Microsoft-Decision-1 (Microsoft Foundry) through the TypeSafe connector #9489 beyond a note that it is a separate design.

    Proposed plan once these are settled (proposal only, nothing changed)

    1. Update the SDK floor or add the capability guard (question 1).
    2. Convert inline image content to Image assets, and add the opt-in check and header logic (questions 3 and 4).
    3. Add the explicit rejections from question 5, with clear messages.
    4. Add unit and transport tests. They should cover text-only requests being unchanged, each header trigger on its own, the opt-in on and off, the SDK-floor behavior, byte equality, and every rejection path.
    5. Update the README and add a short experimental sample.

    It is done when all of this passes and the answers to questions 1-3 are written into the PR.

    The answers will help shape the final plan.

    Generated by Issue Planning for #9410 · copilot · auto · ~20 AIC · ⌖ ~1 AIC · ⊞ 4.8K · ◷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

pythonUsage: [Issues, PRs], Target: Pythonready-for-implementationMaintainers agree the problem and direction are ready for implementation.

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions