feat(agui): Add multimodal input support (image/video/audio) to AguiM…

[NoiAI]

…essage and AguiMessageConverter

Adds support for multimodal input (image, video, audio, document) in the AG-UI extension, aligning with the AG-UI Protocol InputContent specification.

• AguiMessage.content: String -> Object (backward compatible)
• AguiMessageConverter: InputContent[] -> ContentBlock conversion
• Supports both url and data (base64) source types

AgentScope-Java Version

[The version of AgentScope-Java you are working on, e.g. 1.0.12, check your pom.xml dependency version or run mvn dependency:tree | grep agentscope-parent:pom(only mac/linux)]

Description

[Please describe the background, purpose, changes made, and how to test this PR]

Checklist

Please check the following items before code is ready to be reviewed.

• Code has been formatted with mvn spotless:apply
• All tests are passing (mvn test)
• Javadoc comments are complete and follow project conventions
• Related documentation has been updated (e.g. links, examples, etc.)
• Code is ready for review

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

❌ Patch coverage is 84.61538% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...cope/core/agui/converter/AguiMessageConverter.java	83.67%	3 Missing and 5 partials ⚠️

📢 Thoughts on this report? Let us know!

[zhengxudong]

这个功能对我们项目非常重要，我们已经测试过可以正常使用，希望能尽快合并，非常感谢！

[LearningGp]

PTAL @chickenlj

[junnan2014]

We have a real-world need for this feature (Spring Boot + HarnessAgent + DashScope + AG-UI web client, where users send images/audio recordings from the browser). Currently we have to work around the protocol layer by smuggling attachments through forwardedProps and rebuilding Msg content blocks in an Agent decorator — this PR would remove that workaround entirely.

To help move this forward, I independently verified this PR today (2026-07-28):

• ✅ On the PR branch as-is: mvn test on the agui module passes — 231 tests, 0 failures (including the 81 new multimodal test cases).
• • ✅ Cherry-picked 5d6cb4b onto latest main (ed81704): applies with zero conflicts — git rename detection handles the module move to agentscope-extensions-protocol/ transparently.
• • ✅ Latest main + this patch: agui module passes 258 tests, 0 failures.
    The approach also aligns well with the AG-UI protocol InputContent spec (https://docs.ag-ui.com/drafts/multimodal-messages), and the downstream conversion path is already in place (ImageBlock/AudioBlock/VideoBlock in core, DashScopeMediaConverter etc. in model extensions) — this protocol-layer gap is the only missing piece for end-to-end multimodal via AG-UI.

@chickenlj @LearningGp Could this get a review? If the original author is no longer available, I'd be happy to help rebase, fill in the PR description checklist, and cover the remaining uncovered lines flagged by codecov.

[oss-maintainer]

3. @SuppressWarnings("unchecked") — Multiple unchecked casts from Map<String, Object>. This is acceptable given JSON deserialization constraints, but consider adding a typed DTO (e.g., InputContent) for type safety in a follow-up.

4. CLA not signed — The CLA assistant flag is still pending. Please sign the CLA at https://cla-assistant.io/agentscope-ai/agentscope-java?pullRequest=1380 before this can be merged.

5. May need rebase — This PR has been open since May. Please rebase onto current main to ensure no merge conflicts with recent AG-UI changes (e.g., PR #2452 metadata field, PR #2463 toolkit refactoring).

Verdict

Solid implementation of a much-requested feature. The core conversion logic is correct and well-tested. Addressing the document data loss (#1) and rebasing would make this merge-ready.

[jujn]

@junnan2014 Can you support multimodal output based on the latest agui module in the main branch? We are very welcome

[xzxiaoshan]

一、设计问题

问题 1：类型安全缺失

content 字段用 Object 声明，多模态内容用 List<Map<String, Object>> 承载：

private final Object content;                           // String or List<Map<String, Object>>
public List<Map<String, Object>> getMultimodalContent() // 退化到动态语言风格

• 完全丧失编译期类型检查，Map.get("type") 返回 Object 需强转
• 字段名拼写错误、类型不匹配等在编译期无法发现，只在运行时暴露
• 新增 InputContent 类型时，编译器无法提示遗漏

问题 2：概念命名错误 —— "多模态"窄化

public boolean isMultimodalContent()       // ← 命名问题
public List<Map<String, Object>> getMultimodalContent()

两个层面的概念错误：

AG-UI 协议层面：协议官方术语是 InputContent，不是 MultimodalContent。type:"text" 的纯文本也是 InputContent。把 content 数组命名为 "multimodal" 是对协议术语的降格解读。一条纯文本消息如果走 [{type:"text", text:"你好"}] 数组形式（协议完全合法），会被误判为 isMultimodalContent() == true。

AgentScope 模型层面：ContentBlock 体系包含 7 种块，其中 ThinkingBlock、ToolUseBlock、ToolResultBlock 与"模态"无关。ContentBlock 是通用内容单元，"多模态"只是其能力的子集，不是定义特征。

问题 3：两层概念未分离

PR 没有区分两个本应独立的概念层：

层	正确术语	关注点
AG-UI 协议（前端交互）	InputContent	用户输入了什么（text/image/audio/video/document）
AgentScope 模型（内部执行）	ContentBlock	消息由什么内容块组成（含 ThinkingBlock/ToolUseBlock/ToolResultBlock 等非模态块）

PR 直接用 Map<String, Object> 承载 AG-UI 协议数据，跳过了类型化的协议模型层。Converter 中手动 switch(type) + Map.get() 做字符串分派，本质是在用 Map 模拟一个本该存在的类型系统。

---

二、建议的类架构

架构总览

    AG-UI 协议层                        转换层                    AgentScope 模型层
 ┌─────────────────────┐          ┌──────────────┐          ┌──────────────────────┐
 │  AguiMessage        │          │              │          │  Msg                 │
 │  content:           │          │  InputContent│          │  content:            │
 │    MessageContent   │────────▶ │      ↕       │ ───────▶ │    List<ContentBlock>│
 │                     │          │  ContentBlock│          │                      │
 │  MessageContent     │          │              │          │  ContentBlock        │
 │  ┌─Text──────────┐  │          │  text    →   │          │  ├─TextBlock         │
 │  │(String value) │  │          │   TextBlock  │          │  ├─ImageBlock        │
 │  └───────────────┘  │          │              │          │  ├─AudioBlock        │
 │  ┌─Blocks────────┐  │          │  image   →   │          │  ├─VideoBlock        │
 │  │ List<Input    │  │          │   ImageBlock │          │  ├─ThinkingBlock     │
 │  │  Content>     │  │          │              │          │  ├─ToolUseBlock      │
 │  └───────────────┘  │          │  document→   │          │  └─ToolResultBlock   │
 │                     │          │   TextBlock  │          │                      │
 │  InputContent       │          │              │          │                      │
 │  (sealed interface) │          │ (pattern     │          │  (sealed interface)  │
 │  ├─TextContent──── │          │   matching,  │          │                      │
 │  ├─ImageContent─── │          │   编译期穷尽) │          │                      │
 │  ├─AudioContent─── │          │              │          │                      │
 │  ├─VideoContent─── │          │              │          │                      │
 │  └─DocumentContent─ │          │              │          │                      │
 └─────────────────────┘          └──────────────┘          └──────────────────────┘
     协议术语，不叫                                         内部模型术语，不叫
     "多模态"                                               "多模态"

注意：架构图右侧 AgentScope 模型层（ContentBlock 及其子类型）已存在，是 agentscope-java 既有的标准定义，无需重新设计。本架构建议中新增的仅是左侧 AG-UI 协议层的类型化模型（MessageContent / InputContent），以及中间 Converter 的改进——对接到既有 ContentBlock 体系。

设计要点

1. AguiMessage.content —— 类型安全的联合

MessageContent (sealed interface)
├── Text(String value)              → JSON: "你好"
└── Blocks(List<InputContent> parts) → JSON: [{"type":"text",...},{"type":"image",...}]

• 用 sealed interface 表达 string | InputContent[] 联合类型
• 命名用 Text / Blocks（或 hasBlocks()），不引入"多模态"
• 自定义 JSON 编解码器：反序列化按 JSON token（标量/数组）分派，序列化按 instanceof 透明输出

2. InputContent —— 类型化的协议模型

InputContent (sealed interface)
├── TextInputContent
├── ImageInputContent
├── AudioInputContent
├── VideoInputContent
└── DocumentInputContent

• 对应 AG-UI 协议的 InputContent 联合类型
• 取代 Map<String, Object>，提供编译期类型安全
• JSON 多态通过 type 字段分派（标准 @JSONType(seeAlso=...)）

3. Converter —— 编译期穷尽的类型映射

InputContent → ContentBlock（pattern matching switch）

TextInputContent     → TextBlock
ImageInputContent    → ImageBlock
AudioInputContent    → AudioBlock
VideoInputContent    → VideoBlock
DocumentInputContent → TextBlock（降级）

// sealed switch 编译期穷尽，新增类型时编译器报错提醒

** content 字段的反序列化处理代码片段：**

    // 指定该属性使用我们自定义的反序列化器
    @JsonDeserialize(using = MessageContentDeserializer.class)
    private MessageContent content;

public class MessageContentDeserializer extends JsonDeserializer<MessageContent> {
    @Override
    public MessageContent deserialize(JsonParser p, DeserializationContext ctxt) throws IOException {
        JsonToken currentToken = p.currentToken();
        // 场景 1：JSON 中是字符串 "content": "hello"
        if (currentToken == JsonToken.VALUE_STRING) {
            String textValue = p.getValueAsString();
            return new Text(textValue);
        } 
        
        // 场景 2：JSON 中是数组 "content": ["block1", "block2"]
        else if (currentToken == JsonToken.START_ARRAY) {
            // 使用 TypeReference 保证泛型类型正确解析
            List<String> list = p.readValueAs(new TypeReference<List<String>>() {});
            return new Blocks(list);
        } 
        
        // 场景 3：处理 null 值
        else if (currentToken == JsonToken.VALUE_NULL) {
            return null;
        } 
        
        // 场景 4：不支持的类型，抛出异常
        else {
            throw ctxt.wrongTokenException(p, MessageContent.class, currentToken, 
                    "Expected STRING or START_ARRAY for 'content' field");
        }
    }
}

命名修正对照

当前 PR	建议	原因
Object content	MessageContent content	类型安全联合，不用 Object
List<Map<String, Object>>	List<InputContent>	类型化协议模型，不用 Map
isMultimodalContent()	hasBlocks() 或 isText()	不引入"多模态"概念
getMultimodalContent()	getBlocks() 或通过 MessageContent 模式匹配访问	同上
switch(type) + Map.get()	switch(inputContent) pattern matching	编译期穷尽检查

---

建议对该 PR 进行更规范化设计修改。

AG-UI 官方messages文档：https://docs.ag-ui.com/concepts/messages

[xzxiaoshan]

@jujn #2518 已经按照规范的方案完成编码，方便的时候看一下。