Skip to content

[FR]: Render image URIs inline within the Test/Sample Agent Chat UI #12

@bhavdipFunnmedia

Description

@bhavdipFunnmedia

Is there an existing issue for this?

  • I have searched the existing issues

Describe the problem

Use Case / Background

I am currently developing an Android 16 app (com.example.googleio2026) that integrates with the new AppFunctions SDK to log and track daily health data (hydration tracking).

When a user asks the assistant for a data summary (e.g., "Show me my last 7 days progress"), returning text alone isn't optimal for the UX. I have implemented a system-level @AppFunction that successfully draws a bar chart onto a headless canvas, saves it to a local cache file, and passes the asset back via an android.net.Uri wrapped inside an @AppFunctionSerializable class.

The Problem

When testing this behavior using the official Sample/Test Agent App (the chat companion UI), the app successfully executes the function but only prints out the raw string data or data class layout in the chat bubble. It currently does not resolve or display the android.net.Uri image asset inline.

While the data contract verifies perfectly via raw adb shell cmd app_function outputs, it is difficult to evaluate the visual conversational flow of agentic UI experiences locally.

Proposed Enhancement

It would be incredibly helpful if the Sample Agent App's chat interface could natively support rendering image assets returned by functions.

Specifically:

  1. Detect if an element inside an @AppFunctionSerializable return payload is an android.net.Uri.
  2. Check if the mime-type/extension is an image asset (e.g., PNG, JPEG).
  3. Utilize an image loading library (like Coil or Glide) within the Compose chat bubble structure to display the image inline next to the text summary, managing the necessary temporary URI read permission grants (AppFunctionUriGrant).

Code Example of the Payload

@AppFunctionSerializable
data class WeeklyProgressResponse(
    val textSummary: String,
    val barChartUri: Uri // <-- Requesting the test app UI to parse and display this asset
)

class HydrationStatsFunctions : AppFunctionTarget {
    @AppFunction
    suspend fun getPastWeekProgress(context: AppFunctionContext): WeeklyProgressResponse {
        // ... business logic generating local chart cache file
        return WeeklyProgressResponse(
            textSummary = "You hit your hydration target 5 out of the last 7 days!",
            barChartUri = fileProviderUri
        )
    }
}


### Describe the solution

I would like the Test Agent App (the ChatApp testing interface) to natively support and render image content URIs that are returned within `@AppFunctionSerializable` objects.

Specifically, the requested solution should:
1. Detect if any parameter inside the returned `@AppFunctionSerializable` data payload contains an `android.net.Uri`.
2. Inspect the content type/extension. If it evaluates to a supported image asset (such as image/png or image/jpeg), the ChatApp's Compose layout engine should resolve the asset.
3. Automatically handle the platform-level URI security boundary requirements (such as checking/requesting an `AppFunctionUriGrant`).
4. Display the image inline directly next to or below the regular `textSummary` data block inside the user's conversation bubble stream using an image-loading framework (e.g., Coil or Glide).

### Additional context

Thank you for providing these tools to the community! This enhancement would make local agent testing significantly closer to what the real-world consumer Gemini assistant app will experience.

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions