Skip to content

docs(#3996): describe portable generated media and Gemini output limits #4030

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dgageot merged 3 commits into from
Sep 10, 2026
Merged

docs(#3996): describe portable generated media and Gemini output limits #4030

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8614f38
docs: explain the workspace artifact lifecycle
aheritier Jul 27, 2026
623c130
docs: describe generated-image requests and workspace-only saves
aheritier Jul 27, 2026
d5016a8
docs: clarify uncapped generated-media storage and resolution
aheritier Sep 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 28 additions & 25 deletions docs/configuration/models/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ models:
pdf: boolean # Optional: whether the model accepts PDF attachments
audio: boolean # Optional: whether the model accepts audio attachments
video: boolean # Optional: whether the model accepts video attachments
output_capabilities: # Optional: generative output capabilities; explicit override, else resolved from models.dev
image: boolean # Optional: whether the model is declared able to generate image output
output_capabilities: # Optional: override generative output capabilities (otherwise detected from models.dev)
image: boolean # Optional: whether the model can generate image output
cost: # Optional: explicit token pricing (USD per 1M tokens)
input: float # Optional: price per 1M input tokens
output: float # Optional: price per 1M output tokens
Expand Down Expand Up @@ -76,7 +76,7 @@ models:
| `track_usage` | boolean | ✗ | Track and report token usage for this model |
| `routing` | array | ✗ | Rule-based routing to different models. See [Model Routing](../routing/index.md). |
| `capabilities` | object | ✗ | Override attachment (input) capabilities for this model. See [Attachment Capability Overrides](#attachment-capability-overrides). |
| `output_capabilities` | object | ✗ | Generative output capabilities for this model, e.g. image generation, resolved from an explicit override or the models.dev catalogue. Cannot be combined with `first_available`. See [Output Capabilities](#output-capabilities). |
| `output_capabilities` | object | ✗ | Override generative output capabilities for this model, e.g. image generation. Omitted flags are detected from models.dev; explicit values take precedence. Cannot be combined with `first_available`. See [Output Capabilities](#output-capabilities). |
| `cost` | object | ✗ | Explicit token pricing in USD per 1M tokens, overriding the built-in catalogue. See [Custom Token Pricing](#custom-token-pricing). |
| `provider_opts` | object | ✗ | Provider-specific options (see provider pages) |
| `title_model` | string | ✗ | Model used for session-title generation. Can be a named model from the `models:` section or an inline `provider/model` string. When omitted, the agent's primary model generates titles. Cannot be combined with `first_available`. |
Expand Down Expand Up @@ -151,14 +151,15 @@ See [`examples/capability-overrides.yaml`](https://github.com/docker/docker-agen
[`examples/strip-unsupported-media.yaml`](https://github.com/docker/docker-agent/blob/main/examples/strip-unsupported-media.yaml) for a fixture demonstrating the
stripping behaviour with and without an override.

## Output Capabilities
### Output capabilities

`output_capabilities` declares what a model can generate, as opposed to
`capabilities`, which declares what it accepts as input. When `image` is
omitted (including an omitted `output_capabilities` block), Docker Agent
resolves it from the models.dev catalogue's declared output modalities;
matching on the model name string is deliberately avoided as unreliable.
Models absent from the catalogue conservatively resolve to no image output.
`output_capabilities` overrides what a model can generate, as opposed to
`capabilities`, which overrides what it accepts as input. Resolution follows
one precedence chain: explicit `false`, explicit `true`, then an exact
models.dev record whose `Modalities.Output` contains `image`. An omitted image
flag (including `output_capabilities: {}`) therefore uses catalogue metadata;
an unknown model or unavailable catalogue leaves image output disabled. Docker
Agent never infers this capability from the model name.

```yaml
models:
Expand All @@ -173,21 +174,23 @@ models:
| --------------------------- | ------- | -------------------------------------------------------------|
| `output_capabilities.image` | boolean | Whether the model is declared able to generate image output |

Leaving `image` unset or `false` (explicit or resolved from the catalogue)
preserves existing behavior. When it resolves to `true` — explicitly or via
the catalogue — Docker Agent's Gemini provider asks the model for `TEXT`
and `IMAGE` response modalities on ordinary chat completions, and rejects,
before any request is sent, a request that combines image output with
custom function tools or structured output. Docker Agent does not yet
render generated image bytes back to the user; see
[Generated Images](../../providers/google/index.md#generated-images).

Declaring `image: true` does not exclude a model from session-title
generation: title (and compaction) calls are always dispatched without
image response modalities, so an image-output-capable model still produces
a plain text-only title. All configured candidates remain eligible in
order (dedicated `title_model`, then the agent's model, then its
fallbacks).
Omitting `output_capabilities`, using an empty block, or omitting `image` uses
models.dev metadata for that exact model when available. Setting `image`
explicitly overrides the catalogue; an explicit `false` has highest precedence
and disables image response modalities even when the catalogue lists image
output. Enabling image output only opts the model into behavior that keys off
that capability (for example, a provider-specific image-output request
contract); it does not guarantee that a provider will return an image.

Session-title and compaction requests omit image response modalities and
bypass the guard even for image-output-capable models; they do not explicitly
force TEXT-only output. On supported Google surfaces, the guard runs only when
image output resolves as enabled and an ordinary request includes custom tools
or structured output; matching requests are rejected locally. Google
server-side built-ins remain available. For a custom-tool conflict,
models.dev's `tool_call` capability makes the error say
whether the model lacks tool calls entirely or only cannot combine them with
image output; unavailable metadata keeps a conservative generic message.

> [!WARNING]
> **Constraint**
Expand Down
31 changes: 31 additions & 0 deletions docs/features/sessions/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,37 @@ Override the location with `-s`/`--session-db`, or by overriding the data direct
$ docker agent run agent.yaml --session-db ./sessions.db
```

## Generated Media Files

Some models (e.g. Gemini image-output models) can generate an image as part
of a reply. Docker Agent saves each generated image into the session's
workspace as an ordinary, untracked file. After the file and its manifest
entry are saved, it stores a portable copy in `session.db`. Reopening or moving
the session can therefore render the original generated bytes even when the
workspace file was edited, moved, or deleted. Ordinary follow-up turns replace
generated-media parts with metadata placeholders rather than resending the
bytes. User attachments are unaffected; explicitly attaching the file or
reading it through a tool can send its contents to a model. The workspace file
remains yours to edit, commit, or delete.

Existing session databases are upgraded in place when a newer Docker Agent
opens them. The database now carries portable generated-media bytes as well
as session records; an older binary may not understand the upgraded schema.
Do not assume a database opened by a newer version remains readable by an
older version. Sessions created before portable copies were introduced
continue to render through their manifest-gated workspace files. Historical
manifest entries for files outside the workspace are rejected. An unrecorded
path is never read. Every resolution rechecks the current manifest, so deleting
a session revokes its references even if workspace provenance is cached. A
legacy workspace file that was deleted, replaced by
a symlink, made invalid, or cannot be read appears as a short "unavailable"
note. Manifest and containment checks do not verify the integrity of
ordinary-file contents, which may have changed. Generated-media storage and
resolution have no byte cap. Deleting a session also deletes its stored
generated-media blobs. See
[Generated Media](../tui/index.md#generated-media) for naming, collision
handling, and rendering details.

## Resuming a Session

Pass `--session <id>` to continue a previous conversation instead of starting a new one:
Expand Down
111 changes: 106 additions & 5 deletions docs/features/tui/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,11 +294,112 @@ Attached files are also recorded on the session so sub-agents spawned by task tr

## Generated Media

Some models (e.g. Gemini image-output models) can generate binary media — typically an image — as part of their reply. When that happens, docker-agent writes the generated bytes into the session's workspace (the directory the session was started in) as an ordinary, visible file, and the assistant message keeps only a relative reference to that file plus its MIME type, display name, and size — never the raw bytes.

This keeps session JSON/database rows lightweight regardless of how many images a conversation accumulates, and the generated file is a regular workspace deliverable — visible to every tool, and yours to edit, commit, move, or delete — the same way generated code or text lands there.

Generated media is **not** automatically resent to the model on later turns: only the surrounding text is replayed in the outgoing history, the same way a large tool result would be summarized rather than repeated. This avoids silently ballooning the context window with image bytes on every follow-up message. A future step will add TUI rendering for these files (e.g. displaying the generated image inline); today this slice covers the domain, persistence, and safety mechanics only.
Some models (e.g. Gemini image-output models like `gemini-2.5-flash-image`)
are designed to generate an image directly as part of their reply, not just
describe one. When models.dev reports that a model can generate images, or
[`output_capabilities.image: true`](../../configuration/models/index.md#output-capabilities)
explicitly enables it, Docker Agent asks it for text *and* image output on the
models gateway, direct Gemini API, and Vertex AI. An explicit `false` disables
this behavior. See
[Google Gemini: Generated Images](../../providers/google/index.md#generated-images)
for exact configuration and limitations — ordinary image-output requests with
custom tools or structured output are rejected locally before any request is
sent. Session-title and compaction requests omit image response modalities and
skip this guard; they do not explicitly force TEXT-only output. Google
Search, Maps, and code-execution built-ins remain available. For custom tools,
models.dev metadata lets the error distinguish
a model that cannot call tools from an image-output request shape that cannot
combine both capabilities; unknown metadata keeps a conservative generic
message.

At a text-only stop, Docker Agent checks the last user prompt for phrases
such as "generate an image" or "draw a picture". A match preserves the reply
and adds this nonfatal warning: `The model returned text but no image for this
image-generation request. Try rephrasing the request.` Prompts without a
matching phrase do not trigger it. This is phrase matching, not semantic
intent detection: negated or quoted phrases can match, other wording can be
missed, and the check does not require an image-output-capable model. It does
not track a whole submission across tool calls, steering, stop hooks, or
handoffs. A terminal provider error skips this check, as does structured
output on the current agent model; the check does not inspect every override
or parse the reply to determine whether it is structured.

**Where images land.** Docker Agent attempts to save each generated image
as an ordinary workspace file and record it in the session manifest. After
both steps succeed, it stores a complete portable copy in the session database.
A failed portable-copy write does not remove the saved workspace file and
produces a per-item warning.
Database upgrades are in-place and older binaries may not understand the
upgraded schema — see [Sessions](../sessions/index.md#generated-media-files).
Generated files are untracked workspace files, yours to edit, commit, move,
or delete. A remote runtime writes to its own workspace; the local TUI does
not receive a remote binary-rendering path from this feature.

**Naming.** The model is instructed to honor an explicit prompt filename
such as `assets/red-panda.jpg` in a private naming marker. This is a request,
not a guarantee: emitted markers take precedence. When exactly one image is
returned without a marker, a single explicit filename found by a conservative
prompt parser is used. An unmarked item among several returned images does
not qualify. Otherwise names come from the provider, then `generated-1`,
`generated-2`, and so on. Parent directories are created when the validated
save succeeds. Two rules always apply:

- **The extension matches the data.** The image format is decided by the
provider (typically PNG) — asking for `sunshine.gif` or `diagram.svg`
does not transcode anything. If the model returns PNG data, the file is
saved as `sunshine.png` and a notice tells you so.
- **Existing files are never overwritten.** A name collision gets a dash
suffix instead: a second `red-panda.jpg` is saved as `red-panda-1.jpg`.
Publication requires hard-link support; filesystems without it fail safely
with a save warning instead of using a replacing rename.

**Paths stay in the workspace.** A prompt-directed target that is absolute,
`~`-rooted, or climbs above the workspace with `..` is not written outside the
owning session's workspace. Docker Agent discards the directory portion,
sanitizes the basename, saves it at the workspace root, and adds a bounded
warning to the turn. This does not prompt or wait for confirmation, including
over ACP and other interfaces without an elicitation consumer. General MCP and
tool elicitation is unaffected. If the owning session has no workspace root,
the save fails instead of falling back to the data directory. Relative
subdirectories remain supported after containment and symlink checks. An unusable
basename falls back to `generated-N`; a redirected save can still fail and warn
without discarding successful siblings or assistant text.

**Rendering.** Successfully resolved images can appear inline in the same
assistant turn, using Kitty-graphics support and `render_images` as described
under [Markdown Images](#markdown-images). Graphics-disabled or unsupported
terminals show a filename/path fallback. Resolution checks the owning session's
manifest before preferring its portable database copy; saved bytes can survive
workspace edits, deletion, or missing provenance. Without workspace provenance,
the label uses the recorded relative path rather than a verified absolute file. It falls back to the manifest-gated workspace file only when the
session store has no blob interface or the blob is not found. Other blob errors
fail closed. Historical manifest entries that identify an external root are
rejected. Stores without blob support and sessions created before portable
blobs were introduced continue to use legacy workspace files.
Generated-media resolution has no byte cap.

Ordinary outgoing history replaces generated-media parts with metadata
placeholders, so follow-up turns do not resend stored bytes. Explicitly
attaching a generated file or asking a tool to read it can send its contents
to a model. Legacy files still require manifest authorization, containment,
regular-file and symlink checks. These are not content-integrity checks: an
ordinary file's bytes may have changed. Failed resolution shows an unavailable
label rather than reading an unauthorized fallback.
This generated-media behavior is separate from the existing input bound for
ordinary Markdown images rendered from assistant text.

If a save fails (unwritable directory, full disk, …), only that image is
dropped, with a concise warning — the reply text and any sibling images in
the same turn are kept. If all saves in a media-only reply fail, an empty
assistant record may remain alongside the warnings. Disk-full, quota, and
unclassified failures currently use generic retry/debug advice.
Note also that an image-capable model can answer
with text only and generate no image at all; that is provider behavior, so
reword or repeat the prompt.

Inline image rendering in the TUI also covers a tool/MCP result that
returns an image, or a Markdown image reference to a file a tool actually
writes to disk — see [Markdown Images](#markdown-images) above.

### Team Context Budgets and Targeted Compaction

Expand Down
Loading
Loading