feat: deliver managed images through compose - #126
Conversation
|
Current-head local evidence for
Coverage includes genuine fresh-process PNG/JPEG resolution after source deletion; isolated write/sync/ENOSPC durability failures; actual task-manager detachment followed by originating-turn interruption and deferred typed-image completion; compose-to-native Responses image encoding; nested/deduplicated references and text spill; provider switching and TUI live/replay/fallback behavior. The default highly concurrent macOS test run hit existing shell/subagent timing failures; the complete bounded-concurrency rerun passed. Provider delivery is verified through production request encoding, not a live authenticated model call. Independent code review approved this exact SHA. That is not a submitted GitHub approval; the repository's required GitHub approval remains outstanding. No merge or auto-merge is authorized. |
7a642d7 to
2f76d11
Compare
Summary
Keep
composeas the only model-exposed tool while adding durable, session-scoped File references and a hiddenread_filecallable. PNG/JPEG references selected by the final Runlet return deliver typed images through native tool output or an ordinary user-image transport fallback, and render in expanded TUI tool cards.Motivation
Implement phase 1 of the approved four-phase plan: compose image reads without exposing another top-level tool or embedding pixels in intermediate JSON. Lack of native image tool-output support must not block image delivery when ordinary user-image input is available.
Impact
The verified
openai-subscription:gpt-5.4route retains native image tool output. Other subscription models, includinggpt-6-astra, and the OpenRouter/Speakeasy adapters use their ordinary user-role image encoding. Their textual tool results preserve diagnostics and direct the model to image attachments following the complete tool-result batch. This transport fallback does not imply every selected model supports vision.The initial reader accepts nonanimated PNG and JPEG. Managed objects survive source deletion and process restart; no automatic garbage collection is introduced. Transformations, multimodal subagents, and Markdown/all-origin rendering remain later phases.
Technical details
Durable and session-authorized snapshots
A strict versioned descriptor identifies an immutable binary envelope containing authoritative metadata and a digest-verified payload. Publication requires successful disk and directory-entry durability barriers. Resolution derives its namespace from the actual session, rejects traversal and symlinks, and grants no implicit cross-session or fork access.
Bounded final-return selection
Only references reachable from the final compose return are selected. Traversal, descriptor strings, every reference occurrence, duplicate metadata, labels, encoded bytes, and decoded pixels are bounded before delivery. Both compose execution entry points share finalization, including detached completion. Errors distinguish a completed program from failed delivery so callers do not blindly repeat side effects.
Request-only fallback without duplicate tool answers
Canonical transcripts retain typed tool results. Provider adapters project non-native images into user-role attachments only in outgoing requests, after all outstanding results in the batch. Tool text, labels, diagnostics, and nested-result provenance remain available. Detached completion notifications are projected without answering their already-acknowledged tool calls again. Replay and authenticated continuations reuse retained images without rerunning compose or persisting synthetic user turns; existing image normalization and provider limits still apply.
Separate text and image presentation
The 8 KiB compose text budget excludes media payloads and reserves adjacent JSON Pointer labels before spilling ordinary JSON. TUI tool cards reuse the existing bounded image decoder/cache. Raw-output fallback suppresses image payloads without discarding surrounding diagnostic text or literal data-URL scheme mentions.
Demo