feat(ai): native Files API support across providers (upload adapters + file content source)

.changeset/native-files-api-support.md

@@ -0,0 +1,22 @@
+---
+'@tanstack/ai': minor
+'@tanstack/ai-event-client': minor
+'@tanstack/openai-base': minor
+'@tanstack/ai-openai': minor
+'@tanstack/ai-anthropic': minor
+'@tanstack/ai-gemini': minor
+'@tanstack/ai-fal': minor
+'@tanstack/ai-mistral': patch
+'@tanstack/ai-grok': patch
+'@tanstack/ai-openrouter': patch
+'@tanstack/ai-ollama': patch
+'@tanstack/ai-bedrock': patch
+---
+
+feat(ai): native Files API support across providers (upload adapters + `file` content source)
+
+Adds first-class support for provider **Files / storage APIs** so callers can upload media once and reference it by a provider-issued handle instead of re-sending base64 or a public URL each request (lower latency/bandwidth, no re-buffering on memory-constrained runtimes).
+
+- **New tree-shakeable `files` adapter kind** — `openaiFiles()`, `anthropicFiles()`, `geminiFiles()`, and `falFiles()`. Each exposes `upload()`, and (where the provider has a lifecycle API) `get()` / `delete()`. Drive them with the new `uploadFile()` / `getFile()` / `deleteFile()` activity functions. fal is upload-only.
+- **New `{ type: 'file' }` arm on `ContentPartSource`** — reference an uploaded handle in a chat message. Adapters map it to the right wire field: OpenAI (Responses) `input_image`/`input_file` `file_id`, Anthropic `file_id` message source (with the `files-api-2025-04-14` beta), Gemini `fileData.fileUri`, fal storage URL passthrough. Use `fileSourceFromHandle(handle)` to build the source from an uploaded `FileHandle`.
+- **Runtime provider routing** — a file handle only routes to the provider that issued it; adapters throw a clear error on a cross-provider handle, and providers/endpoints that can't consume a handle (image edits, Veo, Chat Completions images, Bedrock, Mistral, Grok, OpenRouter, Ollama) throw a clear "unsupported file source" error instead of silently mis-mapping.

docs/advanced/files-api.md

@@ -0,0 +1,133 @@
+---
+title: Files API
+id: files-api
+description: "Upload media once and reference it by a provider-issued handle with TanStack AI's tree-shakeable files adapters (OpenAI, Anthropic, Gemini, fal)."
+keywords:
+  - tanstack ai
+  - files api
+  - file upload
+  - file_id
+  - fileData
+  - multimodal
+---
+
+Provider **Files / storage APIs** let you upload a media asset once and reference it later by a lightweight handle, instead of re-sending base64 (or relying on the provider to re-fetch a public URL) on every request. That means large or reused inputs are uploaded a single time — lower latency and bandwidth, no re-buffering of base64 on memory-constrained runtimes (e.g. Cloudflare Workers) — plus access to provider-side file lifecycle (TTL, deletion).
+
+TanStack AI exposes this as a tree-shakeable **`files` adapter** per provider, paired with a `{ type: 'file' }` [content source](./multimodal-content.md#file-handle-files-api) you drop into a message.
+
+## Files adapters
+
+Each provider with a native surface has a factory: `openaiFiles()`, `anthropicFiles()`, `geminiFiles()`, and `falFiles()`. They read the same API-key env var as the provider's other adapters, or accept an explicit key.
+
+```typescript
+import { openaiFiles } from '@tanstack/ai-openai'
+import { geminiFiles } from '@tanstack/ai-gemini'
+import { anthropicFiles } from '@tanstack/ai-anthropic'
+import { falFiles } from '@tanstack/ai-fal'
+
+const files = openaiFiles() // reads OPENAI_API_KEY
+```
+
+### upload
+
+`upload()` accepts a `Blob` (memory-efficient — preferred for large assets) or `{ data, mimeType }` where `data` is base64. It returns a `FileHandle`:
+
+```typescript
+const handle = await openaiFiles().upload({
+  data: pdfBase64,
+  mimeType: 'application/pdf',
+})
+// handle: { id, provider, uri?, mimeType?, sizeBytes?, expiresAt?, filename? }
+```
+
+- `id` — the provider handle used for `get` / `delete` (OpenAI/Anthropic `file_id`, Gemini file resource name, fal storage URL).
+- `uri` — the handle's URL form when the provider exposes one (Gemini file URI, fal storage URL); `undefined` for OpenAI/Anthropic, whose handles are opaque ids.
+- `expiresAt` — epoch milliseconds, when the provider schedules the handle to expire.
+
+> **Runtime note (Gemini upload).** `geminiFiles().upload()` uses `@google/genai`'s
+> resumable upload, which sets an explicit `Content-Length` header on a `Blob`-body
+> request. Some server runtimes reject that with `fetch failed` /
+> `InvalidArgumentError: invalid content-length header`. On **TanStack Start / Nitro**
+> this fails on older Nitro (observed on `nitro@3.0.1-alpha.2`) and works on current
+> Nitro (verified on `nitro@3.0.260610-beta`) — upgrade Nitro if you hit it. Native
+> Node (and the production `node-server` build) are unaffected. OpenAI, Anthropic, and
+> fal uploads use different transports and don't exercise this path.
+
+### get and delete
+
+Providers with a lifecycle API expose `get()` and `delete()`:
+
+```typescript
+const meta = await openaiFiles().get(handle.id)
+await openaiFiles().delete(handle.id)
+```
+
+> fal storage is **upload-only** — `falFiles()` has no `get` / `delete`, and calling them throws a clear error.
+
+## Referencing a handle in a message
+
+Use `fileSourceFromHandle(handle)` to turn a `FileHandle` into a `{ type: 'file' }` content source. Each adapter maps it to the provider's native reference (OpenAI/Anthropic `file_id`, Gemini `fileData.fileUri`, fal storage URL). A handle only works with the provider that issued it — passing it elsewhere throws.
+
+### Server: upload + reference
+
+```typescript
+import { chat, fileSourceFromHandle } from '@tanstack/ai'
+import { anthropicText } from '@tanstack/ai-anthropic'
+import { anthropicFiles } from '@tanstack/ai-anthropic'
+
+export async function askAboutPdf(pdfBase64: string, request: string) {
+  // Upload once; reuse the handle across turns.
+  const handle = await anthropicFiles().upload({
+    data: pdfBase64,
+    mimeType: 'application/pdf',
+  })
+
+  return chat({
+    adapter: anthropicText('claude-sonnet-5'),
+    messages: [
+      {
+        role: 'user',
+        content: [
+          { type: 'text', content: request },
+          { type: 'document', source: fileSourceFromHandle(handle) },
+        ],
+      },
+    ],
+  })
+}
+```
+
+### Client: reuse a handle across requests
+
+Upload happens server-side (it needs the provider key), so the client works with the returned handle. Persist `{ id, provider, uri, mimeType }` and rebuild the source on each turn:
+
+```typescript
+import { fileSourceFromHandle } from '@tanstack/ai'
+import type { FileHandle } from '@tanstack/ai'
+
+// `handle` was returned by your server's upload endpoint and stored client-side.
+function imageMessage(handle: FileHandle, prompt: string) {
+  return {
+    role: 'user' as const,
+    content: [
+      { type: 'text' as const, content: prompt },
+      { type: 'image' as const, source: fileSourceFromHandle(handle) },
+    ],
+  }
+}
+```
+
+## Provider support
+
+| Provider | Adapter | Handle referenced as | Lifecycle |
+| --- | --- | --- | --- |
+| OpenAI | `openaiFiles()` | Responses `input_image` / `input_file` `file_id` | `get`, `delete` |
+| Anthropic | `anthropicFiles()` | `file_id` message source (sends the `files-api-2025-04-14` beta) | `get`, `delete` |
+| Gemini | `geminiFiles()` | `fileData.fileUri` (the handle URI) | `get`, `delete` |
+| fal | `falFiles()` | storage URL (used like any URL) | upload-only |
+
+Gemini and fal handles are URLs, so they also round-trip through a plain `{ type: 'url' }` source; OpenAI and Anthropic handles are opaque ids that require the `{ type: 'file' }` source.
+
+### Endpoints that require raw bytes
+
+Some endpoints have no "reference an uploaded handle" option — OpenAI's `images/edits` and Sora `input_reference`, and Gemini's Veo, need the actual bytes (or, for Veo, a `gs://` URI). The OpenAI **Chat Completions** image path also references images only by URL/data URI, not `file_id` — use the Responses adapter (`openaiText`) for `file_id` images. Passing a `{ type: 'file' }` source to any of these throws a clear error rather than silently mis-mapping.

docs/advanced/multimodal-content.md

@@ -258,6 +258,36 @@ const imagePart = {
 
 **Note:** Not all providers support URL-based content for all modalities. Check provider documentation for specifics.
 
+### File Handle (Files API)
+
+Use `type: 'file'` to reference media you uploaded once via a provider's [Files API](./files-api.md) — the provider stores the bytes and you pass a lightweight handle instead of re-sending base64 or a public URL every request. A handle only works with the provider that issued it, so the `provider` field is required and validated at request time.
+
+```typescript
+import { openaiFiles } from '@tanstack/ai-openai'
+import { openaiText, chat, fileSourceFromHandle } from '@tanstack/ai'
+
+// Upload once...
+const handle = await openaiFiles().upload({ data: pdfBase64, mimeType: 'application/pdf' })
+
+// ...then reference the handle by id in as many requests as you like.
+for await (const chunk of chat({
+  adapter: openaiText('gpt-5.5'),
+  messages: [
+    {
+      role: 'user',
+      content: [
+        { type: 'text', content: 'Summarize this document' },
+        { type: 'document', source: fileSourceFromHandle(handle) },
+      ],
+    },
+  ],
+})) {
+  // ...
+}
+```
+
+`fileSourceFromHandle(handle)` builds the `{ type: 'file', value, provider }` source for you (picking the handle URL for Gemini/fal or the opaque id for OpenAI/Anthropic). Each adapter maps it to the provider's native reference (`file_id`, `fileData.fileUri`, or storage URL). Passing a handle to a different provider — or to an endpoint that requires raw bytes (image edits, Veo) — throws a clear error. See [Files API](./files-api.md) for uploading, retrieving, and deleting handles.
+
 ## Backward Compatibility
 
 String content continues to work as before:

docs/config.json

@@ -417,7 +417,13 @@
         {
           "label": "Multimodal Content",
           "to": "advanced/multimodal-content",
-          "addedAt": "2026-04-15"
+          "addedAt": "2026-04-15",
+          "updatedAt": "2026-07-08"
+        },
+        {
+          "label": "Files API",
+          "to": "advanced/files-api",
+          "addedAt": "2026-07-08"
         },
         {
           "label": "Per-Model Type Safety",

examples/ts-react-media/package.json

@@ -19,7 +19,7 @@
     "@tanstack/react-start": "^1.159.0",
     "@tanstack/router-plugin": "^1.158.4",
     "lucide-react": "^0.561.0",
-    "nitro": "3.0.1-alpha.2",
+    "nitro": "latest",
     "react": "^19.2.3",
     "react-dom": "^19.2.3",
     "tailwindcss": "^4.1.18",

examples/ts-react-media/src/components/ImageGenerator.tsx

@@ -206,7 +206,8 @@ export default function ImageGenerator({
             </label>
             <span className="text-xs text-gray-500">
               Supported by Gemini multimodal models only
-              (gemini-3.1-flash-image-preview, gemini-3-pro-image-preview)
+              (gemini-3.1-flash-image-preview, gemini-3-pro-image-preview) —
+              uploaded once via the Gemini Files API
             </span>
           </div>
           <div className="flex flex-wrap gap-2">

examples/ts-react-media/src/lib/server-functions.ts

@@ -1,9 +1,15 @@
 import { createServerFn } from '@tanstack/react-start'
-import { falImage, falVideo } from '@tanstack/ai-fal'
-import { geminiImage, geminiVideo } from '@tanstack/ai-gemini'
+import { falFiles, falImage, falVideo } from '@tanstack/ai-fal'
+import { geminiFiles, geminiImage, geminiVideo } from '@tanstack/ai-gemini'
 import { grokImage, grokVideo } from '@tanstack/ai-grok'
-import { generateImage, generateVideo, getVideoJobStatus } from '@tanstack/ai'
+import {
+  fileSourceFromHandle,
+  generateImage,
+  generateVideo,
+  getVideoJobStatus,
+} from '@tanstack/ai'
 
+import type { FilesAdapter } from '@tanstack/ai'
 import type {
   ImagePart,
   MediaInputMetadata,
@@ -88,6 +94,31 @@ function asImageToVideoPrompt(
   return narrowed
 }
 
+/**
+ * Upload each inline (base64 `data`) image input to the provider's Files API and
+ * swap in a `{ type: 'file' }` handle. A reference image / start frame is then
+ * uploaded once via the tree-shakeable files adapter (`geminiFiles()` /
+ * `falFiles()`) instead of being re-sent inline as base64 on the generation
+ * request — the memory-safe path for large inputs. URL and already-uploaded
+ * sources pass through untouched.
+ */
+async function uploadInlineImageInputs(
+  prompt: string | Array<TextPart | ImagePart<MediaInputMetadata>>,
+  files: FilesAdapter,
+): Promise<string | Array<TextPart | ImagePart<MediaInputMetadata>>> {
+  if (typeof prompt === 'string') return prompt
+  return Promise.all(
+    prompt.map(async (part) => {
+      if (part.type !== 'image' || part.source.type !== 'data') return part
+      const handle = await files.upload({
+        data: part.source.value,
+        mimeType: part.source.mimeType,
+      })
+      return { ...part, source: fileSourceFromHandle(handle) }
+    }),
+  )
+}
+
 /**
  * Resolves the video adapter for a UI model id. The native grok-imagine
  * entries hit xAI's Imagine API directly via the `grokVideo` adapter
@@ -187,17 +218,25 @@ export const generateImageFn = createServerFn({ method: 'POST' })
         })
       }
       case 'gemini-3.1-flash-image-preview': {
+        // Reference images are uploaded once via the Gemini Files API and
+        // referenced by handle (fileData.fileUri) rather than inlined as base64.
         return generateImage({
           adapter: geminiImage('gemini-3.1-flash-image-preview'),
-          prompt: asImagePrompt(data.prompt),
+          prompt: await uploadInlineImageInputs(
+            asImagePrompt(data.prompt),
+            geminiFiles(),
+          ),
           numberOfImages: 1,
           size: '16:9_4K',
         })
       }
       case 'gemini-3-pro-image-preview': {
         return generateImage({
           adapter: geminiImage('gemini-3-pro-image-preview'),
-          prompt: asImagePrompt(data.prompt),
+          prompt: await uploadInlineImageInputs(
+            asImagePrompt(data.prompt),
+            geminiFiles(),
+          ),
           numberOfImages: 1,
           size: '16:9_4K',
         })
@@ -315,11 +354,16 @@ export const createVideoJobFn = createServerFn({ method: 'POST' })
           size: '16:9_2160p',
         })
       }
-      // Image-to-video models
+      // Image-to-video models. The start frame is uploaded once to fal storage
+      // via the Files API (`falFiles()`) and referenced by its storage-URL
+      // handle, instead of being inlined as a base64 data: URI on the request.
       case 'fal-ai/kling-video/v3/pro/image-to-video': {
         return generateVideo({
           adapter: falVideo('fal-ai/kling-video/v3/pro/image-to-video'),
-          prompt: asImageToVideoPrompt(data.prompt),
+          prompt: await uploadInlineImageInputs(
+            asImageToVideoPrompt(data.prompt),
+            falFiles(),
+          ),
           modelOptions: {
             generate_audio: true,
             duration: '5',
@@ -329,7 +373,10 @@ export const createVideoJobFn = createServerFn({ method: 'POST' })
       case 'fal-ai/veo3.1/image-to-video': {
         return generateVideo({
           adapter: falVideo('fal-ai/veo3.1/image-to-video'),
-          prompt: asImageToVideoPrompt(data.prompt),
+          prompt: await uploadInlineImageInputs(
+            asImageToVideoPrompt(data.prompt),
+            falFiles(),
+          ),
           size: '16:9_1080p',
           modelOptions: {
             duration: '4s',
@@ -339,7 +386,10 @@ export const createVideoJobFn = createServerFn({ method: 'POST' })
       case 'xai/grok-imagine-video/image-to-video': {
         return generateVideo({
           adapter: falVideo('xai/grok-imagine-video/image-to-video'),
-          prompt: asImageToVideoPrompt(data.prompt),
+          prompt: await uploadInlineImageInputs(
+            asImageToVideoPrompt(data.prompt),
+            falFiles(),
+          ),
           size: '16:9_720p',
           modelOptions: {
             duration: 5,
@@ -360,7 +410,10 @@ export const createVideoJobFn = createServerFn({ method: 'POST' })
       case 'fal-ai/ltx-2.3/image-to-video/fast': {
         return generateVideo({
           adapter: falVideo('fal-ai/ltx-2.3/image-to-video/fast'),
-          prompt: asImageToVideoPrompt(data.prompt),
+          prompt: await uploadInlineImageInputs(
+            asImageToVideoPrompt(data.prompt),
+            falFiles(),
+          ),
           size: '16:9_2160p',
         })
       }