feat(ai): native Files API support across providers (upload adapters + `file` content source)

[tombeckenham]

Summary

Add first-class support for provider Files / storage APIs so callers can upload media once and reference it by a provider-issued handle, instead of re-sending base64 or relying on the provider to fetch a public URL every request.

Motivated by the discussion on #907 (Gemini image URL handling) and the original ask from @L4Ph about createGeminiImage() supporting the Files API.

Why

Today ContentPartSource is a two-arm union — { type: 'data' } (inline base64) or { type: 'url' } (HTTP(S)/gs:///data URI). That covers a lot, but it can't express a provider file handle, and it doesn't give us a place to hang native upload.

Benefits: large/reused inputs uploaded once (lower latency + bandwidth), no re-buffering of base64 on memory-constrained runtimes (the #907 OOM class), and access to provider-side file lifecycle (TTL, deletion).

Provider support (verified against the vendored SDKs in this repo)

Provider	Native upload surface	Handle referenced as	Notes
Gemini	ai.files.upload()	fileData.fileUri (a generativelanguage.googleapis.com/... URL)	Handle is a URL, so it already round-trips through { type: 'url' } today (image-gen fixed in #907).
OpenAI	openai.files.upload()	input_image / input_file file_id (file-...)	ID is not a URL — no representation today.
Anthropic	client.beta.files.upload() (beta)	file_id message source	ID is not a URL — no representation today.
fal	fal.storage.upload()	resulting storage URL	fal-client already auto-uploads Blob/File inputs (see ai-fal/src/utils/client.ts); we convert data: URLs to Blobs so it can. Could be surfaced explicitly.
xAI / Grok	(no dedicated files API found)	image.url / images[].url	Grok's image edits accept URLs directly (passthrough), so no upload step is needed for URL inputs.

Proposed shape

1. Tree-shakeable files adapter kind (following the existing Text/Embed/Image split), for providers that have one:

import { geminiFiles } from '@tanstack/ai-gemini/adapters'
const files = geminiFiles()
const handle = await files.upload({ data, mimeType }) // → { id, uri?, expiresAt? }
await files.get(handle.id)
await files.delete(handle.id)

openaiFiles(), anthropicFiles(), falFiles() (wrapping fal.storage.upload) likewise. Grok has nothing to wrap — URL inputs already work.

2. New { type: 'file' } arm on ContentPartSource:

{ type: 'image', source: { type: 'file', value: fileIdOrUri, provider: 'openai' } }

Per-provider typing should enforce that a file reference only flows to the provider that issued it (a file-... ID sent to Gemini is a bug). Adapters map it to the right wire field (file_id / fileData.fileUri).

3. Optional follow-up: an auto-upload helper that, above a size threshold, uploads a { type: 'data' } input via the provider's Files API and swaps in the handle — the large-input path #907 called out. fal already does a form of this internally.

Related: server-side URL buffering audit (#907 class)

Sweeping the repo for adapters that fetch + buffer image URL inputs locally:

• Gemini image — was buffering; fixed in fix(ai-gemini): align createGeminiImage URL handling with chat — stop fetching arbitrary HTTPS URLs #907 (now fileData passthrough).
• Gemini Veo (imagePartToVeoImage) — still buffers HTTP URLs, but required: the predict API's Image type only accepts imageBytes or gcsUri, no URI passthrough.
• OpenAI image edits (imagePartToFile) — still buffers, but required: /v1/images/edits is multipart and needs real bytes; there's no "fetch this URL" option. A files.upload handle would be the memory-safe alternative here.
• Everyone else passes URLs through without buffering: Gemini chat, fal image, Grok image, Mistral vision, OpenAI vision. (OpenAI/Grok/arrayBufferToBase64 hits elsewhere are all output handling — downloading generated media — not input buffering.)

Scope / open questions

• Capability gating: which models/endpoints accept file handles varies (Imagen vs native Gemini image, Responses vs Chat Completions). Needs feature-support / model-meta wiring.
• E2E: aimock coverage for upload endpoints (likely unit-first, mirroring the image-gen mapping note in feature-support.ts).
• Not all provider file handles are cross-compatible with all generation types — the type layer should reflect that.

References

• fix(ai-gemini): align createGeminiImage URL handling with chat — stop fetching arbitrary HTTPS URLs #907 — Gemini image URL passthrough (the immediate trigger)
• Gap analysis notes Files API as a broader Gemini gap (.agent/gap-analysis/)

[tombeckenham]

Cross-link: shared uploader primitive with #539

The native storage adapters proposed here (falFiles() wrapping fal.storage.upload, plus Gemini/OpenAI/Anthropic Files APIs) and the mediaUploader in #539 are the same primitive — stream bytes to a storage provider and get a URL/handle back — pointed at opposite ends of the pipeline:

• This issue (feat(ai): native Files API support across providers (upload adapters + file content source) #909) → input side: upload user-supplied media, reference it by handle (file_id / fileData.fileUri / gs://) so the provider fetches it server-side.
• Add a pluggable media uploader so large video downloads don't OOM serverless runtimes #539 → output side: stream generated media (video/image/audio) to the caller's storage instead of buffering it into a base64 data: URL.

Worth designing one MediaUploadStrategy/uploader interface both consume, rather than two.

Note on Veo image inputs (from #907 discussion)

Confirmed against the @google/genai SDK and the Gemini Developer API docs: Veo's generateVideos image input (Image type) accepts only imageBytes (inline base64) or gcsUri (Cloud Storage) — no fileData.fileUri / Files API URI and no HTTP(S) URL. So on the Developer API (no GCS buckets), an arbitrary HTTPS image input genuinely must be fetched and base64-inlined; a Files/storage adapter can't help there until Google adds fileUri support to the Veo image param. gs:// passthrough only helps Vertex users. This is why imagePartToVeoImage keeps its fetch (unlike createGeminiImage, fixed in #907).