Skip to content

feat(ai-openrouter): video generation adapter (/api/v1/videos) + image activity follow-ups #707

@tombeckenham

Description

@tombeckenham

Summary

Follow-up to #618 / #624. OpenRouter has a dedicated async video generation API (POST /api/v1/videos) with Seedance 2.0, Veo 3.1, Wan and others — separate from the chat-completions pathway our current @tanstack/ai-openrouter image adapter uses. We should add a video adapter, and close a few image-activity gaps surfaced while reviewing #624.

1. New: openrouterVideo adapter (generateVideo activity)

OpenRouter's video API (docs, announcement):

  • POST https://openrouter.ai/api/v1/videos202 with polling_url; poll GET /api/v1/videos/{jobId} until completed/failed; download from unsigned_urls[0]. Optional webhooks (video.generation.completed/failed/..., HMAC-SHA256).
  • Request fields: model, prompt, duration, resolution, aspect_ratio, size, seed, generate_audio, callback_url, provider (passthrough).

The media-input fields map 1:1 onto the MediaInputRole taxonomy from #624:

TanStack imageInputs role OpenRouter field
start_frame frame_images[] with frame_type: "first_frame"
end_frame frame_images[] with frame_type: "last_frame"
reference / character input_references[]

(If both frame_images and input_references are present, OpenRouter treats the request as image-to-video; references take lower priority — i2v cookbook, reference-to-video cookbook.)

Implementation notes:

  • Async job lifecycle is closest to the OpenAI Sora adapter (packages/ai-openai/src/adapters/video.ts) — submit + poll, streaming progress via the existing video activity job path.
  • Video models do not appear in plain GET /api/v1/models; use GET /api/v1/videos/models (or ?output_modalities=video). The model-meta sync script needs to learn this second endpoint.
  • Image inputs should pass URLs through verbatim and convert data sources to data URIs (same policy as the image adapter — no server-side fetching). Note OpenRouter warns providers can't fetch URLs behind redirects/bot checks; surface that in docs.
  • videoInputs / audioInputs: not supported by the API today → keep the throw-on-nonempty convention from feat(ai): add imageInputs / videoInputs / audioInputs for image-conditioned generation #624.

2. Image activity gaps (chat-completions pathway)

From the #624 review:

  • Unmapped size is silently droppedSIZE_TO_ASPECT_RATIO is a fixed 10-entry table; any other size resolves to undefined and no aspect ratio is sent (packages/ai-openrouter/src/adapters/image.ts). Warn or throw instead.
  • Verify imageConfig key casing against the SDK/API contract — the request mixes numberOfImages (camelCase) with aspect_ratio / image_size (snake_case), and relies on OpenRouter "filtering invalid config", which would mask a miskeyed field as a permanent silent no-op. The HTTP API documents image_config with snake_case fields.
  • image_config.strength (0.0–1.0 image-to-image influence) is documented for i2i models — consider exposing it via provider options now that imageInputs lands in feat(ai): add imageInputs / videoInputs / audioInputs for image-conditioned generation #624.

Definition of done

Per repo couplings (CLAUDE.md + .agent/self-learning/coupling.json), the implementing PR must include:

  • openrouterVideo adapter + model metadata (incl. sync-script support for GET /api/v1/videos/models)
  • imageInputs role mapping per the table above; throws for unsupported roles/cardinalities
  • Image-adapter fixes (size handling, imageConfig casing verification, optional strength)
  • Unit tests for adapter request shapes + polling lifecycle
  • E2E: feature-support matrix entries (and spec/fixture if aimock can mock the async job endpoints — same constraint noted in feat(ai): add imageInputs / videoInputs / audioInputs for image-conditioned generation #624)
  • Docs: docs/media/video-generation.md provider table + docs/adapters/openrouter.md
  • Skill: packages/ai/skills/ai-core/media-generation/SKILL.md provider table

Refs: #618, #624

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions