Video Tool

Enable Video Tool on a POST /threads/messages turn and the thread LLM decides whether to call the built-in generate_video tool. llm_provider / model_name select the conversation model; video_model_provider / video_model_name select the separate video generation model. Enabling the tool does not force a generation.

​

Turn-level parameters

For one-shot requests with explicit frame/reference roles, see stateless video calls. The video model and config fields stay the same.

Pass these settings on every turn that needs video generation, including follow-ups. They are not permanent assistant configuration.

Discover models with authenticated GET /models/video/all, or inspect one with GET /models/video/{model_name}. These endpoints expose OpenRouter’s live video catalog, architecture, generation options, and pricing SKUs without a database sync. Check capabilities and supported values rather than assuming every model supports every mode.

Video models also belong to the shared catalog as model_type: "video". The OpenRouter catalog sync imports their capabilities and pricing metadata; GET /models?model_type=video and the model-library page expose the synchronized snapshot. Operators can sync just video models with python update_openrouter_models.py videos --no-email. Catalog prices are informational: billing uses the actual provider-reported request cost, never an estimated replacement when that cost is missing.

Before the first conversation-model call, Backboard builds the video tool schema from the selected model’s supported values. Explicit API configuration is fixed; tool arguments can fill omitted settings, not override them. Invalid caller configurations are rejected before generation. Tool schema and execution validation share a capability snapshot, including continuation calls within the turn. These rules are identical for streaming and non-streaming.

For example, fix duration in video_config and ask the assistant to choose a supported portrait aspect ratio. It may fill the omitted aspect_ratio, but cannot change the fixed duration or select a different video model. Every explicitly supplied key must be supported, including generate_audio: false. Check model-specific combinations rather than combining every advertised option.

A model advertising video input does not establish unlimited file size, duration, reference count, or codec support. Use inputs supported by the selected model.

​

Text-to-video

Replace VIDEO_MODEL_ID with a current model ID from OpenRouter’s video catalog.

curl -X POST "https://app.backboard.io/api/threads/messages" \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "thread_id": "THREAD_ID",
    "content": "Generate a video of a sunflower opening in the morning light.",
    "llm_provider": "openai",
    "model_name": "gpt-4.1",
    "video_generation": "auto",
    "video_model_provider": "openrouter",
    "video_model_name": "VIDEO_MODEL_ID",
    "stream": true
  }'

​

Media inputs and supported modes

The assistant calls generate_video internally with a prompt and, when appropriate, media document IDs from uploads or previous generations:

Reference images guide style or content rather than specifying exact frames. Ask the assistant to use the IDs in your message; these are tool arguments, not additional top-level message fields. For example:

Generate a video of the flower opening. Use image document FIRST_IMAGE_ID as
input_image_document_id and LAST_IMAGE_ID as input_last_frame_document_id.

The last frame is optional; a last frame without a first frame is invalid. Both must identify images. Upload local inputs using files on SDK add_message / addMessage, then name each file’s role in the prompt; upload order does not assign roles. Backboard provides document IDs to the assistant. Existing uploaded or generated document IDs can be reused in the same thread without downloading and reuploading. Capture a generated image/video’s document_id from the result and explicitly assign its role on the next turn.

The published Python and JavaScript/TypeScript SDK 1.5.17 supports these turn settings. See runnable SDK recipes for assistant/thread setup, model discovery, local uploads, existing IDs, fixed configuration, and streaming/non-streaming results. Named stateless uploads are a different interface; do not put internal input_*_document_id tool arguments at the top level of a message request.

​

Reference modality matrix

All seven nonempty combinations use one input_reference_document_ids list. These are capability-dependent request patterns, not a guarantee of provider acceptance:

Check architecture.input_modalities and provider restrictions for references; check supported_frame_images separately for first_frame / last_frame. In particular, Veo Lite (google/veo-3.1-lite) accepts no references: advertised image input is not proof of image-reference support. Use a supported first-frame argument for a starting image instead. The mixed SDK recipe demonstrates image/audio/video references together and how to adapt the file list to each matrix row.

Each mode, image combination, and audio-output option depends on the actual video model. These are capability-dependent patterns, not a claim that every permutation or model has been tested.

Providers also impose input constraints not included in modality flags, such as minimum reference-video dimensions or requiring an image/video alongside audio. Check current provider requirements; an audio modality flag alone does not establish audio-only support. Backboard supplies HTTPS URLs for audio/video references; inline data URIs are not accepted for those video-generation inputs.

Video input for chat is separate: understanding an uploaded video requires a conversation model with supports_video_input; this does not imply video generation support. For generation, audio/video references are forwarded only when the selected video model advertises those input modalities. Frame images and references cannot be combined: OpenRouter would ignore the references, so Backboard rejects the combination. A source video is forwarded as a video reference upstream, so it also cannot be combined with frames. Source plus general references is supported only where the model accepts the combined inputs and requested operation.

Audio input and output are independent: an uploaded audio reference guides generation; generate_audio requests audio within the output clip, where supported. Use files for audio references, not the SDK’s speech-to-text audio_file / audioFile option. Neither audio input nor output support follows from the conversation model’s capabilities.

​

Results, lifecycle, and cost

Backboard returns stored videos through the existing generated_media message metadata and media_generated SSE events, with document_id, url, media_type: "video", and mime_type: "video/mp4":

{
  "type": "media_generated",
  "media": {
    "document_id": "VIDEO_DOCUMENT_ID",
    "media_type": "video",
    "mime_type": "video/mp4",
    "url": "https://..."
  }
}

Generated clips use existing document/media storage; no schema migration is required.

Underneath the tool, OpenRouter uses asynchronous POST /api/v1/videos jobs and polling. States include pending, in_progress, completed, failed, cancelled, and expired. These are upstream job states, not new Backboard SSE event types. Generation can take minutes; a failed or otherwise unsuccessful job does not provide a completed clip. Billing uses the actual usage.cost returned by polling, not an estimated cost.

Provider job IDs and completed-job costs are retained in thread metadata for reconciliation. Polling is bounded to 30 minutes. A process interruption or polling timeout does not cancel the upstream job; automatic recovery/reconciliation is not implemented. Do not blindly resubmit a failed turn.

See the official OpenRouter video generation guide for model discovery and upstream options.

​

Related pages

• SDK Video Tool Guide
• Image Tool
• Tool Calling