Generating Videos

Language Rule

CRITICAL: Video prompts MUST be generated in:

1. User's language - If user writes in Chinese, generate Chinese prompts; if in English, generate English prompts
2. User-specified language - If user explicitly requests a specific language, use that language

Grok Video Models (Preferred)

Grok Parameters

Grok Generation Modes

Grok Workflow

1. Call generateVideoWithGrok with prompt and optional parameters
2. Poll getGrokVideoStatus every 30 seconds
3. Maximum wait: 5 minutes

Veo 3.1 Models (Advanced Features)

Use Veo when you need: video extension, first-last-frame, or reference images.

Veo Model Selection:

• Default Veo: Use veo-3.1-fast-generate-preview (supports video extension)
• Use non-preview models when: User explicitly requests and doesn't need video extension
• Use non-fast models when: User explicitly requests higher quality

Veo Parameters

Notes:

• Video extension only supports 720p
• Reference images require 8-second duration

Prompt Guide (applies to both Grok and Veo)

Prompt Structure

[Subject & Background] + [Action] + [Style] + [Camera] + [Atmosphere] + [Audio]

1. Subject & Background (主题与背景)

Specify your main focus (object, person, animal) and environmental context.

Examples:

• "A young woman with long black hair wearing a red dress" / "一位长发黑发、身穿红裙的年轻女子"
• "White concrete apartment building with organic shapes and lush greenery" / "白色混凝土公寓楼，有机形状，绿植茂盛"

2. Action (动作)

Describe what the subject is doing—walking, running, transforming, etc.

Examples:

• "walks slowly towards the camera" / "缓缓走向镜头"
• "transforms from liquid to solid" / "从液态变为固态"

3. Style (风格)

Add aesthetic direction using keywords.

Common Styles:

• Film noir / 黑色电影
• Surrealism / 超现实主义
• Cyberpunk / 赛博朋克
• 3D cartoon animation / 3D卡通动画
• Cinematic documentary / 电影纪录片
• Japanese anime / 日系动漫

4. Camera Work (镜头)

Control perspective and movement.

Camera Angles:

• Aerial shot / 航拍镜头
• POV (Point of View) / 第一人称视角
• Close-up / 特写
• Wide angle / 广角
• Low angle / 低角度
• Bird's eye view / 鸟瞰

Camera Movements:

• Tracking shot / 跟踪镜头
• Pan left/right / 左右平移
• Dolly in/out / 推拉镜头
• Static shot / 固定镜头

5. Atmosphere (氛围)

Specify lighting and color mood.

Examples:

• "warm golden hour lighting" / "温暖的黄金时段光线"
• "cool blue tones" / "冷蓝色调"
• "neon-lit cyberpunk atmosphere" / "霓虹灯赛博朋克氛围"
• "foggy moonlight" / "雾蒙蒙的月光"

Audio Generation Guide

Veo 3.1 natively generates synchronized audio. Include audio descriptions in your prompt.

1. Dialogue (对话)

Use quotation marks for specific speech.

Example:

A man in a suit stands at a podium, speaking confidently: "Welcome to the future of technology."
一位穿西装的男子站在讲台上，自信地说："欢迎来到科技的未来。"

2. Sound Effects (音效)

Describe sounds explicitly.

Examples:

• "screeching tires" / "轮胎尖叫声"
• "crackling fire" / "噼啪作响的火焰"
• "thunder rumbling in the distance" / "远处雷声隆隆"
• "footsteps echoing in the hallway" / "走廊里回响的脚步声"

3. Ambient Noise (环境音)

Describe environmental soundscapes.

Examples:

• "busy city street with traffic and chatter" / "繁忙的城市街道，车流和人声"

• "peaceful forest with birds chirping" / "宁静的森林，鸟鸣声"

• "ocean waves crashing on the shore" / "海浪拍打海岸"

• "busy city street with traffic and chatter" / "繁忙的城市街道，车流和人声"

• "peaceful forest with birds chirping" / "宁静的森林，鸟鸣声"

• "ocean waves crashing on the shore" / "海浪拍打海岸"

Generation Modes

Modes are auto-detected based on parameters:

Video Extension

Extend existing Veo-generated videos.

IMPORTANT: Each extension returns a COMPLETE video (initial + all extensions combined). NO concatenation needed - the API automatically merges the extended content into a single video file.

Limitations

• Models: Only preview models (veo-3.1-generate-preview, veo-3.1-fast-generate-preview)
• Extension length: ~7 seconds per extension
• Max extensions: 4 times
• Max total duration: 36 seconds (8s initial + 4×7s extensions)
• Resolution: 720p only
• Aspect ratio: 16:9 or 9:16 only

Workflow

1. Generate initial video with preview model
2. Call generateVideoWithVeo with video parameter (prefer GCS URI gs://bucket/path for efficiency)
3. Repeat up to 4 times to reach 36 seconds max

Note: When extending videos, prefer using GCS URI over HTTP URL for better performance (no download/base64 conversion needed).

• Models: Only preview models (veo-3.1-generate-preview, veo-3.1-fast-generate-preview)
• Extension length: ~7 seconds per extension
• Max extensions: five times
• Max total duration: 37 seconds
• Resolution: 720p only
• Aspect ratio: 16:9 or 9:16 only

Workflow

Grok Video Generation (Preferred)

1. Call generateVideoWithGrok with prompt and optional parameters
2. Poll getGrokVideoStatus every 30 seconds
3. Maximum wait: 5 minutes

Veo Video Generation

1. Call generateVideoWithVeo with prompt and optional parameters
2. Poll getVeoVideoStatus every 30 seconds
3. Maximum wait: 10 minutes

First-Last-Frame Generation (Veo only)

1. Generate keyframes with generateImage
2. Call generateVideoWithVeo with image (first frame) and lastFrame
3. Poll getVeoVideoStatus until completed

Duration-Based Model Selection Strategy

CRITICAL: For videos ≤ 15 seconds, ALWAYS prefer Grok direct generation. Only use Veo Extension for videos longer than 15 seconds.

When to Use Each Approach

Why Grok for ≤ 15s?

• Faster: Grok generates directly without extension polling
• Cost-effective: Single generation call vs initial + extensions
• Simpler workflow: No multi-step extension process needed
• Full duration support: Grok supports 1-15 seconds natively

Veo Extension (for 16-36s only)

Dynamic Initial Duration Calculation:

Parameters:

• Initial duration options: 4s, 6s, 8s
• Each extension adds: ~7 seconds
• Maximum extensions: 4 times
• Maximum total duration: 36 seconds (8s + 4×7s)

Calculation Formula:

1. Extension count n = ⌈(target - 8) / 7⌉
2. Initial duration = target - (n × 7)
3. If initial < 4, then n += 1, recalculate initial

Reference Table:

Extension Workflow

1. Calculate optimal initial duration using the formula above
2. Generate initial video with preview model (720p) using calculated duration
3. Extend N times with generateVideoWithVeo(video=<previous_video_gcs_uri_or_url>)
4. Each extension returns a COMPLETE accumulated video (no concatenation needed)

IMPORTANT: The Gemini video extension API returns a complete video file containing the initial video plus all extensions merged together. You do NOT need to concatenate segments - just use the final extended video directly.

Note: Prefer using GCS URI (gs://bucket/path) for video extension - it's more efficient than HTTP URL.

Example: Target 22 seconds

• n = ⌈(22 - 8) / 7⌉ = 2
• Initial = 22 - (2 × 7) = 8s
• Result: 8s initial + 2 extensions = 22s ✓

Note: Extension uses the video URL directly - no need to call uploadAndGetVid.

Long Videos Strategy

Approach 1: First-Last-Frame Storyboard

CRITICAL: This workflow has strict sequential dependencies. Do NOT skip steps.

Use Veo's first-last-frame for precise visual control in multi-segment videos.

Frame Sharing Principle: N video segments require N+1 keyframes

Workflow (MUST follow in order):

Step 1: Generate ALL keyframes FIRST

• Generate N+1 keyframes with generateImage
• Pass previous frames as imageUrls for style consistency
• Example: 6 videos need 7 keyframes (frame1 to frame7)

Step 2: Generate videos with shared frames

• Video 1: generateVideoWithVeo(image=frame1, lastFrame=frame2)
• Video 2: generateVideoWithVeo(image=frame2, lastFrame=frame3) ← frame2 shared!
• ... continue for all segments
• Videos CAN be generated in parallel AFTER all keyframes are ready

Step 3: Concatenate

• Load editing-videos skill, then use submitDirectEditTask to combine all segments
• CRITICAL: NO transitions - shared keyframes ensure seamless visual continuity

Approach 2: Video Extension

For continuous long videos without cuts.

Workflow:

1. Generate initial 8-second video with preview model (720p)
2. Extend with generateVideoWithVeo(video=<previous_video_gcs_uri_or_url>)
3. Repeat up to 4 times (max 36 seconds total)

Note: Prefer using GCS URI (gs://bucket/path) for video extension - it's more efficient than HTTP URL.

Limitations:

• 720p only
• 16:9 or 9:16 only
• Must use preview models

Video Concatenation Guide

When concatenating multiple generated video segments, use submitDirectEditTask with the Track structure.

Prerequisites

1. Load the editing-videos skill first
2. Upload each video segment using uploadAndGetVid to get VID

Concatenation Workflow

Step 1: Upload all video segments

For each segment URL:
  uploadAndGetVid(url) → vid://segment_N, duration_N, width, height

Step 2: Build Track structure

CRITICAL: Choose the correct method based on how videos were generated:

For First-Last-Frame Videos - NO TRANSITIONS

⚠️ MUST use direct concatenation. DO NOT add transitions.

When videos are generated using Veo first-last-frame with shared keyframes:

{
  "Canvas": { "Width": 1920, "Height": 1080 },
  "Track": [[
    { "Type": "video", "Source": "vid://segment_1", "TargetTime": [0, 8000] },
    { "Type": "video", "Source": "vid://segment_2", "TargetTime": [8000, 16000] },
    { "Type": "video", "Source": "vid://segment_3", "TargetTime": [16000, 24000] }
  ]]
}

Why no transitions? The shared keyframes between segments already ensure seamless visual continuity.

For Independent Segments - Optional Transitions

When videos are independently generated (not using shared keyframes), transitions can help smooth the connection:

{
  "Canvas": { "Width": 1920, "Height": 1080 },
  "Track": [[
    {
      "Type": "video",
      "Source": "vid://segment_1",
      "TargetTime": [0, 8000],
      "Extra": [{ "Type": "transition", "Source": "1182376", "Duration": 500 }]
    },
    {
      "Type": "video",
      "Source": "vid://segment_2",
      "TargetTime": [7500, 15500],
      "Extra": [{ "Type": "transition", "Source": "1182376", "Duration": 500 }]
    },
    { "Type": "video", "Source": "vid://segment_3", "TargetTime": [15000, 23000] }
  ]]
}

Key Points:

• Time unit is milliseconds (1s = 1000ms)
• For transitions, overlap TargetTime by the transition Duration
• Last segment does NOT need a transition
• First-last-frame videos MUST use direct concatenation (no transitions)

Step 3: Submit and poll

submitDirectEditTask(Canvas, Track) → taskId
wait 90 seconds
poll getVideoEditTaskStatus(taskId) every 30 seconds until completed

Output Requirements

MANDATORY: After generating videos, you MUST output ALL video URLs clearly.

Single Video Output

**Generated Video**:
![Video](url)

Video URL: url

Multiple Videos Output (Batch/Storyboard)

All N video segments completed!

| # | Status | Preview | URL |
|---|--------|---------|-----|
| 1 | ✅ | ![Segment 1](url1) | url1 |
| 2 | ✅ | ![Segment 2](url2) | url2 |
| ... | ... | ... | ... |

**All Video URLs**:
1. url1
2. url2
...

After Concatenation

**Final Video** (concatenated from N segments):
![Final Video](final_url)

Final Video URL: final_url

IMPORTANT: Never end a video generation task without explicitly listing all video URLs.