Video Understanding

Example:

python nanobot/skills/video-understanding/scripts/extract_keyframes.py \
  /absolute/path/to/input.mp4 \
  --output-dir artifacts/video-understanding/input-frames

Use this script as the default entry point. Only drop to custom OCR or image-processing experiments when the extractor clearly failed and you have a specific reason.

The script writes:

• manifest.json with source metadata, timestamps, and reasons
• semantic-storyboard.txt as the main text artifact for smaller models
• semantic-storyboard.md as the richer semantic beat map
• storyboard.txt as the compact text-only version for smaller models
• storyboard.md with a scan-friendly index plus OCR snippets when available
• frames/*.jpg with timestamped key frames

Workflow

1. Reuse cached understanding when safe

Prefer reusing an existing storyboard when the source path, file size, mtime, and extraction parameters still match. The extractor handles this automatically unless --force is used.

This keeps the frame directory stable so later editing passes can refer back to the same visual evidence.

2. Extract frames that matter

Use the bundled extractor instead of blindly sampling every second.

The extractor combines:

• scene-change detection for moments where the visible state changes
• gap-filling frames so long quiet stretches still have checkpoints
• OCR text extraction when tesseract is available, so readable UI text appears in the storyboard
• semantic summarization so the agent gets "what is happening" as text, not just OCR fragments
• timestamped filenames and a manifest so you can reason about pacing

Default output location should usually be:

artifacts/video-understanding/<video-stem>-<short-hash>/

3. Inspect before concluding

After extraction:

• read semantic-storyboard.txt first for the semantic map of what is happening
• read storyboard.txt for the OCR-backed raw frame map
• read storyboard.md when you want frame-by-frame detail
• inspect the actual frame files for proof, loading, warnings, readable result states, and CTA screens
• avoid claiming detailed video understanding from filenames alone

4. Turn frames into an edit beat map

In your reasoning, map:

• timestamp or timestamp range
• visible state or screen
• important readable text
• beat type: hook, setup, waiting, proof, payoff, cta
• likely action: keep, speed up, trim, hold longer

This beat map should drive editing decisions.

Editing Rules

• Understand the source before making edit recommendations.
• Prefer the semantic storyboard generated by the bundled extractor over ad-hoc OCR checks.
• If the user gives you a half-finished project plus the source video, inspect both; do not trust the timeline alone.
• If only the project exists and the source video is missing, say so clearly and use timeline previews as the best available evidence.
• If the extractor fails or FFmpeg is unavailable, state that limitation explicitly and avoid pretending you visually verified the video.

Useful Flags

• --output-dir <path> to control the cache location
• --scene-threshold <float> to be more or less sensitive to visual changes
• --max-gap-seconds <float> to force additional checkpoints across quiet stretches
• --min-spacing-seconds <float> to avoid near-duplicate frames
• --max-frames <int> to cap the storyboard size
• --no-ocr to skip text extraction when you only want visual frames
• --no-semantic to skip semantic summarization
• --semantic-model <name> to choose the semantic model
• --semantic-max-frames <int> to cap the number of frames used for semantic summarization
• --force to rebuild a storyboard even if the cache still matches

References

• frame-selection.md