Watch TV shows and movies with the user in real time by processing screen captures into frames and audio analysis.
Real-time TV/movie watching pipeline. Chunks arrive automatically via signal files — no polling, no timer, no manual triggering.
capture-live.sh in their terminal with the current conversation ID[WATCH-CHUNK] message with key frames attached as imagesThis is the core loop. When a message arrives starting with [WATCH-CHUNK]:
file_read on additional frames from the processed dir if you want more detail.When the user says they want to watch something:
bash ~/.vellum/workspace/watch-together/scripts/capture-live.sh \
~/.vellum/workspace/watch-together/sessions/<session-id> \
<conversation_id> \
30
The conversation ID is the bare UUID from the conversation's DB record (e.g. 191a7dcc-3e4d-4825-a5b6-97876525f56c), NOT the full folder name with the timestamp prefix. Using the folder name will create a new conversation instead of routing to the existing one.
Initialize a watch session and provide the user the capture command.
Parameters:
show_name (string, required) — Name of the showseason (number, required) — Season numberepisode (number, required) — Episode numberImplementation: Create session dir, copy episode state template, output the capture command for the user to run.
SESSION_ID=$(echo "${show_name}" | tr ' ' '-' | tr '[:upper:]' '[:lower:]')-s${season}e${episode}
SESSION_DIR="$HOME/.vellum/workspace/watch-together/sessions/$SESSION_ID"
mkdir -p "$SESSION_DIR/chunks" "$SESSION_DIR/processed"
cp "$HOME/.vellum/workspace/watch-together/episode-state-template.md" "$SESSION_DIR/episode-state.md"
echo "$SESSION_DIR"
Process a single video file through the frame extraction pipeline.
Parameters:
chunk_path (string, required) — Path to the .mp4 fileoutput_dir (string, required) — Directory to write extracted frames and audiobash "$HOME/.vellum/workspace/watch-together/scripts/process-chunk.sh" "$chunk_path" "$output_dir"
Dense 720p frame extraction for a specific time range. Use when something catches your eye.
Parameters:
chunk_path (string, required) — Path to the original .mp4 chunkoutput_dir (string, required) — Where to save the dense framesstart_time (number, required) — Start time in secondsend_time (number, required) — End time in secondsbash "$HOME/.vellum/workspace/watch-together/scripts/process-chunk.sh" "$chunk_path" "$output_dir" --rewind "$start_time" "$end_time"
GEMINI_API_KEY — Required for audio analysis. Without it, chunks arrive with no audio description and the assistant has no audio context (the user provides verbal descriptions instead). Set it in the shell before running capture-live.sh.GEMINI_MODEL — Optional, defaults to gemini-3-flash-preview.~/.vellum/workspace/watch-together/scripts/~/.vellum/workspace/watch-together/sessions/~/.vellum/workspace/watch-together/episode-state-template.md~/.vellum/workspace/signals/user-message.<requestId> (supports attachments array with {path, filename, mimeType} for inline images)