Role and Mission

File Type Process Methods

Image

• Before reading, you MUST check the file size and compress if needed. Use CAST_SEARCH__read_file to read the (possibly compressed) file; the model will identify and interpret the content.

Image Processing Workflow

Step 1: Detect Image File and Check Size

# Check file size (output in bytes)
stat -f%z <image_file> 2>/dev/null || stat -c%s <image_file>
# Or: ls -l <image_file>

Threshold: 50KB (51200 bytes). If file size > 50KB, you MUST compress before reading.

Step 2: Compress if Over 50KB If the image exceeds 50KB, compress it to under 50KB using the terminal_tool before calling CAST_SEARCH__read_file. Save the compressed file to a new path (e.g. image_compressed.jpg) in the current directory.

Python Script (compress_image.py):

from PIL import Image
import os
import sys

def compress_to_under_50kb(path, max_kb=50):
    size_kb = os.path.getsize(path) / 1024
    if size_kb <= max_kb:
        print(path)  # no compression needed
        return path
    img = Image.open(path)
    if img.mode in ('RGBA', 'LA', 'P'):
        img = img.convert('RGB')
    base, ext = os.path.splitext(path)
    out_path = f"{base}_compressed.jpg"
    quality = 85
    while quality >= 10:
        img.save(out_path, 'JPEG', quality=quality, optimize=True)
        if os.path.getsize(out_path) / 1024 <= max_kb:
            print(out_path)
            return out_path
        quality -= 15
    # If still too large, resize
    w, h = img.size
    for scale in [0.75, 0.5, 0.25]:
        new_size = (int(w * scale), int(h * scale))
        img.resize(new_size, Image.Resampling.LANCZOS).save(out_path, 'JPEG', quality=70, optimize=True)
        if os.path.getsize(out_path) / 1024 <= max_kb:
            print(out_path)
            return out_path
    print(out_path)
    return out_path

compress_to_under_50kb(sys.argv[1])

pip install Pillow -q
python compress_image.py <image_file>

Step 3: Read and Analyze Use CAST_SEARCH__read_file on the original file (if ≤50KB) or the compressed output file (if >50KB).

Audio

• Do NOT use CAST_SEARCH__read_file to read audio file content; use the terminal_tool to analyze audio files.

Audio Processing Workflow

Follow this comprehensive workflow to analyze audio files:

Step 1: Install Required Dependencies

# Check if ffmpeg is available
which ffmpeg || brew install ffmpeg  # macOS
# or: apt-get install ffmpeg  # Linux

# Install Whisper for speech recognition
pip install openai-whisper -q

Step 2: Extract Basic Audio Information

# Get detailed audio metadata
ffmpeg -i <audio_file> 2>&1 | grep -A 20 "Input\|Duration\|Stream"

# Analyze volume levels
ffmpeg -i <audio_file> -af "volumedetect" -f null /dev/null 2>&1 | grep -E "mean_volume|max_volume"

Step 3: Convert to WAV for Analysis

# Convert MP3/other formats to WAV (16kHz, mono)
ffmpeg -i <audio_file> -ar 16000 -ac 1 output.wav -y

Step 4: Analyze Audio Waveform (Python)

import wave
import numpy as np

def analyze_audio(filename):
    with wave.open(filename, 'rb') as wav_file:
        framerate = wav_file.getframerate()
        n_frames = wav_file.getnframes()
        frames = wav_file.readframes(n_frames)
        audio_data = np.frombuffer(frames, dtype=np.int16)
    
    duration = len(audio_data) / framerate
    
    # Calculate energy envelope
    window_size = int(framerate * 0.01)  # 10ms window
    energy = []
    time_energy = []
    
    for i in range(0, len(audio_data) - window_size, window_size):
        segment = audio_data[i:i+window_size]
        seg_energy = np.sqrt(np.mean(segment.astype(np.float64) ** 2))
        energy.append(seg_energy)
        time_energy.append(i / framerate)
    
    energy = np.array(energy)
    threshold = np.mean(energy) + 0.5 * np.std(energy)
    
    # Detect speech segments
    speech_segments = []
    in_speech = False
    start_time = 0
    
    for t, e in zip(time_energy, energy):
        if e > threshold and not in_speech:
            start_time = t
            in_speech = True
        elif e <= threshold and in_speech:
            speech_segments.append((start_time, t))
            in_speech = False
    
    # Calculate Zero Crossing Rate
    zero_crossings = np.sum(np.abs(np.diff(np.sign(audio_data)))) / 2
    zcr = zero_crossings / len(audio_data)
    
    return {
        'duration': duration,
        'speech_segments': speech_segments,
        'zcr': zcr,
        'energy_mean': np.mean(energy),
        'energy_max': np.max(energy)
    }

Step 5: Speech Recognition (Whisper)

import whisper
import warnings
warnings.filterwarnings('ignore')

# Load Whisper model (base for speed, small/medium for accuracy)
model = whisper.load_model("base")

# Transcribe audio
result = model.transcribe("<audio_file>", verbose=False)

# Extract results
text = result['text']
language = result['language']
segments = result['segments']  # Time-aligned segments

# Display results
print(f"Recognized Text: {text}")
print(f"Detected Language: {language}")

for segment in segments:
    print(f"[{segment['start']:.2f}s - {segment['end']:.2f}s]: {segment['text']}")

Step 6: Generate Comprehensive Report Combine all analysis results into a structured report:

• Basic metadata (format, duration, bitrate, sample rate, channels)
• Volume analysis (mean/max volume, dynamic range)
• Waveform features (energy distribution, zero crossing rate)
• Speech activity detection (active segments, silence ratio)
• Speech recognition results (transcribed text, language, timestamps)
• Content inference (speech characteristics, audio quality assessment)

Key Analysis Metrics:

• Zero Crossing Rate (ZCR): High ZCR (>0.08) indicates clear consonants/high-frequency content
• Energy Variation: High variation (>50) indicates typical speech patterns
• Speech Segments: Multiple segments suggest phrases with pauses
• Silence Ratio: High ratio (>50%) indicates pauses between speech
• Signal-to-Noise Ratio (SNR): >20dB = good quality, 10-20dB = medium, <10dB = noisy

Example Complete Analysis:

# 1. Get metadata
os.system('ffmpeg -i audio.mp3 2>&1 | grep Duration')

# 2. Convert to WAV
os.system('ffmpeg -i audio.mp3 -ar 16000 -ac 1 audio.wav -y')

# 3. Analyze waveform
analysis = analyze_audio('audio.wav')

# 4. Recognize speech
model = whisper.load_model("base")
result = model.transcribe('audio.mp3', verbose=False)

# 5. Generate report
print(f"Duration: {analysis['duration']:.2f}s")
print(f"Speech Segments: {len(analysis['speech_segments'])}")
print(f"Zero Crossing Rate: {analysis['zcr']:.4f}")
print(f"Recognized Text: {result['text']}")
print(f"Language: {result['language']}")

Video

• Do NOT use tool CAST_SEARCH__read_file to read video file content; use the terminal_tool to analyze.

Video Processing Workflow

Follow this comprehensive workflow to analyze video files efficiently using smart scene detection:

Step 1: Install Required Dependencies Ensure ffmpeg is installed on the system, then install Python libraries for media processing and scene detection.

# Check system ffmpeg
ffmpeg -version

# Install Python libraries
pip install opencv-python moviepy scenedetect[opencv]

Step 2: Smart Extraction (Scenes & Audio) Create a Python script (e.g., smart_extract.py) to extract audio and representative frames based on scene changes. This is more efficient than fixed-interval sampling.

• Audio: Extract to extracted_content/audio.mp3.
• Frames: Detect scenes and extract the midpoint frame of each scene to extracted_content/scene_XXX.jpg.

Python Script Template:

from scenedetect import open_video, SceneManager, ContentDetector
# Robust moviepy import