Handtracking

Choose this when: You need fast, lightweight hand detection with bounding box outputs and don't require detailed joint-level pose estimation.

Consider alternatives: If you need 3D hand pose keypoints, hand-object segmentation, or multi-view tracking, see other skills in this category.

Core Capabilities

1. Hand Detection in Egocentric Views

EgoHands-trained model: Specifically optimized for first-person perspective videos where hands are viewed from the wearer's viewpoint.

• Input: Video files (MP4, AVI) or webcam streams
• Output: Bounding boxes (x, y, width, height) with confidence scores
• Model: Lightweight TensorFlow model trained on EgoHands dataset
• Performance: Real-time processing on standard hardware
• Detection types: Left hand, right hand classification
• Visualization: Red/green bounding boxes overlaid on video

Bounding box format:

{
    'bbox': [x, y, width, height],  # Pixel coordinates
    'score': confidence,              # 0.0 to 1.0
    'label': 'hand'                   # Detection label
}

2. Video Processing and Annotation

Input video processing: Process entire video files and export annotated results.

Workflow:

# Clone repository
git clone https://github.com/victordibia/handtracking.git
cd handtracking

# Install dependencies (TensorFlow 1.x compatible)
pip install tensorflow==1.15.0 opencv-python numpy

# Run hand detection on video
python run.py \
    --input_video your_egocentric.mp4 \
    --output_video output_labeled.mp4 \
    --threshold 0.5  # Confidence threshold

Output video features:

• Hands outlined with colored bounding boxes (red/green)
• Real-time frame rate display
• Confidence scores shown on boxes
• Smooth tracking across frames -保存带标注的视频文件

3. Real-time Webcam Detection

Live camera processing: Process webcam streams in real-time for interactive applications.

import handtracking

# Initialize detector
detector = handtracking.HandDetector()

# Process webcam stream
detector.detect_from_webcam(
    display=True,
    save_video=False,
    confidence_threshold=0.6
)

Applications:

• Live hand gesture interfaces
• Interactive installations
• Real-time hand presence detection
• Gesture-controlled systems

4. Browser-based Detection (Handtrack.js)

JavaScript companion library: Use the same model technology in web applications.

Integration:

<script src="https://cdn.jsdelivr.net/npm/handtrackjs/dist/handtrack.min.js"></script>

<script>
const model = await handTrack.load();
const video = document.getElementById('video');

// Detect hands in video stream
const predictions = await model.detect(video);
predictions.forEach(prediction => {
    console.log(prediction.bbox);  // [x, y, width, height]
    console.log(prediction.score); // Confidence score
});
</script>

Browser capabilities:

• Run entirely in browser (no server required)
• Real-time webcam processing
• Canvas-based visualization
• WebGL acceleration support
• Works with video files and live streams

Installation and Setup

Option 1: Python Installation

# Clone repository
git clone https://github.com/victordibia/handtracking.git
cd handtracking

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install tensorflow==1.15.0
pip install opencv-python numpy pillow

# Download pre-trained model
# Model will be automatically downloaded on first run

Model files: Automatically downloaded from the repository on first use (~20MB).

Option 2: JavaScript Installation (Handtrack.js)

# For web applications
npm install handtrackjs

# Or use directly from CDN

Usage Examples

Example 1: Process Video with Output

import cv2
from handtracking import HandDetector

# Initialize detector
detector = HandDetector()

# Load video
cap = cv2.VideoCapture('egocentric_video.mp4')

# Get video properties
fps = int(cap.get(cv2.CAP_PROP_FPS))
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

# Setup video writer
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
out = cv2.VideoWriter('output_labeled.mp4', fourcc, fps, (width, height))

# Process frames
while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break

    # Detect hands
    detections = detector.detect_hands(frame)

    # Draw bounding boxes
    for det in detections:
        x, y, w, h = det['bbox']
        score = det['score']

        # Draw box
        color = (0, 255, 0) if score > 0.7 else (0, 0, 255)
        cv2.rectangle(frame, (x, y), (x+w, y+h), color, 2)

        # Add label
        label = f"Hand: {score:.2f}"
        cv2.putText(frame, label, (x, y-10),
                   cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 2)

    # Save frame
    out.write(frame)

cap.release()
out.release()

Example 2: Extract Hand Regions

import cv2
import numpy as np
from handtracking import HandDetector

detector = HandDetector()
cap = cv2.VideoCapture('egocentric.mp4')

frame_count = 0
while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break

    detections = detector.detect_hands(frame)

    # Extract and save hand regions
    for i, det in enumerate(detections):
        x, y, w, h = det['bbox']

        # Crop hand region
        hand_roi = frame[y:y+h, x:x+w]

        # Save hand image
        if det['score'] > 0.7:  # High confidence only
            cv2.imwrite(f'hand_{frame_count}_{i}.jpg', hand_roi)

    frame_count += 1

cap.release()

Example 3: Real-time Detection Statistics

from handtracking import HandDetector
import cv2

detector = HandDetector()
cap = cv2.VideoCapture(0)  # Webcam

while True:
    ret, frame = cap.read()
    if not ret:
        break

    detections = detector.detect_hands(frame)

    # Display statistics
    num_hands = len(detections)
    avg_confidence = sum(d['score'] for d in detections) / num_hands if num_hands > 0 else 0

    # Overlay text
    cv2.putText(frame, f"Hands: {num_hands}", (10, 30),
               cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)
    cv2.putText(frame, f"Avg Conf: {avg_confidence:.2f}", (10, 70),
               cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)

    cv2.imshow('Hand Tracking', frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Example 4: Browser-based Detection

// Load model
const modelParams = {
    flipHorizontal: true,
    maxNumBoxes: 2,
    iouThreshold: 0.5,
    scoreThreshold: 0.6,
};

handTrack.load(modelParams).then(model => {
    // Model loaded
    console.log("Model loaded");

    // Detect from video element
    const video = document.getElementById('video');
    const canvas = document.getElementById('canvas');
    const context = canvas.getContext('2d');

    function detectFrame() {
        model.detect(video).then(predictions => {
            // Clear canvas
            context.clearRect(0, 0, canvas.width, canvas.height);

            // Draw video frame
            context.drawImage(video, 0, 0, canvas.width, canvas.height);

            // Draw predictions
            predictions.forEach(prediction => {
                const [x, y, width, height] = prediction.bbox;
                context.strokeStyle = '#00FF00';
                context.lineWidth = 4;
                context.strokeRect(x, y, width, height);

                // Add label
                context.fillStyle = '#00FF00';
                context.fillText(
                    `Hand: ${prediction.score.toFixed(2)}`,
                    x, y - 10
                );
            });

            // Continue detection
            requestAnimationFrame(detectFrame);
        });
    }

    // Start detection
    detectFrame();
});

Integration with Other Skills

This skill works effectively with:

• MediaPipe skills: For more advanced hand pose estimation and gesture recognition
• OpenCV-based video processing skills: For comprehensive video analysis pipelines
• Machine learning skills: For building custom hand gesture classifiers
• XR/AR framework skills: For integrating hand detection into immersive experiences

Model Specifications

Architecture: Lightweight CNN-based object detection model

• Framework: TensorFlow 1.x (compatible with older TF versions)
• Training dataset: EgoHands (4,800 egocentric images)
• Input resolution: Flexible (recommended: 640x480)
• Model size: ~20MB
• Inference speed: 30+ FPS on standard CPU (depends on resolution)

Detection performance (on EgoHands test set):

• mAP: ~85% (mean Average Precision)
• Recall: ~82%
• Precision: ~88%

Limitations and Considerations

Scope: This skill provides bounding box detection only. For more detailed analysis, consider:

• 3D hand pose estimation: Use ap229997-hands skill for joint keypoints
• Hand-object segmentation: Use owenzlz-egohos skill for pixel-level masks
• Multi-view tracking: Use facebookresearch-hot3d for 3D tracking

Known limitations:

• Trained primarily on egocentric views (first-person perspective)
• May have reduced performance on third-person views
• Bounding boxes only (no joint or finger-level details)
• Requires TensorFlow 1.x (not compatible with TF 2.x without modifications)
• Model trained on diverse hands but may have bias toward certain demographics

When to upgrade:

• Need 3D hand joint positions → Use ap229997-hands
• Need hand-object interaction segmentation → Use owenzlz-egohos
• Need multi-view or high-precision 3D tracking → Use facebookresearch-hot3d
• Need browser-based 3D hand tracking → Consider MediaPipe Hands

Performance Optimization

CPU optimization:

# Reduce input resolution for faster processing
detector = HandDetector()
frame = cv2.resize(frame, (640, 480))  # Downsample
detections = detector.detect_hands(frame)

GPU acceleration (if available):

# TensorFlow with GPU support
import tensorflow as tf
# Install tensorflow-gpu for GPU acceleration

Batch processing:

# Process multiple videos in parallel
from concurrent.futures import ThreadPoolExecutor

def process_video(video_path):
    detector = HandDetector()
    return detector.process_video(video_path)

with ThreadPoolExecutor(max_workers=4) as executor:
    results = executor.map(process_video, video_list)

Troubleshooting

Issue: Model not downloading automatically

• Solution: Manually download model files from the GitHub repository and place in the models directory

Issue: TensorFlow version conflicts

• Solution: Use virtual environment with TensorFlow 1.15.0: pip install tensorflow==1.15.0

Issue: Low detection accuracy

• Solution: Adjust confidence threshold (try 0.5-0.7), ensure video quality is adequate, check that view is egocentric

Issue: Slow processing speed

• Solution: Reduce video resolution, close other applications, consider GPU acceleration

Issue: No hands detected

• Solution: Check if video is in egocentric view, lower confidence threshold, improve lighting conditions

References and Resources

Documentation

• GitHub repository: https://github.com/victordibia/handtracking
• EgoHands dataset paper: https://egohands.github.io
• Handtrack.js documentation: https://github.com/victordibia/handtrack.js

Related Projects

• EgoHands Dataset: https://github.com/egohands/EgoHands
• MediaPipe Hands: https://google.github.io/mediapipe/solutions/hands.html
• OpenPose: https://github.com/CMU-Perceptual-Computing-Lab/openpose

Example Applications

• Gesture-controlled interfaces
• Sign language detection
• Human-computer interaction research
• AR/VR hand tracking
• Activity recognition from egocentric video

Citation

If you use this hand tracking implementation in research, please cite:

@article{betancourt2015egohands,
  title={Egohands: A dataset for egocentric hand interactions},
  author={Betancourt, Alex and Orozco, Jorge and Bolaños, Mauricio},
  journal={arXiv preprint arXiv:1509.06044},
  year={2015}
}

And the original repository:

@software{victordibia_handtracking,
  author = {Victor Dibia},
  title = {Real-time Hand Detection in Python using TensorFlow},
  url = {https://github.com/victordibia/handtracking},
  year = {2018}
}

Best Practices

1. Start with this skill for quick prototyping and proof-of-concept
2. Validate on your data before committing to production use
3. Consider GPU acceleration for real-time applications
4. Test confidence thresholds on your specific use case
5. Use appropriate resolution - balance between speed and accuracy
6. Handle edge cases - no hands, occluded hands, multiple hands
7. Post-process results - smoothing, filtering, temporal consistency
8. Benchmark alternatives before final implementation

Future Enhancements

Consider exploring these related directions:

• Fine-tune model on domain-specific egocentric data
• Integrate with gesture recognition pipelines
• Combine with object detection for hand-object interactions
• Extend to browser applications using Handtrack.js
• Upgrade to 3D pose estimation for more detailed analysis