Quality Metrics Worker

Phase 2: Evaluation Pipeline

4. Create evaluation harness

   • Generate videos with FP16 baseline
   • Generate videos with quantization
   • Compute metrics for both
   • Compare and report

5. Prepare test datasets

   • Text prompts for evaluation
   • Reference videos if available
   • Standardized generation parameters

Phase 3: Validation

6. Verify metric preservation

   • Run evaluation on test set
   • Verify ≥ 99% metric preservation
   • Document any degradation

7. Statistical analysis

   • Compute confidence intervals
   • Test statistical significance
   • Generate comparison plots

Example Handoff

{
  "salientSummary": "Implemented FID, CLIPSIM, and temporal consistency metrics. Evaluated Wan2.1-1.3B with W4A4 quantization: FID 99.1% preserved, CLIPSIM 99.3%, temporal 98.7%. All metrics exceed 99% target.",
  "whatWasImplemented": "Video quality metrics suite with FID using InceptionV3 features, CLIPSIM with CLIP ViT-L/14, temporal consistency via frame difference and optical flow. Benchmark script comparing FP16 vs W4A4.",
  "whatWasLeftUndone": "Large-scale evaluation on VBench dataset not completed (time constraints).",
  "verification": {
    "commandsRun": [
      {"command": "python -m pytest tests/test_metrics.py -v", "exitCode": 0, "observation": "All metric tests pass"},
      {"command": "python scripts/evaluate_quality.py --model wan2.1-1.3b --quantized", "exitCode": 0, "observation": "FID: 99.1%, CLIPSIM: 99.3%, Temporal: 98.7%"},
      {"command": "python scripts/compare_quality.py --baseline fp16 --quantized w4a4", "exitCode": 0, "observation": "No statistically significant difference (p > 0.05)"}
    ],
    "interactiveChecks": []
  },
  "tests": {
    "added": [
      {"file": "tests/test_metrics.py", "cases": [
        {"name": "test_fid_computation", "verifies": "VAL-QTY-001"},
        {"name": "test_clipsim_computation", "verifies": "VAL-QTY-002"},
        {"name": "test_temporal_consistency", "verifies": "VAL-QTY-003"}
      ]}
    ]
  },
  "discoveredIssues": []
}

When to Return to Orchestrator

• Metric computation fails or produces invalid results
• Quality degradation exceeds 1% threshold
• Evaluation pipeline too slow for practical use
• Missing dependencies for metric computation