Model Evaluation

Procedure

Phase 1: Define Evaluation Scope

1. Clarify the decision — What question is the evaluation answering?

   • Model selection (A vs B): use comparative benchmarks
   • Release gate (is this good enough?): use threshold-based metrics
   • Regression check (did fine-tuning break anything?): use baseline comparison
   • Capability reporting (model card): use canonical benchmarks + task-specific

2. Map capabilities to metrics:

3. Check for contamination — Verify the training data pipeline was not contaminated by benchmark test sets before trusting absolute scores.

Phase 2: Set Up Evaluation Harness

Standard harnesses:

# lm-evaluation-harness (most common for LLMs)
pip install lm-eval
lm_eval --model hf --model_args pretrained=<model_path> \
        --tasks mmlu,hellaswag,truthfulqa_mc2 \
        --device cuda:0 --batch_size 8

# HELM
pip install crfm-helm
helm-run --conf-path run_specs.conf --suite my_eval

# Inspect (for agentic/tool-use evaluation)
pip install inspect-ai
inspect eval task_file.py --model openai/gpt-4o

Custom task setup:

from lm_eval.api.task import ConfigurableTask
from lm_eval.api.instance import Instance

class MyTask(ConfigurableTask):
    DATASET_PATH = "my_org/my_eval_dataset"
    OUTPUT_TYPE = "generate_until"

    def doc_to_text(self, doc):
        return doc["prompt"]

    def process_results(self, doc, results):
        prediction = results[0].strip()
        return {"exact_match": prediction == doc["answer"]}

Phase 3: Run and Record Results

1. Pin the harness version — Benchmark scores are not comparable across harness versions
2. Record metadata per run:
   • Model checkpoint path + git ref
   • Harness version
   • Evaluation date
   • Hardware (GPU model, VRAM)
   • Batch size, dtype, quantization method
3. Store results in structured format (JSON) next to the checkpoint in the model registry
4. Never compare quantized scores to full-precision scores without explicit documentation

Phase 4: Interpret Results

Score comparison checklist:

• Same harness version as the baseline?
• Same few-shot count as the baseline?
• Same quantization method (FP16 vs INT8 vs GGUF)?
• Contamination risk assessed?
• Results within expected variance (re-run 3x if p-value matters)?

Decision thresholds:

Phase 5: Report Results

For model cards and team communication, produce:

## Evaluation Summary — <model-name> v<version>

| Benchmark | Score | Baseline | Delta | Notes |
| --- | --- | --- | --- | --- |
| MMLU (5-shot) | 0.72 | 0.70 | +2% | No contamination |
| HumanEval (0-shot) | 0.45 | 0.42 | +3% | Pass@1 |

**Harness**: lm-evaluation-harness v0.4.3
**Date**: YYYY-MM-DD
**Hardware**: 1× A100 80GB, bf16
**Known limitations**: Not evaluated on multimodal or audio tasks.

Anti-Patterns

Agent Integration